Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features
Authors: Benyuan Meng
Qianqian Xu
Zitai Wang
Xiaochun Cao
Qingming Huang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Diffusion Models
Representation Learning
Model Property Study
Primary area: diffusion_based_models
TL;DR: We evaluate and exploit previously-ignored activations from diffusion backbones based on our new discoveries of their properties, thus resulting in better features for discrimination.

Abstract: Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at https://github.com/Darkbblue/generic-diffusion-feature.

Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features
Benyuan Meng" 1.2
Qianqian Xu Zitai Wang?
Xiaochun Cao
Qingming Huang 6,3,7
Institute of Information Engineering, CAS
Schooo of Cyber Security, University of Chinese Academy of Sciences
3 Key Lab. of Intelligent Information Processing, Institute of Computing Technology, CAS
*Peng Cheng Laboratory
School of Cyber Science and Tech., Shenzhen Campus of Sun Yat-sen University "Scoool of Computer Science and Tech., University of Chinese Academy of Sciences KKey Laboratory of Big Data Mining and Knowledge Management, CAS mengbenyuanliie ac en (xuqianqian, wwnggitaa) ac. en caoxiaochunOmail, sysu. edu. en qmhuangOucas ac. cn
Abstract
Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting small yet effective subset poses fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advance- ments in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes further step with a much broader range of activations evaluated. Considering the significant increase activations, full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activa- tions that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at this url.
Introduction
Diffusion models [21, 10. 37, 36] are powerful generative models that progressively reconstruct
images from Gaussian noises through series of denoising steps. Typically, a U-Net [38] is trained as the noise predictor backbone perform denoising. Recently, the impressive generative capability inspires the application to discriminative tasks such semantic segmentation [52, 58] or semantic correspondence [29, 56]. In this direction, diffusiio feature is one simple yet effective approach, where the intermediate signals, named activations, are extracted from the pre-trained diffusion U-Net as dense features [2. 58, 56, 29, 53, 13, 14].
CCoresponning authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Diffusion Activations
Inter-Module
VIT Basic Block Output
VIT Attention Activations
We Consider
Previous Studies Consider
Property Analysis & Qualitative Filtering
Quantitative Comparison
Quantitative Comparison
SDv1.5: 5: 66.7 PCK@0.11
SDv1.5: 69.8 PCK@0.11
SDXL: 59.2 PCK@0.11
SDXL: 75.2 PCK@0.11
Figure I: Prior arts only consider a small fraction of potential activations in diffusion models. As a result, more advanced diffusion architecture fails to achieve better performance (SDXL SDv1.5). In contrast, we consider a broader range of candidate activations. To facilitate the quantitative comparison, first make comprehensive and generalizable analysis qualitatively filter out many candidates in advance. On top of this, our method achieves superior performance (75.2 PCK@0.1)
The complex architecture of the diffusion U-Net provides many activations that can serve as features. However, these activations inherently vary quality, inducing significant performance gaps on discrimination. Hence, selecting a small yet effective subset from these activations has become fundamental problem. In the early stage, Baranchuk er. al [2] perform a large-scale quantitative comparison among activations within Guided Diffusion [10]. Later, the activations they select are followed by most studies this field, pursuing other improvements [29, 56, 58, 52, 60, 53, 24, 27].
However. we find that this fundamental issue is far from solved. On one hand, Baranchuk et. al [2] only consider activations between neighboring modules that comprise the main residuals. This means that many potential activations are excluded from the candidate pool, such as the queries and keys in the self-attention blocks. Moreover, recent developments of diffusion architecture, such as gross-attention [37] or embedded deep vision transformers (ViT's [36], have introduced additional types of activations. Hence, shown in upper Figure only a small fraction of potential activations in modern diffusion models have been evaluated for their discriminative ability. which might hinder future work in this direction. For example, lower Figure shows that Stable Diffusion XL (SDXL) [36], which is more advanced than Stable Diffusion v1.5 (SDv1.5) [37], fails to achieve better performance with the feature selection solution proposed in [2].
In this paper, we revisit the fundamental problem of feature selection, considering a more comprehen- sive candidate pool of activations. Due its large volume, a full-scale quantitative comparison is no longer operational, urging us to modify the previous research methodology in [2]. As illustrated Figure instead of a direct quantitative comparison, we first explore the properties of diffusion U-Nets. These properties allow us to qualitatively and efficiently filter out many activations that are highly likely to sub-optimal, shrinking the candidate pool for the quantitative comparison. More importantly, we find these properties are universal among diffusion models, making it possible to generalize our findings to more models beyond those covered this paper.
Specifically, the properties we find, which are distinct to existing knowledge of model properties [1. 38], exactly correspond three top-to-bottom levels of the diffusion U-Net: Diffusson noises at the macro level: the diffusion process induces a new type of noise on both low- and high-frequency signals. (ii) In-resolution granularity changes the in-resolution level: the changes in information granularity are not only across but also within resolutions. (iii) Locality without positional embeddings at the sub-module level: the embedded ViTs in diffusion U-Nets present a new type of local information different from that induced by the conventional positional embeddings [44]. Based on these insights, we develop effective feature selection solutions for several popular diffusion models. Finally, the experiments on three discriminative tasks, including semantic correspondence, semantic segmentation, and label-scarce segmentation, validate the superiority of our solutions over the SOTA methods.

---

In summary, the contributions of this work are three-fold:
Revisit of a fundamental problem: the fundamental issue of feature selection remains unresolved in the realm of diffusion feature.
To the best of our knowledge, we are the first to point out that
Generie insights: The properties we find are universal among different diffusion U-Nets, which can provide valuable insights for future work.
Extensive validation: Extensive experiments on three discriminative tasks validate the effective- ness the feature selection solutions induced by our insights.
2 Related Work
Diffusion models for discrimination. We discuss three mainstream ways to diffusion models for discrimination. (i) Diffusion classifier [26. 7] utilizes Bayes' theorem to transform a pre-trained diffusion model into image classifier. This method enjoys theoretical guarantee and does not need additional training. However, it is limited to image-level tasks. (ii) The second way is to model discriminative tasks image-to-image generation tasks with diffusion models [22, 23). This method is suitable for various dense vision tasks but requires heavy training. (iii) Diffusion feature [2, 58, 56, 41, 29], the focus of this paper, follows the traditional practice of feature extraction to pursue the balance between wider applicability and less training. This makes it adaptable for different tasks and alleviates the training needs for the diffusion model. Only small downstream models may require training.
The diffusion feature approach has seen various improvements. Some techniques toggle the input settings of diffusion models, such as seeking better timesteps [60, 29]. The others add trainable parameters outside the diffusion model to refine the outputs provide efficient fine-tuning alter- native [58, 27, 46]. Additionally, some studies explore completely training-fres methods that utilize spatial attention information [51, 43), and some attempts focus on text-free diffusion models [32, 33).
Despite the progress, previous methods only consider activations between neighboring modules, leaving many potential candidates unevaluated. Our work shows that such an overlook can hinder model performance. By introducing a more comprehensive feature selection solution, our method could generically enhance both existing and future diffusion feature approaches.
Analysis on model properties. The inner properties of neural networks have consistently received much attention [5, 34]. For transformers, Geva et al. [15] show that the feed-forward layers act as key-value memories and are interpretable for humans. Amir al. [1] extend this insight to vision transformers and put it into practical applications such as image classification, while Vilas et, al. [45] try to make more detailed interpretation of ViT activations.
The methodology of these studies provides valuable guidance for our research. However, since diffusion models are trained for generation, our study relies more on qualitative analysis and feature visualization compared to previous work.
3 Preliminaries: Architecture of Diffusion U-Nets
Diffusion models typically involve a forward pass and reverse pass [21]. During the forward pass, noise is gradually added to clean image 2n E Rxxwxh until the image resembles Gaussian noise, where w and h denote the width and height, respectively. This process can be denoted by a ~ q(x, mn), where q represents the noise posterior and is the timestep. In the reverse pass, an end-to-end neural network Eg, as parameterized by 8, learns to predict the noises and thus reconstruct the image. One such denoising step can be denoted as noise, and c the condition that describes the expected image content. Although there are alternative formulations for diffusion models [39, 28, 40], they all rely on this neural network backbone eg. This study focuses on this backbone, typically implemented as U-Net [38], rather than other components of diffusion models. Next, we will detail the architecture of the diffusion U-Net and standardize the terminology referring to different parts of the model, as shown in Figure 2.
c0(x,, t, c). where is the predicted
We start with an overview. Following the initial U-Net design [38], the U-Net has three main stages: down-stage, mid-stage, and up-stage. The down-stage reduces the resolution of activations, while the ut-stage increases it. Both stages contain multiple resolutions, in each of which the activations share
3

---

Down-Stage
Mid-Stage
Up-Stage
In/Out Conv
Res Module
ViT Module
Down/Up Sampler
Resolution
Modules
Self-Attentic
Basic Block
Alt-Atttenioo
Feed-Forward
Figure 2: U-Net architecture (upper) and the VIT module (lower), taking SDXL as example.
the same resolution. Furthermore, each resolution includes several modules, including ResModule (convolutional ResNet [20] structures), Module [12], and Dowzsanpler/Upsamper (simple convolutional layers). Previous diffusion feature approaches only consider activations between these adjacent modules, i.e., inter-module activations.
We next dive into the details below the module level. Among these modules, ResModule and ViTs adopt residual connection [20], where an increment activation is added element-wiie wise to the residual activation refine Specifically, ResModule uses simple convolutional layers produce increments, whereas ViTs use complex attention mechanisms, which will be further explained next. Typically, ViT operates as a standalone model followed by decoder that produces the output predictions for visual tasks [12, 25, 9, 18]. However, in the diffusion U-Net, multiple VITs are embedded into the network, and their outputs serve as increment activations.
Furthermore, each ViT module consists of several stacked basic blocks. A basic block typically has self-attention layer to perform attention on the image itself and feed-forward layer essentially
two-layer MIP [15]. Modern diffusion U-Net introduces additional cross-attention layer between the layers, enabling the fusion of the image and additional textual prompts. Besides, each layer includes a residual connection, meaning that the increment activation added to the outer residual also the internal residual.
In nutshell, the architecture described above provides abundant activations that can serve as dense features. Given the massive activations, it is no longer operational to perform full-scale quantitative comparison. Hence, we next make comprehensive analysis of model properties to better understand these activations, which can help the qualitative filtering.
Distinct Properties of Diffusion U-Nets
The diffusion U-Net has many interesting properties, but now we only focus on those distinct from the knowledge of traditional U-Net [38] ViTs [44]. shown in Figure this section highlights three noticeable properties, each of which corresponds to a different level of the diffusion U-Net architecture described Section 3. Notably, these properties can be universally observed in different samples and diffusion models, though all visualization in the main content is conducted on the same image and SDXL model for consistency. Additional visualization provided Appendix Besides, the omitted properties are available in Appendix E.

---

(b, In-Resolution Level) There are in-resolution granularity changes
Previous Knowledge of Overall U-Net Granularity
(c, Sub-Module Level) Without positional embeddings, there is still locality in self-attention query & key
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency, but its change down/up-stage asymmetric
Noisy
Still Clean
Down-Stage
Mid-Stage
Up-Stage
Figure 3: We highlight three properties of diffusion U-Nets that distinct from existing knowledge about other models: (a) Asymmetric diffusion noises. (b) In-resolution granularity changes. (c) Locality without positional embeddings: pixels within the orange circle resemble nearby background pixels more than distant pixels on the horse's 's neck that are semantically closer.
4.1 Asymmetric Diffusion Noises
The first property is at the macro level and closely related the overall diffusion process. It is common that high-frequency signals are typically noisy [5, 34]. This phenomenon can also be observed in diffusion U-Nets, especially within the increment activations of residual connection. However, this does not mean that low-frequency signal activations in diffusion U-Nets are free from noises. As shown in Figure 3(a), the diffusion process introduces low-frequency signals. This is not surprising since the diffusion process requires the backbone to process noisy inputs and predict noises outputs. As result, activations near the inputs outputs, regardless of their frequency, also suffer from such noises. Considering the special cause, we name such noises diffusion noises.
type of noise that also impacts
How does the influence diffusion noises spread across diffusion U-Nets? As shown in Figure 3(a), diffusion noises exist throughout the entire down-stage, with decreasing magnitude. Remarkably, during the early half of the up-stage, activations are rather clean, with no perceivable diffusion noises. Only in the later half do diffusion noises start resurface. The existence of this asymmetrie behavior can also be indirectly supported by the ablation curves in many fellow studies such as [2, 32, 33). Furthermore, shown in Appendix A, such asymmetric pattern consistent across activations in different diffusion models.
Similar to common high-frequency noises, diffusion noises can degenerate feature quality. Hence, this property can serve as a criterion for identifying and filtering out sub-optimal activations, which will delve into in Section 5.
4.2 En-Reeslution Granularity Changes
The second property is at the in-resolution level and closely related to recent advances in diffusion architectures. Specifically, the design U-Net follows the idea of resolution hierarchy [38]. Conse- quently, the overall architecture displays fine-coarse-fine granularity trend, looking like the alphabet "U". Traditional U-Nets implement this architecture with a relatively large number of resolutions, while each resolution is typically small, equipped with two three simple convolutional layers.
5

---

Hence, the understanding of traditional U-Net focuses on the granularity changes across resolutions, implicitly assuming that the change within a single resolution negligible.
However, diffusion U-Nets become much "fatter". In other words, modern diffusion U-Nets typically have much fewer resolutions, but each resolution is significantly enlarged. For example, SDv11.5 only has four resolutions [37], and SDXL further decreases this number to three [36], as shown in Figure 2. Meanwhile, each resolution can produce much more activations, primarily thanks the embedded ViT modules. This architectural evolution makes the granularity change within a single resolution more significant, as depicted in Figure 3(b).
Different granularity carries varied information and quality, resulting in different discriminative performance downstream tasks [2, 29]. Hence, this discovery of in-resolution granularity changes highlights the necessity to evaluate more activations, especially those within the embedded ViT', as they are of different granularity.
4.3 Locality without Positional Embeddings
For the third distinet property, we delve into the sub-module level, ie., the blocks in embedded ViT modules. Positional embeddings, which are widely used in language transformers [44] and ViTs [12], aim to provide spatial information for each input token. Consequently, the activations of traditional ViTs display strong positional information, where the latent pixel resembles nearby pixels more than
those that are semantically similar but far away [1]. This phenomenon is significant for the layers close to the inputs. When the layer goes deeper, the tokens are refined with semantic information, making the activations display less positional information. However, only in the last few layers, such positional information becomes negligible.
In contrast, ViT modules in diffusion U-Nets do not use positional embeddings [37, 36], perhaps because the other U-Net components have provided sufficient spatial cues. This change results in distinct properties of the activations. On one hand, positional information is negligible for most activations despite how near they are to the imputs. For example, even in the first basic block, cross- attention query activations contain no perceivable positional information. On the other hand, the queries and keys of sell-attention still display non-negligible positional information, marked with orange circles in Figure 3(c). Specifically, the latent pixel on the horse's 's neck is light blue color, similar to the pixels its left that actually represent the background. In contrast, the pixels above the circle are in purple color, though they also represent the horse's 's neck. Such comparison shows that a latent pixel is more similar to other pixels that are spatially near it than those semantically closer to
We name this phenomenon locality since it has a different mechanism from that induced by positional embeddings. As pointed out in [12], self-attention allows ViT integrate global and local information even in the shallow layers, and the attention scope enlarges w.r.f. layer depth. Even without positional embeddings, self-attention activations are generally consistent with this insight, leading to the existence of locality. Nevertheless, the magnitude has indeed greatly reduced, compared to the visualization of conventional ViT activations locality exists but is inherently weaker, as much semantic information is still reserved. In addition, locality quickly diminishes as the layer goes deeper.
[1]. As shown Figure 3()), in shallow activations,
Since positional information can degrade the quality of activations [1], its absence has the potential to enhance the activations in ViT modules. Moreover, locality can play a special role in activation filtering, as presented in Section 5.
4.4 Universality of Three Properties
Although the visualization in Figure 3 is conducted only on SDXL, the scope of these properties is not limited to the specific architecture. Further supporting evidence is available in Appendix A.
Diffusion noises directly arise from the diffusion process. Hence, it is promising to extend this property to other diffusion backbones.
(ii) In-resolution granularity changes come from the "fatness" of U-Nets, making it potentially applicable to more traditional U-Nets.
(iii Locality originates from the self-attention mechanism in ViT architectures, so broadly applicable to standalone or embedded ViTs where positional embeddings are absent.

---

(a)
Resolution#O: Diffusion Noises Are Negligible
(b)
Block#0
Block#5
Block#9
Self-Atten.
Resolution#1: Diffusion Noises in Later Half
PCK@0.1
36.79
58.59
61.99
Cross-Atten.
Resolution#0
PCK@0.1
59.56
60
PCK
(c)
Self-Atten.
Other Activations
Visualization
Resolution
PCK@0.1
41.41
34,58
23.10
Figure 4: (a) Diffusion noises result in significant performance degeneration (Resolution#1), (b) Locality degrades the quality of self-attention activations (Block#O and Block#5). (c) Locality in self-attention activations can suppress diffusion noises, leading better quality than noisy activations (41.41 v.s. 34.58). All ICK@0.limg (1) results are evaluated on the semantic correspondence task.
5 Enhanced Feature Selection from Diffusion U-Nets
So far, we have had more comprehensive understating of the properties of diffusion U-Nets. All these properties, especially in-resolution granularity changes, encourage us to reconsider the feature selection solution, with special emphasis on the activations in ViT modules. With these properties, we are also able to filter out many low-quality activations qualitatively, followed by a thus simplified quantitative comparison.
5.1 Qualitative Filtering
Avoiding Diffusion Noises. As shown in Figure 4(a), diffusion noises tend to degrade the quality of activations. Hence, it is natural to filter out the activations severely affected by diffusion noises from the candidate pool. Specifically, according the asymmetric trend of diffusion noises, we only consider activations in the early half of the up-stage, which are rather clean. This approach will significantly reduce the number of candidate activations and simplify the quantitative comparison.
Avoiding Self-Attention Locality. The locality in self-attention modules another important factor that degrade the activations. The empirical evidence in Figure 4(b) demonstrates that these activations are generally inferior to the others, such as those from cross-attention layers or the outputs of VIT basic blocks. Consequently, it is rather safe filter out most activations in self-attention modules from our candidate pool.
Using Locality to Suppress Diffusion Noises. So far, the activations in the candidate pool are clean and free from locality. However, all these activations are low-resolution ones since high- resolution activations are generally noisy and thus filtered out. This is unfavorable since some detailed information might only exist in high-resolution activations. To address this issue, we exploit side effeet of self-attention locality. Specifically, suppress diffusion noises via its focus on spatial structures. Although locality is sub-optimal, is still superior to severe diffusion noises. In view of this, the candidate pool reserves self-attention activations extracted from the later half of the up-stage.
indicated in Figure 4(c), locality help
We have filtered out many candidate activations based on the distinct properties of diffusion U-Nets. Additionally, all increment activations residual connection can be further filtered out since they introduce high-frequency noises [5, 34]. After such qualitative filtering, small but high-quality candidate pool is available. Taking SDXL. as an example, the number of candidates decreases from 279 to 63, i.c., a 78% reduction. Next, we explain how to conduct this quantitative comparison briefly.

---

Table I: Experimental results on the semantic correspondence task. The best results are in bold font and the runner-up is underlined.
Category
Method
PCK @0.1lmg f
PCK @0. bbox
DINO
51.68
41.04
DHPF
55.28
42.63
SOTA
DIFT DHF
52.90
72.56
64.61
Legaey-v1.5
75.14
66.73
Baseline
Legacy-XL.
66.00
59.16
Ours-v
77.78
69.83
Ours
Ours-Xi XL
81.72
75.18
Ours-XL-t
83.90
76.86
5.2 Quantitative Comparison
The quantitative comparison follows the protocol of [2]. Specifically, given an input image Vo. we first perform the forward pass with pre-defined timestep generate the noisy image æz. Then, the U-Net backbone conducts one denoising step. Instead of collecting the model output we gather U-Net activations and consolidate them to the candidate pool, as described in Section 5.1. Afterward, each activation is individually fed to a downstream model to evaluate its discriminative potential, with the details of the model described in Appendix C. Such comparison is made fair by using the same dataset for all activations. Finally, we can obtain the capability ranking of activations from each resolution, according which features can be selected wisely. Notably, we conduct such comparisons on multiple datasets to guarantee generalizability. since the best activations may differ among different scenes [2]. Thanks to qualitative filtering, the time cost for each (model, dataset) pair has been reduced by more than one week, equipped with Nvidia(R) RTX 3090 GPUs.
Given the capability ranking, feature selection is in fact flexible, as it possible to combine multiple activations and specifically tune the choice for task. For practicality, we provide fff-the-shelf feature combinations for S.v1.5 and SDXL that are likely to generically perform well Appendix B, according to the results detailed in Appendix D. Each of them consists of four activations mainly from ViT modules rather than inter-module positions. Taking SDv1.5 as an example, one of the four selected features is from one self-attention layer in the highest resolution, which utilizes our observation Section 4.
6 Experimental Validation on Multiple Discriminative Tasks
To validate the effectiveness of our feature selection solution, the experiments are conducted on three popular discriminative tasks: semantic correspondence, semantic segmentation, and label-scarce segmentation. The SOTA methods for each task are selected as competitors. Besides, we also compare with two baselines that select the conventional inter-module activations as features [2], named Legacy-vl.5 and Legacy-XL. For our method, we provide the following implementations:
Ours-v1.5 & Ours-XL: Features extracted from SDv1.5 and SDXL, respectively.
Ours-XL-t: For fairness, further enhance SDXL features with some additional techniques that are also adopted by SOTAs. The techniques we select, i.e., attention maps [58] and feature amalgamation [2. 29, 60], relatively simple and lightweight.
More experimental details are in Appendix C, such as task information, evaluation metrics, and implementation details. Besides, Appendix D presents additional results not covered here.
6.1 Empirical Results on Semantic Correspondence
present the experimental results for semantie correspondence in Table From the results, we have the following observations:

---

Table 2: Experimental results on semantic segmentation and its altered version with scarce labeled data, evaluated using mloU? metric. The best results are bold font and the runner-up underlined.
Standard Setting
Label-Scarce Setting Horse-21
Category
Method
Method
ADE20K
CityScapes
MaskCLIP
23.70
SwAVw2 MAE
54.0 + 0.9
ODISE
29.90
63.4 + 1.4
SOTA
VPD
37.63
55.06
DatasetDDPM DDPM
60.8 1.0
Meta Prompts
40.89
71.94
65.0 0.8
Legacy-v1.5
40.26
64.01
Legacy-v1.5
59.4 + 1.3 53.0
Baseline
Legacy-XL
27.78
71.67
Legacy-XL
Ours-v1.5
41.07
64.10
Ours-v1.5
60.2
Ours
Ours-XL
43.45
74.47
Ours-XL
62.7 2 0.7
Ours-XL-t
45.71
75.89
Ours-XL-t
66.3 + 0.9
()) Ours-vI.5 outperforms Legacy-vI.5 (77.78 v.s. 75.14 on PCK@0.1 iigg). The main reason is that Legacy-vI.5 fails effectively handle the diffusion noises high-resolution activations. Previous approaches either reserve the noisy activations and thus suffer from performance degradation [2. 52], or simply discard high-resolution activations and thus suffer from informa- tion loss [58, 46]. In contrast, our approach uses self-attenticn locality to suppress diffusion noises and harvest better high-resolution activations.
(ii) Legacy-XL inferior to Legacy-vI.3 (66.00 v.s. 75.14 on PCK im. img). At first glance, this result is counter-intuitiv since SDXL is more advanced than SDv1.5. 55. However, the analysis in Section can unravel the mystery. Specifically, since SDXL has more ViT modules, the more valuable activations shift from inter-module positions to these embedded ViTs. Since the baseline does not consider ViT modules, Legacy-XI. fails to achieve better performance. In contrast, Ours-XL. shows improvement over Ours-v/.5 (81.72 v.s. 77.78 on PCK @0.1 lmng) This is consistent with the advance model architecture and again validates our analysis.
(iii) Ours-XI-1 significantly outperforms the SOTA method with similar amalgamation technique, i.e., DHF [29] (83.90 v.s. 72.56 on PCK @0. imgg. This performance gain again validates the effectiveness of our method.
6.2 Empirical Results on Semantic Segmentation
As shown in the left part of Table 2, Ours-XL-I and Ours-XL achieve state-of-the-art performance on the semantic segmentation task (45.71 and 43.45 40.89 ADE20K), demonstrating its effectiveness and generalizability. Furthermore, the most competitive SOTA, Meta Prompts [46]. introduces large number of trainable parameters and uses the diffusion U-Net recurrently, which is rather cone-consuming. In contrast, our method delivers superior results with efficiency maintained.
Unlike the results in semantic correspondence, Legacy-XL. outperforms Legacy-vi.5 CityScapes (71.67 v.s. 64.01), and the performance gap between Legacy-v/.5 and Ours-vl.5 is narrow (64.01 vs. 60). This because this task utilizes relatively large-scale downstream model, which can significantly refine the input features and thus reduce the gap in feature quality. Nevertheless, Ours-XL still achieves significant improvement over Legacy-XL. (74.47 71.67).
6.3 Empirical Results Label-Scarce Segmentation
One advantage of diffusion features the applicability to label-scarce scenarios [2]. For validation under such conditions, experiment the label-scarce segmentation task, with results presented in the right part of Table The observations are generally similar to those on the semantic correspon- dence task. For example, Ours-vl.5 outperforms Legacy-vl.5 (60.2 59.4), Legacy-XL is inferior to Legacy-vl.5 (53.0 v.s. 59.4), and Durs-XLi is better than Ours-vl.5 (62.7 v.s. 60.2).
Next, we focus on the comparison with the SOTA method, DDPM [2]. Although DDPM is relatively early study, outperforms the other competitors and our most implementations. This result has two reasons. On one hand, DDPM performs the diffusion process directly in the image space

---

rather than the currently common practice of compressed latent space. Although inefficient, such an implementation yields better discriminative features. On the other hand, DDPM utilizes diffusion models specifically trained on each dataset. In comparison, we use pre-trained general-purposed SDv1.5 or SIXL, be more efficient and thus consistent with the motivation of diffusion feature. Hence, it is rather challenging to surpass this SOTA method. Fortunately, our best implementation, Ours-XL-t, achieves this goal with the help of additional lightweight techniques (66.3 v.s. 65.0).
7 Conclusion and Future Work
In this study, we revisit the fundamental problem of feature selection from diffusion U-Nets. We point out that prior arts only consider a limited range of potential activations. In contrast, we consider a much wider range of activations as candidates, especially those extracted from the embedded ViT modules. Given the large volume of the candidate pool, we first analyze the properties of diffusion U-Nets. The properties we find are universal such that our observations are not limited to the specific diffusion architecture. Based on these properties, we qualitatively filter out many activations with low quality, facilitating the following quantitative comparison. On top of this, concrete feature selection solutions are proposed for two popular diffusion models, i.e., SDv1.5 and SDXL. Finally, extensive experiments three discriminative tasks validate the effectiveness of our method.
However, we are not sure whether our observations can generalize well recently-develope DiT models [35] since they have markedly different architecture from D--ee-bbase diffusion models. Thus, analyzing DiT models is promising topic for future research.
Acknowledgments
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, part by National Natural Science Foundation of China: 62236008, U21B2038, U23B2051, 61931008, 62122075 and 62025604, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680000, in part by the Innovation Funding of ICT, CAS under Grant No. E000000,, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX20240384.
References
[1] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. ECCV Workshop on What is Motion For, 2022.
[2] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In The Tenth International Conference on Learning Representations, 2022
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Annual Conference on Neural Information Processing Systems, 2020.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In /EEF/CVF International Conference on Computer Vision, ICCV, pages 9630-9640, 2021.
[5] Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, 4(3):e15, 2019.
[6] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high- righ-quaiity video diffusion models. CoRR, abs/2401.09047, 2024.
[7] Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu. Your difffusin model is secretly a certifiably robust classifier. CoRR, 162 2024.
10

---

[8] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3213-3223, 2016.
[9] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[10] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Anmual Conference on Neural Information Processing Systems, pages 8780-8794, 8794, 2021.
[11] Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskelip. In International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pages 8090-8102 2023.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, 2021.
[13] Niladri Shekhar Dutt, Sanjeev Muralikrishnan, and Niloy J. Mitra. Diffusion 3d features (diff3f) decorating untextured shapes with distilled semantic features. In /EEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4494-4504, 2024.
[14] Raphael Antonius Frick and Martin Steinebach. Diffseg: inpainting attacks using multi-fcature segmentation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages
Towards detecting eeffusion-basedd
2024.
[15] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, 2021.
layers
[16] Rui Gong, Martin Danelljan, Han Sun, Julio Delgado Mangas, and Luc Van Gool. Prompting diffusion representations for cross-domain semantic segmentation. CoRR, abs/2307.02111 2023.
[17] Boyu Han, Qianqian Xu, Zhiyong Yang, Shilong Bao, Peisong Wen, Yangbangyan Jiang, and Qingming Huang. Aucseg: Auc-oriented pixel-level long-tail semantic segmentation, 2024.
[18) Shijie Hao, Yuan Zhou, and Yanrong Guo. A brief survey semantic segmentation with deep learning. Newrocomputing, 406:302-321, 2020.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick, Masked autoencoders are scalable vision learners. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15979-15988, 2022.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778. 2016.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Annual Conference on Neural Information Processing Systems, 2020.
[22] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. DDP: diffuusion model for dense visual prediction. In IEEE/CVF International Conference on Computer Vision, pages 21684-21695 2023.
[23] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. CORR, abs/2312.02 145, 2023.
[24] Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimarães, and Pietro Perona. Text-image alignment for diffusion- based perception. CoRR, abs/2310.00031, 2023.
11

---

[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Alexander C. Li. Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. IEEE/CVF International Conference on Computer Vision, pages 2206-2217, 2023.
[27] Xinghui Li, Jingyi Lu, Kai Han, and Victor Prisacariu. Sd4match: Learning to prompt stable diffusion model for semantic matching. CoRR, abs/2310.17569. 2023.
[28] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.
[29] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell, Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Annual Conference Neural Information Processing Systems, 2023.
[30] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair A large-scale benchmark for semantic correspondence. CoRR, abs/1908 10543, 2019.
[31] Juhong Min, Jongmin Lec, Jean Ponce, and Minsu Cho. Learning to compose hypercolumns for visual correspondence. In Computer Vision ECCV 2020 16th European Conference, volume 12360, pages 346-363. Springer, 2020.
[32] Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion models beat gans image classification. CoRR, abs/2307.08700 2023.
[33] Soumik Mukhopadhyay, Matthew Gwilliam, Yosuke Yamaguchi, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Tianyi Zhou, and Abhinav Shrivastava. Do text-free diffusion models learn discriminative visual representations? CoRR, abs/2311.17921, 2023.
[34] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11))=7, 2017.
[35] William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision, pages 4172-4182, 2023.
[36] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307, 019552, 2023.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffuusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10674-10685, 2022.
[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assiste Inter- vention, volume 9351, pages 2442411. Springer, 2015.
[39] Yang Song, Jascha Sill-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representutions, 2021.
[40] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In The Eleventh International Conference on Learning Representa- tions, 2023.
[41] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. In Annual Conference on Neural Information Processing Systems, 2023.
12

---

[42] Kashvi Taunk, Sanjukta De, Srishti Verma, and Aleena Swetapadma. A brief review of nearest neighbor algorithm for learning and classification. In 2019 international conference on intelligent computing and control systems (ICCS), pages 225-1266.. IEEE, 2019.
[43] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar González-Franco. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. CoRR, abs/2308. 1..3008.24499 22499, 2023.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Annual Conference on Neural Information Processing Systems, pages 5998-609900 2017.
[45] Martina G. Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. In Annual Conference on Neural Information Processing Systems 2023, 2023.
[46] Qiang Wan, Zilong Huang, Bingyi Kang, Jiashi Feng, and Li Zhang. Harnessing diffusion models for visual perception with meta prompts. CoRR, abs/2312. 14733, 2023.
[47] Zitai Wang. Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Ope- nauc: Towards auc-oriented open-set recognition. In Annual Conference on Neural Information Processing Systems, 2022.
[48] Zitai Wang. Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Optimizing partial area under the top-k curve: Theory and practice. IEEE Trans. Puttern Anal. Mach. Intell., 45(4))5053-500) 2023.
[49] Zitai Wang, Qianqian Xu, Zhiyong Yang. Yuan He, Xiaochun Cao, and Qingming Huang. A unified generalization analysis of re-weighting and logit-adjustment for imbalanced learning. In Annual Conference Neural Information Processing Systems, 2023.
[50] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. In Annual Conference on Neural Information Processing Systems, 2023.
[51] Changming Xiao, Qi Yang, Feng Zhou, and Changshui Zhang. From text to mask: Localizing entities using the attention of text-to-image diffusson models. CoRR, abs/2309.04409 2023.
[52] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE/VVF Conference Computer Vision and Pattern Recognition, pages 2955-2966, 2023.
[53] Xingyi Yang and Xinchao Wang. Diffusion model as representation learner. In IEEE/CVF International Conference on Computer Vision, pages 18892-18903, 2023.
[54] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365. 2015.
[55] David Junhao Zhang, Mutian Xu, Chuhui Xue, Wenqing Zhang, Xiaoguang Han, Song Bai, and Mike Zheng Shou. Free-atm: Exploring unsupervised learning on inffusiion-generated images with free attention masks. CoRR, abs/2308.0677 06739, 2023.
[56] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani. Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements DINO for zero-shot semantic correspondence. In Annual Conference on Neural Information Processing Systems, 2023.
[57] Zhengxin Zhang, Qingjie Liu, and Yunhong Wang. Road extraction by deep residual u-net. IEEE Geosci. Remote. Sens. Lett., 15(5):749-755 2018.
[58] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen La. Unleashing text-to-image diffusion models for visual perception. IEFF/CVE International Conference on Computer Vision, pages 5706-5716, 2023.
13

---

[59] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5122-5130, 2017.
[60] Jingyi Zhou, Jiamu Sheng, Jiayuan Fan, Peng Ye, Tong He, Bin Wang, and Tao Chen. When hyperspectral image classification meets diffusion models: An unsupervised feature learning framework. CoRR, abs/2306.08s 0.9644, 2023.
14

---

Contents
Additional Visualization for Distinct Properties of Diffussin U-Nets
16
A.1 Visualization of Other Sample Images
16
A.2 Visualization from Other Diffusion Models
17
Details of Our Method
22
B.1
Feature Selection Solution for SDv1.5
22
B.2 Feature Selection Solution for SDXL
22
B.3 Feature Selection Solution with Additional Techniques
22
B.4
Alternative Feature Selection Solution with Additional Techniques
23
Experimental Details
23
C.1 Semantic Correspondence
23
C.2 Semantic Segmentation
24
C.3 Label-Scarce Segmentation
24
Additional Experimental Results
24
D.1
Generalizability across Different Scenes
24
D.2
Quantitative Comparison Results
25
E
Other Properties of Diffusion U-Nets
29
E.1
High-Frequency Noises in Increment Activations
29
Detailed In-Resolution Granularity Change
E.3
Collaboration between Embedded ViTs.
29
F
Future Direction
30
G Computation Resources
30
Asset License
31
15

---

A Additional Visualization for Distinct Properties of Diffusion U-Nets
A.1 Visualization of Other Sample Images
We have provided activation visualization from SDXL in simple outdoor scene presenting a horse in the wild. Next, we will show visualization from the same model different scenes demonstrate the universality of the observed properties. Another simple outdoor scene presenting Two simple indoor scenes are visualized in Figure 6 and Figure 7. Two complex outdoor scenes of urban streets visualized in Figure 8 and Figure 9. Two complex indoor scenes are shown in Figure 10 and Figure 11.
is in Figure 5.
(b, In In-Resolution Level) There are in-resolution granularity changes
(c, Sub-Module Level) Without positional embeddings, there is still locality in self-attention query & key
SDXL, Simple Outdoor
(a, Macro Level) Diffusion noises induced by the diffusion process and exist despite signal frequency, but its change in down/up-stage is asymmetric Nolsy
Still Clean
Down-Stage
Mid-Stage
Up-Stage
Figure 5: Visualization of SDXL activations simple outdoor scene.
(b, Resolution Level) There are in-resolution granularity changes
SDXL, Simple Indoor
(c, Sub-Module Level) Without positional embeddings, there is still locality self-attention query key
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency, change in down/up-stage asymmetric
Noisy
Still
Down- Down-Stage
Mid-Stage
Up-Stage
Figure 6: Visualization of SDXL activations simple indoor scene.
16

---

(b, in-Resolution Level) are in-resolution granularity changes
OET & CHANDON
ITEE STAR
PAA GNE
Ag
(c, Sub-Module Level) Without positional embeddings, there still locality self-attention query key
SDXL, Simple Indoor
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency, but change in nown/up-stage asymmetric
Noisy
Still Clean
Down-Stage
Mid-Stage Stage
Up- Stage
Figure 7: Visualization of SDXL activations simple indoor scene.
(b, In- Resolution Level) There are in-resolution granularity changes
(c, Sub-Module Level) Without positional embeddings, there is still locality in self-attention query key
SDXL, Complex Outdoor
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency, but change down/up-stage asymmetric
Noisy
Clean
Sown-Stage
Mid-Stage
Up-Stage
Figure 8: Visualization of SDXL activations complex outdoor scene.
A.2 Visualization from Other Diffusion Models
Playground v2 Activations. Playground v2 shares the same architecture SDXL but is trained independently, and is claimed to be more powerful in generation. In Figure 12, we present its activation visualization using the same horse image the primary SDXL visualization. Compared to SDXL, Playground v2 activations are less noisy, particularly in the down-stage. This supports the claim that Playground v2 is stronger model.
SDv1.5 Activations. SDv1.5, as an older model, has slightly different architecture from SDXL. Specifically, SDv1.5 has four resolutions instead of three, but its ViTs contain only one layer. Moreover, SDXL, only the highest resolution lacks ViT due to efficiency concerns, while in
aesthetic
17

---

(b, Resolution Level) are in-resolution granularity changes
(c, Lub-Module Level) Without positional embeddings, there still locality self-attention query key
SDXL, Complex Outdoor
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency, but change in nown/up-stage asymmetric
Noisy
Still Clean
Down- Down-Stage
Mid-Stage
Up-Stage
Figure Visualization of SDXL activations on complex outdoor scene.
(b, In Resolutiot Level) There are in-resolution granularity changes
(c, Sub-Module Level) Without positional embeddings,
there is still locality in self-attention query & key
SDXL, Complex Indoor
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency, but change in down/up-stage asymmetric
Noisy
Still Clean
Down-Stage
Mid-Stage
Up-Stage
Figure 10: Visualization of SDXL activations complex indoor scene.
SDv1.5, only the lowest resolution lacks ViT due to its low resolution. Despite the architectural changes, the three unique properties still apply to SDv1.5, as shown in Figure 13.
Video Diffusion Activations. To further demonstrate the universality, we even select a diffusion model for video generation [6] for visualization. This model is based on the SDv1.5 architecture, with an additional temporal attention layer inserted after cross-ateention to enable sequential generation. Although this model is designed for different task, we can still observe the three unique properties in Figure 14.
Conventional U-Net Activations. As comparison to diffusion U-Net, we also visualize some activations from conventional U-Net Figure 15, where the three properties of diffusion U-Net can hardly be observed.
18

---

(b, In in-Resolution Level) There are in-resolution granularity changes
(c, Lub-Module Level) Without positional embeddings,
there is still locality in self-attention query & key
SDXL, Complex Indoor
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency. but change in down/up-stage is asymmetric
Noisy
Still Clean
Down- Down-Stage
Mid- Mtid-Stage
Up-Stage
Figure 11: Visualization of SDXL activations on a complex indoor scene.
(b, In-Resolution Level) There are in-resolution granularity changes
(c, Sub-Module Level) Without positional embeddings, there is still locality in self-attention query key
Playground v2
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency, but change in down/up-stage asymmetric
Nolsy
Still Clean
Down-Stage
Mid-Stage
Up-Stage
Figure 12: Visualization of Playground v2 activations on a simple outdoor scene.
19

---

(b, In-Resolution Level) There in-resolution granularity changes
(c, Sub-Module Level) Without positional embeddings, there is still locality in self-attention query & key
SDv1.5
(a, Level) Diffusion noises induced by the diffusion process and exist despite signal frequency, but its change in down/up-stage is asymmetric
Noisy
Still Clean
Down-Stage Stage
Mid-Stage
Up-Stage
Figure 13: Visualization of SDv1.5 activations simple outdoor scene.
(b, In- tn-Resolution Level) There are in-resolution granularity changes
(c, Sub-Module Level) Without positional embeddings,
there is still locality in self-attention query key
Video Diffusion Model
(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency, but change in down/up-stage is asymmetric
Noisy
Still Clean
Down-Stage
Up-Stage
Figure 14: Visualization of the activations of a video diffusion model. The input is short video, and visualize frame of it.
20

---

Down
Up
Mid
Figure 15: Activation visualization from traditional U-Net for semantic segmentation [57].
21

---

B Details of Our Method
In this section, we index all diffusion U-Net components in the order in which they are activated during network forward run. Besides descriptions in natural language, we also index the activations using the same notations as used in our code implementation. These notations denote stages using down/mid/up, resolutions level, and module index repeat. Index starts to count from 0 instead of following the convention coding.
B.1 Feature Selection Solution for SDv1.5
We select four activations from SDv1.5, maintaining the same total feature channels as the conven- tional way to extract all output activations from each resolution.
The cross-attention query activation from the 2nd ViT, the 2nd resolution. This provides coarse information for simple scenes -rppppppaaaliiiibblooo--o-oo----
(ii) The inter-module activation after the 3rd ResModule, the 2nd resolution. This provides coarse information for complex scenes
(iii) The cross-attention query activation from the 2nd ViT, the 3rd resolution. This provides finer information with a higher resolution
(iv) The self- oeffaattntion key activation from the 1st ViT, the 4th resolution. This extracts features from the highest resolution, harnessing the noise suppression effect of self-attention locality hatpee------ppaa--iii-booo------
We omit the index of basic blocks in ViTs, SDv1.5 only contains one-layer ViTs. Additionally, we totally ignore the lowest resolution, as its activations have very low resolution (8 x 8 if the input image is 512 x 512), suggesting inferiority, as supported by the quantitative comparison.
B.2 Feature Selection Solution for SDXL
Four activations are selected from SDXL, trying to get similar total feature channels to the SDv1.5 feature selection solution.
(i)
The output activation after the 8th basic block, the 1st VIT. the 1st resolution. This provides relatively coarse information for simple scenes hop--eeeel-eeepaaat-vvttbbe---
(ii) The output activation after the 6th basic block, the 1st VIT. the 1st resolution. This provides relatively coarse information for complex scenes
(iii) The roossattennionn query activation from the 1st basic block, the 1st VIT, the 2nd resolution. This provides relatively fine information for simple scenes -up-levell-repeat--vii -iooooo---isioooooooooo-ss
q).
(iv) The output activation after the 1st basic block, the 1st VIT, the 2nd resolution. This provides relatively fine information for complex scenes
We ignore the highest resolution since it is affected by diffusion noises and lacks ViTs from which we can extract self-attention activations. However, if the downstream model is strong enough to learn to suppress noises, it may be possible additionally extract inter-module activations from this resolution to harness more information.
B.3 Feature Selection Solution with Additional Techniques
For the setting Ours-XL-t, we mainly utilize two simple techniques that are also adopted in some SOTA methods: (i) We additionally extract attention maps, i.e., the similarity scores of cross-attention query and key, as dense features [58, 52]. Such attention maps are closely related to the semanties of prompts, thus providing important supplementary information. Despite its usefulness, this technique only adds a few additional channels to the features. (ii) We amalgamate features from different models [2, 29, 60] through simple concatenation. This technique is common practice and can be seen as an extension of amalgamating different activations together as whole feature. We next explain what activations selected to implement the two techniquess.
22

---

We first extract features according to the feature selection solution for SDXL. Afterward, we extract additional features from SDv1.5:
(i)
The cross-attention query activation from the 2nd VIT, the 2nd resolution (up-levell-repeatl- vit-blockO-cross-g).
iii) The cross-attention query activation from the 2nd ViT, the 3rd resolution (up-lovel2-repeatl vitbblock-croos--q..
(iii) The upsampler output activation from the 3rd resolution, to harness high-resolution information (up-tevel-uppsampleerou)))
(iv) The self-attention key activation from the 1st VIT, the 4th resolution, to harness high-resolution information paalevel3-repeatt--vv wit--vitt-bbbae blockO--elf-kk...
(v) The attention maps averaged over all cross layers in the up-stage.
We also extract one feature from Playground v2: the output activation after the 4th basic block, the 1st VIT, the 1st resolution (up-level0-repeatt wwwprrrpeeaaaiiilloooo-----
B.4 Alternative Feature Selection Solution with Additional Techniques
Large-scale datasets for semantic segmentation mostly consist of images of complex scenes. In such cases, we find attention maps can too noisy to be useful. Therefore, we discard the attention map technique and the entire SDv1.5 model, as it is weaker compared to the newer SDXL and Playground v2 models. To compensate for the loss of activations, we select additional activations from SDXL and Playground v2.
From SDXL, we select all the activations as described in the feature selection solution for SDXL and select one additional activation: the upsampler output activation from the 2nd resolution, to harness high-resolution information (up-levell uup-leeellupsaapleer activations:
From Playground v2, we select the following
(i) The output activation after the 6th basic block, the 1st ViT, the 1st resolution (up-level0-repeatt- vit-blockk-out).
(ii The cross-attention query activation from the 1st basic block, the 1st ViT, the 2nd resolution -ep-levell-rppeatO--e Sit--loccO--rrsss
(iii The upsampler output activation from the 2nd resolution
C Experimental Details
C.1 Semantic Correspondence
Task and Dataset. Semantic correspondence [18] involves finding a pixel in an image that semanti- cally matches another keypoint pixel in a reference image, such as the hind legs of two different cats. We conduct experiments on the SPair-7Ik dataset [30].
Evaluation Metric. PCK @0.1 img (1) and PCK@0.1 (1) are used, following the widely-adope protocol reported in [30]. These two metrics mean the percentage of correctly predicted keypoints, where predicted keypoint is considered to be correct if it lies within the neighborhood of the corresponding annotation with a radius of 0.1 x mar w). For PCK @0. 10.1.mm//...... 1m00l /PCK @0. bbox. h, w denote the dimension of the entire image/object bounding box, respectively.
SOTA Competitors. We provide the results from four SOTA methods: DINO [4] and DHPF [31] as non-diffiusion-feauur methods, well DIFT [41] and DHF [29] as diffusion feature methods.
Implementation Details. The semantic correspondence task can be done via the nearest neighbor algorithm [42], which is unsupervised and faaniin-frree [30]. We add one additional trainable convolutional layer before applying the nearest neighbor algorithm to refine the input features, which is also adopted by some SOTAs including DHF. The model is trained for two epochs, each containing 5,000 sample pairs, following conventional settings. Our implementation derived from DHF, and keep all sepper-prrameter. at their default settings.
23

---

C.2 Semantic Segmentation
Task and Dataset. Semantic segmentation [18] is essentially pixel-level classification. For this task, we choose the ADE20K dataset [59] with over 20k annotated images of 150 semantic categories, and the CityScapes dataset [8], which contains 5,000 fine annotated images of urban street scenes.
Evaluation Metric. We use mloU metric, which the mean over the IoU performance across all semantic classes [18]. For each image, IoU (Intersection over Union, 1) is defined by #(overlapped pixels between the prediction and the ground truth) #(union pixels them).
SOTA Competitors. We choose three diffusion feature SOTA methods as competitors. ODISE [52] an early method with simple implementation. VPD [58] is another early study of this field, which introduces additional text adapter modules for improvement. Meta Prompts [46] is newer method and shows significant improvements. We also report the performance of MaskCLIP [11], which is included as competitor in the ODISE study.
Implementation Details. Both VPD and Meta Prompts perform full-scale fine-tuning the diffusion U-Net using feedback from the discriminative task. This heavy fine-tuning does not entirely comply with the motivation of diffusion feature, which seeks balance between wider applicability and less training, and is hard to extend to the larger SDXL model. Therefore, we keep the entire diffusion model frozen instead. As the setting has been changed, the performance of VPD and Meta Prompts in Table 2 is based on our experiments, not the reported results from the original papers. Our implementation directly uses the tlype--prammee reported in Meta Prompts.
C.3 Label-Scarce Segmentation
Task and Dataset. Using features from a pre-trained diffusson model ensures good performance even when labeled training data is scarce [2]. For this setting, we use a dataset collected in [2] and experiment on its Horse-21 subset, the data of which is sourced from LSUN [54]. This subset contains only 30 labeled training images to be consistent with the intuition. The semantic segmentation in the label-scarce scenario also uses the mloU metric.
SOTA Competitors. We select the SOTA diffusion feature approach, DDPM [2], as the major competitor. We also include other representative segmentation methods: DatasetDDPM, MAE [19], SwAV [3], which are all reported in [2].
Implementation Details. Following DDPM [2], the downstream model is ensemble of ten simple MLP networks, each conducting pixel-wise classification. The simplicity of the model is intended to demonstrate the innate capability and generalizability of diffusion models. Our implementation is parameters. We use a larger batch smaller does not improve performance. Additionally, this is
derived from DDPM with only batch size changed among all hyper-parameters. size for faster experiments also the setting for the quantitative comparison, as the compact size of the dataset can enhance efficiency.
D Additional Experimental Results
D.1 Generalizability across Different Scenes
As stated in Section 5, it is preferable to conduct the quantitative comparison across multiple datasets and choose activations that are optimal for each. This approach can enhance the generalizability of the selected features. To evaluate the generalizability of our features, we conducted an additional experiment, with results presented Table 3.
In this experiment, we design an alternative feature selection solution for SDXL, based solely on quantitative results from a single dataset consisting of simple scenes. In this solution, we extract both optimal and slightly sub-optimal activations, maintaining the same total number of feature channels as the standard solution. The alternative solution achieves higher performance on the simple scene it is based on but performs significantly worse on the other scene. Therefore, we conclude that our standard feature selection solution achieves generalizability across different scenes, albeit with a slight performance drop compared to features specifically selected for each scene.
24

---

Table 3: Examination of generalizability across different scenes. Generic Solution refers to our standard feature selection solution for SDXL,, while Specific Solution refers the outcome of considering only a simple scene for the quantitative comparison. The experiment is conducted on the label-scarce segmentation task, where the Horse-21 subset used for simple scenes and the Bedroom-28 subset is used for complex scenes. We mark the better results as bold font.
Method
Simple Scene
Complex Scene
Generic Solution
63.34
47.55
Specific Solution
63.70
45.41
D.2 Quantitative Comparison Results
In this part. we present all the quantitative comparison results obtained following the protocol described in Section 5. These results are from dataset of simple scenes (Horse-21 [2]) and dataset complex scenes (Bedroom-28 [2]). Since SDXL. and Playground v2 share the same U-Net architecture, their results shown in the same tables. We display the results from the lowest
resolution in Table 4 and the results from the middle resolution in Table 5. We have also done quantitative comparison SDv1.5, and the results are shown in Table 6.
25

---

Table 4: Quantitative comparison results of SDXL and Playground v2. This table shows the results from the lowest resolution. Activation ID indicates the location of each activation the diffusion U-Net. The best results are in bold font and the runner-up is underlined.
SDXL
Playground v2
Activation ID
Simple
Complex
Simple
Complex
up-level0-repeatt-rre-oo
53.49
38.52
54.57
40.73
56.46
42.31
55.32
41.93
wp.lleeellrrgeeaal--xttbbbooo--
57.36
42.31
56.60
44.52
57.82
42.72
56.44
43.93
58.73
42.62
58.12
45.82
58.27
43.09
57.79
45.90
58.95
44.83
59.04
47.81
57.30
44.36
56.97
47.38
58.70
45.26
58.67
49.77
57.24
43.00
57.69
48,51
58.95
44.83
58.76
48.39
57.06
41.73
56.03
44.68
58.46
43.98
56.59
48.54
up-levee00-rgpeatt0vvii-oou
58.36
41.54
57.99
43.95
up-levell--rapeaatl-res-o-
57.28
39.27
56.59
41.74
55.72
40.96
55.56
43.97
56.70
40.97
57.51
43.35
57.92
41.65
56.37
42.10
57.38
42.25
57.20
43.10
56.55
40.50
55.00
42.50
56.41
40.97
57.54
42.92
54.92
39.66
54.89
40.79
55.86
40.66
56.32
42.48
51.63
39.01
52.72
38.37
53.97
39.79
55.90
41.34
50.27
36.42
36.09
24.81
up.lexel--reppaal--ii-hhhh-----0
53.31
38.58
53.19
40.30
up-lexel0-reeecat--vittoo
57.89
38.91
57.00
42.87
up--lovel---qqecaa2-rr--0000
56.11
36.37
55.95
40.62
56.09
36.68
55.56
41.29
57.14
37.16
56.48
41.14
55.38
34.87
56.15
39.78
up--doeell-<qqaaa222iiiiblllllu-
56.36
35.63
56.71
40.93
55.26
34.40
54.98
38.70
55,50
34.61
56.77
39.36
52.70
33.16
54.70
37.93
54.68
33.75
55.75
38.95
51.91
31.97
52.92
35.58
53.07
32,43
54.07
36.91
49.22
29.34
41.07
18.48
51.41
30.64
51.31
35.64
up-levee0-repeaa2--vitoot
53.77
33.79
54.24
37.53
up-levelO-upsaamplerroot
53.21
32.85
53.97
36.30
26

---

Table 5: Quantitative comparison results of SDXL and Playground v2. This table shows the results from the middle resolution. Activation ID indicates the location of each activation in the diffusion U-Net, The best results are in bold font and the runner-up is underlined.
SDXL
Playground v2 Complex
Activation ID
Simple
Complex
Simple
47.84
30.12
47.49
31.63
49.08
31.75
47.96
32.53
46.74
31.14
46.49
30.71
44.78
30.56
44.21
28.58
-out
42.20
27.32
40.90
27.13
upelexell-repccat0vvitttoott
46.93
29.35
46.08
30.15
upplexeel-reppcall-eesoouuu
41.36
26.81
41.51
28.09
39.25
27.02
39.92
27.72
upplleell-zggpaallxxx--booo----
38.64
25.68
39.47
27.60
36.76
24.57
37.20
25.82
up.eevell-agecatl-viibbblo.....o
34.74
23.20
34.88
24.32
up-levell-repecat--vittoott
39.16
25.48
39.87
26.82
up..eveel--repcat2-res-oot
33.58
23.99
36.30
25.26
31.63
22.98
33.97
23.88
32.16
24.97
34.30
24.83
31.31
23.51
34.81
25.27
30.39
22.99
33.01
25.44
27.65
22.61
32.40
24.01
27.66
22.35
31.44
24.24
up-levell-repeaa--vvitoot
27.61
19.80
26.29
18.61
27

---

Table 6: Quantitative comparison results of SDv1.5. Activation ID indicates the location of each activation in the diffusion U-Net. The best results are in bold font and the runner-up is underlined. both marked per resolution.
SDv1.5
Activation ID
Simple
Complex
up-levell-epeeat---es-ouut up-eewel--eppaao---i-boookoooopp
47.16
37.86
47.59
41.13
48.59
39.75
up-leweel--repcat0-vii-ouuu
48.21
39.91
up-ecwell-repeaal--ressuuu
49.71
40.80
49.04
42.80
49.88
39.43
up-levell-repecat--viteouuu
50.23
40.98
up=levell-reppcal22stsssuuut
50.61
39.47
50.10
38.80
49.09
37.44
up-levell-repcaa2-vit-ouu
49.62
36.69
up-devel2-repccat0-eee-oouu
52.43
37.98
52.59
39.63
51.56
36.62
up-level2-reecaat-vviiooo
52.29
38.78
ups-cwel2-repcaal--ressouut
53.30
36.65
up/ceeel-eppaall-iiibblccccoossp
53.58
39.82
up--dvel--ngeeallxxibbblo00-0.
50.70
35,96
up-level2-reepaat--vittouuu
50.50
35.59
up=levee22-eppeat2-ressouur
48.47
33.70
45.44
32.45
45.69
32.71
46.24
32.95
up=:evel--rreeaa22iiibbbok0--0
45.33
32.23
up--cvee2--ccpaaa2-vii-ou
45.04
28.87
up-level2-upsamplerroott
45.07
29.57
38.81
25.72
37.78
26.04
31.64
24.02
32.07
24.00
29.24
21.27
29.19
21.39
28

---

Common Activations (No Noises)
ResModule Increment (Moderate Noises)
Feed-Forward (Moderate Noises)
Self-Attention (Strong Noises)
Figure 16: This visualization compares the high-frequene frequeeryenny noises in various activations, showing three types of increment activations with strong noises.
Coarse
Fine
Overty Fine
Figure 17: This visualization shows the granularity change across all inter-module activations of one resolution. At the end, the activations start to contain some slight noises, which is sign of over-refinement.
E
Other Properties of Diffusion U-Nets
E.1
Hiigh-Frequency Noises in Increment Activations
Typically, figh-frequency signals are usually noisy [5, 34]. This applies to the diffusion U-Net, and further examine what activations more vulnerable to such noises. To be specific, the diffusion U-Ner contains many residual connection structures, and their increment activations are feggfrreyeeccy signals prone to noises. Such increment activations include:
(i) The increment branch ResModule. These activations are moderately noisy and thus usually less effective than inter-module activations.
(ii) The feed-forward layer activations within ViTs. These activations are also moderately noisy. However, this results from their significantly more channels compared to other activations, which can reduce their noise magnitude.
(iii) The self-attention value activations within ViTs. These activations are severely noisy and suffer significant degradation as they the nested inner increments within embedded ViTs.
These activations are visualized in Figure 16 for clearer illustration. Additionally, the residual activations within VITs at the same time also increments the main inter-module residual. However, these activations are not obviously affected by high-frequency noise, possibly due to their dual role as ViT residuals.
E.2 Detailed In-Resolution Granularity Change
have previously described the existence of in-resolution granularity changes, and this section will further detail the pattern of one such change. In one resolution of the up-stage, activations gradually shift from coarse to fine granularity, which aligns with intuition. However, the diffusion U-Net tends to overly refine the inter-module activations, resulting in slight noises in the last few inter-module activations due to excessive detail. This can be observed from the visualization of Figure 17. Consequently, drop in discriminative performance is often resolution.
the end of one
E.3
Collaboration between Embedded ViTs
Multiple ViT' exist within one resolution; for example, one resolution in the up-stage typically contains three ViTs. These ViTs collaborate refining the main inter-module residual. During this process, each ViT exhibits inner granularity change it produces the increment activation, and the
29

---

ViT
#1
VIT #2
VIT
#3
Shifting Overlapped Granularity Spectra
Figure 18: This visualization shows that the collaborating ViTs within one resolution contain overlapped and shifting granularity changes.
changes of collaborating ViTs form certain pattern. To be specific, the change of one ViT overlaps with the previous one to some extent but also shifts as a whole towards finer granularity, which is shown in Figure 18.
Future Direction
might good future direction focus on more challenging discrimination scenarios. Specifically, long tail and out-of-distribuiin are important problems in discrimination, which can greatly hinder the performance models that work well under i.i.d. settings [49]. There have been few attempts to address this more challenging problem with diffusion models. For example, the disentanglement property prompts might grant diffusion features cross-domain capability [16]. It is also possible to utilize diffusion models to synthesize training samples to adjust training data distribution [50, 55]. Given these attempts, more efforts are still required be put in this direction. Moreover, there recent study [17] that might boost long tail and models. To be specific, AUC is an evaluation metric as well as a loss function that promotes good performance both head and tail samples, which is important tool for long tail and out-of- distribution studies [47, 48]. Conventionally, AUC is applicable to image classification but not semantic segmentation and other pixel-level tasks, but the aforementioned study manages to adopt AUC for semantic segmentation. Given this new tool, it is now made more viable to attempt to enhance diffusion feature long tail and didtof-distribution problems.
discrimination using diffusion
G Computation Resources
We use Nvidia(R) RTX 3090 and Nvidia(R) RTX 4090 GPUs for the experiments, all with 24GB VRAM. Most of our experiments, except label-scarce segmentation, require additional storage besides the necessary space for model checkpoints and datasets. The label-scarce segmentation task first extracts features and stores them the disk, and then loads them for the downstream task, which takes about 4GB.
The codes are designed to be able to run on single GPU or less than 4 GPUs, while multiple experiments can run simultancously if more GPUs are provided. Each experiment on the semantic correspondence task takes about 5 hours. Each experiment the large-scale semantic segmentation task takes to days, depending on the dataset. Each experiment on the label-scarce segmentation task takes about hour, but repeat 5 random splits following [2], which increases the overall time. All the experiments in sum can be done within two weeks.
Our quantitative comparison is based on the label-scarce segmentation task, each run taking about 40 minutes. The time shorter because the quantitative comparison evaluates each activation individually. This comparison uses large storage, which is about 45GB per model dataset. With the help of qualitative filtering, we can finish the quantitative comparison one model on one dataset within 2 days.
30

---

H
Asset License
SPair-7Ik:
Available https://cvlab. http::///wv.bbpppttec......aae yot.ech..c.kk///aaarr////a------
Label-scarce segmentation datasets (sourced from LSUN):: Available https://github.cco com/
fyu/lsun.
ADE20K: Custom (research-only, non-commercial), at https://groups.caall. mit........... edu/ vision/ vision/datasets/ Saatasets/AEE20//eerrt
CityScapes: Custom, at
dataset.com/licenss
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in Section 7, which are mainly related to DiT models [35].
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
32

---

Justification:
We do not make theoretical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The most important information to reproduce our results the exact activations select as features, and it is detailed in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [No]
Justification: We will release the codes after acceptance. Besides, we have provided the necessary information for reproducing the experiments in this manuscript.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCodeSubmis /nipp..cccc. https/ccccpbccc//ccceeecec ( ssion ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have included the experimental details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error bars for some experimental results. Others are omitted as can too computationally expensive, such as about two days per run.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the csGuidel NeurIPS Code of Ethics https: :/te:://neeu..eeeeeeee http::::/arrppp.cppppppc//////// cet ines?
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socetal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers where related assets are first mentioned and list their license in Appendix H.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "K8YtaWZIXs", "reviewer": "Reviewer_AMWL", "summary": "This paper proposes a new feature selection method for diffusion models by evaluating a broader range of activations, particularly those in embedded Vision Transformers (ViTs). The authors identify the limitations of current approaches that consider only a narrow range of activations and introduce a qualitative analysis to filter out low-quality activations in diffusion U-Nets. They develop specific feature selection solutions for popular diffusion models SDv1.5 and SDXL. Experiments demonstrate that this method outperforms state-of-the-art techniques in tasks like semantic correspondence, semantic segmentation, and label-scarce segmentation, showcasing its effectiveness and generalizability.", "questions": "Please refer to the weakness.", "limitations": "Yes, the authors have clarified the limitations in Section 7.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper extends the evaluation to a wider range of activations within diffusion models, especially those in embedded ViT modules, which were previously overlooked. Since feature selection is a basic problem for diffusion features, this paper can boost future work in this direction.\n- The authors introduce a qualitative analysis to effectively filter out low-quality activations, simplifying the subsequent quantitative comparison and improving feature selection efficiency. What’s more, the observed properties are not limited to the discussed models, which can also inspire future work.\n- Extensive experiments demonstrate that the proposed feature selection method significantly outperforms state-of-the-art techniques in various discriminative tasks, validating its effectiveness and generalizability.", "weaknesses": "- As pointed out by the authors, the observations and methods proposed do not necessarily generalize well to recently developed DiT models since their architecture is markedly different from diffusion U-Nets .\n- Some details can be more clarified. For example, in Appenidx C.3, the authors state that “this is also the setting for the quantitative comparison”. In other words, quantitative comparison is conducted on Label-Scarce Segmentation, and the experiments on the other tasks follow this setting. Is this understanding correct? If so, I wonder why the authors select this setting to conduct quantitative comparison. \n- Besides, there are some typos. For example, in the introduction, “a fundamental problem to select” should be “a fundamental problem in selecting”. In the caption of Figure 3, “existing knowledge of other models” should be “existing knowledge about other models”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.  \n\n> **Weakness 1**:\nAs pointed out by the authors, the observations and methods proposed do not necessarily generalize well to recently developed DiT models since their architecture is markedly different from diffusion U-Nets.  \n\n**Response:**\nThanks for this valuable suggestion! Despite this limitation, U-Net-based diffusion models are still being vastly used, so our study can have a broad impact.\nFor more details, please refer to the **global response Question 1**.  \n\n> **Weakness 2:**\nSome details can be more clarified. For example, in Appendix C.3, the authors state that “this is also the setting for the quantitative comparison”. In other words, quantitative comparison is conducted on Label-Scarce Segmentation, and the experiments on the other tasks follow this setting. Is this understanding correct? If so, I wonder why the authors select this setting to conduct quantitative comparison.  \n\n**Response:**\nThanks for the advice. We will try to better clarify details in the refined manuscript. As for the specific question, the reviewer's understanding is correct. We choose to conduct the quantitative comparison on label-scarce segmentation because (i) Experiments on this task take a relatively short time. (ii) This task provides several datasets for scenes of different complexity, allowing us to compare the capability of activations in different scenarios.  \n\n> **Weakness 3:**\nBesides, there are some typos.  \n\n**Response:**\nThanks for the advice. We will pay attention to such typos and fix them in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1vdUUOQYuO", "reviewer": "Reviewer_cCcb", "summary": "Diffusion models have achieved significant success in image generation and show great potential for various discriminative tasks. The authors rethink the foundational problem of feature selection within these models. To this end, they analyze the properties of diffusion models, including asymmetric diffusion noises, in-resolution granularity changes, and locality without positional embeddings, to filter out suboptimal activations. The results demonstrate that their proposed feature selection method outperforms state-of-the-art techniques across multiple discriminative tasks, achieving superior performances.", "questions": "Besides those in weakness, I wonder can we combine multiple features during the qualitative analysis to further improve the performance? And how many features do you select for each image?", "limitations": "Yes, as stated in Sec.7.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper points out three unique properties of diffusion models—asymmetric diffusion noises, in-resolution granularity changes, and locality without positional embeddings—that provide valuable insights into improving discriminative tasks.\n- The authors propose an off-the-shelf solution for feature selection. Besides, the findings and methodologies presented are not limited to the specific models studied (SDv1.5 and SDXL).\n- The paper conducts extensive experiments across multiple discriminative tasks, including semantic correspondence, semantic segmentation, and label-scarce segmentation. The results, as well as the visual illustration, validate the effectiveness of the proposed feature selection method, demonstrating significant improvements over state-of-the-art techniques.", "weaknesses": "-  The authors acknowledge the uncertainty about whether the findings can generalize to newer models like DiT (Diffusion Transformer) due to architectural differences.\n- The qualitative filtering approach proposed is novel, but its scalability and efficiency in very large-scale settings are not fully demonstrated. How do the authors alleviate this problem?\n- The paper focuses on empirical results and qualitative analysis. There is a lack of rigorous theoretical foundation or mathematical proofs to support the proposed feature selection methodology and the identified properties of diffusion U-Nets. A more robust theoretical framework would strengthen the claims made.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.  \n\n> **Weakness 1:**\nThe authors acknowledge the uncertainty about whether the findings can generalize to newer models like DiT (Diffusion Transformer) due to architectural differences.  \n\n**Response:**\nThanks for this valuable suggestion! There are currently still many researchers and casual users using U-Net-based diffusion models. We will leave the study on DiT models to future work.\nFor more details, please refer to the **global response Question 1**.  \n\n> **Weakness 2:**\nThe qualitative filtering approach proposed is novel, but its scalability and efficiency in very large-scale settings are not fully demonstrated. How do the authors alleviate this problem?  \n\n**Response:**\nThe qualitative filtering is mainly conducted through feature visualization. With the prior knowledge of humans, it is relatively easy to do sampling, i.e., only visualizing some activations, for better efficiency. Furthermore, the three discovered properties can help this process be more efficient. More practically, SDXL is one of the current largest diffusion models, and the qualitative filtering for it has been carried out successfully.  \n\n> **Weakness 3:**\nThe paper focuses on empirical results and qualitative analysis. There is a lack of rigorous theoretical foundation or mathematical proofs to support the proposed feature selection methodology and the identified properties of diffusion U-Nets. A more robust theoretical framework would strengthen the claims made.  \n\n**Response:**\nWe follow the utilitarian trend of fellow studies and leave more rigorous theoretical analysis to future work.\nFor more details, please refer to the **global response Question 2**.  \n\n> **Question 1:**\nI wonder can we combine multiple features during the qualitative analysis to further improve the performance?  \n\n**Response:**\nThe major benefit of combining multiple features is that different features might contain complementary information. Therefore, it is better to combine features that are more distinct. If we aim to do this via qualitative analysis, one possible way is to select activations with as distinct color patterns as possible. Note that the colors we can see in the visualization are calculated using PCA analysis with the target dimension as 3, so the colors can reflect the information contained by activations to some extent.  \n\n> **Question 2:**\nAnd how many features do you select for each image?  \n\n**Response:**\nExcept for the Ours-XL-t solution, we select four features per image. This is to ensure the total dimension of features is almost the same as the conventional features [2, 41]. To be specific, the conventional features in our experiment have 3520 dimensions, Ours-v1.5 features have 3520 dimensions, and Ours-XL features have 3840 dimensions. Ours-XL-t does select more features to match the feature amalgamation technique, where 10 or 8 features in total are selected based on the task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EiEJWahhNi", "reviewer": "Reviewer_8YQr", "summary": "This paper highlights the importance of considering a broader range of activations within diffusion models. The authors propose three universal properties of diffusion U-Nets that aid in qualitatively filtering out activations that are clearly sub-optimal. On top of this, the authors can improve the efficiency of feature selection. By leveraging these properties, the proposed method demonstrates superior performance across multiple discriminative tasks, such as semantic correspondence and segmentations. This comprehensive approach to activation selection addresses a fundamental issue in diffusion models, enhancing their applicability and performance in various tasks.", "questions": "I have no more questions.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The authors propose an efficient method for selecting high-quality activations.  With the help of qualitatively filtering out sub-optimal activations before quantitative comparison, the computational costs can be significantly reduced, compared with the prior arts.\n2) The authors present their observations in a clear and organized manner, making it easy for readers to understand the complex concepts and follow the research process. Besides, the main content and the appendix include numerous visual samples and activation visualizations, which help illustrate the properties and effectiveness of the proposed activation selection method.\n3) Extensive experimental results across multiple discriminative tasks validate the superiority of the proposed method over the competitors.", "weaknesses": "W1: This paper could benefit from introducing more notations to clearly define and differentiate between various activations and components within the diffusion models, which would further enhance the clarity of the descriptions.\nW2: While the empirical results are robust, the paper could include more theoretical analysis to provide deeper insights.\nW3: Some more latest related work are commended to be cited:\n[1] Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features, CVPR 2024. [2] DiffSeg: Towards Detecting Diffusion-Based Inpainting Attacks Using Multi-Feature Segmentation, CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.  \n\n> **W1:**\nThis paper could benefit from introducing more notations to clearly define and differentiate between various activations and components within the diffusion models, which would further enhance the clarity of the descriptions.  \n\n**Response:**\nThanks for this good advice. We do use some notations in our codes, but we wanted to make the manuscript look more formal, so the notations were not used in the paper's main body. During refinement, we will try to add these notations back. For example, we use \"up-resolution0-vit0-block1-cross-q\" to represent the activation at cross-attention query, the second basic block, the first ViT, the first resolution, up-stage.  \n\n> **W2:**\nWhile the empirical results are robust, the paper could include more theoretical analysis to provide deeper insights.  \n\n**Response:**\nThanks for your advice! We follow the fellow studies in this direction to take a utilitarian approach and leave more theoretical analysis to future work.\nFor more details, please refer to the **global response Question 2**.  \n\n> **W3:**\nSome more latest related work are commended to be cited.  \n\n**Response:**\nThanks for the advice. These studies will be added to the refined manuscript.  \n\n- [1] is an application of diffusion features, which generates images conditioned on sampled views of 3D models, extracts diffusion features during generation, and unprojects features back to the 3D surface. This work demonstrates that even though features from different views can be inconsistent, the associated features are robust.\n- [2] detects the areas having been inpainted in an image based on segmentation using multiple features, including RGB-based, frequency-based, and noise-based features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UTwAUCIpJM", "reviewer": "Reviewer_xf1j", "summary": "The paper addresses the critical task of enhancing feature selection in diffusion models to improve performance in discriminative tasks such as semantic correspondence, semantic segmentation, and label-scarce segmentation. Previous methods often overlooked many potential activations within diffusion models, leading to suboptimal performance and limitations due to ignoring high-resolution activations and not effectively handling diffusion noises. The authors propose a comprehensive feature selection solution that leverages distinctive properties of diffusion U-Nets, including diffusion noises, in-resolution granularity changes, and locality without positional embeddings, to filter and select the most relevant features. Experimental results demonstrate that their method outperforms state-of-the-art techniques, achieving significant improvements in various metrics across multiple datasets.", "questions": "See the weakness.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper introduces a novel approach to feature selection in diffusion models, focusing on qualitative analysis to filter out suboptimal activations before performing quantitative comparisons. This methodology shifts from the traditional full-scale quantitative comparison, making the process more efficient and potentially more accurate.\n- The authors identify three distinct properties of diffusion U-Nets that are leveraged for feature selection:\n1.  **Asymmetric Diffusion Noises**: The diffusion process introduces unique noises affecting both low- and high-frequency signals.\n2.  **In-Resolution Granularity Changes**: Modern diffusion U-Nets exhibit significant granularity changes within a single resolution due to fewer but larger resolutions.\n3. **Locality without Positional Embeddings**: Self-attention modules in diffusion U-Nets show a new type of locality that enhances activation quality without traditional positional embeddings.\n- The discovered properties means that the findings can be generalized to various diffusion models, providing valuable insights and a solid foundation for future research. This can guide the development of more effective and efficient discriminative models and applications in other fields.\n- The proposed method achieves SOTA performance across multiple discriminative tasks, including semantic correspondence, semantic segmentation, and label-scarce segmentation.", "weaknesses": "- Some more comparisons are recommended. While the authors claim to find unique properties in diffusion U-Nets, it lacks a detailed comparison with traditional U-Nets. A more comprehensive analysis of how these properties differ would provide better context.\n- Some visualizations, such as Figure 3(c) showing positional information in self-attention activations, could be clearer and better explained to enhance understanding of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.  \n\n> **Weakness 1:**\nSome more comparisons are recommended. While the authors claim to find unique properties in diffusion U-Nets, it lacks a detailed comparison with traditional U-Nets. A more comprehensive analysis of how these properties differ would provide better context.  \n\n**Response:**\nWe will add more comparisons against traditional U-Nets during refinement, and here we provide some details. \n\n- Diffusion noises are a special noise type induced by the diffusion process, which inputs partially noisy images to diffusion U-Nets and expects the outputs to be noises. In traditional U-Nets without diffusion process, such as a U-Net for semantic segmentation, the input is image and the output is semantic masks. Therefore, there are no special diffusion noises in the activations of a traditional U-Net. \n- In-resolution granularity change, in theory, also exists in traditional U-Nets. However, in most traditional U-Nets, one resolution only has much fewer network components. In this way, in-resolution granularity change can hardly be observed. \n- Locality without positional embeddings is in fact a unique property compared to typical ViTs instead of traditional U-Nets. Traditional U-Nets are typically comprised of convolutional layers rather than attention layers, so neither locality nor positional information can be seen in traditional U-Nets.\n\nAdditionally, we provide activation visualization from a traditional U-Net for semantic segmentation in the PDF attached to the global rebuttal.  \n\n> **Weakness 2:**\nSome visualizations, such as Figure 3(c) showing positional information in self-attention activations, could be clearer and better explained to enhance understanding of the findings.  \n\n**Response:**\nWe will pay attention to this advice during refinement, and here are some changes we can make. \n\nThe visualization provided throughout the paper is obtained using PCA analysis, reducing the dimension of activations down to 3 and regarding the three dimensions as RGB. Hence, the color of a latent pixel in the visualization can reflect the original information in the activation to some extent. Observe the visualization with an orange circle mark in Figure 3(c), and we can see that the latent pixel on the horse's neck is a light blue color, similar to the pixels to its left that actually represent the background. In contrast, the pixels above the circle are in purple color, though they also represent the horse's neck. Such comparison shows that a latent pixel is more similar to other pixels that are spatially near it than those semantically closer to it, which is the meaning of positional information and locality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear SAC, AC, and reviewers,  

Thank you for your invaluable feedback. Based on your comments, we have revised the details and now offer a global response to some common questions.  

> **Question 1:**
As we have admitted, the conclusion of this study can fail to extend to DiT models. Can this be an important weakness?

**Response:**
Despite the advancements in DiT models, the more conventional U-Net-based diffusion models are still being vastly used by both casual users and researchers. Hence, our study can still benefit many fellow studies.
Moreover, the study on DiT features may require a dedicated study, so we narrow the scope of this paper to more concentrated research.

> **Question 2:**
This study takes a more empirical and qualitative approach, without rigorous theoretical foundation or mathematical proofs.

**Response:**
Theoretical analysis can indeed bring more insights, and we will set this as our future plan. Nevertheless, most fellow studies in this direction take a rather utilitarian approach, placing enhancing the actual performance in the first place. We also follow this trend and argue that the current manuscript has achieved this goal.

---

Please refer to the specific responses below for more information. We will update all these improvements in the next version.

Meta-review: The paper considers the problem of feature selection within diffusion models, analyzing various properties of diffusion models (properties such as asymmetric diffusion noises, in-resolution granularity changes, and locality without positional embeddings), and proposing a feature selection method that outperforms state-of-the-art techniques across multiple tasks.  Empirical results are
detailed and validate the claims.  All reviewers agree with the4 value of the contribution.
Decision: Accept (spotlight)