Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniques
Authors: Benyuan Meng
Qianqian Xu
Zitai Wang
Zhiyong Yang
Xiaochun Cao
Qingming Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models
Representation Learning
Model Property
Primary area: diffusion_based_models
TL;DR: We reveal a hidden yet harmful phenomenon, content shift, in diffusion features, and propose a method to utilize off-the-shelf generation techniques to suppress it.

Abstract: Diffusion models are powerful generative models, and this capability can also be applied to discrimination. The inner activations of a pre-trained diffusion model can serve as features for discriminative tasks, namely, diffusion feature. We discover that diffusion feature has been hindered by a hidden yet universal phenomenon that we call content shift. To be specific, there are content differences between features and the input image, such as the exact shape of a certain object. We locate the cause of content shift as one inherent characteristic of diffusion models, which suggests the broad existence of this phenomenon in diffusion feature. Further empirical study also indicates that its negative impact is not negligible even when content shift is not visually perceivable. Hence, we propose to suppress content shift to enhance the overall quality of diffusion features. Specifically, content shift is related to the information drift during the process of recovering an image from the noisy input, pointing out the possibility of turning off-the-shelf generation techniques into tools for content shift suppression. We further propose a practical guideline named GATE to efficiently evaluate the potential benefit of a technique and provide an implementation of our methodology. Despite the simplicity, the proposed approach has achieved superior results on various tasks and datasets, validating its potential as a generic booster for diffusion features. Our code is available at https://github.com/Darkbblue/diffusion-content-shift.

Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniques
Benyuan Meng Qianqian Xu Zitai Wang*
Zhiyong Yang² Xiaochun Cao Qingming Huang 5,3,7
Institute of Information Engineering, CAS
Schooo of Cyber Security, University of Chinese Academy of Sciences Key Lab. of Intelligent Information Processing, Institute of Computing Technology, CAS
*Peng Cheng Laboratory
5 "School of Computer Science and Tech., University of Chinese Academy of Sciences School of Cyber Science and Tech., Shenzhen Campus of Sun Yat-sen University
Key Laboratory of Big Data Mining and Knowledge Management, CAS mengbenyuan@lie ac. en (xuqianqian, sangzitai /iict ac. en fyangzhiyong21, qmhuang) @ucas. ac cn caoxi aaoxiaochun@mall sysu, edu. cn
Abstract
Diffusion models powerful generative models, and this capability also be applied to discrimination. The inner activations of a pre-trained diffusion model can as features for discriminative tasks, namely, diffusion feature. We discover that diffusion feature has been hindered by hidden yet universal phenomenon that call content shift. To be specific, there content differences between features and the input image, such as the exact shape of a certain object. We locate the cause of content shift as inherent characteristic of diffusion models, which suggests the broad existence of this phenomenon in diffusion feature. Further empirical study also indicates that its negative impact is not negligible even when content shift is not visually perceivable. Hence, we propose to suppress content shift enhance the overall quality of diffusion features. Specifically, content shift is related to the information drift during the process of recovering an image from the noisy input, pointing out the possibility of turning off-the-shelf generation techniques into tools for content shift suppression. We further propose practical guideline named GATE efficiently evaluate the potential benefit of technique and provide an implementation of our methodology. Despite the simplicity, the proposed approach has achieved superior results on various tasks and datasets, validating its potential as generic booster for diffusion features. Our code is available this url.
Introduction
Diffusion models (DMs) [15, 34] are a prevalent family of generative models for various tasks [36, 35, 25]. This strong generative capability be applied to discrimination [21]. Diffusion Feature (DF), a popular approach, extracts inner activations from a pre-trained diffusion model as vision features [1, 50, 60, 57, 37, 62], similarly to how ResNet [14] serves feature extractor. Extracting features with a vastly pre-trained diffusion model grants this approach strong robustness and generalizability. Furthermore, it enjoys the philosophy of hitchhiking, solid research paradigm in the age of large base models [44]: advancements diffusion models can all be transformed into better feature quality. promises this research direction bright future.
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Suppress Content Shift Better Features
Convolutional Features
Content Shift
Off Shelf Generation Techniques
Input
Diffusion Model
Attention Features
Suppress
Content Shift Better Features
Figure 1: Current diffusion features widely suffer from content shift, i.e., content differences between inputs and features. Due to the inherent connection, content shift can be suppressed with off-the-shelf generation techniques.
Have we already obtained satisfying diffusion features? The original role of diffusion models, generation, has provided an inspiring perspective. There is an endless pursuit among AIGC players* for better control over generation [17,58 53, 28]. This implies the inherent difficulty in controlling diffusion models: their generation results may not be exactly the same as intended. Will this property of diffusion models also affect the quality of diffusion features? We visualize some diffusion features in Figure and find that they do contain detail differences from inputs, which might hinder the performance. We name this phenomenon content shift, i.e., content differences between diffusion features and the input image. Although we have just shown an example with very obvious content shift, it is in fact intentionally amplified for observation. Under more practical feature extraction settings, the magnitude of content shift would be significantly weaker for visual perception. In Section 4, however, empirical results show that even in such cases, the negative impact of content shift on discrimination is not negligible, indicating the necessity to suppress it for diffusion features of better quality.
In pursuit of a method to suppress content shift, we need further investigate why this phenomenon exists. We notice in Section 4 that the diffusion backbone reconstructs clean inner representations from noisy inputs in the middle of UNet before predicting noises based on the reconstructed content. The diffusion features we are using are in fact the reconstructed representations, which answers details are
why we can obtain clean features from noisy images. However, since potentially blurred out by noises and then recovered by "imagination", this reconstruction process inherently suffers from the risk of drifting from the original image. Content shift in diffusion features, naturally, is the reflection of such drift during reconstruction. Consequently, to suppress content shift, we need an additional way to directly introduce the original clean image into the reconstruction process and steer it towards the original image. To delight, notice that many off-the-sheit image generation techniques for diffusion models [58, 28, 53] also work by injecting additional information into UNet and thus steering the reconstruction. Through careful evaluation of the effect of generation techniques, we are able to select some techniques that can directly satisfy the goal of suppressing content shift, which eases the implementation and extension of method. To guide the efficient evaluation of generation techniques, we also propose a guideline in Section 5, This method denoted as GenerAtion Techniques Enhanced (GATE) diffusion feature and its effect also shown in Figure 1.
To validate GATE, we implement it (Section 5) by choosing three techniques from few highly popular generation techniques. Since the integrated techniques can benefit from feature amalgamation [24, 62] more than previous approaches, this method is also adopted in our implementation. Although this is a very simple implementation, it achieves impressive performance on various tasks and datasets (Section 6). It demonstrates the effectiveness and potential of GATE, more techniques can still be evaluated, and there will be even more as diffusion models develop.
In summary, the contribution of this work is as follows:
hattps:/civvtaa..

---

To the best of our knowledge, we are the first to reveal and systematically analyze the universal, harmful, yet hidden phenomenon, content shift, in diffusion features.
We point out the possibility of utilizing off-the-shelf generation techniques for content shift suppression and propose the GATE guideline to facilitate technique integration.
Comprehensive experiments two discriminative tasks validate the effectiveness of our method.
2
Related Work
The introduction to diffusion models can be found in [6]. In this section, we solely focus on diffusion features. So far, the research topics in this direction fall into two categories: task exploration and method improvement, i.e., applying the paradigm to different tasks and studying the approach itself for better feature quality, respectively.
Task exploration. There have been attempts on semantic segmentation [1. 50], semantic correspon- dence [57, 37, 22], hyperspectral image classification [62]. domain generalization [9]. training data synthesizing [45, 55], zero-shot referring image segmentation [30], visual grounding [23], novel task involving personalization [47], co-salient object detection [48], and open-worke world segmentation [38].
This extensive and extending list shows the discriminative capability of DMs.
Method improvement. The way to enhance diffusion features can be divided according to the three important factors of a diffusion model: prompt, layer, and timestep. In a basic pipeline, the prompt is manually designed and simple, the layer only refers the activations between convolutional blocks, and the timesteps are manually set. (i) To enhance the use of prompt, a typical improvement is prompt tuning [50. 60, 57, 22], which equivalently fine-tunes the text encoder along with the downstream discriminative task. Another novel method is auto-caption cutoocaptinning [19], replacing manual prompt design with an automatic captioner. (ii) To dig more information out of UNet layers, researchers choose to additionally take attention layers into consideration. However, we find that cross artss-attention is more frequently used [50, 60, 57, 30] while self-attention is less popular [47]. (iii) As for better usage of timesteps, [62] proposes to extract features from many timesteps and dynamically assign weights to them, while [52] employs reinforcement learning for better timestep selection.
Of the two directions, our work aims for better methods instead of new tasks. Furthermore, most existing diffusion feature approaches suffer from content shift, an example of which is the visualization of attention features in Figure 1. Therefore, our GATE can serve as generic performance booster to other diffusion feature approaches, showing its potential for broad application.
3 Preliminaries: Diffusion Feature
Diffusion models consist of a neural network module and a diffusion scheduler. The network is an end-to-end network, which can be formally denoted as ca, where 0 is the parameters. The diffusion scheduler the core of diffusion models. With the scheduler, diffusion models generate images progressively, during which the network module is re-used on each timestep, each time only predicting an incremental noise. Generally, we use a smaller larger timestep to indicate less / more noises. A typical generation process starts from t T (total noises) and ends at t = 0 (clean images).
Next, we will explain how features are extracted using common pipeline for diffusion features, with the visual illustration in Figure 2. Given input image 3xxxx where h, w are height and width, the extraction process includes: (i) A pre-trained VAE encodes the input image into the latent space, inducing E 34xh'xxx xw' as a common practice presented in [34, 33]. (ii) acts as the input of the forward diffusion process, i.e., timestep & 0. As suggested in [1]. it is beneficial
extract diffusion features at non-zero timestep. Following this practice, we set t = 50 and get Ty. (iii) Das along with the timestep and a textual prompt c, sent into the pre-trained diffusion UNet ca, i.e. co ca (xr,t,c) c). The generation techniques selected by our method are also applied here to modify 0, c. (iv) Convolutional and attention features are gathered during the computation of the backbone. For convolutional features, we gather the output activations of each resolution in the upsampling stage of the UNet. For attention features, we obtain the mean value of the similarity maps between query and key in all cross-attentic layers.
3

---

In/Out
ResModule
module (cross-atten.)
Sampler
Sown-Stage
Mid-Stage
Up-Stage
Add naise
via prior ((rlls))
X
Latent image (query)
Prompt (key) horse, high qualitt. best quality. realistic...
The output activations of each resolution the up- Stage
similarity maps
VAE Encoder
average over layers
Attention Feature
Convolutional Feature
Input
Downstream discriminative tasks
Figure 2: The overall process of feature extraction. The original image is first processed into UNet inputs via VAE and noise addition. Afterward, we collect the output activations of each resolution in the upsampling stage convolutional features. the same time, the pross-attentis layers of UNet produce similarity maps, which are averaged over all upsampling layers as attention features.
Dataset
Low Quality
Neutral
High Quality
Horse-21
58.90
59.03
59.38
CIFAR10
91.44
91.21
Figure 3: The averaged results three repeats with quality prompts. Horse-21 (high quality) and CIFARIO (low quality) benefit from prompts closer to the image quality, suggesting the negative effect of content shift at small timesteps.
Exploration of Content Shift
4.1 Impact of Content Shift
qualitatively observe content shift from feature visualization. However, this visualization intentionally amplified for better observation by extracting features at large timesteps. Now we aim examine the impact content shift on quantitative performance under more practical scenarios, ie., when timesteps are small. To this end, we need to toggle the magnitude of content shift in features. Describing the quality the image is widely adopted way control generation. A prompt accurately describing the quality of input images is considered to suppress content shift, while intentionally describing something different from the image will cause more severe content shift. We select the semantic segmentation task [1] as an example of high-quality images and image classification on CIFARIO [20] low-quality images. The results in Figure seem counter-intuitive at first glance, as low-quality prompts surprisingly can improve the performance this in fact complies with the quality input images. Therefore, these results can demonstrate the negative impact of content shift feature quality small timesteps.
In Figure 1,
CIFAR 10, but

---

(c) More Noisy Early Activations
(d) Recovered Clean Activations
Original Image
Noisy Input
Output Noises
Figure 4: Content shift is caused by the reconstruction process within diffusion model activations. This visualization consists of activations obtained during single network forward pass.
4.2 Cause of Content Shift
After the negative impact of content shift is confirmed, we next aim to find its cause. Unlike more conventional feature extractors such ResNet [14], the inputs diffusion models the original image (Figure 4(a)), but its noisy version (Figure 4(b)), as enforced by the diffusion process [15]. The early layers of diffusion models even further add more noise the feature maps (Figure 4(c)). However, the diffusion UNet gains the ability from vast pre-training to reconstruct clean inner representations from noisy inputs (Figure 4(d)), roughly
the middle of the UNet structure. Additionally, the shortcut structures in UNet also help the reconstruction by passing some high-freyuency details. Afterward, the diffusion UNet will further predict noises based on the reconstructed representations (Figure 4(e)).
Notably, the diffusion features are using are in fact the reconstructed representations, which answers why clean diffusion features can be obtained even though the inputs are noisy. Despite the reconstruction ability, however, many high-frequency details are potentially blurred out by input noises, and thus their reconstruction mostly based on "imagination". This leads to possible drift from the original image during reconstruction [7]. Naturally, the content shift phenomenon in extracted diffusion features reflects the drift during reconstruction. Consequently, content shift is an inherent characteristic of difflusion models and diffusion features, which suggests its broad existence across models and timesteps.
5
Suppression of Content Shift
5.1 Utilization of Generation Techniques
According to the cause of content shift, we need to steer the reconstruction process back to the original image to suppress content shift. While it is viable to design new methods for this purpose, we find also possible and more efficient to adopt off-the-shelf generation techniques. Specifically, inherent characteristic of diffusion models, content shift affects not only features but also the original generative purpose of diffusion models. Hence, there have been techniques for generation that are already capable of toggling content shift by steering reconstruction [7, 58]. For example, ControlNet [58] introduces an additional reference image and steers reconstruction by directly modulating activations, pushing the reconstrueted representation towards the reference image. It inspires us to utilize ControlNet a different way: we can set the same input image simultaneously the reference image, enforcing the recovered image to be more similar the input one and thus suppressing content shift. Similarly, it is possible to adopt other generation techniques to suppress content shift.
Furthermore, utilizing off-the-shelf generation techniques is more consistent with the intuition of diffusion feature than designing new method. Specifically, this field is dependent on the generative capability of diffusion models, and thus is important to stay updated with new advancements in the generation field. Utilizing techniques from the generation field helps with this goal, while method that newly designed solely for diffusion feature might hinder Considering it, we decide not to devise new methods, but to develop more detailed guidelines for suitably integrating these off-the the- generation techniques.
We next illustrate the guidelines for utilizing off-the-shelf generation techniques. In the field of generation, the abundant techniques may not all be suitable for suppressing content shift, suggesting the necessity of examining the effect of given technique feature quality. Although it is possible
5

---

Img2lmg Generation
Extract Features with Techniques
Step 1.
Diffusion Model
Input
Referenic
Img2lmg Generation
Step
Diffusion Model
Technique
Input
Evaluation
Evaluation Similar->
Step
versus
Evaluation
Reference
Reference is More Similar-
Feature Amalgemetion
Output
GATE Guideline to Evaluate Techniques
Implementation GATE
Figure 5: Overview of the GATE guideline and our implementation. GATE evaluates if technique can suppress content shift based on the result of Img2Img generation. If a technique makes the result more similar to the input, it is considered be potentially helpful. We further implement GATE by choosing three off-the-shelf generation techniques and amalgamating features obtained with different combinations of the techniques.
to make empirical and quantitative examinations on discriminative tasks, a more efficient evaluation protocol would be preferable. To this end, we propose the GATE guideline for technique evaluation, which presented in Figure 5 along with other components of the framework. To avoid having to integrate technique into the feature extraction pipeline before knowing its potential, utilize Img2lmg generation as an alternative for evaluation:
Gather Img2Img generation result as the reference image, done with high repainting strength to amplify content divergence for observation.
(ii) Apply the technique to evaluated and perform another Img2lmg generation. the technique requires parameters, they should be set such that the output could be more similar to the input.
(iii) the new output is more similar to the input than the reference image, the technique is considered able suppress content shift diffusion features.
5.2
Quantitative Evaluation
We have also developed a quantitative metric for evaluating generation techniques. We set feature extracted at 0 as reference FEATTer E cxx it is less affected by noises. Then use the Laplacian operator to evaluate the contour difference between feature FEAT and the reference:
xxx an [|Laplacia((FAATTar i,5)l,, - |Laplaccia i E (-o0, 2.J
diff
Σ
(I)
We then set a feature with stronger content shift as an anchor and compare di fanchor with other features.
diff)
Score fonooonheon
(2)
Score = means an exact match, and a smaller value indicates more shift. In this way, we can measure the extent of content shift in extracted features using different generation techniques and thus evaluate the suppression effect of techniques. Noticeably, the evaluation of this quantitative metric be well approximated by the previously proposed qualitative evaluation, recommend the qualitative evaluation efficiency is desired.
5.3 Selected Generation Techniques
We select three generation techniques our implementation of GATE, Their Img2Img generation results are provided in Figure 6. Integrating the techniques only slightly impacts efficiency, which will be discussed in Appendix A along with technique details. Additionally, we analyze two failed

---

Input
Reference
Prompts
ControlNet
LoRA
Figure 6: Img2lmg generation results according to the GATE guideline, validating the potential of the three selected techniques for suppressing content shift.
Different Timesteps Basic Model
Different Combinations of Techniques
Figure 7: The first row shows features extracted at different timesteps. The second row is from different combinations of generation techniques and shows stronger diversity.
techniques in Appendix B, which might help better understand how generation techniques influence content shift.
description of expected image content in natural language. For example, "a single horse running in a sports field, with a well-equipped rider on its back, high quality, highly realistic, masterpiece". Modern diffusion models are inherently trained generate images conditioned on the given prompts [34]. Hence, by describing the content of the input image in prompts, is possible steer the reconstruction stay close to the input. The result of fine-grained prompts in Figure 6 is not significantly better than the reference, but this technique enjoys the advantage of being built-in function of diffusion models, requiring code modification.
Fine-Grained Prompts. Prompts
ControlNet. Quite often, the control via prompts is too ambiguous. This when ControlNet [58] be helpful. ControlNet is plug-in module for diffusion models, designed to take an additional control image input and push the reconstructed representation toward the control image. ControlNet presents outstanding control effect generation among the three techniques, shown in Figure 6, implying its potential for suppressing content shift.
LoRA. LoRA [17] is an efficient replacement for model fine-tuning, which bears similar effect to fine-tuning, thus showing to be highly effective for capturing image styles. By injecting additional knowledge of image styles of the given dataset into model weights, LoRA enables reconstruction to end with stronger similarity to the input image. From Figure 6, we can observe that although LoRA does not provide strong control effect ControlNet, can significantly promote image quality, which is believed to bring some unique advantages.
5.4 Feature Amalgamation
The three techniques above are able to improve feature quality individually and can be applied simultaneously for stronger suppression effects. While this can lead single high-quality feature, it is common practice in previous diffusion feature approaches to amalgamate multiple features for further improvement [1, 24, 62, 29]. The conventional way extracts features at different timesteps to obtain more diverse information. We find that, compared to the conventional amalgamation of timesteps, the amalgamation of different combinations of generation techniques can bring stronger diversity, indicated in Figure 7. Therefore, we additionally amalgamate features obtained with
7

---

various generation technique combinations to harness further enhancement. More details of how feature amalgamation is implemented are provided in Appendix A.
Experimental Validation
6.1 Experimental Settings
Task & Dataset. Typically, diffusion feature studies [1, 50, 60] prefer fine-grained pixel-level tasks for evaluation. Following this practice, we select three tasks for experiments: semantic correspondence using SPair-7Ik [26] dataset, label-scarce semantic segmentation using Bedroom- 28 [54] and Horse-21 [54] datasets, and standard semantic segmentation using ADE20K [61] and CityScapes [5] datasets.
Evaluation Metrics. (i) For semantic correspondence, PCK (CKK@..lmgg(" and PCK@0.1bbor (t) are used, following the addpted protocol reported in [26]. (We omit @0.1 to save some space in Table I.) These two metries mean the percentage of correctly predicted keypoints, where a predicted keypoint is considered to be correct if it lies within the neighborhood of the corresponding annotation with a radius of 0.1 x mar(h, w). For PCK @0. /PCK @0. Inhxx h, w denote the dimension of the entire image/object bounding box, respectively. (ii) For semantic segmentation, we use mloU metric, which the mean over the IoU performance across all semantic classes [12]. For each image, IoU (Intersection over Union, 1) is defined by #(overlapped pixels between the prediction and the ground truth) / #(union pixels of them). In addition, we also use aAcc and mAcc, where aAce is the classification accuracy of all pixels and mAcc averages the accuracy over categories.
50. When ControlNet is applied, except for standard semantic segmentation, we additionally start multi-step denoising from = 60. For feature amalgamation, we extract multiple convolutional features and one attention feature per image:
Feature Extraction. All tasks extract features
(i) Semantic correspondence: We obtain six in total convolutional features using individual fine- grained prompt, ControlNet, and LoRA techniques, and one attention feature using a prompt including all object categories, with ControlNet and LoRA.
(ii) Label-scarce semantic segmentation: We obtain convolutional feature using fine-grained prompts, one convolutional feature using ControlNet, and one (Bedroom-28) to two (Horse- 21) features using different LoRA weights. One attention feature is extracted with all three techniques applied.
(iii) Standard semantic segmentation: One convolutional feature is obtained using only fine-grained prompts and two more are extracted additionally with ControlNet and different LoRA weights. One attention feature is extracted with all three techniques applied.
Notably, the ADE20K dataset for standard semantic segmentation contains images of varying scenes, which can test how well the fine-grained prompt technique can generalize in this scenario. To this end, we use a prompt that can cover different scenarios: highly realistic photo of the real world. can be an indoor scene, various scenes for generalizability and describes image quality for fine mne-grained effect.
outdoor scene, photo of nature. high quality". This prompt covers
For more experimental settings, including more detailed feature extraction methods and implementa- tion details, please refer to Appendix C.
6.2 Comparison with SOTA
The experimental results are shown in Table 1 and Table 2. For most SOTA competitors, we borrow the reported results from their original studies. However, MaskCLIP [8] and ODISE [50] only provide results on ADE20K and it is hard to extend their implementations to CityScapes, on CityScapes are missing. Furthermore, the original results reported by VPD [60] are based on full-scale fine-tuning of diffusion UNet, which is not fair we do not train the diffusion model. Therefore, we re-evaluated VPD with the diffusion UNet frozen and reported our results.
their results
Semantic Correspondence. It is pity that the related studies do not perform experiments under exactly the same setting. For fairness, we mainly compare GATE against a baseline method, which uses one feature extracted without any technique, under unified setting. For reference, still
8

---

Table The results of semantic correspondence (left, PCK@0.11 and label-scarce semantic segmen- lation (right, mloUf). Red for the best result and blue for the runner-up.
Method
PCKing t
PCKahox f
Method
Bedroom-25 28
Horse-21 21
DINO
51.68
41.04
ALAE
20.0 + 1.0
Non-DF
DHPF
55.28
42.63
GAN Inversion
13.9 + 0.6
17.7 + 0.4
GAN Encoder
22.4 1.6
26.7 0.7
DIFT
52.90
SWAV
41.0 +
51.7
DF
DHF
72.56
64.61
SwAVw2
42.4
54.0
MAE
45.0 15.0
6.44
Baseline
m 61.15
51.66
DatasetGAN
2.7
1.4
conv 73.96
65.74
DatasetDDPM
47.9
60.8
DDPM
4..4
6..0
n 64.47
55.72
GATE
conv 76.60
69.10
GATE
53.1 2.7
67.2 +
Table 2: Results the two standard semantic segmentation datasets, ADE20K and CityScapes. Red for the best result and blue for the runner-up.
ADE20K
CityScapes aAcet
Category
Method
mloUf
aAcc
mAcef
mloUf
mAcef
MaskCLIP
23.70
SOTA
ODISE
29.90
VPD
37.63
79.16
50.08
55.06
90.14
68.96
Ours
GATE
40.51
79.68
54.90
64.20
92.83
76.98
provide the results from four state-of-the-art methods: DINO [3] and DHPF [27] non-DF methods, as well DIFT [37] and DHF [24] as DF methods. The results are in the left part of Table 1.
Semantic Segmentation. The strong generalizability of a pre-trained diffusion model can ensure good discriminative performance even when labeled training data is scarce. For this scenario, we show the results in the right part of Table 1. The SOTA diffusion feature method for this setting, DDPM [1], serves the major competitor. We also include other segmentation methods: DatasetGAN [59], DatasetDDPM, MAE [13], SwAV [2]. GAN Inversion [40], GAN Encoder,
VDVAE [4], and ALAE [32]. We further validate our method on the more common setting of semantic segmentation using standard datasets: ADE20K [61] and CityScapes [5], with the results
presented Table The competitors are MaskCLIP [8]. ODISE [50], and VPD [60]. where VPD is re-evaluated with the diffusion model frozen for fairness.
Generally, GATE outperforms competitors by large margins all datasets. It demonstrates that previous diffusion feature approaches have been hindered by content shift, and GATE has a promising application generic booster for feature quality.
6.3 Qualitative Analysis
In Figure 8, we provide feature visualization for qualitative analysis of GATE. The visualization is obtained using PCA analysis, reducing the channels of features which are regarded as RGB for visualization. We can observe: (i) The attention features become clearer and closer to the input image when more generation techniques are applied according to GATE, showing the suppression effect on content shift. (ii) Notably, for the second image where person is riding horse, the baseline attention feature fails to follow the instruction, i.e., attending only to the horse and ignoring the person. In contrast, generation techniques applied according to GATE help attention features attend to the correct object. (iii) From convolutional features, we can see the application of generation techniques brings stronger diversity.

---

Baseline
Fine RineGGriined Prompts
ControlNet
LoRA All Combined
Convolutional Features
Attention Features
Convolutional
Features
Attention
Festures
For Attention:
Focus Horse
Ignore Rider
Convolutional Festures
Attention Festures
mloU-58.13
mloU-59.47
mllU+=5.44
mloU-59.43
mloU-59.73
Figure 8: Effect of GATE without feature amalgamation. Images with various scenes are shown for generalizability. the second image, attention feature is asked to focus on the horse and ignore the rider. The mloU performance is single Horse-21 split, with red/blue for the best/worst.
Ablation Study: Effect without Feature Amalgamation
For ablation study, we aim to evaluate the effect of selected techniques without feature amalgamation. The discriminative performance is shown at the bottom line of Figure 8, which is obtained on single Horse-21 split instead of five random repeats for faster evaluation. We can observe: (i) Every individual technique can improve feature quality over baseline. (ii) When multiple techniques are applied simultaneously, stronger improvement can be obtained. This demonstrates that all three selected techniques can benefit feature quality, and their benefits can be combined together.
7 Conclusion and Future Work
In this paper, reveal phenomenon named content shift that has been causing degradation in diffusion features. Based on the analysis of its cause, we propose to suppress with off-the-shelf generation techniques, which allows hitchhiking the advancements in generative diffusion models. This approach, while enjoying simplicity, is experimentally demonstrated to be generically effective.
However, the effectiveness of GATE relies on the selected techniques, for which we propose both a qualitative evaluation guideline and a quantitative metric. Though we selected three effective techniques and reported failed cases, there still is more to explore, which might potentially lead to more effective implementations. Furthermore, we only experimented with three tasks, so the full potential of GATE might remain uxpereex.lored
10

---

Acknowledgments
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008. U2IB2038, U23B2051, 1, 61931008, 62122075, 62025604, 62206264, and 92370102, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680000, in part by the Innovation Funding of ICT. CAS under Grant No. E0.0000000, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX20240384.
References
[1] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In The Tenth International Conference on Learning Representations, 2022. 2022.
[2] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, In Annual Conference on Neural Information Processing Systems, 2020.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in superviee vision transformers. In IEEE/CVF International Conference on Computer Vision, ICCV, pages 9630-9640, 2021.
[4] Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In 9th International Conference on Learning Representations, 2021.
[5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bertt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3213-3223, 2016.
[6] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
[7] Giannis Daras, Yuval Dagan, Alex Dimakis, and Constantinos Daskalakis. Consistent diffusion models: Mitigating sampling drift by learning to consistent, Annual Conference on Neural Information Processing Systems, 2023.
[8] Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskclip. In International Conference Machine Leurning, ICML, volume 202 of Proceedings of Muchine Learning Reseurch, pages 8090-8102, 2023.
[9] Rui Gong, Martin Danelljan, Han Sun, Julio Delgado Mangas, and Luc Van Gool, Prompting diffusion representations for cross-domain semantic segmentation. CoRR, abs/2307.0211 2023.
[10] Rui Gong, Martin Danelljan, Han Sun, Julio Delgado Mangas, and Lue Van Gool. Prompting diffusion representations for cross-domain semantic segmentation. CoRR, abs/2307 02//2307.02133 0218,, 2023.
[11] Boyu Han, Qianqian Xu, Zhiyong Yang, Shilong Bao, Peisong Wen, Yangbangyan Jiang, and Qingming Huang. Aucseg: Auc-oriented pixel-level long-tail semantic segmentation, 2024.
[12] Shijie Hao, Yuan Zhou, and Yanrong Guo. A brief survey semantic segmentation with deep learning. Neurocomputing 406:302-321, 2020.
[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In JEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 15979-15988. 2022.

---

[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Annual Conference on Neural Information Processing Systems, 2020.
[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abs/2207.12598, 2022.
[17] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022.
[18] Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolu- tions. In Annual Conference Neural Information Processing Systems, pages 10236-10245, 2018.
[19] Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimarães, and Pietro Perona. Text-image ali alignment for diffusion-based perception. CoRR, abs/2310.00031 2023.
[20] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[21] Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In JEEE/CVF International Conference on Computer Vision, pages 2206-2217, 2023.
[22] Xinghui Li, Jingyi Lu, Kai Han, and Victor Prisacariu. Sd4match: Learning to prompt stable diffusion model for semantic matching. CoRR, abs/2310.17569, 2023.
[23] Xuyang Liu, Siteng Huang, Yachen Kang, Honggang Chen, and Donglin Wang. Vgdiffzero: Text-to-image diffusion models can be zero-shot visual grounders. CoRR, abs/2309.01 141, 2023.
[24] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Annual Conference on Neural Information Processing Systems, 2023.
[25] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wa, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic diffcrential equations. In The Tenth International Conference Learning Representations, 2022.
[26] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. CoRR, abs/I 900..199....1 10543, 2019.
[27] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Learning to compose hypercolumns for visual correspondence. In Computer Vision ECCV 2020 16th European Conference, volume 12360, pages 346-363. Springer, 2020.
[28] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Hightt AAAI Conference Artificial Intelligence, pages 4296-4304, 2024.
[29] Soumik Mukhopadhyay, Matthew Gwilliam, Yosuke Yamaguchi, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Tianyi Zhou, and Abhinav Shrivastava. Do text-free diffusion models learn discriminative visual representations? CoRR, abs/2311.17922 2023.
[30] Minheng Ni, Yabo Zhang. Kailai Feng, Xiaoming Li, Yiwen Guo, and Wangmeng Zuo. Ref-diff: Zero-shot referring image segmentation with generative models. CoRR, 0abb/2200..16777 2023,
[31] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. CoRR, abs/2306.1 14824, 2023.
12

---

[32] Stanislav Pidhorskyi. Donald A. Adjeroh, and Gianfranco Doretto. Adversarial latent au- toencoders. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14092-14101, 2020.
[33] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307.01952, 2023.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10674-10685, 2022.
[35] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2.500-22510, 2023,
[36] Yang Song, Jascha Sohl- Doch--Dicsttin, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Reprexentations, 2021.
[37] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. In Annual Conference on Neural Information Processing Systems, 2023.
[38] Lv Tang, Peng- Tao Jiang, Haoke Xiao, and Bo Li. Towards training-free free open- open-world segmenta- tion via image prompting foundation models. CoRR, abs/2310. 2023.
[39] Kashvi Taunk, Sanjukta De, Srishti Verma, and Aleena Swetapadma. A brief review of nearest neighbor algorithm for learning and classification. In 2019 international conference on intelligent computing and control systems (ICCS), pages 1255-1260 IEEE, 2019.
[40] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Trans. Graph., 40((4) 11:331114 2021.
[41] Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Ope- nauc: Towards auc-oriented open-set recognition. In Annual Conference on Neural Information Processing Systems, 2022.
[42] Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Optimizing partial area under the top-k curve: Theory and practice. IEEE Trans. Pattern Anal. Mach. Intell., +5(4))50533500)) 2023.
[43] Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. A unified generalization analysis of re-weighting and logit-adjustment for imbalanced learning. In Annual Conference on Neural Information Processing Systems, 2023.
[44] Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. A brief overview of chatgpt: The history, status quo and potential future development. IEEE CAA J. Autom. Sinica, 10(5):1 1at-1122211336 2023.
[45] Weijia Wu, Yuzhong Zhao. Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. In Annual Conference on Neural Information Processing Systems, 2023.
[46] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu. Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. In Annual Conference on Neural Information Processing Systems, 2023.
[47] Changming Xiao, Qi Yang, Feng Zhou, and Changshui Zhang. From text to mask: Localizing entities using the attention of text-to-image diffusion models. CoRR, abs/2309.04109 2023.
[48] Haoke Xiao, Ly Tang, Bo Li, Zhiming Luo, and Shaozi Li. Zero-shot co-salient object detection framework. CoRR, abs/2309.00 094999 2023.
13

---

[49] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. Computer Vision ECCV 2018 15th European Conference, volume 1 1120, pages 432 448, 2018.
[50] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9966, 2023.
[51] Shiqi Yang. Atsushi Hashimoto, and Yoshitaka Ushiku. A critical look at the current usage of foundation model for dense recognition task. CoRR, abs/2307.02862, 2023.
[52] Xingyi Yang and Xinchao Wang. Diffusion model as representation learner In JEEE/CVF International Conference on Computer Vision, pages 18892-18903, 2023.
[53] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. CoRR, abs/2308.06772 06721, 2023.
[54] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365, 2015.
[55] David Junhao Zhang, Mutian Xu, Chuhui Xue, Wenqing Zhang. Xiaoguang Han, Song Bai, and Mike Zheng Shou. Free-atm: Exploring unsupervised learning on diffusion-generatee images with free attention masks. CoRR, abs/2300.06639, 2023.
[56] David Junhao Zhang, Mutian Xu, Chuhui Xue, Wenqing Zhang, Xiaoguang Han, Song Bai, and Mike Zheng Shou. Free-atm: Exploring unsupervised learning on diffusion -ennuriin--eeerteedd images with free attention masks. CoRR, abs/2308.06633, 2023.
[57] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements DINO for zero-shot semantic correspondence. In Annual Conference on Neural Information Processing Systems, 2023.
[58] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to- image diffusion models. In IEEE/CVF International Conference on Computer Vision, pages 3813-3824, 2023.
[59] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, (ean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In IEEE Conference on Computer Vision and Pattern Recognition, pages 10145- 10155, 2021.
[60] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. JEEE/CVF International Conference on Computer Vision, pages 7706-57166. 2023.
[61] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5122 5130, 2017.
[62] Jingyi Zhou, Jiamu Sheng, Jiayuan Fan, Peng Ye, Tong He, Bin Wang, and Tao Chen. When hyperspectral image classification meets diffusion models: An unsupervised feature learning framework. CoRR, abs/2306.00 08964, 2023.
14

---

A Implementation Details and Efficiency Concerns
In this section, we will explain the implementation details of each selected technique and feature amalgamation. Further discussion will also be provided on how these techniques manage to suppress content shift and the efficiency impact of our implementation.
A.1 Fine-Grained Prompts
Fine-grained prompts can be integrated into diffusion feature very simply. We only need to replace the simple and short prompts, such "a photo of horse" in most DF approaches [37, 9, 45, 55, 21], with more complex ones. Since prompts are a built-in function of current diffusion models, the only required integration of fine-gratee fine-grained prompts is to generate these prompts. To fulfill our goal, we need detailed descriptions of the image content, and many image captioning models can serve this purpose, such as 2osmos-2 [31]. It is also possible manually design prompts that can cover various images, which is found to be almost as effective as auto captioners. When this approach is used, our observation is only in the training convolutional features, attention features require more consideration. To be specific, using specific prompts for each image causes inconsistency in the structure of attention features. To fill this gap, we always use manual and fixed prompt for attention features, even when convolutional features utilize auto-captioners. This fixed prompt is designed by observing images and describing the common objects. For example, "bedroom, a bed, some bedroom furniture, lights, a door, ceiling, floor, walls, pillow, quilt, chair and table, window, in good quality".
avoid data leaks. While this is effective for
Regarding the efficiency of integrating fine-grained prompts, we do not make any changes to the original feature extraction pipeline but only introduce the overhead of image captioning. Theoretically, this is still linear complexity. In practice, the actual time consumption depends on what auto-captioner is used. Moreover, these prompts can be re-used for the extraction of many groups of features, further reducing the proportion of overhead in the total time.
A.2 ControlNet
ControlNet requires a parameter, the control signal, which should be set as the input image itself to suppress content shift. More specifically, the control image ControlNet requires not an ordinary image, but a specially processed one, such as depth image or a canny image. Among all control image types, we find that canny images are exceptionally efficient because their process does not use any neural network model. Therefore, we use canny the sole control type for feature extraction.
Although the simple usage is already yielding satisfying results, we find an additional way to further exploit ControlNet. It is reported that taking multiple denoising steps before feature extraction can make the attention feature less blurry [51]. However, taking multiple denoising steps not a common practice for diffusion features, as it also brings severe performance drops caused by content shift. With ControlNet, it becomes possible to harness this good property without failing to suppress content shift. We only do this for attention features, so convolutional features are still extracted in a one-step manner.
Regarding efficiency, processing an image into a canny image takes a very small overhead, So the major overhead introduced by ControlNet should be extracting attention features with multiple denoising steps. Whilst this does take obviously longer time than one-step feature extraction, its impact limited due to two factors: (i) We empirically find that the optimal effect comes with less than 10 denoising steps. If more steps are taken, attention features will meet unacceptable content shift even with ControlNet applied. (ii) Different from the diversity effect in convolutional features, attention features do not see obvious enhancement when amalgamated. Therefore, we only extract one attention feature per image and concatenate it to the amalgamation result of convolutional features. Considering the two factors, taking multiple denoising steps does not in fact contain many steps and its use is relatively rare in the overall feature extraction process. Thus, the efficiency impact of integrating ControlNet is small.

---

A.3 LoRA
The integration of LoRA is also simple. Integrating LoRA into feature extraction only requires switching the base model weights to LoRA weights, so the major effort might be training a good LoRA weight. Fortunately, the community has offered many handy tools for ordinary users, such as kohya_ss", which can be directly utilized for our purpose. Following the tradition from the community, we choose 30 random images from the training set to train a LoRA weight. Specifically, we choose the LoCon type of LoRA, which additionally inserts trainable parameters between convolutional layers, and follow the corresponding guides for training.
The major overhead introduced by LoRA is its training. As a popular tool for AIGC players, the training for LoRA is feasible even on personal computers, showing its efficiency. In our experiments, training one LoRA weight usually takes less than 30 minutes. Moreover, the trained LoRA weights can be re-used, further reducing the proportion of overhead in the feature extraction process.
A.4 Regularized Weight Assignment for Feature Amalgamation
Feature amalgamation can be accomplished by simply concatenating different features together. However, this is found suboptimal, leading to superior strategy for feature amalgamation, which is weight assignment [24, 62]. Roughly, it assigns weights to each feature and amalgamates the features via linear combination according to the weights. We next formalize this process. We have b feature maps rı, 12, ...... rh E wxx for each image, with c, w. h being channel, width, and height, respectively. Weight assigners are simple MLP CNN networks parameterized by 0 and can be denoted as (=) Considering that the downstream task might benefit from various perspectives, there are assigners. Assigning weights wi is denoted as:
w = fr;;;)) E R, s.t. Σ =1j=
(3)
Afterward, we take the linear combinations of features and concatenate them the final feature:
r = Concat
1
wPr)
We notice that this weight assignment strategy is suboptimal. To be specific, the assigners tend to converge to a trivial solution, where they assign weights almost equally to all features and different assigners share similar prediction. To tackle this issue, introduce two extra regularization terms. The first one encourages sparsity, meaning the assigners should concentrate on only a few features. The second one, named diversity, promotes a larger discrepancy between the predictions of different weight assigners. Formally, the loss function is:
fin = E 11
Σ 1=1
2
72 2
[ w* 2
(5)
kØj
where E is the original loss of the downstream task; and 72 are the hyper-parameters of the regularization terms; w == (wf, wg.... , w² visualize the direct effect of the proposed regularization on weight distribution.
and || |la denotes the fn-nrmm. Appendix D.5 will
Since attention features do not show the same diversity effect as convolutional features, the two types of features are processed differently. To be specific, obtain many convolutional features but only one attention feature per image. Feature amalgamation conducted among convolutional features, while the attention feature is concatenated to the final amalgamation result.
Next, we aim to explain why the two proposed regularization terms can achieve their design purpose. Note that the weights of all features are restricted to sum up to 1, which means the fnomr of w always For a vector with a fixed f fnnormm, a larger fg-norm means concentrating on less axes among all dimensions. Therefore, to promote a larger norm.. As for diversity regularization, it simply measures and promotes the discrepancy among the predictions of different weight assigners.
use the negative value of its f-romm as sparsity regularization
Phttps://githur httpa//....//maaaaa//aa/eeaaeeee

---

Input
Reference
Classifier- Free Guidance
Figure 9: Img2lmg generation results of classifier-fres guidance according to the GATE guideline, showing obvious effect for suppressing content shift.
Table 3: Experimental results comparing the influence on suppressing content shift ControlNet and IP-Adapter.
Method
mloU
DDPM
65.0 0.8
GATE (Only IP-Adapter)
65.8 + 1.3
GATE (Only ControlNet)
66.2 1.2
GATE (Full)
67.2 :: 1.1
Compared with previous weight assignment approaches [24, 62]. we only add two regularization terms, thus introducing negligible overhead. Weight assignment itself does trade efficiency for performance. However, since this approach has already been adopted in previous approaches, think it is rational to only consider the additional efficiency impact of our proposed regularization terms.
B Classifier-Free Guidance and IP-Adapter: Failed Case
Classifier-free guidance [16] is a widely adopted generation technique in current diffusion models. Its design purpose is sacrifice diversity for better fidelity, similar to low-temperature sampling [18] in other generative models.
We first show its Img2Img generation results according to the GATE guideline in Figure 9. It is clear that it makes little difference to the similarity between outputs and the input, already suggesting that it lacks the potential for suppressing content shift. Since it can also be implemented simply, we integrate into feature extraction as well and actually examine its effect, leading to the same conclusion the previous quick evaluation following the GATE guideline.
Why classifier-free guidance unable suppress content shift? It designed to gain fidelity the cost of diversity. While better fidelity is irrelevant to content shift, it might be helpful to lower the diversity reconstruction given the blank caused by noises. However, the less diverse reconstruction still not guaranteed be centered on the original content. Therefore, even though classifier free guidance can reduce diversity, content shift will not suppressed. We hope this failed case can provide more insights into how generation techniques affect content shift and how to select them.
IP-Adapter is generation technique designed for image variation, which shares similar architecture to ControlNet. By inputting images, IP- -d-ateee helps generate new images with some elements taken from the reference one. Since the goal of IP-Adapter is image variation instead of strict control, is less effective in mitigating content shift than ControlNet. The weaker effectiveness of AP-Adepeer experimentally demonstrated on the Horse-21 dataset, as shown in Table IP-Adapter is not entirely ineffective for content shift, but less effective than ControlNet and thus not included in our implementation.
C Experimental Details
C.1
Implementation Details
For the diffusion model to extract features, we choose Stable Diffusion v1.5 [34] to be consistent with SOTA competitors. For semantic correspondence, two settings examined: (i) directly performing

---

nearest neighbor algorithm [39] (nn), (ii) adding an extra convolutional layer after each assigner (com)). For label-scarce semantic segmentation. the downstream model follows the framework in [1], keeping nyper-parameter, unchanged. For standard semantic segmentation, we instead use UPerNet [49] as the downstream model. 71 and 7/2 the hyper-parameter for the proposed regularization terms, are tuned for each dataset. Choosing the number of weight assigners is mainly tradeoff between efficiency and performance (Appendix D.4). We use less than three based on our resources.
C.2 Feature Extraction Details for Each Task
C.2.1 Semantic Correspondence
Convolutional features include two features per setting under different random seeds:
(i Two with fine-grained prompt "a photo of acroplane, bicycle, bird, boat, bottle, bus, car, chair, cow, dog, horse, motorbike, person, potted plant, sheep, train, tv monitor, high quality, best quality, highly realistic, masterpiece, high resolution".
(ii) Two with LoRA.
(ii) Two with Nen
Attention features are obtained using a prompt including all object categories, with ControlNet (denoising from = 60) and LoRA also applied.
C.2.2 Semantic Segmentation on Horse-21
Convolutional features include:
(i) A feature using fine-grained prompt "a horse, high quality, best quality, highly realistic, masterpiece".
(ii) A feature using ControlNet, with simple prompt "a horse".
(iii) Two features using different LoRA weights and a simple prompt. The first LoRA weight is trained to generate high-quality images, while the second LoRA weight is trained until it slightly overfits.
Attention features are obtained using prompt "a photo of single horse running in a sports field, with a well-equipped rider on its back, seems they are competition, high quality, best quality, highly realistic, masterpiece", with ControlNet and the first LoRA weight applied. Additionally, we concatenate the output feature of amalgamation with a feature extracted using DDPM [1].
C.2.3 Semantic Segmentation on Bedroom-28
Convolutional features include:
(i) feature using fine-grained prompt "a photo of a tidy and well-designed bedroom". It is found that quality prompts such as "high quality" will be interpreted as the quality of the room instead of the image, such prompts are not utilized.
(ii) A feature using ControlNet, with simple prompt "a bedroom".
(iii) One feature using LoRA weight, which is moderately trained to generate high-quality images.
Attention features are obtained using a prompt "bedroom, a bed, some bedroom furniture, lights, a door, ceiling, floor, walls, pillow, quilt, chair and table, window, in good quality". ControlNet and LoRA are also applied. Additionally, we concatenate the output feature of amalgamation with feature extracted using DDPM [1].
C.2.4 Semantic Segmentation on ADE20K
Convolutional features include:
(i) One feature using only fine-grained prompt "a highly realistic photo of the real world. It can be an indoor scene, or an outdoor scene, or a photo of nature. high quality" plus all category labels.

---

Table Examining GATE on SDXL features. The best result is marked red. For Baseline, three basic feature maps are used. For other settings, one or two basic feature maps are replaced with technique-enhanced ones, keeping the total number of features unchanged.
Category
Basic
Prompt
ControlNet
mloUt
Baseline
53.40
53.61
Individual
53.66
Combined
53.96
(ii) Two features using the same prompt as above and additionally ControlNet and LoRA. The two features use two different LoRA weights, where one is moderately trained and the other slightly overfits.
No attention features are used. We find that very complex scenes attention features can be of low quality and bring slight degradation instead of enhancement.
C.2.5 Semantic Segmentation on CityScapes
Convolutional features include:
(i) One feature using only fine-grained prompt "An urban street scene with multiple lanes, various buildings, traffic lights, cars in the lanes, and pedestrians, highly realistic".
(ii) Two features using the same prompt above and additionally ControlNet and LoRA. The two features use two different LoRA weights, where one is moderately trained and the other slightly overfits.
No attention features are used, based on the same observation as for ADE20K.
C.3
Details for Exploration of Content Shift
For CIFAR1O, high-quality and low-quality prompts follows:
"cinematic shot photo taken by ARRI, photo taken by sony, incredibly detailed, sharpen, masterpiece, best quality, realistic, HD, raytracing. CG unified 8K wallpapers" "low quality, low resolution, blurry, draft, grainy, disfigured, deformed, low contrast, underexposed. overexposed, bad art"
For semantic segmentation, they are:
"a horse, high quality, best quality, highly realistic, masterpiece" "a horse, low quality, worst quality, deformation, broken shape, blurry"
D Additional Experiments
D.1 Additional Qualitative Results
Besides the feature visualization in Figure we provide additional visualization here in Figure 10.
D.2 Demonstration for Generalizability
We would like to further demonstrate the generalizability of GATE by applying it to different diffusion model, SDXL [33]. The architecture of SDXL has been significantly modified to include more complicated attention structures while its overall upsampling blocks are simplified. Due to the architecture change, the previous method for feature extraction cannot fully exploit the potential of SDXL, thus making the performance lower than Stable Diffusion v1.5. Nevertheless, it is enough to
5

---

Prompts
Combined
Convolutional
Attention
Convolutiionll
Attention
Convolutional
Attention
Figure 10: Additional feature visualization for qualitative analysis with different types images, showing the effectiveness of GATE on both convolutional features and attention features.
Table Results Horse-21 study the effect of feature amalgamation, using the same setting as Figure 8. Each column corresponds to group of features. Basic is extracted at different timestep with other settings remaining the same Basic.
Basic
Basic*
Prompts
ControlNet
LoRA
mloU
58.13
59,44
60.12
60.02
60.27
60.74
demonstrate the generalizability of GATE. Furthermore, since LoRA training for SDXL is different and requires more investigation, we have excluded this technique from this small experiment.
Observing the results in Table 4, we can again confirm the benefit of GATE. Both techniques can enhance performance. Moreover, combining two techniques able to bring further improvement. This demonstrates the generalizability of GATE as it can be integrated into different diffusion models.

---

64.5
POKeO, Limg
63.5
0.6
0.15
0.3
Diversity Y:
Sparsity
Figure 11: Sensitivity analysis of the regularization terms, where 71/22 is for sparsity/diversity. These results validate the necessity of these regularization terms.
D.3 Effect of Feature Amalgamation
Now aim examine the effect of regularized weight assignment strategy for feature amalga- mation. We will first demonstrate that amalgamating many features leads to better performance than single feature. Then will conduct an analysis of the hyper-parameter sensitivity regarding the proposed regularization terms.
The first experiment is conducted on the same Horse-21 split as Figure 8. In addition, this experiment only applies techniques individually, not simultancously, for a simpler setting. The following observations can be made from Table 5:
Amalgamating many features generally superior to a single feature. Even if we simply use features obtained from different timesteps, it is beneficial.
(ii) the features are from different technique combinations rather than simply different timesteps, the performance will be higher. This shows that applying various techniques can yield more diverse diffusion features.
(iii) If more features are used, the performance will be better than when only are used.
To analyze the sensitivity property of the two regularization terms, we perform a grid search on the semantic correspondence task. The reported metric is PCK @ img im (f) and the model type is Based on Figure 11, we can conclude that both regularization terms contribute to the performance gain, but they need tuning. We have tuned them for the results in Table 1.
D.4 Ablation Study Number of Weight Assigners
In this part, we provide supplementary ablation study on the number of weight assigners. The results in Table show that when we increase the number of weight assigners, the overall performance will improve gradually. However, we have two following concerns:
(i) More weight assigners will take longer time to converge. The training steps that are enough for fewer assigners to fully converge are insufficient for more assigners, indicated by that the the performance of more assigners still improves with more training steps.
(ii) Increasing the number weight assigners will bring more memory usage and more time consumption, which will affect efficiency.
Consequently, choosing the number weight assigners is mainly tradeoff between performance and efficiency.

---

Table Results (mloUf) to reveal the impact of adjusting the number of weight assigners. The best result is marked as red.
Num
Training Steps
2 x Training Steps
Gain
64.58
63.91
-0.67
2
65.39
65.68
0.29
3
65.44
65.84
0,40
65.19
65.73
0.54
Table 7: Experimental results image classification in comparison to both ResNet and diffusion feature baseline without GATE.
Method Acc(%)
ResNet-50
93.62
Baseline
94.55
GATE
95.21
D.5 Impact Regularization on Weights
We aim to visualize the impact of our proposed regularization terms on weight distribution, show that their effect aligns with the design purpose. The statistics are shown in Figure 12. We select the semantic correspondence task under nn setting with two weight assigners.
The first four figures visualize the impact of sparsity. The first two show the Gaussian probability density fitted the mean and variance top-1 and top-2 weights. The other two show the average value of all weights. Without regularization, top-1 weights will be rather small. With regularization, top-1 weights will become much larger, showing that the prediction of assigners becomes more concentrated.
The last two figures look into diversity. We compare the predictions of two assigners. If overlap is observed in top-1 nor top-2 weights, it counts as no overlap, which is the best case. If overlap occurs in top-I weights, is regarded as overlap @ top-1. If overlap only occurs when we consider both top-1 and top-2 weights, this is overlap @ top-2. It is clear that diversity regularization can effectively force assigners to focus on different feature maps, thus providing information from various perspectives.
D.6 Image Classification
Although image-level tasks are not typically utilized for the evaluation of diffusion features, have used this task for early empirical studies. Hence, in Table 7, we provide some results on CIFARIO, in comparison standard ResNet backbone and baseline where diffusion features used without GATE.
E
Future Direction
Long tail and out-of-diitribbution scenarios are challenging problems for discrimination, where a model that works well under i.i.d. settings degrade significantly [43]. There have been few attempts to apply diffusion models to such challenging scenarios [10, 46, 56], but more efforts are still considered helpful. Recently, there has been study [1]] that might open more opportunities this topic. To be specific, AUC is useful metric and loss function for long tail and out-of-distribution studies [41, 42], but it used to be not applicable to pixel-level tasks such as semantic segmentation. This recent study, however, has successfully introduced new AUC-based tool that can be applied to segmentation as well, enabling more future studies in this direction.

---

top-1 weight top-2 weight
top-1 weight top-2 weight
probability
probability
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
value
value
(a) Probability density of weights w/o regularization.
(b) Probability density of weights w/ regularization.
1.0
1.0
0.8
0.8
percentage
0.6
percentage
0.6
0.4
0.4
0.2
0.2
0.0
0.0
top-1 top-2 top-3 top-4 top-5 tap-6
top-1 top-2 top-3 top-4 top-5 top-6 rank in all weights
rank all weights
(c) Average values of all weights w/o regularization.
(d) Average values of all weights w/ regularization.
600
500
530
number
number
400
300
300
200
100
overiap @ top-1
overtap @ top-2
no overlap
overlap @ top-1
overiap @ top-2
no overlap
case
case
(c) Overlap issues of 2 weights w/ regularization.
(f) Overlap issues of top-2 weights w/ regularization.
Figure 12:
Visualization of weight distribution with and without regularization, The first four focus sparsity and the other visualize the impact of diversity.

---

Computation Resources
We use Nvidia(R) RTX 3090 and Nvidia(R) RTX 4090 GPÜs for the experiments, all with 24GB VRAM.
The experiments semantic correspondence take about hours per and about 50GB of disk storage. The experiments on Horse-21 and Bedroom-28 take about 5 hours per run, where one run consists of repeats different splits. Furthermore, each uses about 45GB of disk storage. The experiments on ADE20K and CityScapes take about 2 days per run with no additional disk usage besides what is needed to store model checkpoints and datasets. Our early exploration of content shift utilizes relatively lightweight experiments for efficiency.
G Asset License
SPair-71k: Available https://cvlab www.b..pssech...o.om ppsteebh.a...a/reeeerr////aaa
Horse-21 and Bedroom-28 (LSUNN): Available httpss///gtthubb.cm//////aaa
ADE20K: Custom (research-only, vision/ vision/datasets, (latasets/ADDe0K/eeettttt
CitySeapes: Custom, at non-commercial), at https://groups.ccalll mit, edu/
10

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We make clear claims of of contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
We make no theoretical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details in Appendix A and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
12

---

Answer: [No]
Justification: We will release the codes after acceptance. Besides, we have provided the necessary information for reproducing the experiments in this manuscript.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: /CodeSubmis /nipp..cccc. httpsscccppbccc//cceeecccc / ssion ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are provied in Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars for Horse-21 and Bedroom-22 datasets. For other datasets, the same practice would be too costly.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
13

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Related information is provided Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : //te:://neuu...eeeeeee cc/ htte//ublcc/ttticeeeeeeecee csGuidel ines?
Answer: [Yes]
Justification:
We conform with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
14

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers where related assets are first mentioned and list their license in Appendix G.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
15

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16


Reviews:
[{"review_id": "eJriETniCg", "reviewer": "Reviewer_z9si", "summary": "This paper addresses the issue of content shift in diffusion features, which are inner activations extracted from pre-trained diffusion models used for discriminative tasks. The content shift refers to the phenomenon where there are content differences between the features and the input images. In the manuscript, the authors showed the cause of content shift as an inherent characteristic of diffusion models, related to information drift during the image recovery process from noisy inputs. The proposed method utilizes off-the-shelf generation techniques to suppress content shift and introduces GATE for evaluating and implementing the techniques. The proposed method is simple yet novel.", "questions": "1. Please provide a more detailed explanation and additional diagrams to clarify how attention features are extracted from the diffusion model in Figure 1 and 2.\n2. Please provide details of the overall process of the proposed method.\n3. How does the proposed method perform on tasks involving different types of images or datasets with varying levels of noise and complexity?", "limitations": "1. The paper should improve the clarity of the proposed method, especially in explaining the extraction of features from the diffusion model. Providing a more detailed and step-by-step explanation would help readers understand the method better.\n2. The authors should consider demonstrating the experimental results in a better way with additional evaluation metrics and extensive qualitative evaluation.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The paper identifies and systematically analyzes the content shift in diffusion features, which has not been studied before. it is a simple but novel method.\n+ The related work and the explanation of the phenomenon are clear. They supported the explanation with visual examples and empirical results.\n+ They compared the proposed method with different state-of-the-art methods on different benchmark datasets to show the effectiveness of the proposed method.", "weaknesses": "- The paper lacks clarity in explaining the exact process of how attention features are extracted from the diffusion model. Figures 1 and 2, which are supposed to illustrate this process, do not provide enough detail.\n- The details of the overall process of the proposed method are missing. \n- The effectiveness of the GATE guideline is demonstrated using a limited set of generation techniques. A broader range of techniques should be evaluated to strengthen the claim of its general applicability.\n- Even though the authors provided comparison experiments in the manuscript and ablation study in the supplementary materials, the results do not clearly demonstrate the superiority of the proposed method. More detailed and extensive comparative analyses with additional evaluation metrics and qualitative evaluation are needed to show the advantages of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.  \n\n> **Q1:**\nPlease provide a more detailed explanation and additional diagrams to clarify how attention features are extracted from the diffusion model in Fig.1 and Fig.2.  \n\n**A1:**\nModern diffusion models use cross-attention between the latent image and prompts to enable user interaction. We extract the similarity scores between query (latent image) and key (prompt) and average them over all layers as attention features.\nFor more details, please refer to the **global response GQ1** and the attached PDF.  \n\n> **Q2:**\nPlease provide details of the overall process of the proposed method.  \n\n**A2:**\nPerhaps due to the omitted details of **Extract Features with Techniques** in Fig.7, the overall pipeline is a bit confusing. In fact, this step follows the practice in the prior arts [2]. We first encode an input image with VAE, then add noises to it, and finally send the image into diffusion UNet to extract features.\n\nFor more details, please refer to the **global response GQ1** and the attached PDF.  \n\n> **Q3:**\nA broader range of techniques should be evaluated to strengthen the claim of its general applicability.  \n\n**A3:**\nWe have also evaluated two techniques, classifier-free guidance and IP-Adapter. As the results presented in Appendix and **global response GQ2** show, the two techniques fail to suppress content shift and do not achieve significant performance gain. These results again strengthen the general applicability of our GATE guideline.\n\n> **Q4:**\nMore detailed and extensive comparative analyses with additional evaluation metrics and qualitative evaluation are needed to show the advantages of the proposed method.  \n\n**A4:**\nAccording to your suggestion, we have provided more comparative analyses with additional metrics, including mIoU, aAcc, and mAcc. Please refer to the **global response GQ4** for more details.  \n\nFor more qualitative results, besides the visualization in Fig.10, we also include additional feature visualization in the PDF attached to the global response. From the visualization, we can observe that:\n\ni) The attention features become clearer and closer to the input image when more generation techniques are applied according to GATE, showing the suppression effect on content shift.\nii) Notably, for the second image where a person is riding a horse, the baseline attention feature fails to follow the instruction, i.e., attending only to the horse and ignoring the person. In contrast, generation techniques applied according to GATE help attention features attend to the correct object.\niii) From convolutional features, we can see the application of generation techniques brings stronger diversity. \n\n> **Q5:**\nHow does the proposed method perform on tasks involving different types of images or datasets with varying levels of noise and complexity?  \n\n**A5:**\nAmong the three generation techniques, only fine-grained prompts can be sensitive to image types and noise levels. For simplicity, we tackle this challenge via general prompts, whilst a pre-trained image captioner is also an option. \n\nFor more details, please refer to the **global response GQ3**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1vT6HaZWoP", "reviewer": "Reviewer_cofK", "summary": "The authors focus on the task of suppressing content shift in diffusion features using off-the-shelf generation techniques. The authors observe that diffusion features, which are extracted from pre-trained diffusion models, suffer from a phenomenon called content shift, where there are content differences between the input image and the features extracted. The strength of the proposed method lies in utilizing off-the-shelf generation techniques to suppress this content shift, leveraging their ability to control the recovering process in diffusion models. Experimental validation shows that the proposed GATE framework effectively enhances diffusion features across various tasks and datasets, outperforming state-of-the-art methods.", "questions": "See the weakness.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(1) The authors identify that even minor, visually imperceptible content shifts can negatively impact the discriminative performance of diffusion features, highlighting the necessity to suppress this phenomenon to enhance feature quality.\n(2) The method leverages off-the-shelf generation techniques to suppress content shift, which maintains computational efficiency and achieves significant improvements.\n(3) Extensive experiments confirm that the method outperforms state-of-the-art methods in multiple benchmarks, validating its potential for widespread application.\n(4) The paper is well organized, making it easy to understand.", "weaknesses": "(1) Some details can be more clarified. For example, the authors propose to use  fine-grained prompts to surpass content shift. I wonder how to design these prompts. Does it require us to design prompts for each input? \n(2) The paper focuses on empirical validation and lacks a thorough theoretical analysis of why and how the proposed techniques effectively suppress content shift. It is recommended to provide more insights or explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.  \n\n> **Q1:**\nI wonder how to design these prompts. Does it require us to design prompts for each input?  \n\n**A1:**\nWe observe the common characteristics of the training images and design a general prompt for all images.\nIn this way, we do not need to design prompts for each input nor see the images of the testing set.\nMore advanced methods can also be utilized, such as a pre-trained image captioner. We do not use image captioners in the manuscript, but some additional results taking this approach are listed in the **global response GQ3**.  \n\n> **Q2:**\nThe paper focuses on empirical validation and lacks a thorough theoretical analysis of why and how the proposed techniques effectively suppress content shift. It is recommended to provide more insights or explanations.  \n\n**A2:**\nThanks for this constructive suggestion! We focus this paper on enhancing the actual performance and thus leave thorough theoretical analysis to future work. Nevertheless, we have made some preliminary discoveries.\nAs shown in Fig.4, from pre-training, diffusion models obtain the ability to reconstruct clean inner latent representations from noisy inputs. However, noises may make some details blurry, thus making the reconstruction drift from the input image, which leads to content shift in diffusion features.\nThe three selected techniques can suppress content shift due to two factors.\nOn one hand, they introduce additional information into this reconstruction process.\nOn the other hand, the techniques are designed to steer the reconstruction towards the introduced information.\nThe two factors combined together can guide the reconstructed latent representation towards the input image, thus suppressing content shift.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bN6cPGeCj6", "reviewer": "Reviewer_ECpW", "summary": "This paper explores how diffusion models, typically used for generative tasks, can also serve discriminative purposes by using inner activations as features. The authors point out that these features often suffer from a phenomenon named content shift, i.e., the features are semantically different from the input image. This content shift negatively impacts the performance of diffusion features. The authors propose to use off-the-shelf generation techniques to suppress content shift. To this end, they introduce a practical guideline called GATE. Their approach, which is validated through extensive experiments, significantly improves performance on various tasks and datasets.", "questions": "Please see the weakness.", "limitations": "Yes, the authors discuss their limitations in Section 7.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) As far as I know, this paper is the first to reveal and systematically analyze the phenomenon of content shift in diffusion features, highlighting its universal and harmful effects on discriminative tasks.\n2) The authors propose a practical guideline (GATE) that utilizes off-the-shelf generation techniques to suppress content shift. This method leverages existing tools and advancements in the field, providing a convenient approach to enhancing diffusion feature quality.\n3) The effectiveness of the proposed approach is demonstrated through extensive experiments on various tasks and datasets, showing significant improvements over state-of-the-art methods.\n4) The proposed solution is not limited to a specific task or model but is shown to be a generic booster for diffusion features. This broad applicability enhances the potential impact and utility of the approach in different scenarios and for various applications.", "weaknesses": "1) The authors propose the GATE guideline for evaluating generation techniques. Then, the actual implementation includes three successful techniques and a failed one. However, as pointed out by the authors, there are abundant techniques for generation. Hence, it is recommended to provide more failure cases to validate the negative impact of the content shift.\n2) Some details can be further clarified. To be specific, which results are implemented by the authors or borrowed from previous works could be more explicitly stated. Besides, some results in Table 1 and Table 2 are missing. The authors should make a further explanation.\n3) Besides, some typos and minor errors found in the paper:\n- Section 2, Line 66: \"diffusion feature\" should be \"diffusion features\".\n- Section 4.2, Line 136: \"consist the diffusion features we extract\" should be \"constitute the diffusion features we extract\".\n- In reference [1], “ECCV Workshopt” should be “ECCV Workshop”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.  \n\n> **Q1:**\nHence, it is recommended to provide more failure cases to validate the negative impact of the content shift.  \n\n**A1:**\nWe have included another failure case, IP-Adapter, in the rebuttal.\nFor details, please refer to the **global response GQ2**.  \n\n> **Q2:**\nTo be specific, which results are implemented by the authors or borrowed from previous works could be more explicitly stated. Besides, some results in Table 1 and Table 2 are missing. The authors should make a further explanation.  \n\n**A2:**\nThanks for your patience and advice. We will refine these details further and make some explanation here.\nIn Table 1, Baseline for semantic correspondence and GATE for both tasks are implemented by us, and all the other results are borrowed from previous studies [1-4, 12, 23, 26, 30, 36, 39, 53].\nIn Table 2, MaskCLIP and ODISE results are borrowed from previous studies [8, 45]. As they do not provide results on CityScapes and it is hard to extend the codes on this dataset, we do not have CityScapes results for the two methods. Both our GATE and VPD are implemented by us. Specifically, the original results reported by VPD are based on full-scale fine-tuning of diffusion UNet, which is not fair as we do not train the diffusion model in our setting. Therefore, we re-evaluated VPD with the diffusion UNet frozen and reported our results.\n\n> **Q3:**\nBesides, some typos and minor errors found in the paper.  \n\n**A3:**\nThanks for your patience. We will pay attention to these typos and errors during refinement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "54milp0IIx", "reviewer": "Reviewer_RvKH", "summary": "The paper describes a pipeline for improving feature extraction from pre-trained diffusion models for general-purpose task such as semantic segmentation. The focus is on \"content shift\", where extracted feature maps at a certain time step in the diffusion process show differences in the content composition compared to the input image. Three previous methods for controlling image to image transformation with diffusion models are used for extracting features and attention maps. Subsequently, these are fused by a learned weighted linear combination, or amalgamation, providing updated image features for use in the downstream application. The pipeline is tested for semantic correspondence on the SPair-71k dataset, and for semantic segmentation on the Bedroom-28, Horse-21, ADE20K, and CityScapes datasets. The results show improvement over the compared baselines (vanilla feature extraction, and some other previous feature extraction techniques).", "questions": "* In my view, there are too many details left out in the main paper, related to, e.g., method, datasets, and experiments. In its current state, it relies too much on details in the appendix. For example, the dataset used for semantic correspondence in Tab. 1 is not even specified, metrics are not explained, implementation details are missing, etc. To me, this is more important compared to the lengthy and speculative motivation of content shift in Section 4. It is rather obvious that features can shift away from the input image with excessive noise added. I would suggest moving most of these discussions and figures to the appendix, and instead provide the necessary details for understanding the pipeline and experiments.\n\n* Although the appendix provides further information, there are still missing details and experiments. For example, one central hyper-parameter is the noise level used for feature extraction. This is neither specified nor tested in experiments. What noise levels were used? How were these selected? How is the improved performance of the suggested pipeline reflected over different selections of noise levels? Is it possible to combine a larger number of different noise levels, e.g., in the learned combination?\n\n* Rather than the speculative motivation in Section 4, is there a more systematic way of quantifying the content shift? As it looks now, this is rather speculative and only verified through a few example images. Since this is the main focus, it would be important to measure and compare, e.g., how it behaves over different noise levels and for different feature extraction techniques.\n\n* The experimental setup now relies on having rather specific types of images (horses, bedrooms, street view, etc.), so that a general text prompt can be provided for the fine-grained prompt image generation. How can this generalize to more diverse types of images?\n\n* In Fig. 5, there are some examples for classification on CIFAR-10, but there are no experimental results on classification. How does the proposed method perform in classification tasks? Isn't this the most straightforward strategy for evaluation of feature quality?\n\n* How are clean features obtained from the noisy features, in Fig. 4?\n\n* In summary, although there are interesting observations, the manuscript is to me not yet mature enough, and lacking both in details and experiments.", "limitations": "* Limitations are only very briefly mentioned in the conclusions. These could be elaborated on, in connection to the points discussed above.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ The focus on improving features from diffusion models for downstream applications is interesting, and to my knowledge a direction with relatively little previous work\n+ The evaluation shows favorable results, where the proposed pipeline can improve on vanilla features and some other feature extraction methods", "weaknesses": "- The paper is written on a quite high level, with many of the relevant details left out or put in the appendix\n- The technical contribution is rather limited, combining previous techniques for the purpose of improving diffusion features\n- The evaluation is lacking, with many details missing and without proper ablation study", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for constructive comments. We would like to response as follows.\n\n>**Q1:**\nThe paper is written on a quite high level with many details left out or in appendix.\n\n**A1:**\nWe focused on content shift and how generation techniques affect it. Some details were thus omitted in the main body due to the space limitation, which might be confusing. We will revise according to your advice.\n\nFor pipeline, please refer to **global question GQ1**.  \nFor experimental details, we show **part of** refinement below due to space limit.  \n\nWe extract features at $t=50$.\n- Semantic Segmentation\n\t- Convolutional features:\n\t\t- One with fine-grained prompt \"a highly realistic photo of the real world. It can be an indoor scene, or an outdoor scene, or a photo of nature. high quality\" plus all category labels (ADE20K).\n\t\t- Two features: same prompt as above, ControlNet and two different LoRA weights.\n\t- No attention features are used, as they do not bring benefit in complex scenes.\n\n>**Q2:**\nThe technical contribution is limited, combining previous techniques.\n\n**A2:**\nPerhaps due to presentation, it is a pity to leave you this impression.\nIn fact, our main contribution is the pioneering discovery of content shift, a widespread, harmful, yet hidden behavior of diffusion features. \nWe systematically investigated its cause and thus found that existing generation techniques are enough to mitigate such shift effectively.\nHence, we choose these techniques for easier implementation and extension.\nThe key point is: among numerous techniques, which can serve for discrimination tasks?\nOur GATE guideline bridges this gap and inspires the necessary adaptation to these techniques.\nWe hope all these observation, analysis, guidance, and adaptation can benefit future work.\n\n\n>**Q3:**\nMore experimental details and ablation study.\n\n**A3:**\nFor more evaluation and details, please refer to **GQ4** and your **Q1**.\nFig.10 serves as ablation. For your convenience, we list results below, which validate the effectiveness of each technique and their combination.\n\n|Method|mIoU (on a single split)|\n|:-|:-:|\n|Baseline|58.13|\n|GATE (Fine-grained prompts)|59.47|\n|GATE (ControlNet)|59.41|\n|GATE (LoRA)|59.43|\n|GATE (Full)|**59.73**|\n\n>**Q4:**\nProvide necessary details for pipeline and experiments in the main content.\n\n**A4:**\nWe will refine manuscript according to your suggestions.\nThe pipeline in **GQ1** will replace Fig.2 in Preliminary.\nThe experimental details in **Q1** will be added.\n\n>**Q5:**\nWhat noise levels were used? Is it possible to combine different noise levels?\n\n**A5:**\nIn fact, the choice of noise level is orthogonal to our work.\nEarly diffusion feature work [2] systematically analyzed this issue and suggested $t=50$ as a general choice. We observe similar results and thus follow this practice.  As shown below, we test ControlNet with other noise levels on a single Horse-21 split, where GATE outperforms baseline across noise levels, especially when the noise level is stronger.\n\n|Method|t=400|t=700|\n|:-|-|-|\n|Baseline|47.09|24.39|\n|GATE|**56.35**|**52.65**|\n\nIt is reasonable to combine different noise levels, which is actually the focus of some prior arts [23, 56], including a semantic correspondence SOTA.\nNevertheless, The results in Tab.1 show that amalgamating different generation techniques brings stronger enhancement.\n\n>**Q6:**\nA more systematic way of quantifying content shift.\n\n**A6:**  \nWe also developed a quantitative metric.\nWe set feature extracted at $t=0$ as reference $FEAT_{ref} \\in \\mathbb{R}^{c\\times h\\times w}$ as it is less affected by noises. We use Laplacian operator to evaluate the contour difference between feature $FEAT$ and the reference:\n\n$$\ndiff=\\sum_{i,j}^{h\\times w}|\\left\\|Laplacian(FEAT_{ref},i,j)\\right\\|_2-\\left\\|Laplacian(FEAT,i,j)\\right\\|_2|\n$$\n\nWe then set a feature with stronger content shift as anchor and compare $diff_{anchor}$ with other features.\n\n$$\nScore=\\frac{(diff_{anchor}-diff)}{diff_{anchor}}\\in(-\\infty,1]\n$$\n\n$Score=1$ means exact match, and a smaller value indicates more shift. Below are quantitative results obtained on a single Horse-21 split, showing $Score$ is consistent with mIoU performance.\n\n|mIoU|$Score$|\n|-|-|\n|59.3|0.73|\n|58.7|0.73|\n|54.2|0.20|\n|50.1|-0.17|\n\nIn our manuscript, we use qualitative GATE guideline because it can well approximate $Score$ and is much simpler. According to your advice, we will add this metric in refinement.\n\n\n>**Q7:**\nHow can the method generalize to more diverse types of images?\n\n**A7:**\nAmong the three techniques, ControlNet and LoRA are insensitive. For fine-grained prompt, generalization is clarified in **GQ3**.\n\n>**Q8:**\nHow does the method perform in classification tasks?\n\n**A8:**\nDiffusion features have much more channels (3520) than input images (RGB). Hence, prior arts prefer pixel-level task to image-level task like image classification.\nIn Sec.4, we aim to demonstrate quality described by prompts should resemble image quality, so the classification dataset CIFAR10 is introduced for low-quality images.\n\nAccording to your advice, we provide additional results on classification in **GQ4**.\n\n>**Q9:**\nHow are clean features obtained from noisy features?\n\n**A9:**\n- Diffusion models learn from pre-training to reconstruct clean representations from noisy inputs. Notably, this is also related to content shift as noisy inputs cause information loss.\n- Shortcut structures in UNet pass high-frequency details to upsampling stage, helping reconstruction.\n\n> **Q10:**\nLimitations are too brief.\n\n**A10:**\nWe focus on content shift and its solution. Due to space limit, we omit many implementation details, which will be revised.\nBesides, effectiveness of GATE relies on selected techniques, for which we propose qualitative evaluation guideline. Though we select three effective techniques and report fail cases, there still is more to explore.\nFurthermore, we only experiment on three tasks, so the full potential of GATE might remain under-explored.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear SAC, AC, and reviewers,  

Thanks for your valuable feedback. Based on your comments, we first offer a global response to some common questions.  

> **GQ1:**
More details about the overall pipeline and how attention features are extracted.
**(Q1, Q2 of z9si, Q1 of RvKH)**

**GA1:**
Thanks for this constructive suggestion! In the original manuscript, we provided an overview of the proposed GATE framework in Fig.7 but did not detail the process of extracting attention features. According to this advice, the PDF attached to the global rebuttal details the extraction process, i.e., the top-right module **Extract Features with Techniques** in Fig.7. In the revised manuscript, we will replace Fig.2 in Preliminary section with the new figure. 

Specifically, given an input image $x\in\mathbb{R}^{3\times h \times w}$, where $h, w$ are height and width, the extraction process has the following steps:

i) A pre-trained VAE encodes the input image into the latent space, inducing $x_0\in\mathbb{R}^{4\times h' \times w'}$, as a common practice presented in [31, 32].

ii) $x_0$ acts as the input of the forward diffusion process, i.e., timestep $t=0$. As suggested in [2], it is beneficial to extract diffusion features at non-zero timestep. Following this practice, we set $t=50$ and get $x_t$.  

iii) $x_t$, along with the timestep $t$, a textual prompt $c$, is sent into the pre-trained diffusion UNet $\epsilon_\theta$, where $\theta$ denotes the UNet backbone, i.e., $\epsilon_\theta(x_t,t,c)$. The generation techniques are used here to modify $\theta, c$.   

iv) Finally, diffusion features can be obtained by:
 - **Convolutional Feature**: The pre-trained diffusion UNet is comprised of multiple resolutions, which can be divided into downsampling stage, middle stage, and upsampling stage. We extract the output activations of each resolution in the upsampling stage as convolutional features. 
 - **Attention Feature**: In modern diffusion models, there are many cross-attention layers, which calculate the attention between the latent image and the textual prompt. These cross-attention layers enable users to control the generated image content using natural language prompts. We extract the similarity maps between query (image) and key (prompt). Afterward, we average the maps over all layers and obtain an attention map for each token in the prompt. All these maps consist attention features.  

> **GQ2:**
Evaluate more generation techniques.
**(Q3 of z9si, Q1 of ECpW, Q2 of RvKH)**

**GA2:**
Besides fine-grained prompt, ControlNet, and LoRA, we have also evaluated classifier-free guidance [15] and IP-Adapter [48]. According to our qualitative evaluation guideline, these two techniques fail to suppress content shift effectively. Furthermore, our quantitive results also show that they do not bring significant benefits, which again validates the effectiveness of our qualitative guideline.

To be specific, classifier-free guidance aims to enhance generation quality. However, better image quality does not necessarily suppress content shift. We have included this technique in the Appendix as a failure case study.  

IP-Adapter shares similar architecture with  ControlNet. However, its goal is image variation, instead of strict control. Hence, it is less effective on mitigating content shift than ControlNet. To support this claim, we include some ablation experimental data below, which is conducted on the Horse-21 dataset.

|Method|mIoU|
|:--|:-:|
|DDPM|65.0 ± 0.8|
|GATE (Only IP-Adapter)|65.8 ± 1.3|
|GATE (Only ControlNet)|66.2 ± 1.2|
|GATE (Full)|**67.2 ± 1.1**|

In conclusion, if a generation technique can make the generation result closer to the additional information introduced to the generation process, the technique tends to be helpful. This property can be efficiently evaluated using our GATE guideline for more techniques.  

> **GQ3:**
How to apply the selected techniques to datasets with various types of images and noise levels?
**(Q5 of z9si, Q1 of cofK, Q7 of RvKH)**

**GA3:**  
Among the three selected techniques, ControlNet and LoRA are essentially insensitive to image types, noise levels, and complexity. Hence, they can be applied to various types of images and datasets.

Fine-grained prompts can be sensitive to these factors if we use image-specific prompts. To tackle this challenge, we have two solutions: i) a relatively general prompt or ii) a pre-trained image captioner to generate specific prompts for each image. For simplicity, the manuscript chooses the first approach for the ADE20K dataset, which contains images of varying types. The corresponding performance gain shows that GATE with general prompts can apply to this scenario. Additionally, according to this suggestion, we provide experimental results of the second approach below, using captioner Kosmos-2 [29]. We can find that GATE (Captioner) achieves comparable performance to GATE (Standard). Hence, we choose general prompts in practice for simplicity and efficiency.

|Method|mIoU|
|:--|:-:|
|DDPM|65.0 ± 0.8|
|GATE (Captioner)|**67.3 ± 1.4**|
|GATE (Standard)|67.2 ± 1.1|

> **GQ4:**
More comparative experiments are needed. **(Q4 of z9si, Q3, Q8 of RvKH)**

**GA4:**
For additional evaluation metrics, we report the mIoU, aAcc and mAcc results on ADE20K and CityScapes.
- mIoU is as defined in the manuscript;
- aAcc is the classification accuracy of all pixels;
- mAcc averages the accuracy over categories.

We also report image classification results, where Baseline is diffusion feature without GATE.  

These results again validate the effectiveness of GATE.  

- ADE20K
|Method|mIoU|aAcc|mAcc|
|:-|:-:|:-:|:-:|
|VPD|37.63|79.16|50.08|
|GATE|40.51|79.68|54.90|

- CityScapes
|Method|mIoU|aAcc|mAcc|
|:-|:-:|:-:|:-:|
|VPD|55.06|90.14|68.96|
|GATE|64.20|92.83|76.98|

- CIFAR10
|Method|Acc(\%)|
|:-|:-:|
|ResNet-50 [a]|93.62|
|Baseline|94.55|
|GATE|95.21|

[a] https://github.com/kuangliu/pytorch-cifar (5.9k star)

Meta-review: This paper is about suppressing the content shift in recently popular "diffusion features" (using pre-trained diffusion models to retrieve feature representations). The phenomenon of content shift is newly discovered by the authors, i.e., the content of the feature representation shifts from the original input due to the "noisy" nature of diffusion models. To resolve this problem, the authors propose to utilize popular conditioning techniques (e.g., prompt-based, ControlNet, and LoRA) in a very unusual but clever way: conditioning the features based on the very original image. This is very simple but quite effective, achieving state-of-the-art performance with substantial improvements. The proposed method is also quite efficient in that off-the-shelf techniques can be directly used, which also philosophically conforms with the recent trends of AI. All reviewers have agreed on the potential impact of the topic itself and the proposed method.

However, there was a not-so-small issue during the author-reviewer discussion: The original submission lacked details and was written at a higher level, so a large number of additional details were submitted as a rebuttal. This amount concerned some of the reviewers, and in response, the authors have posted a partial revision of the paper as another rebuttal. This is not allowed in the NeurIPS policy, and there was a heated discussion between the authors and the reviewers regarding its appropriacy. After a discussion between AC and SAC, it was decided that the reviewers should ignore the revision and continue the discussion with AC. The issue still resided in that the amount of revision needed is quite substantial.

After the discussion, the AC personally reviewed the paper in detail to yield the following conclusion: The required revision is mostly about details and not the core contribution, as the authors argued. The method is largely based on very well-known off-the-shelf techniques (diffusion features, conditioning techniques like prompt-based, ControlNet, and LoRA), and the only novel twist introduced in the paper is simply changing the conditioning input to an unusual one (the original input itself), which is in fact not quite different from the usual ones (some other input to steer the generated result) in characteristics. The details needed here are more about tuning issues rather than deal-breaking design issues. Accordingly, the required revision is not too substantial (in a qualitative sense), considering the merit of the topic/message and the effectiveness/efficiency of the proposed method.

That being said, the additional details discussed in the rebuttal are indeed required and must definitely be added to the final camera-ready version. Under this strict condition, an accept decision is recommended.
Decision: Accept (poster)