Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniques

Authors: benyuan  meng
qianqian  xu
zitai  wang
zhiyong  yang
xiaochun  cao
qingming  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 08:45:16

Abstract: Diffusion models are powerful generative models, and this capability can also be applied to discrimination. The inner activations of a pre-trained diffusion model can serve as features for discriminative tasks, namely, diffusion feature. We discover that diffusion feature has been hindered by a hidden yet universal phenomenon that we call content shift. To be specific, there are content differences between features and the input image, such as the exact shape of a certain object. We locate the cause of content shift as one inherent characteristic of diffusion models, which suggests the broad existence of this phenomenon in diffusion feature. Further empirical study also indicates that its negative impact is not negligible even when content shift is not visually perceivable. Hence, we propose to suppress content shift to enhance the overall quality of diffusion features. Specifically, content shift is related to the information drift during the process of recovering an image from the noisy input, pointing out the possibility of turning off-the-shelf generation techniques into tools for content shift suppression. We further propose a practical guideline named GATE to efficiently evaluate the potential benefit of a technique and provide an implementation of our methodology. Despite the simplicity, the proposed approach has achieved superior results on various tasks and datasets, validating its potential as a generic booster for diffusion features. Our code is available at https://github.com/Darkbblue/diffusion-content-shift.

Main paper: 
 introduction: 
 Diffusion models (DMs) [15, 34] are a prevalent family of generative models for various tasks [36, 35, 25] . This strong generative capability can be applied to discrimination [21] . Diffusion Feature (DF), a popular approach, extracts inner activations from a pre-trained diffusion model as vision features [1, 50, 60, 57, 37, 62] , similarly to how ResNet [14] serves as a feature extractor. Extracting features with a vastly pre-trained diffusion model grants this approach strong robustness and generalizability. Furthermore, it enjoys the philosophy of hitchhiking, a solid research paradigm in the age of large base models [44] : advancements in diffusion models can all be transformed into better feature quality. It promises this research direction a bright future.Figure 1 : Current diffusion features widely suffer from content shift, i.e., content differences between inputs and features. Due to the inherent connection, content shift can be suppressed with off-the-shelf generation techniques.Have we already obtained satisfying diffusion features? The original role of diffusion models, generation, has provided an inspiring perspective. There is an endless pursuit among AIGC players 2for better control over generation [17, 58, 53, 28] . This implies the inherent difficulty in controlling diffusion models: their generation results may not be exactly the same as intended. Will this property of diffusion models also affect the quality of diffusion features? We visualize some diffusion features in Figure 1 and find that they do contain detail differences from inputs, which might hinder the performance. We name this phenomenon content shift, i.e., content differences between diffusion features and the input image. Although we have just shown an example with very obvious content shift, it is in fact intentionally amplified for observation. Under more practical feature extraction settings, the magnitude of content shift would be significantly weaker for visual perception. In Section 4, however, empirical results show that even in such cases, the negative impact of content shift on discrimination is not negligible, indicating the necessity to suppress it for diffusion features of better quality.In pursuit of a method to suppress content shift, we need to further investigate why this phenomenon exists. We notice in Section 4 that the diffusion backbone reconstructs clean inner representations from noisy inputs in the middle of UNet before predicting noises based on the reconstructed content. The diffusion features we are using are in fact the reconstructed representations, which answers why we can obtain clean features from noisy images. However, since high-frequency details are potentially blurred out by noises and then recovered by "imagination", this reconstruction process inherently suffers from the risk of drifting from the original image. Content shift in diffusion features, naturally, is the reflection of such drift during reconstruction. Consequently, to suppress content shift, we need an additional way to directly introduce the original clean image into the reconstruction process and steer it towards the original image. To our delight, we notice that many off-the-shelf image generation techniques for diffusion models [58, 28, 53] also work by injecting additional information into UNet and thus steering the reconstruction. Through careful evaluation of the effect of generation techniques, we are able to select some techniques that can directly satisfy the goal of suppressing content shift, which eases the implementation and extension of our method. To guide the efficient evaluation of generation techniques, we also propose a guideline in Section 5. This method is denoted as GenerAtion Techniques Enhanced (GATE) diffusion feature and its effect is also shown in Figure 1 .To validate GATE, we implement it (Section 5) by choosing three techniques from a few highly popular generation techniques. Since the integrated techniques can benefit from feature amalgamation [24, 62] more than previous approaches, this method is also adopted in our implementation. Although this is a very simple implementation, it achieves impressive performance on various tasks and datasets (Section 6). It demonstrates the effectiveness and potential of GATE, as more techniques can still be evaluated, and there will be even more as diffusion models develop.In summary, the contribution of this work is as follows:• To the best of our knowledge, we are the first to reveal and systematically analyze the universal, harmful, yet hidden phenomenon, content shift, in diffusion features.• We point out the possibility of utilizing off-the-shelf generation techniques for content shift suppression and propose the GATE guideline to facilitate technique integration.• Comprehensive experiments on two discriminative tasks validate the effectiveness of our method. 
related work: 
 The introduction to diffusion models can be found in [6] . In this section, we solely focus on diffusion features. So far, the research topics in this direction fall into two categories: task exploration and method improvement, i.e., applying the paradigm to different tasks and studying the approach itself for better feature quality, respectively.Task exploration. There have been attempts on semantic segmentation [1, 50] , semantic correspondence [57, 37, 22] , hyperspectral image classification [62] , domain generalization [9] , training data synthesizing [45, 55] , zero-shot referring image segmentation [30] , visual grounding [23] , a novel task involving personalization [47] , co-salient object detection [48] , and open-world segmentation [38] . This extensive and extending list shows the discriminative capability of DMs.Method improvement. The way to enhance diffusion features can be divided according to the three important factors of a diffusion model: prompt, layer, and timestep. In a basic pipeline, the prompt is manually designed and simple, the layer only refers to the activations between convolutional blocks, and the timesteps are manually set. (i) To enhance the use of prompt, a typical improvement is prompt tuning [50, 60, 57, 22] , which equivalently fine-tunes the text encoder along with the downstream discriminative task. Another novel method is auto-captioning [19] , replacing manual prompt design with an automatic captioner. (ii) To dig more information out of UNet layers, researchers choose to additionally take attention layers into consideration. However, we find that cross-attention is more frequently used [50, 60, 57, 30] while self-attention is less popular [47] . (iii) As for better usage of timesteps, [62] proposes to extract features from many timesteps and dynamically assign weights to them, while [52] employs reinforcement learning for better timestep selection.Of the two directions, our work aims for better methods instead of new tasks. Furthermore, most existing diffusion feature approaches suffer from content shift, an example of which is the visualization of attention features in Figure 1 . Therefore, our GATE can serve as a generic performance booster to other diffusion feature approaches, showing its potential for broad application. 
preliminaries: diffusion feature: 
 Diffusion models consist of a neural network module and a diffusion scheduler. The network is an end-to-end network, which can be formally denoted as ϵ θ , where θ is the parameters. The diffusion scheduler is the core of diffusion models. With the scheduler, diffusion models generate images progressively, during which the network module is re-used on each timestep, each time only predicting an incremental noise. Generally, we use a smaller / larger timestep to indicate less / more noises. A typical generation process starts from t = T (total noises) and ends at t = 0 (clean images).Next, we will explain how features are extracted using a common pipeline for diffusion features, with the visual illustration in Figure 2 . Given an input image x ∈ R 3×h×w , where h, w are height and width, the extraction process includes: (i) A pre-trained VAE encodes the input image into the latent space, inducing x 0 ∈ R 4×h ′ ×w ′ , as a common practice presented in [34, 33] . (ii) x 0 acts as the input of the forward diffusion process, i.e., timestep t = 0. As suggested in [1] , it is beneficial to extract diffusion features at non-zero timestep. Following this practice, we set t = 50 and get x t . (iii) x t , along with the timestep t and a textual prompt c, is sent into the pre-trained diffusion UNet ϵ θ , i.e., ϵ θ (x t , t, c). The generation techniques selected by our method are also applied here to modify θ, c. (iv) Convolutional and attention features are gathered during the computation of the backbone. For convolutional features, we gather the output activations of each resolution in the upsampling stage of the UNet. For attention features, we obtain the mean value of the similarity maps between query and key in all cross-attention layers.  
exploration of content shift 4.1 impact of content shift: 
 In Figure 1 , we can qualitatively observe content shift from feature visualization. However, this visualization is intentionally amplified for better observation by extracting features at large timesteps. Now we aim to examine the impact of content shift on quantitative performance under more practical scenarios, i.e., when timesteps are small. To this end, we need to toggle the magnitude of content shift in features. Describing the quality of the image is a widely adopted way to control generation. A prompt accurately describing the quality of input images is considered to suppress content shift, while intentionally describing something different from the image will cause more severe content shift. We select the semantic segmentation task [1] as an example of high-quality images and image classification on CIFAR10 [20] for low-quality images. The results in Figure 3 seem counter-intuitive at first glance, as low-quality prompts surprisingly can improve the performance on CIFAR10, but this in fact complies with the quality of input images. Therefore, these results can demonstrate the negative impact of content shift on feature quality at small timesteps. After the negative impact of content shift is confirmed, we next aim to find its cause. Unlike more conventional feature extractors such as ResNet [14] , the inputs to diffusion models are not the original image (Figure 4(a) ), but its noisy version (Figure 4 (b)), as enforced by the diffusion process [15] . The early layers of diffusion models even further add more noise to the feature maps (Figure 4(c) ). However, the diffusion UNet gains the ability from vast pre-training to reconstruct clean inner representations from noisy inputs (Figure 4(d) ), roughly at the middle of the UNet structure. Additionally, the shortcut structures in UNet also help the reconstruction by passing some high-frequency details. Afterward, the diffusion UNet will further predict noises based on the reconstructed representations (Figure 4(e) ).Notably, the diffusion features we are using are in fact the reconstructed representations, which answers why clean diffusion features can be obtained even though the inputs are noisy. Despite the reconstruction ability, however, many high-frequency details are potentially blurred out by input noises, and thus their reconstruction is mostly based on "imagination". This leads to possible drift from the original image during reconstruction [7] . Naturally, the content shift phenomenon in extracted diffusion features reflects the drift during reconstruction. Consequently, content shift is an inherent characteristic of diffusion models and diffusion features, which suggests its broad existence across models and timesteps. 
utilization of generation techniques: 
 According to the cause of content shift, we need to steer the reconstruction process back to the original image to suppress content shift. While it is viable to design new methods for this purpose, we find it also possible and more efficient to adopt off-the-shelf generation techniques. Specifically, as an inherent characteristic of diffusion models, content shift affects not only features but also the original generative purpose of diffusion models. Hence, there have been techniques for generation that are already capable of toggling content shift by steering reconstruction [7, 58] . For example, ControlNet [58] introduces an additional reference image and steers reconstruction by directly modulating activations, pushing the reconstructed representation towards the reference image. It inspires us to utilize ControlNet in a different way: we can set the same input image simultaneously as the reference image, enforcing the recovered image to be more similar to the input one and thus suppressing content shift. Similarly, it is possible to adopt other generation techniques to suppress content shift. Furthermore, utilizing off-the-shelf generation techniques is more consistent with the intuition of diffusion feature than designing a new method. Specifically, this field is dependent on the generative capability of diffusion models, and thus it is important to stay updated with new advancements in the generation field. Utilizing techniques from the generation field helps with this goal, while a method that is newly designed solely for diffusion feature might hinder it. Considering it, we decide not to devise new methods, but to develop more detailed guidelines for suitably integrating these off-the-shelf generation techniques.We next illustrate the guidelines for utilizing off-the-shelf generation techniques. In the field of generation, the abundant techniques may not all be suitable for suppressing content shift, suggesting the necessity of examining the effect of a given technique on feature quality. Although it is possible to make empirical and quantitative examinations on discriminative tasks, a more efficient evaluation protocol would be preferable. To this end, we propose the GATE guideline for technique evaluation, which is presented in Figure 5 along with other components of the framework. To avoid having to integrate a technique into the feature extraction pipeline before knowing its potential, we utilize Img2Img generation as an alternative for evaluation:(i) Gather an Img2Img generation result as the reference image, done with high repainting strength to amplify content divergence for observation. (ii) Apply the technique to be evaluated and perform another Img2Img generation. If the technique requires parameters, they should be set such that the output could be more similar to the input. (iii) If the new output is more similar to the input than the reference image, the technique is considered able to suppress content shift in diffusion features.We have also developed a quantitative metric for evaluating generation techniques. We set feature extracted at t = 0 as reference F EAT ref ∈ R c×h×w as it is less affected by noises. Then we use the Laplacian operator to evaluate the contour difference between feature F EAT and the reference:EQUATIONWe then set a feature with stronger content shift as an anchor and compare dif f anchor with other features.EQUATIONScore = 1 means an exact match, and a smaller value indicates more shift. In this way, we can measure the extent of content shift in extracted features using different generation techniques and thus evaluate the suppression effect of techniques. Noticeably, the evaluation of this quantitative metric can be well approximated by the previously proposed qualitative evaluation, so we recommend the qualitative evaluation if efficiency is desired.We select three generation techniques as our implementation of GATE. Their Img2Img generation results are provided in Figure 6 . Integrating the techniques only slightly impacts efficiency, which will be discussed in Appendix A along with technique details. Additionally, we analyze two failed Fine-Grained Prompts. Prompts are a description of expected image content in natural language. For example, "a single horse running in a sports field, with a well-equipped rider on its back, high quality, highly realistic, masterpiece". Modern diffusion models are inherently trained to generate images conditioned on the given prompts [34] . Hence, by describing the content of the input image in prompts, it is possible to steer the reconstruction to stay close to the input. The result of fine-grained prompts in Figure 6 is not significantly better than the reference, but this technique enjoys the advantage of being a built-in function of diffusion models, requiring no code modification.ControlNet. Quite often, the control via prompts is too ambiguous. This is when ControlNet [58] can be helpful. ControlNet is a plug-in module for diffusion models, designed to take an additional control image input and push the reconstructed representation toward the control image. ControlNet presents an outstanding control effect over generation among the three techniques, as shown in Figure 6 , implying its potential for suppressing content shift.LoRA. LoRA [17] is an efficient replacement for model fine-tuning, which bears a similar effect to fine-tuning, thus showing to be highly effective for capturing image styles. By injecting additional knowledge of image styles of the given dataset into model weights, LoRA enables reconstruction to end with stronger similarity to the input image. From Figure 6 , we can observe that although LoRA does not provide as strong a control effect as ControlNet, it can significantly promote image quality, which is believed to bring some unique advantages.The three techniques above are able to improve feature quality individually and can be applied simultaneously for stronger suppression effects. While this can lead to a single high-quality feature, it is a common practice in previous diffusion feature approaches to amalgamate multiple features for further improvement [1, 24, 62, 29] . The conventional way extracts features at different timesteps to obtain more diverse information. We find that, compared to the conventional amalgamation of timesteps, the amalgamation of different combinations of generation techniques can bring stronger diversity, as indicated in Figure 7 . Therefore, we additionally amalgamate features obtained with various generation technique combinations to harness further enhancement. More details of how feature amalgamation is implemented are provided in Appendix A.6 Experimental Validation 
experimental settings: 
 Task & Dataset. Typically, diffusion feature studies [1, 50, 60] prefer fine-grained pixel-level tasks for evaluation. Following this practice, we select three tasks for experiments: semantic correspondence using SPair-71k [26] dataset, label-scarce semantic segmentation using Bedroom-28 [54] and Horse-21 [54] datasets, and standard semantic segmentation using ADE20K [61] and CityScapes [5] datasets.Evaluation Metrics. (i) For semantic correspondence, PCK@0.1 img (↑) and PCK@0.1 bbox (↑) are used, following the widely-adopted protocol reported in [26] . (We omit @0.1 to save some space in Table 1 .) These two metrics mean the percentage of correctly predicted keypoints, where a predicted keypoint is considered to be correct if it lies within the neighborhood of the corresponding annotation with a radius of 0.1 × max(h, w). For PCK@0.1 img /PCK@0.1 bbox , h, w denote the dimension of the entire image/object bounding box, respectively. (ii) For semantic segmentation, we use mIoU metric, which is the mean over the IoU performance across all semantic classes [12] . For each image, IoU (Intersection over Union, ↑) is defined by #(overlapped pixels between the prediction and the ground truth) / #(union pixels of them). In addition, we also use aAcc and mAcc, where aAcc is the classification accuracy of all pixels and mAcc averages the accuracy over categories.Feature Extraction. All tasks extract features at t = 50. When ControlNet is applied, except for standard semantic segmentation, we additionally start multi-step denoising from t = 60. For feature amalgamation, we extract multiple convolutional features and one attention feature per image:(i) Semantic correspondence: We obtain six in total convolutional features using individual finegrained prompt, ControlNet, and LoRA techniques, and one attention feature using a prompt including all object categories, with ControlNet and LoRA.(ii) Label-scarce semantic segmentation: We obtain one convolutional feature using fine-grained prompts, one convolutional feature using ControlNet, and one (Bedroom-28) to two (Horse-21) features using different LoRA weights. One attention feature is extracted with all three techniques applied.(iii) Standard semantic segmentation: One convolutional feature is obtained using only fine-grained prompts and two more are extracted additionally with ControlNet and different LoRA weights. One attention feature is extracted with all three techniques applied.Notably, the ADE20K dataset for standard semantic segmentation contains images of varying scenes, which can test how well the fine-grained prompt technique can generalize in this scenario. To this end, we use a prompt that can cover different scenarios: "a highly realistic photo of the real world. It can be an indoor scene, or an outdoor scene, or a photo of nature. high quality". This prompt covers various scenes for generalizability and describes image quality for fine-grained effect.For more experimental settings, including more detailed feature extraction methods and implementation details, please refer to Appendix C.The experimental results are shown in Table 1 and Table 2 . For most SOTA competitors, we borrow the reported results from their original studies. However, MaskCLIP [8] and ODISE [50] only provide results on ADE20K and it is hard to extend their implementations to CityScapes, so their results on CityScapes are missing. Furthermore, the original results reported by VPD [60] are based on full-scale fine-tuning of diffusion UNet, which is not fair as we do not train the diffusion model. Therefore, we re-evaluated VPD with the diffusion UNet frozen and reported our results.Semantic Correspondence. It is a pity that the related studies do not perform experiments under exactly the same setting. For fairness, we mainly compare GATE against a baseline method, which uses one feature extracted without any technique, under a unified setting. For reference, we still provide the results from four state-of-the-art methods: DINO [3] and DHPF [27] as non-DF methods, as well as DIFT [37] and DHF [24] as DF methods. The results are in the left part of Table 1 .Semantic Segmentation. The strong generalizability of a pre-trained diffusion model can ensure good discriminative performance even when labeled training data is scarce. For this scenario, we show the results in the right part of Table 1 . The SOTA diffusion feature method for this setting, DDPM [1] , serves as the major competitor. We also include other segmentation methods: DatasetGAN [59] , DatasetDDPM, MAE [13] , SwAV [2] , GAN Inversion [40] , GAN Encoder, VDVAE [4] , and ALAE [32] . We further validate our method on the more common setting of semantic segmentation using standard datasets: ADE20K [61] and CityScapes [5] , with the results presented in Table 2 . The competitors are MaskCLIP [8] , ODISE [50] , and VPD [60] , where VPD is re-evaluated with the diffusion model frozen for fairness.Generally, GATE outperforms competitors by large margins on all datasets. It demonstrates that previous diffusion feature approaches have been hindered by content shift, and GATE has a promising application as a generic booster for feature quality.In Figure 8 , we provide feature visualization for qualitative analysis of GATE. The visualization is obtained using PCA analysis, reducing the channels of features to 3, which are regarded as RGB for visualization. We can observe: (i) The attention features become clearer and closer to the input image when more generation techniques are applied according to GATE, showing the suppression effect on content shift. (ii) Notably, for the second image where a person is riding a horse, the baseline attention feature fails to follow the instruction, i.e., attending only to the horse and ignoring the person. In contrast, generation techniques applied according to GATE help attention features attend to the correct object. (iii) From convolutional features, we can see the application of generation techniques brings stronger diversity. In the second image, the attention feature is asked to focus on the horse and ignore the rider. The mIoU performance is on a single Horse-21 split, with red/blue for the best/worst.For ablation study, we aim to evaluate the effect of selected techniques without feature amalgamation. The discriminative performance is shown at the bottom line of Figure 8 , which is obtained on a single Horse-21 split instead of five random repeats for faster evaluation. We can observe: (i) Every individual technique can improve feature quality over baseline. (ii) When multiple techniques are applied simultaneously, stronger improvement can be obtained. This demonstrates that all three selected techniques can benefit feature quality, and their benefits can be combined together. 
conclusion and future work: 
 In this paper, we reveal a phenomenon named content shift that has been causing degradation in diffusion features. Based on the analysis of its cause, we propose to suppress it with off-the-shelf generation techniques, which allows hitchhiking the advancements in generative diffusion models. This approach, while enjoying simplicity, is experimentally demonstrated to be generically effective.However, the effectiveness of GATE relies on the selected techniques, for which we propose both a qualitative evaluation guideline and a quantitative metric. Though we selected three effective techniques and reported failed cases, there still is more to explore, which might potentially lead to more effective implementations. Furthermore, we only experimented with three tasks, so the full potential of GATE might remain under-explored. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original papers where related assets are first mentioned and list their license in Appendix G.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this section, we will explain the implementation details of each selected technique and feature amalgamation. Further discussion will also be provided on how these techniques manage to suppress content shift and the efficiency impact of our implementation.Fine-grained prompts can be integrated into diffusion feature very simply. We only need to replace the simple and short prompts, such as "a photo of a horse" in most DF approaches [37, 9, 45, 55, 21] , with more complex ones. Since prompts are a built-in function of current diffusion models, the only required integration of fine-grained prompts is to generate these prompts. To fulfill our goal, we need detailed descriptions of the image content, and many image captioning models can serve this purpose, such as Kosmos-2 [31] . It is also possible to manually design prompts that can cover various images, which is found to be almost as effective as auto captioners. When this approach is used, our observation is only in the training set to avoid data leaks. While this is effective for convolutional features, attention features require more consideration. To be specific, using specific prompts for each image causes inconsistency in the structure of attention features. To fill this gap, we always use manual and fixed prompt for attention features, even when convolutional features utilize auto-captioners. This fixed prompt is designed by observing images and describing the common objects. For example, "bedroom, a bed, some bedroom furniture, lights, a door, ceiling, floor, walls, pillow, quilt, chair and table, window, in good quality".Regarding the efficiency of integrating fine-grained prompts, we do not make any changes to the original feature extraction pipeline but only introduce the overhead of image captioning. Theoretically, this is still linear complexity. In practice, the actual time consumption depends on what auto-captioner is used. Moreover, these prompts can be re-used for the extraction of many groups of features, further reducing the proportion of overhead in the total time.ControlNet requires a parameter, the control signal, which should be set as the input image itself to suppress content shift. More specifically, the control image ControlNet requires is not an ordinary image, but a specially processed one, such as a depth image or a canny image. Among all control image types, we find that canny images are exceptionally efficient because their process does not use any neural network model. Therefore, we use canny as the sole control type for feature extraction.Although the simple usage is already yielding satisfying results, we find an additional way to further exploit ControlNet. It is reported that taking multiple denoising steps before feature extraction can make the attention feature less blurry [51] . However, taking multiple denoising steps is not a common practice for diffusion features, as it also brings severe performance drops caused by content shift. With ControlNet, it becomes possible to harness this good property without failing to suppress content shift. We only do this for attention features, so convolutional features are still extracted in a one-step manner.Regarding efficiency, processing an image into a canny image takes a very small overhead, So the major overhead introduced by ControlNet should be extracting attention features with multiple denoising steps. Whilst this does take obviously longer time than one-step feature extraction, its impact is limited due to two factors: (i) We empirically find that the optimal effect comes with less than 10 denoising steps. If more steps are taken, attention features will meet unacceptable content shift even with ControlNet applied. (ii) Different from the diversity effect in convolutional features, attention features do not see obvious enhancement when amalgamated. Therefore, we only extract one attention feature per image and concatenate it to the amalgamation result of convolutional features. Considering the two factors, taking multiple denoising steps does not in fact contain many steps and its use is relatively rare in the overall feature extraction process. Thus, the efficiency impact of integrating ControlNet is small.The integration of LoRA is also simple. Integrating LoRA into feature extraction only requires switching the base model weights to LoRA weights, so the major effort might be training a good LoRA weight. Fortunately, the community has offered many handy tools for ordinary users, such as kohya_ss 3 , which can be directly utilized for our purpose. Following the tradition from the community, we choose 30 random images from the training set to train a LoRA weight. Specifically, we choose the LoCon type of LoRA, which additionally inserts trainable parameters between convolutional layers, and follow the corresponding guides for training.The major overhead introduced by LoRA is its training. As a popular tool for AIGC players, the training for LoRA is feasible even on personal computers, showing its efficiency. In our experiments, training one LoRA weight usually takes less than 30 minutes. Moreover, the trained LoRA weights can be re-used, further reducing the proportion of overhead in the feature extraction process.Feature amalgamation can be accomplished by simply concatenating different features together. However, this is found suboptimal, leading to a superior strategy for feature amalgamation, which is weight assignment [24, 62] . Roughly, it assigns weights to each feature and amalgamates the features via a linear combination according to the weights. We next formalize this process. We have b feature maps r 1 , r 2 , • • • , r b ∈ R c×w×h for each image, with c, w, h being channel, width, and height, respectively. Weight assigners are simple MLP or CNN networks parameterized by θ and can be denoted as f (•; θ). Considering that the downstream task might benefit from various perspectives, there are n assigners. Assigning weights w j i is denoted as:EQUATIONAfterward, we take the linear combinations of features and concatenate them as the final feature:EQUATIONWe notice that this weight assignment strategy is suboptimal. To be specific, the assigners tend to converge to a trivial solution, where they assign weights almost equally to all features and different assigners share a similar prediction. To tackle this issue, we introduce two extra regularization terms. The first one encourages sparsity, meaning the assigners should concentrate on only a few features. The second one, named diversity, promotes a larger discrepancy between the predictions of different weight assigners. Formally, the loss function is:EQUATIONwhere L is the original loss of the downstream task; γ 1 and γ 2 are the hyper-parameters of the regularization terms; w j := (w j 1 , w j 2 , • • • , w j b ) T , and ∥ • ∥ 2 denotes the ℓ 2 -norm. Appendix D.5 will visualize the direct effect of the proposed regularization on weight distribution. Since attention features do not show the same diversity effect as convolutional features, the two types of features are processed differently. To be specific, we obtain many convolutional features but only one attention feature per image. Feature amalgamation is conducted among convolutional features, while the attention feature is concatenated to the final amalgamation result.Next, we aim to explain why the two proposed regularization terms can achieve their design purpose. Note that the weights of all features are restricted to sum up to 1, which means the ℓ 1 -norm of w is always 1. For a vector with a fixed ℓ 1 -norm, a larger ℓ 2 -norm means concentrating on less axes among all dimensions. Therefore, we use the negative value of its ℓ 2 -norm as sparsity regularization to promote a larger ℓ 2 -norm. As for diversity regularization, it simply measures and promotes the discrepancy among the predictions of different weight assigners. Compared with previous weight assignment approaches [24, 62] , we only add two regularization terms, thus introducing negligible overhead. Weight assignment itself does trade efficiency for performance. However, since this approach has already been adopted in previous approaches, we think it is rational to only consider the additional efficiency impact of our proposed regularization terms.Classifier-free guidance [16] is a widely adopted generation technique in current diffusion models. Its design purpose is to sacrifice diversity for better fidelity, similar to low-temperature sampling [18] in other generative models.We first show its Img2Img generation results according to the GATE guideline in Figure 9 . It is clear that it makes little difference to the similarity between outputs and the input, already suggesting that it lacks the potential for suppressing content shift. Since it can also be implemented simply, we integrate it into feature extraction as well and actually examine its effect, leading to the same conclusion as the previous quick evaluation following the GATE guideline.Why is classifier-free guidance unable to suppress content shift? It is designed to gain fidelity at the cost of diversity. While better fidelity is irrelevant to content shift, it might be helpful to lower the diversity of reconstruction given the blank caused by noises. However, the less diverse reconstruction is still not guaranteed to be centered on the original content. Therefore, even though classifier-free guidance can reduce diversity, content shift will not be suppressed. We hope this failed case can provide more insights into how generation techniques affect content shift and how to select them.IP-Adapter is a generation technique designed for image variation, which shares a similar architecture to ControlNet. By inputting images, IP-Adapter helps generate new images with some elements taken from the reference one. Since the goal of IP-Adapter is image variation instead of strict control, it is less effective in mitigating content shift than ControlNet. The weaker effectiveness of IP-Adapter is experimentally demonstrated on the Horse-21 dataset, as shown in Table 3 . IP-Adapter is not entirely ineffective for content shift, but less effective than ControlNet and thus not included in our implementation.For the diffusion model to extract features, we choose Stable Diffusion v1.5 [34] to be consistent with SOTA competitors. For semantic correspondence, two settings are examined: (i) directly performing nearest neighbor algorithm [39] (nn), (ii) adding an extra convolutional layer after each assigner (conv).For label-scarce semantic segmentation, the downstream model follows the framework in [1] , keeping hyper-parameters unchanged. For standard semantic segmentation, we instead use UPerNet [49] as the downstream model. γ 1 and γ 2 , the hyper-parameters for the proposed regularization terms, are tuned for each dataset. Choosing the number of weight assigners is mainly a tradeoff between efficiency and performance (Appendix D.4). We use less than three based on our resources.C. Attention features are obtained using a prompt including all object categories, with ControlNet (denoising from t = 60) and LoRA also applied.(i) A feature using fine-grained prompt "a horse, high quality, best quality, highly realistic, masterpiece". (ii) A feature using ControlNet, with a simple prompt "a horse". (iii) Two features using different LoRA weights and a simple prompt. The first LoRA weight is trained to generate high-quality images, while the second LoRA weight is trained until it slightly overfits.Attention features are obtained using a prompt "a photo of a single horse running in a sports field, with a well-equipped rider on its back, seems they are in a competition, high quality, best quality, highly realistic, masterpiece", with ControlNet and the first LoRA weight applied. Additionally, we concatenate the output feature of amalgamation with a feature extracted using DDPM [1] .Convolutional features include:(i) A feature using fine-grained prompt "a photo of a tidy and well-designed bedroom". It is found that quality prompts such as "high quality" will be interpreted as the quality of the room instead of the image, so such prompts are not utilized. (ii) A feature using ControlNet, with a simple prompt "a bedroom". (iii) One feature using LoRA weight, which is moderately trained to generate high-quality images.Attention features are obtained using a prompt "bedroom, a bed, some bedroom furniture, lights, a door, ceiling, floor, walls, pillow, quilt, chair and table, window, in good quality". ControlNet and LoRA are also applied. Additionally, we concatenate the output feature of amalgamation with a feature extracted using DDPM [1] .(i) One feature using only fine-grained prompt "a highly realistic photo of the real world. It can be an indoor scene, or an outdoor scene, or a photo of nature. high quality" plus all category labels. No attention features are used. We find that in very complex scenes attention features can be of low quality and bring slight degradation instead of enhancement.Convolutional features include:(i) One feature using only fine-grained prompt "An urban street scene with multiple lanes, various buildings, traffic lights, cars in the lanes, and pedestrians, highly realistic".(ii) Two features using the same prompt as above and additionally ControlNet and LoRA. The two features use two different LoRA weights, where one is moderately trained and the other slightly overfits.No attention features are used, based on the same observation as for ADE20K.For CIFAR10, high-quality and low-quality prompts are as follows:"cinematic shot photo taken by ARRI, photo taken by sony, incredibly detailed, sharpen, masterpiece, best quality, realistic, HD, raytracing, CG unified 8K wallpapers" "low quality, low resolution, blurry, draft, grainy, disfigured, deformed, low contrast, underexposed, overexposed, bad art"For semantic segmentation, they are:"a horse, high quality, best quality, highly realistic, masterpiece" "a horse, low quality, worst quality, deformation, broken shape, blurry"Besides the feature visualization in Figure 8 , we provide additional visualization here in Figure 10 .We would like to further demonstrate the generalizability of GATE by applying it to a different diffusion model, SDXL [33] . The architecture of SDXL has been significantly modified to include more complicated attention structures while its overall upsampling blocks are simplified. Due to the architecture change, the previous method for feature extraction cannot fully exploit the potential of SDXL, thus making the performance lower than Stable Diffusion v1.5. Nevertheless, it is enough to Observing the results in Table 4 , we can again confirm the benefit of GATE. Both techniques can enhance performance. Moreover, combining two techniques is able to bring further improvement. This demonstrates the generalizability of GATE as it can be integrated into different diffusion models.Figure 11 : Sensitivity analysis of the regularization terms, where γ 1 /γ 2 is for sparsity/diversity. These results validate the necessity of these regularization terms.Now we aim to examine the effect of our regularized weight assignment strategy for feature amalgamation. We will first demonstrate that amalgamating many features leads to better performance than a single feature. Then we will conduct an analysis of the hyper-parameter sensitivity regarding the proposed regularization terms.The first experiment is conducted on the same Horse-21 split as Figure 8 . In addition, this experiment only applies techniques individually, not simultaneously, for a simpler setting. The following observations can be made from Table 5 :(i) Amalgamating many features is generally superior to a single feature. Even if we simply use features obtained from different timesteps, it is beneficial.(ii) If the features are from different technique combinations rather than simply different timesteps, the performance will be higher. This shows that applying various techniques can yield more diverse diffusion features.(iii) If more features are used, the performance will be better than when only two are used.To analyze the sensitivity property of the two regularization terms, we perform a grid search on the semantic correspondence task. The reported metric is PCK@0.1 img (↑) and the model type is nn.Based on Figure 11 , we can conclude that both regularization terms contribute to the performance gain, but they need tuning. We have tuned them for the results in Table 1 .In this part, we provide a supplementary ablation study on the number of weight assigners. The results in Table 6 show that when we increase the number of weight assigners, the overall performance will improve gradually. However, we have two following concerns:(i) More weight assigners will take a longer time to converge. The training steps that are enough for fewer assigners to fully converge are insufficient for more assigners, indicated by that the the performance of more assigners still improves with more training steps.(ii) Increasing the number of weight assigners will bring more memory usage and more time consumption, which will affect efficiency.Consequently, choosing the number of weight assigners is mainly a tradeoff between performance and efficiency. We aim to visualize the impact of our proposed regularization terms on weight distribution, to show that their effect aligns with the design purpose. The statistics are shown in Figure 12 . We select the semantic correspondence task under nn setting with two weight assigners.The first four figures visualize the impact of sparsity. The first two show the Gaussian probability density fitted to the mean and variance of top-1 and top-2 weights. The other two show the average value of all weights. Without regularization, top-1 weights will be rather small. With regularization, top-1 weights will become much larger, showing that the prediction of assigners becomes more concentrated.The last two figures look into diversity. We compare the predictions of two assigners. If no overlap is observed in top-1 nor top-2 weights, it counts as no overlap, which is the best case. If overlap occurs in top-1 weights, it is regarded as overlap @ top-1. If overlap only occurs when we consider both top-1 and top-2 weights, this is overlap @ top-2. It is clear that diversity regularization can effectively force assigners to focus on different feature maps, thus providing information from various perspectives.Although image-level tasks are not typically utilized for the evaluation of diffusion features, we have used this task for early empirical studies. Hence, in Table 7 , we provide some results on CIFAR10, in comparison to a standard ResNet backbone 4 and a baseline where diffusion features are used without GATE.Long tail and out-of-distribution scenarios are challenging problems for discrimination, where a model that works well under i.i.d. settings can degrade significantly [43] . There have been a few attempts to apply diffusion models to such challenging scenarios [10, 46, 56] , but more efforts are still considered helpful. Recently, there has been a study [11] that might open more opportunities to this topic. To be specific, AUC is a useful metric and loss function for long tail and out-of-distribution studies [41, 42] , but it used to be not applicable to pixel-level tasks such as semantic segmentation. This recent study, however, has successfully introduced a new AUC-based tool that can be applied to segmentation as well, enabling more future studies in this direction. We use Nvidia(R) RTX 3090 and Nvidia(R) RTX 4090 GPUs for the experiments, all with 24GB VRAM.The experiments on semantic correspondence take about 5 hours per run and about 50GB of disk storage. The experiments on Horse-21 and Bedroom-28 take about 5 hours per run, where one run consists of 5 repeats on different splits. Furthermore, each run uses about 45GB of disk storage. The experiments on ADE20K and CityScapes take about 2 days per run with no additional disk usage besides what is needed to store model checkpoints and datasets. Our early exploration of content shift utilizes relatively lightweight experiments for efficiency. Justification: Related information is provided in Appendix F.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: We conform with the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://civitai.com/https://github.com/bmaltais/kohya_sshttps://github.com/kuangliu/pytorch-cifar 


Decision: This paper is about suppressing the content shift in recently popular "diffusion features" (using pre-trained diffusion models to retrieve feature representations). The phenomenon of content shift is newly discovered by the authors, i.e., the content of the feature representation shifts from the original input due to the "noisy" nature of diffusion models. To resolve this problem, the authors propose to utilize popular conditioning techniques (e.g., prompt-based, ControlNet, and LoRA) in a very unusual but clever way: conditioning the features based on the very original image. This is very simple but quite effective, achieving state-of-the-art performance with substantial improvements. The proposed method is also quite efficient in that off-the-shelf techniques can be directly used, which also philosophically conforms with the recent trends of AI. All reviewers have agreed on the potential impact of the topic itself and the proposed method.

However, there was a not-so-small issue during the author-reviewer discussion: The original submission lacked details and was written at a higher level, so a large number of additional details were submitted as a rebuttal. This amount concerned some of the reviewers, and in response, the authors have posted a partial revision of the paper as another rebuttal. This is not allowed in the NeurIPS policy, and there was a heated discussion between the authors and the reviewers regarding its appropriacy. After a discussion between AC and SAC, it was decided that the reviewers should ignore the revision and continue the discussion with AC. The issue still resided in that the amount of revision needed is quite substantial.

After the discussion, the AC personally reviewed the paper in detail to yield the following conclusion: The required revision is mostly about details and not the core contribution, as the authors argued. The method is largely based on very well-known off-the-shelf techniques (diffusion features, conditioning techniques like prompt-based, ControlNet, and LoRA), and the only novel twist introduced in the paper is simply changing the conditioning input to an unusual one (the original input itself), which is in fact not quite different from the usual ones (some other input to steer the generated result) in characteristics. The details needed here are more about tuning issues rather than deal-breaking design issues. Accordingly, the required revision is not too substantial (in a qualitative sense), considering the merit of the topic/message and the effectiveness/efficiency of the proposed method.

That being said, the additional details discussed in the rebuttal are indeed required and must definitely be added to the final camera-ready version. Under this strict condition, an accept decision is recommended.

Reviews: review: paper_summary: paper_summary: This paper addresses the issue of content shift in diffusion features, which are inner activations extracted from pre-trained diffusion models used for discriminative tasks. The content shift refers to the phenomenon where there are content differences between the features and the input images. In the manuscript, the authors showed the cause of content shift as an inherent characteristic of diffusion models, related to information drift during the image recovery process from noisy inputs. The proposed method utilizes off-the-shelf generation techniques to suppress content shift and introduces GATE for evaluating and implementing the techniques. The proposed method is simple yet novel.
strength_weakness: strength_weakness: - The paper lacks clarity in explaining the exact process of how attention features are extracted from the diffusion model. Figures 1 and 2, which are supposed to illustrate this process, do not provide enough detail.
- The details of the overall process of the proposed method are missing. 
- The effectiveness of the GATE guideline is demonstrated using a limited set of generation techniques. A broader range of techniques should be evaluated to strengthen the claim of its general applicability.
- Even though the authors provided comparison experiments in the manuscript and ablation study in the supplementary materials, the results do not clearly demonstrate the superiority of the proposed method. More detailed and extensive comparative analyses with additional evaluation metrics and qualitative evaluation are needed to show the advantages of the proposed method.
questions: questions: 1. Please provide a more detailed explanation and additional diagrams to clarify how attention features are extracted from the diffusion model in Figure 1 and 2.
2. Please provide details of the overall process of the proposed method.
3. How does the proposed method perform on tasks involving different types of images or datasets with varying levels of noise and complexity?
limitations: limitations: 1. The paper should improve the clarity of the proposed method, especially in explaining the extraction of features from the diffusion model. Providing a more detailed and step-by-step explanation would help readers understand the method better.
2. The authors should consider demonstrating the experimental results in a better way with additional evaluation metrics and extensive qualitative evaluation.
paper_summary: paper_summary: The authors focus on the task of suppressing content shift in diffusion features using off-the-shelf generation techniques. The authors observe that diffusion features, which are extracted from pre-trained diffusion models, suffer from a phenomenon called content shift, where there are content differences between the input image and the features extracted. The strength of the proposed method lies in utilizing off-the-shelf generation techniques to suppress this content shift, leveraging their ability to control the recovering process in diffusion models. Experimental validation shows that the proposed GATE framework effectively enhances diffusion features across various tasks and datasets, outperforming state-of-the-art methods.
strength_weakness: strength_weakness: (1) Some details can be more clarified. For example, the authors propose to use  fine-grained prompts to surpass content shift. I wonder how to design these prompts. Does it require us to design prompts for each input? 
(2) The paper focuses on empirical validation and lacks a thorough theoretical analysis of why and how the proposed techniques effectively suppress content shift. It is recommended to provide more insights or explanations.
questions: questions: See the weakness.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper explores how diffusion models, typically used for generative tasks, can also serve discriminative purposes by using inner activations as features. The authors point out that these features often suffer from a phenomenon named content shift, i.e., the features are semantically different from the input image. This content shift negatively impacts the performance of diffusion features. The authors propose to use off-the-shelf generation techniques to suppress content shift. To this end, they introduce a practical guideline called GATE. Their approach, which is validated through extensive experiments, significantly improves performance on various tasks and datasets.
strength_weakness: strength_weakness: 1) The authors propose the GATE guideline for evaluating generation techniques. Then, the actual implementation includes three successful techniques and a failed one. However, as pointed out by the authors, there are abundant techniques for generation. Hence, it is recommended to provide more failure cases to validate the negative impact of the content shift.
2) Some details can be further clarified. To be specific, which results are implemented by the authors or borrowed from previous works could be more explicitly stated. Besides, some results in Table 1 and Table 2 are missing. The authors should make a further explanation.
3) Besides, some typos and minor errors found in the paper:
- Section 2, Line 66: "diffusion feature" should be "diffusion features".
- Section 4.2, Line 136: "consist the diffusion features we extract" should be "constitute the diffusion features we extract".
- In reference [1], “ECCV Workshopt” should be “ECCV Workshop”.
questions: questions: Please see the weakness.
limitations: limitations: Yes, the authors discuss their limitations in Section 7.
paper_summary: paper_summary: The paper describes a pipeline for improving feature extraction from pre-trained diffusion models for general-purpose task such as semantic segmentation. The focus is on "content shift", where extracted feature maps at a certain time step in the diffusion process show differences in the content composition compared to the input image. Three previous methods for controlling image to image transformation with diffusion models are used for extracting features and attention maps. Subsequently, these are fused by a learned weighted linear combination, or amalgamation, providing updated image features for use in the downstream application. The pipeline is tested for semantic correspondence on the SPair-71k dataset, and for semantic segmentation on the Bedroom-28, Horse-21, ADE20K, and CityScapes datasets. The results show improvement over the compared baselines (vanilla feature extraction, and some other previous feature extraction techniques).
strength_weakness: strength_weakness: - The paper is written on a quite high level, with many of the relevant details left out or put in the appendix
- The technical contribution is rather limited, combining previous techniques for the purpose of improving diffusion features
- The evaluation is lacking, with many details missing and without proper ablation study
questions: questions: * In my view, there are too many details left out in the main paper, related to, e.g., method, datasets, and experiments. In its current state, it relies too much on details in the appendix. For example, the dataset used for semantic correspondence in Tab. 1 is not even specified, metrics are not explained, implementation details are missing, etc. To me, this is more important compared to the lengthy and speculative motivation of content shift in Section 4. It is rather obvious that features can shift away from the input image with excessive noise added. I would suggest moving most of these discussions and figures to the appendix, and instead provide the necessary details for understanding the pipeline and experiments.

* Although the appendix provides further information, there are still missing details and experiments. For example, one central hyper-parameter is the noise level used for feature extraction. This is neither specified nor tested in experiments. What noise levels were used? How were these selected? How is the improved performance of the suggested pipeline reflected over different selections of noise levels? Is it possible to combine a larger number of different noise levels, e.g., in the learned combination?

* Rather than the speculative motivation in Section 4, is there a more systematic way of quantifying the content shift? As it looks now, this is rather speculative and only verified through a few example images. Since this is the main focus, it would be important to measure and compare, e.g., how it behaves over different noise levels and for different feature extraction techniques.

* The experimental setup now relies on having rather specific types of images (horses, bedrooms, street view, etc.), so that a general text prompt can be provided for the fine-grained prompt image generation. How can this generalize to more diverse types of images?

* In Fig. 5, there are some examples for classification on CIFAR-10, but there are no experimental results on classification. How does the proposed method perform in classification tasks? Isn't this the most straightforward strategy for evaluation of feature quality?

* How are clean features obtained from the noisy features, in Fig. 4?

* In summary, although there are interesting observations, the manuscript is to me not yet mature enough, and lacking both in details and experiments.
limitations: limitations: * Limitations are only very briefly mentioned in the conclusions. These could be elaborated on, in connection to the points discussed above.

Comments: title: Please respond to the authors' rebuttal.
comment: Dear Reviewer z9si,

We greatly appreciate your diligence, expertise, and efforts!

We are delighted to hear from you that our detailed rebuttal and clarifications have effectively addressed your concerns. Through the rebuttal and discussion, we pursue better ways to clarify some important concepts of our method (such as the feature extraction pipeline and some experimental details), while the core idea and main contributions stay unchanged. Perhaps due to the way of our writing, these concepts are not fully clarified in the original version. Furthermore, it seems that the rebuttal policies allow such revision for clarifications and analyses, according to the additional page for the camera-ready version.

For your convenience, we attach the related policies right here.

> Author responses: Authors will have one week to view and respond to initial reviews. Author responses may not contain any identifying information that may violate the double-blind reviewing policy. Authors may not submit revisions of their paper or supplemental material, but may post their responses as a discussion in OpenReview. This is to reduce the burden on authors to have to revise their paper in a rush during the short rebuttal period.

> After the initial response period, authors will be able to respond to any further reviewer/AC questions and comments by posting on the submission’s forum page. The program chairs reserve the right to solicit additional reviews after the initial author response period.  These reviews will become visible to the authors as they are added to OpenReview, and authors will have a chance to respond to them.

> The main text of a submitted paper is limited to nine content pages, including all figures and tables. Additional pages containing references don’t count as content pages. If your submission is accepted, you will be allowed an additional content page for the camera-ready version.
Thank you for your detailed rebuttal and the clarifications provided. The additional explanations and diagrams are clear and helpful to clarify several aspects of the proposed method. The rebuttal have effectively addressed the concerned I raised. However, I want to mention that the authors' responses include substantial clarifications and additional analyses that may border on revisions rather than the typical rebuttal. I am not sure that this level of change aligns with NeurIPS policies on rebuttals.
(Part 4 of 4)
## 6.3 Qualitative Analysis

In Figure 10, we provide feature visualization for qualitative analysis of GATE.
The visualization is obtained using PCA analysis, reducing the channels of features to 3, which are regarded as RGB for visualization.
We can observe:
(i) The attention features become clearer and closer to the input image when more generation techniques are applied according to GATE, showing the suppression effect on content shift.
(ii) Notably, for the second image where a person is riding a horse, the baseline attention feature fails to follow the instruction, *i.e.*, attending only to the horse and ignoring the person.
In contrast, generation techniques applied according to GATE help attention features attend to the correct object.
(iii) From convolutional features, we can see the application of generation techniques brings stronger diversity.

## 6.4 Ablation Study: Effect without Feature Amalgamation

For ablation study, we aim to evaluate the effect of selected techniques without feature amalgamation.
The discriminative performance is shown at the bottom line of Figure 10, which is obtained on a single Horse-21 split instead of five random repeats for faster evaluation.
We can observe:
(i) Every individual technique can improve feature quality compared to baseline.
(ii) When multiple techniques are applied simultaneously, stronger improvement than each individual technique can be obtained.
This demonstrates that all three selected techniques can benefit feature quality, and their benefits can be combined together.

---

# 7 Conclusion and Future Work

In this paper, we reveal a phenomenon named content shift that has been causing degradation in diffusion features.
Based on the analysis of its cause, we propose to suppress it with off-the-shelf generation techniques, which allows hitchhiking the advancements in generative diffusion models.
This approach, while enjoying simplicity, is experimentally demonstrated to be generically effective.

However, the effectiveness of GATE relies on the selected techniques, for which we propose both a qualitative evaluation guideline and a quantitative metric.
Though we selected three effective techniques and reported failed cases, there still is more to explore, which might potentially lead to more effective implementations.
Furthermore, we only experiment on three tasks, so the full potential of GATE might remain under-explore

---

>In Appendix

# C IP-Adapter: Failed Case
IP-Adapter is a generation technique designed for image variation, which shares a similar architecture to ControlNet.
By inputting images, IP-Adapter helps generate new images with some elements taken from the reference one.
Since the goal of IP-Adapter is image variation instead of strict control, it is less effective in mitigating content shift than ControlNet.
The weaker effectiveness of IP-Adapter is experimentally demonstrated on the Horse-21 dataset, as shown in Table 3.

|Method|mIoU|
|:--|:-:|
|DDPM|65.0 ± 0.8|
|GATE (Only IP-Adapter)|65.8 ± 1.3|
|GATE (Only ControlNet)|66.2 ± 1.2|
|GATE (Full)|**67.2 ± 1.1**|

---

>In Appendix

## E.5 Image Classification

Although image-level tasks are not typically utilized for the evaluation of diffusion features, we have used this task for early empirical studies.
Here, we provide some results on CIFAR10, in comparison to a standard ResNet backbone and a baseline where diffusion features are used without GATE.

|Method|Acc(\%)|
|:-|:-:|
|ResNet-50|93.62|
|Baseline|94.55|
|GATE|95.21|
(Part 3 of 4)
## 6.1 Experimental Settings

**Task \& Dataset.**
Typically, diffusion feature studies \cite{baranchuk2021label, xu2023open, zhao2023unleashing} prefer fine-grained pixel-level tasks for evaluation.
Following this practice, we select three tasks for experiments: semantic correspondence using SPair-71k \cite{min2019spair} dataset, label-scarce semantic segmentation using Bedroom-28 \cite{yu2015lsun} and Horse-21 \cite{yu2015lsun} datasets, and standard semantic segmentation using ADE20K \cite{zhou2017scene} and CityScapes \cite{Cordts16city} datasets.

**Evaluation Metrics.**
(i) For semantic correspondence, $\text{PCK\@0.1}\_{\text{img}}(\uparrow)$ and $\text{PCK\@0.1}\_{\text{bbox}}(\uparrow)$ are used, following the widely-adopted protocol reported in \cite{min2019spair}. (We omit @0.1 to save some space in Table 1.) These two metrics mean the percentage of correctly predicted keypoints, where a predicted keypoint is considered to be correct if it lies within the neighborhood of the corresponding annotation with a radius of $0.1 \times max(h, w)$. For $\text{PCK\@0.1}\_{\text{img}}$/$\text{PCK\@0.1}\_{\text{bbox}}$, $h, w$ denote the dimension of the entire image/object bounding box, respectively.
(ii) For semantic segmentation, we use mIoU metric, which is the mean over the IoU performance across all semantic classes \cite{hao2020brief}. For each image, IoU (Intersection over Union, $\uparrow$) is defined by \#(overlapped pixels between the prediction and the ground truth) / \#(union pixels of them).
In addition, we also use aAcc and mAcc, where aAcc is the classification accuracy of all pixels and mAcc averages the accuracy over categories.

**Implementation Details.**
For all tasks, we extract features at $t=50$, which is an optimal noise level suggested in \cite{baranchuk2021label}.
When ControlNet is applied, except for standard semantic segmentation, we additionally start multi-step denoising from $t=60$.
For feature amalgamation, we extract multiple convolutional features and one attention feature per image:
- Semantic correspondence: We obtain six in total convolutional features using individual fine-grained prompt, ControlNet, and LoRA techniques, and one attention feature using a prompt including all object categories, with ControlNet and LoRA.
- Label-scarce semantic segmentation: We obtain one convolutional feature using fine-grained prompts, one convolutional feature using ControlNet, and one (Bedroom-28) to two (Horse-21) features using different LoRA weights. One attention feature is extracted with all three techniques applied.
- Standard semantic segmentation: One convolutional feature is obtained using only fine-grained prompts and two more are extracted additionally with ControlNet and different LoRA weights. One attention feature is extracted with all three techniques applied.

Notably, the ADE20K dataset for standard semantic segmentation contains images of varying scenes, which can test how well the three techniques can generalize in this scenario.
While ControlNet and LoRA are essentially insensitive to this scenario, very specific fine-grained prompts can meet some problems.
To this end, we use a prompt that can cover different scenarios: "a highly realistic photo of the real world. It can be an indoor scene, or an outdoor scene, or a photo of nature. high quality".
This prompt both covers various scenes for generalizability and describes the intended image quality for fine-grained effect.

## 6.2 Comparison with SOTA

The experimental results are shown in Table 1 and Table 2.
For most SOTA competitors, we borrow the reported results from their original studies.
However, MaskCLIP and ODISE only provide results on ADE20K and it is hard to extend their implementations to CityScapes, so their results on CityScapes are missing.
Furthermore, the original results reported by VPD are based on full-scale fine-tuning of diffusion UNet, which is not fair as we do not train the diffusion model.
Therefore, we re-evaluated VPD with the diffusion UNet frozen and reported our results.

(Remaining parts of the original subsection)
(Part 2 of 4)
## 4.2 Cause of Content Shift

After the existence of content shift is confirmed in diffusion features, we next aim to find its cause.
Unlike more conventional feature extractors such as ResNet \cite{he2016deep}, the inputs to diffusion models are not the original image (Figure 4(a)), but its noisy version (Figure 4(b)), as enforced by the diffusion process \cite{ho2020denoising}.
The early layers of diffusion models even further add more noise to the inner representations (Figure 4(c)).
However, the diffusion UNet gains the ability from vast pre-training to reconstruct clean inner representations from noisy inputs (Figure 4(d)), roughly at the middle of the UNet structure.
Additionally, the shortcut structures in UNet also help the reconstruction by passing some high-frequency details.
Afterward, the diffusion UNet will further predict noises based on the reconstructed representations (Figure (e)).

Notably, the diffusion features we are using are in fact the reconstructed representations, which answers why clean diffusion features can be obtained even though the inputs are noisy.
Despite the reconstruction ability, however, many high-frequency details are potentially blurred out by input noises, and thus their reconstruction is mostly based on "imagination".
This leads to possible drift from the original image during reconstruction \cite{Daras23consistent}.
Naturally, the content shift phenomenon in extracted diffusion features reflects the drift during reconstruction.
**Consequently, content shift is an inherent characteristic of diffusion models and diffusion features, which suggests its broad existence across models and timesteps**.

---

## 5.2 Quantitative Evaluation

We also propose a quantitative metric for the evaluation of generation techniques.
We set feature extracted at $t=0$ as reference $FEAT_{ref} \in \mathbb{R}^{c\times h\times w}$ as it is less affected by noises. We use the Laplacian operator to evaluate the contour difference between feature $FEAT$ and the reference:

$$    
diff=\sum_{i,j}^{h\times w}|\left\|Laplacian(FEAT_{ref},i,j)\right\|_2-\left\|Laplacian(FEAT,i,j)\right\|_2|\in(-\infty,1]
$$

We then set a feature with stronger content shift as an anchor and compare $diff_{anchor}$ with other features.

$$
    Score=\frac{(diff_{anchor}-diff)}{diff_{anchor}}
$$

$Score=1$ means an exact match, and a smaller value indicates more shift.
In this way, we can measure the extent of content shift in extracted features using different generation techniques and thus evaluate the suppression effect of techniques.
Noticeably, the evaluation of this quantitative metric can be well approximated by the previously proposed qualitative evaluation, so we recommend the qualitative evaluation if efficiency is desired.
(Part 1 of 4)
>In Introduction line 41-51

In pursuit of a method to suppress content shift, we need to further investigate why this phenomenon exists.
We notice in Section 4 that the diffusion backbone reconstructs clean inner representations from noisy inputs in the middle of UNet before predicting noises based on the reconstructed content.
The diffusion features we are using are in fact the reconstructed representations, which answers why we can obtain clean features from noisy images.
However, since high-frequency details are potentially blurred out by noises in the inputs and recovered by "imagination" gained from vast pre-training, this reconstruction process inherently suffers from the risk of drifting from the original image.
Content shift in diffusion features, naturally, reflects the drift during reconstruction.
Consequently, to suppress content shift, we need an additional way to directly send the original clean image into the middle of UNet and steer the reconstruction towards the original image.
To our delight, we notice that many off-the-shelf image generation techniques for diffusion models \cite{zhang2023adding, mou2023t2i, ye2023ip} also work by sending additional information into UNet and thus steering the reconstruction.
Hence, it is possible to select some techniques that can directly satisfy the goal of suppressing content shift, which eases the implementation and extension of our method.
Although the huge number of generation techniques might pose difficulty to the selection, we propose a guideline in Section 5 to bridge the gap.
This method is denoted as *GenerAtion Techniques Enhanced* (**GATE**) diffusion feature and its effect is also shown in Figure 1.

---

# 3 Preliminaries: Diffusion Feature

Diffusion models consist of a neural network module and a diffusion scheduler.
The network is an end-to-end network, which can be formally denoted as $\epsilon_\theta$, where $\theta$ is the parameters.
The diffusion scheduler is the core of diffusion models.
While previous generative models such as GAN expect to generate an image directly in one step, the diffusion scheduler turns the generation process into a timeline comprised of many timesteps.
Generally, we use a **smaller / larger timestep** to indicate **less / more noises**, and $t=0$ / $t=T$ for clean images / total noises.

Next, we will explain how convolutional features and attention features are extracted using a common pipeline for diffusion features, with the visual illustration in Figure 2.
Given an input image $x\in\mathbb{R}^{3\times h \times w}$, where $h, w$ are height and width, the extraction process has the following steps:
(i) A pre-trained VAE encodes the input image into the latent space, inducing $x_0\in\mathbb{R}^{4\times h' \times w'}$, as a common practice presented in \cite{rombach2022high, podell2023sdxl}.
(ii) $x_0$ acts as the input of the forward diffusion process, *i.e.*, timestep $t=0$.
As suggested in \cite{baranchuk2021label}, it is beneficial to extract diffusion features at non-zero timestep. Following this practice, we set $t=50$ and get $x_t$.
(iii) $x_t$, along with the timestep $t$, a textual prompt $c$, is sent into the pre-trained diffusion UNet $\epsilon_\theta$, *i.e.*, $\epsilon_\theta(x_t,t,c)$.
The generation techniques selected by our method are also applied here to modify $\theta, c$.
(iv) Convolutional and attention features are gathered during the computation of the backbone.
For convolutional features, we gather the output activations of each resolution in the upsampling stage of the UNet.
For attention features, we obtain the mean value of the similarity maps between query and key in all cross-attention layers.
We greatly appreciate your diligence, expertise, and efforts! Through the responses, we pursue better ways to clarify some important concepts of our method, while the core idea and main contributions stay unchanged. Thanks to the discussion with all reviewers, we have polished our presentation and integrated the responses. It is a pity that according to the policy, we cannot provide the revised PDF directly. To dispel your concern about the amount of such revision, we would like to clarify how each response will be included in the revised manuscript. 

Briefly, although with additional information, most responses require no additional space or only need tolerable space to complete the revision (**GQ1** - **GQ4**, **Q2** - **Q5**, and **Q7** - **Q10**). Although **Q1** and **Q6** do require additional space, the policy has considered such revision and allows one additional page in the camera-ready version. Hence, if this paper is fortunately accepted, we can easily add all these responses in a revision. Even if we do not have the additional page, the revision is also feasible by following your advice to re-organize Fig.3 - Fig.6.

**GQ1**: In this response, we add more details about the pipeline, especially how these features are extracted. In the revised version, these details will replace part of the previous preliminary section (Sec.3). Meanwhile, the new figure about pipeline details will replace Fig.2 in the preliminary section. All the revisions will take up little additional space.

**GQ2**: We analyze two failure cases in this response, one of which has already been included in Appendix. For revision, we will put the other one in Appendix as well, not taking up the space of the main body.

**GQ3**: This response explains how the three selected techniques generalize to images of various types, which was not clearly illustrated before. According to your advice, the experiment section (Sec.6) will include some necessary implementation details on the ADE20K dataset to clarify this issue.

**GQ4**: The experimental results in this response will be organized as follows.
- We add more metrics for the experiments on ADE20K and CityScapes. These new results will be directly integrated into the blank space in Tab.2, requiring no additional vertical space. 
- We provide the classification performance on CIFAR10. These results will be put in Appendix, since it is not a common practice to conduct experiments on this task.

**Q1**: We provide many experimental details in this response. In the revision, according to your advice, we will summarize some important details such as datasets, metrics, and implementation details. The full version, which has been presented in the global comment, will be put in the Appendix due to space limitation.

**Q2**: We will revise the introduction to better clarify the focus and novelty of this study.

**Q3**: This response clarifies the ablation study that has been presented in Fig.10. Perhaps due to our presentation, the reader might ignore the ablation results presented at the bottom of Fig.10. For better clarity, we will split the discussion on ablation study and qualitative analysis, which only requires simple re-organization.

**Q4**: please refer to the revision of **GQ1** and **Q1**.

**Q5**: We will add necessary experimental details to clarify the noise level we use.

**Q6**: In this response, we add the quantitative metric to measure content shift. We will add this content in Sec.5, requiring some additional space.

**Q7**: please refer to the revision of **GQ3**.

**Q8**: please refer to the revision of **GQ4**.

**Q9**: This response clarifies how clean features are obtained from noisy features. We will revise the discussion in Sec.4. to clarify this issue. According to your advice, we can reduce some empirical results if it is necessary.

**Q10**: We will revise the conclusion section.

---

For your convenience, we attach the revised content right here.
Thanks for your detailed responses and most of my concerns are addressed. I maintain my score.
Dear Reviewer z9si,  

As the rebuttal deadline approaches, could you please let us know if our newly added response has addressed your concerns?
Thank you once again for your time and valuable feedback!
Thanks for the exhaustive reply and clarifications! I agree that the analysis and method are of good value, and the clarifications made in the rebuttal addresses many of the problems with limited information. I will revise my score accordingly. However, my worry is that the large amount of additional information provided through the rebuttal is too much to add in a revision.
Dear Reviewers,

The authors have posted their rebuttals to the reviews. Could you please respond to the rebuttals?
Please engage in the discussion with the authors. Your help is much appreciated.

Thanks,

AC
We provide the full-scale experimental details here.  

We extract features at $t=50$. When ControlNet is applied, except for ADE20K and CityScapes, we additionally start multi-step denoising from $t=60$.
- **Semantic Correspondence**
	- Convolutional features: two features per setting under different random seeds:
		- Two with fine-grained prompt "a photo of x, high quality, best quality, highly realistic, masterpiece, high resolution", where "x" is the concatenation of all class labels.
		- Two with LoRA.
		- Two with ControlNet.
	- Attention features are obtained using a prompt including all object categories, with ControlNet (denoising from $t=60$) and LoRA also applied.
- **Label-Scarce Segmentation: Horse-21**
	- Convolutional features extracted at $t=50$:
		- A feature using fine-grained prompt "a horse, high quality, best quality, highly realistic, masterpiece".
		- A feature using ControlNet, with a simple prompt "a horse".
		- Two features using different LoRA weights and a simple prompt. The first LoRA weight is trained to generate high-quality images, while the second LoRA weight is trained until it slightly overfits.
	- Attention features are obtained using a prompt "a photo of a single horse running in a sports field, with a well-equipped rider on its back, seems they are in a competition, high quality, best quality, highly realistic, masterpiece", with ControlNet and the first LoRA weight applied.
	- Additionally, we concatenate the output feature of amalgamation with a feature extracted using DDPM [2].
- **Label-Scarce Segmentation: Bedroom-28**
	- Convolutional features extracted at $t=50$:
		- A feature using fine-grained prompt "a photo of a tidy and well-designed bedroom". It is found that quality prompts such as "high quality" will be interpreted as the quality of the room instead of the image, so such prompts are not utilized.
		- A feature using ControlNet, with a simple prompt "a bedroom".
		- One feature using LoRA weight, which is moderately trained to generate high-quality images.
	- Attention features are obtained using a prompt "bedroom, a bed, some bedroom furniture, lights, a door, ceiling, floor, walls, pillow, quilt, chair and table, window, in good quality". ControlNet and LoRA are also applied.
	- Additionally, we concatenate the output feature of amalgamation with a feature extracted using DDPM [2].
- **Semantic Segmentation: ADE20K**
	- Convolutional features extracted at $t=50$:
		- One feature using only fine-grained prompt "a highly realistic photo of the real world. It can be an indoor scene, or an outdoor scene, or a photo of nature. high quality" plus all category labels.
		- Two features using the same prompt as above and additionally ControlNet and LoRA. The two features use two different LoRA weights, where one is moderately trained and the other slightly overfits.
	- No attention features are used. We find that in very complex scenes attention features can be of low quality and bring slight degradation instead of enhancement.
- **Semantic Segmentation: CityScapes**
	- Convolutional features extracted at $t=50$:
		- One feature using only fine-grained prompt "An urban street scene with multiple lanes, various buildings, traffic lights, cars in the lanes, and pedestrians, highly realistic".
		- Two features using the same prompt as above and additionally ControlNet and LoRA. The two features use two different LoRA weights, where one is moderately trained and the other slightly overfits.
	- No attention features are used, based on the same observation as for ADE20K.