Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence
Authors: Grace Luo
Lisa Dunlap
Dong Huk Park
Aleksander Holynski
Trevor Darrell
Venue: NeurIPS 2023 poster (2023)
Keywords: semantic correspondence
hypercolumns
diffusion models
generative model representations

Abstract: Diffusion models have been shown to be capable of generating high-quality images, suggesting that they could contain meaningful internal representations. Unfortunately, the feature maps that encode a diffusion model's internal information are spread not only over layers of the network, but also over diffusion timesteps, making it challenging to extract useful descriptors. We propose Diffusion Hyperfeatures, a framework for consolidating  multi-scale and multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks. These descriptors can be extracted for both synthetic and real images using the generation and inversion processes. We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence: our method achieves superior performance on the SPair-71k real image benchmark. We also demonstrate that our method is flexible and transferable: our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features of synthetic image pairs with unseen objects and compositions. Our code is available at https://diffusion-hyperfeatures.github.io.

Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence
Grace Luo!
Lisa Dunlap'
Dong Huk Park 1*
Aleksander Holynski
1:2
Trevor Darrell
'CC Berkeley
Google Research
Abstract
Diffusson models have been shown to be capable of generating high-quality images, suggesting that they could contain meaningful internal representations. Unfor- tunately, the feature maps that encode a diffusion model's internal information are spread not only over layers the network, but also over diffusion timesteps, making it challenging to extract useful descriptors. We propose Diffusion Hyperfea- tures, framework for consolidating multi- selti-scale and multi-timestte feature maps into per-pixel feature descriptors that can be used for downstream tasks. These descriptors can be extracted for both synthetic and real images using the generation and inversion processes. We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence: our method achieves superior performance on the SPair-71k real image benchmark. We also demonstrate that our method flexible and transferable: our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features of synthetic image pairs with unseen objects and compositions. Our code available at https: /ttp:/::///ffffoo//////oo http:///dfffuuuuooooorreretrrree hyperfeatures github.ie.
Introduction
Much of the recent progress in computer vision has been facilitated by representations learned by deep models. Features from ConvNets [52, 5, 41, 39], Vision Transformers [9, 2], and GANs [45, 26] have demonstrated great utility number of applications, even when compared hand-crafted feature descriptors [24, 7]. Recently, diffusion models have shown impressive results for image generation, suggesting that they too contain rich internal representations that can used for downstream tasks. Existing works that use features from a diffusion model typically select particular subset of layers and timesteps that best model the properties needed for given task (e.g., features for semantic-level correspondence may be most prevalent in middle layers, whereas textural content may be at later layers). This selection not only requires a laborious discovery process hand-crafted for a specific task, but it also leaves behind potentially valuable information distributed across other features in the diffusion process.
In this work, we propose a framework for consolidating all intermediate feature maps from the diffusion process, which vary both over scale and time, into single per-pixel descriptor which we dub Diffusion Hyperfeatures. In practice, this consolidation happens through feature aggregation network that takes as input the collection of intermediate feature maps from the diffusion process and produces output a single descriptor map. This aggregation network is interpretable, as it learns mixing weights identify the most meaningful features for given task (e.g., semantic correspondence). Extracting Diffusion Hyperfeatures for a given image is simple performing the diffusion process for that image (the generation process for synthetic images, and inversion for real images) and feeding all the intermediate features to our aggregator network.
Equal advising contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Diffusion Process
Diffusion Hyperfeatures
LiNet Layers
Aggregation Network
t-0
[-T
Timesteps
Figure I: Unlike prior work that hand-selects a subset of raw diffusion features, we extract all feature maps from the diffusion process, varying across both timesteps and layers, and use lightweight aggregation network to consolidate them into Diffusion Hyperfeatures. For real images, we extract these features from inversion process, and for synthetic images we extract these features from the generation process. Given pair of images, we find semantic correspondences by performing nearest-neighbor search over their Diffusion Hyperfeatures.
We evaluate this approach by training and testing our descriptors on the task of semantic keypoint correspondence, using real images from the SPair-7IK Ik benchmark [28]. We present an analysis the utility of different layers and timesteps of diffusion model features. Finally, we evaluate our trained feature aggregator on synthetic images generated by the diffusion model and show that our Diffusion Hyperfeatures generalize to out-of-domall data.
2 Related Work
The term hypercolunn, originally from neuroscience literature [18]. was first coined for neural network features by Hariharan el al. [15] to refer to the set of activations corresponding pixel across layers of convolutional network, an idea that has also been studied in the context of texture [25], optical flow [46], and stereo [20]. One central idea of this line of work is that for precise localization tasks such as keypoint detection and segmentation [15, 50], it essential to reason at both coarse and fine scales rather than the typical coarse-to- to-fine setting. The usage of hypercolumns has also been popular for the task of semantic correspondence, where approaches must be coarse enough to be robust to illumination and viewpoint changes and fine enough to compute precise matches [27, 30, 44, 29, 1]. Our work revisits the idea of hypercolumns to leverage the features of a recently popular network architecture, diffusion models, which primarily differ from prior work in that the underlying feature extractor is trained on a generative objective and offers feature variation along the axis of time in addition to scale.
Hypercolumn Features.
Deep Features for Semantic Correspondence. There has been recent interest in transferring representations learned by large-scale models for the task of semantic correspondence, Peebles el al. [34] addressed the task of congealing [23] by supervising a warping network with synthetic GAN data produced from a learned style vector representing shared image structure. While prior work [34, 43] has demonstrated that generative models can produce aligned image outputs, which is useful supervision for semantic correspondence, we study how well the raw intermediate repre- sentations themselves would perform for the same task. Utilizing self-supervised representations, in particular from DINO [4], has also been especially popular. Prior work has demonstrated that is possible to extract high-quality descriptors from DINO that are robust and versatile across domains and challenging instances of semantic correspondence [2]. These descriptors have been used for downstream tasks such as semantic segmentation [14], relative camera pose estimation [10], and dense visual alignment [31, 11]. While DINO does indeed contain rich visual representation,

---

Cat sitting Hoing room.
Predicted
Layer 4
Layer 10
Predicted Xy
Layer 4
Layer 10
10 generation steps
50 generation steps
Figure 2: We show an example pair of synthetic images for the prompt "cat sitting in a living room" and the PCA of the features from Layers 4, 10 during both early and late generation step. While different layers capture different image characteristics (here Layer 4 delineates the face vs. body and Layer 10 captures the edges), these features also evolve and become more fine-grained time.
diffusion features are under-explored for the task of semantic correspondence and likely contain enhanced semantic representations due to training on image-text pairs.
Diffusion Model Representations, There have been few works that have analyzed the underlying representations in diffusion models and proposed using them for downstream tasks. Plug-and- Play [43] injects intermediate features from a single layer of the diffusion UNet during second generation process to preserve image structure in text-guided editing. FeatureNeRF [49] distills diffusion features from this same layer into a neural radiance field. DDPMSeg [3] and ODISE [48] aggregate features from hand-selected subset of layers and timesteps for semantic and panoptic segmentation respectively. While these works also consider a subset of features across layers and/or time, work primarily differs in the following ways: first, rather than hand-selecting subset of features, we propose a learned feature aggregator building on top of Xu et al. [48] that weights all features and distills them into concise descriptor map of fixed channel size and resolution. Furthermore, all of these methods solely use the generation process to extract representations, even for real images. In contrast, use the inversion process, where are able to extract higher-quality features at these same timesteps, as seen in Figure 3.
3 Diffusion Hyperfeatures
In diffusion process, one makes multiple calls to a UNet to progressively denoise the image (in the case of generation) or noise the image (in the case of inversion). In either case, the UNet produces number of intermediate feature maps, which can be cached across all steps of the diffusion process amass set of feature maps that vary over timestep and layers of the network. This feature is rather large and unwieldy, varying in resolution and detail across different axes, but contains rich information about texture and semantics. We propose using lightweight aggregation network to learn the relative importance of each feature map for given task (in this paper we choose semantic correspondence) and consolidate them into a single descriptor map (our Diffusion Hyperfeatures). To assess the quality of this descriptor map, using a nearest- neiihstt search on the extracted descriptors, and compare these correspondences to the ground-truth annotations [28, 47]. An overview of our method is shown in Figure 1.
compute semantic correspondences for an image pair, by
Our approach is composed of two core components. Extraction (Section 3.1): We formulate simplified and unified extraction process that accounts for both synthetic and real images, which means we are able to use the same aggregation network on features from both image types. Aggregation (Section 3.2): We propose an interpretable aggregation network that learns mixing weights across the features, which highlights the layers and timesteps that provide the most useful features unique the underlying model and task.

---

Layer
Layer 10
Layer 4
Layer
Generation, t=25 (noise then denoise)
Inversion, t=25
(continuous chain)
Figure 3: We show an example pair of real images from SPair-7Ik and the PCA of the features from Layers 4, 10 when extracted at the middle timestep = 25. While prior work extracts generation features by noising and denoising the image independently approach we extract inversion features from one continuous chain (right). Extracting features from the same timestep of the inversion chain can produce features more true to original image content.
the specific timestep (left), in
3.1 Diffusion Process Extraction
One popular sampling procedure for a trained diffusion model is DDIM sampling [40] of the form
VonTo + v1 - a where 6 ~ 1))
where so is the clean image, is the noise prediction from the diffusion model conditioned on the timestep and noisy image 74+1 from the previous timestep, and xy is the prediction for the timestep. To run generation, one runs the reverse process from t = T to O, with the input set to pure noise sampled from (0, 1). To run inversion, one runs the forward process from & 0 T. with the input set to the clean image.
Generation. When synthesizing an image, we can cache the intermediate feature maps across the generation process, which already contain shared representations that can be used to relate the image to other synthetic images, as seen in the PCA visualization of the feature maps in Figure 2. In this example, we see that the head and body of the two cats share a corresponding latent representation throughout almost the entire generation process, even as early as the first ten steps, where the inputs to the UNet are almost pure noise. This can be explained by the preliminary prediction for the final image To, which already lays out the general idea of the image including the structure, color, and main subject. As generation progresses, the principal components of the features also evolve, with Layer 4 changing from displaying coarse to more refined common semantic sub-parts and Layer 10 changing from displaying no shared characteristics to tigh-frequency image gradients. These observations indicate that the diffusion model provides coarse and fine features that capture different image characteristics (i.e. semantic textural information) throughout different combinations of layers and timesteps. Hence, we find it important to extract features from all layers and timesteps in order adequately tune our final descriptor map to represent the appropriate level of granularity needed for given task.
useful features can be extracted for real images through the inversion process. Although inversion is a process that destructs the real image into noise, we observe that its features contain useful information akin the generation process for synthetic images. In Figure 3, we can see that our inversion features are able to reliably capture the full body of both cats and their common semantic subparts (head, torso, legs) in Layer 4 and their edges in Layer 10 even timestep when the input to the model is relatively noisy. In contrast, using the generation process to analyze real images (as done in prior work) leads hyperparameter tuning and tradeoffs. For example, at timesteps close to t = T where in-distribution inputs are close to noise, the features start to diverge from information present in the real image and may even hallucinate extraneous details, as seen in Figure 3. In this example, because the color of the top cat's stomach is white like the background, the generation features from Layer merge the stomach with the background. Similarly, because there is low contrast between the bottom cat and the background, the generation features from
Inversion. These

---

Layer 10 fail to capture the silhouette of the cat and instead depict random texture edges. Intuitively. inversion features are more trustworthy because the notion of chaining, where at every step the input is some previous output of the model rather than random mixture of an image and noise, and every extracted feature map therefore interrelated. Extracting features from continuous inversion process also induces symmetry with the generation process, which in Section 4.3 we demonstrate allows us to use both feature types interchangeably with the same aggregation network.
3.2 Diffusion Hyperfeatures Aggregation
Given a dense set of feature maps from the diffusion process, we now must efficiently aggregate them into single descriptor map without omitting the information from any layers timesteps. A naive solution would be to simply concatenate all feature maps into one very deep feature map, but this proves to be too high dimensional for most applications. We address this issue with our aggregation network, which standardizes the feature maps with tuned bottleneck layers and sums them according to learned mixing weights. Specifically, for a given feature map r we upsample it to a standard resolution, pass through a bottleneck layer B [16, 48] to a standard channel count, and weight it with a mixing weight w. The final Diffusion Hyperfeatures take on the form
W1,s Bi(ris)
where is the number of layers and S is the number of timesteps. Note that we run the diffusion process for a total number of T timesteps but only select a subsample of S timesteps for aggregation to conserve memory. We opt to share bottleneck layers across timesteps, meaning we use a total of L bottleneck layers. However, we learn L S unique mixing weights for every combination of layer and timestep. We then tune these bottleneck layers and mixing weights using task-specific supervision. For semantic correspondence, the cosine similarity between every possible pair of points. We then supervise with the labeled corresponding keypoints using a symmetric cross entropy loss in the same fashion as CLIP [36]. During training, we downscale the labels according to the resolution of our descriptor maps. When running inference, we upsample the descriptor maps before performing nearest neighbors matching to predict semantic keypoints. We demonstrate that this lightweight parameterization of our aggregation network is performant (see Section 4.1) and interpretable (see Section 4.2) without degrading the open-domain knowledge represented in the diffusion features (see Section 4.3).
flatten the descriptor maps for a pair of images and compute
Experiments
Baselines. We compare against both zero-shot and supervised methods for the task of semantic correspondence. For our zero-shot baselines, we compare against the DINO [2, 4] Layer 9 key features and DINOv2 [32] Layer 11 token features, derived from a self-supervised model trained on ImageNet [8] or LVD-142M [32] respectively. For our supervised baselines, we compare against DHPF [29] and CATS++ [6], which aggregate ResNet- 101 [16] hypercolumn features. The features are derived from a supervised model pretrained on ImageNet and finetuned on SPair-71k [28]. We also compare against two diffusion baselines, selecting single feature map from a known semantic layer [43] (SD-Layer-4) and concatenating feature maps from all layers (SD-Concat-All).
Experimental Details. Our method extracts descriptors from Stable Diffusion v1-5 [37], a generative latent diffusion model trained on LAION-5B [38]. We tune our bottleneck layers and mixing weights on SPair-71k for up to 5000 steps with a batch size of 2 and a learning rate of 1e-3 using an AdamW optimizer [21]. We extract features from the UNet decoder layers (denoted as Layers I to 12), specifically the outputs of the residual block before the self- and cross- attention blocks. Our full method runs the inversion or generation process over the standard T = 50 scheduler timesteps and subsamples every 5 steps, thereby aggregating features over S = 11 total steps. We also ablate using a "one-step" process (T = 1, = 1). When operating on real images, we feed the empty prompt ... We find that extracting features only from the unconditional model removes the need for manual prompting and yields high-quality results. When operating on synthetic images, we extract features only from the conditional model, or the branch conditioned on the prompt that generated the image.
Metries. We report results in terms of PCK @a, or the percentage of correct keypoints at the threshold a. The predicted keypoint is considered to be correct if it lies within radius of a max (h, w) of
5

---

Layers
Timesteps S
SPair-7Ik
CUB
1 PCK
1 PCKGO. 1ator
PCKK0.11m
1 PCK60, 1
DINO [2]
51.68
41.04
72.72
55.90
DINOv2 [32]
68.33
56.98
89.96*
76.83*
DHPP [29]
34
55.28
42.63
77.30
61.42
CATS++ [6]
30
70.26
57.06
75.92
59.49
SD-Layer-4
1
1
58.80
46.58
78.43
61.22
SD-Concat-All
12
1
52.12
41.83
70.22
54.05
Ours
12
11
72.56
64.61
82.29
69.42
Ours-One-Step
12
63.74
54.69
76.59
62.11
SD-Layer-Pruned
1
57.69
48.16
80.67
67.21
Ours-Prume Pruseed
1
64.02
53.74
79.10
63.95
Ours-SDv2-1
12 11
70.74
64.85
80.39
68.04
Table 1:
We evaluate our semantic keypoint matching performance on real images from SPair-71k and CUB. For our CUB evaluation, transfer the model tuned on SPair-71k. We compare against Stable Diffusion baselines that extract features from a single layer (SD-Layer-4) or concatenation of all layers (SD-Concat-Al)) We ablate pruning the single feature map with the highest mixing weight selected by our method, either as the raw feature map (SD-Layer-Prunee) Preeed) or after the bottleneck layer Durr-Prunedd, We ablate tuning our method with only one timestep (One-Step) or features from another Stable Diffusion variant (SDv2-1). *Note that DINOv2 was trained on samples from CUB [32].
the ground truth annotation, where h, w denotes the dimensions of the entire image (0img) object bounding box (Abboz) in the original image resolution. Following Amir el al. [2] we use the threshold 0.1 [2]. We use the default input resolution for each baseline. and we "compute the metrics on the standard setting, i.e. the original image size" [42] to ensure a fair comparison. For DINO, DHPF, and CATS++ we use the resolutions 224, 240, and 512 respectively. For DINOv2 we use resolution 770 to account for its larger patch size and for Stable Diffusion variants we use resolution 224 in line with our DINO baseline.
Additional Experiments. In the Supplemental we provide additional experiments ablating perfor- mance of features across individual decoder layers and model variants, performance of raw inversion vs. generation features, the effect of aggregating DINO and DINOv2 features, and applying Diffusion Hyperfeatures for dense warping.
4.1 Semantic Keypoint Matching on Real Images
We evaluate on the semantic keypoint correspondence benchmark SPair-71k [28), which is composed of image pairs from 18 object categories spanning animals, vehicles, and household objects. Following prior work [2]. evaluate 360 random image pairs from the test split, with 20 pairs per category. To test the method's s ability to transfer to unseen data, we also evaluate on CUB [47], which is variety of bird species. Our validation set is 360 random image pairs taken from the Kulkarni et al. [22] validation split.
composed of semantic keypoints
As seen in Table 1, even when compared with the competitive baselines DINOv2 and CATS++ our method achieves the best result with 72.56 PCK @. and 82.29 PCK @0.1 on SPair-71k and CUB respectively. Selecting single diffusion feature map (SD-Layer-4) already outperforms DINO and DHPF by a margin at least 4% in PCK @0.1 1img likely due larger and more diverse training set. Conversely, naively concatenating all feature maps (SD-Concat- All), degrades this improvement, likely because each map captures a different level of granularity and therefore should not be equally weighted for coarse-level semantic similarity task. When aggregating feature maps across time and space using our Diffusson Hyperfcatures, we see significant boost of 14% in PCK @CK@0.L. Ling over SD-Layer-4, This sizeable performance improvement indicates that (1) while single layer may capture a good deal of semantic information, the other layers also capture complementary information that be useful for disambiguating points finer level and (2) the diffusion model's internal knowledge about the image is not fully captured by a single pass through the UNet but rather spread out multiple timesteps.
underlying model's
In Figure 4, we show qualitative examples of our predicted semantic correspondences compared with the strongest baselines. In general, while both baselines are able to relate broad semantic regions

---

Ground Truth
SD -Lyyer
DINOv2
CATS+
Ours
SPair-71k
SPain-7lk
CUB
Figure 4: Example images from SPair-71k and CUB, the ground-truth user-annotated correspon- dences, and predicted correspondences from each method.
such as the head, arms, or legs, they struggle with fine-grained subparts and confuse them with other visually similar regions. For example, SD-Layer-4 and DINOv2 confuse the headlight (yellow) of the blue car with the rear light of the red car, whereas our method is able to correctly reason about the front vs. back of the cars, In a similar fashion, unlike our method both of these baselines also confuse the back of the seagull's 's neck (orange) with the front of the northern flicker bird's 's neck.
4.2 Ablations
Number of Difffusion Steps. We ablate the importance of the time dimension by tuning variant of our method on feature maps from a one-step inversion process (Ours-One- One-Step), where there is only one possible timestep extract from. While aggregating over all layers performs better than single layer selection (SD-Layer-4) or simple concatenation ADD-Concat-All) by a margin of at least 5 5% in PCK @. it also heavily lags behind our full method. As indicated by our mixing weights in Figure 5, the most useful information for semantic correspondence are concentrated in the early timesteps of the diffusion process, where the input image is relatively clean but contains some noise. Timestep selection can be thought of as a knob for the amount of high frequency detail present in the image to analyze. where at these early timesteps the model is implicitly mapping the noisy input to a smoother image, similar to the oversmoothed cats for the predicted To in Figure 2. Evidently not all texture and detail is necessary for the task of semantic correspondence, and the model is able to produce more useful features at different timesteps which highlight different image frequencies.
Since method employs interpretable mixing weights, have ranking of the relative importance of each layer and timestep combination for the task of semantic correspondence. To verify this ranking, we ablate pruning to the single feature map with the highest mixing weight. As seen in Figure 5, for Ours-SDvI- 1-5 this is the feature map associated with Layer 5, Timestep 10. Referring to Table 1, this automatically selected raw feature map (SD-Layer-Pruned) performs comparably to the feature map manually selected by prior work (SD--Layer-4) Interestingly. if only viewing features from one-step inversion process it would seem that Layer features are significantly worse than Layer 4 features, as discussed further in the Supplemental, but the story changes after tuning selection over both time and space. After passing this pruned feature map to our learned bottleneck layer, the feature map performs comparably on CUB and 6% better in PCK @0. ling on SPair-7lk, -1k,, as seen when comparing SD--LyerrPPrin Pruned and Ours-Pruned in Table 1. This trend validates that our bottleneck layer does not degrade the power of the original representation, but rather refines it in a way that is likely helpful for complex object categories present in SPair- beyond birds. Considering the 9% gap in PCK @ img between our pruned and full method (Ours-Pruned vs. Ours), it becomes evident that it is not a single feature map that drives our strong performance but rather the soft mixing of many feature maps.
Pruning.
7

---

Mixing Weights Tuned on Spair-7ik SDv1-5 SDv2-1
Ground Truth
SDv2-1 Layer
Ours SDv2-1
2013 2035 00-13
Figure 5: The learned mixing weights when aggregating SDv1-5 vs. SDv2-1 features across multiple layers and timesteps. Bright yellow denotes a high weighting, and dark blue denotes low weighting. We also depict predicted correspondences from SDv2-I-Layer-4 vs. Ours-SDv2-1, While Layer features from SDv1-5 perform well in semantic correspondence, this same layer in SDv2-1 performs extremely poorly. Our method automatically learns the best layers depending on the model variant.
Model Variant. We also ablate using different model variant for our diffusion feature extraction, namely SDv2-1. Most notably, SDv2-1 differs from SDv1-5 in its large-scale text encoder, which scales from CLIP [36] to OpenCLIP [19]. The mixing weights learned for both model variants, depicted in Figure 5, showcase the same high-level trends, where the features found to be the most useful for semantic correspondence are concentrated in middle Layers 4-9 and early Timesteps 5-15. However, on a more nuanced level, the behavior starts to diverge with regards to the relative importance of layers and timesteps within this range. Namely, layer selection moves from Layers 4-5 to higher resolution Layers 5-7 from SDv1-5 to v2-1. This behavior is confirmed by our early hyperparameter sweeps of raw feature maps across model variants discussed in the Supplemental, where in fact the Layer 4 feature map of SDv2-1 I performs extremely poorly for the task of semantic correspondence. Timestep selection also moves from Timestep 10 to Timestep 5 from SDv1-5 v2-1, which is surprising because this means that SDv2-1 tends to select higher resolution feature maps from timesteps with higher frequency inputs for task where it essential abstract fine grained details into semantically meaningful matches. These trends seem to imply that more powerful text encoder produces shared semantic representations at increased levels of detail, possibly because the model is better able to connect more distantly related visual concepts via text instead of giving them completely disjoint representations. Hence, our aggregation network is able to dynamically adjust to the representations being aggregated and the task at hand, both of which influence the most important set of features to select from the diffusion process.
4.3 Transfer on Synthetic Images
In addition to evaluating the transfer of our aggregation network to other datasets such as CUB, we also evaluate on synthetic images. Specifically, we take the same aggregation network tuned on inversion features and simply flip the timestep ordering operate generation features. Therefore in this setting, we are testing our network's ability to generalize (1) to a completely unseen feature type from different diffusion process (inversion vs. generation) and (2) out-of-domain object categories that are not present in SPair-71k. Surprisingly, our network generalizes well, outperforming predictions from both DINO and the raw feature map from the last step of the generation process (SD-Layer-4) as seen in Figure 6. Although DINO and SD-LLayer-4 are generally able to correspond broad semantic regions correctly, they sometimes have difficulty with relative placement of subparts. In the case of the rungs of the Eiffel Tower (purple, pink, green), DINO collapses all of its predictions onto the middle rung and SD-Layer-4 collapses them onto the left rung, whereas our method is able to correctly correspond the middle and side rungs of the tower in a triangle formation. The basclines can also be distracted by other objects in the scene that are visually similar. In the case of the mermaid's legs (red, yellow), both baselines incorrectly correspond certain points with the rock in the right image, whose contours and silhouette resemble the legs. On the other hand, our method is able to predict more reliable correspondences for fine-gra tne-grained subparts, even in these challenging cases with unseen textures (lego, snow) and categories (hat, cactus). The ability of our aggregation network to extend to open-domain synthetic images opens up the exciting possibility of generating custom synthetic datasets with pseudo ground-truth semantic correspondences, which we demonstrate are more precise than correspondences derived from the raw feature maps.
8

---

Ground Truth
DINO
SD-Layer-4
A colorful hot air balloon soaring over city skyline and trees with the Eäffel Tower bac buckground.
A mermaid reading book on sunken pirate shrp.
The Statue Liberty made out of Lego.
A snowmmn wearing hat and scarf sitting stop mountain.
robot watering garden flowers and cacti.
Figure 6: Example synthetic images and their text prompts, the ground-truth user-annotated corre- spondences, and predicted correspondences from DINO, SD-Layer-4, and our method. We transfer the aggregation network tuned on inversion features of real images to generation features of synthetic images that are completely out-of-domain compared to the SPair-7Ik categories.
Conclusion
We demonstrate that our Diffusion Hyperfeatures able distill the information distributed across time and space from diffusion process into a single descriptor map. Our intepretable aggregation network also enables automatic analysis of the most useful layers and timesteps based both the underlying model and task of interest. We outperform methods that use supervised hypercolumns or self-supervised descriptors by a large margin on semantic keypoint correspondence benchmark comprised of real images. Although we tune on small set of real images with limited categories, we demonstrate that our method able retain the open- open-domain capabilities of the underlying diffusion features by demonstrating strong performance in predicting semantic correspondences in challenging synthetic images, especially compared using the raw feature maps. Our ability to predict high-quality correspondences derived from the same feature maps used to produce the synthetic image could potentially employed to create synthetic image sets with aseudo-labeles semantic keypoints, which would be valuable for downstream tasks such as image-to-image translation or 3D reconstruction.
Acknowledgements
We thank Angjoo Kanazawa, Yossi Gandelsman, Norman MU, David Chan, and Jitendra Malik for helpful discussions. This work was supported in part by DoD including DARPA's SemaFor, PTG and/or LwLL programs, well BAIR's industrial alliance programs.

---

References
[1] Kíir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best-buddies: Sparse cross-domain correspondence. ACM Trunsactions on Gruphics (TOG), 37(4)11-1 2018.
[2] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. Proceedings of the European Conference Computer Vision Workshops (ECCVW), 2022.
[3] Dmitry Baranchuk, Andrey Voynov. Ivan Rubachev. Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. In International Conference on Learning Represen- rations, 2022.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[5] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE international conference on computer vision, pages 1511-1520, 2017. [6] Seokju Cho, Sunghwan Hong. and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. arXiv preprin! arXiv:2202.068
2022.
[7] Navnect Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR '05), volume 1, pages 886-893. Icee, 2005.
[8] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Iece, 2009.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[10] Walter Goodwin, Sagar Vaze, Joannis Havoutis, and Ingmar Posner.
Zeno-shot category-leve object pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[11] Kamal Gupta, Varun Jampani. Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia. Noah Snavely, and
Abhishek Kar. Asic: Aligning sparse in-the-wild image collections. 2023.
[12] Bumsub Ham, Minsu Cho. Cordelia Schmid, and Jean Ponce. Proposal flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[13] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow; Semantic correspondences
from object proposals. In PAMI, 2017.
[14] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsu- pervised semantic segmentation by distilling feature correspondences. In International Conference on
Learning Reprexentations, 2022.
[15] Bharath Hariharan, Pablo Arbelácz, Ross Girshick, and Jitendra Malik. Hypercolumns for object segmen- tation and fine- grained localization. In Proceedings of the IEEE conference computer vision and pattern recognition, 2015.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[17] Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin C.K. Chan, and Ziwei Liu. Verssn:::: Diffusion-bar based relation inversion from images. arXiv preprint arXiv:2303. 13495, 2023.
[18) D. H. Hubel and T. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology. 160(1), 1962.
[19] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. If you use this software, please cite it as below.
[20] D. G. Jones and J. Malik. Determining three-dimensions
shape from orientation and spatial frequency disparities. In European Conference on Computer Vision, 1992.
[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
1rXiv:1412.660 1412,6980, 2014.
[22] Nilesh Kulkarni, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani. Articulation-awanl canonical
surface mapping. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 452-461, 2020.
[23] Erik G. Learned-Millen Data driven image models through continuous joint alignment. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 28(2):236-2) 250, 2006.
[24] David G. Lowe. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision,
60(2):91-100, nov 2004.
[25] J. Malik and P. Perona. Preattentive texture discrimination with early vision mechanisms. Journal of the
Optical Society of America A, 7(5), 1990.
[26] Xin Mao, Zhaoyu Su, Pin Siang Tan, Jun Kang Chow, and Yu-Hsing Wang. Is discriminator a good feature
extractor? arXiv preprint arXiv: 2wt..99220000 997899 2019.
[27] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Hyperpixel flow: Semantic correspondence with multi-layer neural features. In ICCV, 2019.
10

---

[28] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. arxiv prepreint arXiv: 10543, 2019.
[29]
Juhong Min, Jongmin Lec, Jean Ponce, and Minsu Cho. Learning compose hypercolumns for visual correspondence. In ECCV, 2020.
[30] David Novotny, Diane Larlus, and Andrea Vedaldi. Auchornet: A weakly supervised network leam geometry-senaee sensitive features for semantic matching, In Proceedings of the IEEE conference on computer
vision and pattern recognition, 2017.
[31] Dolev Amar, Michal Goyer, Yoni Kasten, and Tali Dekel. Neural congealing: Aligning images to a joint semantic atlas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[32] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Ya- Huang. Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.
Dinav2: Learning robust visual features without supervision, 2023.
[33] Dong Huk Park, Grace Luo, Clayton Toste, Samanch Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell. Shape-guided diffusion with insidc-outsie attention. arXiv, 2022.
[34] William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei Efros, and Eli Shechtman. Gan-supervised dense visual alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022.
[35] Jordi Pont-Tusst, Federico Perazzi, Sergi Caelles, Pablo Arbelaez, Alexander Sorkine- and Luc Van Gool. The 2017 DAVIS challenge on video object segmentation. CoRR, abs/1704.00677 2017. [36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Arxiv:2103 00020, 2021.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution
image synthesis with latent diffusion models, 2021.
[38] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarzzyk, and Jenia Jitsev. LAION-5b; An open large-scale dataset for training next generation image-text models. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[39] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-
tion. arXiv preprint arXiv: 1409.1556 2014.
[40] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 2826, 2016.
[41]
[42] Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Prohabilistic warp consistency for weakly- supervised semantic correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8708-87 8718, June 2022.
[43] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text- driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[44]
Nikolai Ufer and Bjorn Ommer. Deep semantic feature matching. In Proceedings of the IEEE conference
on computer vision and pattern recognition, 2017.
[45] Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthcsis and semantic manipulation with conditional gans. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[46] J. Weber and J. Malik. Robust computation of optical flow in a multi-scale differential framework. International Journal of Computer Vision. 14(1), 1995.
[47] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200.
Technical Report (NS-TR-2010-00
California Institute of Technology, 2010.
[48] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Odise: Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEFF/CVF segmee
Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[49] Jianglong Ye, Naiyan Wang, and Xiaolong Wang. Featurenerf: Learning generalizable nerfs by distilling pre- rre-traine vision foundation models, arXiv preprint arXiv:2303. 12786, 2023.
[50] Fisher Yu, Dequan Wang. Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018
[51] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong. Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan. Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. 2022.
[52] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechuman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-50 595, 2018
11


Reviews:
[{"review_id": "84zTQRAOOk", "reviewer": "Reviewer_LUzf", "summary": "This paper explores semantic correspondence tasks with stable diffusion model. Specifically, the authors proposed to first extract feature maps varying across timesteps and layers from the diffusion process and trains a lightweight neural network to aggregate them together for semantic correspondence.\nExperimental results on CUB-200 and SPair-71k datasets show that proposed method outperforms other baselines.\n\n", "questions": "please refer to the weaknesses section.", "limitations": "no", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "\n- This paper shows that one could explore diffusion models for semantic correspondence tasks.\n\n- the writing is clear and easy to follow.\n\n- The experiments show that the proposed method could outperform other baselines on the SPair-71k real image benchmark.\n\n- the authors have shown numerous visual examples to demonstrate the correspondence capability.\n\n", "weaknesses": "1. The proposed method, that \"distill the information  distributed across time and space from a diffusion process into a single descriptor map\", seems to have more potential than correspondence tasks. Have the authors explore other tasks, like video label propagation, Homography estimation or perception tasks like classification?\n\n\n2. Concurrent works: there are multiple papers presenting correspondence ability of diffusion models:\n\n[1*] Hedlin, Eric, et al. \"Unsupervised Semantic Correspondence Using Stable Diffusion.\" arXiv preprint arXiv:2305.15581 (2023).\n[2*] Tang, Luming, et al. \"Emergent Correspondence from Image Diffusion.\" arXiv preprint arXiv:2306.03881 (2023).\n[3*] Zhang, Junyi, et al. \"A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence.\" arXiv preprint arXiv:2305.15347 (2023).\n\nCould the authors explain the differences if possible?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *Have the authors explored other tasks [...]?* \n\nPlease see the global response PDF for new applications in semantic appearance transfer and video mask propagation.\n\n2. *Concurrent works: there are multiple papers presenting correspondence ability of diffusion models: [...] Could the authors explain the differences if possible?* \n\nIndeed there are a few concurrent works that have appeared in unofficial preprints after the submission of this project. **One main conceptual difference with these concurrent works is that we aggregate features across all timesteps of the diffusion process**, which we motivate in Figure 2 of the main paper and ablate in our experimental results (Ours vs Ours - One-Step in Table 1 of the main paper, where we achieve a 10\\% boost in PCK\\@0.1_bbox). In contrast, these works all use a hand-selected *single timestep.* In Figure 5 of the main paper we demonstrate why heuristics for hand-selecting specific diffusion features may not be generalizable, since SDv1-5 and SDv2-1 display significantly different behavior in which combinations of layers and timesteps are the most “semantic,” as automatically learned by our aggregation network. **Finally, we also report the best keypoint matching performance on SPair-71k (64.6 PCK\\@0.1_bbox) compared with 45.4 [1], 52.9 [2], 62.9 [3] PCK\\@0.1_bbox respectively.**\n\n[1] Hedlin et. al. \"Unsupervised Semantic Correspondence Using Stable Diffusion.\" arXiv 2023. \\\n[2] Tang et. al. \"Emergent Correspondence from Image Diffusion.\" arXiv 2023. \\\n[3] Zhang et. al. \"A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence.\" arXiv 2023.\n"}, {"review_id": "mW4vA9OlW7", "reviewer": "Reviewer_HmuP", "summary": "This paper proposes an approach for extracting useful features for pre-trained diffusion models for application to dense visual correspondence tasks. How to do this is not clear due to presence of features both through the network, and over diffusion steps. The proposed approach is to learn which features to use using the hypercolumns framework, by passing activations through bottleneck layers then averaging over layers and time. Experiments show that this method outperforms unsupervised DINO features and supervised hypercolumns.", "questions": "- How does it quantitatively compare against more recent supervised approaches?\n- Why share bottleneck layers over time steps? They contain very different representations.\n- Why not use attention features from the diffusion model in a similar manner to DINO?\n- Aggregating features over time steps is difficult, hence the trained layers on top. Could using a distilled diffusion model such as a consistency model give better representations?", "limitations": "As stated in the submission details, limitations are not discussed. There are a number of limitations including having to train additional components (unlike DINO), not comparing with more recent supervised methods, and sharing layers across time steps.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The premise of extracting useful features from diffusion models is an important one; we know that those features are present but it is not clear how to extract useful ones.\n- Using inversion to get features for real images is a good idea and the explanation that they are more reliable than sampling from the posterior as used in prior work makes sense.\n- The exploration into what feature representations are learned at different layers/times is interesting (Figures 2 and 3).\n- The approach is shown to outperform unsupervised DINO features as well as hypercolumns (Table 1) showing that useful features have been extracted from the diffusion model; the baselines including using a single layer from the diffusion model demonstrate the benefit of combining over layers/time; and visual examples shown in Figures 4-6 demonstrate the approach working.\n", "weaknesses": "- Unlike using DINO features, the proposed approach requires training additional components for downstream tasks, making it much less versatile.\n- The baseline models in Table 1 are a poor representation of current approaches. DINO features are not trained specifically for keypoint matching and newer supervised approaches such as CATS++ (Cho et al. TPAMI 2022) and VAT (Hong et al. ECCV 2022) substantially outperform DHPF.\n- The approach shares bottleneck layers across time steps; this seems potentially problematic since features differ substantially across different times.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *How does it quantitatively compare against more recent supervised approaches?*\n \n**Please see the global response for a comparison to CATS++ and DINOv2, where we outperform both methods on SPair-71k by 2\\% and 4\\% PCK\\@0.1_img respectively.**\n\n2. *Why share bottleneck layers over time steps?* \n\nPlease see the global response for an ablation of individual bottleneck layers per timestep, where **we demonstrate that our final method with shared bottleneck layers performs comparably within 1\\% PCK\\@0.1_img on SPair-71k with significant savings in memory consumption.**\n\n3. *Why not use attention features from the diffusion model in a similar manner to DINO?*\n\n**We find that the residual block (resblock hidden) and the best performing attention features (attn value) perform comparably, which we ablate in Table 7.** For an individual layer (SD-Layer-4) resblock hidden outperforms against attn value by 3\\% PCK\\@0.1_img, and when concatenating all layers (SD-Concat-All) resblock hidden underperforms against attn value by 4\\% PCK\\@0.1_img. In practice it is more straightforward to use these residual block features because selecting amongst the key, query, value, or token features requires additional hyperparameter tuning and can significantly vary across models. For example, while Amir et. al. [1] finds that the Layer 9 key features perform the best in DINOv1, in Table 5 of the global response we demonstrate that these same features perform the worst in DINOv2.\n\n**Table 7. Residual Block Features vs. Self-Attention Features  (SPair-71k PCK\\@0.1_img)**\n| Model                | resblock hidden | attn key | attn query | attn value | attn token |\n|:---------------------|:-----------:|:------------:|:------------:|:------------:|:------------:|\n| SD-Layer-4 |  **58.80** | 50.73 | 53.14 | 55.60\t| 49.62 |\n| SD-Concat-All |  52.12 | 44.84 | 48.68 | **55.79** | 55.32 |\n\n4. *Could using a distilled diffusion model such as a consistency model give better representations?* \n\nSince consistency models [2] directly map noise to data, without an iterative sampling process, it cannot provide features that vary across timesteps. While this may yield a faster feature extraction process, it provides fewer features for our aggregation network to select from, which can degrade performance. For example, our method that uses features from a single timestep (Ours - One-Step) performs 9\\% worse in PCK\\@0.1_img than our method that uses features from all timesteps (Ours) in Table 1 of the main paper.\n\n5. *As stated in the submission details, limitations are not discussed.* \n\nWe will be sure to include a more extensive discussion of our method’s limitations in the final manuscript.\n\n[1] Amir et. al. “Deep ViT Features as Dense Visual Descriptors.” ECCV-W 2021.\\\n[2] Song et. al. “Consistency Models.” ICML 2023."}, {"review_id": "xtFMGtZRvR", "reviewer": "Reviewer_sN43", "summary": "This paper proposes improving feature distillation from diffusion models for representation learning by aggregating information from the feature maps of the U-Net at varying timesteps, weighting them with a tunable aggregation network. The authors show that even at timesteps from which features are usually discarded for feature learning (i.e. early steps in the generation process) useful features can be found. After formulating their approach to extracting these features using a weighted aggregation of a set of standardized input feature maps, authors show validity of their approach by improving considerably over baseline methods and prior work that only selects a single diffusion timestep for feature extraction. Lastly, authors show the ability of their framework to generalize to unseen synthetic data (generated by the diffusion model), by extracting features from the generation process for a diffusion model trained on a different data distribution.\n", "questions": "- For figures 2, 3, would it be possible to show also the network input at these timesteps for reference? \n- Line 117, you mention that “these observations indicate that the diffusion model provides coarse and fine features that capture different image characteristics, throughout different combinations of layers and timesteps”. I’m wondering to what extent this is owing to the formulation of the diffusion process itself, and to what extent this is due to the use of a very specific multi-resolution processing architecture (U-Net) in the diffusion process. I.e. is the extraction of both coarse and fine image features a result of diffusion or a result of using a U-Net? Would your method also work with diffusion models that make use of other architectures?\n- Line 135, I have a bit of trouble understanding your reasoning for the more “trustworthy” inversion features. Could you elaborate? Why would repeated application of the model necessarily lead to more “trustworthy” features?\n- You apply your feature extraction specifically for semantic correspondence detection. Could the extracted features also be used in other settings? Are there any limiting factors that prevent your framework from being used in such settings? Have you tried e.g. classification of extracted features?\n", "limitations": "The authors do not discuss the limitations or societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The idea of extracting features from powerful generative models such as diffusion models is very relevant, and the authors show that their approach to this extraction process obtains great results.\n- The paper is very well-written and reads very smoothly. Authors are very descriptive in their wording and use figures to clearly illustrate their approach. The idea is simple and seems very effective, as authors illustrate with a number of experiments and ablation over baselines.\n- Argumentation for design choices is very solid, and authors motivate their approach with an experiment showing the validity/salience of features found in earlier timesteps of the diffusion model.\n- The authors also show impressive transfer performance to an unseen dataset, highlighting possible applications in pseudo-label generation for semantic correspondence.\n\n", "weaknesses": "- I have no major concerns. A minor concern I have is with regards to novelty, given that extracting hypercolumn features has been attempted before, even for diffusion models. \n- Another minor concern is the requirement for task-specific fine-tuning of the aggregation network. The fact that this network requires explicit supervision for a task means that the actual representation extracted from the diffusion model that can be used in arbitrary downstream tasks is the whole set of features maps across timesteps. Would it be possible to train an aggregation network on a self-supervised task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *For figures 2, 3, would it be possible to show also the network input at these timesteps for reference?* \n\nIn Figure 2 of the main paper, the input to the diffusion model is the text prompt “Cat sitting in a living room” and random noise for $x_{T}$. In Figure 3 of the main paper, the input to the diffusion model is the empty text prompt “” and a noisy version of the real image for $x_{25}$. We agree that the inputs can be unclear and will revise these figures in the final manuscript.\n\n2. *Line 117 [...] is the extraction of both coarse and fine image features a result of diffusion or a result of using a U-Net?* \n\nWe agree that the multi-scale nature of our features across layers is in large part due to the U-Net architecture used in the underlying diffusion model. However, prior work such as Amir et. al. [1] also finds that ViT architectures can also contain this type of coarse vs. fine information, where “shallow features mostly contain positional information” and “deeper layers [...] favor [...] more semantic features.” We note that the evolution of these features over time, owing to the nature of the diffusion process itself, is also a critical component of our method. In Table 1 of the main paper our final method that uses all layers across all timesteps significantly outperforms the variant of our method that uses all layers at a single timestep (Ours - One-Step) by 9\\% in PCK\\@0.1_img. Intuitively, timesteps when the input is noisier in the diffusion process produce features that capture more low frequency statistics that provide orthogonal information also useful for the semantic correspondence task, compared to when the input is clean as seen in Figure 2 of the main paper.\n\n3. *Line 135 [...] more “trustworthy” inversion features”* \n\n**In the PDF document attached to the global response Figure 1 quantitatively compares the performance of inversion vs. generation features at each timestep, where inversion features generally perform better across the board with as much as a 1-5\\% increase in PCK\\@0.1_img in noisier timesteps $t=25$ to $t=50$.** Because the DDIM inversion process should be able to deterministically recover the real image and the model predicts the noise added to the image at each timestep, we hypothesize that the predicted noise carefully destructs information at a specific band of frequencies appropriate for the timestep, compared with using random noise. Beyond DDIM inversion, one interesting further direction of research is exploring the quality of features derived from other inversion processes proposed in the community.\n\n4. *Could the extracted features also be used in other settings?* \n\nYes, it certainly should be able to be used for other tasks such as classification, for example if one were to add a classification head to process the features and train it with the appropriate loss. We have also included in the global response PDF some new applications in semantic appearance transfer and video mask propagation.\n\n[1] Amir et. al. “Deep ViT Features as Dense Visual Descriptors.” ECCV-W 2021."}, {"review_id": "KOMXZ0WP3K", "reviewer": "Reviewer_c4Xr", "summary": "This paper proposes diffusion hyperfeatures, a framework for integrating different scale and timestep features to form a representative feature descriptors in dense level. Within the U-Net architecture, unlike other works that uses hand-crafted methods to select a particular subset of layers for further processing, this paper proposes a simple aggregation network that aggregates the intermediate features. The effectiveness of this method is evaluated on standard benchmark of semantic correspondence, including SPair-71k. ", "questions": "See weaknesses above ", "limitations": "Limitations or failure cases are not presented. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. SOTA performance is achieved with large gap to existing works.\n\n2. One of the first attempt to tackle semantic correspondence task with diffusion. \n\n3. Although paper is easy-to-read, each paragraphs are too long. Also it is quite hard to understand some sentences. This is a minor issue though. ", "weaknesses": "1. This paper is one of the first attempt to tackle semantic correspondence with diffusion concept. In this manner, this paper looks novel, since this is very new to this task. However, to me, this work severely lacks contributions. What this paper does is simply using a series of features from stable diffusion, a large-scale model, feeds to a simple aggregation network that simply performs weighted sum to obtain the final features and it is exploited for Winner-Takes-All (WTA) for finding correspondences. \n (1) An attempt to select features are not novel as this is already smartly done in HPF [23] and DHPF [25], while this work simply uses all the features to feed to an aggregation network.\n (2) Using different feature representation  that is much powerful than standard backbone (resnet 101) will guarantee apparent performance boosts. This is already demonstrated in CATs++: Boosting Cost Aggregation With Convolutions and Transformers (TPAMI). \n(3) No novel diffusion techniques are proposed as far as i understood. \n\n\n\n3. Lacking implementation details. What resolution was used for evaluation and training? This is very important in this task as the performance highly correlates to it. Unless this is evaluated in the same resolution and compared with other works, the comparisons are not fair at all. \n\n4. Why did the authors use only 2 datasets for evaluation? Traditionally, existing methods would also evaluate their methods on Pf-PASCAL, PF-WILLOW, TSS as well. This needs to be included, since evaluation on 2 datasets seem very insufficient. \n\n5.  In supplementary material, section 6.1 explains computational resources. This is very abstract and not very helpful. Comparison to other works is not a must, but at least provide some measures for completeness.\n\n6. DINOv1 is a quite old-model, since DINO-v2 is already out. DINO-v2 is a much powerful model that will yield much stronger performance. \n\n7. Many other semantic correspondence works are not cited and compared. As far as I know, the current SOTA is Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence, arxiv'22, which would be better to be included in the paper. Also when it is compared with DINO, only 1 layer is used. In semantic correspondence, multi-scale and multi-level features are highly important which would have detrimental effects on performance without them. So the comparison may not be fair here as well. However, as the authors also provided results for single layer results, this is not an issue, but a suggestion to include results that use more layers of DINOv1 or DINOv2.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *This paper is one of the first attempt to tackle semantic correspondence with diffusion concept.* \n\nOur method is novel. We are not claiming to make a contribution for a feature extraction or matching algorithm hand-crafted for the task of semantic correspondence; rather, we propose a simple and general framework for utilizing features across the diffusion process. **Additionally, our final method achieves state-of-the-art performance on the SPair-71k benchmark, even when compared to DINOv2 and CATS++ (Table 1 of the global response).** According to the NeurIPS reviewer guidelines on the topic of originality, it states “Is the work a novel combination of well-known techniques? (This can be valuable!)”. We present a simple and effective framework that achieves superior performance compared with prior work, similar in spirit to other published works of this nature such as ODISE [1].\n\n2. *What resolution was used for evaluation and training?* \n\nWe acknowledge that input resolution does have a large impact on performance as shown in Cho et. al. [2], and we use the same evaluation protocol as Truong et. al. [4]. Specifically, we run all the baselines from their respective codebases using their default hyperparameter settings, which we display in Table 6 above, and “compute the metrics on the standard setting, i.e. the original image size, and re-compute the PCK in this setting” [4]. **It is important to note we evaluate our method with downsampled images of resolution 224, following Amir et. al. [3], which is the *lowest input resolution* out of all the methods and is a fair comparison of all methods in the main paper.** In our additional experiments, to account for DINOv2’s large patch size, we use input images of resolution 770 so that it produces feature maps at resolution 55x55, the same as DINOv1. In a similar vein we use input images of resolution 512 for CATS++ to compare against the best possible variant of the method. During training time we use input images of resolution 64x64, the resolution of Stable Diffusion’s latent space. \n\n**Table 6. Method Statistics**\n| Model                | Input Image Resolution |\n|:---------------------|:--------------:|\n| DINO            |       224       |\n| **Ours**             |       **224**       |\n| DHPF              |       240       |\n| CATS++            |       512       |\n| DINOv2           |       770       |\n\n3. *Why did the authors use only 2 datasets for evaluation?* \n\n**Please see the global response where we run additional experiments on PF-PASCAL and PF-WILLOW, where we outperform other methods like DINOv2 by 2\\% and 3\\% respectively.** \n\n4. *In supplementary material, section 6.1 explains computational resources.* \n\nWe agree that this section could be more informative, are there any metrics in particular you would like?\n\n5. *DINOv1 is a quite old-model, since DINO-v2 is already out.* \n\nWe did not initially compare to DINOv2 because it was released shortly before the submission deadline and has yet to be published. Nevertheless, we have included the results for DINOv2 and we will include it in the final manuscript. **Please see the global response where we compare against DINOv2 and outperform the method by 4\\% PCK\\@0.1_img on SPair-71k.**\n\n6. *​​Many other semantic correspondence works are not cited and compared.* \n\n**Please see the global response where we compare against CATS++ and outperform the method by 2\\% PCK\\@0.1_img on SPair-71k.** Unfortunately, IFCAT [5] does not have code available for us to re-run their method in our standardized evaluation setting, and for us to evaluate their transfer performance from SPair-71k to other datasets, but **comparing our results with their reported results we still outperform IFCAT [5] by 0.2\\% PCK\\@0.1_bbox (64.61 vs 64.40 respectively).** Our DINO baseline uses the method proposed by Amir et. al. [3], which uses a single layer, so we provide results using a single layer of Stable Diffusion (SD-Layer-4) for comparability.\n\n[1] Xu et. al. “Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models.” CVPR 2023. \\\n[2] Cho et. al. “CATs++: Boosting Cost Aggregation with Convolutions and Transformers.” TPAMI 2022. \\\n[3] Amir et. al. “Deep ViT Features as Dense Visual Descriptors.” ECCV-W 2021. \\\n[4] Truong et. al. “Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences.” CVPR 2022. \\\n[5] Hong et. al. “Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence.” arXiv 2022."}, {"review_id": "mtgL2TOhaW", "reviewer": "Reviewer_coKq", "summary": "The authors propose a method to extract per-pixel feature descriptors from multi-scale and multi-timestep feature maps generated by diffusion models. These descriptors can be utilized for various downstream tasks.\n\nThe framework is evaluated on the task of semantic keypoint correspondence, specifically on the SPair-71k real image benchmark. The authors claim that their method achieves superior performance on this benchmark compared to other approaches. Additionally, they demonstrate that their method is flexible and transferable, as the feature aggregation network trained on real image pairs can also be used on synthetic image pairs with unseen objects and compositions.\n\nOverall, the proposed Diffusion Hyperfeatures framework aims to enhance the internal representations of diffusion models and improve their utility in computer vision tasks like semantic keypoint correspondence.", "questions": "Clarification on Consolidation Process: Could the authors provide more details regarding the feature aggregation network used to consolidate the intermediate feature maps? How does it handle variations in scale and time? Are there any specific design choices or architectural considerations that impact its performance?\n\nExperimental Setup: It would be helpful to understand the specific experimental setup employed in the evaluation of Diffusion Hyperfeatures. Could the authors elaborate on the dataset used, the selection of keypoint correspondence task, and the metrics used for evaluation? Additionally, is there any consideration given to computational efficiency or runtime performance when applying the framework to real-time applications?\n\nComparison with Existing Methods: The authors briefly mention that existing works select specific subsets of layers and timesteps from diffusion models for different tasks. Could the authors provide a more detailed comparison or discussion on how their approach differs and potentially outperforms these existing methods? Are there any specific limitations or challenges associated with the subset selection approach that Diffusion Hyperfeatures address?\n\nTransferability and Generalization: While the paper mentions generalization to synthetic images, it would be valuable to explore the transferability and generalization capabilities of Diffusion Hyperfeatures across other domains or datasets as well. Can the authors provide insights or experiments on how the framework performs on different benchmark datasets or real-world scenarios beyond the SPair-71k dataset?\n", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper discusses a framework called \"Diffusion Hyperfeatures\" for consolidating feature maps in diffusion models. Here's a substantive assessment of the strengths of the paper across various dimensions:\n\nOriginality: The paper introduces a novel approach to consolidate and extract per-pixel descriptors from intermediate feature maps generated by diffusion models. While existing works focus on selecting specific layers and timesteps, this paper proposes a comprehensive framework that considers all intermediate features. This consolidation process through a feature aggregation network is a new concept, which contributes to the originality of the paper.\n\nQuality: The paper appears to be of high quality as it references relevant literature and state-of-the-art techniques in computer vision, such as ConvNets, Vision Transformers, and GANs. The authors also evaluate their proposed framework on the task of semantic keypoint correspondence using real images from the SPair-71k benchmark. The evaluation includes an analysis of different layers and timesteps of diffusion model features, demonstrating a thorough investigation. Additionally, the generalization of Diffusion Hyperfeatures to out-of-domain data is examined by evaluating them on synthetic images generated by the diffusion model.\n\nClarity: The paper presents its ideas and contributions in a clear manner. It provides a concise overview of the challenges faced with existing methods and describes the proposed Diffusion Hyperfeatures framework step-by-step. The use of terminology and technical language is appropriate for the target audience of researchers in computer vision. However, without the full paper, it is difficult to assess the clarity of the detailed methodology and experimental setup.\n\nSignificance: The paper addresses an important problem in computer vision, namely how to effectively utilize the internal representations of diffusion models for downstream tasks. By proposing the Diffusion Hyperfeatures framework, the authors aim to improve the utility of diffusion models beyond image generation. The evaluation results on semantic keypoint correspondence and the demonstration of generalization to synthetic images indicate the potential significance of this approach. If the proposed framework proves to be effective, it could contribute to advancements in various computer vision applications.\n\nOverall, based on the provided text, the paper demonstrates strengths in terms of originality, quality, clarity, and potential significance. ", "weaknesses": "Based on the provided paper, here are a few potential areas where the paper could benefit from improvement:\n\nComparison with Existing Methods: While the paper mentions that features from ConvNets, Vision Transformers, and GANs have demonstrated significant capabilities, it would be beneficial to provide a more detailed comparison with existing methods that extract feature descriptors from diffusion models. This would help establish the novelty and superiority of the proposed Diffusion Hyperfeatures framework.\n\nExperimental Evaluation: The paper briefly mentions evaluating the proposed framework on the task of semantic keypoint correspondence using real images from the SPair-71k benchmark. To strengthen the paper, it would be valuable to include a thorough analysis of the experimental results, including quantitative metrics, comparative evaluations with state-of-the-art methods, and possibly visual illustrations or qualitative assessments of the generated keypoint correspondences.\n\nGeneralization to Diverse Domains: The paper mentions evaluating Diffusion Hyperfeatures on synthetic images generated by the diffusion model. While this demonstrates some level of generalization, it would be useful to explore the performance and robustness of the framework across a broader range of datasets and domains. This could involve testing the framework on different benchmark datasets and real-world scenarios to validate its effectiveness in diverse settings.\n\nClear Methodology Description: The provided text offers a high-level overview of the proposed framework, but for a comprehensive evaluation, it is crucial to have a clear and detailed description of the methodologies employed. It would be helpful to include information about the architecture of the feature aggregation network, the specific techniques used for consolidation, and any additional preprocessing steps or modifications made to the diffusion model.\n\nPotential Limitations and Future Directions: The paper could benefit from discussing potential limitations or challenges associated with the proposed framework. Identifying these limitations and suggesting future directions for improvement would enhance the depth and completeness of the research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *Clarification on Consolidation Process.* \n\nWe would be more than happy to provide additional details regarding the aggregation network, what specific points were unclear? We discuss details regarding the feature aggregation network as well as how we handle variations in scale and time of the diffusion features in L140-146 of the main paper. We ablate specific design choices of our method such as the number of diffusion steps or the underlying model variant in Section 4.2 of the main paper.\n\n2. *Experimental Setup.* \n\nWe train our aggregation network on the SPair-71k dataset, and we compare our method against the baselines on both SPair-71k and CUB in Table 1 of the main paper. **We also compare against additional methods such as CATS++ and DINOv2 in Table 1 of the global response, where we outperform both methods by 2\\% and 4\\% PCK\\@0.1_img respectively.** We report using the percentage of correct keypoints (PCK), discussed further in L169 of the main paper. We depict visualizations of our predicted correspondences in Figures 4, 5, 6 of the main paper.\n\n3. *Comparison with Existing Methods.* \n\nWe compare against SD-Layer-4, a baseline that selects a specific subset of layers and timesteps. We provide a detailed discussion comparing against this baseline in Section 4.1 of the main paper, including a discussion of how subset selection can result in subpart performance on the keypoint matching task.\n\n4. *Transferability and Generalization.* \n\nWe provide additional results on real-world datasets such as CUB in Table 1 of the main paper and **PF-WILLOW, PF-PASCAL in Table 3, 4 of the global response where we outperform other methods like DINOv2 by 2\\% and 3\\% respectively.**"}]

Author rebuttal:
### Summary
We thank the reviewers for their helpful feedback and suggestions, which we will integrate into the final manuscript. In this work we present a “simple and [...] very effective” (Reviewer sN43) framework for consolidating the internal representations of a diffusion model for tasks such as semantic correspondence.

To re-state our motivation: prior work shows that features from a hand-selected layer & timestep can be useful for downstream applications. Our work demonstrates that the most useful features are not contained within a single layer and timestep, but rather are distributed across _all layers and timesteps_ of the diffusion sampling process (different choices of time and layer often contain complementary information, as seen in Figure 2 of the main paper). Unfortunately, naive concatenation across all features results in excessively high dimensional descriptors that equally weigh all source features, making distance metrics less practical and useful. Our final proposed approach solves this by aggregating these high-dimensional features into a more useful low-dimensional descriptor map that is trainable for a given target task. **We have performed additional evaluations on PF-WILLOW and PF-PASCAL to demonstrate that our method is transferable. We provide them below, along with new applications in semantic appearance transfer and video mask propagation (see PDF).**

The reviewers agree that “the premise of extracting useful features from diffusion models is an important one” (Reviewer HmuP) and that it is “very relevant” (Reviewer sN43). **Moreover, we clearly demonstrate that we achieve “SOTA performance [...] with large gap to existing works” (Reviewer HmuP), which we further validate with new comparisons to CATS++ and DINOv2 in Table 1 below.**

**Table 1. SPair-71K**
| Model                | PCK\@0.1_img | PCK\@0.1_bbox |
|:---------------------|:-----------:|:------------:|
| DINOv2 [1]            |    68.33    |     56.98    |
| CATS++ [2]           |    70.26    |     57.06    |
| Ours - Indiv. Bottleneck per Timestep       |    73.07    |     65.09    |
| Ours                | **72.56**   | **64.61**   |

**Table 2. CUB**
| Model                | PCK\@0.1_img | PCK\@0.1_bbox |
|:---------------------|:-----------:|:------------:|
| DINOv2 [1]            |    89.96*    |     76.83*   |
| CATS++ [2]           |    75.92    |     59.49    |
| Ours                | **82.29**   | **69.42**  |
|*Please note that DINOv2’s training set included CUB|

### Comparison to CATS++ and DINOv2
**In Table 1, we display results for both CATS++ and DINOv2 on SPair-71k. While these methods are competitive, our method still achieves the best result at 72.56 in PCK\@0.1_img, with a 2\% increase over CATS++ and 4\% increase over DINOv2.** For CUB, PF-PASCAL, and PF-WILLOW (Table 2, 3, 4) we outperform CATS++ by 6\%, 19\%, and 11\% in PCK\@0.1_img respectively. Similarly, DINOv2 performs worse than our method across the board, except for CUB where it exhibits unusually high performance *because it was trained on samples from CUB* (Table 15 of the Appendix in [1]). In Table 5 we display the hyperparameter sweep we conducted for DINOv2 ViT-S/14, leading us to use the Layer 11 token features for our DINOv2 baseline.

### Individual Bottleneck Layer per Timestep Ablation
Our choice to share bottleneck layers is an effort to reduce model size at the cost of a slight performance decrease. **As seen in Table 1, both the method with individual bottleneck layers per timestep and our final method perform similarly, with less than a 1\% degradation in PCK\@0.1_img.** Training a bottleneck layer for each timestep would require 132 projection layers (812.85 MB) compared to just 12 projection layers (26.12 MB) when the bottleneck layers are shared.

**Table 3. PF-PASCAL [3]**
| Model         | PCK\@0.1_img | PCK\@0.1_bbox |
|:---------------------|:-----------:|:------------:|
| DINOv2 [1]    |       84.30 |       78.99 |
| CATS++ [2]    |       68.02 |       62.96 |
| Ours      |    **86.67** |     **82.85** |

**Table 4. PF-WILLOW [4]**
| Model         | PCK\@0.1_img | PCK\@0.1_bbox |
|:---------------------|:-----------:|:------------:|
| DINOv2 [1]    |       86.64 |       71.34 |
| CATS++ [2]    |       78.87 |       66.09 |
| Ours          |       **89.61** |       **77.98** |

### Additional Evaluation Datasets
We focused on SPair-71k and CUB because they presented more complex and varied examples than the other benchmarks, which are largely composed of simple image pairs with “similar viewpoints and scales” [5]. That being said, we are happy to include results on PF-PASCAL and PF-WILLOW (Table 2, 3 above). Note that across all these datasets we transfer the model purely trained on SPair-71k for evaluation. **When transferred to PF-PASCAL and PF-WILLOW, our method outperforms DINOv2 by 2\% and 3\% PCK\@0.1_img respectively.**


**Table 5. DINOv2 Hyperparameter Selection (SPair-71k PCK\@0.1_img)**
| Model                | key | query | value | token |
|:---------------------|:-----------:|:------------:|:------------:|:------------:|
| dinov2_vits14 - Layer 9 |  18.74 | 19.10 | 52.47	| 55.08 |
| dinov2_vits14 - Layer 11 |  36.97 | 36.46 | 64.25 | **68.33** |

**References**\
[1] Oquab et. al. “DINOv2: [...].” arXiv 2023.\
[2] Cho et. al. “CATs++: [...].” TPAMI 2022.\
[3] Ham et. al. “Proposal flow.” CVPR 2016.\
[4] Ham et. al. “Proposal flow: Semantic correspondences from object proposals.” PAMI 2017.\
[5] Min et. al. “SPair-71k: [...].” arXiv 2019.

Meta-review: The authors' extensive rebuttal has addressed the critical points raised by the reviewers. After considering the rebuttal, discussions, and also taking the authors' additional comments into account, all reviewers unanimously agreed to accept the paper -> accept. Congratulations! 
Please take the reviewer comments into account when preparing your camera-ready version.
Decision: Accept (poster)