Title: Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence
Venue: NeurIPS 2023
Accepted: yes
Keywords: semantic correspondence

hypercolumns

diffusion models

generative model representations
Abstract: Diffusion models have been shown to be capable of generating high-quality images, suggesting that they could contain meaningful internal representations. Unfortunately, the feature maps that encode a diffusion model's internal information are spread not only over layers of the network, but also over diffusion timesteps, making it challenging to extract useful descriptors. We propose Diffusion Hyperfeatures, a framework for consolidating  multi-scale and multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks. These descriptors can be extracted for both synthetic and real images using the generation and inversion processes. We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence: our method achieves superior performance on the SPair-71k real image benchmark. We also demonstrate that our method is flexible and transferable: our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features of synthetic image pairs with unseen objects and compositions. Our code is available at https://diffusion-hyperfeatures.github.io.

Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence

Abstract

Diffusion models have been shown to be capable of generating high-quality images, suggesting that they could contain meaningful internal representations. Unfortunately, the feature maps that encode a diffusion model's internal information are spread not only over layers of the network, but also over diffusion timesteps, making it challenging to extract useful descriptors. We propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks. These descriptors can be extracted for both synthetic and real images using the generation and inversion processes. We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence: our method achieves superior performance on the SPair-71k real image benchmark. We also demonstrate that our method is flexible and transferable: our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features of synthetic image pairs with unseen objects and compositions. Our code is available at https://diffusion-hyperfeatures.github.io .

1 Introduction

Much of the recent progress in computer vision has been facilitated by representations learned by deep models. Features from ConvNets [52, 5, 41, 39], Vision Transformers [9, 2], and GANs [45, 26] have demonstrated great utility in a number of applications, even when compared to hand-crafted feature descriptors [24, 7]. Recently, diffusion models have shown impressive results for image generation, suggesting that they too contain rich internal representations that can be used for downstream tasks. Existing works that use features from a diffusion model typically select a particular subset of layers and timesteps that best model the properties needed for a given task (e.g., features for semantic-level correspondence may be most prevalent in middle layers, whereas textural content may be at later layers). This selection not only requires a laborious discovery process hand-crafted for a specific task, but it also leaves behind potentially valuable information distributed across other features in the diffusion process.

In this work, we propose a framework for consolidating all intermediate feature maps from the diffusion process, which vary both over scale and time, into a single per-pixel descriptor which we dub Diffusion Hyperfeatures . In practice, this consolidation happens through a feature aggregation network that takes as input the collection of intermediate feature maps from the diffusion process and produces as output a single descriptor map. This aggregation network is interpretable, as it learns mixing weights to identify the most meaningful features for a given task (e.g., semantic correspondence). Extracting Diffusion Hyperfeatures for a given image is as simple as performing the diffusion process for that image (the generation process for synthetic images, and inversion for real images) and feeding all the intermediate features to our aggregator network.

1

∗

Aggregation

Network

0=

t=T

UNet Layers

Timesteps

We evaluate this approach by training and testing our descriptors on the task of semantic keypoint correspondence, using real images from the SPair-71k benchmark [28]. We present an analysis of the utility of different layers and timesteps of diffusion model features. Finally, we evaluate our trained feature aggregator on synthetic images generated by the diffusion model and show that our Diffusion Hyperfeatures generalize to out-of-domain data.

2 Related Work

Hypercolumn Features. The term hypercolumn , originally from neuroscience literature [18], was first coined for neural network features by Hariharan et al. [15] to refer to the set of activations corresponding to a pixel across layers of a convolutional network, an idea that has also been studied in the context of texture [25], optical flow [46], and stereo [20]. One central idea of this line of work is that for precise localization tasks such as keypoint detection and segmentation [15, 50], it is essential to reason at both coarse and fine scales rather than the typical coarse-to-fine setting. The usage of hypercolumns has also been popular for the task of semantic correspondence, where approaches must be coarse enough to be robust to illumination and viewpoint changes and fine enough to compute precise matches [27, 30, 44, 29, 1]. Our work revisits the idea of hypercolumns to leverage the features of a recently popular network architecture, diffusion models, which primarily differ from prior work in that the underlying feature extractor is trained on a generative objective and offers feature variation along the axis of time in addition to scale.

Deep Features for Semantic Correspondence. There has been a recent interest in transferring representations learned by large-scale models for the task of semantic correspondence. Peebles et al. [34] addressed the task of congealing [23] by supervising a warping network with synthetic GAN data produced from a learned style vector representing shared image structure. While prior work [34, 43] has demonstrated that generative models can produce aligned image outputs, which is useful supervision for semantic correspondence, we study how well the raw intermediate representations themselves would perform for the same task. Utilizing self-supervised representations, in particular from DINO [4], has also been especially popular. Prior work has demonstrated that it is possible to extract high-quality descriptors from DINO that are robust and versatile across domains and challenging instances of semantic correspondence [2]. These descriptors have been used for downstream tasks such as semantic segmentation [14], relative camera pose estimation [10], and dense visual alignment [31, 11]. While DINO does indeed contain a rich visual representation,

Catsittinginalivingroom.

Predicted xo

Layer 4

Layer 10

Predicted xo

Layer 4

Layer 10

10 generation steps

50 generation steps

diffusion features are under-explored for the task of semantic correspondence and likely contain enhanced semantic representations due to training on image-text pairs.

Diffusion Model Representations. There have been a few works that have analyzed the underlying representations in diffusion models and proposed using them for downstream tasks. Plug-andPlay [43] injects intermediate features from a single layer of the diffusion UNet during a second generation process to preserve image structure in text-guided editing. FeatureNeRF [49] distills diffusion features from this same layer into a neural radiance field. DDPMSeg [3] and ODISE [48] aggregate features from a hand-selected subset of layers and timesteps for semantic and panoptic segmentation respectively. While these works also consider a subset of features across layers and/or time, our work primarily differs in the following ways: first, rather than hand-selecting a subset of features, we propose a learned feature aggregator building on top of Xu et al. [48] that weights all features and distills them into a concise descriptor map of a fixed channel size and resolution. Furthermore, all of these methods solely use the generation process to extract representations, even for real images. In contrast, we use the inversion process, where we are able to extract higher-quality features at these same timesteps, as seen in Figure 3.

3 Diffusion Hyperfeatures

In a diffusion process, one makes multiple calls to a UNet to progressively denoise the image (in the case of generation) or noise the image (in the case of inversion). In either case, the UNet produces a number of intermediate feature maps, which can be cached across all steps of the diffusion process to amass a set of feature maps that vary over timestep and layers of the network. This feature set is rather large and unwieldy, varying in resolution and detail across different axes, but contains rich information about texture and semantics. We propose using a lightweight aggregation network to learn the relative importance of each feature map for a given task (in this paper we choose semantic correspondence) and consolidate them into a single descriptor map (our Diffusion Hyperfeatures). To assess the quality of this descriptor map, we compute semantic correspondences for an image pair, by using a nearest-neighbors search on the extracted descriptors, and compare these correspondences to the ground-truth annotations [28, 47]. An overview of our method is shown in Figure 1.

Our approach is composed of two core components. Extraction (Section 3.1): We formulate a simplified and unified extraction process that accounts for both synthetic and real images, which means we are able to use the same aggregation network on features from both image types. Aggregation (Section 3.2): We propose an interpretable aggregation network that learns mixing weights across the features, which highlights the layers and timesteps that provide the most useful features unique to the underlying model and task.

Layer 4

Layer 10

Layer 4

Layer 10

Generation,t=25

Inversion, t=25

(noise then denoise)

(continuous chain)

3.1 Diffusion Process Extraction

One popular sampling procedure for a trained diffusion model is DDIM sampling [40] of the form

where x 0 is the clean image, /epsilon1 t is the noise prediction from the diffusion model conditioned on the timestep t and noisy image x t +1 from the previous timestep, and x t is the prediction for the next timestep. To run generation, one runs the reverse process from t = T to 0 , with the input x T set to pure noise sampled from N (0 , 1) . To run inversion, one runs the forward process from t = 0 to T , with the input x 0 set to the clean image.

Generation. When synthesizing an image, we can cache the intermediate feature maps across the generation process, which already contain shared representations that can be used to relate the image to other synthetic images, as seen in the PCA visualization of the feature maps in Figure 2. In this example, we see that the head and body of the two cats share a corresponding latent representation throughout almost the entire generation process, even as early as the first ten steps, where the inputs to the UNet are almost pure noise. This can be explained by the preliminary prediction for the final image x 0 , which already lays out the general idea of the image including the structure, color, and main subject. As generation progresses, the principal components of the features also evolve, with Layer 4 changing from displaying coarse to more refined common semantic sub-parts and Layer 10 changing from displaying no shared characteristics to high-frequency image gradients. These observations indicate that the diffusion model provides coarse and fine features that capture different image characteristics (i.e. semantic or textural information) throughout different combinations of layers and timesteps. Hence, we find it important to extract features from all layers and timesteps in order to adequately tune our final descriptor map to represent the appropriate level of granularity needed for a given task.

Inversion. These same useful features can be extracted for real images through the inversion process. Although inversion is a process that destructs the real image into noise, we observe that its features contain useful information akin to the generation process for synthetic images. In Figure 3, we can see that our inversion features are able to reliably capture the full body of both cats and their common semantic subparts (head, torso, legs) in Layer 4 and their edges in Layer 10 even at a timestep when the input to the model is relatively noisy. In contrast, using the generation process to analyze real images (as done in prior work) leads to hyperparameter tuning and tradeoffs. For example, at timesteps close to t = T where in-distribution inputs are close to noise, the features start to diverge from information present in the real image and may even hallucinate extraneous details, as seen in Figure 3. In this example, because the color of the top cat's stomach is white like the background, the generation features from Layer 4 merge the stomach with the background. Similarly, because there is low contrast between the bottom cat and the background, the generation features from

Layer 10 fail to capture the silhouette of the cat and instead depict random texture edges. Intuitively, inversion features are more trustworthy because of the notion of chaining , where at every step the input is some previous output of the model rather than a random mixture of an image and noise, and every extracted feature map is therefore interrelated. Extracting features from a continuous inversion process also induces symmetry with the generation process, which in Section 4.3 we demonstrate allows us to use both feature types interchangeably with the same aggregation network.

3.2 Diffusion Hyperfeatures Aggregation

Given a dense set of feature maps from the diffusion process, we now must efficiently aggregate them into a single descriptor map without omitting the information from any layers or timesteps. A naive solution would be to simply concatenate all feature maps into one very deep feature map, but this proves to be too high dimensional for most applications. We address this issue with our aggregation network, which standardizes the feature maps with tuned bottleneck layers and sums them according to learned mixing weights. Specifically, for a given feature map r we upsample it to a standard resolution, pass through a bottleneck layer B [16, 48] to a standard channel count, and weight it with a mixing weight w . The final Diffusion Hyperfeatures take on the form

where L is the number of layers and S is the number of timesteps. Note that we run the diffusion process for a total number of T timesteps but only select a subsample of S timesteps for aggregation to conserve memory. We opt to share bottleneck layers across timesteps, meaning we use a total of L bottleneck layers. However, we learn L · S unique mixing weights for every combination of layer and timestep. We then tune these bottleneck layers and mixing weights using task-specific supervision. For semantic correspondence, we flatten the descriptor maps for a pair of images and compute the cosine similarity between every possible pair of points. We then supervise with the labeled corresponding keypoints using a symmetric cross entropy loss in the same fashion as CLIP [36]. During training, we downscale the labels according to the resolution of our descriptor maps. When running inference, we upsample the descriptor maps before performing nearest neighbors matching to predict semantic keypoints. We demonstrate that this lightweight parameterization of our aggregation network is performant (see Section 4.1) and interpretable (see Section 4.2) without degrading the open-domain knowledge represented in the diffusion features (see Section 4.3).

4 Experiments

Baselines. We compare against both zero-shot and supervised methods for the task of semantic correspondence. For our zero-shot baselines, we compare against the DINO [2, 4] Layer 9 key features and DINOv2 [32] Layer 11 token features, derived from a self-supervised model trained on ImageNet [8] or LVD-142M [32] respectively. For our supervised baselines, we compare against DHPF [29] and CATS++ [6], which aggregate ResNet-101 [16] hypercolumn features. The features are derived from a supervised model pretrained on ImageNet and finetuned on SPair-71k [28]. We also compare against two diffusion baselines, selecting a single feature map from a known semantic layer [43] (SD-Layer-4) and concatenating feature maps from all layers (SD-Concat-All).

Experimental Details. Our method extracts descriptors from Stable Diffusion v1-5 [37], a generative latent diffusion model trained on LAION-5B [38]. We tune our bottleneck layers and mixing weights on SPair-71k for up to 5000 steps with a batch size of 2 and a learning rate of 1e-3 using an AdamW optimizer [21]. We extract features from the UNet decoder layers (denoted as Layers 1 to 12), specifically the outputs of the residual block before the self- and cross- attention blocks. Our full method runs the inversion or generation process over the standard T = 50 scheduler timesteps and subsamples every 5 steps, thereby aggregating features over S = 11 total steps. We also ablate using a 'one-step' process ( T = 1 , S = 1 ). When operating on real images, we feed the empty prompt ''. We find that extracting features only from the unconditional model removes the need for manual prompting and yields high-quality results. When operating on synthetic images, we extract features only from the conditional model, or the branch conditioned on the prompt that generated the image.

Metrics. We report results in terms of PCK@ α , or the percentage of correct keypoints at the threshold α . The predicted keypoint is considered to be correct if it lies within a radius of α ∗ max ( h, w ) of

Table 1: We evaluate our semantic keypoint matching performance on real images from SPair-71k and CUB. For our CUB evaluation, we transfer the model tuned on SPair-71k. We compare against Stable Diffusion baselines that extract features from a single layer (SD-Layer-4) or concatenation of all layers (SD-Concat-All). We ablate pruning to the single feature map with the highest mixing weight selected by our method, either as the raw feature map (SD-Layer-Pruned) or after the bottleneck layer (Ours-Pruned). We ablate tuning our method with only one timestep (One-Step) or features from another Stable Diffusion variant (SDv2-1). *Note that DINOv2 was trained on samples from CUB [32].

the ground truth annotation, where h, w denotes the dimensions of the entire image ( α img ) or object bounding box ( α bbox ) in the original image resolution. Following Amir et al. [2] we use the threshold α = 0 . 1 [2]. We use the default input resolution for each baseline, and we 'compute the metrics on the standard setting, i.e. the original image size' [42] to ensure a fair comparison. For DINO, DHPF, and CATS++ we use the resolutions 224, 240, and 512 respectively. For DINOv2 we use resolution 770 to account for its larger patch size and for Stable Diffusion variants we use resolution 224 in line with our DINO baseline.

Additional Experiments. In the Supplemental we provide additional experiments ablating performance of raw features across individual decoder layers and model variants, performance of raw inversion vs. generation features, the effect of aggregating DINO and DINOv2 features, and applying Diffusion Hyperfeatures for dense warping.

4.1 Semantic Keypoint Matching on Real Images

We evaluate on the semantic keypoint correspondence benchmark SPair-71k [28], which is composed of image pairs from 18 object categories spanning animals, vehicles, and household objects. Following prior work [2], we evaluate on 360 random image pairs from the test split, with 20 pairs per category. To test the method's ability to transfer to unseen data, we also evaluate on CUB [47], which is composed of semantic keypoints on a variety of bird species. Our validation set is 360 random image pairs taken from the Kulkarni et al. [22] validation split.

As seen in Table 1, even when compared with the competitive baselines DINOv2 and CATS++, our method achieves the best result with 72.56 PCK@ 0 . 1 img and 82.29 PCK@ 0 . 1 img on SPair-71k and CUB respectively. Selecting a single diffusion feature map (SD-Layer-4) already outperforms DINO and DHPF by a margin of at least 4% in PCK@ 0 . 1 img , likely due to the underlying model's larger and more diverse training set. Conversely, naively concatenating all feature maps (SD-ConcatAll), degrades this improvement, likely because each map captures a different level of granularity and therefore should not be equally weighted for a coarse-level semantic similarity task. When aggregating all feature maps across time and space using our Diffusion Hyperfeatures, we see a significant boost of 14% in PCK@ 0 . 1 img over SD-Layer-4. This sizeable performance improvement indicates that (1) while a single layer may capture a good deal of semantic information, the other layers also capture complementary information that can be useful for disambiguating points on a finer level and (2) the diffusion model's internal knowledge about the image is not fully captured by a single pass through the UNet but is rather spread out over multiple timesteps.

In Figure 4, we show qualitative examples of our predicted semantic correspondences compared with the strongest baselines. In general, while both baselines are able to relate broad semantic regions

Ground Truth

SD-Layer4

DINOv2

CATS++

Ours

SPair-71k

OTJUST

SPair-71k

CUB

such as the head, arms, or legs, they struggle with fine-grained subparts and confuse them with other visually similar regions. For example, SD-Layer-4 and DINOv2 confuse the headlight (yellow) of the blue car with the rear light of the red car, whereas our method is able to correctly reason about the front vs. back of the cars. In a similar fashion, unlike our method both of these baselines also confuse the back of the seagull's neck (orange) with the front of the northern flicker bird's neck.

4.2 Ablations

Number of Diffusion Steps. We ablate the importance of the time dimension by tuning a variant of our method on feature maps from a one-step inversion process (Ours-One-Step), where there is only one possible timestep to extract from. While aggregating over all layers performs better than single layer selection (SD-Layer-4) or simple concatenation (SD-Concat-All) by a margin of at least 5% in PCK@ 0 . 1 img , it also heavily lags behind our full method. As indicated by our mixing weights in Figure 5, the most useful information for semantic correspondence are concentrated in the early timesteps of the diffusion process, where the input image is relatively clean but contains some noise. Timestep selection can be thought of as a knob for the amount of high frequency detail present in the image to analyze, where at these early timesteps the model is implicitly mapping the noisy input to a smoother image, similar to the oversmoothed cats for the predicted x 0 in Figure 2. Evidently not all texture and detail is necessary for the task of semantic correspondence, and the model is able to produce more useful features at different timesteps which highlight different image frequencies.

Pruning. Since our method employs interpretable mixing weights, we have a ranking of the relative importance of each layer and timestep combination for the task of semantic correspondence. To verify this ranking, we ablate pruning to the single feature map with the highest mixing weight. As seen in Figure 5, for Ours-SDv1-5 this is the feature map associated with Layer 5, Timestep 10. Referring to Table 1, this automatically selected raw feature map (SD-Layer-Pruned) performs comparably to the feature map manually selected by prior work (SD-Layer-4). Interestingly, if only viewing features from a one-step inversion process it would seem that Layer 5 features are significantly worse than Layer 4 features, as discussed further in the Supplemental, but the story changes after tuning selection over both time and space. After passing this pruned feature map to our learned bottleneck layer, the feature map performs comparably on CUB and 6% better in PCK@ 0 . 1 img on SPair-71k, as seen when comparing SD-Layer-Pruned and Ours-Pruned in Table 1. This trend validates that our bottleneck layer does not degrade the power of the original representation, but rather refines it in a way that is likely helpful for complex object categories present in SPair-71k beyond birds. Considering the 9% gap in PCK@ 0 . 1 img between our pruned and full method (Ours-Pruned vs. Ours), it becomes evident that it is not a single feature map that drives our strong performance but rather the soft mixing of many feature maps.

Mixing

WeightsTunedonSpair-71k

GroundTruth

SDv2-1-Layer4

Ours-SDv2-1

SDv1-5

SDv2-1

Layer

Layer

11

11

12

12

05 10 152025303540 4550

Timestep

05 10 15 20 2530 35 40 45 50

Timestep

Model Variant. We also ablate using a different model variant for our diffusion feature extraction, namely SDv2-1. Most notably, SDv2-1 differs from SDv1-5 in its large-scale text encoder, which scales from CLIP [36] to OpenCLIP [19]. The mixing weights learned for both model variants, depicted in Figure 5, showcase the same high-level trends, where the features found to be the most useful for semantic correspondence are concentrated in middle Layers 4-9 and early Timesteps 5-15. However, on a more nuanced level, the behavior starts to diverge with regards to the relative importance of layers and timesteps within this range. Namely, layer selection moves from Layers 4-5 to higher resolution Layers 5-7 from SDv1-5 to v2-1. This behavior is confirmed by our early hyperparameter sweeps of raw feature maps across model variants discussed in the Supplemental, where in fact the Layer 4 feature map of SDv2-1 performs extremely poorly for the task of semantic correspondence. Timestep selection also moves from Timestep 10 to Timestep 5 from SDv1-5 to v2-1, which is surprising because this means that SDv2-1 tends to select higher resolution feature maps from timesteps with higher frequency inputs for a task where it is essential to abstract fine-grained details into semantically meaningful matches. These trends seem to imply that a more powerful text encoder produces shared semantic representations at increased levels of detail, possibly because the model is better able to connect more distantly related visual concepts via text instead of giving them completely disjoint representations. Hence, our aggregation network is able to dynamically adjust to the representations being aggregated and the task at hand, both of which influence the most important set of features to select from the diffusion process.

4.3 Transfer on Synthetic Images

In addition to evaluating the transfer of our aggregation network to other datasets such as CUB, we also evaluate on synthetic images. Specifically, we take the same aggregation network tuned on inversion features and simply flip the timestep ordering to operate on generation features. Therefore in this setting, we are testing our network's ability to generalize (1) to a completely unseen feature type from a different diffusion process (inversion vs. generation) and (2) out-of-domain object categories that are not present in SPair-71k. Surprisingly, our network generalizes well, outperforming predictions from both DINO and the raw feature map from the last step of the generation process (SD-Layer-4) as seen in Figure 6. Although DINO and SD-Layer-4 are generally able to correspond broad semantic regions correctly, they sometimes have difficulty with relative placement of subparts. In the case of the rungs of the Eiffel Tower (purple, pink, green), DINO collapses all of its predictions onto the middle rung and SD-Layer-4 collapses them onto the left rung, whereas our method is able to correctly correspond the middle and side rungs of the tower in a triangle formation. The baselines can also be distracted by other objects in the scene that are visually similar. In the case of the mermaid's legs (red, yellow), both baselines incorrectly correspond certain points with the rock in the right image, whose contours and silhouette resemble the legs. On the other hand, our method is able to predict more reliable correspondences for fine-grained subparts, even in these challenging cases with unseen textures (lego, snow) and categories (hat, cactus). The ability of our aggregation network to extend to open-domain synthetic images opens up the exciting possibility of generating custom synthetic datasets with pseudo ground-truth semantic correspondences, which we demonstrate are more precise than correspondences derived from the raw feature maps.

Ground Truth

DINO

SD-Layer-4

Ours

Acolorfulhotair

balloon soaring over

a city skyline and

treeswiththeEiffel

Towerinthe

background.

Amermaid readinga

bookonasunken

pirate ship.

TheStatueofLiberty

made out of Lego.

Asnowmanwearing

a hat and a scarf

sitting atopa

mountain.

Arobotwateringa

garden of flowers

5 Conclusion

We demonstrate that our Diffusion Hyperfeatures are able to distill the information distributed across time and space from a diffusion process into a single descriptor map. Our intepretable aggregation network also enables automatic analysis of the most useful layers and timesteps based on both the underlying model and task of interest. We outperform methods that use supervised hypercolumns or self-supervised descriptors by a large margin on a semantic keypoint correspondence benchmark comprised of real images. Although we tune on a small set of real images with limited categories, we demonstrate that our method is able to retain the open-domain capabilities of the underlying diffusion features by demonstrating strong performance in predicting semantic correspondences in challenging synthetic images, especially compared to using the raw feature maps. Our ability to predict high-quality correspondences derived from the same feature maps used to produce the synthetic image could potentially be employed to create synthetic image sets with pseudo-labeled semantic keypoints, which would be valuable for downstream tasks such as image-to-image translation or 3D reconstruction.

References

Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best-buddies: Sparse cross-domain correspondence. ACM Transactions on Graphics (TOG) , 37(4):1-14, 2018.

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. Proceedings of the European Conference on Computer Vision Workshops (ECCVW) , 2022.

Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Labelefficient semantic segmentation with diffusion models. In International Conference on Learning Representations , 2022.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE international conference on computer vision , pages 1511-1520, 2017.

Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. arXiv preprint arXiv:2202.06817 , 2022.

Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) , volume 1, pages 886-893. Ieee, 2005.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Walter Goodwin, Sagar Vaze, Ioannis Havoutis, and Ingmar Posner. Zero-shot category-level object pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022.

Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. Asic: Aligning sparse in-the-wild image collections, 2023.

[12]

Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow. In

Proceedings of the IEEE

Conference on Computer Vision and Pattern Recognition (CVPR)

, 2016.

Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow: Semantic correspondences from object proposals. In PAMI , 2017.

Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In International Conference on Learning Representations , 2022.

Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns for object segmentation and fine-grained localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin C.K. Chan, and Ziwei Liu. ReVersion: Diffusion-based relation inversion from images. arXiv preprint arXiv:2303.13495 , 2023.

D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology , 160(1), 1962.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.

D. G. Jones and J. Malik. Determining three-dimensional shape from orientation and spatial frequency disparities. In European Conference on Computer Vision , 1992.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Nilesh Kulkarni, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani. Articulation-aware canonical surface mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 452-461, 2020.

Erik G. Learned-Miller. Data driven image models through continuous joint alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence , 28(2):236-250, 2006.

David G. Lowe. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision , 60(2):91-110, nov 2004.

J. Malik and P. Perona. Preattentive texture discrimination with early vision mechanisms. Journal of the Optical Society of America A , 7(5), 1990.

Xin Mao, Zhaoyu Su, Pin Siang Tan, Jun Kang Chow, and Yu-Hsing Wang. Is discriminator a good feature extractor? arXiv preprint arXiv:1912.00789 , 2019.

Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Hyperpixel flow: Semantic correspondence with multi-layer neural features. In ICCV , 2019.

Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. arXiv prepreint arXiv:1908.10543 , 2019.

Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Learning to compose hypercolumns for visual correspondence. In ECCV , 2020.

David Novotny, Diane Larlus, and Andrea Vedaldi. Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2017.

Dolev Ofri-Amar, Michal Geyer, Yoni Kasten, and Tali Dekel. Neural congealing: Aligning images to a joint semantic atlas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.

Dong Huk Park, Grace Luo, Clayton Toste, Samaneh Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell. Shape-guided diffusion with inside-outside attention. arXiv , 2022.

William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei Efros, and Eli Shechtman. Gan-supervised dense visual alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbelaez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 DAVIS challenge on video object segmentation. CoRR , abs/1704.00675, 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv:2103.00020 , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Probabilistic warp consistency for weaklysupervised semantic correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8708-8718, June 2022.

Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for textdriven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Nikolai Ufer and Bjorn Ommer. Deep semantic feature matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2017.

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Highresolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

J. Weber and J. Malik. Robust computation of optical flow in a multi-scale differential framework. International Journal of Computer Vision , 14(1), 1995.

P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Odise: Openvocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Jianglong Ye, Naiyan Wang, and Xiaolong Wang. Featurenerf: Learning generalizable nerfs by distilling pre-trained vision foundation models. arXiv preprint arXiv:2303.12786 , 2023.

Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2018.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation, 2022.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to extract per-pixel feature descriptors from multi-scale and multi-timestep feature maps generated by diffusion models. These descriptors can be utilized for various downstream tasks.\n\nThe framework is evaluated on the task of semantic keypoint correspondence, specifically on the SPair-71k real image benchmark. The authors claim that their method achieves superior performance on this benchmark compared to other approaches. Additionally, they demonstrate that their method is flexible and transferable, as the feature aggregation network trained on real image pairs can also be used on synthetic image pairs with unseen objects and compositions.\n\nOverall, the proposed Diffusion Hyperfeatures framework aims to enhance the internal representations of diffusion models and improve their utility in computer vision tasks like semantic keypoint correspondence.\n\nSTRENGTHS:\nThis paper discusses a framework called \"Diffusion Hyperfeatures\" for consolidating feature maps in diffusion models. Here's a substantive assessment of the strengths of the paper across various dimensions:\n\nOriginality: The paper introduces a novel approach to consolidate and extract per-pixel descriptors from intermediate feature maps generated by diffusion models. While existing works focus on selecting specific layers and timesteps, this paper proposes a comprehensive framework that considers all intermediate features. This consolidation process through a feature aggregation network is a new concept, which contributes to the originality of the paper.\n\nQuality: The paper appears to be of high quality as it references relevant literature and state-of-the-art techniques in computer vision, such as ConvNets, Vision Transformers, and GANs. The authors also evaluate their proposed framework on the task of semantic keypoint correspondence using real images from the SPair-71k benchmark. The evaluation includes an analysis of different layers and timesteps of diffusion model features, demonstrating a thorough investigation. Additionally, the generalization of Diffusion Hyperfeatures to out-of-domain data is examined by evaluating them on synthetic images generated by the diffusion model.\n\nClarity: The paper presents its ideas and contributions in a clear manner. It provides a concise overview of the challenges faced with existing methods and describes the proposed Diffusion Hyperfeatures framework step-by-step. The use of terminology and technical language is appropriate for the target audience of researchers in computer vision. However, without the full paper, it is difficult to assess the clarity of the detailed methodology and experimental setup.\n\nSignificance: The paper addresses an important problem in computer vision, namely how to effectively utilize the internal representations of diffusion models for downstream tasks. By proposing the Diffusion Hyperfeatures framework, the authors aim to improve the utility of diffusion models beyond image generation. The evaluation results on semantic keypoint correspondence and the demonstration of generalization to synthetic images indicate the potential significance of this approach. If the proposed framework proves to be effective, it could contribute to advancements in various computer vision applications.\n\nOverall, based on the provided text, the paper demonstrates strengths in terms of originality, quality, clarity, and potential significance.\n\nWEAKNESSES:\nBased on the provided paper, here are a few potential areas where the paper could benefit from improvement:\n\nComparison with Existing Methods: While the paper mentions that features from ConvNets, Vision Transformers, and GANs have demonstrated significant capabilities, it would be beneficial to provide a more detailed comparison with existing methods that extract feature descriptors from diffusion models. This would help establish the novelty and superiority of the proposed Diffusion Hyperfeatures framework.\n\nExperimental Evaluation: The paper briefly mentions evaluating the proposed framework on the task of semantic keypoint correspondence using real images from the SPair-71k benchmark. To strengthen the paper, it would be valuable to include a thorough analysis of the experimental results, including quantitative metrics, comparative evaluations with state-of-the-art methods, and possibly visual illustrations or qualitative assessments of the generated keypoint correspondences.\n\nGeneralization to Diverse Domains: The paper mentions evaluating Diffusion Hyperfeatures on synthetic images generated by the diffusion model. While this demonstrates some level of generalization, it would be useful to explore the performance and robustness of the framework across a broader range of datasets and domains. This could involve testing the framework on different benchmark datasets and real-world scenarios to validate its effectiveness in diverse settings.\n\nClear Methodology Description: The provided text offers a high-level overview of the proposed framework, but for a comprehensive evaluation, it is crucial to have a clear and detailed description of the methodologies employed. It would be helpful to include information about the architecture of the feature aggregation network, the specific techniques used for consolidation, and any additional preprocessing steps or modifications made to the diffusion model.\n\nPotential Limitations and Future Directions: The paper could benefit from discussing potential limitations or challenges associated with the proposed framework. Identifying these limitations and suggesting future directions for improvement would enhance the depth and completeness of the research.\n\nQUESTIONS:\nClarification on Consolidation Process: Could the authors provide more details regarding the feature aggregation network used to consolidate the intermediate feature maps? How does it handle variations in scale and time? Are there any specific design choices or architectural considerations that impact its performance?\n\nExperimental Setup: It would be helpful to understand the specific experimental setup employed in the evaluation of Diffusion Hyperfeatures. Could the authors elaborate on the dataset used, the selection of keypoint correspondence task, and the metrics used for evaluation? Additionally, is there any consideration given to computational efficiency or runtime performance when applying the framework to real-time applications?\n\nComparison with Existing Methods: The authors briefly mention that existing works select specific subsets of layers and timesteps from diffusion models for different tasks. Could the authors provide a more detailed comparison or discussion on how their approach differs and potentially outperforms these existing methods? Are there any specific limitations or challenges associated with the subset selection approach that Diffusion Hyperfeatures address?\n\nTransferability and Generalization: While the paper mentions generalization to synthetic images, it would be valuable to explore the transferability and generalization capabilities of Diffusion Hyperfeatures across other domains or datasets as well. Can the authors provide insights or experiments on how the framework performs on different benchmark datasets or real-world scenarios beyond the SPair-71k dataset?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes diffusion hyperfeatures, a framework for integrating different scale and timestep features to form a representative feature descriptors in dense level. Within the U-Net architecture, unlike other works that uses hand-crafted methods to select a particular subset of layers for further processing, this paper proposes a simple aggregation network that aggregates the intermediate features. The effectiveness of this method is evaluated on standard benchmark of semantic correspondence, including SPair-71k.\n\nSTRENGTHS:\n1. SOTA performance is achieved with large gap to existing works.\n\n2. One of the first attempt to tackle semantic correspondence task with diffusion. \n\n3. Although paper is easy-to-read, each paragraphs are too long. Also it is quite hard to understand some sentences. This is a minor issue though.\n\nWEAKNESSES:\n1. This paper is one of the first attempt to tackle semantic correspondence with diffusion concept. In this manner, this paper looks novel, since this is very new to this task. However, to me, this work severely lacks contributions. What this paper does is simply using a series of features from stable diffusion, a large-scale model, feeds to a simple aggregation network that simply performs weighted sum to obtain the final features and it is exploited for Winner-Takes-All (WTA) for finding correspondences. \n (1) An attempt to select features are not novel as this is already smartly done in HPF [23] and DHPF [25], while this work simply uses all the features to feed to an aggregation network.\n (2) Using different feature representation  that is much powerful than standard backbone (resnet 101) will guarantee apparent performance boosts. This is already demonstrated in CATs++: Boosting Cost Aggregation With Convolutions and Transformers (TPAMI). \n(3) No novel diffusion techniques are proposed as far as i understood. \n\n\n\n3. Lacking implementation details. What resolution was used for evaluation and training? This is very important in this task as the performance highly correlates to it. Unless this is evaluated in the same resolution and compared with other works, the comparisons are not fair at all. \n\n4. Why did the authors use only 2 datasets for evaluation? Traditionally, existing methods would also evaluate their methods on Pf-PASCAL, PF-WILLOW, TSS as well. This needs to be included, since evaluation on 2 datasets seem very insufficient. \n\n5.  In supplementary material, section 6.1 explains computational resources. This is very abstract and not very helpful. Comparison to other works is not a must, but at least provide some measures for completeness.\n\n6. DINOv1 is a quite old-model, since DINO-v2 is already out. DINO-v2 is a much powerful model that will yield much stronger performance. \n\n7. Many other semantic correspondence works are not cited and compared. As far as I know, the current SOTA is Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence, arxiv'22, which would be better to be included in the paper. Also when it is compared with DINO, only 1 layer is used. In semantic correspondence, multi-scale and multi-level features are highly important which would have detrimental effects on performance without them. So the comparison may not be fair here as well. However, as the authors also provided results for single layer results, this is not an issue, but a suggestion to include results that use more layers of DINOv1 or DINOv2.\n\nQUESTIONS:\nSee weaknesses above\n\nLIMITATIONS:\nLimitations or failure cases are not presented.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes improving feature distillation from diffusion models for representation learning by aggregating information from the feature maps of the U-Net at varying timesteps, weighting them with a tunable aggregation network. The authors show that even at timesteps from which features are usually discarded for feature learning (i.e. early steps in the generation process) useful features can be found. After formulating their approach to extracting these features using a weighted aggregation of a set of standardized input feature maps, authors show validity of their approach by improving considerably over baseline methods and prior work that only selects a single diffusion timestep for feature extraction. Lastly, authors show the ability of their framework to generalize to unseen synthetic data (generated by the diffusion model), by extracting features from the generation process for a diffusion model trained on a different data distribution.\n\nSTRENGTHS:\n- The idea of extracting features from powerful generative models such as diffusion models is very relevant, and the authors show that their approach to this extraction process obtains great results.\n- The paper is very well-written and reads very smoothly. Authors are very descriptive in their wording and use figures to clearly illustrate their approach. The idea is simple and seems very effective, as authors illustrate with a number of experiments and ablation over baselines.\n- Argumentation for design choices is very solid, and authors motivate their approach with an experiment showing the validity/salience of features found in earlier timesteps of the diffusion model.\n- The authors also show impressive transfer performance to an unseen dataset, highlighting possible applications in pseudo-label generation for semantic correspondence.\n\nWEAKNESSES:\n- I have no major concerns. A minor concern I have is with regards to novelty, given that extracting hypercolumn features has been attempted before, even for diffusion models. \n- Another minor concern is the requirement for task-specific fine-tuning of the aggregation network. The fact that this network requires explicit supervision for a task means that the actual representation extracted from the diffusion model that can be used in arbitrary downstream tasks is the whole set of features maps across timesteps. Would it be possible to train an aggregation network on a self-supervised task?\n\nQUESTIONS:\n- For figures 2, 3, would it be possible to show also the network input at these timesteps for reference? \n- Line 117, you mention that “these observations indicate that the diffusion model provides coarse and fine features that capture different image characteristics, throughout different combinations of layers and timesteps”. I’m wondering to what extent this is owing to the formulation of the diffusion process itself, and to what extent this is due to the use of a very specific multi-resolution processing architecture (U-Net) in the diffusion process. I.e. is the extraction of both coarse and fine image features a result of diffusion or a result of using a U-Net? Would your method also work with diffusion models that make use of other architectures?\n- Line 135, I have a bit of trouble understanding your reasoning for the more “trustworthy” inversion features. Could you elaborate? Why would repeated application of the model necessarily lead to more “trustworthy” features?\n- You apply your feature extraction specifically for semantic correspondence detection. Could the extracted features also be used in other settings? Are there any limiting factors that prevent your framework from being used in such settings? Have you tried e.g. classification of extracted features?\n\nLIMITATIONS:\nThe authors do not discuss the limitations or societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach for extracting useful features for pre-trained diffusion models for application to dense visual correspondence tasks. How to do this is not clear due to presence of features both through the network, and over diffusion steps. The proposed approach is to learn which features to use using the hypercolumns framework, by passing activations through bottleneck layers then averaging over layers and time. Experiments show that this method outperforms unsupervised DINO features and supervised hypercolumns.\n\nSTRENGTHS:\n- The premise of extracting useful features from diffusion models is an important one; we know that those features are present but it is not clear how to extract useful ones.\n- Using inversion to get features for real images is a good idea and the explanation that they are more reliable than sampling from the posterior as used in prior work makes sense.\n- The exploration into what feature representations are learned at different layers/times is interesting (Figures 2 and 3).\n- The approach is shown to outperform unsupervised DINO features as well as hypercolumns (Table 1) showing that useful features have been extracted from the diffusion model; the baselines including using a single layer from the diffusion model demonstrate the benefit of combining over layers/time; and visual examples shown in Figures 4-6 demonstrate the approach working.\n\nWEAKNESSES:\n- Unlike using DINO features, the proposed approach requires training additional components for downstream tasks, making it much less versatile.\n- The baseline models in Table 1 are a poor representation of current approaches. DINO features are not trained specifically for keypoint matching and newer supervised approaches such as CATS++ (Cho et al. TPAMI 2022) and VAT (Hong et al. ECCV 2022) substantially outperform DHPF.\n- The approach shares bottleneck layers across time steps; this seems potentially problematic since features differ substantially across different times.\n\nQUESTIONS:\n- How does it quantitatively compare against more recent supervised approaches?\n- Why share bottleneck layers over time steps? They contain very different representations.\n- Why not use attention features from the diffusion model in a similar manner to DINO?\n- Aggregating features over time steps is difficult, hence the trained layers on top. Could using a distilled diffusion model such as a consistency model give better representations?\n\nLIMITATIONS:\nAs stated in the submission details, limitations are not discussed. There are a number of limitations including having to train additional components (unlike DINO), not comparing with more recent supervised methods, and sharing layers across time steps.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper explores semantic correspondence tasks with stable diffusion model. Specifically, the authors proposed to first extract feature maps varying across timesteps and layers from the diffusion process and trains a lightweight neural network to aggregate them together for semantic correspondence.
Experimental results on CUB-200 and SPair-71k datasets show that proposed method outperforms other baselines.

STRENGTHS:
- This paper shows that one could explore diffusion models for semantic correspondence tasks.

- the writing is clear and easy to follow.

- The experiments show that the proposed method could outperform other baselines on the SPair-71k real image benchmark.

- the authors have shown numerous visual examples to demonstrate the correspondence capability.

WEAKNESSES:
1. The proposed method, that "distill the information  distributed across time and space from a diffusion process into a single descriptor map", seems to have more potential than correspondence tasks. Have the authors explore other tasks, like video label propagation, Homography estimation or perception tasks like classification?


2. Concurrent works: there are multiple papers presenting correspondence ability of diffusion models:

[1*] Hedlin, Eric, et al. "Unsupervised Semantic Correspondence Using Stable Diffusion." arXiv preprint arXiv:2305.15581 (2023).
[2*] Tang, Luming, et al. "Emergent Correspondence from Image Diffusion." arXiv preprint arXiv:2306.03881 (2023).
[3*] Zhang, Junyi, et al. "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence." arXiv preprint arXiv:2305.15347 (2023).

Could the authors explain the differences if possible?

QUESTIONS:
please refer to the weaknesses section.

LIMITATIONS:
no
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10