SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Number of Diffusion Steps. We ablate the importance of the time dimension by tuning a variant of our method on feature maps from a one-step inversion process (Ours-One-Step), where there is only one possible timestep to extract from. While aggregating over all layers performs better than single layer selection (SD-Layer-4) or simple concatenation (SD-Concat-All) by a margin of at least  $5\%$  in PCK@0.1img, it also heavily lags behind our full method. As indicated by our mixing weights in Figure 5, the most useful information for semantic correspondence are concentrated in the early timesteps of the diffusion process, where the input image is relatively clean but contains some noise. Timestep selection can be thought of as a knob for the amount of high frequency detail present in the image to analyze, where at these early timesteps the model is implicitly mapping the noisy input to a smoother image, similar to the oversmoothed cats for the predicted  $x_0$  in Figure 2. Evidently not all texture and detail is necessary for the task of semantic correspondence, and the model is able to produce more useful features at different timesteps which highlight different image frequencies.
Pruning. Since our method employs interpretable mixing weights, we have a ranking of the relative importance of each layer and timestep combination for the task of semantic correspondence. To verify this ranking, we ablate pruning to the single feature map with the highest mixing weight. As seen in Figure 5, for Ours-SDv1-5 this is the feature map associated with Layer 5, Timestep 10. Referring to Table 1, this automatically selected raw feature map (SD-Layer-Pruned) performs comparably to the feature map manually selected by prior work (SD-Layer-4). Interestingly, if only viewing features from a one-step inversion process it would seem that Layer 5 features are significantly worse than Layer 4 features, as discussed further in the Supplemental, but the story changes after tuning selection over both time and space. After passing this pruned feature map to our learned bottleneck layer, the feature map performs comparably on CUB and  $6\%$  better in PCK@0.1img on SPair-71k, as seen when comparing SD-Layer-Pruned and Ours-Pruned in Table 1. This trend validates that our bottleneck layer does not degrade the power of the original representation, but rather refines it in a way that is likely helpful for complex object categories present in SPair-71k beyond birds. Considering the  $9\%$  gap in PCK@0.1img between our pruned and full method (Ours-Pruned vs. Ours), it becomes evident that it is not a single feature map that drives our strong performance but rather the soft mixing of many feature maps.
Figure 5: The learned mixing weights when aggregating SDv1-5 vs. SDv2-1 features across multiple layers and timesteps. Bright yellow denotes a high weighting, and dark blue denotes a low weighting. We also depict predicted correspondences from SDv2-1-Layer-4 vs. Ours-SDv2-1. While Layer 4 features from SDv1-5 perform well in semantic correspondence, this same layer in SDv2-1 performs extremely poorly. Our method automatically learns the best layers depending on the model variant.
Model Variant. We also ablate using a different model variant for our diffusion feature extraction, namely SDv2-1. Most notably, SDv2-1 differs from SDv1-5 in its large-scale text encoder, which scales from CLIP [36] to OpenCLIP [19]. The mixing weights learned for both model variants, depicted in Figure 5, showcase the same high-level trends, where the features found to be the most useful for semantic correspondence are concentrated in middle Layers 4-9 and early Timesteps 5-15. However, on a more nuanced level, the behavior starts to diverge with regards to the relative importance of layers and timesteps within this range. Namely, layer selection moves from Layers 4-5 to higher resolution Layers 5-7 from SDv1-5 to v2-1. This behavior is confirmed by our early hyperparameter sweeps of raw feature maps across model variants discussed in the Supplemental, where in fact the Layer 4 feature map of SDv2-1 performs extremely poorly for the task of semantic correspondence. Timestep selection also moves from Timestep 10 to Timestep 5 from SDv1-5 to v2-1, which is surprising because this means that SDv2-1 tends to select higher resolution feature maps from timesteps with higher frequency inputs for a task where it is essential to abstract fine-grained details into semantically meaningful matches. These trends seem to imply that a more powerful text encoder produces shared semantic representations at increased levels of detail, possibly because the model is better able to connect more distantly related visual concepts via text instead of giving them completely disjoint representations. Hence, our aggregation network is able to dynamically adjust to the representations being aggregated and the task at hand, both of which influence the most important set of features to select from the diffusion process.
4.3 Transfer on Synthetic Images
In addition to evaluating the transfer of our aggregation network to other datasets such as CUB, we also evaluate on synthetic images. Specifically, we take the same aggregation network tuned on inversion features and simply flip the timestep ordering to operate on generation features. Therefore in this setting, we are testing our network's ability to generalize (1) to a completely unseen feature type from a different diffusion process (inversion vs. generation) and (2) out-of-domain object categories that are not present in SPair-71k. Surprisingly, our network generalizes well, outperforming predictions from both DINO and the raw feature map from the last step of the generation process (SD-Layer-4) as seen in Figure 6. Although DINO and SD-Layer-4 are generally able to correspond broad semantic regions correctly, they sometimes have difficulty with relative placement of subparts. In the case of the rungs of the Eiffel Tower (purple, pink, green), DINO collapses all of its predictions onto the middle rung and SD-Layer-4 collapses them onto the left rung, whereas our method is able to correctly correspond the middle and side rungs of the tower in a triangle formation. The baselines can also be distracted by other objects in the scene that are visually similar. In the case of the mermaid's legs (red, yellow), both baselines incorrectly correspond certain points with the rock in the right image, whose contours and silhouette resemble the legs. On the other hand, our method is able to predict more reliable correspondences for fine-grained subparts, even in these challenging cases with unseen textures (lego, snow) and categories (hat, cactus). The ability of our aggregation network to extend to open-domain synthetic images opens up the exciting possibility of generating custom synthetic datasets with pseudo ground-truth semantic correspondences, which we demonstrate are more precise than correspondences derived from the raw feature maps.
Figure 6: Example synthetic images and their text prompts, the ground-truth user-annotated correspondences, and predicted correspondences from DINO, SD-Layer-4, and our method. We transfer the aggregation network tuned on inversion features of real images to generation features of synthetic images that are completely out-of-domain compared to the SPair-71k categories.
We demonstrate that our Diffusion Hyperfeatures are able to distill the information distributed across time and space from a diffusion process into a single descriptor map. Our interpretable aggregation network also enables automatic analysis of the most useful layers and timesteps based on both the underlying model and task of interest. We outperform methods that use supervised hypercolumns or self-supervised descriptors by a large margin on a semantic keypoint correspondence benchmark comprised of real images. Although we tune on a small set of real images with limited categories, we demonstrate that our method is able to retain the open-domain capabilities of the underlying diffusion features by demonstrating strong performance in predicting semantic correspondences in challenging synthetic images, especially compared to using the raw feature maps. Our ability to predict high-quality correspondences derived from the same feature maps used to produce the synthetic image could potentially be employed to create synthetic image sets with pseudo-labeled semantic keypoints, which would be valuable for downstream tasks such as image-to-image translation or 3D reconstruction.
We thank Angjoo Kanazawa, Yossi Gandelsman, Norman Mu, David Chan, and Jitendra Malik for helpful discussions. This work was supported in part by DoD including DARPA's SemaFor, PTG and/or LwLL programs, as well as BAIR's industrial alliance programs.