Fuse2Match: Training-Free Fusion of Flow, Diffusion, and Contrastive Models for Zero-Shot Semantic Matching
Authors: Jing Zuo
Jiaqi Wang
Yonggang Qi
Yi-Zhe Song
Venue: NeurIPS 2025 poster ()
Keywords: Dense Correspondence
Diffusion models
Zero-Shot Semantic Matching
Primary area: applications
TL;DR: We explore a training-free confidence-aware fusion framework that adaptively combines features from SD3, SD, and DINO for zero-shot semantic matching.

Abstract: Recent work shows that features from Stable Diffusion (SD) and contrastively pretrained models like DINO can be directly used for zero-shot semantic correspondence via naive feature concatenation. In this paper, we explore the stronger potential of Stable Diffusion 3 (SD3), a rectified flow-based model with a multimodal transformer backbone (MM-DiT). We show that semantic signals in SD3 are scattered across multiple timesteps and transformer layers, and propose a multi-level fusion scheme to extract discriminative features. Moreover, we identify that naive fusion across models suffers from inconsistent distributions, thus leading to suboptimal performance. To address this, we propose a simple yet effective confidence-aware feature fusion strategy that re-weights each model’s contribution based on prediction confidence scores derived from their matching uncertainties. Notably, this fusion approach is not only training-free but also enables per-pixel adaptive integration of heterogeneous features. The resulting representation, Fuse2Match, significantly outperforms strong baselines on SPair-71k, PF-Pascal, and PSC6K, validating the benefit of combining SD3, SD, and DINO through our proposed confidence-aware feature fusion. Code is available at https://github.com/panda7777777/fuse2match

S Fuse2Match: Training-Free Fusion of Flow, Diffusion, and Contrastive Models for Zero-Shot Semantic Matching
Jing Zuo Jiaqi Wang' Yonggang Qi'
Yi-Zhe Song?
SSchoo of Artificial Intelligence, Beijing University of Posts and Telecommunications SketchXX CVSSP, University of Surrey
zuoj0723@gmail.. com, {wang- jiaqi, aeiiaii,iyee qiyg) @bupt. qiyg)&bupt. edu. en, y. y.ongesuerrr ac. uk
Corresponding author
Abstract
Recent work shows that features from Stable Diffusion (SD) and contrastively pretrained models like DINO can be directly used for zero-shot semantic corre- spondence via naive feature concatenation. In this paper, we explore the stronger potential of Stable Diffusion 3 (SD3), a rectified flow-based model with multi- modal transformer backbone (MM-DiT), We show that semantic signals in SD3 are scattered across multiple timesteps and transformer layers, and propose multi-level fusion scheme to extract discriminative features. Moreover, we iden- tify that naive fusion across models suffers from inconsistent distributions, thus leading to suboptimal performance. To address this, we propose a simple yet effective confidence-aware feature fusion strategy that re-weights each model's contribution based on prediction confidence scores derived from their matching uncertainties. Notably, this fusion approach is not only training-free but also enables per-pixel adaptive integration of heterogencous features. The resulting representation, Futh2Match, significantly outperforms strong baselines on SPair- 71k, PF-Pascal, and PSC6K, validating the benefit of combining SD3, SD, and DINO through our proposed confidence-awant feature fusion. Code available https://githnn com/ wwwm///aaa77777/fuuuuo
Introduction
Dense correspondence prediction is fundamental computer vision task that aims to establish pixel-level matches between images. Conventional methods often rely on large amount of task-specific data and fine-tuning. In contrast, humans can naturally align semantically related regions without supervision. This has motivated growing attention in zero-shot semantic dense correspondence, which serves as critical benchmark for assessing the generalizability of large-scale foundation models.
Recent advancements large-scale diffusion models for text-to-image generation have sparked interest in their potential beyond generative tasks. In particular, works such as DIFT (27 and
SD+DINO 333 demonstrate that features extracted from diffusion models (i.e., Stable Diffusion 25))
either alone in combination with contrastive models (i.e., DINO (3), enable strong performance in zero-shot semantic correspondence, without any fine-tuning, However, existing research typically adopts UNet-based Stable Diffusion and has not fully explored the latest architectures or the question of how to effectively combine multiple pretrained models. Specifically, we ask: (i) Can the latest flow-based diffusion models, such SD3, bring further benefits? (ii) More importantly, can we
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

better integrate diverse pretrained models with complementary strengths for dense matching, in training-free way?
A common strategy for model fusion is knowledge distillation 11 where a student network learns to mimic multiple teacher models, such as AM-RADIO (24.. While effective, distillation requires costly training, and often fails to preserve the diversity of the source models. In this work, we introduce a confidence-aware feature fusion framework that is training-freee features from different off-the-shelf models are fused at inference time, based on their estimated matching confidence. This allows us to leverage complementary inductive biases across different model families (e.g., contrastive B vs. generative pertaining (25] VIT 5 vs. UNet (26 vs. DiT (1 backbones), in a lightweight and effective manner.
Importantly, our work is not a simple extension of prior methods like SD+DINO or DIFT. While we draw inspiration from these studies, we tackle much more challenging and unexplored scenario: the use of rectified flow models with transformer-based backhones (i.e., MM-DiT in SD3) for zero-shot correspondence. Due to substantial differences in architecture and training objectives (e.g.. flow- matching vs. denoising), extracting meaningful features from SD3 is highly non-trivial. Following previous wisdom naively select single timestep and transformer block from SD3 even leads to inferior performance compared to UNet-based SD, despite SD3 being a more powerful model overall. Through systematic exploration of hyperparameters such as timesteps, transformer blocks, and attention facets (e.g., key, query, value, token), we reveal that semantic information is dispersed across multiple blocks and facets in SD3. Thus, we explored multi-level fusion across timesteps and layers, enabling SD3 to achieve performance on par with DIFT (SD) and DINOv2.
In addition, we investigate whether incorporating features from earlier UNet-based Stable Diffusion (SD) models and contrastively pretrained ViTs (i.c., DINO) can further enhance the SD3 representa- tions. However, our empirical analysis reveals substantial inconsistencies across these foundation models, stemming from their heterogeneous architectures and training objectives. As a result, naive feature concatenation leads suboptimal performance due to confficting feature distributions. To address this, we propose a simple yet effective confidence-awart fusion strategy that adaptively re- re-weights each model's contribution in per-pixel fashion, guided by matching uncertainty-baat based confidence score. This design differs from SD+DINO, which performs feature fusion by treating each model's contribution equally across all pixels. Moreover, we show that our fusion mechanism is generalizable across different model combinations, demonstrating its flexibility and robustness.
Our main contributions are as follows: (i) To our best knowledge, it is the first attempt leverage SD3, a large-scale text-to-image generation pretrained model built on rectified flow, for zero-shot semantic correspondence without any task-specific fine-tuning. (ii) A simple yet effective method is devised for pixel-wise adaptive fusion of features from diverse foundation models, leveraging their complementary knowledge arising from distinct architectures and pretext training tasks. As result, robust and generalizable features for dense matching across categories, viewpoints, and domains are obtained. (iii) Substantial performance gains have been achieved across various benchmarks (SPair-71k, PF-Pascal, and PSC6K datasets), validating the effectiveness.
2 Related Work
Zero-Shot Semantic Correspondence. The goal of semantic correspondence is identify match- ing object locations that share the same semantics, regardless of differences in categories (27), viewpoints 18), deformations [8] or domains Many recent works 2 27 33 focus on the zero-shot setting, which requires no task-specific fine-tuning. This is achieved by directly utilizing the features obtained from foundation models, which are shown to be effective in learning implicit seman- tic correspondence. DINOv1 I was first explored by Amir et al. [2) as a feature extractor with localized semantic information, demonstrating its applicability various zero-shot tasks such as segmentation and correspondence. DINOv2 20). trained on larger and nigher-quality datasets, has been shown to achieve superior performance in zero-shot correspondence tasks [33]. Text-to-image diffusion models, such as DIFT, have been leveraged for semantic correspondence lately since stable diffusion (SD) features have a strong sense of spatial layout, thereby facilitating pert-level correspondence. SD+DINO demonstrates that SD and DINO features have different properties that are complementary. By simply concatenating these two features, significant performance improvements can be achieved. In this work, we show that the latest flowd model, i.c., SD3, trained by text-to-image generation

---

can tackle sero semantic correspondence. In addition, we observe significant performance gains over SD+DINO by fusing features of SD3, SD, and DINO.
Representation Learning with Difflusion Models. Diffusion models have demonstrated excep- tional performance in image and video generation 77] By generative modeling trained using large- scale rext-photo pairs, diffusion models have been shown to be an effective representation learner exhibiting robust generalization to novel scenarios [30]. It has been observed that the intermediate layers of the UNet decoder can capture rich semantic information across various granularity levels by utilizing different timesteps and layer depths during denoising [2 292. Consequently, representations derived from diffusion models have proven to be highly effective in downstream tasks involving image segmentation (28 classification [1 and semantic correspondence [7 33 In this work, we
focus on directly utilizing features obtained from multimodal diffusion transformers (MM-DiT) for
zero-shot semantic correspondence, which is less studied.
Preliminary on Stable Diffusion
We begin by introducing background on Stable Diffusion 3 (SD3), which serves the feature extractor for semantic correspondence prediction.
Rectified Flow Model. One of the key properties of Stable Diffusion (SD3) 7 is that it is built
upon Rectified Flow (RF) [[[45]] which connects data po and noise distribution p) =O I)
via linear paths, hence can mitigate error accumulation during sampling. Coupled with the novel Logit-Normal sampler, SD3 outperforms early versions of diffusion models, such as SDXI. 22 The
forward process is defined as:
2t = (1 - t) + te,
(1)
where zo is the data sample, (0, I) is Gaussian noise, and t [0, 1] denotes the timestep. The backward process reverses this trajectory using a velocity field ve (Zx, t), governed by the ordinary differential equation (ODE):
dz: di
= ve (2t,
The model is trained via the Conditional Flow Matching (CFM) 112.. which minimizes the discrepancy between the learned velocity field ue (2t, t) and the target velocity field (z(le)):
LCrM B.,,, (2 t) u. (=
MM-DiT Backbone. Unlike the commonly adopted UNet architecture in the early versions of Stable Diffusion, another key difference in SD3 is the employed diffusion transformer backbone DIT. Notably, DiT backbone exhibits better scalability and facilitates bidirectional information exchange between text and image modalities, resulting in fine-grained, semantically rich image representations.
4 Methodology
In this section, we first introduce how we extract pixel-level matching features from Stable Diffusion 3 and utilize them to perform the nearest search for semantic correspondence. We then introduce our method to fuse features from diverse large-scale pretrained models.
4.1 Problem Setup
semantic correspondence aims to find paired locations that share similar semantics between two images. We simply extract dense pixel features in both photos then match them. Given a feature map F; for image I, the feature vector for pixel at location p is (p). Then we can match the most relevant pixel from image I2 given pixel p1 from image I, which can be formally defined as:
Following the common practice 2
27 33
arg min ((F (2)))
(4)
where d denotes the cosine distance used to mature the feature vector similarity.

---

4.2 Extracting Matching Features From MM-DiT
Forward Process. Similar the Latent Diffusion Models (LDM) 25 given an image Ty Rhxuxx, we first project it into the latent space through a pretrained VAE, i.., 2n = (zo), where 20 "xxx> represents the latent image, and & denotes the pretrained encoder. Gaussian noise is then added to corrupt at noise level determined by timestep / using Eq. Next, MM-DiT blocks are used to predict noise at each timestep.
Feature Extraction via Backward Process. Given the corrupted latent image 24, we can extract features from the MM-DiT blocks of the pretrained SD3 during the backward denois- ing process. Apart from 2; and timestep t, text prompt embedding is fed into the net- work as well for image denoising. Follow- ing DIFT, we simply adopt "A photo of a fobject category). as the text prompt, which encoded into an embedding E R..xD. using pretrained, frozen text encoders. To construct the image embedding, we first add positional embed- ding and patchify the latent image žt. then project this patch encoding and the text embedding to the same dimension. The text and image embeddings are concatenated as a sequence, which is then pro- cessed by the MM-DiT blocks to predict the in- jected noise at timestep t. Notably, each modality is handled by its own independent transformer, yet the input sequence contains both text and image tokens. This allows each representation operate within its own domain while still exchanging knowledge from the other modality.
photo dog"
Layernorm & Mod Linear
Layemarn Mad Linear
Sou porreaseetee Oor katis
Vc
Sie hor BMitteme
Q
Attention
NN
NN
Figure I: We extract SD3 features from intermedi- ate MM-DiT blocks and empirically identify that the query (Qz) and value (V²) from the image-branch at- tention blocks serve as the most effective features for semantic matching.
Feature Facets Selection. While also built upon the ViT architecture, SD3 provides broader set of feature facets by allowing cross- mrosll interactions. As shown in Figure I it feeds both a text prompt embedding (i.e., e,) and image embedding (i.e., x;)) into the transformer blocks, enabling the emergence self-attention, cross-attention, and joint attention, in addition to the conventional feature facets used in DINO- VIT 2 such tokens, queries, keys, and values. In this context, self-attention refers to interactions among tokens within the same modality (e.g.. image-to-imee to-image or text-to-textt text), while cross-attention facilitates information exchange between different modalities (e.g., text-to-image). Joint attention denotes the process of concatenating text and image embeddings prior to attention computation, enabling unified reasoning over fused representations. To identify the most effective representations for semantic matching, we perform greedy search to select the optimal feature facets for establishing correspondences between two images. Note that once the hyperparameters are determined via greedy search on the training set, they remain fixed during inference, introducing no additional computational overhead.
4.3 Confidence-aware Feature Fusion
We hypothesize that different large-scale pretrained models, including DINOv2, UNet-based Stable Diffusion (SD), and DiT-based Rectified Flow Transformers (SD3), all learn implicit and complemen- tary knowledge of semantic correspondence due to their diverse network architectures and pretraining tasks. SD and SD3 perform latent space generative pertaining, while DINOv2 is obtained via con- trastive learning. As recent evidence shows that simply fusing features from SD and DINO yields surprisingly strong performance in zero-shot semantic correspondence. Therefore, are interested in exploring whether combining SD3 with other pretrained models would provide additional benefits for zero-shot semantic correspondence.
straightforward approach to directly concatenate SD3 features with those from other pretrained models, such as DINO and SD. However, due to fundamental differences in network architectures and training objectives, foundation models naturally exhibit diverse feature characteristics when performing semantic matching, which inevitably leads to inconsistencies across models. To address

---

this, we propose confidence-aweer aware feature fusion strategy that effectively integrates the most reliable predictions from each model. Intuitively, we place greater trust in models that exhibit lower uncertainty in their semantic correspondences between two images. In practice, given a pair of image feature maps, i.e., F1 E TR x XC and F2 E TR X/>2xw2xx extracted from a pretrained model, feature vector of query pixel P1 at location (I. v) in Fi is defined as x,,y E TRC Then similarity vector M E h2 representing the distances between P1 and all locations in P2 can be obtained by:
M Ux.y flatten (F2)²
where flatten F2) is a flattened matrix derived from feature map F2. containing hy pixel vectors. Consequently, we define the prediction certainty by measuring the gap between the maximum and average value in the similarity vector M:
max M avg M 3.
(6)
Intuitively, a sharp activation in M suggests high confidence in matching, whereas a more evenly distributed similarity across M implies lower confidence. To this end, fuse features from different pretrained models by concatenating their re-weighted features with the normalized confidence scores. This ensures that models with higher matching confidence contribute more prominently, with the fusion operating at the pixel level to allow for highly localized and adaptive feature integration.
5 Experiments
5.1 Experimental Settings
Datasets. We evaluate the MM-DiT features and its fused features on three challenging benchmarks, namely SPair- P1air-71K KK [19. PF-Pascal [10), and PSC6K [16]. Following the tuning-free setting, we directly test pixel matcher the corresponding test sets. SPair-71K and PF-Pascal are datasets constructed for photo-to-pe -hoto matching. SPair-7IK contains 12.714 test image pairs across 18 categories, featuring wide variety of object keypoints and viewpoints, which pose significant challenges for dense matching. PF-Pascal, on the other hand, has 299 test image pairs distributed across 20 categories. PSC6K photo-sketch paired dataset consisting of 6,250 photo-sketch pairs across 125 categories, with 150,000 keypoint annotation pairs. Establishing semantic correspondence between photos and sketches particularly challenging since free-hand sketches are inherently abstract, lacking the texture and color cues present in RGB images, which are crucial for accurate dense correspondence prediction. Evaluation on the PSC6K dataset is significant it verifies whether pretrained models trained with photorealistic images can generalize to vastly different domains, such as human-draww sketches
Competitors. We compare our method against the current ttate-of-the-art semantic correspondence approaches, specifically focusing on those in the zero-shot setting, where point matching is performed directly based on their distance in the feature maps of image pairs, without additional task-specific fine-tuning, involving SD+DINO [33), DIFT [27), OpenCLIP(23) DINOv2 [0), DINOv [3], and SigLip(32]. To gain insights about their comparison results over conventional unsupervised and super- vised methods with task-specific fine-tuning, we also include ISCVGSM ASIC CATs++[4),
DHF 17
and SD4Match 13 for comparison.
we adopt the Percentage of Correct Keypoints (PCK) metric to assess the accuracy of keypoint correspondences. Essentially, predicted keypoint is deemed correct if it is sufficiently close to the corresponding ground truth keypoint, i.e., the distance is within max(h, w), where 7 is positive scalar in the range [0. 1] controlling the strictness of the criterion, and (h, w) denote the dimensions of the object bounding boxes in the SPair-7Ik dataset, or the image resolutions the PF-Pascal and PSC6K dataset.
Evaluation Metric. Following 27 33
Implementation details. We implement our approach using PyTorch. We choose the Stable Diffusion 3.5 large model for extracting MM-DiT features. During feature fusion, the Stable Diffusion 2.1 and DINO v2 large models are adopted following the experimental setups in 27 33 In accordance with the common practice, we retain the original image resolutions used by each foundation model during training, i.c.. 768 x 768 for SD, 480 480 for DINOv1, 840 x 840 for
5

---

Table 1: Comparison results (PCK @0. 1) on SPair-71k dataset. S, U, zs", and ZSM represent supervised. unsupervised, zero-shot using single model, and zero-shot with multiple models, respectively. Our proposed SD3-based approaches are color-coded in gray. The best and second-best scores are marked in bold and underlined, respectively.
Method
Acro
Bike
Bird
Bost
Bottee
Bus
Car
Ca
Chuir
Cow
Dog
Home
Mottor
Person
Plant
Shoep
Thain
CAIs++
60.5
46.9
82,5
41.6
56.8
64.9
50.4
72.8
29.
75.8
65.4
62.5
50.9
56.1
54.8
48.2
80.9
74,9
59.9
DHF
74.0
61.0
87.2
40.7
47.8
70.0
74,4
80.9
38.5
76.1
00.9
6688
66.6
70.3
58.0
54.3
87.4
60.3
64.9
SD4Match
75.3
67.4
85.7
62.9
36.7
73.0
70.9
U
ISCVGSM
74.8
64.5
87.1
45.6
52.7
77.8
21.4
82.4
47.7
82.0 67.3 73.9
67.6
60.0
49.9
69.88
78.5
59.1
67.3
ASIC
57.9
25.2
68.1
24.7
35.
28.4
30.9
54.8
21.6
45.0 47.2 39.9
26.2
48.8
14.5
24.5
49.0
24.6
36.9
28*
Sigl.p
51.5
41.7
74.4 25.1
34.1
61.1
47.5
18.7
DINO41
41.3
27.5
67.2
23.1
45.3
39.7
35.8
65.5
25.8 55.1 45.9
31.8
27.5
44.5
34.1
36.7
45.7
42,0
42.5
DINO<2
75.5
67.5
86.4
47,9
46.2
58.0
53.9
71.0
37.8
09.5
67.8
09.6
67.0
65.6
56.9
31.6
OpenCLIP
53.2
33.4
69.4
28.0
33.3
41.0
41.8
55.8
23.3
47.0
43.9
44.1
43.5
55.1
23.6
31.7
21.8
DIFT (SD2)
63.2
54.3
80.3
34.4
46.0
52.2
48.5
77.1
38.9
76.6
55.2
61.4
53.2
45.8
57.6
56.3
70.9
63.5
59.4
63.7
50.9
79.4 37.0
51.9
58.1
52.8 73.9
70.7
52.2
55.1
57.2
54.6
63.8
SD-DINO
73.4
64.0 86.5 39.8 53.0 55.2 $3.9 78.4 46.3 77.6 65.0
69.8
63.2
50.1
58.9
68.01
66.7
54.5
63.3
SD3+DINOV2
75,5
66.4 86.8 15.1 58.5 61.5 56.2 77.0 48.5 76.6 61.9
69.2
69.7
69.0
57.2
66.3
61.5
62,9
65.9
SD4SD34DINOv2
74.3
63.8
86.5 43.0 56.1
80.2
81.3
65.3
68.1
67.9
Fuse2Match
75.1 66.2 87.7
43,4 55.3 61.4 57.1
81.1 51.9 82.1
67.1
71.6
67.9
68.8
66.2
69.6
75.4
69.9
68.8
DINOv2, and 1024 x 1024 for SD3. Given an input image in resolution w, the extracted features using those models will be upsampled by bilinear interpolation into an image size feature map, resulting feature map size (h x w x 9728) for SD3, (h x w x 1280) for SD, and (/ w x 1024) for DINOv2. All those features are normalized before concatenation, resulting in the combined feature map of size (h x w x 12,032). substantially larger than those used in DIFT and SD+DINO. To reduce the computational cost, apply matrix factorization, which significantly improves efficiency and allows inference on a single A100 40G GPU.
5.2 Semantic Correspondence Results
Following the same setting in DIFT and SD+DINO (i.e., SD-2.1+DINOW more precisely), the metric PCK @011 (i.e., setting T 0.1) is reported on the SPair-71k dataset. Results are shown in Table D we can see that: (i) SD3 (Ours) can achieve slightly better results (PCK @0.1 59.8) among competitors using features from single pretrained models, where OpenCLIP, SigL.i. and DINOv1/v2 demonstrate relatively weaker performance compared to DIFT and SD3, suggesting the advantages of generative diffusion and flow models over discriminatively contrastively trained models. (ii) Feature fusion using UNet-based diffusion and DINOv2 (i.e., SD+DINO) achieves PCK@0.1 score of 63.3, clearly outperforming all single pretrained models. Compared to SD+DINO, feature fusion using SD3 and DINO yields consistently improved performance (63.3 vs. 65.9). highlighting the superior compatibility and representational capacity of SD3 in conjunction with DINO. Interestingly, this advantage emerges despite SD3 and SD- 2.1 achieving comparable results when used individually, suggesting that the improvements stem not from raw performance differences, but from enhanced complementarity in the fused representation. (iii) The highest performance is obtained by integrating SD3, stable diffusion (SD), and DINO via our proposed confidence-and between SD3 (DiT-based) and earlier SD variants (UNet-based), which enables more complementary feature representations for semantic matching. Notably, the fusion scheme guided by estimated confidence scores (i.e., Fuse2Match) consistently outperforms naive feature concatenation (i.e., SD SD3 + DINOv2), underscoring the effectiveness of our adaptive weighting mechanism.
Quantitative Results on SPair-71k.
fusion strategy. We attribute this gain to the architectural heterogeneity

---

SD3
SD2
DINOv2
20
24
12
23
Figure 2: Visualization of feature maps from SD3, SD, and DINO via PCA and K-Means clustering.
'Conflict' and 'conplementary' effects among foundation models. We evaluate (i) whether different single foundation models can achieve correct matching the same locations, and (ii) whether the combination of foundation models can achieve improved per-point matching results compared to single models, to explicitly study their conflict and complementary effects. The idea is that the matched locations would be dif- ferent when conflicts among foundation models
Table 2: Semantic matching statistics to reflect the "con- flict" and "complementary" effects among SD3, SD, and DINOv2.
None (1984)
One (1974)
Two (2557)
All (5719)
Fusion
Naive Con. Ours
6.4%
40.9%
88.3%
99.5%
12.7%
55.7%
90.9%
99.5 %
exist, and improved matching results are expected due to the complementary effects offered by feature combination, either via naive concatenation or our proposed confidence-aware feature fusion. Specifically, given the test set of SPair-71k, 1k, which contains 12,234 image pairs, we evaluate whether each foundation model (i.e., SD3, SD, DINOv2) can achieve correct matching for least half of the annotations within each image pair. As shown in Table 2 we present the number of image pairs that could be correctly matched by different numbers of foundation models. For example, none of the models succeeded in matching 1,984 image pairs (denoted as None), while only one model succeeded another 1, 1,974 image pairs, and so forth. is evident that models often conflict with each other, as some models fail at specific matchings while others succeed, indicating contradictory predictions. Regarding the complementary effects among foundation models, we can see that, for example, simply concatenating features from all models improves correct matches from none to 6.42%, which further increases to 12.75% when adopting our confidence-awaan aware feature fusion approach, validating their complementary effects. Similar trends can be observed even though saturation occurs in easier cases, such as the 5,719 image pairs where all single foundation models succeed. In addition, we visualize the feature maps from SD3, SD, and DINO using PCA and K-Means clustering. The resulting clusters shown in Figure demonstrate that each model captures the data with distinct structural patterns.
Table 3: Comparisons under various conditions.
viewpoint diff.
scale diff.
truncation diff.
occlusion diff'.
easy
medi.
hard
easy
medi.
hard
none
source
target
both
none
source
target
both
SD3
61.2
46.7
43.9
56.5
54.0
47.7
57.1
51.4
50.4
48.8
56.7
49.5
49.2
50.0
SD2
61.9
42.9
36.9
55.8
52.1
43.8
56.1
48.8
48.4
45.5
54.9
48.2
48.0
19.4
DINOv2
55.7
58.1
57.4
58.5
$7.0
50.6
59.4
54.4
52.9
50.1
57.2
55.5
55.3
58.0)
Fuse2Match
70.7
56.8
50.7
65.5
63.5
58.6
65.7
62.1
61.2
58.6
65.1
61.0
61.0
61.7
Model behavior in various cases. Apart from evaluating the conflict and complementary effects among different foundation models, we provide additional experimental results on how these models behave in various cases. Specifically, we follow SPair-71k, which splits the image pairs in the test set into diverse variations, resulting different levels (i.e., easy, medium, and hard) of viewpoint and scale, and various situations regarding truncation and occlusion. As shown in Table (PCK
7

---

Image - Image
Sketch-sPhoto
Sketch
Source Imoge
DINOV2
OIT
503
503-Sb+0ING
Figure 3: Comparison of semantic correspondence results using features from different pretrained models on the SPair-7Ik and PSC6K dataset. Key points in images color-coded. The incorrect matches highlighted cross mark corresponding color. Zoom for the best view.
we can find out that (i) features from DiT-based SD3 alone do not exhibit clear advantage over UNet-based SID, but SD3 indeed performs better diffieult cases viewpoint and scale. (ii) DINOv2 outperforms both SD and SD3 generally, especially for hard cases, suggesting that DINOv2 offers accurate matching. while SD and SD3 excel at establishing rough correspondences. (iii) Utilizing our proposed feature fusion approach can significantly improve the performance of relying on any single model, verifying the efficacy of the confidence weight used in the process.
Quantitative results on PF-Pascal and PSC6K. As shown in Ta- ble 4 similar conclusions can be drawn that our integrated features of SD3+SD+DINO achieve the best performance across various matching thresholds on both the PF-Pascal and PSC6K datasets. Surprisingly, the per- formance of utilizing SD3 features alone
Table 4: PF-PASCAL and PSC6K results.
PF.PSCAL
PSC6K
Method
0.05
0.1
0.15
0.05
0.1
0.15
DINOvI
55.6
74.2
38.98
78.76
DINOv2
63.4
82.6
35.91
61.02
74.45
DIFT
82.2
88.1
35.80
58.67
72.37
SD3
72.0
86.5
92.2
41.86
67.50
81.02
SD+DINO
68.1
85.7
91.5
surpass that of the com- bined SD+DINO features on the PF- Pascal dataset, highlighting its strong capability for matching similar semantic locations between images. On the PSC6K dataset, we observe that SD3 features still outperform DIFT when matching sketch-photo pairs. Interestingly, while SD3 alone does not surpass the contrastively trained DINOv1/v2 models, the combination of features significantly improves the matching accuracy. This implies their complementary properties, which generalize effectively to image pairs with significant domain shifts using the integrated features.
Fuse2Match
78.6
90.6
94.6
49.62
74.31
86.28
To better understand the performance of different pretrained models in differ- ent matching scenarios, i.e., photo-to-photte sketch-to-photo, and sketch-to-sketchh we present some qualitative comparison results in Figure 3 We can find that our proposed fused feature is capable of
Qualitative Results.

---

Sketch-PPhoto
Skeech-- Sketch-+Skete Sketch
F
Figure 4: Visualization of semantic correspondence and matching heat maps by our fused SD3+SD+DINO
feature on the PSDC6K dataset.
Acto.
PCK-00.1
PCK-20.1
60
Bike
Bird
40
Boal
Bottle
All
2 5 666
(a) Block Number
(b) TimeStep
Facets
Figure 5: Ablation study results (PCK@0.1) on (a) block number, (b) timestep, and (c) facets of MM-DiT blocks evaluated on the first five categories of SPair-71k dataset.
more accurately identifying pixel pairs with similar semantics across different objects and varying viewpoints. Our approach demonstrates strong robustness to geometric transformations, such as aligning the left and right legs of a chair in the second row the front and back wheels of bicycle in the last row. In addition, our fused SD3+SD+DINO feature archives highly precise matching even between sketch-photo pairs two sketches, demonstrating the robustness and generalizability of the feature. Moreover, we visualize more sketch2photo and sketch2sketch matching results coupled with attention heat maps on the PSC6K dataset. As shown in Figure 4 reasonable attention heat maps can be obtained, indicating the domain-agnostic semantic knowledge contained in the fused feature.
5.3 Ablation Study
We ablate several key hyperparameter choices of the feature extraction network (MM-DiT) on subset of the SPair-71k training split, which is constructed by randomly selecting 10 photo pairs across all the 18 categories.
Timestep and block number. Similar to DIFT, the selection of timestep crucial for feature extraction. DIFT shows that large timestep and earlier UNet upsampling layer yield more semantically
block number
features. On the contrary, for extracting feature from SD3, our ablation results in Figure 55 a) and (b) reveal that moderate timestep t around 521 and later MM-DiT blocks around 24th yield better performances. In practice, we select three different timesteps, i.c., f 521, 621, 721]. and conduct feature extraction from two attention blocks, namely 24 and 25, which have been proven to be effective our case.
Facets MM-DIT. We apply different facets in MM-DiT blocks serving as pixel-level features for semantic matching study the impacts, involving the image tokens, queries, values, keys, self-

---

Table Performance gains after performing confidence-aware feature fusion to various model combinations on
a subset of SPair dataset.
Models
Aero.
Bike
Bird
Boat
Bottle
SDI.5+DINOv2
+0.99
+1.73
-0.60
-0.99
+1.73
+0.25
SD3+SD2
+0.25
-0.75
+0.87
+0.27
+0.48
+0.27
SD2+DINOv2
+0.26
-0.13
+0.35
+0.45
+0.12
+0.20
SD3+DINOv2
+0.84
+1.50
-0.02
+2,28
-0.07
+0.74
Token
query
key
Valuc
joint attn
self attn
cross
query
Source Point
volue
self
cross o1io
query
value
Figure 6: Visualization matching probability maps using different MM-DiT facets. Features from query and value concentrate more precisely to the desired object parts.
attention, cross-attention, and joint attention. Experimental results Figure 5 show that employing either the query value features for matching outperforms other options. In practice, we combine their strengths by concatenating query and value the feature, which produces more accurate and sharper activations, as validated in Figure 6
Generalization of confidence-aware feature fusion. We evaluate the generalization capability of our proposed confidence-aware feature fusion strategy by applying different combinations of foundation models, involving SD3, DIFT (SD2/SD1.55 .),, and DINOv2. As shown in Table 5 our proposed confidence-awart feature fusion achieves superior matching performance compared naive concatenation, demonstrating its strong generalization ability across different model combinations.
6 Conclusions
We demonstrated that SD3, transformer-basee diffusion model, provides strong zero-shot semantic correspondence performance without task-specific fine-tuning. Through systematic exploration of SD3's architecture, identified feature configurations that yield robust pixel-level matching. Moreover, we showed that SD3's global semantic features can be effectively complemented by SD and DINO's fine-grained representations. By introducing a confidence-aware fusion strategy, we adaptively integrate features from diverse models, significantly improving generalization across tasks such as photo-to-photo and sketch-to-photo matching.
Linitations. The text prompt adopted in approach is coarse-grained, which might be suboptimal and could limit the SD3 features for dense semantic matching, which demands fine-grained image understanding. Therefore. exploring ways enhance SD3 features by prompt tuning further will be an important endeavor in the future.
Acknowledgment
This work was supported by the Hainan Provincial Joint Project of Li'an International Education Innovation Pilot Zone (Grant No.624LALHO08, BUPT Kunpeng&Ascend Center of Cultivation, NSFC (Grant No.61601042), the Program for Youth Innovative Research Team of BUPT (Grant No. 2023QNTD02), and the Super Computing Platform of BUPT.
10

---

References
Michael S Albergo and Eric Vanden- Vanden-Eijnden. Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209. 15571, 2022.
[2] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features dense visual descriptors. ECCVW What is Motion For?, 2022.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf--sue vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[4] Seokju Cho, Sunghwan Hong. and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence. 45(6):7174-7194, 2022.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
[6] Johan Edstedt, Qiyu Sun, Georg Bökman, Marten Wadenbäck, and Michael Felsberg, Roma: Robust dense feature matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17790-19800, 2024.
[7] Patrick Esser, Sumith Kulal, Andreas Blattmann. Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the 41th International Conference on Machine Learning, 2024.
[8] Agrim Gupta, Jiajun Wu, Jia Deng. and Fei-Fei Li. Siamese masked autoencoders. Advances in Neural Information Processing Systems, 36:40676-40695 2023.
[9] Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishck Kar. Asic: Aligning sparse in-the-wild image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4 4134-4145, 2023.
[10] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow: Semantic correspondences from object proposals. IEEE transactions on pattern analysis and machine intelligence, 4017):1711-1255, 2017.
[11] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 02531, 2015.
[12] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives the elbo with simple data augmentation. Advances Neural Information Processing Systems, 36, 2024.
[13] Xinghui Li. Jingyi Lu, Kai Han, and Victor Adrian Prisacariu. Sd4match: Learning to prompt stable diffusion model for semantic matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 97558-27565, 2024.
[14] Yaron Lipman, Ricky T. Q Chen, Heli Ben Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. The 1thh International Conference on Learning Representations, 2023.
[15] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectificd flow. arXiv preprint arXiv:2209.0300 2022.
[16] Xuanchen Lu, Xiaolong Wang, and Judith E Fan. Learning dense correspondences between photos and sketches. In International Conference on Machine Learning, pages 22899-223 22916, 2023.
[17] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfea- tures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
[18] Octave Mariotti, Oisin Mac Aodha, and Hakan Bilen. Improving semantic correspondence with viewpoint- guided spherical maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19521-19530, 2024.
[19] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-7lk: A large-scale benchmark for semantic correspondence. arXiv prepreint 1908 10543, 2019.

---

[20] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Ya-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, pages 2835 2835-8856, 2024.
[21] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 4195-42000 2023.
[22] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:330.001952, 2023.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, 2021.
[24] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce domains into one. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490-12500, 2024.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the JEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[26] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asisstd -ntervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[27] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances Neural Information Processing Systems, 36:1363-1389, 2023.
[28] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco. Zsolt Kira, and Mar Gonzalez-France Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3554-3563, 2024.
[29] Jinglong Wang, Xiawei Li, Jing Zhang, Qingyuan Xu, Qin Zhou Qian Yu, Lu Sheng. and Dong Xu. Diffusion model is secretly a training-fiee free open vocabulary semantic segmenter. arXiv preprint arXiv:2309.022. 027709999077..... 2023.
[30] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-imaae diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1955-2966, 2023.
[31] Xingyi Yang and Xinchao Wang. Diffusion model representation learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18938- 11949, 2023.
[32] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986 2023.
[33] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Appendix & Supplementary Materials
A
Semantic Matching
In this section, further present visual results of semantic matching across different models in Figure [ We also provide heatmap visualizations in Figure D which highlight the matching quality and indicate the domain-agnostic semantic knowledge contained in the fused feature.
Source
DINOv2
DIFT
SD+DINO
SD3
SD3NSD+DINO
11cK
Figure 7: More comparison results SPair-71k datasets.
13

---

Source
DINOv2
DIFT
SD+DINO
SD3
DD3+
Figure 8: Comparison results for semantic correspondence between sketches and photos.
14

---

Source Sketch Targer Image
Source Sketch Torget
Figure 9: Heat maps of matching similarity between sketch-photo and sketch-sketch pairs.
15

---

B Instance-level Image Swapping
In this section, we further investigate the application of semantic matching instance-level image swapping. Given the dense correspondence between two images established by our pixel matcher, we can transplant an instance from source image onto the instance in a target image. The obtained new image preserves the identity of the source instance, while adapting to the pose or layout present in the target instance. Specifically, we replace the pixels of target image with the pixels in source image by nearest neighbor lookup, producing a swapped image. The swapped image is then refined using Stable Diffusion. As shown in Figure TO, the object in the swapped image resembles the pose and size of the object in target image (works well for sketch too) while the appearance comes from the source. After refinement using SD, high-quality images are obtained. We show more swap results in Figure Ind 12
Source
Target
Swap
Inversion
Figure 10:
Instance swapping with FuseeMatchfeaaures. Swap: image after pixel replacement. Inversion: image refined by SD.
16

---

Source
Target
Swap
Inversion
Figure 11: More instance swapping results between photos.
17

---

Source
Target
Swap
Inversion
Figure 12: More instance swapping results from sketch to photo.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state the main claims clearly and the experiment results support the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All necessary experimental details are discussed in this paper. We will release the code to ensure full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: All datasets used in the paper are publicly available. We will release our code with detailed instructions for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The proposed method is zero-shot and the test details is reported in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow prior work in the dense matching field that doesn't 't include statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the information in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics and respect it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See at the end of the Experiments section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The paper does not release any data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We don't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
25


Reviews:
[{"review_id": "GxjLykuRP3", "reviewer": "Reviewer_S5hJ", "summary": "The paper explores whether the recently released Stable Diffusion 3 (SD3) transformer backbone, trained with a rectified-flow objective, contains spatial cues useful for zero-shot semantic correspondence, a task previously dominated by UNet-based Stable Diffusion 2 and contrastive ViTs such as DINO. After an exhaustive greedy search that shows semantic signals are scattered across mid-noise timesteps and late MM-DiT layers, the authors build a multi-level SD3 descriptor and then fuse it with SD 2.1 and DINO v2 through a confidence-aware weighting that amplifies high-certainty pixels, defined as the gap between the maximum and mean of each similarity vector (L185-L190). This training-free ensemble, dubbed E-SD3, improves PCK@0.1 from 59.8 for SD3 alone to 68.8 on SPair-71k (L234-L244) and beats prior tuning-free baselines on PF-Pascal and PSC6K as well (L289-L299), while requiring only inference-time feature extraction.", "questions": "Could the authors clarify the wall-clock latency and peak memory when processing a pair, and whether those numbers include the VAE encoder-decoder passes demanded by SD3? Have they tried a lightweight student network or linear adapter trained to emulate the fused representation, and if so how does its accuracy-vs-speed trade-off look?\nWhy was the particular confidence metric chosen over entropy-based or temperature-scaled alternatives, and does its effectiveness persist if all three backbones are similarly uncertain? Given that the greedy search for timesteps, layers, and facets was conducted on SPair-71k training data (L318-L324), can the authors report performance on a held-out dataset unseen during that search to rule out subtle over-tuning?", "limitations": "Please see questions and weaknesses. I am not a specialist in this sub-field, so my current recommendation is tentative and could shift after reading the authors’ rebuttal.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The work is timely, being the first to repurpose SD3 for dense matching (L63-L65), and it demonstrates that transformer-based diffusion features can complement both UNet-diffusion and contrastive ViT signals, yielding clear accuracy gains without any fine-tuning.\n\nHowever, I suppose those gains arrive at a steep computational cost as a single image produces a SD3 map that must be compressed just to fit on an A100-40 GB GPU (L230-L232). Memory footprint and inference time comparison seems required. “confidence-aware fusion” sounds a hand-crafted re-weighting based on the max-minus-mean gap (L185-L190), a surrogate for saliency that closely mirrors the mutual-nearest-neighbor or peak-ratio heuristics. Apart from being the first to adaping SD3 for matching, what is the paper's main technical contribution?", "was_revised": true, "final_justification": "All concerns were adaquately addressed in the rebuttal, raising final recommendation to borderline accept.", "rebuttal": "We sincerely thank reviewer **S5hJ** for the critical and constructive feedback! Below, we provide point-by-point responses to the specific questions and concerns raised, and we hope these clarifications adequately address the issues.\n***\n**Q1: Performance gains arrive at a steep computational cost.**\n\nA1: We thank the reviewer for pointing out this important aspect. Raw SD3 features are large indeed, and we adopt a compression step not merely as a workaround but as a core design to promote alignment across heterogeneous models and reduce resource consumption. To quantify this following your suggestion, we report memory and latency comparisons in the Table bellow. With compression (we concatenate all per-pixel features and apply PCA to reduce the dimension to 1K), SD3 inference requires ~3.5 GB GPU peak memory (less than DINOv2 and SD2) and has manageable matching runtime (\\~0.03s per pair, 9x faster than the SD3 only using matrix factorization) while still maintaining competitive results (PCK@0.1 at 58.98 on SPair-71k), making it feasible for real-world, training-free zero-shot scenarios. We agree that further improvements in efficiency (e.g., via distillation) are promising directions for future work.\n\nTable: Comparison of Inference Time (Including VAE encoder) and Peak Memory Usage per Pair across Different Models\n| Model             | Feature Extraction Time (s) | Dense Matching Time (s) | Peak Memory (GB) | PCK\\@0.1|\n|------------------|-----------------------------|--------------------------|------------------|------------------|\n| SD3              | 0.530                       | 0.312                    | 29               | 59.48\n| SD3 (compressed) | 0.530                       | 0.039                    | 3.5              | 58.98\n| SD2              | 0.294                       | 0.040                    | 4.7              | - \n| DINOv2           | 0.022                       | 0.054                    | 3.8              | - \n\n**Q2: Confidence-aware fusion strategy is a surrogate for saliency that mirrors the mutual-nearest-neighbor or peak-ratio.**\n\nA2: Thank you for the inspiring observation. We conducted additional experiments using peak-ratio [a]—the ratio between the top-1 and top-2 values in the similarity vector M (Eq. 5)—as a confidence measure. While conceptually intuitive, peak-ratio yields lower performance (PCK\\@0.1: 66.94) than our proposed max-minus-mean strategy (PCK\\@0.1: 68.8). We attribute this gap to peak-ratio’s sensitivity to local noise, i.e., when multiple candidates have similar similarity scores, the ratio becomes unstable and less indicative of true confidence. In contrast, our max-minus-avg captures global contrast by penalizing non-peak similarities collectively, leading to more robust fusion.\n\nRegarding the resemblance to mutual nearest neighbor (MNN) [b], we appreciate the reviewer's perspective but currently do not see a direct conceptual link. MNN is primarily used for establishing reliable correspondences between two sets, while our strategy operates at the feature aggregation level. We would be grateful if the reviewer could elaborate on the intended analogy.\n\n[a] Ensemble Learning for Confidence Measures in Stereo Vision. CVPR, 2013.\n\n[b] Best-buddies similarity for robust template matching. CVPR, 2015.\n\n**Q3: Have you tried a lightweight student network or linear adapter trained for feature fusion?**\n\nA3: Thank you for the insightful suggestion. Our work specifically focuses on a training-free fusion strategy that directly integrates heterogeneous pre-trained features without any fine-tuning or additional modules. While training a lightweight student network or linear adapter is indeed a promising direction for future work, it falls beyond the scope of our current objective.\n\n**Q4: Why was the particular confidence metric chosen over entropy-based or temperature-scaled alternatives, and does its effectiveness persist if all three backbones are similarly uncertain?**\n\nA4: We appreciate the reviewer’s insightful question. Our confidence metric was selected due to its training-free nature and effectiveness in reflecting semantic saliency across heterogeneous feature spaces. Unlike entropy-based or temperature-scaled metrics which rely on probability distributions or require tuning, our method operates directly on normalized feature activations and scales efficiently across diverse models. We have also experimented with entropy-based confidence metric. As shown in the table below, our confidence metric yields better results. \n\nWhen all backbones are similarly uncertain, the confidence scores tend to flatten, leading to more balanced fusion weights according to eq.6. We believe this acts as a natural safeguard against overfitting to unreliable representations.\n\nTable: PCK\\@0.1 results on SPair-71k with different feature fusion strategies.\n\n| Strategy           | PCK\\@0.1 |\n|--------------------|------------------------|\n| Entropy-based            | 60.5                   |\n| Ours     | 68.8                  |\n\n**Q5: Performance on held-out datasets**\n\nA5: Sorry for the confusion. In fact, our search for fusion hyperparameters (i.e., timestep, layer, and facet) was conducted exclusively only on the SPair-71k training split, with the test split held out for final evaluation (as reported in Table 1 of the paper). Moreover, to assess the generalizability of the selected configuration, we directly applied the same hyperparameters to PF-PASCAL and PSC6K without any re-tuning. The consistent performance gains observed in Table 1 and Table 4 in the paper suggest that our method is not overfitted to the SPair-71k training set and that the selected configuration is robust across datasets."}, {"review_id": "uQDyDtGu5a", "reviewer": "Reviewer_Aycd", "summary": "This work further combines the Stable Diffusion 3, Stable Diffusion and DINO v2 called E-SD^3 for semantic correspondence. In E-SD^3, a multi-level fusion scheme  is designed to extract discriminative features and a confidence-aware feature fusion strategy to handle the issue of nconsistent distributions across models. This methos outperforms state-of-the-arts on three SC datasets.", "questions": "See weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. E-SD^3 performs well on all the datasets especially in the matching tasks of skeleton and image (PSC6K).\n\n2. The designed feature fusion strategy is effective to make full use of strong backbones.\n\nWeakness\n\n1. In table 1, best and second best should be labeled for each settings instead of labeled together.\n\n2. How do the three backbones contribute to the SC result after fusion? How is the effect of using the same fusion idea to process SD and DINO2?\n\n3. Figure 4. Why are there multiple highlighted regions for geometrically similar objects such as wheels and chair arms in matching heat maps?", "was_revised": true, "final_justification": "Thanks for the detailed responses. My concerns are solved completely so I keep the rating and tend to accept this paper.", "rebuttal": "We sincerely thank reviewer **Aycd** for the insightful feedback! The suggestions have helped us better refine the presentation and further clarify key aspects of the work.\n***\n**Q1: How do the three backbones contribute to the SC result after fusion?**\n\nA1: Thank you for the insightful question. To understand the contributions of different backbones in the fused representation, we analyzed the confidence-aware fusion weights across the test set. As shown in the table below, we empirically observed that SD3 tends to contribute more in semantically complex scenes, likely due to its robust but coarser correspondence capability. DINOv2 appears to play a larger role in structure-sensitive regions (e.g., vehicles), possibly benefiting from its precise yet noisier features. Meanwhile, SD provides relatively more stable cues in low-texture regions. While these insights are based on limited observations and preliminary statistics, they are consistent with the feature behaviors illustrated in Figure 2 and offer useful insights into the complementary strengths of each backbone. We will add more qualitative and quantatitive results accordingly in our revised version.\n\n**Table: Fusion weights analysis.** Here, *Complex* refers to scenes with multiple objects or cluttered backgrounds (e.g., ''pottedplant'' and ''person''). *Structure-sensitive* targets are man-made objects with stable geometry (e.g., ''aeroplane and ''motorbike''). *Low-texture* denotes smooth-surfaced objects lacking rich local details (e.g., ''tv-monitors'' and ''bottle'').\n\n|                      | SD3    | SD2    | DINOv2 |\n|----------------------|--------|--------|--------|\n| Complex              | **0.3511** | 0.3417 | 0.3070 |\n| Low-texture          | 0.3300 | **0.3433** | 0.3265 |\n| Structure Sensitive  | 0.3281 | 0.3242 | **0.3486** |\n\n**Q2: What is the effect of using the same fusion idea to process SD and DINO2?**\n\nA2: Thank you for the question. In fact, we have included the comparison results of fusing SD(1.5) with DINOv2 using different fusion methods (i.e., naive concatenation vs confidence-aware fusion) in Table 5, which suggests that our confidence-aware fusion strategy can be effectively generalized to other settings.\n\n**Q3: Why are there multiple highlighted regions for geometrically similar objects in Figure 4.**\n\nA3: Sorry for the confusion. In Figure 4, given a query point in the source image, we compute its feature similarity to all points in the target image, resulting in a dense attention map. This map is then color-coded by the similarity scores, where warmer colors indicate higher similarity. As such, multiple highlighted regions may appear when the model assigns moderately high attention to several geometrically similar areas (such as the wheels and chair arms you noticed). However, only the point with the highest attention score (e.g., the left chair aim which has both geometry and location correct) is selected for the final match. The purpose of the visualization is to show the entire attention distribution, rather than just the final matched point. We will further clarify this point in our revised version.\n\n**Q4: Format issue in Table 1**\n\nA4: Thank you! We will fix it accordingly."}, {"review_id": "p4pdUf1WER", "reviewer": "Reviewer_1cVT", "summary": "This paper proposes a confidence-aware feature fusion strategy that re-weights contributions of different models based on prediction confidence scores. The key advantage of this strategy lies in its training-free nature. Experimental results on three semantic matching datasets demonstrate the effectiveness of the proposed method.", "questions": "1.Can you provide an analysis explaining why the performance gain on SPair-71k is limited?\n2.How are other methods of calculating the prediction certainty? For example, entropy-based methods or variance.\n3.The study should be enhanced by incorporating comparative analyses with advanced fusion methodologies beyond naive concatenation.", "limitations": "See weeknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1.\tThe training-free architecture offers enhanced practical utility.\n2.\tThe methodology and experimental procedures are presented with notable clarity.\n3.\tThe comprehensive experimental framework conclusively demonstrates the method's effectiveness through multi-faceted validation.\nWeaknesses：\n1.\tThe proposed fusion strategy essentially extends weight adaptation upon naive concatenation, failing to introduce structural or theoretical innovations, thus constituting a marginal contribution.\n2.\tThe confidence computation in the proposed method, while computationally straightforward, and the empirical demonstration of the fusion strategy's effectiveness notwithstanding, exhibit a conspicuous absence of theoretical justification and experimental validation for the predictive certainty quantification approach.\n3.\tThe experimental section only compares the proposed method against naive concatenation, revealing marginal performance gains. More critically, it omits comparative analysis with state-of-the-art model fusion techniques.\n4.\tWhile experiments are conducted on three datasets, the method exhibits limited performance improvement on SPair-71k compared to substantial gains observed on the other two benchmarks. Crucially, the paper fails to account for this significant performance discrepancy across datasets.", "was_revised": true, "final_justification": "Most of my concerns are addressed, so I raise my rating to borderline accept.", "rebuttal": "We sincerely thank reviewer **1cVT** for the critical and constructive feedback! The following provides our detailed responses to each of the concerns raised. We hope our clarifications resolve the outstanding issues and help strengthen the paper.\n***\n**Q1: The proposed fusion strategy essentially extends weight adaptation upon naive concatenation, failing to introduce structural or theoretical innovations, thus constituting a marginal contribution.**\n\nA1: Thank you for the feedback. We respectfully disagree that our method constitutes a marginal extension of naive concatenation. While our fusion strategy may appear structurally simple, it addresses a critical but underexplored issue: the substantial feature inconsistency across foundation models due to their heterogeneous architectures and training objectives. Naive concatenation leads to suboptimal performance, as it treats all models equally regardless of their local reliability. \n\nIn contrast, our confidence-aware fusion adaptively re-weights each model’s contribution in a per-pixel fashion, guided by matching uncertainty. This design departs from fixed-weight methods (e.g., SD+DINO) and shows superior flexibility and robustness across diverse model combinations.\n\n**Q2: The confidence computation in the proposed method, while computationally straightforward, and the empirical demonstration of the fusion strategy's effectiveness notwithstanding, exhibit a conspicuous absence of theoretical justification and experimental validation for the predictive certainty quantification approach.**\n\nA2: We thank the reviewer for the insightful comments. While our confidence metric is structurally simple, it is designed as a training-free, semantics-aware proxy to reflect local matching reliability. To support its effectiveness, we conducted a correlation analysis between the confidence scores and keypoint matching errors (measured by the Euclidean distance between predicted and ground-truth target points) on 100 randomly sampled cases using SD3, SD2, and DINOv2.\n\nAs shown in the table below, we observe consistent moderate negative correlations (Pearson *r* ≈ –0.35 to –0.41, *p*-value < 1e–70), indicating that features with lower confidence scores tend to yield larger localization errors. This aligns with our design intuition that unreliable regions should be down-weighted in the fusion process. Here, Pearson *r* quantifies the linear relationship between the confidence score and matching error across individual features, while the associated *p*-value reflects the statistical significance of the correlation. We will incorporate these analyses into the revised version.\n\n**Table. Confidence–Error Correlation Analysis**\n| Feature  | Pearson *r* | *p*-value      |\n|---------|-----------|--------------|\n| SD3     | -0.3975   | 9.83e-99     |\n| SD2     | -0.4124   | 6.76e-107    |\n| DINOv2  | -0.3537   | 3.82e-77     |\n\n**Q3: The experimental section only compares the proposed method against naive concatenation, revealing marginal performance gains.**\n\nA3: Thank you for this valuable comment. Our work is primarily focused on training-free fusion of heterogeneous frozen backbones in zero-shot settings. And, it is important to note that in zero-shot scenarios with frozen encoders, even small yet consistent improvements across diverse settings are challenging to achieve. Our method shows reliable gains with negligible computational overhead, highlighting its robustness and plug-and-play usability. \n\n**Q4: More critically, it omits comparative analysis with state-of-the-art model fusion techniques.**\n\nA4: We appreciate the reviewer’s suggestion. We acknowledge the value of benchmarking against more sophisticated fusion frameworks (e.g., adaptive feature modulation [a], attention-based fusion [b], cross-modal distillation [c]). However, many of these methods rely on task-specific training or supervision, which diverges from our target: a **training-free, plug-and-play** fusion mechanism that emphasizes generality and applicability across diverse tasks and datasets. Exploring how our confidence-aware framework could be integrated with learnable fusion modules in a lightweight manner is a promising direction, and we plan to investigate this in future work.\n\n[a] Revisiting Feature Fusion for Image Classification. CVPR, 2021.\n\n[b] A Simple Framework for Contrastive Learning of Visual Representations. ICML, 2020.\n\n[c] Exploring Category-Agnostic Vision-Language Pretraining for Open-World Visual Understanding. CVPR, 2023.\n\n**Q5: While experiments are conducted on three datasets, the method exhibits limited performance improvement on SPair-71k compared to substantial gains observed on the other two benchmarks. Crucially, the paper fails to account for this significant performance discrepancy across datasets.**\n\nA5: We appreciate the reviewer for pointing out this important observation. Indeed, the performance gain on SPair-71k is less prominent compared to the other two benchmarks. We attribute this discrepancy to the inherent challenges of SPair-71k, which involves larger viewpoint changes, stronger non-rigid deformation, and noisier annotations. These factors significantly degrade the performance of all pretrained models involved (e.g., SD3, DINOv2), thus limiting the headroom for additional gains through fusion. Nevertheless, our method still consistently outperforms individual models and naive fusion strategies on SPair-71k, as reported in Table 3 in our paper. This indicates that even under such adverse conditions, our confidence-aware fusion remains effective and robust. We will clarify this point more explicitly in the revised version.\n\n**Q6: How are other methods of calculating the prediction certainty? For example, entropy-based methods or variance.**\n\nA6: Thank you for the suggestion. We agree that uncertainty estimation methods such as entropy or variance have been widely used, especially in classification tasks. In our case, however, the proposed confidence score is computed based on the feature-level consistency across multiple pretrained models, which is **more suitable for training-free fusion and semantic matching tasks where no explicit distribution is available**. \n\nIn contrast, entropy- or variance-based methods typically require either class probability distributions or Bayesian/posterior modeling, which is non-trivial in our zero-shot or training-free setting. Nevertheless, to evaluate their applicability, we conducted preliminary comparisons with entropy- and variance-based certainty scores adapted to our context. As shown in the table below, our method consistently outperforms these alternatives, highlighting its effectiveness in guiding fusion without requiring distributional assumptions.\n\n| Feature Fusion Strategy           | PCK\\@0.1 |\n|--------------------|------------------------|\n| Entropy-based            | 60.5                   |\n| Variance-based           | 66.8                   |\n| Ours     | 68.8                  |"}, {"review_id": "v0eY9GVArW", "reviewer": "Reviewer_k3NR", "summary": "This paper explores Stable Diffusion 3 (SD3) for the task of semantic correspondence. It makes two major contributions: First, it investigates the optimal configurations of the timestep in the diffusion process, the layer of the MM-DiT blocks, and the facets of the attention process to achieve the best zero-shot semantic matching. Second, it proposes a weighted combination method to combine features from SD3, SD, and DINOv2 and construct a hyperfeature to further improve the accuracy of zero-shot semantic matching. The paper compares the proposed method against the previous zero-shot methods and achieves state-of-the-art performance in this category.", "questions": "I believe this paper contains sufficient details, and I do not require anything substantial from the author. Maybe including the result of supervised methods using SD as the foundation in Table 1 helps improve the fairness of the Table.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. This paper investigates SD3 for the semantic correspondence task, which is a novel attempt in this field. \n2. Most of the claims are well supported by experimental evidence. The effectiveness of combining features from SD3, SD, and DINOv2 is demonstrated in Table 1, while the benefit of the proposed Confidence-aware Feature Fusion module is shown in the final two rows of the same table. Moreover, the feature fusion’s generalizability is strengthened by further experiments in Table 5, where it is applied to various model combinations.\n3. Thorough details are given in the experiments, including the image size used for each method and hardware requirements for the method, which are useful to show the whole picture of the method. \n4. The paper is well-written and easy to follow.\n\n**Weakness**\n1. While the use of SD3 is novel, the idea of combining features from SD and DINO is not. This concept was first introduced in SD+DINO [32], which this paper builds upon. As such, the core conceptual novelty is somewhat limited. That said, extending an existing research path in a meaningful way remains a valid and valuable contribution. I do not consider this overlap a significant weakness.\n2. The paper selects two supervised methods, CATs++ and DHF, for comparison against these lines of work. However, using these works to create an impression that the proposed method can achieve a higher number in benchmarks over the supervised method is not fair. First, CATs++ employs ResNet101 as the backbone. ResNet101 is a much weaker general feature extractor than SD, so using CATs++ to highlight the performance of the proposed method is not fair. Second, although DHF employs SD as the backbone, the paper selectively ignores other supervised SD-based methods with much stronger performance, such as SD4Match [1] and GeoAware-SC [2]. Despite this unfairness, the proposed method is a zero-shot semantic matching method, and it should not be compared with supervised methods in the first place, as the supervised methods will most likely outperform zero-shot methods. Therefore, this flaw does not negatively impact the contribution of the paper.\n\n**Conclusion**\nThis paper offers a meaningful contribution to the semantic correspondence literature by leveraging SD3 and proposing a confidence-aware fusion strategy. Despite some concerns about conceptual novelty and comparative evaluation, the experimental support is solid, and the direction is promising. I lean toward a borderline accept recommendation.\n\n[1] Li, Xinghui, Jingyi Lu, Kai Han, and Victor Adrian Prisacariu. \"Sd4match: Learning to prompt stable diffusion model for semantic matching.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27558-27568. 2024. \n\n[2] Zhang, Junyi, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. \"Telling left from right: Identifying geometry-aware semantic correspondence.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3076-3085. 2024.", "was_revised": true, "final_justification": "In my opinion, this paper makes a solid contribution to the semantic correspondence field by investigating SD3 feature in this task. The author's rebuttal addressed the most of my concerns. Therefore, I would like to maintain my original rating (borderline accept)", "rebuttal": "We sincerely thank reviewer **k3NR** for the positive and encouraging feedback! We are glad that the contributions of our work are well recognized and valued.\n***\n**Q1: To compare with supervised methods (CATs++ and DHF) to create an impression that the proposed method can achieve a higher number in benchmarks over the supervised method is not fair.**\n\nA1: We thank the reviewer for this constructive comment and fully agree that zero-shot methods should not be directly compared with supervised counterparts. Our primary goal in including CATs++ and DHF was to contextualize our method's performance, not to claim superiority over supervised methods. To avoid misleading readers, we will remove the comparisons to CATs++ and DHF in Table 1.\n\n**Q2: Maybe including the result of supervised methods using SD as the foundation in Table 1 helps improve the fairness of the Table.**\n\nA2: Thank you for the constructive suggestion. We agree that including results from recent supervised methods using SD as the foundation can improve the completeness and fairness of Table 1. In response, we will include the results of SD4Match (shown in below), a representative supervised method built upon SD, for comparison.\n\n| Method                  | Aero | Bike | Bird | Boat | Bottle | Bus  | Car  | Cat  | Chair | Cow  | Dog  | Horse | Motor | Person | Plant | Sheep | Train | TV   | All  |\n|-------------------------|------|------|------|------|--------|------|------|------|--------|------|------|-------|-------|--------|-------|--------|--------|------|------|\n| SD4Match                | 75.3 | 67.4 | 85.7 | 64.7 | 62.9   | 86.6 | 76.5 | 82.6 | 64.8   | 86.7 | 73.0 | 78.9  | 70.9  | 78.3   | 66.8  | 64.8   | 91.5   | 86.6 | 75.5 |\n\nAs expected, SD4Match outperforms our training-free zero-shot approach, which is reasonable given that it benefits from task-specific supervision and fine-tuning. Nonetheless, our method remains competitive in several settings despite requiring no additional training, which highlights its efficiency and flexibility. \n\n**Q3: While the use of SD3 is novel, the idea of combining features from SD and DINO is not. This concept was first introduced in SD+DINO [32], which this paper builds upon. As such, the core conceptual novelty is somewhat limited. That said, extending an existing research path in a meaningful way remains a valid and valuable contribution. I do not consider this overlap a significant weakness.**\n\nA3: Thank you for the comment and kind words! While inspired by SD+DINO and DIFT, we want to emphasize that, our work goes beyond prior methods by tackling a more challenging setting: using rectified flow models with transformer backbones (SD3) for zero-shot correspondence. Due to SD3’s distinct flow-matching training and transformer architecture, feature extraction is highly non-trivial—naive choices underperform older UNet-based SD. We found that naive fusion hurts performance due to inconsistent feature distributions among SD3 and other foundation models. To address this, we introduce a confidence-aware fusion strategy that adaptively re-weights features per-pixel based on uncertainty, differing from uniform fusion in SD+DINO."}]

Author rebuttal:

Meta-review: This paper addresses the task of semantic correspondences and investigates the quality of flow-matching features (in the form of SD3) for the task. The paper finds that the features that are useful for the task are spread across timesteps and designs a fusion technique to gather these features. The paper then creates an ensemble of feature sets and demonstrates strong performance on semantic correspondence benchmarks.

The reviews were positive for this paper (4 borderline accept). The reviewers commended the performance of the approach as well as the clean analysis and presentation. The main drawback was seen as the relatively weak technical novelty of the paper. I generally agree with all these points but also agree with the authors that there is value in understanding the difference in features of different styles of generative models.

I advocate for acceptance.
Decision: Accept (poster)