Diffusion Model for Dense Matching
Authors: Jisu Nam
Gyuseong Lee
Sunwoo Kim
Hyeonsu Kim
Hyoungwon Cho
Seyeon Kim
Seungryong Kim
Venue: ICLR 2024 oral (2024)
Keywords: Diffusion Models
Visual Correspondence
Primary area: generative models

Abstract: The objective for establishing dense correspondence between paired images con- sists of two terms: a data term and a prior term. While conventional techniques focused on defining hand-designed prior terms, which are difficult to formulate, re- cent approaches have focused on learning the data term with deep neural networks without explicitly modeling the prior, assuming that the model itself has the capacity to learn an optimal prior from a large-scale dataset. The performance improvement was obvious, however, they often fail to address inherent ambiguities of matching, such as textureless regions, repetitive patterns, large displacements, or noises. To address this, we propose DiffMatch, a novel conditional diffusion-based framework designed to explicitly model both the data and prior terms for dense matching. This is accomplished by leveraging a conditional denoising diffusion model that explic- itly takes matching cost and injects the prior within generative process. However, limited input resolution of the diffusion model is a major hindrance. We address this with a cascaded pipeline, starting with a low-resolution model, followed by a super-resolution model that successively upsamples and incorporates finer details to the matching field. Our experimental results demonstrate significant performance improvements of our method over existing approaches, and the ablation studies validate our design choices along with the effectiveness of each component. Code and pretrained weights are available at https://ku-cvlab.github.io/DiffMatch.

Published as a conference paper at ICLR 2024
DIFFUSION MODEL FOR DENSE MATCHING
Jisu Nam
Gyuseong Lee" Sunwoo Kim 3+
Hyeonsu Kim
Hyoungwon Cho
Seyeon Kim
Seungryong Kim
Korea University
LG Electronics
3K
ABSTRACT
The objective for establishing dense correspondence between paired images sists of two terms: a data term and a prior term. While conventional techniques focused on defining hand-designed prior terms, which are difficult to formulate, re- cent approaches have focused on learning the data term with deep neural networks without explicitly modeling the prior, assuming that the model itself has the capacity to learn an optimal prior from a large-scale dataset. The performance improvement was obvious, however, they often fail address inherent ambiguities of matching, such as textureless regions, repetitive patterns, large displacements, noises. To address this, we propose DiffMatch, a novel conditional diffusion-basee framework designed to explicitly model both the data and prior terms for dense matching. This is accomplished by leveraging a conditional denoising diffusion model that explic- itly takes matching cost and injects the prior within generative process. However, limited input resolution of the diffusion model a major hindrance. We address this with cascaded pipeline, starting with low-resolution model, followed by super-resolution model that successively upsamples and incorporates finer details to the matching field. Our experimental results demonstrate significant performance improvements of our method over existing approaches, and the ablation studies validate our design choices along with the effectiveness each component. Code and pretrained weights are available at www.ai----aaaaaiibbb...... ho/.ol//D//////////
(a) Source
(b) Target
(c) GLU-Net
(d) GOCor
(e) DiffMatch
(f) Ground-truth
Figure 1: Visualizing the effectiveness of the proposed Diff'atch: (a) source images, (b) target images, and warped source images using estimated correspondences by (c-d) state-of-the-arr ap- proaches (Truong et al. (2000bla). (c) our DiffMatch, and (f) ground-truth. Compared to previous methods (Truong al. 2020b that discriminatively estimate correspondences, our diffusion- generative framework effectively learns the matching field manifold, resulting in better estimating correspondences particularly textureless regions, repetitive patterns, and large displacements.
INTRODUCTION
Establishing pixel-wise correspondences between pairs of images has been one of the crucial prob- lems, as it supports a wide range of applications, including structure from motion (SfM) (Schonberger & Frahm 2016), simultaneous localization and mapping (SLAM) -hurre & Bailey) 2006
"Work done while at Korea University.
1 "Correspnnding author.

---

Published as a conference paper at ICLR 2024
Bailey & Durrant- Whyte 2006), image editing (Barnes et al. 2009
Cheng et al. 2010 Zhang
et al. 2020). and video analy ysis (Hu et al. 2018 Lai & Xie 2019). In contrast to sparse correspon-
dence (Calonder al. 2010; Lowe 2004:] Sarlin et al.] 2020) which detects and matches only
sparse set of key points, dense correspondence (Pérez et al. 2013) Rocco et al. 2017 Kim et al. 2018 Cho et al. 2021) aims to match all the points between input images.
In the probabilistic interpretation, the objective for dense correspondence can be defined with a data term, measuring matching evidence between source and target features, and a prior term, encoding prior knowledge of correspondence. Traditional methods (Pérez et al. 2013; Drulea & Nedevschi
2011 Werlberger et al., 2010; Lhuillier & Quan 2000; Liu al. 2010 Ham al. (2016) explicitly incorporated hand-designed prior terms to achieve smoother correspondence, such as total variation (TV) or image discontinuity-awarr smoothness. However, the formulation of the hand-crafted prior term is notoriously challenging and may vary depending on the specific dense correspondence tasks,
such as geometric matching (Liu et al. 2010 Duchenne et al., 2011 Kim et al.) 2013) or optical
flow (Weinzaepfel et al. 2013 Revaud al. 2015).
Unlike them, recent approaches (Kim et al. 201 Sun et al., 2018 Rocco et al. 2017 2020 Truong
et al. 2020b, MMin & Cho |021] Kim et a 2018 Jiang et al.]|2111_. (021]2221]]] | Cho et 2001 2022) have focused on solely learning the data term with deep neural networks. However, despite demonstrating certain performance improvements, these methods still struggle with effectively addressing the inherent ambiguities encountered dense correspondence, including challenges posed by textureless regions, repetitive patterns, large displacements, noises. We argue that it is because they concentrate on maximizing the likelihood, which corresponds to learning the data term only, and do not explicitly consider the matching prior. This limits their ability learn ideal matching field manifold, and leads to poor generalization.
On the other hand, diffusion models (Ho et al. 2020 Song et al.) 2020a Song & Ermon 2019 Song
et al.] (2020b) have recently demonstrated a powerful capability for learning posterior distribution and have achieved considerable success in the field of generative models (Karras et al. 2020). Building on these advancements, recent studies (Rombach et al., 2022; Seo et al. 2023 Saharia et al. 2022a
Lugmayr et al. 2022) have focused on controllable image synthesis by leveraging external conditions.
Moreover, these advances in diffusion models have also led to successful applications in numerous
discriminative tasks, such as depth estimation (Saxena et al. 2023b Kim et al. 2022 Duan et al.
2023), object detection (Chen et al. 2022a), segmentation (Gu et al.
2022
Giannone al. 2022
and human pose estimation (Holmquist & Wandt 2022).
Inspired by the recent success of the diffusion model (Ho et al.) 2020 Song et al. 2020aa Song &
Ermon| 2019; Song et al., 2020b), we introduce DiffMatch, a conditional diffusion-based framework designed to explicitly model the matching field distribution within diffusion process.
Unlike existing discriminative learning- beadinng-based methods (Kim et al., 2017a| Jiang et al. 2021 Rocco
et al., 2017 2020 TTedd & Deng 2020) that focus solely maximizing the likelihood, DiftMatch aims to learn the posterior distribution of dense correspondence. Specifically, this is achieved by conditional denoising diffusion model designed learn how generate correspondence field given feature descriptors as conditions. However, limited input resolution of the diffusion model significant hindrance. To address this, we adopt cascaded diffusion pipeline, starting with low-resolution diffusion model, and then transitioning to super-resolution diffusion model that successively upsamples the matching field and incorporates higher-resolution details.
We evaluate the effectiveness of DiffMatch using several standard benchmarks (Baintas et al. 2017] Schops et al. 2017), and show the robustness our model with the corrupted datasets (Hendrycks) & Dietterich, 2019: Balntas et al., 2017 Schops et al. 2017). We also conduct extensive ablation studies to val vali our design choices and explore the effectiveness of each component.
2 RELATED WORK
Dense correspondence. Traditional methods for dense correspondence (Horn & Schunck
198]
Lucas & Kanade| 1981) relied on send-designed matching priors. Several techniques (Sun et al.
2010:
Brox & Malik 22010 Liu al., 2010 Taniai et al.) 2016 Kim et al.) [017b]] Ham al.
2016 Kim et al. 2013) introduced optimization methods, such as SIFT Flow (Liu et al. (010).
which designed smoothness and small displacement priors, and DCTM (Kim et al. 2017b). which
2

---

Published as a conference paper at ICLR 2024
introduced discontinuity- aawar prior term. However, manually designing the prior term is difficult.
To address this, recent approaches (Dosovitskiy et al.] 2015]] Rocco et al. 2017] Shen et al. 2019
Mclekhov et al.) 2019: Ranjan &
Teed & Deng 2020 Sun et al. 2018] Truong et al.
2020b 2021 Jiang et al. 2021) have shifted to learning paradigm, formulating an objective function
to solely maximize likelihood. This assumes that an optimal matching prior be learned from large-scale dataset. DGC-Net (Melekhov et al. 2019) and GLU-Net (Truong et al.]
2020b) proposed 202T) employed
coarse-to-fine framework using feature pyramid, while COTR GJang al.,
transformer-baseee network. GOCor (Truong et al. (020a)) developed a differentiable matching module learn spatial priors, addressing matching ambiguities. PDC-Net+ (Truon et al. (023) presented dense matching using a probabilistic model, estimating a flow field paired with confidence map. DKM (Edstedt et al., 2023) introduced a kernel regression global matcher to find accurate global matches and their certainty.
Diffusion models. Diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020) have been exten-
sively researched due to their powerful generation capability. The Denoising Diffusion Probabilistic Models (DDPM) (Ho et al. 2020) proposed a diffusion model in which the forward and reverse processes exhibit the Markovian property. The Denoising Diffusion Implicit Models (DDIM) (Song et al. 2020a) accelerated DDPM by replacing the original diffusion process with non-Markovian chains to enhance the sampling speed. Building upon these advancements, conditional diffusion models that leverage auxiliary conditions for controlled image synthesis have emerged. Palette (Sa-
haria et al., 2022a) proposed a general framework for image-to-imago translation by concatenating the source image as an additional condition. Similarly, InstructPix2Pix (Brooks et al., (2023) trains a conditional diffusion model using a paired image and text instruction, specifically tailored for instruction- based image editing. On the other hand, several studies (Ho et al. 2022, Saharia et al.
2022b Ryu & Ye 2022 Balaji et al. (022) have turned their attention to resolution enhancement, as the Cascaded Diffusion Model (Ho et al.) 2022) adopts cascaded pipeline to progressively interpolate the resolution of synthesized images using the diffusion denoising process.
Diffusion model for discriminative tasks. Recently, the remarkable performance of the diffusion model has been extended to solve discriminative tasks, including image segmentation (Chen et al.
2022b Gu et al. 2022; Ji et al. 2023), depth estimation (Saxena al. 2023b Kim et al. 2022 Duan et al. 20 et al. (2023). object detection (Chen et al.) 2002a), and pose estimation Tevet et al.
2022 Holmquist& & Wandt 2022). These approaches have demonstrated noticeable performance
improvement using diffusion models. Our method represents the first application of the diffusion model the dense correspondence task.
3 PRELIMINARIES
Probabilistic interpretation of dense correspondence. Let us denote a pair of images, i.e., source and target, as Tare and Tug that represent visually or semantically similar images, and feature descriptors extracted from Isre and Jeg. as Dare and Digt, respectively. The objective of dense correspondence is to find correspondence field F that is defined at each pixel i, which warps Larc towards Lig. such that I (1) ~ liro(i + F())) or (1) Dare + F(i)).
This objective can be formulated within probabilistic interpretation (Simoncelli et al. 1991 Sun ctt al. 2008 Ham et al.) 2016 Kim et al. 2017b). where we seek to find F that maximizes the posterior probability of the correspondence field given pair of feature descriptors Dero and Digt, ie.. p( (F) FDrre. D. According to Bayes' theorem Joyce 2003), the posterior can be decomposed such that p(F]Darc Digt) x p( Dere, Degt [F) p(F). ). To find the matching field F* that maximizes the posterior, we can use the maximum a posteriori (MAP) approach (Greig et al. 1989):
F" argmax p( Dere, Dig) = argmax (( Deve. DigilF) -(F)
argmax (logg(DDe D(Drrs Degol logg( ((F)
data term
prior term
In this probabilistic interpretation, the first term, referred to data term, represents the matching evidence between feature descriptors Dare and Degt- and the second term, referred to as prior term, encodes prior knowledge of the matching field F.
3

---

Published as a conference paper at ICLR 2024
X(T
Backbone
Feature
U-Net
Global Cost
Computation
Local Cost
Computation
src
sre
Feature
Back Hackbone
Argmax
Finit Ft.
Pt-1
Degt
Pinit
Local Cost and Initial Correspondence Computation
Conditional Denoising Diffusion Model
Figure Overall network architecture of DiftMatch. Given source and target images, our conditional diffusion based network estimates the dense correspondence between the two images. We leverage two conditions: the initial correspondence Finit and the local matching cost C², which finds long-range matching and embeds local pixel-wise interactions, respectively.
Conditional diffusion models. The diffusion model is type of generative model, and can be divided into two categories: unconditional models (Sohl- SSoh--Dickstein et al. 2015 Ho et al., 2020) and conditional models (Batzolis al.] [2021] Dhariwal & Nichol| (0211. Specifically, unconditional diffusion models learn an explicit approximation of the data distribution, denoted as p(X). On the
other hand, conditional diffusion models estimate the data distribution given certain condition K,
e.g., text prompt (Dhariwal & Nichol 2021). denoted as p(X]
In the conditional diffusion model, the data distribution approximated by recovering data sample from the Gaussian noise through an iterative denoising process. Given a sample Xo. it is transformed X2 through the forward diffusion process at time step E {T. T 1...., ... 1}, which consists of
Gaussian transition at each time step (X,, (X, == (VVI-- X, B,T). The forward diffusion process follows the pre-defined variance schedule Be such that
X, Vo + aiz, Z ~ N),,T),
where o2 = -I 3,). After training, can sample data from the learned distribution through iterative denoising with the pre-defined range time steps. called the reverse diffusion process, following the Mankovian process of DDIM (Song et al. 2020a). which is parametrized as another Gaussian transition (X, X,) == NN 1; /50 Xx, e, f), o0 (X,, t).). To this end, the diffusion network Fo t; K) predicts the denoised sample Xo,: given X,, and K. One step in the reverse diffusion process can be formulated such that
X,-1 -=1= v1 04-1 oo (x=_Va=F(Xx,t; + 0,
V1 Ci.
where o, is the covariance value of Gaussian distribution time step 4.
This iterative denoising process can be viewed as finding X argmax logg(p X K') through the relationship between the conditional sampling process of DDIM (Song al. 2020a) and conditional score-based generative models (Batzolis et al. 2021).
METHODOLOGG
4.1 MOTIVATION
Recent Cearning-based methods (Kim et al.) 2017a Sun et al.) 2018; Rocco et al.) 2017: 2020
Truong et al.) 2020b Min & Chol 2021 Kim et al. 2018 Jiang et al. 2021; Cho et al. 2021 2022) have employed deep neural networks F(.) to directly approximate the data term, i.e., argmax, log pl Dare. Digu without explicitly considering the prior term. For instance, GLU- Net (Truong et al. (020b) and GOCor (Truong et al. 2020a) construct cost volume along candidates F between source and target features Dare and Digt- and regresses the matching fields F* within
deep neural networks, which might be analogy to argmax log Dare, Degt F). In this setting,
4

---

Published as a conference paper at ICLR 2024
Figure 3: Visualization the reverse diffusion process in DiffMatch: (from left to right) source and target images, and warped source images by estimated correspondences as evolving time steps. The source image is progressively warped into the target image through an iterative denoising process.
dense correspondence FM estimated as follows:
F = Fo Degs) se argmax log D Dexe, Degs F),
(4)
F
where Fo(-) and 0 represent a feed-forward network and its parameters, respectively.
These approaches assume that the matching prior can be learned within the model architecture by leveraging the high capacity of deep networks (Truong et al.) 2020b Jiang et al.) [202]] Cho et al. 2021 2022 Min Cho 202T) y and the availability of large-scale datasets. While there exists obvious performance improvement, they typically focus the data term without explicitly considering the matching prior. This can restrict ability of the model learn the manifold of matching field and result in poor generalization.
4.2
FORMULATION
To address these limitations, for the first time, we explore conditional generative model for dense correspondence to explicitly learn both the data and prior terms. Unlike previous discriminative
learning-based approaches (Pérez et al.) 2013 Drulea & Nedevschi, 2011 Werlberger et al. 2010
Kim et al. 201 Sun al., [2018] Rocco et al. (207)), achieve this by leveraging conditional generative model that jointly learns the data and prior through optimization of the following objective
that explicitly leam argmax pP(F) Dare, Digt):
F* = Fo(Dar, Drat) se argmax p( Derc Degt)
argmax flog p( Dares Degi FF) + log (og)(F):
(5)
data term
prior term
We leverage the capacity a conditional diffusion model, which generates high-fidelity and diverse samples aligned with the given conditions, to search for accurate matching within the learned correspondence manifold.
Specifically, we define the forward diffusion process for dense correspondence as the Gaussian transition such that q (F-1) == (VVT- F,-,, B,T), where 3, is predefined variance schedule. The resulting latent variable F; can be formulated as Eq. 2
Va.oo a, ~~ N0,11.
(6)
where Fo the ground-truth correspondence. addition, the neural network Fo(.) is subsequently trained to reverse the forward diffusion process. During the reverse diffusion phase, the initial latent variable Fr iteratively denoised following the sequence FT-1, FT. Fo. using Eq. 3
F = Vol-1Fo(X t; Dare, Dege) + VI a:-1-00 of Vi (xi- VanFolFi,t Dare. Digs)) + 0,
(7)
where Fo Dare, Degt) directly predicts the denoised correspondence For, with source and target features, Dare and Digt, as conditions.
The objective of this denoising process is to find the optimal correspondence field F that satisfies argmax, y log lo pFF|Dere Dtgt). The detailed explanation of the objective function of the denoising process will be explained in Section 4.5
5

---

Published as a conference paper at ICLR 2024
4.3 NETWORK ARCHITECTURE
In this section, we discuss how to design the network architecture Fa(-). Our goal is to find accurate matching fields given feature descriptors Dere and Digt from fre and litt. respectively, as conditions. An overview of our proposed architecture is provided in Figure 2
Cost computation. Following conventional methods (Rocco et al. 2020 Truong et al. 2020a we first compute the matching cost by calculating the pairwise cosine similarity between localized deep features from the source and target images. Given image features Dare and Digt. the matching cost is constructed by taking scalar products between all locations in the feature descriptors, formulated as:
C(i.)) = Dere (i) Degt (i)
(8)
|| (i) Pigt (ill
where i E [0, harc) X [0, Warc), E hegs) x [0, Wegs), and denotes (-2 normalization.
Forming the global matching cost by computing all pairwise feature dot products is robust to long- range matching. However, it is computationally unfeasible due to its high dimensionality such that C' E Rhre X X X To alleviate this, we can build the local matching cost by narrowing down the target search region j within neighhorhood of the source location i, constrained by search radius R. Compared to the global matching cost, the local matching cost C¹ E TR. /ure ware x Rx is suitable for small displacements and, thanks its constrained search range of R. is more feasible for large spatial sizes and can be directly used as a condition for the diffusion model. Importantly the computational overhead remains minimal, with the only significant increase being R x R in the channel dimension.
Conditional denoising diffusion model. As illustrated in Figure 2 we introduce a modified U-Net architecture based on Nichol & Dhariwal [02I]. Our aim is to generate an accurate matching field that aligns with the given conditions. A direct method condition the model simply concatenating Dare and Det with the noisy flow input Ft. However, this led to suboptimal performance in our tests. Instead, we present two distinct conditions for our network: the initial correspondence and the local matching cost.
First, our model is designed to learn the residual the initially estimated correspondence, which leads to improved initialization and enhanced stability. Specifically, we calculate the initial correspondence Fint using the soft-argmax operation (Cho et al. 202I) based on the global matching cost C' between Dere and Degt. This assists the model to find long range matches. Secondly, interactions between paired images. For this, each pixel in the source image is mapped to i' in the target image through the estimated initial correspondence Finit- We then compute the local matching cost C" as an additional condition with Pinit. This local cost guides the model to focus on the neighborhood of the initial estimation, helping to find more refined matching field. With these combined, our conditioning strategies enable the model to precisely navigate the matching field manifold while preserving its generative capability. Finally, under the conditions Finit and C*, the noised matching field F, at time step / passes through the modified U-net (Nichol & Dhariwal, 2021). which comprises convolution and attention, and generates the denoised matching field Fr.o aligned with the given conditions.
integrate pixel-wise
4.4 FLOW UPSAMPLING
The inherent input resolution limitations of the diffusion model is major hindrance. Inspired by recent super-resolution diffusion models (Ho et al. 2022, Ryu & Ye 2022), we propose a cascaded pipeline tailored for flow upsampling. Our approach begins with low-resolution denoising diffusion model, followed by a super-resolution model, successively upsampling and adding fine-grained details to the matching field. To achieve this, we simply finetune the pre-trained conditional denoising diffusion model, which was trained at a coarse resolution. Specifically. instead of using Fintt from the global matching cost, opt for downsampled ground-truth flow field Finit- This simple modification effectively harnesses the power of the pretrained diffusion model for flow upsampling. The efficacy of our flow upsampling model is demonstrated in Table

---

Published as a conference paper at ICLR 2024
Quantitative evaluation on HPatches (Balntas et al. 2017) and ETH3D (Schops al. 2017) with common corruptions from ImageNet-O (Hendrycks & Dietterich, 2019). All results evaluated at corruption severity For simplicity, we denote GoCoNet-GOCor as GOCor (Truong
Table
et al. 2020a).
Dusect
Atsomitaz
Neree
C.D0
Vinttn
Weather
106
CGFF34
Drgial
1160
IP10C
Avo.
GUIUNA
Thossr
NONE
14.15
25.70
25.31
65.36
50.75
46 16
67.81
25.35
57.45
$0.85
44.31
16.94
40:51
GOCR
27.35
27:31
36.63
23.58
30.25
$7.35
88
19 KG
63.05
16 56
71 4.4
25.65
731 (1
11.43
12.33
6.
20.07
34.05
16.95
$3.03
56.82
16
18.10
DiMManch
31.00
24.21
20.14
31.96
15.55
36.16
97.22
22.40
94.74
55.66
20.21
27.22
22.62
37.17
11.63
34.86
Tiozes PONN
47:11
ETHED
POCNeT
26.60
38.56
22.00
10.03
13,00
37:93
76.18
45.06
69.50
55.90
10.00
17.46
18.06
18.09
15.63
29.50
POCNet
PIoN w. 20p0
3049
20.15
25.08
3.99
8.32
33.40
68.79
39.14
00.35
31.30
8.59
$.99
14.55
14.55
13.25
22.11
(a) Source
(b) Target
(c) GLU-Net
(d) GOCor
(c) PDCNet+
(f) DiffMatch
(g) GT
Figure 4: Qualitative results HPatches (Baintas et al. 2017) using motion blur in Hendrycks
& Dietterich (2019). The source images are warped to the target images using predicted correspon- dences.
4.5 TRAINING
In training phase, the denoising diffusion model, as illustrated in Section 4.3 learns the prior knowledge of the matching field with the initial correspondence Fini to give matching hint and the local matching cost C' to provide additional pixel-wise interactions. In other words, we redefine the network Fo FF; 1; Dare: Digt) Fo Funit C²), given that Pinit and C' are derived from Dac and Digt as described in Section 4.3 The loss function for training diffusion model is defined as
follows:
Fo FF( Fini, 1-)
L C= Pisa
Note that for the flow upsampling diffusion model, finetune the pretrained conditional denoising diffusion model with the downsampled ground-truth flow as Fiit.
4.6 INFERENCE
During the inference phase, Gaussian noise Fr is gradually denoised into a more accurate matching field Fo under the given features Dare and Digt conditions through the diffusion reverse process. To account for the stochastic nature diffusion-based models, we propose utilizing multiple hypotheses by computing the mean of the estimated multiple matching fields from multiple initializations Fr, which helps to reduce stochasticity model while improving the matching performance. Further details and analyses are available in Appendix C.2
EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
For the feature extractor backbone, we used VGG-16 (Chatfield et al. 2014) and kept all parameters frozen throughout all experiments. Our diffusion network based on (Nichol & Dhariwall 1021)
with modifications to the channel dimension. The network was implemented using PoTch (Paszke) al. 2019) and trained with the AdamW optimizer (Loshchilov Hutter, 2017) at learning rate of Te-4 for the denoising diffusion model and 3c-5 for flow upsampling model. We conducted comprehensive experiments in geometric matching for four datasets: HPatches (Baintas et al. 2017). ETH3D (Schops et al. 2017), ImageNet-C (Hendrycks & Dietterich) 2019) corrupted HPatches and ImageNet-C corrupted E ETH3D. Following (Truong et 2020b a. 2023). we trained our network
using OPED-CityScape ADE (Ignatov al. 2017] Cordts et al. 2016; Zhou al.] 2019) and
COCO (Lin et al. 2014)- augmente DPED OPtED-CitSccpe--// for evaluation on Hpatches and ETH3D,
respectively. For fair comparison, benchmark method against PDCNet (Truong al. 2021
7

---

Published as a conference paper at ICLR 2024
Table Quantitative evaluation on HPatches (Balntas et al. 2017) and ETH3D (Schops al. 2017). Lower AEPE indicates better performance. Higher scene labels or rates (e.g., V or 15) comprise more challenging images with extreme geometric deformations. The best results are highlighted in bold, and the second-best results are underlined. *; COTR (Jiang et al. (021) is examined separately since provides only confident correspondences and evaluation is limited to this subset. f: This indicates that dense evaluation is performed without zoom-in techniques and confidence thresholding for a fair comparison.
HPavees Oliginal Balnus
2017
ETHED Schops et al. 2017)
Methods
A0g.
[310-3
7800-1
ralc-)
F012-11
rate-13
IMC--13
AVE.
COTR
Dlang cçail AOLT
7.25
1,56
1.82
1.97
2.13
2.27
2.41
2.61
2.12
COTR
1imepp Dlang el.
2021
7.98
1.71
1.92
2.15
2.47
2.85
3.23
3.76
2:59
DGC-Ne1
Melschew
2010
62.01
GLU Net Thoingseen F. KMA00
1.55
12.66
27.54
32.0
52.47
25.08
1.98
2.54
3.89
4.24
5.61
7.55
10.78
5.17
GLUNN
COCOE Lroxi
20200
1.29
10.07
23.56
27.17
38.41
20.16
1.93
2.28
2.64
3.01
3.62
4.79
7,80
3.72
DMP
Hose & KimA-MMM
KMM
3.21
15.32
32.54
38.62
63.43
30.64
2.43
3.31
4.41
5.56
6.93
9.55
14.20
6.62
COTR
19.65
33.81
45.51
62.03
66.28
45.52
8.76
9.86
11.23
12.44
13.77
14.94
16,09
12.46
PDCNs
200
1.30
11.92
28.00
33.97
42.41
24.04
1.77
2.10
2,50
2.58
3.47
4.88
7.57
3.60
PDCNs-
Triing 21 2025
1.44
8.97
22.04
50.13
31.77
18.01
1.70
1.96
2.34
2.57
3.04
4.50
6.25
3.14
DirMaatch
1.85
10.33
19.18
26.38
35.96
18.84
2.08
2.30
2.59
2,94
3.29
3.86
4.54
3.12
(a) Source
(b) Target
(c) GeU-Net
(d) GOCor
(e) DiffMatch
(f) Ground -ruuh
Figure 5: Qualitative results on HPatches (Balntas et al. 2017). the source images are warped to
the target images using predicted correspondences.
and PDCNet+ (Trongg et al., 2023). both trained on the same synthetic dataset. Note that we strictly adhere to the training settings provided in their publicly available codebase. Further implementation details can be found in Appendix A
5.2 MATCHING RESULTS
Our primary aim is to develop a robust generative prior that can effectively address inherent ambigui- ties in dense correspondence, such as textureless regions, repetitive patterns, large displacements. or noises. To evaluate the robustness of the proposed diffusion--asee generative prior in challenging matching scenarios, we tested our approach against a series of common corruptions from ImageNet- C (Hendrycks & Dietterich 2019). This benchmark includes 15 types of algorithmically generated corruptions, organized into four distinct categories. Additionally, we validate our method using the standard HPatches and ETH3D datasets. Further details on the corruptions and explanations for each evaluation dataset can be found in Appendix
ImageNet-C corruptions. In real-world matching scenarios, image corruptions such as weather vari- ations or photographic distortions frequently occur. Therefore, is crucial to establish robust dense correspondence under these corrupted conditions. However, existing discriminative methods (Truong et al. 2020b a (202 2023) solely rely on the correlation layer, focusing on point-to-point feature relationships, resulting in degraded performance in harsh-corrupted settings. In contrast, our frame- work learns not only the likelihood but also the prior knowledge of the matching field formation. We evaluated the robustness of our approach against the aforementioned methods ImageNet-C corrupted scenarios (Hendrycks & Dietterich] 2019) of HPatches and ETH3D. As shown in Table [ our method exhibits outstanding performance harsh corruptions, especially in noise and weather. This is also visually evident in Figure 4 More qualitative results are available in Appendix D
HPatches. We evaluated DiffMatch on five viewpoints HPatches (Balntas al. 2017). Table summarizes the quantitative results and demonstrates that our method surpasses state-of-thh-aan discriminative learning- presented in Figure The effectiveness of our approach is evident from the quantitative results in Figure
methods (Truong et al. 2020b a 202 2023). The qualitative result is
[ This success can be attributed to the robust generative prior that learns a matching field manifold, which effectively addresses challenges faced by previous discriminative methods, such as textureless regions, repetitive patterns, large displacements or noises. More qualitative results are available in Appendix
8

---

Published as a conference paper at ICLR 2024
ETH3D. As indicated in Table our method demonstrates highly competitive performance compared to previous discriminative works (Truong et al., 2020b aj 2021] (023) on ETH3D (Schops et al. 2017h. Notably, Difffatch surpasses these prior works by a large margin, especially at interval rates of 13 and 15, which represent the most challenging settings. Additional qualitative results can be
found in Appendix D
5.3 ABLATION STUDY
Effectiveness of generative prior. We aim to validate our hypothesis that diffusion-based generative prior effective for finding more accurate matching field. To achieve this, we train our network by directly regressing the matching field. Then we compare its performance with our diffusion- -based method. As demonstrated Table our generative approach outperforms the regression- based baseline, thereby emphasiz- ing the efficacy of the generative prior in dense correspondence tasks. The effectiveness of the generative matching prior is further analyzed in Appendix
Table 3: Results via different learn- ing schemes.
HPatches AEPE
ETH3D AEPE
Learning schemes
DinMatch wio difllusion DiftMatch
23.34
3.96
18.82
Component analysis. In this ablation study, we provide a quantitative comparison between different configurations. The results are sum- marized in T (I) refers to the complete architecture of the conditional denoising dif- fusion model, as illustrated Figure [ (II) and (III) denote the conditional denoising dif-
Table 4: Ablations on components. C-ETH3D indi- cates ImageNet-C corrupted ETH3D.
ETHEDD AEPE 1
C.ETHDD
Components
AEPE
Conditional demoising diff.
3,44
(II)
(I) w/o local cost
4.26
32.07
(1I1)
w/o init Hlw
10.28
80.81
fusion model without the local cost and initial
(IV)
+ Flow upsampling dill. (DilMMrch))
3.12
26.45
flow conditioning, respectively. (IV) repre-
sents the flow upsampling diffusion model. Notably, (I) outperforms both (II) and (III), emphasizing
the effectiveness of the proposed conditioning method. The comparison between (I) and (IV) under- lines the benefits of the flow upsampling diffusion model, which has only a minor increase in training time leverages the pretrained (I) at lower resolution.
Time complexity. In this ablation study, we compare the time consumption of our model against existing works (Truong et al., [222]] 2023). As mentioned in Sec. [4.6] our method employs multiple hypotheses dur- ing inference, averaging them for the final output. Table presents the computing times using 1, 2, and samples for multiple hypotheses. Note that we employ batch processing for these multiple inputs instead processing them sequen- tially, improving time efficiency. With fixed sampling time step 5,, the time required for DiffMatch with a single input is comparable to that of previous methods (Truong) et al., 2021 2023), while ensuring comparable performance. Processing more samples leads to enhanced performance with only negligible increase in time. Additionally, this time complexity can
Table 5: Ablations on time complex- ity. C-ETH3D indicates ImageNet-C corrupted ETH3D.
C-ETH3D
Time
Method
AEPE J
[ms]
PDCNet (Troon et al 2021
29.50
112
PDCNe Trwing ara zuz4
27.11
112
DiMMarch (1 sample, stops)
27.52
112
DiffMatch (2 samples, steps)
27.41
123
DifMarch (3 samples, steps)
26.45
140
be further mitigated by decreasing the number of sampling time steps, as discussed in Appendix C.]
6 CONCLUSION
In this paper, we propose DiffMatch, a novel diffusion- framework for dense correspondence, which jointly models the likelihood and prior distribution of matching fields. This is achieved by the conditional denoising diffusion model, based on initial correspondence and local costs derived from feature descriptors. To alleviate the resolution constraint. we further propose a flow upsampling diffusion model that fine-tunes the pretrained denoising model, thereby injecting fine details into the matching field with minimal optimization. For the first time, we highlight the power of the generative prior in dense correspondence, achieving state-of-the-an performance on standard benchmarks. We further emphasize the effectiveness of our generative prior in harshly corrupted settings of the benchmarks. As a result, we demonstrate that our diffusion- generative approach outperforms discriminative approaches in addressing the inherent ambiguities present in dense correspondence.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was supported by the MSIT, Korea (IIT--2024-2000-00111119 ICT Creative Consilience Program, No.2021-0-02063. Artificial Intelligence Innovation Hub).
REFERENCES
Julien Abi-Nahed, Marie-Pierre Jolly, and Guang-Zhong Yang. Robust active shape models: A robust, generic and simple automatic segmentation tool. Medical Image Computing and Computer-Assister Intervention MMCCCI 2006: 9th International Conference, Copenhagen, Den- mark, October 1-6, 2006. Proceedings, Part II 9, pp. 1-8. Springer, 2006.
Tim Bailey and Hugh Durrant- Whyte. Simultancous localization and mapping (slam): Part ii. IEEE robotics & automation magazine, 13(3):108-117, 2006.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:221 1..iv:22/10033:: 1.01324, 2022.
Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajezyk. Hpatches: benchmark and evaluation of handcrafted and learned local descriptors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5173-5182, 2017.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
Georgios Batzolis, Jan Stanczuk, -iroal-Bibiane Schönlicb, and Christian Etmann. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:21 1..1::///1..3066, 2021.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392- 18402, 2023.
Thomas Brox and Jitendra Malik. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE transactions on pattern analysis and machine intelligence, 33(3):500-513, 2010.
Andrés Bruhn and Joachim Weickert. confidence measure for variational optic flow methods. Computational Imaging and Vision, 31:283, 2006.
Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua. Brief: Binary robust inde- pendent elementary features. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp- 778-792. Springer, 2010.
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint urXiv:1405.335 2014.
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprini 2rXiv:2211.09788 2022a.
Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David Fleet. A generalist framework for panoptic segmentation of images and videos. arXiv preprint 1rXiv:2210.06366 2022b.
Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma. Sanja Fidler, and Raquel Urtasun. Monocu- lar 3d object detection for autonomous driving. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2147-2156. 2016.
Ming-Ming Cheng, Fang-Lue Zhang, Niloy J Mitra, Xiaolei Huang, and Shi-Min Hu. Repfinder: finding approximately repeated scene elements for image editing. ACM transactions on graphics (TOG), 29(4):1-8, 2010.
10

---

Published as a conference paper at ICLR 2024
Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee. Kwanghoon Sohn, and Seungryong Kim. Cats: Cost aggregation transformers for visual correspondence. Advances in Neural Information Processing Systems, 34:9011 1-903,, 2021.
Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780-8794, 2021.
Alexey Dasovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 2758-2766, 2015.
Marius Drulea and Sergiu Nedevschi. Total variation regularization of local-global optical flow. In 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 318-323. IEEE, 2011.
Yiqun Duan, Xianda Guo, and Zheng Zhu. Diffusiondepth: Diffusion denoising approach for monocular depth estimation. arXiv preprint arXiv:2303. 05021,, 2023.
Olivier Duchenne, Armand Joulin, and Jean Ponce. A graph-matching kernel for object categorization. In 2011 International Conference on Computer Vision, pp. 1792-1799. IEEE, 2011.
Hugh Durrant- Whyte and Tim Bailey. Simultaneous localization and mapping: part IEEE robotics & automation magazine, 13(2):99-115 110, 2006.
Johan Edstedt, Ioannis Athanasiadis, Märten Wadenbäck, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17765-17775, 2023.
Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models. arXiv preprint arXiv:2205.15463, 2022.
Dorothy M Greig, Bruce T Portcous, and Allan H Scheult. Exact maximum a posteriori estimation for binary images. Journal of the Royal Statistical Sociery: Series B (Methodological), 51(2): 271-279, 1989.
Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, and Weiqiang Wang. Diffu- sioninst: Diffusion model for instance segmentation. arXiv preprint arXiv:2212.00277, 2022.
Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3475-3484, 2016.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: 6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-228.. 2022.
Karl Holmquist and Bastian Wandt. Diffpose: Multi-hypothesis human pose estimation using diffusion models. arXiv preprint arXiv:2211. 16487, 2022.

---

Published as a conference paper at ICLR 2024
Sunghwan Hong and Seungryong Kim. Deep matching prior: Test-time optimization for dense correspondence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9907-9917, 2021.
Berthold K.P. Horn and Brian Schunck. Determining optical flow. Artificial Intelligence, 17(1): 185-203, 1981.
Yuan-Ting Hu, Jia-Bin Huang, and Alexander G Schwing. Videomatch: Matching based video object segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 54-70, 2018.
Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8981-8989 2018.
Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. DsIr- quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 3277-3285, 2017.
Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. arXiv preprint 3rXiv:2303.17552 2023.
Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspon- dence transformer for matching across images. In P Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6207-6217, 2021.
James Joyce. Bayes' theorem. 2003.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 8110-8119, 2020.
Gyeongnyeon Kim, Wooseok Jang, Gyuseong Lee, Susung Hong, Junyoung Sco, and Seungryong Kim. Dag: Depth-aware guidance with denoising diffusion probabilistic models. arXiv preprint arXiv:2212.0888 0208661, 2022.
Jaechul Kim, Ce Liu, Fei Sha, and Kristen Grauman. Deformable spatial pyramid matching for fast dense correspondences. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2307-2314, 2013.
Seungryong Kim, Dongbo Min. Bumsub Ham, Sangryul Jeon. Stephen Lin, and Kwanghoon Sohn. Fess: Fully convolutional self-similarity for dense semantic correspondence. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6560-6569, 2017a.
Seungryong Kim, Dongbo Min, Stephen Lin, and Kwanghoon Sohn. Detm: Discrete-continuon
transformation matching for semantic flow. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4529 4539-4538, 2017b.
Seungryong Kim, Stephen Lin, Sang Ryul Jeon, Dongho Min, and Kwanghoon Sohn. Recurrent transformer networks for semantic correspondence. Advances in neural information processing systems, 31, 2018.
Claudia Kondermann, Daniel Kondermann, Bernd Jähne, and Christoph Garbe. An adaptive confi- dence measure for optical flows based on linear subspace projections. Pattern Recognition: 29th DAGM Symposium, Heidelberg, Germany, September 12-14, 2007. Proceedings 29, pp. 132-141. Springer, 2007.
Claudia Kondermann, Rudolf Mester, and Christoph S Garbe. A statistical confidence measure for optical flows. ECCV (3), 5304:290-301, 2008
Jan Kybic and Claudia Nieuwenhuis. Bootstrap optical flow confidence and uncertainty measure. Computer Vision and Image Understanding, 144999)4449-1466 2011.
12

---

Published as a conference paper at ICLR 2024
Zihang Lai and Weidi Xie. Self- learning for video correspondence flow. arXiv preprint arXiv:190S.00875. 2019.
Jae Yong Lee, Joseph DeGol, Victor Fragoso, and Sudipta N Sinha. Patchmatch- neighborhood consensus for semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 13153-13163, 2021.
Maxime Lhuillier and Long Quan. Robust dense matching using local and global geometric con- straints. In Proceedings 15th International Conference on Pattern Recognition. ICPR-2000, volume 1, pp. 968-972. IEEE, 2000.
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2041-2050 2018.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-1 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: applications. IEEE transactions on pattern analysis and machine intelligence, 3355)9778994, 2010.
Dense correspondence across scenes and its
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05001, 2017.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
Bruce D. Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference Artificial Intelligence- Volume 2, pp. 674-679, 1981.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Oisin Mac Aodha, Ahmad Humayun, Marc Pollefeys, and Gabriel J Brostow. measure for optical flow. IEEE transactions on pattern analysis and machine intelligence, 35(5): 1107-1120, 2012.
Learning confidence
laroslav Melekhov, Alcksei Tiulpin, Torsten Sattler, Mare Pollefeys, Esa Rahtu, and Juho Kan- nala. Dec-net: Dense geometric correspondence network. In 2019 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 1042- IEEE, 2019.
Juhong Min and Minsu Cho. Convolutional hough matching networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2940-2950, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuusion probabilistic models. In ICMI,, pp. 3171- PMLR, 2021.
David Nistér, Oleg Naroditsky, and James Bergen. Visual odometry. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pp. I-I. Icee, 2004.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Javier Sánchez Pérez, Enric Meinhardt-Llopis, and Gabriele Facciolo. Tv-ll optical flow estimation. Image Processing On Line, 2013:137-1500 2013.
13

---

Published as a conference paper at ICLR 2024
Anurag Ranjan and Michael J Black. Optical flow estimation using spatial pyramid network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4161-4170, 2017.
Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Epicflow: Edge- preserving interpolation of correspondences for optical flow. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 1164-1172 2015.
Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convolutional neural network architecture for geometric matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 61 6148-6157, 2017.
Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Nc- net: Neighbourhood consensus networks for estimating image correspondences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(2):020-1034, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. arXiv preprint Nttt:22008008864 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4743)44777777666 2022b.
Baul-Edouard Sarlin, Daniel DoTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4938 4947, 2020.
Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J Fleet. The surprising effectiveness diffusion models for optical flow and monocular depth estimation. arXiv preprint arXiv:2306.0192. 2023a.
Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J. Fleet. Monocular depth estimation using diffusion models, 2023b. URL
14816
Johannes L Schonberger and Jan-Michael Frahm. Stracture-frommmotion revisited. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4104-4113, 2016.
Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2260-3269, 2017.
Junyoung Seo, Gyuseong Lee, Seokju Cho, Jiyoung Lee, and Seungryong Kim. Midms: Matching interleaved diffusion models for exemplar based image translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2191-2199, 2023.
baase
Tianwei Shen, Lei Zhou, Zixin Luo, Yao Yao, Shiwei Li, Jiahui Zhang, Tian Fang, and Long Quan. Self-supervised learning of depth and motion under photometric inconsistency. In Proceedings of the IEEF/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Eero P Simoncelli, Edward H Adelson, and David Heeger. Probability distributions of optical flow. In CVPR, volume 91, pp. 310-315, 1991.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.
14

---

Published as a conference paper at ICLR 2024
Jascha Sohl- Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilirrum thermodynamics. In JCML, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint N.Iv::2000005002 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurlPS, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2020b.
Deqing Sun, Stefan Roth, John P Lewis, and Michael J Black. Learning optical flow. In Computer Vision-ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part III 10, pp. 83-97. Springer, 2008.
Deqing Sun, Stefan Roth, and Michael Black. Secrets of optical flow estimation and their principles. In 2010 IEEE computer society conference computer vision and pattern recognition, pp. 2432- 2439. IEEE, 2010.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz, Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 8934-894 2018. 8.
Tatsunori Taniai, Sudipta N Sinha, and Yoichi Sato. Joint recovery of dense correspondence and cosegmentation in two images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4246-4255, 2016.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. Computer Eccion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen -Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint wrXiv:2209.14916 2022.
Prune Truong, Martin Danelljan, Lue V Gool, and Radu Timofte. Gocor: Bringing globally optimized correspondence volumes into your neural network. Advances in Neural Information Processing Systems, 33:14278: 1478--14290 2020a.
Prune Truong, Martin Danelljan, and Radu Timofte. Glu-net: Global-local universal network for dense flow and correspondences. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 6258-6268 2020b.
Prune Truong, Martin Danelljan, Lue Van Gool, and Radu Timofte. Learning accurate dense correspondences and when to trust them. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5714-5724, 2021.
Prune Truong, Martin Danelljan, Radu Timofte, and Lue Van Gool. Pdc-nett: Enhanced probabilistic dense correspondence network. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5038-5047, 2017.
Anne S Wannenwetsch, Margret Keuper, and Stefan Roth. Probflow: estimation. In Proceedings of the IEEE international conference on computer vision, pp. 1173- 1182, 2017.
Joint optical flow and uncertainty
Philippe Weinzaepfel. Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid. Deepflow: Large displacement optical flow with deep matching. In Proceedings of the IEEE international conference computer vision, pp. 1385-1392, 2013.
15

---

Published as a conference paper at ICLR 2024
Manuel Werlberger, Thomas Pock, and Horst Bischof. Motion estimation with non-local total variation regularization. In 2010 IEEE Computer Society Conference on Computer Vision and Pattem Recognition, pp. 2464-24444444441 2471. IEEE, 2010.
Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar-based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5143-5153, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
16

---

Published as a conference paper at ICLR 2024
Appendix
In the following, we describe more comprehensive implementation details, additional analyses, additional experimental results, limitations, future works, and broader impacts of our work.
A MORE IMPLEMENTATION DETAILS
Our baseline code is built upon the DenseMatching repository We implemented the network in PyTorch (Paszke et al., 2019) and used the AdamW optimizer (Loshchilov & Hutter (007)). All our experiments were conducted on 6 24GB RTX 3090 GPUs. For diffusion reverse sampling, we employed the DDIM sampler (Song et al. 2020a) and set the diffusion timestep to 5 during both the training and sampling phases. We set the default number of samples for multiple hypotheses to for evaluations on ETH3D and to 3 for HPatches, respectively.
In our experiments, trained two primary models: the conditional denoising diffusion model and the flow upsampling diffusion model. For the denoising diffusion model, we train 121M modified U-Net based on (Nichol & Dhariwal 2021) with the learning rate to x 10-4 and trained the model for 130,000 iterations with a batch size of 24. For the flow upsampling diffusion model, we used learning rate of 3 x 10-5 5 and finetuned the pretrained conditional denoising diffusion model for 20,000 iterations with a batch size 2.
For the feature extraction backbone, we employed VGG-16, as described in [Truong et al. (2020b a). We resized the input images to H x W = 512 x 512 and extracted feature descriptors at Conv3-3, Conv4-3, Conv5-3, and Conv6-1 with resolutions
W We used these feature descriptors to establish both global and local matching costs. The conditional denoising diffusion model was trained at a resolution of 64, while the flow upsampling diffusion model was trained at resolution of 256 to upsample the flow field from 64 to 256.
W. H W x T6. and 32 32 respectively, 5 T6
W
B EVALUATION DATASETS
We evaluated DiffMatch on standard geometric matching benchmarks: HPatches (Baintas et al. 2017).
ETH3D (Schops et al. 2017). To further investigate the effectiveness of the diffusion generative prior, also evaluated Mathh under the harshly corrupted settings (Hendrycks & Dietterich 2019) of HPatches (Balntas et al.) (017)) and ETH3D (Schops et al., (017). Here, we provide detailed information about these datasets.
HPatches. We evaluated our method on the challenging HPatches dataset (Baintas et al. 2017). consisting of 59 image sequences with geometric transformations and significant viewpoint changes. The dataset contains images with resolutions ranging from 450 x 600 to 1, 613 x 1, 210..
ETH3D. We evaluated our framework on the ETH3D dataset (Schops et al. 2017). which consists of multi-view indoor and outdoor scenes with transformations not constrained to simple homographies. ETH3D comprises images with resolutions ranging from 480 x 752 to 514 x 955 and consists of 10 image sequences. For a fair comparison, we followed the protocol of (Truong et al., (2020b), which collects pairs of images at different intervals. We selected approximately 500 image pairs from these intervals.
Corruptions. Our primary objective to design a powerful generative prior that can effectively address the inherent ambiguities in dense correspondence tasks, including textureless regions, repeti- tive patterns, large displacements, or noises. To this end, to assess the robustness of our generative prior against more challenging scenarios, ImageNet-C (Hendrycks & Dietterich] 2019). This benchmark consists of 15 types of algorithmically generated corruptions, which are grouped into four distinct categories: noise, blur, weather, and digital. Each corruption type includes five different severity levels, resulting in a total of 75 unique corruptions. For our evaluation, we specifically focused on severity level 5 to highlight the effective- ness of our generative prior. Note that we use all scenes and rate 15 for the Imagenet-C corrupted versions of HPatches and ETH3D, respectively. In the following, offer a detailed breakdown of each corruption type.
subjected it to series of common corruptions from
Matshcat http:///wwo....ooooo//o////////n
17

---

Published as a conference paper at ICLR 2024
accdds SeTee
(a) Source
(b) Target
(c) DiffMatch
(d) Error map
(e) Variance map (l) Ground-truth
Figure 6: Uncertainty estimation. Our framework can measure the pixel-wise mean and variance of estimated matching fields by sampling from different Gaussian noises. We observe that the variance maps are formed almost the same as the error map, which shows that our variance map successfully expresses the uncertainty of dense correspondence.
Gaussian noise is specific type of random noise that typically arises in low-light conditions. Shot noise, also known as Poisson noise. is an electronic noise that originates from the inherent discreteness of light. Impulse noise, a color analogue of halt-and-pepper noise, occurs due to bit errors within an image. Defocus blur occurs when an image is out of focus, causing loss of sharpness, Frosted glass blur commonly seen on frosted glass surfaces, such as panels or windows. Motion blur arises when the camera moves rapidly, while zoom blur occurs when the camera quickly zooms towards an object. Snow, a type of precipitation, can cause visual obstruction in images. Frost, created when ice crystals form on lenses windows, obstruct the view. Fog, which conceals objects and is usually rendered using the diamond-squar algorithm, also affects visibility. Brightness is influenced by daylight intensity. Contrast depends lighting conditions and the color of an object. Elastic transformations apply stretching or contraction to small regions within an image. Pixelation arises when low-resolution images undergo upsampling. JPEG, lossy image compression format, introduces artifacts during image compression.
C ADDITIONAL ANALYSES
C.1 TRADE-OFF BETWEEN SAMPLING TIME STEPS AND ACCURACY.
Figure iillustrates the trade-off between sampling time steps and matching accuracy. As the sampling time steps increase, the matching performance progressively im- proves in our framework. After time step 5, it outperforms all other existing methods (Truong et al. 2020bb a Hui et al. 2018 Sun et al. [2188) Hong & Kim 2021), and the performance also converges. In comparison, DMP (Hong & Kim 202I), which optimizes the neural network to learn the matching prior of an image pair at test time, requires approximately 300 steps. These results highlight that Diff- Match finds shorter and better path to accurate matches in relatively fewer steps during the inference phase.
DifMatchh
LiteFlowNer
PWCNet
GOCor
steps
Figure 7: Time steps PCK.
C.2 UNCERTAINTY ESTIMATION
Interestingly. DiffMatch naturally derives the uncertainty of estimated matches by taking advantage of the inherent stochastic property of a generative model. We accomplish this by calculating the pixel-level variance in generated samples across various initializations of Gaussian noise Fr. On the other hand, it is crucial to determine when and where to trust estimated matches in dense correspondence (Truong 2001 Kondermann et al.] 2008 Mac Aodha et al.] 2012 Bruhn & Weickert, 2006) Kybic & Nieuwenhuis [2011] Wannenwetsch et 2 Ummenhofer et al.) 2017).
Earlier approaches (Kondermann al. 2007
2008 Mac Aodha al. 2012) relied post-hoc
18

---

Published as a conference paper at ICLR 2024
Table Quantitative evaluation on HPatches (Balntas et al. 2017) and ETH3D (Schops al. 2017) with common corruptions from ImageNet-CC (Hendrycks & Dietterich, 2019). All results are evaluated at corruption severity 5. For simplicity, we denote raw correlation volume and GLU- Net-GOCor as Raw corr. and GOCor (Truong et al. 2020a), respectively. We additionally report matching performance of the raw correlation volume to demonstrate the effect of our proposed generative matching prior.
Dutect
Alaonitaz
Geass
Nouse Stoc
Erpulic
Defocos
Class
Bix
Modio
Zook:
Saow
Frost
Weether
Fis
Beilnt
Coctrat
Exsts
Digitll
Fiee
JPBC
Avz.
104.0
94.25
244.3
227.8
254.4
222.1
19F.5
H6500
GLU
Trguss
34:55
32.60
34.18
25.74
25:31
63.25
99.25
46.16
66.50
47.81
25.3
57.45
32.85
44.31
26.94
42.31
HPockes
GOCC
23.35
27.31
36.3
23.56
20.25
51:35
89.25
00.55
36.98
21.44
25.65
28.48
33.62
22:20
36,00
FOCNer
31.00
INMMaad
31.00
26.21
20.14
31.96
19.55
36.06
97.22
57.40
50.74
35.06
20.21
27.22
27.43
33.33
21.63
$4.86
GLUUN
19:30
27.51
20.11
14.15
13.15
15.90
73.25
47.11
57.25
15.89
36.52
19.11
23.25
31.65
ETHED
GUcce
21.4
23.56
23.94
04
11:39
10.85
10.98
10000
33.33
73:36
18.18
45.38
00.00
19.1 11.11
10.06
18.35
15.55
15:43
15.30
28.96
niMMacch
25:11
23.56
34,1
8.62
5.48
16.47
27.62
41.48
64.82
25.08
5.13
15.12
12.86
13:37
14.85
26,45
(a) Source
(b) Target
(c) Raw corr. (d) GLU-Net
(c) GOCor
(f) PDCNet+ (g) DiffMatch
Figure 8: Visualizing the effectiveness of the proposed generative matching prior. The input images are corrupted by fog and snow corruptions (top and bottom, respectively). Compared to raw correlation and previous methods (Trong et al.) 2020b a that focus solely on point-to-point feature relationships, our approach yields more natural and precise matching results by effectively learning the matching field manifold.
techniques to assess the reliability of models, while more recent model-inherent approaches (Truong
et al. 2021: Bruhn & Weickeri [2006) Kybic & Nieuwenhuis] 2011: Wannenwetsch et al. 2017 Ummenhofer et al. (2017) have developed frameworks specifically designed for uncertainty estimation.
The trustworthiness of this uncertainty is showcased in Figure 6 We found direct correspondence
between highly erroneous locations and high-variance locations, emphasizing the potential to interpret the variance uncertainty. We believe this provides promising opportunities for applications demanding high reliability, such as medical imaging (Abi-Nahed et al.] 2006) and autonomous driving (Nistér al. 2004 Chen et al. 2016).
C.3 THE EFFECTIVENESS OF GENERATIVE MATCHING PRIOR
Diff'Match effectively learns the matching manifold and finds natural and precise matches. In contrast, the raw correlation volume, which is computed by dense scalar products between the source and target descriptors, fails to find accurate point-to-point feature relationships in inherent ambiguities in dense correspondence, including repetitive patterns, textureless regions, large displacements, or noises. To highlight the effectiveness of our generative matching prior, we compare the matching performance evaluated by raw correlation and our method under harshly corrupted settings in Table and Figure
The corruptions introduced by (Hendrycks Dietterich 2019) contain the inherent ambiguities in dense correspondence. For instance, snow and frost corruptions obstruct the image pairs by creating repetitive patterns, while fog and brightness corruptions form homogeneous regions. Under these conditions, raw correlation volume fails to find precise point-to-point feature relationships. Conversely, our method effectively finds natural and exact matches within the learned matching manifold, even under severely corrupted conditions. These results highlight the efficacy of our generative prior, which learns both the likelihood and the matching prior, thereby finding the natural matching field even under extreme corruption.
As carlier methods (Pérez et al. 2013) Drulea & Nedevschi 2011 Werlberger et al, 2010 Lhuillier
& Quan 2000 Liu et al.) 2010 Ham et al. 2016) design hand-crafted prior term smoothness
19

---

Published as a conference paper at ICLR 2024
constraint, assume that the smoothness of the flow field is included in this prior knowledge of the matching field. Based on this understanding, we reinterpret Table and Figure | showing the comparison between the results of raw correlation and learning-base baaedd based methods (Truong al. 2020b at 2023 2021 Previous learning-basee approaches predict the matching field with raw correlation between image pair as condition. We observe that despite the absence of an explicit prior term in these methods, the qualitative results from them exhibit notably smoother results compared to raw correlation. This difference serves indicative evidence that the neural network architecture may implicitly learn the matching prior with a sarge-scale dataset.
However, it important note that the concept of prior extends beyond mere smoothness. This broader understanding underlines the importance of explicitly learning both the data and prior terms simultaneously, as demonstrated in our performance.
C.4
COMPARISON WITH DIFFUSION-BASEI DENSE PREDICTION MODELS
Previous works (Ji et al. 2023 Gu et al. 2022 Saxena et al. 2023b
Duan et al. 2023 Saxena et al. 2023a), applying diffusion model for
Table 7: Results via differ- ent conditioning schemes.
dense prediction, such as semantic segmentation (Ji et al. 2023 Gu
2022), or monocular depth estimation (Ji 2023 Duan al.
2023 Saxena et al. 2023a), use single RGB image or its feature descriptor as a condition to predict specific dense predictions, such as segmentation or depth map. aligned with the input RGB image. concurrent study (Saxena et al. (023a) has applied a diffusion model to predict optical flow, concatenating feature descriptors from both source and target images as input conditions. However, it is notable that this model is limited to scenarios involving small displacements, typical in optical flow tasks, which differ from the main focus our study. In contrast, our objective to predict dense correspondence between two RGB images, source Isre and target Jugt, in more challenging scenarios such as image pairs containing textureless regions, repetitive patterns, large displacements, or noise. To achieve this, introduce novel conditioning method which leverages a local cost volume C² and initial correspondence Fint between two images as conditions, containing the pixel-wise interaction between the given images and the initial guess of dense correspondence, respectively.
ETH3D
Learning schemes
AEPE ~
Feature concat.
106.83 3.12
Dift'atch
To validate the effectiveness of our architecture design, we further train our model using only feature descriptors from source and target, D sre and Digt. conditions. This could be similar architecture design to DDP (Ji et al., (023) and DDVM (Saxena et al.) [023a)), which only condition we present quantitative results compare
the feature descriptors from input RGB images. Table | different conditioning methods and observe that the results with our conditioning method significantly outperform those using two feature descriptors. We believe that the observed results are attributed to the considerable architectural design choice, specifically tailored for dense correspondence.
D ADDITIONAL RESULTS
D.1 MORE QUALITATIVE COMPARISON ON HPATCHES AND ETH3D
We provide a more detailed comparison between our method and other kate-of-the-at methods on HPatches (BaIntas et al. 2017) in Figure and ETH3D (Schops et al. 2017) in Figure I0
D.2 MORE QUALITATIVE COMPARISON IN CORRUPTED SETTINGS
We also present qualitative comparison on corrupted HPatches (Balntas et al. 2017) and ETH3D (Schops et al. 2017) in Figure II and Figure I2 respectively.
D.3 MEGADEPTH
To further evaluate the generalizability of our method, expanded our evaluation to include the MegaDepth dataset (Li & Snavely. 2018), known for its extensive collection of image pairs exhibiting extreme variations in viewpoint and appearance. Following the procedures used in PDC-Net+ (Truong et al. 2023), we tested our method on 1,600 images.
20

---

Published as a conference paper at ICLR 2024
The quantitative results, presented in T demonstrate that our approach surpasses PDC-Net+ in performance on the MegaDepth dataset, thereby highlighting the potential for generalizability of our method.
Table 8: Results on MegaDepth
MegaDepth AEPE 1
Methods
PDC-Net+ [Tuong et al. 2021
63.97 59.73
DiffMarch
E LIMITATIONS AND FUTURE WORK
To the best of our knowledge, we are the first to formulate the dense correspondence task using a generative approach. Through various experiments, have demonstrated the significance of learning the manifold of matching fields in dense correspondence. However, our method exhibits slightly lower performance ETH3D (Schops al. 2017) during intervals with small displacements. We believe this is attributed to the imput resolution of our method. Although we introduced the flow upsampling diffusion model, our resolution still remains lower compared to prior works (Truong et al.) 2020b a 2021 2023). We conjecture that this limitation could be addressed by adopting higher resolution and by utilizing inference techniques specifically aimed at detailed dense correspondence, such as zoom-in (Jiang et al., (021) and patch-match techniques (Barnes et al., 2009) Lee et al.) 2.2I). In future work, we aim to enhance the matching performance by leveraging feature extractors more vancee than VGG-16 (Simonyan & Zisserman 2014). Moreover, we plan to improve architectural designs, increase resolution, and incorporate advanced inference techniques to more accurately capture matches.
F BROADER IMPACT
Dense correspondence applications have diverse uses, including simultaneous localization and mapping (Durrant Whyte & Bailey, 2006 Bailey & Durrant- Whyte 2006), structure from motion ( (Schonberger & Frahm 2016), image editing (Barnes al. Cheng al. 2010 Zhang et al. 2020), and video analysis (Hu et al. 2018 Lai & Xie 2019). Although there is no inherent misuse of dense correspondence, it can be misused in image editing produce doctored images of real people. Such misuse of our techniques can lead to societal problems. We strongly discourage the use of our work for disseminating false information tarnishing reputations.
21

---

Published as a conference paper at ICLR 2024
(a) Source
(b) Target
(c) GLU-Net
GOCor
(c) DifiMatch
(f) Ground
Figure 9: Qualitative results on HPatches (Schops et al. 2017 The source images are warped to
target images using predicted correspondences.
(a) Source
(b) Target
(c) GLU-Net
(d) GOCor
(c) DiftMatch
Figure 10: Qualitative results ETH3D Schops al. 2017). The source images warped to the target images using predicted correspondences.
22

---

Published as a conference paper at ICLR 2024
Gaussian
Noise
Shot Noise
Impulse Noise
Defocus
Glass
Motion
Blur
Zoom Blur
Fog
Bright
Contrast
Transform
Pixelate
JPEG Compression
Source
(b) Target
(e) GLU-Net
(d) GOCor
DiMMatch
(i) GT
Figure 11: Qualitative results on HPatches (Schops et al. Dietterich (2019). The source images are warped to the target images using predicted correspon- dences.
2017 using corruptions in Hendrycks
23

---

Published as a conference paper at ICLR 2024
Gaussian
Noise
Noise
Impulse Noise
Defocus Blur
Glass
Motion
Blur
Zoom Blur
Snow
Frost
Fog
Bright
Contrast
Elastic Transform
Pixelate
JPEG
Compression
(a)
Targel
(c) GLU-Net
(d) GoCor
DiflMatch
Figure 12: Qualitative results on ETH3D (Schops et al. 2017 using corruptions in Hendryeks & Dietterich (2019). The source images are warped to the target images using predicted correspon- dences.
24


Reviews:
[{"review_id": "VYoO8USKGL", "reviewer": "Reviewer_8Vji", "summary": "The paper presents DiffMatch, a novel framework for dense matching that explicitly models both the data and prior terms. DiffMatch leverages a conditional denoising diffusion model to address inherent ambiguities of matching, resulting in significant performance improvements over existing techniques. The paper argues that recent approaches have focused on learning the data term with deep neural networks without explicitly modeling the prior, but these approaches often fail to address inherent ambiguities of matching. DiffMatch addresses these issues by explicitly modeling both the data and prior terms using a diffusion model, which is trained to denoise the input image conditioned on the output of the matching network. The paper provides experimental results demonstrating the effectiveness of DiffMatch on several benchmark datasets, achieving state-of-the-art performance in terms of accuracy and efficiency.", "questions": "1. It is common for dense matching models to also test on various tasks such as optical flow estimation (KITTI) and two-view geometry estimation (ScanNet / YFCC100M). Are there specific reason your model cannot achieve these similar tasks?\n2. Adding more discussions / baselines to some diffusion-enabled dense prediction networks would further strengthen my recommendation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper presents a novel framework for dense matching, DiffMatch, and shows significant performance improvements over existing techniques. I believe this is one of the first work to apply diffusion model to solve dense correspondence (flow estimation) tasks and the results are very encouraging. The proposed approach tries to address inherent ambiguities of matching, such as textureless regions, repetitive patterns, large displacements, or noises. The approach also seems to be efficient and scalable, making it suitable for real-world applications. Overall, the paper's contributions are significant in advancing the field of dense matching.\n\nThe paper is well-written and clearly demonstrates the proposed approach and experimental results. The authors provide a detailed explanation of the diffusion model and its application to dense matching, as well as a thorough evaluation of the proposed approach on several benchmark datasets. The paper's contributions are supported by the experimental results, which demonstrate the effectiveness of the proposed approach. The paper is well-organized and easy to follow. The authors provide a clear explanation of the proposed approach and its application to dense matching, as well as a detailed evaluation of the approach on several benchmark datasets. The paper's contributions are clearly presented and supported by the experimental results.", "weaknesses": "I do not have major concerns on the paper less lacking some details. One notable improvement will be adding more discussions to diffusion based dense prediction networks, especially methods like DDP [1]. It is questionable to me why DDP is not directly applicable to the task of dense matching. Another possible improvement is to add diffusion-based dense prediction models as baselines to the method (\\eg a DDP model trained on dense flow supervision).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9x5vu2EdxA", "reviewer": "Reviewer_kgjX", "summary": "This paper proposes a new technique for finding dense correspondences between two 2D RGB images, using a generative rather than discriminative model, specifically a conditional diffusion model. The key insight is that this allows optimizing the full posterior (data and prior terms in the Bayesian formulation) instead of the likelihood. The authors propose additional technical components to get the pipeline to work robustly and accurately.", "questions": "Minor:\n- Please don't write $1e^{-4}$ when (I assume) you mean $10^{-4}$. $e$ is the base of natural logarithms. If you must use scientific number formats (please do it only in code, not papers!), do note that it's written $1e-4$, not $1e^{-4}$.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "With the caveat that this is not my precise area of specialization: I enjoyed reading the paper and think that the proposed method is elegant and interesting. The idea of treating the correspondence field as an image to be synthesized is compelling. The additional components in the pipeline (e.g. for super-res) seem appropriately chosen. The results are good -- even if they don't always beat state-of-the-art baselines -- and definitely good enough given that the technique is of independent methodological interest.", "weaknesses": "\"These approaches assume that the matching prior can be learned within the model architecture by leveraging the high capacity of deep networks\"\n\nFor the argument in the paper to be more compelling, the above statement needs to be clarified. Exactly how is the prior \"learned within the model architecture\"? Can we say something more precise about how the prior is captured, and how much of it, in these earlier methods?\n\nHow many samples were used to compute the MAP estimates used for statistics in the tables, as per Section 4.6? And were these samples chosen i.i.d. from the standard normal distribution?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "muIqbuEyS6", "reviewer": "Reviewer_nL9F", "summary": "This work proposes to use a diffusion model to model a data prior for dense correspondence matching. In particular the authors use a standard feature extraction and dense matching stage to have an initial guess for a dense matching field, then they refine it using a diffusion model trained to predict the residual over the initial guess and finally upsample it using a second diffusion model. With this structure they are able to achieve competitive results compared to the SOTA in dense correspondence matching. The use of a diffusion model to model the data prior of their system implies that the proposed solution is actually a generative system that given an initial dense matching field can sample plausible output ones; as a by-product of this formulation the authors propose to model matching uncertainty as discrepancies in the sampling process with some preliminary interesting results.", "questions": "### Questions\n\n1. Can you comment on weakness (b.) and clarify whether the reported numbers are with/without multiple sampling?\n\n2. Could the proposed conditional denoising diffusion module be plugged on top of other dense correspondence methods to enhance their performance (possibly with retraining)? For example could step (b) of this method be combined with PDCNet+ to further boost the performance? \n\n\n### Suggestions\n\n* Tab. 6 in the appendix is a repetition of Tab. 1 in the main paper\n* I would suggest using a more obvious color map for Fig. 6 in the appendix, right now is a bit hard to parse.\n* I would also suggest rename DiffMatch to PDCNet+ for the qualitative comparison in Fig. 4-5 since that’s the anime used in the table?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ **Novel formulation of the problem**: to the best of my knowledge I have not seen a diffusion model used in this context to refine a matching field. \n\n+ **Possibility to model uncertainty**: the proposed formulation models a distribution of plausible matching fields given an initial guess and therefore models implicitly the uncertainty of the matching process. Fig. 6 in the supplementary shows some preliminary analysis of the modeled uncertainty. I found this emerging property of the formulation extremely interesting although only a preliminary exploration is reported in the paper.", "weaknesses": "a) **Possible generalization concerns and limited experimental evaluation**: modeling a prior on what a good matching field looks like using a diffusion model exposes the proposed solution to generalization problems since the prior will only model the type of matching flows seen during training. For example in the extreme case where the method is trained only with match fields coming from homographies it will probably not generalize well to other types of non-rigid transformations between frames. The competitors have this type of limitation in a less pronounced way since they focus on improving feature extraction and matching rather than modeling a global prior on what a “good matching field” should look like. Whether this problem arises in practice is hard to estimate from the current paper since the experimental validation is rather limited compared to the main competitors.T he proposed method is evaluated only on two datasets for dense correspondence matching and on two corrupted versions of the same datasets. Competitors like GOCor, PDCNet and PDCNet+ are evaluated on other datasets (e.g., MegaDepth and RobotCar) and additional correspondence tasks (e.g., Optical Flow on KITTI, Pose Estimation on YFCC100M and ScanNet).\n\nb) **Inference time concerns**: Tab. 5 of the paper compares the inference time of PDCNet(+) vs the proposed method with 5 sampling steps and shows that the two proposals are comparable. However in Sec. 4.6 the authors mention that in practice they sample multiple times and average the diffused fields to get the final performance. Depending on how many samples are drawn it will have an impact on the runtime making it grow significantly. From the current paper it is unclear if this multiple sampling strategy is used in the experimental validation ro only in Appendix C.2 and whether the inference time of Tab.5 are taking this into account or not. If not (as it seems like from the text) the inference cost will be significantly higher than competitors.\n\nc) **More ablation studies and unclear dependency on the initialization**: the core of the work is the use of a diffusion model to refine an initial estimation of a matching field ($F_{init}$). From the paper it is unclear how much the prior is able to recover in case of a bad initialization or not and whatever, if possible, the model will need more diffusion steps to recover from a bad conditioning. I would have liked these aspects to be discussed as part of the ablation study. Another interesting ablation that would have nicely complemented the work would have been using the dense cost volume as conditioning to the diffusion process. If the concern is around hardware limitations a test should still be possible at lower resolutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FzdFFDYnv4", "reviewer": "Reviewer_tm6R", "summary": "The authors propose a framework for dense matching for a pair of images using conditional diffusion models. In particular, the framework consists of 2 stages. In the first stage an initial cost volume is computed from the features of the source and target images. The cost volume is used to calculate initial global and local flow information. In the second stage, the initial flow field is refined using a multiscale conditional diffusion model to predict dense correspondence between source and target models. Comparisons are provided with recent baselines and state of the art performance is demonstrated.", "questions": "1. In this setting is $F_{init}$ extracted from VGG-16 considered the prior term ?\n2. What is effect of choice of feature extractor on $F_{init}$. Do features extracted from different extractors like ResNet variants still provide reasonable $F_{init}$ for further optimization? In particular, since this is only used as initialization, would the downstream performance be fairly agnostic to this choice?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Paper quality**: The paper is well-written and clearly presented, with attention to detail. The authors have clearly put a lot of effort into making the paper easy to read and understand.\n2. **Comparisons**: The paper provides adequate comparisons to several baselines, on two different datasets, demonstrating the effectiveness of the proposed approach.\n3. **Ablation**: The ablation studies clearly highlight the need for all of the introduced components, which provides additional evidence for the effectiveness of the proposed approach. The appendix also provides an insightful ablation on the effect of matching quality for number of sampling steps. \n4. **Related Works**: An adequate and detailed treatment of related works has been provided to place this work in the context of literature related to dense correspondence computation. \n4. **Approach** The proposed approach is simple and elegant, which makes it easy to understand and implement. Represents a good demonstration of the correspondence matching algorithm.\n5. **Design of Prior** Computing initial value from a cost volume constructed from a pre-trained VGG-16 network and only learning residual is an efficient strategy as the displacement to be learned is smaller than what would need to be learned otherwise.\n6. **Reproducibility** : All the details of the feature extraction network to compute the cost volume and the details of the diffusion model is provided in detail, aiding in the reproducibility of the proposed approach. \n6. **Appendix** The authors provide a clear and detailed appendix section, which is helpful for readers who want to learn more about the proposed approach.", "weaknesses": "1. **Novelty** : The novelty is somewhat limited to the specific design of the conditioning to the diffusion model. Computation of cost volume using pretrained network have been used in many flow computation networks (as in Glu-Net). In particular, the main novelty is the smart choice of inputs and outputs for the diffusion model. Elaborating a bit more on the challenges of the design will help highlight the novelty of this approach. \n3. **Generalization**: The framework shows impressive performance on dense matching for the given datasets, but providing a sense of how generalizable this is to in-the-wild captures, is potentially helpful.\n4. **Performance limits**: The qualitative example demonstrated show dense correspondence matching for relative simple transformations between source and target. Providing some insights about how the framework performs for wide baselines or for settings with large viewpoint changes would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)