Neural Matching Fields: Implicit Representation of Matching Fields for Visual Correspondence
Authors: Sunghwan Hong
Ji Su Nam
Seokju Cho
Susung Hong
Sangryul Jeon
Dongbo Min
Seungryong Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Implicit neural representation
semantic correspondence

Abstract: Existing pipelines of semantic correspondence commonly include extracting high-level semantic features for the invariance against intra-class variations and background clutters. This architecture, however, inevitably results in a low-resolution matching field that additionally requires an ad-hoc interpolation process as a post-processing for converting it into a high-resolution one, certainly limiting the overall performance of matching results. To overcome this, inspired by recent success of implicit neural representation, we present a novel method for semantic correspondence, called Neural Matching Field (NeMF). However, complicacy and high-dimensionality of a 4D matching field are the major hindrances, which we propose a cost embedding network to process a coarse cost volume to use as a guidance for establishing high-precision matching field through the following fully-connected network. Nevertheless, learning a high-dimensional matching field remains challenging mainly due to computational complexity, since a na\"ive exhaustive inference would require querying from all pixels in the 4D space to infer pixel-wise correspondences. To overcome this, we propose adequate training and inference procedures, which in the training phase, we randomly sample matching candidates and in the inference phase, we iteratively performs PatchMatch-based inference and coordinate optimization at test time. With these combined, competitive results are attained on several standard benchmarks for semantic correspondence. Code and pre-trained weights are available at~\url{https://ku-cvlab.github.io/NeMF/}.

Neural Matching Fields: Implicit Representation of Matching Fields for Visual Correspondence
Sunghwan Hong Korea University
Jisu Nam Korea University
Seokju Cho Korea University
Susung Hong Korea University
Sangryul Jeon UC Berkeley
Dongho Min Ewha Womans University
Seungryong Kim
Korea University
Abstract
Existing pipelines of semantic correspondence commonly include extracting high- level semantic features for the invariance against intra-class variations and back- ground clutters. This architecture, however, inevitably results low-resolution matching field that additionally requires ad-hoc interpolation process post- processing for converting into high-resolution one, certainly limiting the overall performance of matching results. To overcome this, inspired by recent success implicit neural representation, we present a novel method for semantic corre- spondence, called Neural Matching Field (NeMF). However, complicacy and high- dimensionality of a 4D matching field are the major hindrances, which we propose cost embedding network to process coarse cost volume to establishing high-precision matching field through the following fully-connected network. Nevertheless, learning a high-dimensional matching field remains chal- lenging mainly due to computational complexity, since a naïve exhaustive inference would require querying from all pixels in the 4D space to infer pixel-wise cor- respondences. To overcome this, we propose adequate training and inference procedures, which in the training phase, randomly sample matching candidates and in the inference phase, we iteratively performs PatchMatch- based inference and coordinate optimization test time. With these combined, competitive results are attained on several standard benchmarks for semantic correspondence. Code and pre-trained weights are available at httphttt: h/hp::://ku-eee cvlab. .it.tgttthhh io/NeMF/
guidance for
Introduction
Establishing visual correspondence across semantically similar images is a fundamental problem in computer vision, which has been facilitating many applications including visual localization [68] 38).
strucure-frommmmotion (69 image editing I and autonomous driving [33]. Unlike traditional dense correspondence tasks 20 where visually similar images of the same scene are used as inputs. semantic correspondence problem poses additional challenges due to intra-class appearance and
severe geometry variations among object instances 15
16]
77 in semantic correspondence literature attempt to address above challenges by leveraging Convolutional Neural Networks (CNNs)- based features thanks to their greater semantic invariance than traditional hand-crafted descrip- 10 66 3 that only capture low-level local structure. As shown in Fig. [a), they typically perform matching with deeper features that contain high-level semantics to obtain a low-resolution correspondence map, and they enforced to hand-crafted interpolation techniques, e.g., bilinear
Much research
65
26
47
31
tors 37
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Target dim.
hg
lhg
NeMF
Target dim.
Source
hg
hg
dim.
MLP
Source
dim.
& (C, p)
We
P
We
Cost Feature Representation
Cost Volume
7(p)
Ws
(a) Existing works
(b) NeMF (Ours)
Figure I:
Intuition of the proposed neural matching field (NeMF): (a) existing works 47 g and
(b) the proposed NeMF. Unlike existing methods that explicitly compute and store discrete matching field defined at low resolution, we implicitly represent a high-dimensional 4D matching field with deep fully-connected networks defined arbitrary original image resolution.
interpolation 665 64, 26 77 9 or TPS warping with sparse keypoints [48] 50 47 significantly
reducing localization precision in matching details. Instead of these hand-crafter designs, several works [25 76 19] attempted to formulate a coarse-to-fine approach by utilizing multi-level features, but they often suffer from the propagation of initial error from the early coarse level.
Inspired by recent success of Implicit Neural Representation (INR) 44 57 45 where coordinate-based neural network allow to model continuous field, we propose novel learnable framework, dubbed Neural Matching Field (NeMF), that aims to establish high-precision correspon- dence at arbitrary original image resolution. However, typically, matching field between a pair of images is complicated and high-ding sigh-dimensional, where a simple fully-connece commonly used in INR, may fail to implicitly represent such a high- ligh-dimensional matching field. To better structure the intricate matching field, we propose a cost embedding network that takes a coarse cost volume to learn cost feature representation and use it as guidance for generating figh-precision matching field through the following fully collecedneeeted network. This is accomplished by designing the cost embedding network with convolutions (18) and self-attention layers (78) to encapsulate local contexts and impart to all pixels with global receptive fields of self-attention, which also helps to compensate for the lack of inductive bias of Transformer by injecting convolutional inductive bias.
network, which is
The intuition of the proposed method is illustrated in Fig. (b).
Although leveraging a cost representation may alleviate the issues for learning the matching field with details preserved, naïvely performing feed-forward for all pixels of matching field to find pixel-wise correspondences that used for providing supervisory signals or inference would be computationally intractable. To this end, we learn a neural matching field by enforcing the network to predict the correctness of correspondence given and the ground-truth point. Furthermore, as the intractability issue applies similarly at inference phase, we propose novel test-time optimization method that not only adopts PatchMatch [I] -asedd search space sampling strategy in the learned neural matching, but also optimizes the coordinates for a means of correction that lead find better correspondences as the iteration progresses. We alternatively perform both PatchMatch-liie inference and coordinate optimization, which works as an exploration and exploitation solution.
consisting of randomly sampled points
In the experiments, we evaluate the effectiveness of the proposed method using the standard bench- marks for semantic correspondence 49 [6]. We demonstrate that the proposed implicit matching field effectively finds high precision correspondences, reporting the dramatically boosted perfor- mances in comparison to that of hand-crafted interpolation techniques. We also conduct extensive ablation study to validate our design choices and explore the effectiveness of each components.
2 Related Work
Semantic Correspondence. The earliest works [10 66 37
focused on feature extraction stage by proposing the hand-crafted feature descriptors. Although these works are probably based on the most well-known traditional hand-crafted feature descriptors, they exhibit limited capability to capture high-level semantics. Resolving such an issue, convolutional neural networks (CNNs) 722 I8 made a paradigm shift thanks to their robust representations to deformations, at first replacing hand-crafted features with deep features, rapidly converging towards end-to-end learning. Since

---

(a) Source
(b) CAT'S K
(c) NeMF (Ours)
(d) CATS
(c) NcMF (Ours)
Figure 2: Visualization of matching fields: (a) source image, where the keypoint is marked as green triangle, (b), (c) 2D contour plots of cost by CATs [8] and the NeMF (ours), respectively, and (d), (e) 3D visualization of cost by CATs and NeMF, with respect to the keypoint (a). Note that all the visualizations are smoothed by Gaussian kernel. Compared to CATs [8) NeMF has higher peak near ground-trun and makes a more accurate prediction.
then, leveraging deep features has become the de facto standard. Rocco al. [62 first proposed an end-to-end geometric matching networks based on the deep feature maps extracted using CNNs and correlation maps computed between the extracted feature maps. Since then, using correlation map which contains all pairs of similarities between descriptors has become popular by numerous
matching networks (65
63 43
76 19
31 48, 50
47
64
26 34
82
32
40 8). However, not
only exhaustively computing and storing all pairwise similarities require quadratic memory and computation complexity with respect to the input spatial size, which a major downside, but also is infeasible to compute them as the resolution increases. This inevitably caused existing methods to utilize correlation map defined at low resolutions.
On the other hand, notable methods include NC-Net (65] which first proposes to employ 4D convo- lutions identify spatially consistent matches by exploring neighbourhood consensus. DHPF (50] applied probabilistic Hough matching (PHM) 7 to find the matching points. CHM 47 extends the PHM by employing -ighesiimenional convolutional kernels to aggregate 6D correlation maps. CATs [8] and its extension [9] use transformers [78 12 to explore global consensus from correlation maps thanks to transformers' ability consider long-range interactions. All these works exploit rich semantics present at high-level features for robust matching across semantically similar images. However, this inevitably necessitates up-sampling the predicted correspondence field to original image-level resolutions, which may result in losing precision. Unlike them, we implicitly represent matching field at arbitrary image-level resolution, eliminating such a loss and ensure high-precision correspondence field to be found, as shown in Fig. 2
Implicit Neural Representation. Implicit neural representation (INR), also known as coordinate- based representations, is continuous, differentiable signal representation parameterized by neural networks [45). INR recently received huge attention, and substantial progress has been made in this direction. INR is not coupled spatial resolution, making the memory requirements to parameterize the input signals orthogonal to spatial resolution.
Notable contributions to INR include COIN 13 that proposes a compression method with INR and LIIF (5) learns a continuous representation for images that can be presented in arbitrary resolution. DeepSDF 5 was a pioneering work that enables high quality representation from 3D input data by leveraging a learned continuous signed distance function. Occupancy networks [4] implicitly represent the 3D surface as the continuous decision boundary. IM-Net [6] also takes similar approach, learning a mapping from coordinates conditioned by shape feature vectors to determine whether point is outside inside the 3D shape.
Since then, IaRedased works consistently have attained state-of-the-art performance in 3D computer vision. As pioneering work, NeRF 45 represents 3D scenes as neural radiance fields for novel view synthesis. Inspired by NeRF, a large number works [8] 54 28 53
59 [4
58 2
73 4 51
80 61 5 made progress in this direction. Although flourished in 3D computer vision tasks, INR has never been properly studied or explored in visual correspondence tasks, which we address in this work.
3

---

Extraction
Cost Computation
Source dim.
hg
Target dim.
Cost Embedding
Source h2
hgg
Target dim.
r(p)
(C. p)
Matching
ML
Field
Feature Extraction
M())
We
We
(Aas Wyy Ry, w))
(hgs War hy W)
Wy
Figure 3: Overview of network architecture: Given a pair of images as an input, we first extract
features using CNNs [18] and compute an initial noisy cost volume at low resolution. We feed the noisy cost volume with the proposed encoder consisting of convolution [T8] and Transformer [78), and decode with deep fully connected networks by taking the encoded cost and coordinates inputs.
Preliminary
Neural Radiance Field (NeRF) is continuous function fw with parameters w that computes volume density and RGB color value с = (r. r. b) taking as an input 3D location o (Ir, y: 2) and 2D viewing direction d = (0, Φ), such that fw (o, d) + (o,, c). In practice, as shown in 45 74], mapping low dimensional inputs x and the neural networks enables better representing high frequency variations.
higher dimensional features before passing them through
Specifically, denoting
7-) an encoding function and I. the number of frequency octaves (t) [sin(2"tm] (x), cos( c2s(22tm) tr)) (2 tx), to)))
(1)
the overall process fu : R x Ru -> IR+ x R is defined as
(o,c) = (n()) (d))
(2)
where To and La denote output dimension of the encoded coordinate o and viewing direction d. respectively. The function fw is formulated representations are not coupled to spatial resolution for using continuous functions, making the
fully-connected deep network. This implicit neural
memory consumption required to parameterize the signal independent of spatial resolution 45 74
4 Neural Matching Fields (NeMF)
4.1 Problem Statement and Overview
The overview of NeMF is shown in Fig. Given pair of source and target images as 1, E H.,,,, and 4 E H, xW,, our objective to find a dense correspondence field F(x) that is defined for each pixel in original image resolution, which warps towards I satisfy (x) I,(x F(x)).
Following traditional matching pipeline, we first extract dense features D, and D, from the input source and target images, and then compute full pair-wise similarity scores between them using cosine distance such that:
D.(x). Dr(y)
C(x,y) y) (x), Di(r)l'
where x E [0, h.) x [0, w.) and y (0, [0, w.). and || denotes 1-2 normalization. Previous extracted features from the deep layers of CNN to have high-level semantic invariance, resulting the spatial resolution of D, and D, to be reduced, i.e., h < H and w < W. Consequently, utilizing the coarse similarity scores C'xx, y) to infer correspondences inevitably yields low-resolution correspondence map, which additionally requires post-processing to be interpolated into a high-resolution map [8] 47].
approaches [50 47
To alleviate this issue, we propose an INR-based learnable framework, called neural matching field (NeMF), that implicitly represents a high- nigh-dimensional 4D matching field to infer nigh-precision correspondences at arbitrary scales without any post-prn pest-erocssiig procedure. Specifically, we formulate a continuous function fa as multi layer perceptron (MLP) with parameters in which encoded position and its corresponding cost feature vector are taken as an input. Formally, denoting 4D coordinates defined original image resolution as p [x. y] where x [o, H.) x [0, W.) and
y [0, H:) x [0, W,), our neural matching field M E R X.. H.XW, is computed
(p) fo(7(p), ((C. [0.
(4)

---

Target dim.
Source ltg] dim.
hg
MLP
M M
pC).p)
bg W/
P
We
Cost Feature Ws Representation
7(p)
Figure 4: Overview of neural matching field optimization: Given an encoded cost, we randomly sample coordinates from uniform distribution. The random coordinates and ground-truth coordinate are then processed altogether to obtain the matching scores and the cross-entropy loss is computed for the training signal.
where (p) is encoded point of p, and p) denotes the cost feature vector at p extracted from coarse cost volume C. A possible way for the architecture design of function fo is first concatenating the two imputs r(p) and @CC, p). and then passing them through the fully-connected network. This approach, however, may impose memory intensive batch normalization operation [.. To address this, we adopt the architecture of [52), where oCC, p) is added to the input features of each fully- connected block. In addition, to guarantee the value of 4D matching cost fields to be lied within the range from 0 to 1,
sigmoid function 17 the end of the networks.
4.2
Cost Embedding Network
Our assumption that formulating the function fa(.) without any condition may be challenged when representing complicated and high-dimensional continuous field. We address this by introducing cost embedding network where the raw cost volume further embedded into cost feature volume oCC, p). which is used guidance for establishing high-precision matching field through the following fully- conneccd network fo(-).
Motivated by recent works R 22 that aggregate matching costs for better correspondence hypothesis. we further embed the raw cost volume through the global receptive fields of self-attention layer 77 [2]. Although these representations are explicitly encoded from all pixels of cost volume, the absence of operations that impart inductive bias, i.e., translation equivariance by convolutions or relative positioning bias, may yield representations with errors. To this end, combine Transformer architecture 78 with convolution operator to compensate the lack of inductive bias, allowing local and global integration of matching cues by encapsulating the local contexts and imparting them to all pixels via self-attention. Concretely, before providing matching cost to the function fo. the 4D raw cost volume C' is embedded into 5D cost feature volume C" e R, W, xW, x K with K channels which is still defined at low resolution. We then use a quadlinear interpolation on C' for query point p cost feature vector ((C,, p) R 1
4.3 Training
As shown in Fig. 4 to train the networks, we use a ground-truth keypoint pair (x, x'} between an input image pair in manner that if query point p [x,y] is classified the ground-truth correspondence. i.e., = x', the network output should be encouraged to be 1, and 0 otherwise. We formulate this as a classification problem, and thus we apply cross-entropy loss learn to predict the correctness of correspondence for a query x in the source image among sampled negative keypoints y the target image (where y A x') and that of ground-truth x'. Even though the better negative sampling techniques, e.g., hard negative mining [27 24). can be used, in experiments, we simply adopt random sampling from uniform distribution as negative samples, as background clutters or extreme geometric variations inherently present across semantic correspondence datasets [15] 16 49 would contribute to robust representation learning. Formally, the loss function defined as follows:
K
C
Σ Mflog[M(px)
(5)
where P& denotes k-thh query samples for Is E (1,..., K}. and Mt is a ground-truth -ruth matching score, e.g., Mi = 1 if pę is ground- truth keypoint pair, and otherwise.
5

---

R1 Target dim.
Hej
H. Target dim..
Ho] Target dim.
Source
Source
Source
Backpropagation
dim.
argmax
MLP
logf Mr(
P
W
W.
W/
pt
F'O))
x N
x N.
PatchMatch Inference
Coordinate Optimication
Figure 5: Illutratiion of the proposed PatchMatch and coordinate optimization: With the learned neural matching field, the proposed PatchMatch injects explicit smoothness and reduces the search range. The subsequent optimization strategy searches for a location that maximizes the score of MLP.
Although this would provide sufficient supervisory signal, we found is beneficial provide an additional explicit supervisory signal for learning better cost features, which positively affect our PatchMatch
[ basedd inference strategy as will be further detailed in Sec. 4.4 To this end, we use end-point-error [8] between the predicted keypoints using the cost feature representations o(C, p) directly and the ground-truth keypoints. Concretely, we obtain a channel
average pooled cost feature volume V avgpool (CC p)) E IR x W. x xW, and compute Fpred by applying soft
argmax function V. We then calculate the Euclidean distance between the ground-truth flow map computed using the ground-truth keypoints and the predicted flow map as
L. = Fg - Fpred|12-
(6)
Combining with Eq. 6 we define the final objective function Liotal with balancing weights A and Ae: fooal C + AcCo-
4.4 Inference
PatchMatch-based Sampling. At the inference stage, we aim to find dense correspondence field F F(x) by leveraging the trained network determine the correct correspondence for each query = y). However, searching the best match for each coordinate over all possible matching candidates y exhaustive manner results H₃ x W. x H, x W, number of feed-forward per sample, which is an extremely time consuming and computationally intensive.
To alleviate the issue, we propose PatchMatch -ibbeded sampling. PatchMatch [1] runs sequence of propagation and update steps to reduce a search space. We use the learned NeMF so as a scoring function to determine the correspondence. To overcome its time-consuming process induced by serial processing inherited from PatchMatch [1]. we propose a GPU-friendly PatchMatch optimization that performs propagation and update in a parallel manner.
Specifically, for initialization, we utilize an average pooled cost feature volume V introduced in Section 4.3 Note that the objective of Eq. 6 directly connects to the initialization step of this approach. This implies that the better cost feature representations would help to obtain a better initialization for PatchMatch-based inference. Then for a query we sample correspondences by considering adjacent pixels such that 2 = [2'-122]}z for adjacent pixels z (1 1-tth iteration. In addition, few random points sampled from uniform distribution are used to augment z such that y U ( for randomly sampled pixels y where U denotes an union of the sets. Then the correspondence fields are undated by considering the set of matching candidate y- such that
of candidate
F((x) argmaxyey=- (M (Mx.))) yl)).
This process is iterated until the convergence. In practice, this candidate selection and scoring run in parallel for every target pixel. This makes the inference process efficient and GPU-friendly in the original resolution, compared to serial propagation and update in 30
Coordinate Optimization. Although the proposed PatchMatch- inference strategy could prevent exhaustive searching by effectively sampling the search range for determining the correct correspondence for each pixel, this may degrade the performance due several reasons including insufficient number of iterations that may result a sub-optimal solution and a limited search range that provides relatively fewer candidates for consideration. To address this issue, we provide means

---

(a) Source
(b) Target
(c) =5
(d) N 10
(c) N =
(f) N = 20
Figure 6: Visualization of flow maps for different N iterations: (a) source image, (b) target image. As the number of iteration we set increases along (c), (d), (e) and (f) at inference phase, NeMF with trained MLP predicts more precise matching fields by PatchMatch-based sampling and coordinate optimization. Note that different colors of flow maps indicate directions and magnitudes of the flows.
to reduce the potential erroneous inference by adopting test-time optimization strategy that directly optimize coordinates y that maximizes the correctness of the correspondence using the learned the network fo-
Concretely, because the network So is naturally differentiable, as shown in Fig. we use gradient descent to optimize the target coordinate y in the direction of decreasing the negative log likelihood of the matching score, with respect to the corresponding source coordinate Formally, the coordinate optimization performed by iterative updates which can be formulated as:
C = ( ([x, y))), vl),
(8)
y y yeyy-vy, aV
where a denotes step size. Any advanced optimizer can also be used for improved optimiza- tion 60 39]. Note that the source coordinate not updated during this optimization. With the proposed coordinate optimization, we combine with PatchMatch-basse sampling to establish a final correspondence field as shown in Fig. [S] Each iteration number is defined as N. As exemplified in Fig. @ NeMF predicts more precise matching fields by PatchMatch-baas sampling and coordinate optimization as evolving iterations.
Note that the key difference of this test-time optimization to that of DMP 19 that we optimize the coordinates to correct themselves to find better correspondence with by leveraging the already learned network, while DMP optimizes the parameters of the networks.
Experiments
5.1
Implementation Details
For backbone feature extractor, we use ResNet-101 [8]] pre-trained on ImageNet II] We use the feature maps resized to 16x 16 for constructing coarse cost volume. For the cost embedding network, we build upon S and its implementations. We implemented our network using Py PyTorc (56), and W (41) optimizer with initial learning rate of 3e 5. We N = 10 for both PatchMatch and coordinate optimizations, and learning rate of 3e-4 is used for coordinate optimization. Additional details are provided in the supplemenatry material.
5.2 Experimental Settings
Datasets. We use three benchmarks, which include SPair-7Ik 149), PF-PASCAL [6 and PF- WILLOW (15), to evaluate the effectiveness of the proposed method. SPair-7Ik 49] provides total 70,958 image pairs, PF-PASCAL [66] contains 1,351 image pairs from 20 categorics, and WF-WILLWW [5] contains 900 image pairs from 4 categories. Each dataset contains ground-truth annotations, which we use them for evaluation and training.
Evaluation Metric. For the evaluation metric, we employ a percentage of correct keypoints (PCK), which is computed as the ratio of estimated keypoints within the threshold from ground-truths to the total number of keypoints. Assume a predicted keypoint kpred and ground-truth keypoint KeT, the number of correctly predicted keypoints are counted, and the condition for deciding the

---

Table I: Quantitative evaluation on standard benchmarks 49 15
16 75
Higher PCK is better.
The best results are in bold, and the second best results are underlined. All results are taken from the papers. Eval. Reso.: Evaluation Resolution, Flow Reso.: Flow Resolution.
SPair77K [2)
PI-PASCAL [Td
PF-WILLOW UiST
Methods
Eval. Reso,
Flww Reso.
PCK
Chivss
PCK @ cling 0.03
PCK @ cheesdy
0.01
0.03
0.05
0.1
0.01
0.05
0.1
0.01
0.03
0.05
0.1
CNNGeo
62
ori
20.6
41.0
69.5
36.9
69.2
A2Net 81
22.3
42.8
70.8
36.3
WeakAliga
63
20.9
49.0
74.8
37.0
70.2
75.9
SFNet 833
288/ofi
20
53.6
81.9
46.3
74.0
PARN
25
26.8
49.1
FMD
35
20
37.4
90.7
75.6
PMNN
31
50.4
82.4
90.6
MMNei
82
224x320
40.9
77.6
89.1
DCC Nei
241Mori/
55.6
82.3
43.6
73.8
HPF 48
300
28.2
60.1
84,8
45.9
74.4
GSF 20
36.1
65.6
87.8
49.1
78.7
ANC-Neet
34
240
15
86.
NC-Net
65
240Vori/-
15
20.1
54.3
78.9
33.8
67.0
50
CHM
+
240
15
46.3
80.1
91.6
69.6
CAT'
2.3
13.8
27.7
2.9
NeMF
ori
3,2
19.5
34,2
53.6
18.6
61.6
80.6
93.6
3.8
25.4
60.8
75.0
Table 2: Per-class quantitative evaluation on SPair-71k [49] benchmark.
aeco.
hus
cbai.
cow
dog
pess.
plan.
CNNGoo [0
23,4
16.7
17.7
32.7
12.7
27.4
13.7
20.9
17.5
10.3
34.1
20.6
Weak-llan 63
22.2
17.6
41.9
15.1
38.1
27.4
21.2
31.8
12.8
26.3
22.6
14.2
20.0
22:2
17.9
10.4
32.2
35.1
20.9
NC-Nat
151
17.9
12:2
52.1
11.7
29.0
19.9
16.1
39.2
9.9
23.9
18.8
15.7
17.4
15.9
14.8
9.6
24.2
31.1
20.1
HPF 65
25.2
18.9
52.1
15.7
38.0
22.8
19.1
52.9
17.9
33.0
3288
20.6
24.4
27.9
21.1
15.9
31.5
35.6
28.2
SCUT
20.7
23.3
24.4
DHPF
47.5
35.1
MMNNi
43.5
27.0
62.4
27.3
40.1
50.1
37.5
60.0
21.0
56.3
50.3
41.3
30.9
30
33.2
64.2
43.6
40.9
PMIC 30
34.1
35.9
74.9
16.3
42.1
43.8
40.0
72.6
21.1
67.6
58.1
30.5
40.1
54.1
43.3
35.7
74.3
39.9
10.4
CATs
32.0
34.7
72.
40.9
53.5
43.6
66.5
24.4
63.3
56.3
52.0
42.6
41.7
43.0
33.6
72.6
38.0
40.9
NeMI
55.6
37.
76.2
36,9
54.1
62.1
47.5
70.5
26.2
67.6
59.3
$7.1
48.0
40.2
42.1
36.7
96.7
66.1
53.6
correctness is defined as follows: d(ckpred < a mmxx W), where d( and denote Euclidean distance and a threshold. When we evaluate on PF-PASCAL, we use aiing following other works [16] 48 50 (). SPair-7Ik 7 and PF-WILLOW with abboxi H and W denote height and width of the object bounding box or entire image, respectively.
5.3 Matching Results
To ensure a fair comparison, the model evaluated on SPair-71k 49 is trained on training split of SPair-71k [49] and the model evaluated on PF-PASCAL [16 and WF-WILWWW [5] is trained on training split of PF-PASCAL [6].
The results are summarized in Table and the qualitative results are shown in Fig. 7 We note the resolution which the method evaluated, since 77 observe that the resolution of images affect the PCK performance, and the resolution of which the method outputs the correspondence field. It shown that NeMF achieves competitive performance or even attains sfate-of E--hhe-an art performance for several alpha thresholds. More concretely, for lower alpha thresholds, we tend to achieve higher PCK compared to other works. This implies that the existing works, which rely on interpolation techniques that prevent from fine-grained matching due to the use of matching field defined at low resolution, may have suffered from the large resolution gap between the predicted flow and that of ground-truth. For example, CATs [8) processes the cost volume 16 and infers flow map at this resolution. On the contrary, NeMF avoids this by implicitly representing a matching field at higher resolution, demonstrating its advantageous approach.
5.4
Ablation Study
In this section, we conduct ablation study to investigate the effectiveness of different configurations for cost embedding network and effectiveness of the proposed inference strategies. We train the networks on the training split of SPair-71k 49 and evaluated on the test split.

---

(a) CAT'
(b) NeMF
(c) CAT'
(d) NeMF
Figure 7:
Qualitative results on PF-PASCAL [66]
keypoint transfer results by (a), (c) CAT' g
and (b), (d) NeMF. Green and red line denote correct and wrong prediction (Aring 0.1), respectively. Note that correspondences are estimated at the original resolutions of iamges.
Different Cost Feature Representation. Table summarizes the comparison between the effec- tiveness of cost feature representations learned from different configurations of cost embedding network. In this ablation study, we compare four configurations. raw cost volume defined at coarse level. From (II) to (IV), we show the effectiveness of extracting cost features via convolutions, self-attention layers and integration of both, respectively.
(I) shows the results for exploiting
We observe that simply leveraging a raw cost volume struggles learn complicated match- ing field, as it does not provide sufficient struc- tural detailed information among pixel-wise similarities. As the cost embedding network is introduced learn feature representations, the
Table Cost feature representation.
SPair 49 PCK @ Ostos
Components
0.05
0.1
0.15
(
Coarse cost volume
0.3
2.3
5.8
15.5
25.7
Conv.
14.6
28.3
48.9
59.4
performance is dramatically boosted, and our approach clearly helps to attain the best perfor- mance by learning more powerful representa- tions than (II) and (III).
Self-attention
2.5
16.4
30.6
51.7
61.7
Cony, self-attention
19.5
34,2
53.6
63.3
Inference Strategies. In this ab- lation study, we aim to show a
Table 4: Inference strategy. T7. denotes Too Long.
quantitative comparison between different strategies at the inference phase. Table summarizes the re- sults. Note that we included (I) highlight that the inference
SPair-Ik (49
Average
Components
PCK @
nunsime per sample
(
Exhaustive infer.
7L
7L
71
71
71.
> 300k
(I)
PatchMaach infer.
1.1
7.7
15.3
31.4
41.9
7.75
(ID) coordinate opt.
19.5
phase, the evaluation on pair of images with original resolution, for example, would take approx- imately more than 300k seconds. This clearly shows the infeasibility of adopting naïve inference strategy.
From (II) to (III), we observe an apparent performance boost, which demonstrates that the proposed test-time coordinate optimization helps to correct the coordinates for finding better correspondences. However, this approach has downside. Applying coordinate optimization inevitably increases the time taken for the inference, which is typical limitation of test-time optimization. However, 0.5 second is minor sacrifice for better performance. Note that further improvement could be made by adopting better optimizing strategy, i.e., learning rate, search range or optimizer.

---

Computational Complexity. Although the proposed inference strategy enables significantly re- duced time for establishing correspondence field between a pair of images, practice, we observe that assuming we set N = 10, the time taken at the inference phase for a single sample is approximately 8-9 seconds on a single GPU Geforce RTX 3090, which prevents from a real-time inference. This an apparent limitation of the proposed approach, but we refer the readers to supplementary material where we show that without affecting the performance, the memory consumption and run-time can be controlled.
In addition, we also provide experimental re- sults that demonstrates the efficiency of the pro- posed approach in comparison existing works, which is summarized in Table Let us assume four
Table 5: Memory Comparison. 00M: Out of Memory
Method
Train
Inference
16
32"
64
128*
that we are representing cost volumes different resolutions, e.g., 16, 32, 64 and 128. At the training phase, unlike other works that inevitably consume more memory as the resolu- tion increases, the proposed approach success- fully deviates from it thanks to the proposed training strategy. Furthermore, at the inference phase, observe that the proposed approach has advantage other methods. Although NeMF may suffer from relatively larger computation and memory consumption than CATs (8] and CHM [4]] when the resolution is low, it has advantage when the resolution is high, allowing the network to exploit highly accurate cost volume with relatively less memory consumption.
X
708
1,538
0OM
OOM
CHM 47
371
433
0OM
/
X
454
3,523
0OM
OOM
CAT'
188
302
1882
0OM
NeMF
/
4,205
4,205
4,205
4,205
X
1,528
1,528
2,443
6,309
Conclusion
In this paper, we proposed a novel INR-based architecture, called neural matching fields (NeMF), that implicitly represent 4D matching field to find proposed an architecture and training and inference procedures targeted to handle complicacy and high-dimensionality of matching field that acts major hindrances. Specifically, we embed the raw cost volume with convolutions and Transformer to obtain local and global integration of matching cues to handle the complicacy, and sampling-base -aaedd training and inference procedure to handle the high-dimensionality. We have shown that the proposed method attains -fateof-tthe-a art performance on several benchmarks for semantic correspondence. We also conducted extensive ablation study to validate our choices.
correspondences. This method
Broader Impact
Our implicit representation of volume may be beneficial for other domains that utilize cor-
relation map, which include semantic segmentation (67 75 46). object detection [6), and image editing [37]. It can help to boost the performance by preserving the fine-detailed information within the cost volume. However, as the proposed approach aims to implicitly represent the cost volume, on its own, it is not feasible to use for malicious purpose.
This research was supported by the MSIT, Korea 2TTT-2022-2020-0-00 ICT Creative Consilience program, No. 2020-0-00368, A Neural-Symbolic Model for Knowl- edge Acquisition and Inference Techniques), and National Research Foundation of Korea (NRF. 202IRIC1C1006891
Acknowledgements.
References
[1] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. Patchmatch: randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
[2] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Brualla, and Pratul Srinivasan. Mip-nerf: A multiscale representation for anti- aaii-alasinng neural radiance fields. In Proceedings of the JEEE/CVF International Conference Computer Vision, pages 5864, 2021.
10

---

[3] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In European conference compuder vision, pages 404-417. Springer, 2006.
[4] Eric R Chan, Marco Montciro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gam: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceddiggs of the IEFF/CVF conference on computer vision and pattern recognition, pages 5799-5809, 2021.
[5] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 8628 8638, 2021.
[6] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939-5948. 2019.
[7] Minsu Cho, Suha Kwak, Cordelia Schmid, and Jcan Ponce. Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1201-1210, 2015.
[8] Seokju Cho, Sunghwan Hong. Sangryul Jeon, Yunsung Lee. Kwanghoon Sohn, and Seungryong Kim. Cats: Cost aggregation transformers for visual correspondence. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[9] Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. arXiv preprint 2202:2202,06817,, 2022.
[10] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR Workshops), 2005.
[11] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /9929, 2020.
[13] Emilien Dupont, Adam Goliáski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.0311 00303333....,.... 2021.
[14] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8649-8658, 2021.
[15] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow. In CVPR, 2016.
[16] Bumsub Ham, Minsu Cho. Cordelia Schmid, and Jean Ponce. Proposal flow: Semantic correspondences from object proposals. IEEE transactions on pattern analysis and machine intelligence, 2017.
[17] Jun Han and Claudio Moraga. The influence of the sigmoid function parameters the speed of back- propagation learning. In Intermational workshop on artificial neural networks, pages 195-201 Springer, 1995.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
[19] Sunghwan Hong and Seungryong Kim. Deep matching prior: Test-time optimization for dense corre- spondence. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2021.
[20] Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz. Fast cost-volume filtering for visual correspondence and beyond. PAMI, 2012.
[21] Shuaiyi Huang, Qiuyue Wang, Songyang Zhang, Shipeng Yan, and Xuming He. Dynamic context correspondence network for semantic alignment. In ICCV, 2019.
[22] Zhaoyang Huang, Xianyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. arXiv preprint ar 22v: 16/94, 2022.
[23] Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy. and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.
11

---

[24] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as contrastive random walk. Advances in neural information processing systems, 33:19545-19560, 2020.
[25] Sangryul Jeon, Seungryong Kim, Dongho Min, and Kwanghoon Sohn. Parn: Pyramidal affine regression networks for dense semantic correspondence. Proceedings of the European Conference on Computer Vision (ECCV), pages 351 351-366, 2018.
[26] Sangryul Jcon, Dongho Min, Seungryong Kim, Jihwan Choe, and Kwanghoon Sohn. Guided semantic flow. In ECCV. Springer, 2020.
[27] Sangryul Jeon, Dongbo Min, Scungryong Kim, and Kwanghoon Sohn. Mining better samples for contrastive learning of temporal correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1034-1044, 2021.
[28] Yoonwoo Jeong. Seokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5846-5854, 2021.
[29] Scungryong Kim, Stephen Lin, Sang Ryul Jeon, Dongbo Min, and Kwanghoon Sohn. Recurrent transformer networks for semantic correspondence. Advances in neural information processing systems, 31, 2018.
[30] Jae Yong Lee, Joseph DeGol, Victor Fragoso, and Sudipta Sinha. Patchmatch-bass based neighborhood consensus for semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[31] Jae Yong Lee, Joseph DeGol, Victor Fragoso, and Sudipta N Sinha. Patchmatch-based neighborhood consensus for semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13153- 13163, 2021.
[32] Junghyup Lee, Dohyung Kim, Jean Ponce, and Bumsub Ham. Sfnet: Learning object-aware semantic correspondence. In CVPR, 2019.
[33] Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Socren Kammel, Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, et al. Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE intelligent vehicles symposium (IV), pages 163-168. IEEE, 2011.
[34] Shuda Li, Kai Han, Theo W Costain, Henry Howard-Jenkins, and Victor Prisacariu. Correspondence networks with adaptive neighbourhood consensus. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10196-10205, 2020.
[35] Xin Li, Deng- Ping Fan, Fan Yang, Ao Luo, Hong Cheng, and Zicheng Liu. Probabilistic model distillation for semantic correspondence. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7505-7514, 2021.
[36] Tsung- Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[37] Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE transactions on pattern analysis and machine intelligence, 33(5):978 0978899994411 2010.
[38] Liu Liu, Hongdong Li, and Yuchao Dai. Efficient global 2d-3d matching for camera localization in large-scale 3d map. In Proceedings of the IEEE International Conference Computer Vision, pages 2372-2381, 2017.
[39] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint
2019.
[40] Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence as an optimal transport problem. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05 2017.
[42] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7219- 2021.
12

---

[43] laroslav Melekhov, Aleksei Tiulpin, Torsten Sattler, Mare Pollefeys, Esa Rahtu, and Juho Kannala. Dge-net: Dense geometric correspondence network. In WACV, 2019.
[44] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[45] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 105-421..0 Springer, 2020.
[46] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. arXiv preprint arXiv: (0t::2/04.05530 2021.
[47] Juhong Min, Scungwook Kim, and Minsu Cho. Convolutional hough matching networks for robust and efficient visual correspondence. arXiv preprint arXiv:2109.0522 2021.
[48] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Hyperpixel flow: Semantic correspondence with multi-layer neural features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[49] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-7Ik: A large-scale benchmark for semantic correspondence. arXiv preprint arXiv:1908. 10543, 2019.
[50] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Learning to compose hypercolumns for visual correspondence. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16. Springer, 2020.
[51] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proceedings the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 1 11453-1 1164, 2021.
[52] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-35111 2020.
[53] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven I Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 174, 2019.
[54] Keunhong Park, Utkarsh Sinha, Jomathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerties: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[55] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- BBruall,, and Steven Seitz. Hypemerrf: A higher-dimensional representation for topologi- cally varying neural radiance fields. arXiv preprint arXiv:2106. 13228, 2021.
[56] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alhan Desmaison, Luca Antiga, and Adam Leter. Automatic di diffcrentiation in pytorch. 2017.
[57] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, pages 523-540. Springer, 2020.
[58] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 10318-10321 2021.
[59] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14153-14161, 2021.
[60] Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.0923 000004009977,00.11
[61] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335-1432 2021.
13

---

[62] Ignacio Rocco, Relja Arandjelovie, and Josef Sivie. Convolutional neural network architecture for geometric matching. In CVPR, 2017.
[63] Ignacio Rocco, Relja Arandjelovié, and Josef Sivic. End-to-end weakly-supervised semantic alignment. In CVPR, 2018.
[64] Ignacio Rocco, Relja Arandjelovié, and Josef Sivie. Efficient neighhourhood consensus networks via submanifold sparse convolutions, In ECCV, 2020.
[65] Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivie. Neigh- bourhood consensus networks, arXiv preprint arXiv: 105 2018.
[66] Edward Rosten and Tom Drummond. Machine learning for high-speed corner detection. In European conference on computer vision, pages 430-443. Springer, 2006.
[67] Michael Rubinstein, Armand Joulin, Johannes Kopf, and Ce Liu. Unsupervised joint object discovery and segmentation in internet images. In CVPR, 2013.
[68] Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Efficient & effective prioritized matching for large-scale image-based localization, IEEE transactions on pattern analysis and machine intelligence, 39(9):1744- 1756, 2016.
[69] Johannes L. Schonberger and Jan-Michael Frahm. Structure-from-noee mooti revisited. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1104-4113 2016.
[70] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154--00166. 2020.
[71] Paul Hongsuck Seo, Jongmin Lee, Deunsol Jung, Bohyung Han, and Minsu Cho. Attentive semantic aligmment with affset-aware correlation kernels. In Proceedings of the European Conference on Computer Vision (ECCV), pages 349-364, 2018.
[72] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409. 140...56,,. 2014.
[73] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances Neural Information Processing Systems, 33:7462-7473, 2020.
[74] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kei,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances Neural Information Processing Systems. 33:7537-7547, 2020.
[75] Tatsunori Taniai, Sudipta N Sinha, and Yoichi Sato. Joint recovery of dense correspondence and cosegmen- tation in images. In Proceedings the IEEE conference on computer vision and pattern recognition. pages 4246-4255, 2016.
[76] Prune Truong, Martin Danelljan, and Radu Timofte. Glu-net: Global-local universal network for dense flow and correspondences. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6258-6268, 2020.
[77] Prune Truong, Martin Danelljan, Fisher Yu, and Lue Van Gool. Probabilistic warp consistency for wcakly-supervised semantic correspondences. arXiv preprint arXiv:2203.04279. 2022.
[78] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances neural information processing systems, 2017.
[79] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021.
[80] Alex Yu, Vickie Ye, Matthew Tancik. and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEFF/CVP Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[81] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[82] Dongyang Zhao, Ziyang Song, Zhenghao Ji. Gangming Zhao, Weifeng Ge, and Yizhou Yu. Multi- scale matching networks for semantic correspondence. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Psuedo-code will be provided in supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "mxiqeWtajp3", "reviewer": "Reviewer_JRSW", "summary": "This paper introduces neural matching fields into semantic correspondence. To the best my knowledge, this approach should be the first method to do the task using implicit neural representation. There are two problems: the computation for 4D matching field and the inference efficiency. Authors provide effect method to address the two problems.", "questions": "I only have one concern. Traditional Implicit Neural Representation method such as LIIF and NeRF records images into the weights of neural network. One neural network represents one image or one scene. Does NeMF take a neural network to represent a semantic correspondence or a matching cost.   If so, how much time will your method cost to train a network? If not so, what is the difference between your method and other semantic correspondence methods.", "limitations": "According to my understand, NeMF takes a network to represent a matching cost. In practice, people need a method to compute different matching cost for different image pairs. How does NeMF to deal with this situation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper employs implicit neural representation to do semantic correspondence. This should be the major contribution. According to the statement of authors, I can follow the idea easily and this idea should work.\n\nThe disadvantage of this work is the experiments. There are too many quantitative comparisons. According to the data, the performance of this method seems OK. However, authors should provide more visual experiments to convince readers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jJkA9bAdLPb", "reviewer": "Reviewer_vfUA", "summary": "This paper proposed a novel method based on Implicit neural representation(INR) for semantic correspondence, named neural matching field (NeMF). Also a cost embedding network containing convolution and self-attention layers is established in order to represent complicated and high-dimensional continuous field better. During the inference phase, this paper introduced the patchmatch-based sampling and coordinate optimization which make the NeMF performs better and more efficient. Experiments on several standard benchmarks and ablation study validate the proposed architecture with the full resolution input image pairs.", "questions": "Figure 3 illustrates the proposed network architecture but it is not clear, especially the concrete architecture of the cost embedding network.\n\nFigure 6 demonstrates the flow maps for different N iterations, the author should give more explanations on the meanings of different colors.\n\nFigure 7 shows several semantic matches on PF-PASCAL, however, it seems that the results of the third pair in the first two columns and the second pair in the last two columns look almost the same while some correspondences predicted by CATs are marked red. Additionally, the correspondence marked red of the last image pair in the third columns looks different from that reported in the original paper(CATs)\n\nqualitative results of CATs on PF-WILLOW are different from that reported in the original paper (i.e. 50.3 when α=0.05 and 79.2 when α=0.1 )\n", "limitations": "This paper formulates predicting semantic correspondence as a classification problem which lacks of generalization between datasets, Concretely, the network trained on one datasets with fixed number of categories may not performer well on other datasets with unseen categories", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: Comparing to existing methods, this paper firstly leverage the implicit neural representation in semantic correspondence.\n\nQuality: Benefiting from INR which is not coupled to spatial resolution, images can be input to the proposed network with their original resolution and SOTA performance is attained with competitive inference time. \n\nClarity: The paper is somewhat clear, but some important details are missing or unclear\n\nSignificance: The paper is likely to have moderate impact on semantic correspondence\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ft_ME3go5uS", "reviewer": "Reviewer_iiVf", "summary": "This paper proposes a new matching field generation method called a neural matching field (NeMF). It uses a cost embedding network consisting of convolution and self-attention layers to process the coarse cost volume to obtain cost feature representation. Through such a mechanism, NeMF can realize high-resolution semantic matching. During the training, to reduce the computational cost of the 5-D matching field, NeMF combines the convolution operations with the self-attention operations to get initial matching tensors.  NeMF also incorporates random sampling when calculating the local matching fields. ", "questions": "1. What is the biggest difference between NeMF and other semantic matching methods based on 5-D matching cost volume?\n2. It seems that the output of PatchMatch is used as pseudo labels to guide the training. \n3. During the inference, it's unclear why NeMF needs back-propagation. ", "limitations": "The computational cost of NeMF is relatively high. I don't find any other negative societal impact.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strength***\n1. This paper is well-organized and can be easily understood by readers. The technical details are introduced clearly.\n2. The authors conducted extensive experiments on multiple benchmarks to investigate the effectiveness of different modules and designs in this paper.\n3. NeMF aims to get matching fields in high resolutions which can resolve the matching ambiguity during semantic matching.\n\n***Weakness***\n1. Some core techniques of this paper have been used in some disparity or optical flow estimation papers. In Figures 3 and 4, the way to construct 5-D matching cost volumes has been widely used in disparity estimation. Besides, although the 5-D matching cost volumes can get richer matching information, it takes in a huge amount of computational cost. When the resolution of the matching field is very large, it's too expensive to process these 5-D matching volumes.\n2. In lines 137-138, to get the semantic matching field in high resolution, NeMF uses interpolation methods (as that in MMNet, ICCV 2021) to upscale the matching fields. But such an interpretation will lead to matching ambiguity obviously. Can these ambiguities brought by interpolation methods be eliminated in subsequent modules?\n3.  In section 4.3 (lines 180-182), if NeMF uses random sampling, the training will encounter the long-tail problem. Because most of the point pairs will be classified as negative matching, and only very few matched point pairs can be classified as positive matching.  In fact, the negative point pairs are much more than the positive pairs. If the random sampling strategy is adopted, there will be no positive samples. And this will lead to the crash of the model training. I feel a little confused about the random sampling.\n4. In lines 185-190, how to generate dense matching flow through sparse annotations is not well-introduced.\n5. In Figure 5, is the inference conducted on the validation or test set? Why there is back-propagation during the inference process? Does NeMF use the output of PatchMatch as pseudo labels to train the model?\n6. It seems that NeMF uses the output of PatchMatch as the pseudo label. But there is no ablation study to support it. It will make readers confused about whether the performance gain comes from the model itself or its supervision.\n7. It seems that the computational cost of NeMF is relatively high.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "WCk0WCu-O96", "reviewer": "Reviewer_a2FL", "summary": "Deep learning based image matching algorithms rely on creating a 4D tensor representing the cost of matching each pair of pixels. Due to large memory consumption this 4D tensor has to be kept at a coarse scale which affects the quality of matching at a finer scale (i.e., small localization errors are prominent). Inspired from the concept of implicit neural representations, this paper proposes to learn an MLP which can efficiently interpolate in the space of matching costs. To recover the matching during inference, the paper proposes two ways: sampling and gradient based optimization. Experiments show that this approach indeed can match images significantly better at a finer scale than competitive approaches. Overall this idea of implicit neural representation combined with test time optimization can also have applications in other tasks which also compute cost volumes e.g., depth estimation, optical flow etc. ", "questions": "C. Major questions and suggestions:\n1. In equation 6, is $y$ initialized from $F_{pred}$ similar to PatchMatch? \n2. In line 272, CATs[8] are said to have a cost tensor of size $16^4$ which at first glance seems to be more than this approach of NMT. However in line 9 of Appendix it is mentioned that NMT needs even more ($16^5$). I would suggest to change the wording in line 272.\n3. I did not find details about training time, batch size, compute hardware used for training etc. Also peak memory consumption during training and inference needs to be compared with CATs. I think NMF would need more due to 5D cost representation instead of 4D.\n4. The details of cost embedding network 4.2 are fuzzy. In Table 2 please define the components concretely. See Table 1 in [R-MVSNet](https://openaccess.thecvf.com/content_CVPR_2019/supplemental/Yao_Recurrent_MVSNet_for_CVPR_2019_supplemental.pdf) for an example.\n\nD. Minor comments and suggestions (do not need to be addressed in rebuttal):\n1. It would be better to have a list of contributions at the end of introduction. This can help the reader in exactly understanding what is novel and what is not in this article.  \n2. Continuing further on the point B1, have the authors tried making the cost tensor even smaller (e.g. $12^5$) to 'stress-test' the MLP? How worse are the results? It might decrease memory consumption and inference time. I understand if these results cannot be reported within the rebuttal period, but it would be good to have these for final version.\n3. Relating to point C4 above, it would be good to know the exact details of architectural changes done on top of CATs possibly in Appendix. Lines 7-9 of Appendix are not detailed enough in my opinion.\n3. I assume the concept of neural fields presented in this paper is also interesting for optical flow and depth estimation applications. Thus in broader impact these applications can also be added. \n4. The reference of [30] is missing the field of year.\n ", "limitations": "Only limitation is more computational burden (compute time and memory) than competitive approaches. This is not a major limitation because the idea itself is novel. Moreover the increase in memory consumption is still linear instead of $O(M^4)$ for $M \\times M$-sized images for conventional approaches. Nonetheless these limitations need to be made more clear in the final manuscript. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A. Strengths:\n1. The idea of using MLP for recovering fine details from matching costs is novel and possibly useful in other applications as well.\n2. The paper is mostly written well and easy to understand.\n3. Extensive experiments do show improvement of this approach over competitive approaches on several datasets. \n4. Even though this approach incurs slightly more memory burden than best competitive approach (CATs[8]) but it allows to compute the matching costs at any arbitrary resolution foregoing the need for upsampling of the cost tensor.\n\nB. Weakness:\n1. The concept of implicit representation was motivated by the need for reducing memory consumption. However, this approach ended-up ultimately using a 5D cost tensor requiring 16x more memory than competitive approach of CATs[8] which uses 4D. \n2. Computation: Inference time of this approach is significantly more than competitive approaches. Although the authors do mention this limitation but memory consumption is not reported (for both training and testing).  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept