Keypoint Matching via Random Network Consensus
Authors: Siyan Dong
Shuzhe Wang
Daniel Barath
Juho Kannala
Marc Pollefeys
Baoquan Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Computer Vision
Keypoint Matching
Randomized Networks
Visual Descriptor
Detector

Abstract: Visual description, detection, and matching of keypoints in images are fundamental components of many computer vision problems, such as camera tracking and (re)localization. Recently, learning-based feature extractors on top of convolutional neural networks (CNNs) have achieved state-of-the-art performance. In this paper, we further explore the usage of CNN and present a new approach that ensembles randomly initialized CNNs without any training. Our observation is that the CNN architecture inherently extracts features with certain extents of robustness to viewpoint/illumination changes and thus, it can be regarded as visual descriptors. Consequently, randomized CNNs serve as descriptor extractors and a subsequent consensus mechanism detects keypoints using them. Such description and detection pipeline can be used to match keypoints in images and achieves higher generalization ability than the state-of-the-art methods in our experiments. 

Under review as a conference paper at ICLR 2023
KEYPOINT MATCHING VIA RANDOM NETWORK CON- SENSUS
Anonymous authors Paper under double-blind review
ABSTRACT
Visual description, detection, and matching of keypoints in images are fundamen- tal components of many computer vision problems, such as camera tracking and (re)localization. Recently, learning-based feature extractors on top of convolu- tional neural networks (CNNs) have achieved state-of state-of-the-arr performance. In this paper, we further explore the usage of CNNs and show that it's possible to lever- age randomly initialized CNNs without training. Our observation is that the CNN architecture inherently extracts features with certain extents of robustness to view- pointiillmmintioon changes and thus, it can be regarded as descriptor extractor. Consequently, randomized CNNs serve as descriptor extractors and a subsequent consensus mechanism detects keypoints using them. Such description and de- tection pipeline can be used to match keypoints in images and achieves higher generalization ability than the ffute-of-the-ald methods in experiments.
INTRODUCTION
Keypoint detection, description, and matching in images are fundamental building blocks in many
computer vision tasks, such as visual localization (Satter et al. 20016 Taira et al.) 2018 Dusmanu
ct al. 2019 Revaud et al. 2019 Sarlin et al. 2219 2020 Tang et al. [221]) ceructure-from-Motiee
(SfM) (Snavely et al. 2006 Wu 2013 Cui & Tan 2015 Schönberger Frahm 2016 Lindenberger
et al. 2021). Simultaneous Localization and Mapping (SLAM) (Mur-Artal et al. 2013 Mur-Artal
& Tardós 2017 Dai et al.||200 (017)) object detection (Csurka et al. 2004] Yang et al. (201)) 9. and pose
estimation (Suwajanakorn et al., 2018 Kundu et al. (018)). The keypoints, in general, refer to the salient pixels that are then matched across images forming point-to-point correspondences. They should be discriminative and robust to viewpoin//iluminatio changes to be accurately matched.
pipeline that first detect salient pix- els (Harris al., 1988 Lowe 2004; Mikolajczyk & Schmid, 2004) then compute local descrip- tors (Lowe 1999 Bay et al. 2006] Calonder et al. 1 Rublee et al. 201T) on top of those pixels. Typically, the detectors consider low-level 2D geometry information such as corners and blobs. To deal with large viewpoint distances, scaled pyramids are applied with Laplace of Gaussian (LOG), Difference of Gaussian (DOG), etc. For description, local statistics such gradients and histograms are computed and used as visual descriptors. To name a few, SIFT (Lowe) 1999), and its variant RootSIFT (Arandjelović & Zisserman 2012). are still popular nowadays due their generality.
Traditional approaches follow
deteetionnthen-deeeree
In recent years, bearning-based approaches on top of convolutional neural networks (CNNs) (Yi al., 2016 Noh et al.] 2017 Ono et al. 2018 Mishkin et al., 2018, De et al. 2018 Dus-
manu et al. 2019 Revaud et al.) 2019) achieve promising results, especially in extreme appearance changes, such images taken day or night (Zhou et al., 2016). and across seasons (Sattler al.) 2018). Compared with traditional handcrafted approaches, the key advantage of introducing deep learning is the ability to learn robust keypoint representations from large-scale datasets. The afore- mentioned methods apply either supervised or self-supervised learning mechanisms to train their networks. After training, the off-the-shelf detectors and descriptors generalize well to several new datasets. Benefiting from their simplicity and effectiveness, the learned features such Super- Point TeeTone et al., 2018), D2-Net (Dusmanu et al. 2019), and R2D2 (Revaud et al., 2019) are widely used nowadays.
In this paper, we further explore CNNs in keypoint detection, description, and matching, without requiring the deep networks be trained. Our observation is that the CNN architecture itself inher-

---

Under review as a conference paper at ICLR 2023
Figure I: Keypoint matching results on two pairs of color images with certain extents viewpoint change. On the left part, we apply the trained off- shellff SuperPoint (DeTone et al. 2018) and visualize its correct matches shown in green line segments. On the right part, we make use of the same network architecture of SuperPoint, but apply 3 different sets of random parameters. The cor- rect matches produced by the random CNNs are visualized in blue, yellow, and red line segments, respectively. Different network parameters produce different matches, yet observe their overlaps in several regions. The images are from the 7-Scenes dataset (Shotton et al. 2013
ently extracts features with certain extent of robustness to viewpoin/illumination changes. There- fore, the extracted features can be directly used as visual descriptors, as shown in Figure [] Since no training is required, it is free to obtain set of visual feature descriptors by changing the random seed that generates the network parameters. In order to minimize the number of incorrect matches (e.g., due similar and ambiguous image regions), we propose consensus mechanism considering several randomly generated descriptors simultaneously to filter incorrect matches. This consensus design can be regarded as keypoint matching and, our experiments, it successfully filters out large amount of wrong matches. The final set of matches, consistent with the epipolar geometry, is found by the widely-used RANSAC (Fischler Bolles 1981) algorithm.
We summarize our contributions as follows:
We show the possibility that using CNNs for keypoint description, detection, and matching. without requiring the deep networks to be trained. This allows the algorithm to generalize well across multiple modalities (domains).
Benefiting from our no-training design, we can freely generate multiple descriptors for each keypoint. This allows for introducing a consensus mechanism to detect robust keypoint matches among the candidates produced by randomized CNNs.
The proposed pipeline achieves similar performance compared with otate-of-the-art detec- tors and descriptors while performing better on the images of new modalities.
2 RELATED WORK
Traditional keypoint detection and description. In general, a good keypoint should be easy to find and ideally the location of the keypoint is suitable for computing a visual descriptor. There- fore, early works (Harris et al. 1988 Shi et al. 1994, Lowe 2004:; Mikolajezyk & Schmid 2004) detect keypoint various types of edges, corners, blobs, shapes, etc. In recent decades, detectors and descriptors like SIFT (Lowe) 2004), SURF (Bay et al. (0066,, and RootSIFT (Arandjelović & Zisserman 2012) are widely used due to their generality. Benefiting from time efficiency, bi- nary descriptors such as BRIEF (Calonder et al.; 2011), BRISK (Leutenegger et al., 2011). and ORB (Rublee et al. 2011) are also popular in many real-tin real-time applications, namely the series works of ORB-SLAM (Mur-Artal et al., al, 2 Mur- Artal & Tardós 2017). The descriptors designed in the aforementioned traditional approaches, in general, are local statistics with certain extents of invari-
2

---

Under review as a conference paper at ICLR 2023
ance scale and rotation. In this paper, we showease that random statistics stem from convolutional neural networks (CNNs) can also be used as visual descriptors.
Learning-based keypoint detection and description. FAST (Rosten & Drummood 2006) is the
first approach that introduces machine learning for corner detection. Recent works (Savinov et al,
2017b. Zhang & Rusinkiewicz, 20188: Di Febbo et al. 2018 Laguna & Mikolajezyk 2022) make
use of deep learning with CNNs to boost the performance. Most of the learning- based methods
focus on description (Simonyan et al., 2014; (Simo-Serra et al., 2015 Balntas et al. 2016 Savinov cet al. 2017a; Mishchuk et al.) 2017: He et al.] 2018 Luo et al. 2019). Based on the traditional detection- ciwiiion-tee--eee dehe-deesriiiion pipeline, LIFT (Yi et al. 2016) takes both keypoint detection and de- scription into account. SuperPoint (DeTone et al. 2018) is the first approach to perform both tasks in a single network. One problem with supervised learning of keypoint detectors is that how to define the saliency. SuperPoint first makes use of synthetic dataset consisting of different shapes and regards the junetions as keypoints for pre-training- Then homographic adaptation applied to other datasets (c.g.. MS-COCO (Lin et al. 2014)) for self-supervised learning. D2-Net (Dusmanu ctt al, 2019) proposes to perform detection after description, an additional loss term is added to seek repeatability. Meanwhile, the keypoints should be not only repeatable but also reliable, which motivates the approach of R2D2 (Revaud et al., (019). Other recent works (Noh et al. [017] Ono al., 2018 Luo et al.] 2020 Tyszkiewicz et al. 2020 Li et al.
2022) apply a similar pipeline and contribute on network designs and training mechanisms. In this paper, we focus on the approaches with simple yet effective network architectures and explore the impact of randomness and consen- sus mechanism. Specifically, SuperPoint (DeTooe et al. 2018), D2-Net (Dusmanu et al. (2019), and R2D2 (Revaud et al. 2019) are chosen as representatives. There are also learning-based dense or semantic correspondence predictions such as UCN (Choy et al. 2016) and NBB (Aberman et al., 2018 which are beyond our scope.
Consensus mechanism. Robust estimation is the problem of simultaneously estimating the param- eters of an unknown mathematical model and finding the points consistent with it (i.e., inliers) in a set of noisy inliers and large-scale measurement errors (i.e., outliers). One of the most popular robust estimators is the RANdom SAmple Consensus (RANSAC) (Fischler & Bolles 1981) that iteratively selects minimal sets of data points, estimates the model parameters, and calculates the support (i.e., number of inliers). There are many variants (Brachmann et al., 2017: Barath & Matas, 2018 Barath et al.] 2020; [vashechkin al., 202T) and the idea of voting and consensus are widely used in computer vision problems such as visual localization (Brachmann & Rother 2019 Huang al. 2021), 1) object detection (Qi et al. 2019), and pose estimation (Peng et al. 2019). In this paper, we apply the idea of voting and perform a consensus mechanism to detect robust keypoint matches.
METHOD
Problem statement. Given a pair of images containing overlapping scene regions, the task of keypoint matching is to find a set of pixel-wise matches that correspond to the same underlying 3D) scene points. These matches enable downstream tasks, e.g., pose estimation and Structure-from- Motion (SIM). Note that the aforementioned camera pose estimation is a minimal problem that requires only a few high-precision matches. However, in practice, due to low precision and outliers existing, a certain amount of matches are required to run robust estimation.
Method overview. Figure 2 illustrates our proposed method. The input is a pair of images, and the output is set of pixel-wise matches. We make use of m VGG-style (Simonyan & Zisserman, 2014) convolutional neural networks (CNNs) with random parameters, i.e., there are m different visual descriptor extractors. Therefore, for each pixel in each image, we obtain descriptors. Next, we apply a matcher (e.g.. the nearest neighbor matcher) across images to select similar pixels based on the extracted descriptors. Note that the matching process is executed independently for each extractor. Consequently, we obtain m sets of match candidates. These candidates are then fed into a consensus mechanism to produce the final matches. Below, we first describe the randomized CNNs to generate match candidates in Section and then introduce the consensus mechanism to produce final matches in Section 3.2
3

---

Under review as a conference paper at ICLR 2023
CNN
Image
Descriptors
Matching
CNN
Final matches produced by consensus among candidates
Image B
Figure 2: The pipeline our method. set of convolutional neural networks (CNNs) with random parameters serve visual descriptor extractors. The input two images are fed into each CNN to extract pixel-wise features as descriptors. Next, for the descriptors outputted by the extractors, a matcher is applied to compute match candidates between the two images. The candidates are then fed into a consensus mechanism to select the final matches.
Local softmax Ratio-to-man
CNN
Image
Descriptor
Local maxima
Saliency
Figure 3: The keypoint extraction (description and saliency estimation) pipeline for a single image with single randomized CNN. The CNN produces dense feature maps pixel-wise visual descrip- tors. Salient descriptors are then used for the keypoint matching process, in which the saliency is estimated by normalization on top of the dense descriptors.
3.1 RANDOM DESCRIPTION
The keypoint extraction process of single CNN is illustrated in Figure 3 Following Super- Point (DeTone et al. 2018), we apply simplified network architecture as the descriptor extractor that takes the full image x as input and produces the feature map FHxwxN xN f(I as pixel-wise descriptors, where H. W E N refer the image height and width, and N N refers to the dimension of descriptors. Before applying descriptor matcher, the feature map is processed to saliency map filter homogeneous regions.
Descriptor. In our method, the CNNs are randomly initialized without any training. Our intuition is that convolution kernel computes certain type of local statisties inside its receptive field, just like traditional methods that count handcrafted gradients and histograms. Therefore, a CNN is a com- bination of kernels to count statistics of statistics. In the literature, there machine learning-like algorithms that apply random statistics to solve computer vision problems, such as in place recog- nition (Glocker al. 2014) and visual localization (Cavallari al.) 2019). which demonstrate the statistics. Note that, in our method, the parameters of CNNs
effectiveness of randomized-then-ffe are also fixed after the random initialization, that each CNN computes consistently the same type of statistics at inference time. Consequently, descriptor can only be used to match the same type of descriptors extracted by the same CNN. Since we employ multiple CNNs independently. the process of keypoint extraction and matching can be deployed in parallel.
Saliency. Before matching, we leverage a saliency detection process to reduce the matching space. Directly matching the deseriptors from randomized CNN results in many candidates lying ho- mogeneous regions, such as textureless floors and walls. To filter out these meaningless candidates, adopt the keypoint detection formulation proposed in D2-Net (Dusmanu al. 2019). The key idea is to detect local maxima in the high-level visual descriptor space, rather than detecting local 2D patterns in the low-level image color space. Specifically, the detection formulation considers two aspects: a local softmax a among nearby pixels in each feature channel, and a ratio / among the feature channels of each pixel. The and 3 scores are defined as follows:
exp (Fp...))))
aijkk EmeeeNas exp (Fv S. F.J.K/ /ax Fis,t
4

---

Under review as a conference paper at ICLR 2023
where jii.j) refers to the neighbor pixels' locations around the pixel (t,j), including itself. The saliency score is defined as $i = max; (Ak,,.k B1.g.s) and then image-level normalized. Note that the aforementioned process
forward computation without additional parameters. In D2-
Net (Dusmanu et al., 2019), the formulation is used to perform soft detection during training. To our experiments, observe that the process effectively assigns high scores to salient pixels even if the CNN parameters are randomized.
Matching. As for keypoint matching, we make use of classical nearest neighbor matcher that, for each descriptor, in one image, it retrieves the top-2 similar descriptors in another image and computes ratio test (Lowe] 2004) to filter out ambiguous descriptors. Then, mutual nearest neighbor check is applied to keep only those matches that are stable in the two matching directions, i.e,, from the left to the right image and vice versa. A match candidate is represented as p (i), (2)) where i.j E N refer to the 2D) locations of the two associated keypoints. For each type the deseriptors from the same CNN fi, the aforementioned matching process executed independently to generate set of match candidates M;- {p,, Piz .... Pig J. As result, we obtain m sets of match candidates as the input for the following consensus mechanism.
3.2 CONSENSUS MECHANISM
Directly taking all the match candidates M M = - Mj to the downstream task such as pose esti- mation often fails due large proportion of wrong matches, In this section, we introduce simple and effective consensus mechanism that rejects incorrect matches early. For each keypoint in M, our goal to find correct match or discard The idea, inspired by RANSAC (Fischler & Bolles 1981). is to first generate model hypotheses using random minimal samples and then vote for each hypothesis using the rest of the samples to select the most consensus one. our prob- lem, randomly selected candidate E .M serves the minimal sample (i.e. a model hypothesis) that gives a match between keypoints (11. and (22. Then the rest of the match candidates correlated with the two keypoints vote if they support the hypothesis. This process is achieved by keypoint clustering and consensus scoring, which are introduced detail below.
DOUF
Illustration of our consensus status on match hypotheses. The associated keypoints of each hypothesis are shown in the same color. The keypoints in the two images (left and middle) are separately grouped into clusters represented as center points (right).
Figure 4:
Keypoint clustering, Given a match hypothesis generated from P.- = ((āe))Jz)) (Zuz) Jea)] E M, the objective is to check if it is in consensus with other correlated match candidates M. The correlated candidates M C .M are obtained by seeking all the candidates that are associated with keypoints (Z fry) or (Izz:Jag)) With the keypoints Mrs we apply 2D location clustering in each image separately. According to the clustering results, we compute consensus score as the measurement of the distribution. If the keypoints are well distributed, h passes the consensus check and update the keypoint locations with the center points of the most consensus clusters. Otherwise, the hypothesis h will be discarded. The hypotheses with optimized keypoint locations output the final robust keypoint matches.
Consensus scoring. To quantitatively measure the consensus status (keypoint distribution) after clustering, introduce the consensus score on top of the clusters Q. First, the clusters containing only one keypoint are immediately discarded. For each remaining cluster E Q, we compute a density score defined
dg = lal /std((g)))
(2)
5

---

Under review as a conference paper at ICLR 2023
where [4] refers to the number of keypoints, and std(q) refers to the standard deviation of the 2D locations to approximate the cluster radius. Finally, the consensus score is defined as
i
(3)
maxy maxx(dy)) E dg otherwise.
Three examples of the consensus status are illustrated in Figure 4 and the set of keypoints in orange gains the best score among the three.
Generality. The proposed consensus mechanism above is agnostic to keypoint descriptors, detec- tors, and matchers. Therefore, their alternatives such as trained SuperPoint and SuperGlue (Sarlin et al. 2020) can also be ensembled into the framework.
4 EXPERIMENTS
In this section, validate the effectiveness of our method. We first elaborate the implementation details in Section 4.1 Then, we conduct comparisons with state-of-the-al representative methods on both matching performance in Section 4.2 and pose estimation in Section 4.3) Last, we perform analysis and ablation studies on our method in Section 4.4.
4.1 IMPLEMENTATION DETAILS
In all the experiments, we make use of 7 layers convolutional neural network (CNN) as a basic descriptor extractor. Each layer of the network is followed by ReLU activation, except for the last layer, and each of the first 3 activations is followed by a pooling layer. The last feature maps containing N = 256 dimensional descriptors are normalized to unit vectors before output. We apply bilinear interpolation to resize the descriptors to align the input resolution.
By default, we employ m 8 randomized CNNs for visual feature extraction. With the saliency map, we only select the keypoints with higher scores than the median for the following matching process. The ratio test threshold during the matching process is set to 0.95.
As for the consensus mechanism, we apply the MeanShift (Pedregosa et al. 2011) algorithm with 24 pixels bandwidth to perform keypoint clustering. The threshold the consensus score is set to 1.0 by default, i.e., we output match if there is only one valid and compact cluster.
4.2 KEYPOINT MATCHING
Competitors. There are various related approaches on top of CNN based keypoints, and we believe the following methods are the most relevant and representative to be our competitors: DELF (Noh
let al. 2017). SuperPoint (DeTTone et al. (2018), D2-Net (Dusmanu al. 2019). and R2D2 (Revaud et al. 2019 We also apply RootSIFT (Arandjelović & Zisserman 2012) as the representative of traditional handerafted approaches.
Datasets. We conduct experiments on the 7-Scenes (Shotton et al. 2013) and MegaDepth (Li & Snavely 2018) datasets since they provide depth images that can be used verify matches densely. The 7-Scenes dataset consists of indoor scenes with RGB-D images, and ground truth camera poses. Several sequences are officially divided into training and test sets for each scene. Since noisy poses exist in the training set, we only use the test set for our evaluation. To avoid view selection biases, we uniformly sample each test sequence to form our test pairs. We use the original color images, calibrated depth images, and normal images (computed based on depth images) our test input to evaluate the generalization ability to different modalities (domains). The MegaDepth dataset contains outdoor scenes with RGB-D images; we use subset with ground truth camera poses from Tyszkiewicz et al.] 2020 Sun et al. 2021 as the test set. Since the MegaDepth evaluation set is part of the training of Det-ett, we report the results of D2-Net only the depth and normal images in our evaluation.
Metrics. To quantitatively evaluate the keypoint matching results, apply matching accuracy (i.e.. the proportion of correct matches) and the number correct matches as our metrics. On the 7-Scenes dataset, match is considered correct if the distance between the corresponding 3D points

---

Under review as a conference paper at ICLR 2023
is lower than the thresholds (lem and 5cm). On the MegaDepth dataset, due to scale ambiguity, apply thresholds (5 pixels and 20 pixels) on reprojection error instead of absolute 3D distance.
The results shown in Table II On the 7-Scenes dataset, our method obtains more correct matches while the accuracy is comparable with the competitors. For depth and normal im- ages, compared with those in color images, all the methods suffer from performance drops, but are overall the best. It demonstrates the effectiveness of the randomized CNNs with the consensus mechanism. The MegaDepth dataset is much more challenging than 7-Scenes due the large view- point changes, and we observe that our method obtains reasonable results. Besides the quantitative results, we visualize the keypoint matches of two samples in Figure 5
Results.
7-Seenes (% Accuracy/ / /Matches)
MegaDepth (% Accuracy / (Maachee)
Methods
Coler
Depih
Nomual
Color
Depch
Nornal
cm
cm
cm
5 cm
cm
cmm
5 ps
20 px
5 px
20pp
5 p
20 ps
RootSIFT
13.12/11
87.50/
0.0000
12.9733
0.00:0
31.25/6
79.64/87
83.49/90
5/1/2
11.76/4
41.58 19
52.34 /4
DELF
2
61.71/16 16
0,6/1
11,96/20
0.0/0
13.8/111
6.35/32
31.51/ 160
0.90/4
6.28 28
2.10/6
14.11 /:4
SuperPoint
/2.19/22
85.02/ 176
2.00/2
30,71/22
3.03/2
36.16/31
74:55/213
82262/229
7.31
1585/18 18
30.29/588
44.05
Net
16
90.05 /60
0.00/0
27.27/7
0.0010
18.5511
17,71/2
5044/77
53
96.30/4
R2D2
12.50/12
94.29/ 108
0.00/0
45,4575
0.00/0
0.00/1
98 29/
100,00 64
30.00
66.6713
92.86/14 14
100.00/15
7.75/
78.03
/226
2,53/2
43.89/37
6,78/3
71.58/37
7.09/2
21,95/6
4.80/9
19,80/
19,00/19
42.15 /:46
Table 1: Results of keypoint matching on the 7-Scenes and MegaDepth datasets. second-bes numbers are labeled red and blue, respectively.
The best and
4.3 POSE ESTIMATION
Solvers and metrics. On the 7-Scenes dataset, we apply PnP (Lepetit et al., 2009)
algorithm with
RANSAC (Fischler& & Bolles 1981) to solve absolute poses, and we report median translation and rotation errors. On the MegaDepth dataset, due to scale ambiguity, we solve relative poses from essential matrix estimation (Stewenius et al. (006) with RANSAC, and we report the average under the recall curve (AUC) as in (Sarlin et al 2020).
The results are shown in Table On the 7-Scenes dataset, our poses on depth and normal images are marginally worse than SuperPoint, although our matching results are better. For the MegaDepth dataset, our results depth images are close to the state-of-the art, while the results on color and normal images lag behind. The aforementioned evaluation reveal limitations of our method, which are detailly discussed in Section 4.4
Results.
Metluds
7-Seemes (cm/ /)
Megsllepth (AUCSS MAALCS///UCSSR /AUCG2F)
Coler
Dapeh
Neenal
Coler
Depik
Noraral
ReSnSIFT
2,630.93
36.10/ 12.54
14.5544.5
38.09% 67. 87%%
13.9%
5.90% / 22.555
DELF
5.78/ 1.94
18.31
20.32/9.32
0.97% 9.46%
0.53% 033%5/3.848
0,38% 138%/330 3.91%
SuperPoint
2.39/0.57
500/1/71
62.6
2,99%/11
7.30% 29.6416
D2 Net
3.40/ 1.13
20:35/ 9.16
128.2/35.62
0.445 /5.13%
1,248/ 7.45%
K202
2/75/099
157.7/ 28.5
/4.00/20000
241 47.62%
1.99% 78.98
5.20% / 17.40%
3.31/
10.11/307
5.45/138
/
1.55% 111.478
3.60% 160096
Table 2: Results of camera pose estimation on the 7-Scenes and MegaDepth datasets.
4.4 ANALYSIS
Stable random statistics. We first validate the sensitivity of the descriptors stemming from random- ized CNNs. To do so, we test a single CNN (without the saliency computation) with 10 different random seeds the 7-Scenes dataset. The mean matching accuracies for color, depth, and normal images are 66.49%, 23.06%, and %. with standard deviations 2.43e-03, 1.59e-03, and 2.29e- 03. The mean number of correct matches is 462.25, 75.00, and 96.75, with standard deviations of 8.66, 1.83, and 3.10. From these results, we observe that the performance of random descriptors is stable, with mean coefficient of variations of 1.54e-02 (i.e., 2%).
7

---

Under review as a conference paper at ICLR 2023
RootSIFT
DELF
SuperPoint
D2-Not
R222
Ours
Figure 5: Visualization of matching results. On the left, we sample pair of depth images from the 7-Scenes dataset. On the right, we sample a pair of normal images from the MegaDepth dataset.
Effectiveness of saliency. To evaluate the help the saliency computed on top of descriptors, conduct experiments the 7-Scenes dataset with single randomized CNN. When select only the descriptors with higher saliency scores than the median, the matching accuracies achieve (+3.58%)) (4.43%(+1..37%) and 48. 48.73%(+14 (1..444%)) for color, depth, and normal images. In Figure 6 we sample images from 7-Scenes and MegaDepth datasets and visualize the locations of the top 50 salient descriptors in each image. We are glad observe that the visualized deseriptor locations overall locate around salient regions.
Figure 6: Visualization of the salient descriptors in our method using a single randomized CNN. The left three samples are color, depth, and normal images from the 7-Scenes dataset and the right two are depth and normal images from the MegaDepth dataset. The salient locations are shown in dots each image.
8

---

Under review as a conference paper at ICLR 2023
Ablation studies. In Table report ablation studies of our method using different numbers of CNNs and ensemble the tramed SuperPoint and SuperGlue. The result of the single CNN does not apply the consensus check. With consensus check, when there are 3 CNNS, get high accuracy but a low number of correct matches. As the number of CNNs increases, the accuracy drops slightly while get more correct matches. The 7+SP refers to total of CNNs, one of which is the trained SuperPoint, The SuperGlue matcher is only applied to SuperPoint. As they have only 1/8 proportion, SuperPoint and SuperGlue do not impact much.
#CNNs
m
m
m
m
m (7+SP)
m = (7+SP4SG)
Color Image
70.07%
182
82.70%
/95
79.46% / 165
78.55% 207
78.57% 212
78.43%
221
Depth Image
34.43%
50.00%
45.45%
43.69%
43.80%
43.83%
33
Normal Image
48.73%
83.33%
75.51%
73.68%
72.73%
72.73%
Table Ablation studies on the 7-Scenes dataset. We report the matching accuracy and the number of correct matches for each variant of our method.
Figure 7: A failed pose estimation: visualize inliers from the pose solver containing incorrect matches in blue line segments while the ground truth matches are shown in green line segments.
Limitations. Below we analyze the limitations of our method. From Table D we observe that for depth and normal images on the 7-Scenes dataset, we are better than SuperPoint on both matching accuracy and number of matches under different thresholds. However, our camera pose estimations shown in Table 2 are less precise. We blame it on the distribution of the matched keypoints, as
shown in Figure 7 Therefore, improving the camera pose solver deal with the misleading of well-structured outliers will be a good future research direction. Another limitation of our method is that there is no guarantee for the descriptors to scale/rotation invariant. As shown in Figure [8] on the HPatches (Balntas et (017) dataset, our method overall underperforms R2D2 and SuperPoine The main reason is that our method fails on the images with very large viewpoint changes.
Overall
Illumination
Viewpoint
0.8
MMA
5FT
0.4
SuperPont
DELF
02-tae
R262
Oun
0.0
3
4
5
Threshold [px]
Figure 8: Results of keypoint matching the HPatches dataset, As in Dusmanu et al. 2019
apply the metric mean matching accuracy (MMA).
5 CONCLUSION
In this paper, we present a new approach that makes use of random statistics extracted by randomized convolutional neural networks (CNNs) visual descriptors, followed by consensus mechanism to perform keypoint matching among images. Incorporating scale/rotation invariance will definitely improve performance. Also, it is worth more exploration and research network architectures.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best-buddies: Sparse cross dons-d correspondence. ACM Transactions on Graphics (TOG), 37 (4):1-14, 2018.
Relja Arandjelović and Andrew Zisserman. Three things everyone should know to improve object retrieval. In 2012 IEEE conference computer vision and pattern recognition, pp. 291 1-191.... IEEE, 2012.
Vassilcios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Bmvc, volume 1, pp. 3, 2016.
Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajezyk. Hpatches: A bench- mark and evaluation of handcrafted and learned local descriptors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5173-5182, 2017.
Daniel Barath and Jifí Matas. Graph-cut ransac. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 0733-6741 2 2018.
Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiri Matas. Magsac++, fast, reliable and accurate robust estimator. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1304-1312, 2020.
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded robust features. In European conference on computer vision, pp. 404-417. Springer, 2006.
Eric Brachmann and Carsten Rother. Expert sample consensus applied to camera re-localization. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 7525-7534,
2019.
Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. Dsac-differentien ransac for camera localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6684-6686 0694-6692, 2017.
Michael Calonder, Vincent Lepetit, Mustafa Ozuysal, Tomasz Trzcinski. Christoph Strecha, and Pascal Fua. Brief: Computing a local binary descriptor very fast. IEEE transactions on pattern analysis and machine intelligence, 14(7):1281-12)) 2011.
Tommaso Cavallari, Stuart Golodetz, Nicholas A Lord, Julien Valentin, Victor A Prisacariu, Luigi Di Stefano, and Philip HS Torr. Real-time rgb-d camera pose estimation in novel scenes using relocalisation cascade. IEEE transactions on pattern analysis and machine intelligence, 42(10): 2465 2465-2477, 2019.
Christopher B Choy, Jun Gwak, Silvio Savarese, and Manmohan Chandraker. Universal correspondence network. Advances neural information processing systems, 29, 2016.
Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. Visual cate- gorization with bags of keypoints. Workshop on statistical learning in computer vision, ECCV, volume 1, pp. 1-2. Prague, 2004.
Zhaopeng Cui and Ping Tan. Global structure-from-ne frommmooooo by similarity averaging. In Proceedings of the IEEE International Conference Computer Vision, pp. 864-872, 2015.
Angela Dai, Matthias Nießner, Michael Zollhöfer. Shahram Izadi, and Christian Theobalt. Bundle- fusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration.
ACM Transactions on Graphics (ToG), 36(4): 2017.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. Proceedings of the IEEE conference on computer vision and paftern recognition workshops, pp. 2018.
10

---

Under review as a conference paper at ICLR 2023
Paolo Di Febbo, Carlo Dal Mutto, Kinh Tieu, and Stefano Mattoccia. Kenn: Extremely-efffcice hardware keypoint detection with a compact convolutional neural network. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 682-690, 2018
Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable enn for joint description and detection of local features. In Proceedings of the ieee/cvf conference on computer vision and patlern recognition, pp. 8092- 1101,2019
Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications image analysis and automated cartography. Communications of the ACM, 24 (6):381 1981.
Ben Glocker, Jamie Shotton, Antonio Criminisi, and Shahram Izadi. Real-time rgb-d camera relo- calization via randomized ferns for keyframe encoding. IEEE transactions on visualization and computer graphics, 21(5):571-583, 2014.
Chris Harris, Mike Stephens, et al. A combined corner and edge detector. In Alvey vision conference, volume 15, pp. 10-5244. Citeseer, 1988.
Kun He, Yan Lu, and Stan Sclaroff.
Local descriptors optimized for average precision. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 596-605, 2018.
Zhaoyang Huang, Han Zhou, Yijin Li, Bangbang Yang, Yan Xu, Xiaowei Zhou, Hujun Bao, Guofeng Zhang, and Hongsheng Li. Vs-net: Voting with segmentation for visual localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
6101-6111 11,1, 2021.
Maksym Ivashechkin, Daniel Barath, and Jifí Matas, Vsac: Efficient and accurate estimator for and f. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 2021.
Jogendra Nath Kundu, MV Rahul, Aditya Ganeshan, and R Venkatesh Babu. Object pose estimation from monocular image using multi-view keypoint correspondence. European Conference on Computer Vision, pp. 298-313. Springer, 2018.
Axel Barroso Laguna and Krystian Mikolajczyk. Key. net: Keypoint detection by handcrafted and learned cnn filters revisited. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Vincent Lepetit, Francese Moreno-Noguet and Pascal Fua. Epnp: An accurate (n) solution to the pup problem. International journal of computer vision, 81(2)155--166 2009.
Stefan Leutenegger, Margarita Chli, and Roland Y Siegwart. Brisk: Binary robust invariant scalable keypoints. In 2 International conference on computer vision, pp.
leee, 2011. 1.
Kunhong Li, Longguang Wang, Li Liu, Qing Ran, Kai Xu, and Yulan Guo. Decoupling makes weakly supervised local feature better. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18838-15844 2022.
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2041 2050, 2018
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Mare Pollefeys. Pixel-perfect ceuncture-from-moor with featuremetric refinement. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 5987-5997, 2021.
David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference computer vision, volume 2, pp. 1150-1157. leee, 1999.

---

Under review as a conference paper at ICLR 2023
David G Lowe. Distinctive image features from inveriinvariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Zixin Luo, Tianwei Shen, Lei Zhou, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan. Contextdesc: Local descriptor augmentation with cross-modality context. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2527-2536, 2019.
Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang. and Long Quan. Aslfeat: Learning local features of accurate shape and localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6589-6598, 2020.
Krystian Mikolajezyk and Cordelia Schmid. Scale & affine invariant interest point detectors. Inter- national journal of computer vision, 60(1):63-86, 2004.
Anastasiaa Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. Advances in neural information processing systems, 30, 2017.
Dmytro Mishkin, Filip Radenovic, and Jiri Matas. Repeatability is not enough: Learning affine regions via discriminability. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 284-300, 2018.
Raul Mur-Artal and Juan Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics, 33(5):1255-1265 2017.
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accu- rate monocular slam system. IEEE transactions on robotics, 31(5):1147-1163 2015.
Hyconwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In Proceedings of the IEEE international conference on computer vision, pp. 3456-3465 2017.
Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-net: Learning local features from images. Advances in neural information processing systems, 31, 2018.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao. Pynet: Pixel-wise voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4561-4570, 2019.
Charles Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9277-9286, 2019.
Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. Advances in neural information processing systems, 32, 2019.
Edward Rosten and Tom Drummond. Machine learning for high- corner detection. In Euro- pean conference on computer vision, pp. 430-443. Springer. 2006.
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 2011 International conference on computer vision, pp. 2564-2571. Ieee, 2011.
Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From coarse fine: Robust hierarchical localization at large scale. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12716-12722 2019.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 4938-4947, 2020.
12

---

Under review as a conference paper at ICLR 2023
Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Efficient & effective prioritized matching for large- scale image-based localization. IEEE transactions on pattem analysis and machine intelligence, 39(9):1744-1753 1756, 2016.
Torsten Sattler, Will Maddem, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localization in changing conditions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8601-8610, 201 8.
Nikolay Savinov, Lubor Ladicky, and Mare Pollefeys. Matching neural paths: transfer from recog- nition to correspondence search. Advances in Neural Information Processing Systems, 30, 2017a.
Nikolay Savinov, Akihito Seki, Lubor Ladicky, Torsten Sattler, and Marc Pollefeys. Quad-networks: unsupervised learning to rank for interest point detection, In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1822-1830. 2017b.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- commmotioon revisited. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016.
Jianbo Shi al. Good features to track. In 1994 Proceedings of IEEE conference computer vision and pattern recognition, pp. 600. IEEE, 1994.
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 2930-2937, 2013.
Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno- Noguer. Discriminative learning of deep convolutional feature point descriptors. In Proceedings of the IEEE international conference computer vision, pp. 118 126, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1exxiv::400..0 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Learning local feature descriptors using convex optimisation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(8): 1573-1585, 2014.
Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers, pp. 835-846. 2006.
Henrik Stewenius, Christopher Engels, and David Nistér. Recent developments direct relative orientation. ISPRS Journal of Photogrammetry and Remote Sensing, 60(4):284-294, 2006.
Jiaming Sun, Zehong Shen, Yuang Wang. Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8922-8931, 2021.
Supasorn Suwajanakom,, Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi. Discov- ery of latent 3d keypoints via end-to-end geometric reasoning. Advances in neural information processing systems, 31, 2018.
Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivie, Tomas Pajdla, and Akihiko Torii. Inloc: Indoor visual localization with dense matching and view synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7199-7209, 2018.
Shitao Tang, Chengzhou Tang, Rui Huang, Siyu Zhu, and Ping Tan. Learning camera localization via dense scene matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1831-1841, 2021.
Michat Tyszkiewicz, Pascal Fua, and Eduard Trulls. Disk: Learning local features with policy gradient. Advances in Neural Information Processing Systems, 33:14254-1142 14265, 2020.
13

---

Under review as a conference paper at ICLR 2023
Changchang Wu. Towards linear-time incremental structure from motion. In 2013 International Conference on 3D Vision-3DV 2013, pp. 127-134. IEEE, 2013.
Ze Yang. Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9657-9666, 2019.
Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. European conference on computer vision, pp. 467-483. Springer, 2016.
Linguang Zhang and Szymon Rusinkiewicz. Learning to detect features texture images. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6325-6333, 2018.
Hao Zhou, Torsten Sattler, and David W Jacobs. Evaluating local features for day-night matching. In European Conference on Computer Vision, pp. 724-736. Springer. 2016.
A APPENDIX
Times, We run all the experiments on GeForce GTX 1080 Ti. Our single CNN takes about 7 ms to infer a pair of images in x80x640 resolution, while SuperPoint takes around ms. The consensus processing for each match hypothesis takes about 0.3 ms. Note that both the CNNs inference and hypotheses consensus can be implemented parallel for further acceleration.
14


Reviews:
[{"review_id": "s3mHg5IXwna", "reviewer": "Reviewer_Miyx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to apply an ensemble of siamese CNNs to detect and describe keypoints. The parameters of these CNNS are not trained but set randomly. The keypoint descriptors produced by each siamese CNN are matched using a classical nearest neighbor with ratio test followed by a mutual nearest neighbor test. All these match candidates are filtered using a novel consensus mechanism.\n\nThe proposed approach is evaluated on 7-scenes and MegaDepth against RootSIFT, SuperPoint, D2-Net and R2D2 on three different modalities : color images, depth images and normal images.", "summary_of_the_review": "The paper is not far from the acceptance threshold but the novelty and the performances are not sufficient (to me).", "strength_and_weaknesses": "### Strengths\n\n1. The paper is overall well written and easy to read, except section 3.2 Consensus mechanism which is difficult to understand.\n2. Employing untrained CNNs is an interesting idea\n\n### Weaknesses\n\n1. The consensus mechanism section 3.2 is difficult to understand.\n2. The computational time is probably high compared to other methods as it requires 8 forward passes (8 siameses CNNs are employed in the experiments).\n3. Concerning MegaDepth, thresholds of 10 cm and 50 cm have been defined, but if I am not mistaken, for each MegaDepth scene the 3D point cloud is obtained using SfM up to a scale factor. How did you obtain this scale factor ?\n4. Why did not you run experiments on color images for MegaDepth ? (column \"color\" is missing both in Table 1 and 2)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nIplohqhTQ", "reviewer": "Reviewer_m1xk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a surprising new method for keypoint matching across image pairs. The idea is essentially to use randomly-initialized CNNs to generate the features. The random features are first preprocessed slightly to get local peaks, and then the features which do not have mutual neighbors are discarded. Finally matches are found with a consensus mechanism similar to RANSAC, which can be split into a clustering stage (generating hypotheses) followed by a scoring stage (selecting hypotheses). The experiments show that this method is competitive with existing methods.\n", "summary_of_the_review": "I like the paper, because it shows a surprising result. I think many people would want to see this. ", "strength_and_weaknesses": "I think this paper presents some surprising findings. I would not have expected that randomly-initialized CNNs could be made to compete so closely with models like SuperPoint and D2 and R2D2. I also appreciate the analysis on the number of CNNs, and the analysis over distance in the HPatches dataset. It's not the best method, but I learned something from reading the paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Km9bGpcdMD", "reviewer": "Reviewer_cMkJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed method adopts randomized CNNs ensemble and uses it as descriptor extractors and detects keypoints. Comparing to the state-of-the-art requires training specifically with supervised or self-supervised learning, the method doesn't need any of that and the author claims to have high robustness against illumination and other appearance variations.", "summary_of_the_review": "Overall the idea is simple and interesting. However, I believe image matching task has been well studied and there is no difficulty to adapt some state-of-the-art method onto depth or normal image for retraining. Therefore, the authors need to rethink about the task and the relation between the proposed method and the baselines. The question you might want to ask yourself: \"Is there any need for using this unsupervised method in this task?\" Otherwise it's just interesting but without any practical impact.", "strength_and_weaknesses": "The idea is quite interesting. Basically the method seeks matching generalization with an ensemble of purely random parameters. The idea can potentially be used in some data requires matching which contains no training samples.\n\nHowever, the task that authors targeted is specifically for appearance matching which is a well studied domain. The claim of missing matches with depth or nomal are failing (Table.1, Table.2, Figure.5 ...) are simply because those methods (e.g. Superpoint, D2-Net, R2D2) are not trained with those inputs. It's not very hard to swapping those inputs and re-train them.\n\nDespite I know the idea is more like unsupervised vs supervised method, there is simply not much hard work you need to do with supervised method. For example, R2D2 is basically a self-supervised method, the training data can be generated with a set of normal or depth images. The authors need to motivate the idea more than showing the results.\n\nApart from that, the proposed method doesn't really work on any view point changes (Figure 8.) which is essential for image matching. The paper also missed important baseline such as DELF (multi-scale) which is also very robust for illumination changes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject