Consensus Learning with Deep Sets for Essential Matrix Estimation
Authors: Dror Moran
Yuval Margalit
Guy Trostianetsky
Fadi Khatib
Meirav Galun
Ronen Basri
Venue: NeurIPS 2024 poster (2024)
Keywords: Computer Vision
Consensus learning
Relative pose estimation
Deep Sets
Primary area: machine_vision

Abstract: Robust estimation of the essential matrix, which encodes the relative position and orientation of two cameras, is a fundamental step in structure from motion pipelines. Recent deep-based methods achieved accurate estimation by using complex network architectures that involve graphs, attention layers, and hard pruning steps. Here, we propose a simpler network architecture based on Deep Sets. Given a collection of point matches extracted from two images, our method identifies outlier point matches and models the displacement noise in inlier matches. A weighted DLT module uses these predictions to regress the essential matrix. Our network achieves accurate recovery that is superior to existing networks with significantly more complex architectures.

Consensus Learning with Deep Sets for Essential Matrix Estimation
Dror Moran
Yuval Margalit
Guy Trostianetsky
Fadi Khatib
Meirav Galun
Ronen Basri
Department of Computer Science and Applied Mathematics Weizmann Institute of Science
Abstract
Robust estimation of the essential matrix, which encodes the relative position and orientation of two cameras, is a fundamental step structure from motion pipelines. Recent deep-based methods achieved accurate estimation by using complex network architectures that involve graphs, attention layers, and hard pruning steps. Here, we propose a simpler network architecture based on Deep Sets. Given collection of point matches extracted from two images, our method identifies outlier point matches and models the displacement noise in inlier matches. A weighted DLT module uses these predictions to regress the essential matrix. Our network achieves accurate recovery that is superior to existing networks with significantly more complex architectures.
Introduction
Estimating the relative pose of two cameras depicting a stationary scene is fundamental computer
vision task and basic step multiview structure from motion (SFM) 33 24
45
51
43
20
26 35
pipelines. Both classical and recent deep network- -ased algorithms (see a review in Section use point matches compute the essential matrix, which encodes the relative position and orientation of the two cameras. Identifying such point matches by existing heuristics, however, is prone to mistakes, due to possibly large viewpoint changes, illumination differences, and the presence of ambiguous repetitive scene structures, resulting in noisy matches and extremely large numbers of outlier matches (often as many as 95%) that must be identified and pruned to enable accurate pose recovery.
and simultaneous localization and mapping (SLAM) [22] 47
Classical SFM algorithms use RANSAC [16] robustly identify inliers and estimate pose parameters. While RANSAC has been used effectively for consensus recovery, learning-based based deep network approaches have introduced a competitive alternative, making steady progress in accuracy while allowing for efficient inference and demonstrating resilience to very large fractions of outliers. This progress was obtained at the price of complicating the network architecture, e.g., using message passing in local, near-neighbour graphs [58 28
31 52) expensive attention (transformer) layers
28 52,
along with the addition of hard pruning steps
52
In this paper, we introduce a simpler network architecture for consensus learning based on the Deep Sets framework [55]. Deep Sets architectures are based on shared, element-wise layers that are combined with global features produced by summing the element-wise features. Zaheer et al. and others [55] 50 proved that such architectures can express universal permuuation-eqqiiaaiiin functions over sets. In our network, the input set elements include pairs of keypoints, i.e., the coordinates of matching pairs of points. each layer, element-wiss features are produced by linear layer
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

with shared weights, followed by SoftPlus activation. Global features are obtained by averaging the clement-wiss features, where averaging used to maintain invariance to set cardinality. The network utilizes stack of such permutation equívariant layers to classify point matches as either inliers or outliers and identify a consensus set to enable accurate relative camera pose regression. We further improve accuracy by integrating a noise regression module that aims to predict the displacement, due to noise, of the (clean) positions of inlier keypoints. Finally, we observe that training in two stages, i.e. first on noise-free version of the real data (while including the outliers) and subsequently on the original real data, improves the accuracy of the predicted pose. Our network achieves accurate pose recovery that is superior to existing networks with significantly more complex architectures.
In summary, our contributions include:
NACNet, a Noise Aware Consensus Network, for consensus learning tasks and robust geometric model estimation.
A DeepSets based architecture that includes inlier displacement error estimation.
An effective noise-free pretraining scheme: first, pretrain on denoised version of the real data, then train on the real (noisy) data.
Experiments demonstrate that NACNet achieves superior results compared baselines on indoor and outdoor image pairs applied on various descriptors.
Our code is available at htttps:// com http:://gtttabbcommmmommmmrraaa/
2 Related work
Classical methods. RANSAC [16] and its successors, LO-RANSAC 8, USAC 39, MAGSAC 3. and MAGSAC+ 4 search over minimal point configurations to find consensus sets from noisy and corrupted data and estimate a corresponding parametric model. These are applied to matched keypoints with distinct descriptors obtained by filtering with Lowe's 's ratio test [30]. These classical methods are regarded the standard solutions for finding consensus in data consisting of mixtures of inlier and outlier point matches.
Learning-based methods. Deep learning-based methods have been used recently to regress geometric model and outlier classification. DFE [40] used a deep-based itcratively reweighted least squares (IRLS) scheme to predict inlier/outlier scores. LFGC [34] utilized an architecture that involves an inlier/outlier classifier and weight sharing, followed by context normalization, and applied geometric loss to the output of the weighted 8-point algorithm (also called weighted DLT (17).
Follow-up works improve prediction results by introducing more complex network designs. OANet (56) introduced an order-aware block, which contains differentiable permutation invariant pooling and unpooling operators that capture local context by utilizing soft clustering of correspondences in the feature space. CLNet (58) used this order- awdre block together with pruning and local-to-globa, consensus learning procedure strategy to classify the correspondences by employing convolutions on local and global graphs built based the Euclidean distance in feature space. All the methods mentioned above suffer from the leakage of outliers to the consensus set. Consequently, they all use RANSAC at the end of their inference step. In contrast, NCMNet (28), MGNet [31), and BCLNet
[52 used weighted DLT also at inference, showing that the performance of the results is not improved further when RANSAC is applied in addition. NCMNet (28) proposed local-to- bocal-to-global consensus learning scheme in which it first creates a local spatial graph, then a local feature space graph, and finally global graph based the inlier scores from the local graphs. BCLNet (52) introduced the
idea of Bilateral Consensus, adopting the local graph from CLNet (58) as their projection step in a channel-wise transformer that learns global consensus. MGNet [3]] used similar scheme, building both implicit and explicit local graphs and a global graph. Unlike previous methods, this method
does not prune correspondences inside the network.
In contrast to these methods, we use an architecture based on Deep Sets 55 Deep Sets enable efficient information transfer between the point matches through global features without the need to construct and manipulate graphs. Our newly proposed noise regressioo module further improves our results. Finally, with recent methods, method too does not require a final RANSAC step.
based on both CNNs and Transformers have been used in recent years to replace the handcrafted features
Learned feature matching. Deep learning- detectors and descriptors II
54 K 44

---

Noise Aware Consensus Network (NACNet)
Lg
0
L
Lz
W
NAC Block
NAC Block
Model Regression
Block
Y33
&
8z
8
Noise Aware Consensus (NAC) Block
X
Denoised keypoints
Class.
W
4
Latent features
L1-1
Set Layer
Set Layer
Set Layer
Head
P2
Predicted labels
W2 Predicted weights
Set Layer
L
Predicted essential matrix
8
Noise Head
6
X
next block
Pass to model regression
Figure I: Network architecture. Noise Aware Consensus Network (NACNet) architecture, see text
for details.
30 36 5 used in classical methods. Those methods trained with challenging and diverse data have improved the accuracy and robustness of matching even with the classical nearest neighbor matching. Yet the main problems of high outlier remained. Consequently, learned matchers that match keypoints while rejecting non-matchable ones have merged 42 27], combining Transformers with optimal transport[37] (37] to produce more accurate matches even with large camera movement. These matchers rely on the descriptors for the matching and keypoint rejection and require RANSAC as post-processing step. In contrast, our method, similar to (28 BI. [22 gets as an input the keypoints (point correspondences) only and does not incorporate RANSAC.
have shown that correcting keypoints position could positively influence the results of geometric model estimation. All of those works use visual and learned features (SIFT descriptors or features obtained from applying convolutional network the input images) to correct the positions. To our knowledge, our paper is the first to use the input point directly to correct keypoint position, as opposed previous works, which rely on either an estimated geometric model or visual features.
Keypoint refinement. Previous works 15
48
12
26
3 Method
Consider pair of images captured by (internally) calibrated cameras expressed with 3 x matrices, P= [4, 0] and P [R, t). where R and t respectively denote the relative rotation and translation between the two views. The essential matrix E = [t) R, determines the epipolar geometry between the views, so that for any two corresponding points, and p. projected from 3D point, it holds that p7' Ep = O. Existing algorithms commonly estimate the essential matrix directly from a set of putative matches between the two views, i.e., pairs of keypoints.
Our aim in this work is to construct a network that identifies consensus set of point matches (a set of inliers), given set of putative matches as input (generally contaminated with noise and outliers), and, based on this consensus set, predicts the essential matrix between the two images. We seek to construct network that overcome positional noise, which can reside in the inlier matches, and cope with a considerable fraction of outlier matches, up and above 95%. addition, we aim for method that can generalize to unseen image pairs and work with varying number of point matches and variety of fractions of outliers.
Those goals are achieved by employing a permutation-equiveat
network architecture with the following key properties: (1) a two-stage noise-aware training scheme, (2) noise head for predicting positional inlier noise, and (3) classification head to discriminate between inliers and outliers. These three key properties are the core of our method. Hence, we refer to our network as a Nwise-Aware Consensus Network (NACNet).

---

Ground truth outllers
Ground truth inliers
Denoised inliers
Figure 2: NACNet point location denoising on line-fitting task. The set X (right panel) is composed of 90% outliers (marked in grey) and (noisy) inliers (red). Our model predicts the denoised version X (purple, left panel). Evidently, the prediction of the positional noise, yielding noise-free inliers, agrees with the line model.
Formally, let X {xi...., X C R+ denote a set of point correspondences, with x, (pi, 46. Pi, ):) denoting match between keypoint p; (Ps:9s) in the left image and keypoint P: p== (Di, (i) in the right image. Our aim is, given X as an input to classify each matching pair x, as an inlier (v:- 1) outlier (y: 0) and associate with it (inlier) confidence score C2 IR. Those predictions are used to estimate the essential matrix that relates the two views.
Network architecture. Our network comprises of three noise-aware consensus (NAC) blocks. Each NAC block uses a set encoder to map the input matches to a latent representation and to correct their positions due to positional noise. The third block further classifies the points inliers outliers and produces their corresponding confidence scores. Its outputs feed the model regression block, which implements a weighted differentiable Direct Linear Transformation (DLT) algorithm the confidence scores, to predict the essential matrix. We refer the reader to Figure for a detailed scheme of our network architecture.
34] based on
Noise aware consensus (NAC) block. Each NAC block comprises of a set encoder, noise head, and classification head. The set encoder uses DeepSets layers to map the coordinates of the input point matches to a latent representation E ixdd Each DeepSets layer includes a linear, permutation equivariant layer followed by SoftPlus activation. These layers apply a linear transformation each set member and an additional (different) linear transformation to their average. (We replace the sum in [55] with an average to maintain invariance set cardinality.)
The noise and classification heads are implemented with simple two-layer MLPs. The noise head uses the latent representation to predict displacement vectors for all input points, 8 E Rax4, These displacement vectors are subtracted from the input points, X, producing their predicted denoised locations, X. The classification head uses as input the latent representation and outputs predictions for the inlier/outlier classification labels Ý E [0, 1]mh with their corresponding weights W E IR".
The predicted denoised version of the keypoints X and the latent representation L are passed to next NAC block. the third block, the inlier/outlier predicted labels Y, the weights W, and the denoised keypoints X are passed to the model regression block.
We demonstrate the NAC block denoising effect simple
task. We randomly sample 100 noisy points on a line and, in addition, 900 outliers. An example shown Figure 2 where our NACNet significantly reduces the positional noise in the inlier points.
Model regression block. The model regression block uses the predicted weights, W, and the classification labels, Y. obtained from the classification head, and the denoised version of the keypoints, X, obtained from the noise head to predict the essential matrix in the following way.
È = -(,, N,YYY,)), W).
(I)

---

where denotes the differentiable DLT algorithm (also called the weighted eight-point algorithm, see formulation 34 Section 3). Similarly [66,, we calculate confidence scores follows
y exp((W)))
C: = Σ, exp(W,)
The confidence scores are used as the weights corresponding to the denoised keypoints X in the weighted DLT algorithm.
3.1 Loss function
We minimize a loss composed of three terms
= Les (Ý; Y) + O'mod Lanod (e,un(E,,), (E,EE + am Lm L,,
(3)
The first term Lets uses a weighted binary cross entropy loss, due to the imbalanced of the inliers and outliers in the data, to penalize for inlier/outlier classification errors
"
Less (Y.Y)
Σ
Binneers y: log((s)) + Pouttiers (1-y:) -lg((1 ú:)]
(4)
Here, n is the cardinality the keypoint set, X, and Ballies and Soutiers are determined by a hyperpa- rameter search.
The second term T-mad penalizes for emros in the predicted essential matrix, similarly the suggestion in [40]. We start with grid of point pairs and use the Optimal Triangulation Method (OTM) [77 (page 318) find the closest points that satisfy the ground truth epipolar constraints. Specifically, given pair of grid points (p., P.), OTM seeks to find the global minimum for the following optimization problem:
(qi q:) argmin d(p p¹)² + d(p: p1)² subject E = 0,
(5)
P.P
where E is the ground truth essential matrix, and d is the Euclidean distance between the points. Given the optimal points {q, qi
1 we define the loss using the Symmetric Epipolar Distance:
k
Liod É, E)
Σ Ca Eq:)²
(6)
|ETQiB
Egq
1=1
where is the predicted essential matrix and k: = 400.
The last term Las is used to minimize the distance between the noise-free keypoints, X, and the predicted denoised version of the keypoints, X, over the set of the ground truth inlicrs, as follows
[X , F) = || X
To determine the ground truth, noise-free inlier keypoints, X inlierss apply the Optimal Triangulation Method (177) page 318). The parameters aynod and ans are determined by a hyperparameter search.
3.2 Training
Training our model to remove outlier matches is complicated by the presence of noise in the positions of inlier matches, potentially resulting in a small classification margin. This, turn, has been shown (in the case of kernel SVM) to have a negative effect on sample complexity and generalization error [44] (Pages 205-206, 221). 1). A further complication the lack of ground truth labels; i.c. our inlier/outlier labels set by applying preset threshold to the deviation of the points from the projections derived by the Optimal Triangulation Method (17), page 318)(see Section
To approach this problem, we train our model by applying a two-stage, noise-aware optimization process. The input to the first stage includes the set X containing the noise-free inlier matches along with the outlier matches. The optimization in this stage, therefore, uses only the first two terms of the loss (3) and the noise head muted. the second stage, the input to network includes the original set of keypoints X, and the full loss, i.e., including (7). is optimized. Our experiments and ablations indicate that this two-stage training process significantly improves the performance of our method.
5

---

200
140
600
700
120
150
500
400
50
200
20
100
0.2
0.4
0.0
0.5
0.2
0.4
0.5
0.8
0.2
0.4
0.5
0.5
(a) YFCC SIFT
(b) YFCC SuperPoint
(c) SUN3D SIFT
Figure 3: Distributions of outliers in the different datatsets. Histograms showing for each dataset
(YFCC and Sun3D) and feature descriptor (SIFT or SuperPoint) the number of image pairs (the Y-axis) with a given fraction of outlier matches (the X-axis). The means and standard deviations (from left to right) are 0.89 0.06, 0.77 + 0.14, 0.92 + 0.08
4 Experiments
4.1 Datasets and baselines
Datasets. We trained and tested method on both indoor and outdoor datasets. For outdoor dataset, we used Yahoo's YFCC dataset[49] which contains 100 million images from flicker later reconstructed using SFM(I8). For an indoor dataset, we used the SUN3D [53]. For both datasets, we used the same preprocessing and dataset split as in [56), i.c., the camera poses are extracted from an SFM pipeline, and the test set split into in-scene and cross-scene generalization. In contrast to previous methods that use the Symmetric Epipolar Distance for "ground truth" inlier/outlier labeling, we determined the labels by the deviation of the points from the projections derived by the Optimal Triangulation Method ([17), page 318) using a threshold of 3 x 10 3 In practice, changing the labeling paradigm did not affect the results. Additionally, we used the Phototourism dataset [9] to test our model's generalization across datasets. For keypoint detection, we used SIFT (30) ORB 33
and SuperPoint [1] followed by the preprocessing steps suggested [58]. is shown in Figure 3 consensus learning these datasets is highly challenging due to the high fraction of outliers in all datasets and with all descriptors.
Baselines. We compare our methods with RANSAC(I6), DEGENSAC GC-RANSAC MAGSAC+ 4. PointNet+++[81 DFE LFGC OA-Net ACNe 32 LMC-Ne CL- Net[58] DG-Net[OO) ConvMatch 57].
U- U-Match NCMNet BI], BCLNet 52). and SuperGlue All the evaluations of deep learning-basd based methods are taken from their re- spective papers unless specifically stated otherwise. We used the official SuperGlue repository for evaluation on SuperPoint, and the paper 42 results for evaluation on SIFT. For the RANSAC-based methods 2
4 9 16). we set the maximal number of iterations to 100K and use Lowe's ratio test 30) filter the initial matches, with a threshold tuned differently for the SIFT and SuperPoint descriptors to maximize performance.
Evaluation metries. We use the mean average precision (mAP) to evaluate our model predictions as suggested in [34]. We compute the mAP over the maximum between the translation and rotation angular errors of our predicted essential matrix up to the threshold of 5
4.2 Essential matrix estimation
Our results shown in Table 3 (Additional evaluations are shown in the Appendix.) The results demonstrate that our model outperforms the current SOTA in almost all conditions, including with indoor (SUN3D data) and outdoor (YFCC) images, with keypoint matches obtained with SIFT and SuperPoint, in in-scene (unseen image pairs from scenes included in training), cross-scene, and even cross-dataset Tourism) experiments. Specifically, in the YFCC/SIFT task (Table our model outperforms the other methods by significant margin in the in-scene generalization task and with smaller margin in the cross-scene generalization task. Likewise, on the SUN3D dataset, our method outperforms the other methods in both in-scene and cross-scene generalization, improving over the

---

previous SOTA by 3.6% in the cross-scene test. Qualitative results can be seen in Figure and in Appendix A.3
Figure 4: NACNet inlier/outlier classification. An example from the SUN3D dataset. Left to right: input image pairs, input matches, and our model's predicted inliers. Color mark ground truth labels: inlier matches are marked in green; outliers are marked red.
Table I: SIFT evaluation. Evaluation of essential matrix estimation on the YFCC and SUN3D datasets with keypoint matching obtained with SIFT. mAP5 (%) is reported, and the best result in each column is in bold. In-scene denote results novel image pairs taken from scenes that were included the training data and cross scene denote results on image pairs taken from unseen scenes. The first set of methods (above the middle line) includes methods that incorporate RANSAC.
YFCC(%)
SUN3D(%)
Method
In-scene
Cross-scens
In-scene
Cross- scene
RANSAC
31.57
42.78
20.88
15.79
GC-RANSAC
30.88
42.55
18.69
13.57
LO-RANSAC
30.96
42.60
19.01
13.85
MAGSAC++
31.01
42.57
19.55
14.23
SuperGluc
59.25
Point-Net++
10.49
16.48
10.58
8.10
DFE
19.13
30.27
14.05
12.06
LFGC
13.81
23.95
11.55
9.30
OA-Net++
32.57
38.95
20.86
16.18
ACNe
29.17
33.06
18.86
14.12
LMC-Net
33.73
47.50
19.92
16.82
CL-Net
39.16
53.10
20.35
17.03
MS DG-Net
38.36
49.13
22.20
17.84
ConyMatch
43.48
54.62
25.36
21.71
U-Match
46.78
60.53
24.98
21.38
NCMNet
52.33
63.43
26.12
20.66
MGNet
51.43
64.63
25.96
21.27
BCLNet
52.62
66.08
24.59
19.96
Ours
60.10
66.14
34.15
25.36
We also tested our model using the deep-learning based descriptor SuperPoint on both the YFCC and SUN3D datasets. Table 2 shows that our model improves over existing SOTA methods that do not incorporate RANSAC by 6.9% on the cross-scene YFCC test and by 2.96% on the cross-scene SUN3D test. Overall, our model achieves SOTA results over both datasets and descriptors.
4.3 Generalization different descriptors and datasets
We next examined our model's ability to generalize across different descriptors and datasets. For these experiments we used our model trained on the YFCC dataset with SIFT matches. We then tested this model on image pairs from novel scenes (i.e., cross-scene experiment) from the YFCC dataset with matches obtained using ORB and SuperPoint. Additionally, we applied the YFCC/SIFT model to image pairs from the test set of the Phototurisem dataset with matches obtained with SIFT and SuperPoint. The results are shown in Table 3 While our model slightly underperforms the SOTA with ORB matches, it outperforms existing methods with SP matches by almost 3.9%. On

---

Table SuperPoint evaluation. Evaluation of camera pose estimation in experiments on outdoor and indoor datasets using matches obtained with SuperPoint. mAP5 (%) is reported, and the best result each column were included in the training data and cross-scent denote results on image pairs taken from unseen scenes. The first set of methods (above the middle line) includes methods that incorporate RANSAC.
bold. In-scene denote results on novel image pairs taken from scenes that
YFCC(%)
SUN3D(%)
Method
In-scene
Cross-scene
In-scene
Cross-scene
RANSAC
20.36
25.60
21.25
15.89
GC-RANSAC
17.27
22.27
19.32
14.17
LO-RANSAC
17.25
22.12
19.22
14,46
MAGSAC+
18.09
23.47
20.23
15.02
SuperGlue
39.71
57.45
24.09
19.45
Point-Net++
I1.87
17.95
11.40
9.38
DFE
18.79
29.13
13.35
12.04
LFGC
12.18
24.25
12.63
10.68
OA-Net++
29.52
35,27
20.01
15.62
ACNe
26.72
32.98
18.35
13.82
CL-Net
29.35
38.99
15.89
14.03
MS*DG-Net
30.40
37.38
20.28
16.08
U-Match
35.12
45.72
22.73
18.87
Match
38.34
48.80
25.36
21.71
NCMNet
52.20
MGNet
41.53
49.37
24.58
20.65
BCL.et
40.56
48.07
Ours
55.94
59.10
33.97
24.67
Table 3: Generalization across descriptors and datasets. This table shows the results obtained with our model trained the YFCC dataset with SIFT matches applied in inference to the YFCC dataset with the ORB and SuperPoint (SP) feature extractors and to the Tourism dataset with SIFT and SuperPoint (SP). mAP5 (%) is reported, and the best result in each column is marked in bold. & indicates evaluation conducted using published models.
YFCC(%)
PhotoTourism %)
ORB
SIFT
SP
LFGC
7.40
14.78
20.17
5.89
OA-Net++
12.05
19.40
40.39
8.99
CL-Net
14.75
21.00
45.54
9.41
DG-NN-
11.38
21.05
45.53
12.91
U-Match
16.70
28.38
54.43
11.48
NCMNet
19.95
33.20
54.73
30.60
BCINet
18.70
25.85
54.29
23.34
MGNet
20.00
32.88
57.64
20.41
Ours
19.17
37.14
60.81
49.03
generalization to the Phototurism dataset our method performs best with both SIFT and SuperPoint matches.
4.4 Resource utilization
In Table we provide an account of the resources used by our model including the number of parameters, GPU memory usage, and runtime. It can be seen that while our model uses more parameters than NCMNet and BCLNet, which use graph attention architectures. it is 4-6 times faster than these methods and consumes less GPU memory at inference. We further compare our runtime to RANSAC-based methods. With 100K iterations, RANSAC is significantly slower than our method. We note that RANSAC is implemented in CPU. We further considered the recent Kornia's 's GPU implementation of RANSAC for fundamental matrix estimation (1). Using batch size of 10000 samples, their model runtime and maximum GPU memory usage were 40.94ms and 414.66MB, respectively, which are times higher than our model. For GPU resource usage, we used an NVIDIA GeForce RTX 2080Ti and an Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz for CPU.

---

Table 4: Resource utilization. The table shows the resource usage of our model compared to previous methods in terms of number of parameters, GPU memory usage, and runtime. Tested on YFCC with SIFT descriptors. Note that GC-RANSAC and MAGASAC+ are implemented in CPU.
#Params
Max GPU
Runtime Avg-(ms)
Methods
(M)
Mem (MB)
GC-RANSAC
217.3
MAGASAC++
295.29
NCMNet
4.77
174.52
67.43
BCLNet
4.87
140.91
46.94
Ours
22.14
130.75
11.12
4.5 Ablation study
Keypoint denoising. In ablation studies we tested the importance of our noise head, i.c., keypoint denoising process, by training our model with and without this head. In both cases, we used our two stage training scheme. The results in Table 5 demonstrate that our keypoint denoising improves our model performance in both indoor (SUN3D) and outdoor (YFCC) scenes and using both SIFT and SuperPoint matches. This improvement is more noticeable in the more challenging indoor scenario, as it includes fewer inliers and less accurate positions of keypoints.
Two-stage noise-aware training. To test the importance of our two-stage training scheme, we trained model in single stage the original (noisy) of keypoints X, while the noise head muted on the YFCC dataset with SIFT matches. The results are shown 3 (first row vs. second row). The model trained in single stage performs similarly the two-stage trained model in the in-scene generalization test and significantly worse in the cross-scene test.
used correspondence pruning to reduce the effect the outliers' distribution on the final prediction. Specifically in these schemes, matches with the lowest classification scores were removed after each block in their networks. To test the effect of iterative pruning in our model, we implemented a similar scheme, removing half of the input matches with the lowest scores after each block. We test this pruning strategy on model trained YFCC with SIFT matches without keypoint denoising. In contrast to results reported for previous methods, the pruning process had slightly negative effect mAP5 cross-scene score from 65.32% to 64.52%, probably because pruning may also remove some inlier matches. This experiment suggests that our NAC blocks can handle large numbers of outliers successfully without the need for additional pruning.
Correspondences pruning. Previous works [58 28 2
model prediction, decreasing its
Table Ablation studies. Evaluation our model without keypoint denoising and 2-stage training. mAP5% (%) is reported, and the best result in each column and dataset/descriptors is marked in bold. In-scene denote results on novel image pairs taken from scenes that were included in the training data and cross-scene denote results on image pairs taken from unseen scenes.
2-stage
Keypoint denoising
Dataset
Descriptor
In-scene
Cross-scene
training
57.31
59.70
SIFT
58,95
65.32
YFСC
/
60.10
66.14
54.65
58.64
SuperPoint
-
/
55.94
59.10
30.73
23.02
SIFT
-
SUN3D
/
34.15
25.36
SuperPoint
/
-
27.19
21.67
33.97
24.67
Hyper-parameter search. We tested our model with different hyper-parameters, including the number of NAC blocks (which affects the number of DeepSet layers) and the encoder dimension. Using two NAC blocks instead of three reduces the mAP5 to 53.52% and 59.75% for in-scene and

---

cross-scene, respectively. Setting the encoder dimension to 256 instead of 512 reduces the mAP5 to 54.86% and 63.47%. These aaper-parameters search further justify our choices.
4.6 Implementation details
Training, At first, we trained our model on the YFCC dataset with SIFT matches (25 epochs in the noise-free pretraining stage and an additional 10 epochs in the second stage). Then, save on resources, we finetuned the noise-free pretrained YFCC/SIFT model to initialize the training of the rest of the models (SUN3D and YFCC with SuperPoint features), for another epochs with the respective noise-free dataset. Lastly, we train these pretrained models for 10 more epochs using the original (noisy) data. Training was run on an NVIDIA Quadro RTX 6000/ DGX V100/ A40 GPUs, with maximum memory usage of 5GB.
For the loss function set Bialiers 1, Poutters 10, mod and O's 100. We used a threshold of 3 x 10-3 for labeling inliers and outliers. In training. we used the ADAM optimizer with batch size of 32 image pairs and a learning rate of 10 We note that in the noise-free pretraining stage, the predictions of all three blocks are considered in the loss function, whereas in the second stage, only the prediction of the third block is considered.
Architecture details. The network consists of three consecutive NAC blocks, where we only use the output of the last block inference time. The Set Encoders in the NAC blocks combine 12 set layers interleaved with SoftPlus activation, layer normalization, and skip connections in a ResNet-like architecture. We the dimension of the Set Encoder to 512. The Classification and Noise Heads consist of two-layer MIPs interleaved with an activation function. We used a SoftPlus activation for the classification head and LeakyReLU for the noise head. The classification head produces an 2 vector. We apply sigmoid function on the first coordinate to predict Y and use the second coordinate for the weight prediction W.
Conclusion
We presented NACNet, a Noise-Aware Deep Sets framework to estimate relative camera pose, given set of putative matches extracted from two views of a scene. We demonstrated that a position denoising of inliers and noise-free pretraining enable accurate estimation of the essential matrix. Our experiments indicate that our method can handle large numbers of outliers and achieve accurate pose estimation superior to previous methods. We generally observed good cross-dataset and cross- descriptor generalization compared existing methods, but hope further improve on those in future work. In addition, we believe adding a block performing degeneracy test, can further help properly utilizing non-degeee configurations of matches and consequently improve the results of the DLT block. Finally, in future work, we will seek to incorporate our work in multiview structure from motion pipelines.
Acknowledgments and Disclosure of Funding
This research was supported in part by the Israel Science Foundation, grant No. 1639/19, by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center, by the MBZUAI- AAIIIII-SSSIA.. Joint Program for Artificial Intelligence Research and by research grants from the Estates of Bernice Bernath and Marni Josephs Grossman; Joel B. Levey; Tully and Michele Plesser and the Anita James Rosen and Harry Schutzman Foundations.
References
[1] Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski, R.: Building rome in day. Communications of the ACM 54(10), 105-112 (2011)
[2] Barath, D., Matas, J.: Graph-Cut RANSAC. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1733-6741 (2018)
[3] Barath, D., Matas, J., Noskova, J.: MAGSAC: marginalizing sample consensus. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 10197-10205 (2019)
10

---

[4] Barath, D., Noskova, J., Ivashechkin, M., Matas, J.: MAGSAC++, a fast, reliable and accurate robust estimator. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1304-1312 (2020)
[5] Bay, H., Tuytelaars, T., Gool, L.V.: SURF: Speeded up robust features. In: European conference on computer vision. pp. 404-417. Springer (2006)
[6] Bradski, G.: The opencv library. Dr. Dobb's Journal: Software Tools for the Professional Programmer 25(11), 120-123 (2000)
[7] Campos, C., Elvira, R., Rodríguez. JJ.G., Montiel, J.M., Tardós, J.D.; URB-SLAM3: An accurate
open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics 37(6), 1874-1890 (2021)
[8] Chum. O., Matas, J., Kittler, J.: Locally optimized RANSAC. In: Joint Pattern Recognition Symposium. pp. 236-243. Springer (2003)
Chum, O., Werner, T., Matas, J.: Two-view geometry estimation unaffected by dominant plane. In: 2005
IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'OS). OSS). vol. pp. 772-779. IEEE (2005)
[10] Dai, L.., Liu, Y., Ma, J., Wei, L.., Lai, T., Yang, C., Chen, R.: DG-Net: Progressive Correspondence Learning Multiple Sparse Semantics Dynamic Graph. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8973 8982 (2022)
[11] DeTone, D., Malisiewicz, T., Rabinovich, A.: SuperPoint: Self-supervised interest point detection and description. Proceedings the IEEE conference on computer vision and pattern recognition workshops. pp. 224-236 (2018)
[12] Dusmanu, M., Schönberger, J.L., Pollefeys. M.: Multi-view optimization of local feature geometry. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28. 2020, Proceedings, Part 16. pp. 670-68800000 Springer (2020)
[13] Edstedt, J., Bökman, G., Wadenbäck, M., Felsberg, M.: DeDoDe: Detect, Don't Describe Describe. Don't Detect for Local Feature Matching. In: 2024 International Conference 3D Vision (3DV). IEFE (2024)
[14] Edstedt, J., Bökman, G., Zhao, Z: DeDoDe v2: Analyzing and Improving the DeDoDe Kcypoint Detector In: IEEE/CVF Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2024)
[15] Eichhardt, L. Barath, D.: Optimal multi-view correction of local affine frames. arXiv preprint 9X0x51905.00519 (2019)
[16] Fischler, M.A., Bolles, R.C.: Random sample consensus: paradigm for model fitting with applications to
image analysis and automated cartography. Communications of the ACM 24(6), 381-395 (1981)
[17] Hartley, R., Zisserman, A.; Multiple view geometry computer vision. Cambridge university press (2003)
[18] Heinly, J., Schonberger, J.L., Dunn, E., Frahm, J.M.: Reconstructing the world* in six days" captured by the yahoo 100 million image dataset). In: Proceedings of IEEE conference on computer vision and pattern recognition. pp. 3287-3295 (2015)
[19] Jin, Y., Mishkin, D., Mishehuk, A., Matas, J., Fua, P., Yi, K.M., Trulls, E.: Image matching across wide basclines: From paper practice. International Journal of Computer Vision 129(2), 517-547 (2021)
[20] Kasten, Y., Geifiman, A., Galun, M., Basri, R.: Algebraic characterization of essential matrices and their averaging in multiview settings. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 5895-5903 (2019)
[21] Kazerouni, LA.,, Fitzgerald. L.., Dooly, G., Toal, D.: A survey of theh--rtt on visual SLAM. Expert Systems with Applications 205, 117734 (2022)
[22] Khairuddin, A.R., Talib, M.S., Haron, H.: Review on simultancous localization and mapping (SLAM). In: 2015 IEEE international conference on control system, computing and enginecring (ICCSCE). pp. 85-90. IEEE (2015)
[23] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint uxXiv11412.6988 (2014)
[24] Klopschitz, M., Irschara, A., Reitmayr, G., Schmalstieg, D.: Robust incremental structure from motion. In: Proc. 3DPVT. vol. 2, pp. 1-8 (2010)
[25] Li, Z., Zhang, S., Ma, J.: U-Match: Two-view Correspondence Learning with Hicrarchy-awaid Local
Context aggregation. In: International Joint Conference Artificial Intelligence (ICCAI) (2023)
[26] Lindenberger, P., Sarlin, P.E., Larsson, V., Pollefeys, M.: Pixel-perfect structure- www.-rrroottttt motion with fea- turemetric refinement. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5987-5997 (2021)

---

Revisited. In: Conference on Computer Vision and [27] Lindenberger. P., Sarlin, P.E., Pollefeys, M.: LightGlue: Local Feature Matching Light Speed. In: ICCV (2023)
[28] Liu, X., Yang, J.: Progressive Neighbor Consistency Mining for Correspondence Pruning. In; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9527-9537 (2023)
[29] Liu, Y., Liu, L., Lin, C., Dong. Z., Wang. W.: LeamableMottin Coherence for Correspondence pruning. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 3237-3246 (2021)
[30] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2), 91-1 10 (2004)
[31] Luanyuan, D., Du, X., Zhang, H., Tang. J.: MGNet: Learning Correspondences via Multiple Graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 3945-3953 (2024)
[32] Ma, J.. Zhao, J. Jiang, J., Zhou, H., Guo, X.: Locality preserving matching. International Journal of Computer Vision 127(5). 111-531 (2019)
[33] Martinec, D., Pajdla. T.: Robust rotation and translation estimation in multiview reconstruction. In: 2007
IEEE Conference on Computer Vision and Pattern Recognition. pp. 1-8. IEEE (2007)
[34] Moo Yi, K., Trulls, E., Ono, Y., Lepetit, V., Salzmann, M., Fua, P:: Learning to find good correspondenees.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2666 2674
(2018). https://doi.orr org// 10. (09/cvpr2011800222
[35] Moran, D., Koslowsky, H., Kasten, Y.. Maron, H., Galun, M.. Basri, R.: Deep permutation equivariant structure from motion. In: Proceedings the IEEE/CVF International Conference on Computer Vision. pp. 5976-5986 (2021)
[36] Mur-Artal, R., Montiel, J.M.M.. Tardos, J.D.: ORB-SLAM: versatile and accurate monocular slam system. IEEE transactions on robotics 31))), 1147-1163 (2015)
[37] Peyré, G., Cuturi, M., et al.: Computational optimal transport: With applications to data science. Founda- tions and Trends@ in Machine Learning 1 (1(5-)), 355-607 (2019)
[38] Qi. C.R., Yi, L., Su. H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30, 5099-5999199 5108 (2017)
[39] Raguram, R., Chum, O., Pollefeys, M.. Matas, J., Frahm, J.M.: USAC: A universal framework for random
sample consensus. IEEE transactions on pattern analysis and machine intelligence 35(8), 2022 2038 (2012)
[40] Ranftl, R., Koltun, V:: Deep fundamental matrix estimation. In: Proceedings of the European Conference Computer Vision (ECCV). pp. 284-299 (2018)
[41] Riba, E.. Mishkin, D., Ponsa, D., Rublee, E., Bradski, G.: Kormia: an open source differentiable computer vision library for pytorch. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3674-3683 (2020)
[42] Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.; SuperGlue: Learning Feature Matching With Graph Neural Networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR) (June 2020)
[43] Schönberger, J.L.,, Frahm, J.M.:
Pattern Recognition (CVPR) (2016)
[44] Shalev- Shwartz, S., Ben- Ben-David, S.: Understanding machine learning: From theory to algorithms. Cam- bridge university press (2014)
[45] Snavely, N., Seitz, S.M., Szeliski, R.: Modeling world from internet photo collections. International journal of computer vision 80, 189-210 (2008)
[46] Sun, W., Jiang, W., Trulls, E., Tagliasacchi, A., Yi, K.M.: ACNe: Attentive Context Normalization for Robust Permutation- Eqquaaaiinnn Learning. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11286-11299 1295 (2020)
[47] Taketomi, T.. Uchiyama, H., Ikeda. S.: Visual SLAM algorithms: A survey from 2010 to 2016. IPSJ Transactions on Computer Vision and Applications 9(1), 1-11 (2017)
[48] Tang. J.. Kim, H., Guizilini, V., Pillai. S., Ambrus, R.: Neural outlier rejection for self-supervised keypoint learning. arXiv preprint arXiv: 10615 (2019)
[49] Thomee, B., Shamma, D.A., Friellnd,, G., Elizalde, B., Ni, K., Poland, D., Borth, D., L.J.J: YFCC100M: The new data in multimedia research. Communications of the ACM 59(2), 64-73 (2016)
[50] Wagstaff, E., Fuchs, F.B., Engelcke, M., Osborne, M.A., Posner, L:: Universal approximation of functions sets. The Journal of Machine Learning Research 23(1). 3762-6817 (2022)
12

---

[51] Wilson, K., Snavely, N.: Robust global translations with Idsfm. In: Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part III 13. pp. 61-75. Springer (2014)
[52] Xiangyang Miao, Guobao Xiao", S.W., Yu. J.: BCLNet: Bilateral Consensus Learning for Two-Viee -iew Correspondence Pruning. In: Thirty-Eighth AAAI Conference Artificial Intelligence (AAAI). pp. 4225 4232 (2024)
[53] Xiao, J., Owens, A., Torralba, A.: Sun3d: A database of big spaces reconstructed using sfm and object labels. In: Proceedings of the IEEE international conference computer vision. pp. 1625-1632 (2013)
[54] Yi, K.M., Trulls, E., Lepetit, V., Fua, P:: Lift: Learned invariant feature transform. In: European conference
on computer vision. pp. 467-483. Springer (2016)
[55] Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep Sets. Advances in neural information processing systems (2017)
[56] Zhang, J., Sun, D., Luo, Z., Yao, A., Zhou, L.,, Shen, T., Chen, Y., Quan, L., Liao, H.: Learning two-view correspondences and geometry using order-aware network. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5845-5854 (2019)
[57] Zhang. S., Ma, J.: Convmatch: Rethinking network design for two-view correspondence learning. IEEE Transactions Pattern Analysis and Machine Intelligence (2023)
[58] Zhao, C.. Ge, Y., Zhu, F., Zhao, R., Li, H.. Salzmann, M.: Progressive Correspondence Pruning by Consensus Learning, In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp- 6464-6473 (2021)
13

---

A
Appendix
Below, we include further quantitative evaluations, comparison visual results between our method and previous methods, and list of used assets' licenses.
A.1 Denoising evaluation
Below we evaluate the utility of the denoising component of our network. Figure shows the noise distribution before and after our keypoint denoising, measured according to the ground truth essential matrix. The median of the mean reprojection error (over each pair) reduces due to this component by 0.202 pixels and even more (0.246 pixels) for image pairs with pose error (maximum between translation and rotation errors) lower than five degrees.
Figure 5: Denoising evaluation. Reprojection error of inlier keypoints before and after applying our denoising scheme, computed using the ground truth pose. The box plots show the 0.25, 0.5, and 0.75 quantiles. The two left bar plots represent the evaluation over all the image pairs the YFCC dataset. The right two bar plots focus on image pairs whose pose prediction was accurate (i.e., pose error below 5", where the pose error is defined as maximum of the translation and rotation angular errors.). Evaluation was conducted on the YFCC dataset using SIFT descriptors.
3.5
Reprojection (pixels)
1.5
Noisy
Denoised
Noisy
Denoised
Posc Err<5
Pose Err=
A.2
Classification evaluation
Our inlier/outlier classification results are shown in Table "Ground truth" labels are determined by triangulation, discussed in Section 3.2 It can be seen that model achieves higher Fl scores compared to previous methods, possibly explaining our overall improved regression accuracy.
A.3 Qualitative results
Figure/Jshowe results obtained with our method, compared with CLNet and MGNet. Here we use the same indoor and outdoor image pairs shown in the MGNet paper. In Figure 7 we compare our results to NCMNet (using their published checkpoint) on randomly selected image pairs from the YFCC dataset. It can be seen in both figures that our method generally outputs fewer outliers than previous methods.
A.4 License of used assets
YFCC100M dataset images are under common -rrattvveereeve license, and each media file in the dataset is subject to the Creative Commons licenses chosen by their creators/uploaders.
SUN3D dataset[5) is published under MIT license
14

---

Table Classification evaluation. Inlier/outlier classification results on the YFCC dataset and SIFT descriptors. Precision(P), Recall(R), and Fl score are reported.
In-scene
Cross-scene
Methods
P(%)
R(%)
F1(%)
P(%)
R(%)
F1((%)
RANSAC
47.4
52.6
49.9
43.5
50.6
46.8
PointNet++
49.8
86.4
46.6
84.1
59.9
LFGC
56.6
86.3
68.3
54.6
84.7
66.4
OANet++
60.0
89.3
71.8
55.7
85.9
67.6
MSA-Net
61.9
90.5
73.5
58.7
87.9
70.4
CL.ect
76.0
79.2
77.6
75.0
76.4
75.7
MS² DG-Net
63.1
90.9
74.5
59.1
88.4
70.8
ConvMatch
63.0
91.5
74,6
58.7
89.3
70,9
NCMNet
78.4
81.7
79.6
77.0
78.2
77.4
BCLNet
78.4
82.5
79.7
78.3
Ours
84.6
82.9
83.2
82.2
79.1
80.2
Phototourism dataset [9] is published under Creative Commons
Attribution-
NonCommeercial-Sharekre 4.0 International License: CC 4.0
SuperGlue 42 code and weights are published under license for f "ACADEMIC OR NON PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY"
SuperPoint[ [1] code and weights are published under a license for ACCADEMIC OR
PROFPR ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY"
part of our code is adopted from (L which is published under GPL-3.0 license
BCLNet and NCMNet 28 code is published without specifying license.
OPENCV 6 and Kornia are published under Apache License 2.0
15

---

(a) Input
(b) NACNe((Ours)
(c) CLNNt
(d) MGNet
Figure Qualitative results. Visualization results of two-view correspondence pruning on unknown outdoor and indoor scenes. From left to right are the input pairs and the results of NACNet, CLNet, and MGNet, respectively. Inliers are marked in green and outliers are marked in red.
16

---

(a) Input
(b) NACNet(Ours)
(c) NCMNet
Figure 7: Qualitative results. Visualization results of two-view correspondence pruning unknown outdoor scenes. From left to right are the input pairs and the results of NACNet and NCMNet, respectively. Inliers are marked in green and outliers are marked in red.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All of the claims made in the abstract and introduction are described in the method and experiment parts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our model limitations in the conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
There are no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper describes our method's implementation details and training scheme in detail.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
Our code and data are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee //nips.com htte://alllllgiiiee/e//eeeuaaass ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All of our training and test details are written the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The common practice in the outlier removal papers is not report error bars and statistical significance of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: the implementation details section, we indicate the GPU types and memory consumption used in our training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have read the code of ethics, and our research and paper are in line with
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our work does not have any direct societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: There is no risk for misuse of our model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All of the models and data are cited correctly, their license terms of use are respected, and explicitly mentioned in the supplementary.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have yet to release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We didn't use crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We didn't use crowdsourcing or human subject.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "7hBsTfzPEC", "reviewer": "Reviewer_THYF", "summary": "The authors propose a method to tackle the traditional computer vision problem of estimating the essential matrix between two camera views of the same scene based on a set of point matches. The method distinguishes between inlier / outlier matches, and explicitly models the displacement noise in the inlier matches using a “Noise Aware Consensus Network”. The model is trained in two stages: first, the model sees only noise-less inlier matches with outlier matches, and secondly real world blends of noisy inlier and outlier matches. The authors show that this framework is able to compute accurate essential matrices with a variety of different image descriptors over different datasets.", "questions": "It would be great to read the authors’ responses to the issues raised in the weaknesses section.", "limitations": "The authors discussed the limitations of their work well with respect to the requirements in the Checklist.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presented convincing performance as compared to the baselines, especially in the cross-scene / cross dataset settings. This attribute is perhaps most important for practical applications, and shows that the explicit noise reasoning scheme appears to work well. \n\n2. The ablation studies shed light on the contributions of the 2-stage training and keypoint denoising steps. These will help guide future research. \n\n3. The architecture and training details are quite clear. Together with the authors’ promise of releasing the code, this work is a meaningful contribution to the research community. \n\n4. The paper is easy to read and navigate; different parts of the design are motivated clearly.", "weaknesses": "1. It would be important to understand how the capacity of the NACNet architecture compares with the baseline methods - while it indeed appears simpler and more streamlined than pipelines such as BCNet, it would be important to understand whether the complexity has simply been wrapped into higher capacity models.\n\n2. It seems that correspondence pruning should have a large impact on the performance of the model as well, but not a lot of attention is given to this area (e.g. in ablation studies); it seems that only some qualitative results are shown in Fig 4 and in the appendix. However, what is the actual performance of the prediction of Y_i (inlier / outlier labeling) in the different settings (in-scene, cross-scene, cross-dataset, etc)? Does this have a large impact on the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these comments.\n\nW1. Network capacity and complexity:\n\nWe address this question in Table R.1. While our model uses more parameters than NCMNet and BCLNet, which use graph attention architectures, it is 4-6 times faster than these methods and consumes less GPU memory at inference.\n\nW2. Pruning and classification evaluation:\n\nWe discuss the effect of correspondence pruning in our paper (ablation is mentioned in Section 4.4, lines 246-255). Similar to MGNet, we see that pruning has a slight negative effect (~-0.8%) on accuracy. We further attach Table R.3 to show classification accuracy. It can be seen that our model achieves higher F1 scores compared to previous methods, possibly explaining our overall improved regression accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "He38FMXR2k", "reviewer": "Reviewer_HiXm", "summary": "In this work, authors propose NACNet (Noise Aware Consensus network) for the robust essential matrix estimation. For this purpose authors apply DeepSets based architecture that predicts inlier / outlier class as well as inlier displacement error estimation. Authors also propose a two-stage training: (1st) train with noise-free matches by disabling noise-prediction head and (2nd) train on real data with noise.\n\nExtensive experiments with in-scene, cross-scene, cross-dataset, cross-feature generalization and extensive ablation study shows that propose simpler method is superior compared to other deep-learning based consensus methods with complicated architectures.", "questions": "L97: \"Our work is first one to apply keypoint position correction without using a geometric model\" -- Last loss that defines distance between noise-free keypoints and prediction uses \"geometry\" (as in optimal triangulation) to get noise-free points. What does it mean here when authors say \"noise-free\"?\n\nWhat is the size of the model? How fast is the inference? It would be interesting to know how close are we compared to RANSAC in-terms of runtime?", "limitations": "Authors discuss limitation of their approach especially not-perfect generalization to other datasets / features, adding degeneracy test block inside the model.\n\nThis work doesn't have societal impact, so authors ignore that.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, paper is well written with sufficient mathematical rigor to prove authors claim that \"Our network achieves accurate recovery that is superior to existing networks with significantly more complex architectures.\"\n\n(1) Predicting noise in key-points without feature-descriptor is well-motivated and proven to work through ablation study.\n\n(2) Architecture design is simple with DeepSet based 3 NAC block with Model Regression head on top. The network predicts confidence along with inlier-outlier classification (similar to [1]) that allows robust estimation of Essential Matrix via differentiable DLT. \n\n(3) Whole network is end-to-end differentiable.\n\n(4) Benchmarkin &, experiments across different datasets shows good generalization capability of the proposed method. This can have high impact in relative-pose estimation that is essential to multi-view geometry.\n\nReferences\n\n[1] Yi, Kwang Moo, et al. \"Learning to find good correspondences.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.", "weaknesses": "Paper discusses predicting noise-free key-points as an output, however lacks sufficient discussion around difference between noisy and noise-free points.\n\n(1) How does keypoint noise changes feature points? It would have been nicer to see some visualization / empirical matrix of change in key-point locations. of noise-free key-points predicted. \n\n(2) Another interesting fact that is not tested: does the change in key-point position is to fit into the \"Model\" or is it actually becoming sub-pixel accurate as shown in [2]\n\n(3)  Some more comparison when input has less \"outlier\" / more \"outlier\". Empirical evidence around effect of cardinality is also missing. \n\n\nReferences \n\n[2] Lindenberger, Philipp, et al. \"Pixel-perfect structure-from-motion with featuremetric refinement.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these comments.\n\nW1.Denoising evaluation:\n\nWe include a summary box plot (Figure R.1) of the noise distribution before and after our key-point denoising, measured with respect to the ground truth essential matrix. The median of the mean reprojection error (over each pair) is reduced by 0.202 pixels and even more (0.246 pixels) for image pairs with pose error (maximum between the translation and rotation errors) lower than five degrees.\n\nW2. Noise prediction:\n\nIn principle, our denoising module is trained with correspondences cleaned with the ground truth essential matrix, and so it is expected to predict noise-free key-point locations independent of the predicted essential matrix. Moreover, as our ablation indicates (Table 4 in the paper), this denoising process improves the accuracy of the essential matrix regression, indicating that it yields a better fit to true key-point positions rather than to the predicted model. \n\nW3. Performance with different outlier ratios:\n\nWe address this question in Table R.4. As expected, accuracy drops as the fraction of outliers increases or as the number of inliers decreases. Somewhat counter-intuitively, we obtain somewhat low accuracy with small fractions of outliers when SIFT features are used. This occurs because our test data in this segment is small (only 30 pairs).\n\nQ1. Meaning of “noise-free”:\n\nWe apologize for this confusion. Existing methods use the essential matrix estimated *in inference* to correct the positions of key-points. Our network, in contrast, learns to denoise these positions independently from the estimated essential matrix, as we further explained in W2 above. Of course, the noise-free points used *in training* are obtained by applying geometry (triangulation). We will rephrase this statement\n\nQ2. Model size and Inference time:\n\nWe address this question in Table R.1. While our model uses more parameters than NCMNet and BCLNet, which use graph attention architectures, it is 4-6 times faster than these methods and consumes less GPU memory at inference. We further compare our runtime to RANSAC-based methods. With 100K iterations, RANSAC is significantly slower than our method. We note that RANSAC is implemented in CPU. Kornia[1] introduced a GPU RANSAC implementation for fundamental matrix estimation. We tested their resource usage on the same data to demonstrate the potential of GPU implementations for RANSAC. Using a batch size of 10000 samples, their model runtime and maximum GPU memory usage were 40.94ms and 414.66MB, respectively, which are 4 times higher than our model. \n\n[1] Riba, Edgar, et al. \"Kornia: an Open Source Differentiable Computer Vision Library for PyTorch.” Winter Conference on Applications of Computer Vision. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cnFEeyQXGV", "reviewer": "Reviewer_wTxF", "summary": "The paper proposes a deepsets based architecture for essential matrix estimation. The proposed architecture is required to overcome positional nosie, be capable of handling outlier matches which comprise a significant portion of the input data, generalize to unseen image pairs and be capable of handling arbitrary number of point matches.\n\nThe propose archchitecture comes with the following:\n\n  - a two-stage noise-aware training scheme\n\n  - a noise head for predicting positional inlier nosie, and\n\n  - a classification head for inlier/outlier discrimination.\n\nThe architecture consists of \n\n  - Nosie-aware Concensus Blocks (NAC)\n\n    - These are stacks of set encoders (ie Deepsets) used to obtain latent representation of input sets and for classification of points as inliers or outliers.\n\n  The ouput of these blocks are used to estimate the essential matrix.", "questions": "- Can an ablation on the choice of set functions be provided? For instance, recent methods such as [1] and [2] have been shown to perform better than deepsets for modeling interactions over sets and these might provide further performance boost to the proposed architecture.\n\n- Also, how does the number of deepset layers affect performance? [3] shows that special techniques are required to ensure effective learning when multiple deepsets layers are stacked. Curious how this correlates with your experiments.\n\n## References\n\n[1] Lee, Juho, et al. \"Set transformer: A framework for attention-based permutation-invariant neural networks.\" International conference on machine learning. PMLR, 2019.\n\n[2] Bruno, Andreis, et al. \"Mini-batch consistent slot set encoder for scalable set encoding.\" Advances in Neural Information Processing Systems 34 (2021): 21365-21374.\n\n[3] Zhang, Lily, et al. \"Set norm and equivariant skip connections: Putting the deep in deep sets.\" International Conference on Machine Learning. PMLR, 2022.", "limitations": "While no discussion of limitations is provided, i do not see any immediate negative societal impact of this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A simple architecture is proposed that offers better empirical performance compared to current methods.", "weaknesses": "- Ablation on the choice of set functions is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these comments.\n\nQ1.Set transformer ablation:\n\nWe tested the SetTransformer in a quick synthetic experiment. We trained both the SetTransformer and our DeepSet model on noise-free data. While our model achieves a highly accurate pose estimation of 86.52% mAP5 (due to the lack of noise), the SetTransformer model achieves much less accurate results of 66.79% mAP5. We believe the SetTransformer architecture suffers from the lack of global features and a narrow bottleneck in the ISAB block. We leave improvement of the SetTransformer architecture to future work.\n\nQ2. Number of DeepSet layers:\n\nWe provide further ablations in Table R.5, showing that the removal of one NAC block (two blocks instead of three) reduces accuracy by nearly 7%.\n\nThank you also for bringing up the work of Zhang et. al [3]. Despite some architectural differences, our implementation respects the principle of keeping a “clean path” for the gradient descent (e.g., by utilizing skip connections in a similar design as in [3]), enabling the usage of multiple DeepSet layers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBYLHoooRH", "reviewer": "Reviewer_1GwE", "summary": "This work proposes a simpler yet effective network architecture based on Deep Sets for the estimation of essential matrices.\n\nThe method is based on three key properties:\n\n1. A two-stage noise-aware training scheme first uses noise-free inlier matches for training, and then uses original matches and full loss for training;\n\n2. A noise head predicts the displacement noise in inlier matches.\n\n3. A classification head classifies point matches as either inliers or outliers.\n\nEvaluations show that the method outperforms previous methods in terms of accuracy and generalization.", "questions": "1. While the DeepSets-based architecture is shown to be effective, traditional fully convolutional networks, such as UNet, are also applicable. Is there any strong reason why a DeepSets-based network is preferred and explored first?\n\n2. The authors remove half of the input matches to test the effect of outlier pruning. Is this approach reasonable? Is this value adjustable? It would be better to add an ablation study showing performance changes as the number of removed matches varies. Furthermore, this is an indirect method to validate the effectiveness of the proposed inlier/outlier classification head. Why was the ablation study not conducted with and without this head?\n\n3. While the paper provides an ablation study on the proposed key properties, it lacks an ablation of the network architecture and hyperparameters, such as the number of set layers, the dimension of the Set Encoder, etc, but this would also be valuable to know — for example, how much of an impact do these parts of the network have on the final results?\n\n4. The paper lacks details on applying the Optimal Triangulation Method to set inlier/outlier labels and obtain ground truth, noise-free inlier keypoints. Including these details would help readers better understand the training process.\n\n5. What is the unit of the y-axis in Fig. 3?\n\n6. To the best of my knowledge, the ground truth poses in the SUN3D dataset are inaccurate (see supplemental material in {1}). Why did the authors use this dataset solely to follow the evaluation protocol of previous methods? For indoor scenes, is it more suitable to use ScanNet dataset {2} for evaluation as [42]?\n\n7. While the paper claims that the network is simpler than existing methods, the multiple NAC blocks and the noise regression module may introduce a certain level of complexity. It would be better to provide experiments and discussions on the computational efficiency of the proposed method compared to existing methods, including runtime and model parameter scale, which are critical for real-world applications.\n\n{1} Bellavia, Fabio. “Image Matching by Bare Homography.” IEEE Transactions on Image Processing 33 (2023): 696-708.\n{2} Dai, Angela, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser and Matthias Nießner. “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 2432-2443.", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed network architecture is simple yet effective.\n\n2. NAC block represents denoising and inlier/outlier classification effects, well validated through a toy example, i.e., a line-fitting task in Fig. 2.\n\n3. The key properties proposed are shown to be effective in the ablation study.\n\n4. The paper is well-structured and is generally easy to follow.", "weaknesses": "1. While the DeepSets-based architecture is shown to be effective, traditional fully convolutional networks, such as UNet, are also applicable. Is there any strong reason why a DeepSets-based network is preferred and explored first?\n\n2. The authors remove half of the input matches to test the effect of outlier pruning. Is this approach reasonable? Is this value adjustable? It would be better to add an ablation study showing performance changes as the number of removed matches varies. Furthermore, this is an indirect method to validate the effectiveness of the proposed inlier/outlier classification head. Why was the ablation study not conducted with and without this head?\n\n3. While the paper provides an ablation study on the proposed key properties, it lacks an ablation of the network architecture and hyperparameters, such as the number of set layers, the dimension of the Set Encoder, etc, but this would also be valuable to know — for example, how much of an impact do these parts of the network have on the final results?\n\n4. The paper lacks details on applying the Optimal Triangulation Method to set inlier/outlier labels and obtain ground truth, noise-free inlier keypoints. Including these details would help readers better understand the training process.\n\n5. What is the unit of the y-axis in Fig. 3?\n\n6. To the best of my knowledge, the ground truth poses in the SUN3D dataset are inaccurate (see supplemental material in {1}). Why did the authors use this dataset solely to follow the evaluation protocol of previous methods? For indoor scenes, is it more suitable to use ScanNet dataset {2} for evaluation as [42]?\n\n7. While the paper claims that the network is simpler than existing methods, the multiple NAC blocks and the noise regression module may introduce a certain level of complexity. It would be better to provide experiments and discussions on the computational efficiency of the proposed method compared to existing methods, including runtime and model parameter scale, which are critical for real-world applications.\n\n{1} Bellavia, Fabio. “Image Matching by Bare Homography.” IEEE Transactions on Image Processing 33 (2023): 696-708.\n{2} Dai, Angela, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser and Matthias Nießner. “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 2432-2443.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these comments.\n\nQ1. UNet:\n\nOur network obtains as input an unordered set of point matches of apriori unknown cardinality. Such input is naturally suitable for DeepSets and other permutation equivariant architectures. Standard convolutional and the UNet architecture cannot handle such input, since it requires input on a grid. Graph convolutional architectures can be applied (and indeed are applied in previous methods, including CLNet, NCMNet, and also U-Match, which uses a graph UNet architecture). These methods, however, are more complex and, as our experiments show, are slower (see the attached Table R.1) and yield inferior accuracies (see Tables 1-2 in the paper).\n\nQ2. Pruning amount:\n\nFor our ablation, we followed previous approaches, including CLNet, NCMNet, and BCLNet, which pruned half of the input matches. MGNet further performed an ablation study with different pruning ratios and showed that pruning reduces the accuracy of the predictions. We should emphasize that our confidence scores (Eq. 2 in the paper) perform “soft pruning,” which appears to be more effective than “hard pruning.”\n\nAblating the classification head:\n\nWe evaluate the performance of our model without the classification head. Due to the prevalence of outliers, performance deteriorates to chance levels (0.014% and 0.025% mAP5 score for in-scene and cross-scene, respectively). We further demonstrate the effectiveness of our classification head by providing its precision, recall, and F1 scores (Table R.3).\n\nQ3. Hyperparameters ablation studies:\n\nWe include in Table R.5 an ablation of the number of NAC blocks (which affects the number of DeepSet layers) and the encoder dimension. These ablations further justify our choices.\n\nQ4. Optimal Triangulation Method:\n\nWe will add these details to the paper.\n\nQ5. Y-axis units in Fig. 3:\n\nThe Y-axis represents the number of image pairs. \n\nQ6. ScanNet evaluation:\n\nIndeed, we used SUN3D to follow the evaluation protocol of previous methods. For the rebuttal, we include in Table R.2 the results of cross-dataset generalization on ScanNet. We compared our results to NCMNet predictions. Our method achieves superior results on this highly challenging dataset. Since previous methods have not published their model trained on SUN3D, we trained NCMNet from scratch using the official code release.\n\nQ7. Resources usage:\n\nWe address this question in Table R.1. While our model uses more parameters than NCMNet and BCLNet, which use graph attention architectures, it is 4-6 times faster than these methods and consumes less GPU memory at inference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their comments.

We addressed each of your questions individually. Please note the attached pdf.

Meta-review: All reviewers were initially positive and remained positive after the rebuttal. Reviewers commented on the good results, and as an application paper, this paper is worth noting. The AC thus follows this unanimous recommendation and recommends accepting the paper. A note that was made by one of the reviewers during the discussions was that the paper could have had a larger impact, if the paper evaluated beyond essential matrices, e.g., other robust estimation problems. But without it, the AC agrees that the paper's impact could be limited as while this is a plausible hypothesis, it remains untested.
Decision: Accept (poster)