Consensus Learning with Deep Sets for Essential Matrix Estimation

Authors: dror  moran
yuval  margalit
guy  trostianetsky
fadi  khatib
meirav  galun
ronen  basri
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:19:54

Abstract: Robust estimation of the essential matrix, which encodes the relative position and orientation of two cameras, is a fundamental step in structure from motion pipelines. Recent deep-based methods achieved accurate estimation by using complex network architectures that involve graphs, attention layers, and hard pruning steps. Here, we propose a simpler network architecture based on Deep Sets. Given a collection of point matches extracted from two images, our method identifies outlier point matches and models the displacement noise in inlier matches. A weighted DLT module uses these predictions to regress the essential matrix. Our network achieves accurate recovery that is superior to existing networks with significantly more complex architectures.

Main paper: 
 introduction: 
 Estimating the relative pose of two cameras depicting a stationary scene is a fundamental computer vision task and a basic step in multiview structure from motion (SFM) [33, 24, 45, 1, 51, 43, 20, 26, 35] and simultaneous localization and mapping (SLAM) [22, 47, 7, 21] pipelines. Both classical and recent deep network-based algorithms (see a review in Section 2) use point matches to compute the essential matrix, which encodes the relative position and orientation of the two cameras. Identifying such point matches by existing heuristics, however, is prone to mistakes, due to possibly large viewpoint changes, illumination differences, and the presence of ambiguous repetitive scene structures, resulting in noisy matches and extremely large numbers of outlier matches (often as many as 95%) that must be identified and pruned to enable accurate pose recovery.Classical SFM algorithms use RANSAC [16] to robustly identify inliers and estimate pose parameters. While RANSAC has been used effectively for consensus recovery, learning-based deep network approaches have introduced a competitive alternative, making steady progress in accuracy while allowing for efficient inference and demonstrating resilience to very large fractions of outliers. This progress was obtained at the price of complicating the network architecture, e.g., using message passing in local, near-neighbour graphs [58, 28, 31, 52] or expensive attention (transformer) layers [28, 52] , along with the addition of hard pruning steps [58, 28, 52] .In this paper, we introduce a simpler network architecture for consensus learning based on the Deep Sets framework [55] . Deep Sets architectures are based on shared, element-wise layers that are combined with global features produced by summing the element-wise features. Zaheer et al. and others [55, 50] proved that such architectures can express universal permutation-equivariant functions over sets. In our network, the input set elements include pairs of keypoints, i.e., the coordinates of matching pairs of points. In each layer, element-wise features are produced by a linear layer with shared weights, followed by SoftPlus activation. Global features are obtained by averaging the element-wise features, where averaging is used to maintain invariance to set cardinality. The network utilizes a stack of such permutation equivariant layers to classify point matches as either inliers or outliers and identify a consensus set to enable accurate relative camera pose regression. We further improve accuracy by integrating a noise regression module that aims to predict the displacement, due to noise, of the (clean) positions of inlier keypoints. Finally, we observe that training in two stages, i.e., first on a noise-free version of the real data (while including the outliers) and subsequently on the original real data, improves the accuracy of the predicted pose. Our network achieves accurate pose recovery that is superior to existing networks with significantly more complex architectures.In summary, our contributions include:• NACNet, a Noise Aware Consensus Network, for consensus learning tasks and robust geometric model estimation. • A DeepSets based architecture that includes inlier displacement error estimation.• An effective noise-free pretraining scheme: first, pretrain on a denoised version of the real data, then train on the real (noisy) data. • Experiments demonstrate that NACNet achieves superior results compared to baselines on indoor and outdoor image pairs applied on various descriptors.Our code is available at https://github.com/drormoran/NACNet.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: All of the claims made in the abstract and introduction are described in the method and experiment parts. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Classical methods. RANSAC [16] and its successors, LO-RANSAC [8] , USAC [39] , MAGSAC [3] , and MAGSAC++ [4] search over minimal point configurations to find consensus sets from noisy and corrupted data and estimate a corresponding parametric model. These are applied to matched keypoints with distinct descriptors obtained by filtering with Lowe's ratio test [30] . These classical methods are regarded as the standard solutions for finding consensus in data consisting of mixtures of inlier and outlier point matches.Learning-based methods. Deep learning-based methods have been used recently to regress a geometric model and outlier classification. DFE [40] used a deep-based iteratively reweighted least squares (IRLS) scheme to predict inlier/outlier scores. LFGC [34] utilized an architecture that involves an inlier/outlier classifier and weight sharing, followed by context normalization, and applied a geometric loss to the output of the weighted 8-point algorithm (also called weighted DLT [17] ).Follow-up works improve prediction results by introducing more complex network designs. OANet [56] introduced an order-aware block, which contains differentiable permutation invariant pooling and unpooling operators that capture local context by utilizing soft clustering of correspondences in the feature space. CLNet [58] used this order-aware block together with pruning and local-to-global consensus learning procedure strategy to classify the correspondences by employing convolutions on local and global graphs built based on the Euclidean distance in feature space. All the methods mentioned above suffer from the leakage of outliers to the consensus set. Consequently, they all use RANSAC at the end of their inference step. In contrast, NCMNet [28] , MGNet [31] , and BCLNet [52] used weighted DLT also at inference, showing that the performance of the results is not improved further when RANSAC is applied in addition. NCMNet [28] proposed a local-to-global consensus learning scheme in which it first creates a local spatial graph, then a local feature space graph, and finally a global graph based on the inlier scores from the local graphs. BCLNet [52] introduced the idea of Bilateral Consensus, adopting the local graph from CLNet [58] as their projection step in a channel-wise transformer that learns global consensus. MGNet [31] used a similar scheme, building both implicit and explicit local graphs and a global graph. Unlike previous methods, this method does not prune correspondences inside the network.In contrast to these methods, we use an architecture based on Deep Sets [55] . Deep Sets enable efficient information transfer between the point matches through global features without the need to construct and manipulate graphs. Our newly proposed noise regression module further improves our results. Finally, as with recent methods, our method too does not require a final RANSAC step.Learned feature matching. Deep learning-based detectors and descriptors [11, 54, 13, 14] based on both CNNs and Transformers have been used in recent years to replace the handcrafted features [ 30, 36, 5] used in classical methods. Those methods trained with challenging and diverse data have improved the accuracy and robustness of matching even with the classical nearest neighbor matching.Yet the main problems of high outlier rate remained. Consequently, learned matchers that match keypoints while rejecting non-matchable ones have merged [42, 27] , combining Transformers with optimal transport [37] to produce more accurate matches even with large camera movement. These matchers rely on the descriptors for the matching and keypoint rejection and require RANSAC as a post-processing step. In contrast, our method, similar to [28] , [31] , [52] , gets as an input the keypoints (point correspondences) only and does not incorporate RANSAC.Keypoint refinement. Previous works [15, 48, 12, 26] have shown that correcting keypoints position could positively influence the results of geometric model estimation. All of those works use visual and learned features (SIFT descriptors or features obtained from applying a convolutional network to the input images) to correct the positions. To our knowledge, our paper is the first to use the input point set directly to correct keypoint position, as opposed to previous works, which rely on either an estimated geometric model or visual features.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss our model limitations in the conclusion section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Consider a pair of images captured by (internally) calibrated cameras expressed with 3 × 4 matrices, P = [I, 0] and P = [R, t], where R and t respectively denote the relative rotation and translation between the two views. The essential matrix E = [t] × R, determines the epipolar geometry between the two views, so that for any two corresponding points, p and p, projected from a 3D point, it holds that pT Ep = 0. Existing algorithms commonly estimate the essential matrix directly from a set of putative matches between the two views, i.e., pairs of keypoints.Our aim in this work is to construct a network that identifies a consensus set of point matches (a set of inliers), given a set of putative matches as input (generally contaminated with noise and outliers), and, based on this consensus set, predicts the essential matrix between the two images. We seek to construct a network that can overcome positional noise, which can reside in the inlier matches, and cope with a considerable fraction of outlier matches, up and above 95%. In addition, we aim for a method that can generalize to unseen image pairs and work with a varying number of point matches and a variety of fractions of outliers.Those goals are achieved by employing a permutation-equivariant network architecture with the following key properties: (1) a two-stage noise-aware training scheme, (2) a noise head for predicting positional inlier noise, and (3) a classification head to discriminate between inliers and outliers. These three key properties are at the core of our method. Hence, we refer to our network as a Noise-Aware Consensus Network (NACNet). Formally, let X = {x 1 , . . . , x n } ⊂ R 4 denote a set of point correspondences, with x i = (p i , q i , pi , qi ) denoting a match between a keypoint p i = (p i , q i ) in the left image and a keypoint pi = (p i , qi ) in the right image. Our aim is, given X as an input to classify each matching pair x i as an inlier (y i = 1) or outlier (y i = 0) and associate with it an (inlier) confidence score C i ∈ R. Those predictions are used to estimate the essential matrix that relates the two views.Network architecture. Our network comprises of three noise-aware consensus (NAC) blocks. Each NAC block uses a set encoder to map the input matches to a latent representation and to correct their positions due to positional noise. The third block further classifies the points as inliers or outliers and produces their corresponding confidence scores. Its outputs feed the model regression block, which implements a weighted differentiable Direct Linear Transformation (DLT) algorithm [34] , based on the confidence scores, to predict the essential matrix. We refer the reader to Figure 1 for a detailed scheme of our network architecture.Noise aware consensus (NAC) block. Each NAC block comprises of a set encoder, a noise head, and a classification head. The set encoder uses DeepSets layers to map the coordinates of the input point matches to a latent representation L ∈ R n×d . Each DeepSets layer includes a linear, permutation equivariant layer followed by SoftPlus activation. These layers apply a linear transformation to each set member and an additional (different) linear transformation to their average. (We replace the sum in [55] with an average to maintain invariance to set cardinality.)The noise and classification heads are implemented with simple two-layer MLPs. The noise head uses the latent representation to predict displacement vectors for all input points, δ ∈ R n×4 . These displacement vectors are subtracted from the input points, X, producing their predicted denoised locations, X. The classification head uses as input the latent representation and outputs predictions for the inlier/outlier classification labels Ŷ ∈ [0, 1] n with their corresponding weights W ∈ R n .The predicted denoised version of the keypoints X and the latent representation L are passed to the next NAC block. In the third block, the inlier/outlier predicted labels Ŷ , the weights W , and the denoised keypoints X are passed to the model regression block.We demonstrate the NAC block denoising effect on a simple line-fitting task. We randomly sample 100 noisy points on a line and, in addition, 900 outliers. An example is shown in Figure 2 , where our NACNet significantly reduces the positional noise in the inlier points.Model regression block. The model regression block uses the predicted weights, W , and the classification labels, Ŷ , obtained from the classification head, and the denoised version of the keypoints, X, obtained from the noise head to predict the essential matrix in the following way.EQUATIONwhere g denotes the differentiable DLT algorithm (also called the weighted eight-point algorithm, see formulation in [34] , Section 3). Similarly to [46] , we calculate confidence scores as followsEQUATIONThe confidence scores are used as the weights corresponding to the denoised keypoints X in the weighted DLT algorithm.We minimize a loss composed of three termsEQUATIONThe first term L cls uses a weighted binary cross entropy loss, due to the imbalanced of the inliers and outliers in the data, to penalize for inlier/outlier classification errorsEQUATIONHere, n is the cardinality of the keypoint set, X, and β inliers and β outliers are determined by a hyperparameter search.The second term L mod penalizes for errors in the predicted essential matrix, similarly to the suggestion in [40] . We start with a grid of k point pairs and use the Optimal Triangulation Method (OTM) [17] (page 318) to find the closest points that satisfy the ground truth epipolar constraints. Specifically, given a pair of grid points (p i , pi ), OTM seeks to find the global minimum for the following optimization problem:EQUATIONwhere E is the ground truth essential matrix, and d is the Euclidean distance between the points. Given the optimal points {q i , qi } k i=1 , we define the loss using the Symmetric Epipolar Distance:EQUATION)where Ê is the predicted essential matrix and k = 400.The last term L ns is used to minimize the distance between the noise-free keypoints, X, and the predicted denoised version of the keypoints, X, over the set of the ground truth inliers, as followsEQUATIONTo determine the ground truth, noise-free inlier keypoints, Xinliers , we apply the Optimal Triangulation Method ( [17] , page 318). The parameters α mod and α ns are determined by a hyperparameter search.Training our model to remove outlier matches is complicated by the presence of noise in the positions of inlier matches, potentially resulting in a small classification margin. This, in turn, has been shown (in the case of kernel SVM) to have a negative effect on sample complexity and generalization error [44] (Pages 205-206, 221). A further complication is the lack of ground truth labels; i.e., our inlier/outlier labels are set by applying a preset threshold to the deviation of the points from the projections derived by the Optimal Triangulation Method ( [17] , page 318)(see Section 4.1).To approach this problem, we train our model by applying a two-stage, noise-aware optimization process. The input to the first stage includes the set X containing the noise-free inlier matches along with the outlier matches. The optimization in this stage, therefore, uses only the first two terms of the loss (3), and the noise head is muted. In the second stage, the input to the network includes the original set of keypoints X, and the full loss, i.e., including (7) , is optimized. Our experiments and ablations indicate that this two-stage training process significantly improves the performance of our method. Datasets. We trained and tested our method on both indoor and outdoor datasets. For an outdoor dataset, we used Yahoo's YFCC dataset [49] , which contains 100 million images from flicker later reconstructed using SFM [18] . For an indoor dataset, we used the SUN3D [53] . For both datasets, we used the same preprocessing and dataset split as in [56] , i.e., the camera poses are extracted from an SFM pipeline, and the test set is split into in-scene and cross-scene generalization. In contrast to previous methods that use the Symmetric Epipolar Distance for "ground truth" inlier/outlier labeling, we determined the labels by the deviation of the points from the projections derived by the Optimal Triangulation Method ( [17] , page 318) using a threshold of 3 × 10 -3 . In practice, changing the labeling paradigm did not affect the results. Additionally, we used the Phototourism dataset [19] to test our model's generalization across datasets. For keypoint detection, we used SIFT [30] , ORB [36] ,and SuperPoint [11] followed by the preprocessing steps suggested in [58] . As is shown in Figure 3 , consensus learning on these datasets is highly challenging due to the high fraction of outliers in all datasets and with all descriptors.Baselines. We compare our methods with RANSAC [16] , DEGENSAC [9] , GC-RANSAC [2] , MAGSAC++ [4] , PointNet++ [38] , DFE [40] , LFGC [34] , OA-Net [56] , ACNe [32] , LMC-Net [29] , CL-Net [58] , MS 2 DG-Net [10] , ConvMatch [57] , U-Match [25] , NCMNet [28] , MGNet [31] , BCLNet [52] ,and SuperGlue [42] . All the evaluations of deep learning-based methods are taken from their respective papers unless specifically stated otherwise. We used the official SuperGlue repository for evaluation on SuperPoint, and the paper [42] results for evaluation on SIFT. For the RANSAC-based methods [2, 4, 9, 16] , we set the maximal number of iterations to 100K and use Lowe's ratio test [30] to filter the initial matches, with a threshold tuned differently for the SIFT and SuperPoint descriptors to maximize performance.Evaluation metrics. We use the mean average precision (mAP) to evaluate our model predictions as suggested in [34] . We compute the mAP over the maximum between the translation and rotation angular errors of our predicted essential matrix up to the threshold of 5 • .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: Our code and data are publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
essential matrix estimation: 
 Our results are shown in 1 ), our model outperforms the other methods by a significant margin in the in-scene generalization task and with a smaller margin in the cross-scene generalization task. Likewise, on the SUN3D dataset, our method outperforms the other methods in both in-scene and cross-scene generalization, improving over the previous SOTA by 3.6% in the cross-scene test. Qualitative results can be seen in Figure 4 and in Appendix A.3. We also tested our model using the deep-learning based descriptor SuperPoint on both the YFCC and SUN3D datasets. Table 2 shows that our model improves over existing SOTA methods that do not incorporate RANSAC by 6.9% on the cross-scene YFCC test and by 2.96% on the cross-scene SUN3D test. Overall, our model achieves SOTA results over both datasets and descriptors.We next examined our model's ability to generalize across different descriptors and datasets. For these experiments we used our model trained on the YFCC dataset with SIFT matches. We then tested this model on image pairs from novel scenes (i.e., cross-scene experiment) from the YFCC dataset with matches obtained using ORB and SuperPoint. Additionally, we applied the YFCC/SIFT model to image pairs from the test set of the Phototurisem dataset with matches obtained with SIFT and SuperPoint. The results are shown in generalization to the Phototurism dataset our method performs best with both SIFT and SuperPoint matches.In Table 4 , we provide an account of the resources used by our model including the number of parameters, GPU memory usage, and runtime. It can be seen that while our model uses more parameters than NCMNet and BCLNet, which use graph attention architectures, it is 4-6 times faster than these methods and consumes less GPU memory at inference. We further compare our runtime to RANSAC-based methods. With 100K iterations, RANSAC is significantly slower than our method. We note that RANSAC is implemented in CPU. We further considered the recent Kornia's GPU implementation of RANSAC for fundamental matrix estimation [41] . Using a batch size of 10000 samples, their model runtime and maximum GPU memory usage were 40.94ms and 414.66MB, respectively, which are 4 times higher than our model. For GPU resource usage, we used an NVIDIA GeForce RTX 2080Ti and an Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz for CPU. Correspondences pruning. Previous works [58, 28, 52] used correspondence pruning to reduce the effect of the outliers' distribution on the final prediction. Specifically in these schemes, matches with the lowest classification scores were removed after each block in their networks. To test the effect of iterative pruning in our model, we implemented a similar scheme, removing half of the input matches with the lowest scores after each block. We test this pruning strategy on a model trained on YFCC with SIFT matches without keypoint denoising. In contrast to results reported for previous methods, the pruning process had a slightly negative effect on our model prediction, decreasing its mAP5 • cross-scene score from 65.32% to 64.52%, probably because pruning may also remove some inlier matches. This experiment suggests that our NAC blocks can handle large numbers of outliers successfully without the need for additional pruning. Hyper-parameter search. We tested our model with different hyper-parameters, including the number of NAC blocks (which affects the number of DeepSet layers) and the encoder dimension. Using two NAC blocks instead of three reduces the mAP5 • to 53.52% and 59.75% for in-scene and cross-scene, respectively. Setting the encoder dimension to 256 instead of 512 reduces the mAP5 • to 54.86% and 63.47%. These hyper-parameters search further justify our choices.Training. At first, we trained our model on the YFCC dataset with SIFT matches (25 epochs in the noise-free pretraining stage and an additional 10 epochs in the second stage). Then, to save on resources, we finetuned the noise-free pretrained YFCC/SIFT model to initialize the training of the rest of the models (SUN3D and YFCC with SuperPoint features), for another 5 epochs with the respective noise-free dataset. Lastly, we train these pretrained models for 10 more epochs using the original (noisy) data. Training was run on an NVIDIA Quadro RTX 6000/ DGX V100/ A40 GPUs, with a maximum memory usage of 5GB.For the loss function we set β inliers = 1, β outliers = 10, α mod = 1 and α ns = 100. We used a threshold of 3 × 10 -3 for labeling inliers and outliers. In training, we used the ADAM [23] optimizer with a batch size of 32 image pairs and a learning rate of 10 -4 . We note that in the noise-free pretraining stage, the predictions of all three blocks are considered in the loss function, whereas in the second stage, only the prediction of the third block is considered.Architecture details. The network consists of three consecutive NAC blocks, where we only use the output of the last block at inference time. The Set Encoders in the NAC blocks combine 12 set layers interleaved with SoftPlus activation, layer normalization, and skip connections in a ResNet-like architecture. We set the dimension of the Set Encoder to 512. The Classification and Noise Heads consist of two-layer MLPs interleaved with an activation function. We used a SoftPlus activation for the classification head and a LeakyReLU for the noise head. The classification head produces an n × 2 vector. We apply a sigmoid function on the first coordinate to predict Ŷ and use the second coordinate for the weight prediction W . 
conclusion: 
 We presented NACNet, a Noise-Aware Deep Sets framework to estimate relative camera pose, given a set of putative matches extracted from two views of a scene. We demonstrated that a position denoising of inliers and noise-free pretraining enable accurate estimation of the essential matrix.Our experiments indicate that our method can handle large numbers of outliers and achieve accurate pose estimation superior to previous methods. We generally observed good cross-dataset and crossdescriptor generalization compared to existing methods, but hope to further improve on those in future work. In addition, we believe adding a block performing degeneracy test, can further help properly utilizing non-degenerate configurations of matches and consequently improve the results of the DLT block. Finally, in future work, we will seek to incorporate our work in multiview structure from motion pipelines. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All of our training and test details are written in the experiments section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The common practice in the outlier removal papers is not to report bars and statistical significance of the results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have read the code of ethics, and our research and paper are in line with it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work does not have any direct societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: Below, we include further quantitative evaluations, a comparison of visual results between our method and previous methods, and a list of used assets' licenses.Below we evaluate the utility of the denoising component of our network. Figure 5 shows the noise distribution before and after our keypoint denoising, measured according to the ground truth essential matrix. The median of the mean reprojection error (over each pair) reduces due to this component by 0.202 pixels and even more (0.246 pixels) for image pairs with pose error (maximum between the translation and rotation errors) lower than five degrees. The right two bar plots focus on image pairs whose pose prediction was accurate (i.e., pose error below 5 • , where the pose error is defined as the maximum of the translation and rotation angular errors.). Evaluation was conducted on the YFCC dataset using SIFT descriptors.Our inlier/outlier classification results are shown in Table 6 . "Ground truth" labels are determined by triangulation, as discussed in Section 3.2. It can be seen that our model achieves higher F1 scores compared to previous methods, possibly explaining our overall improved regression accuracy.Figure 6 shows results obtained with our method, compared with CLNet and MGNet. Here we use the same indoor and outdoor image pairs shown in the MGNet paper. In Figure 7 we compare our results to NCMNet (using their published checkpoint) on randomly selected image pairs from the YFCC dataset. It can be seen in both figures that our method generally outputs fewer outliers than previous methods.• YFCC100M dataset [49] images are under a common-creative license, and each media file in the dataset is subject to the Creative Commons licenses chosen by their creators/uploaders.• SUN3D dataset [53] is published under MIT license. • Phototourism dataset [19] is published under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License: CC BY-NC-SA 4.0 • SuperGlue [42] code and weights are published under a license for:"ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY" • SuperPoint [11] code and weights are published under a license for:"ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY" • part of our code is adopted from CLNet [58] which is published under GPL-3.0 license.• BCLNet [52] and NCMNet [28] code is published without specifying a license.• OPENCV [6] and Kornia [41] are published under Apache License 2.0 NeurIPS Paper ChecklistJustification: There is no risk for misuse of our model.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All of the models and data are cited correctly, their license terms of use are respected, and explicitly mentioned in the supplementary.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 


Decision: All reviewers were initially positive and remained positive after the rebuttal. Reviewers commented on the good results, and as an application paper, this paper is worth noting. The AC thus follows this unanimous recommendation and recommends accepting the paper. A note that was made by one of the reviewers during the discussions was that the paper could have had a larger impact, if the paper evaluated beyond essential matrices, e.g., other robust estimation problems. But without it, the AC agrees that the paper's impact could be limited as while this is a plausible hypothesis, it remains untested.

Reviews: review: paper_summary: paper_summary: The authors propose a method to tackle the traditional computer vision problem of estimating the essential matrix between two camera views of the same scene based on a set of point matches. The method distinguishes between inlier / outlier matches, and explicitly models the displacement noise in the inlier matches using a “Noise Aware Consensus Network”. The model is trained in two stages: first, the model sees only noise-less inlier matches with outlier matches, and secondly real world blends of noisy inlier and outlier matches. The authors show that this framework is able to compute accurate essential matrices with a variety of different image descriptors over different datasets.
strength_weakness: strength_weakness: 1. It would be important to understand how the capacity of the NACNet architecture compares with the baseline methods - while it indeed appears simpler and more streamlined than pipelines such as BCNet, it would be important to understand whether the complexity has simply been wrapped into higher capacity models.

2. It seems that correspondence pruning should have a large impact on the performance of the model as well, but not a lot of attention is given to this area (e.g. in ablation studies); it seems that only some qualitative results are shown in Fig 4 and in the appendix. However, what is the actual performance of the prediction of Y_i (inlier / outlier labeling) in the different settings (in-scene, cross-scene, cross-dataset, etc)? Does this have a large impact on the performance?
questions: questions: It would be great to read the authors’ responses to the issues raised in the weaknesses section.
limitations: limitations: The authors discussed the limitations of their work well with respect to the requirements in the Checklist.
paper_summary: paper_summary: In this work, authors propose NACNet (Noise Aware Consensus network) for the robust essential matrix estimation. For this purpose authors apply DeepSets based architecture that predicts inlier / outlier class as well as inlier displacement error estimation. Authors also propose a two-stage training: (1st) train with noise-free matches by disabling noise-prediction head and (2nd) train on real data with noise.

Extensive experiments with in-scene, cross-scene, cross-dataset, cross-feature generalization and extensive ablation study shows that propose simpler method is superior compared to other deep-learning based consensus methods with complicated architectures.
strength_weakness: strength_weakness: Paper discusses predicting noise-free key-points as an output, however lacks sufficient discussion around difference between noisy and noise-free points.

(1) How does keypoint noise changes feature points? It would have been nicer to see some visualization / empirical matrix of change in key-point locations. of noise-free key-points predicted. 

(2) Another interesting fact that is not tested: does the change in key-point position is to fit into the "Model" or is it actually becoming sub-pixel accurate as shown in [2]

(3)  Some more comparison when input has less "outlier" / more "outlier". Empirical evidence around effect of cardinality is also missing. 


References 

[2] Lindenberger, Philipp, et al. "Pixel-perfect structure-from-motion with featuremetric refinement." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
questions: questions: L97: "Our work is first one to apply keypoint position correction without using a geometric model" -- Last loss that defines distance between noise-free keypoints and prediction uses "geometry" (as in optimal triangulation) to get noise-free points. What does it mean here when authors say "noise-free"?

What is the size of the model? How fast is the inference? It would be interesting to know how close are we compared to RANSAC in-terms of runtime?
limitations: limitations: Authors discuss limitation of their approach especially not-perfect generalization to other datasets / features, adding degeneracy test block inside the model.

This work doesn't have societal impact, so authors ignore that.
paper_summary: paper_summary: The paper proposes a deepsets based architecture for essential matrix estimation. The proposed architecture is required to overcome positional nosie, be capable of handling outlier matches which comprise a significant portion of the input data, generalize to unseen image pairs and be capable of handling arbitrary number of point matches.

The propose archchitecture comes with the following:

  - a two-stage noise-aware training scheme

  - a noise head for predicting positional inlier nosie, and

  - a classification head for inlier/outlier discrimination.

The architecture consists of 

  - Nosie-aware Concensus Blocks (NAC)

    - These are stacks of set encoders (ie Deepsets) used to obtain latent representation of input sets and for classification of points as inliers or outliers.

  The ouput of these blocks are used to estimate the essential matrix.
strength_weakness: strength_weakness: - Ablation on the choice of set functions is missing.
questions: questions: - Can an ablation on the choice of set functions be provided? For instance, recent methods such as [1] and [2] have been shown to perform better than deepsets for modeling interactions over sets and these might provide further performance boost to the proposed architecture.

- Also, how does the number of deepset layers affect performance? [3] shows that special techniques are required to ensure effective learning when multiple deepsets layers are stacked. Curious how this correlates with your experiments.

## References

[1] Lee, Juho, et al. "Set transformer: A framework for attention-based permutation-invariant neural networks." International conference on machine learning. PMLR, 2019.

[2] Bruno, Andreis, et al. "Mini-batch consistent slot set encoder for scalable set encoding." Advances in Neural Information Processing Systems 34 (2021): 21365-21374.

[3] Zhang, Lily, et al. "Set norm and equivariant skip connections: Putting the deep in deep sets." International Conference on Machine Learning. PMLR, 2022.
limitations: limitations: While no discussion of limitations is provided, i do not see any immediate negative societal impact of this work.
paper_summary: paper_summary: This work proposes a simpler yet effective network architecture based on Deep Sets for the estimation of essential matrices.

The method is based on three key properties:

1. A two-stage noise-aware training scheme first uses noise-free inlier matches for training, and then uses original matches and full loss for training;

2. A noise head predicts the displacement noise in inlier matches.

3. A classification head classifies point matches as either inliers or outliers.

Evaluations show that the method outperforms previous methods in terms of accuracy and generalization.
strength_weakness: strength_weakness: 1. While the DeepSets-based architecture is shown to be effective, traditional fully convolutional networks, such as UNet, are also applicable. Is there any strong reason why a DeepSets-based network is preferred and explored first?

2. The authors remove half of the input matches to test the effect of outlier pruning. Is this approach reasonable? Is this value adjustable? It would be better to add an ablation study showing performance changes as the number of removed matches varies. Furthermore, this is an indirect method to validate the effectiveness of the proposed inlier/outlier classification head. Why was the ablation study not conducted with and without this head?

3. While the paper provides an ablation study on the proposed key properties, it lacks an ablation of the network architecture and hyperparameters, such as the number of set layers, the dimension of the Set Encoder, etc, but this would also be valuable to know — for example, how much of an impact do these parts of the network have on the final results?

4. The paper lacks details on applying the Optimal Triangulation Method to set inlier/outlier labels and obtain ground truth, noise-free inlier keypoints. Including these details would help readers better understand the training process.

5. What is the unit of the y-axis in Fig. 3?

6. To the best of my knowledge, the ground truth poses in the SUN3D dataset are inaccurate (see supplemental material in {1}). Why did the authors use this dataset solely to follow the evaluation protocol of previous methods? For indoor scenes, is it more suitable to use ScanNet dataset {2} for evaluation as [42]?

7. While the paper claims that the network is simpler than existing methods, the multiple NAC blocks and the noise regression module may introduce a certain level of complexity. It would be better to provide experiments and discussions on the computational efficiency of the proposed method compared to existing methods, including runtime and model parameter scale, which are critical for real-world applications.

{1} Bellavia, Fabio. “Image Matching by Bare Homography.” IEEE Transactions on Image Processing 33 (2023): 696-708.
{2} Dai, Angela, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser and Matthias Nießner. “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 2432-2443.
questions: questions: 1. While the DeepSets-based architecture is shown to be effective, traditional fully convolutional networks, such as UNet, are also applicable. Is there any strong reason why a DeepSets-based network is preferred and explored first?

2. The authors remove half of the input matches to test the effect of outlier pruning. Is this approach reasonable? Is this value adjustable? It would be better to add an ablation study showing performance changes as the number of removed matches varies. Furthermore, this is an indirect method to validate the effectiveness of the proposed inlier/outlier classification head. Why was the ablation study not conducted with and without this head?

3. While the paper provides an ablation study on the proposed key properties, it lacks an ablation of the network architecture and hyperparameters, such as the number of set layers, the dimension of the Set Encoder, etc, but this would also be valuable to know — for example, how much of an impact do these parts of the network have on the final results?

4. The paper lacks details on applying the Optimal Triangulation Method to set inlier/outlier labels and obtain ground truth, noise-free inlier keypoints. Including these details would help readers better understand the training process.

5. What is the unit of the y-axis in Fig. 3?

6. To the best of my knowledge, the ground truth poses in the SUN3D dataset are inaccurate (see supplemental material in {1}). Why did the authors use this dataset solely to follow the evaluation protocol of previous methods? For indoor scenes, is it more suitable to use ScanNet dataset {2} for evaluation as [42]?

7. While the paper claims that the network is simpler than existing methods, the multiple NAC blocks and the noise regression module may introduce a certain level of complexity. It would be better to provide experiments and discussions on the computational efficiency of the proposed method compared to existing methods, including runtime and model parameter scale, which are critical for real-world applications.

{1} Bellavia, Fabio. “Image Matching by Bare Homography.” IEEE Transactions on Image Processing 33 (2023): 696-708.
{2} Dai, Angela, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser and Matthias Nießner. “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 2432-2443.
limitations: limitations: NA

Comments: title: Thanks for the authors' feedback.
Thanks for taking time to write rebuttal.
comment: Thanks for clarifying. This would have been amazing result / ablation to add to the paper. However, I have no further questions.
Thank you for this positive feedback!
Thank you for your response and questions.


1. Here are the results, with two more baseline methods:
| Method                  | MAP5  | MAP10  | MAP20  | MAP30  |
|----------------------|-------|--------|--------|--------|
| U-Match              |  6.53                 | 13.16 | 22.84  | 30.44  |
| ConvMatch w/RANSAC   | 6.53                 | 15.15 | 26.54  | 33.77  |
| NCMNet               |8.6   | 15.16  | 25.65  | 33.56  |
| NACNet (Ours)        |  **10.33** | **17.9**   | **28.55** | **36.08**  |
We note that other previous methods did not provide pre-trained models for SUN3D or config files with hyperparameters. 

2. Indeed, our method appears to achieve a faster runtime (11.12ms) despite using more parameters. This is due to the simplicity of our network’s architecture, which utilizes 82 standard fully-connected layers. In comparison, BCLNet (runtime 46.94), interleaves k-nearest neighbors (k-nn), graph convolution, and attention layers within a total of 190 layers. Further profiling of their code shows that the k-nn (which involves no parameters) takes 16% of this time (\~7.5ms), and the convolutions take 12% (\~5.6ms). Moreover, the forward pass over 190 layers is sequential and cannot be parallelized as efficiently as shallower networks.
Thank you for these comments. We will modify the paper accordingly.

W1. With high pose error, we see nearly no correction at all. Perhaps we can answer your question regarding the noise prediction block with an informal discussion. Our noise prediction block is a function of a match $x_i$ and the full set of input matches $X$. This function is trained to minimize a loss that depends on $x_i$ and the ground truth essential matrix $E$, and we expect it to produce the maximal a postriori noise vector given the set $X$. Here the prior encodes the noise distribution of the feature extractor. For SIFT, this may be an isotropic Gaussian distribution. The set $X$ imposes a distribution over possible essential matrices, and those, in turn, narrow down the likely directions and magnitudes of the predicted error.
I read the rebuttal and all other reviews. This rebuttal has addressed most of my concerns, but some questions still remain.

1. For the reply to Q6, could you please provide the results of other baseline methods on ScanNet dataset?

2. For the reply to Q7, somewhat counterintuitively, the model achieves a faster runtime despite using more parameters.
Thank you for the clarifications. It is great to see that the proposed model is more efficient than the baselines, and that the inlier/outlier classification seems to work well. I have no other concerns. Consequently, I've also adjusted the score up by 1 point.
Reply W1: Thanks for providing the graph. What happens for image pairs with high pose error? (or what are some highest key-point movement you see between before and after denoising?)  What I am trying to understand is what the noise-prediction block is learning? Is it learning through image gradient / feature gradients? Or is there something else going on here.

Reply W2: Thanks, all clear. 

Reply W3: Make sense. This is a good insight, if possible I'll encourage authors to make space for it in the paper / appendix.

Reply Q1: Thanks, this clarifies.

Reply Q2: Amazing! I think this is very important. 11ms is highly desirable. Also, low memory usage shows that this method can also be used on smaller memory constrained devices. 

Overall, I am satisfied with authors response and the paper does show practical and novel algorithm for consensus learning. I will keep my rating to 8.