Visual Correspondence Hallucination
Authors: Hugo Germain
Vincent Lepetit
Guillaume Bourmaud
Venue: ICLR 2022 Poster (2022)
Keywords: visual correspondence hallucination
camera pose estimation

Abstract: Given a pair of partially overlapping source and target images and a keypoint in the source image, the keypoint's correspondent in the target image can be either visible, occluded or outside the field of view. Local feature matching methods are only able to identify the correspondent's location when it is visible, while humans can also hallucinate its location when it is occluded or outside the field of view through geometric reasoning.  In this paper, we bridge this gap by training a network to output a peaked probability distribution over the correspondent's location, regardless of this correspondent being visible, occluded, or outside the field of view.  We experimentally demonstrate that this network is indeed able to hallucinate correspondences on pairs of images captured in scenes that were not seen at training-time.  We also apply this network to an absolute camera pose estimation problem and find it is significantly more robust than state-of-the-art local feature matching-based competitors.

Published as a conference paper at ICLR 2022
VISUAL CORRESPONDENCE HALLUCINATION
Hugo Germain', Vincent Lepetit' and Guillaume Bourmaud
LIGM, École des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Van Vallée, France IIS, University of Bordeaux, Bordeaux INP, CNRS, Bordeaux, France
(firstname. last @enpc. fr, gui 1 aumee wtwtmmaudduuooorddeeee fr
ABSTRACT
Given pair of partially overlapping source and target images and keypoint in the source image, the keypoint's correspondent in the target image can be either visible, occluded or outside the field of view. Local feature matching methods are only able to identify the correspondent's location when it is visible, while humans can also hallucinate (i.e. predict) its location when it occluded or outside the field of view through geometric reasoning. In this paper, we bridge this gap by training a network to output a peaked probability distribution over the correspondent' location, regardless of this correspondent being visible, occluded, or outside the field of view. We experimentally demonstrate that this network indeed able to hallucinate correspondences not seen at training-time time. We also apply this network to an absolute camera pose estimation problem and find it is significantly more robust than state-of-the-art local feature matching-hasee based competitors.
pairs of images captured in scenes that were
INTRODUCTION
Establishing correspondences between two partially overlapping images a fundamental computer vision problem with many applications. For example, state-of-theeart methods for visual localization from an input image rely on keypoint matches between the input image and a reference image (Sattler al. 2018, Sarlin al. 2019 2020 Revaud et al.) 2019). However, these local feature matching methods will still fail when few keypoints are covisible, i.e. when many image locations in one image outside the field of view or become occluded in the second image. These failures to be expected since these methods are pure pattern recognition approaches that seek to identify correspondences, i.e. to find correspondences in covisible regions, and consider the non-covisible regions as noise. By contrast, humans explain the presence of these non-covisible regions through geometric reasoning and consequently are able hallucinate (i.e. predict) correspondences those locations. Geometric reasoning has already been used in computer vision for image matching, but usually as an a posteriori processing (Fischler Bolles 1981 1 Luong & Faugeras) 1996] |Barath Matas 2018 Chum et al. 2003; 2005; Barath et al.) 2019; 2020). These methods seek to remove outliers from the set of correspondences produced by a local feature matching approach using only limited geometric models such as epipolar geometry or planar assumptions.
Contributions. In this paper tackle the problem of correspondence hallucination. In doing so we seek to answer two questions: (f) can we derive a network architecture able to learn to hallucinate correspondences? and (iz) is correspondence hallucination beneficial for absolute pose estimation? The answer to these questions is the main novelty of this paper. More precisely, we consider a network that takes as input pair of partially overlapping source/target images and keypoints in the source image, and outputs for each keypoint probability distribution over its correspondentss 's location in the target image plane. We propose train this network both identify and hallucinate the keypoints' correspondents. We call the resulting method NeurHal, for Neural Hallucinations. To the best of our knowledge, learning to hallucinate correspondences is a virgin territory, thus we first provide an analysis of the specific features of that novel learning task. This analysis guides us towards employing an appropriate loss function and designing the architecture of the network. After training the network, we experimentally demonstrate that it is indeed able to hallucinate correspondences of images captured in novel scenes. We also apply this network to a camera pose estimation problem and find it is significantly more robust than otate-of-the-art local feature matching- based competitors.
unseen pairs

---

Published as a conference paper at ICLR 2022
O
Identified
Inpainted
X
Outpainted
Figure Visual correspondence hallucination. Our network, called NeurHal, takes as input a pair of partially overlapping source and target images and a set of keypoints detected in the source image, and outputs for each keypoint a probability distribution over its correspondent's 's location in the target image. When the correspondent is actually visible, its location can be identified; when is not, its location must be hallucinated. Two types of hallucination tasks can be distinguished: 1) if the correspondent is occluded, its location has to be inpainted; 2) if it is outside the field of view of the target image, its location needs to be ouspainted. NeurHal generalizes to scenes not seen during training: For each of these three pairs of source/target images coming from the scenes of ScanNet (Dai et al., 2017) and MegaDepth (Li & Snavely) 2018), we show (top row) the source image with small subset of keypoints, and (bottom row) the target image with the probability distributions predicted by our network and the ground truth correspondents: o for the identified correspondents, + for the inpainted ones, and X for the outpainted correspondents.
RELATED WORK
To the best of our knowledge, aiming at hallucinating visual correspondences has never been done but the related fields of local feature description and matching immensely vast, and we focus here only on recent dearning-basee approaches.
Learning- based local feature description. Using deep neural networks to learn compute local feature descriptors have shown to bring significant improvements in invariance to viewpoint and illumination changes compared to handcrafted methods (Csurka & Humenberger 2018 Gauglitz et al. 2011 Salahat & Qasaimeh] (017)) Balntas et al., (017). Most methods learn descriptors locally
around pre-computed covisible interest regions both images (Yi et al.) 2016 Detone al. 2018 8
Balntas et al. 201 Luo et al.]
2019), using convolutional-basee siamese architectures trained with
a contrastive loss (Gordo et al.
2016 Schroff et al., 2015) Balntas et al. 2016b Radenović et al. Mishchuk et al. 2017; Simonyan et al. 2014), using pose (Wang et al.) [020 Zhou et al. or self (Yang et 202 supervision. To further improve the performances, (Dusmanu et al.
2016
2021
2019
Revaud et al. 2019) propose to jointly learn to detect and describe keypoints in both images,
while Germain et al. 2020 only detects in one image and densely matches descriptors in the other.
Learning-based local feature matching. All the methods described in the previous paragraph establish correspondences by comparing descriptors using a simple operation such as a dot product. Thus the combination of such a simple matching method with siamese architecture inevitably produces outlier correspondences, especially in non-covisible regions. To reduce the amount of outliers, most approaches employ so-called Mutual Nearest Neighbor (MNN) filtering. However, is possible to go beyond simple MNN and learn to match descriptors. Learning-basee matching methods (Zhang et al. 2019; Brachmann & Rother, 2019: Moo Yi et al. 2018 Sun et al. 2020 Choy et al. 2020 2016) take input local descriptors and/or putative correspondences, and learn to output correspondences probabilities. However, all these matching methods focus only on predicting correctly covisible correspondences.
2

---

Published as a conference paper at ICLR 2022
Jointly learning local feature description and matching. Several methods have recently pro- posed to jointly learn to compute and match descriptors (Sarlin et al. 2020; Sun et al.. 2021 Li et al.
2020: Rocco et al. 2018 2020). All these methods use a stamese Convolutional Neural Network
(CNN) to obtain dense local descriptors, but they significantly differ regarding the way they establish matches. They actually fall into two categories. The first category of methods (Li et al. 2020 Rocco ctt al. 2018; 2020) computes a 4D correlation tensor that essentially represents the scores of all the possible correspondences. This 4D correlation tensor is then used as input to a second network that learns to modify it using soft-MNN and 4D convolutions. Instead of summarizing all the information into a 4D correlation tensor, the second category of methods (Sarlin al. 2020; Sun et al. 2021
rely on Transformers (Vaswani et al. 2017: Dosovitskiy et al. |2020] Ramachandran et al. 2019
Caron et al., 202T Cordonnier et al. 2020 Thao et al. 2020; Katharopoulos et al. 2020) to let the descriptors of both images communicate and adapt to each other. All these methods again focus on identifying correctly covisible correspondences and consider non-covisible correspondences as noise. While our architecture is closely related to the second category of methods as we also rely on Transformers, the motivation for using it is quite different since it is our goal of hallucinating correspondences that calls for non-siamese architecture (see Sec
Visual content hallucination. Yang et al. 2019) proposes to hallucinate the content of RGB-D scans to perform relative pose estimation between two images. More recently (Chen et al. 2021 regresses distributions over relative camera poses for spherical images using joint processing of both images. The work of (Yang et al. 2020; Qian et al. 2020 Jin et al. (021) shows that employing hallucinate-then-matee paradigm can be reliable way of recovering 3D geometry or relative pose from sparsely sampled images. In this work, we focus on the problem of correspondence hallucination which unlike previously mentioned approaches does not aim recovering explicit visual content or directly regressing a camera pose. Perhaps closest to our goal is et al. (202T) that seeks to estimate relative rotation between two non-overlapping images by learning to reason about "hidden" cues such as direction of shadows in outdoor scenes, parallel lines vanishing points.
OUR APPROACH
Our goal is to train a network that takes as input a pair of partially overlapping source/target images and keypoints in the source image, and outputs for each keypoint probability distribution over its correspondent's location in the target image plane, regardless of this correspondent being visible, occluded, or outside the field of view. While the problem of learning to find the location of visible correspondent received a lot of attention in the past few years (see Sec. [ to the best of our knowledge, this paper is the first attempt of learning to find the location of correspondent regardless of this correspondent being visible, occluded, or outside the field of view. Since this learning task is virgin territory, we first analyze its specific features below, before defining a loss function and network architecture able to handle these features.
3.1 ANALYSIS OF THE PROBLEM
The task of finding the location of a correspondent regardless of this correspondent being visible, occluded, or outside the field of view actually leads to three different problems. Before stating those three problems, let us first recall the notion of correspondent it is the keystone of our problem.
Correspondent. Given keypoint ps E R² in the source image Is, its depth da E RR, and the relative camera pose Rys E S3))), trg E RR3 between the coordinate systems of Is and the target image Ir. the correspondent Pr R2a of ps in the target image plane is obtained by warping Ps: Pr : w (ds, Ps, Bys, trs) := Kr (dgRrsKy "p + trs). where Kg and Ky are the camera calibration matrices of source and target images and (u) == /uz/u.- un/ /u. up/ug, 1]" is the projection function. In
a slight abuse of notation, we do not distinguish homogencous 2D vector from non-homogeneous 2D vector. Let us highlight that the correspondent pr of ps may not be visible, i.e. may be occluded or outside the field of view.
Identifying the correspondent. In the case where a network has to establish a correspondence between a keypoint ps in Ig and its visible correspondent pr in I,, standard approaches, such as
3

---

Published as a conference paper at ICLR 2022
comparing local descriptor computed at ps in Is with local descriptors computed at detected keypoints in Ir, are applicable to identify the correspondent Pr-
Outpainting the correspondent. When Pr is outside the field of view of Ir, there is nothing to identify, i.e. neither can Pr be detected as a keypoint nor can local descriptor be computed at that location. Here the network first needs to identify correspondences in the region where I, overlaps with Is and realize that the correspondent Pr is outside the field of view to eventually outpaint it (see Fig. D. We call this operation "outpainting the correspondent" as the network needs predict the location of pr outside the field of view of Ir.
Inpainting the correspondent. When Pr is occluded in I7, the problem is even more difficult since local features can be computed at that location but will not match the local descriptor computed at Ps in Ig. As in the outpainting case, the network needs to identify correspondences in the region where Ir overlaps with Ig and realize that the correspondent pr is occluded to eventually inpaint the correspondent Pr (see Fig. D. We call this operation "inpainting the correspondent" as the network needs to predict the location of pr behind the occluding object.
Let us now introduce a loss function and an architecture that are able to unity the identifying, inpainting and outpainting tasks.
3.2 LOSS FUNCTION
The distinction we made between the identifying, inpainting and outpainting tasks come from the fact that the source image Ig and the target image In are the projections of the same 3D environment from two different camera poses. In order to integrate this idea and obtain a unified correspondence learning task, we rely on the Neural Reprojection Error (NRE) introduced by (Germain et al., 2021). In order to properly present the NRE, first recall the notion of correspondence map.
Correspondence map. Given Is. I7 and a keypoint in the image plane of Ig, the correspondence map Cr of Ps in the image plane of IT is 2D tensor of size He We such that Cr (Pr) (Pr Ps. Is. I7) is the likelihood of Pr being the correspondent of ps. The likelihood can only be evaluated for Pr E fc. where 92c- is the set of all the pixel locations in Cr. Here, we implicitly defined that the likelihood of Pr falling outside the boundaries of C, is zero. In practice, correspondence map Cr implemented as neural network that takes as input ps. Ig and Ir, and outputs a softmaxed 2D tensor. A correspondence map Cr may not have the same number of lines and columns than Ir especially when the goal is to outpaint a correspondence. Thus, in the general case, to transform a 2D point from the image plane of In to the correspondence plane of Cr, we will need another affine transformation matrix Kg. Let us highlight that this likelihood is obtained using the visual content of Is and In only.
Neural Reprojection Error. The NRE (Germain et al. 2021 is loss function that warps a keypoint Рa into the image plane of Ir and evaluates the negative log-liiclihood at this location. In our context, the NRE can be written as:
NRE (ps, Cr, Rrs, trs, ds) := In Cr (xr) where Xy = Kow (ds: Ps, Rrs, trs)
(1)
In general, Xy does not have integer coordinates and the notation In Cy (x7) corresponds performing a bilinear interpolation after the logarithm. For more details concerning the derivation of the NRE, the reader is referred to Germain al. (20TT)
The NRE provides us with a framework to learn to identify, inpaint or outpaint the correspondent of Ps in Ir in a unified manner since Eq. ( is differentiable w.E.t. Cr and there is no assumption regarding covisibility. The main difficulty to overcome is the definition of a network architecture able to output a consistent Cr being given only ps. Ig and Ir as inputs, i.e. the network must figure out whether the correspondent of ps in In can be identified or has to be inpainted or outpainted.
3.3 NETWORK ARCHITECTURE
The analysis from Sec. 3.I and the use of the NRE as a loss (Sec. 3.2 call for: non-siamese architecture to be able to link the information from Ig with the information from Ir
4

---

Published as a conference paper at ICLR 2022
to outpaint inpaint the correspondent if needed; an architecture that outputs matching score for all the possible locations in Ir as well as loca- tions beyond the field of view of Ir the network could decide to identify, inpaint or outpaint correspondent at these locations.
To fulfill these requirements, propose the following: Our network takes as input Is and Ir well as a set of keypoints (Ps..) in the source image plane of Ig. A siamese CNN backbone is applied to Is and Ir to produce compact dense local descriptor maps H3 and Hy. In order to be able outpaint correspondents in the target image plane, we pad Hr with a learnable fixed vector X. This padding step allows initialize descriptors at locations outside the field of view of Ir. We note y the relative ouiput-to-inge tn-ut correspondence map resolution ratio.
The dense descriptor maps H3 and prd and the keypoints (Ps.. are then used as inputs of cussattentiionbbsed backbone F with positional encoding. This part of the network outputs feature vector ds... for each keypoint Ps.. and dense feature vectors Dr,p of the size of Had This coos-attentionnbasee backbone allows the local descriptors Hş and Hr, pad to communicate with each other. Thus, during training, the network will be able to leverage this ability to communicate, to learn to hallucinate peaked inpuinted and oulpainted correspondence maps.
The correspondence map Cr,, of Ps.n in the image plane of I is computed by applying a Ix convolution to prd using ds,n filter, followed by a 2D softmax.
softmax
H
F
An overview of our architecture, that we call NeurHal, is presented in Fig. 2 In practice, in order to keep the required amount of memory and the computational time reason- ably low, the correspondence maps (Cr,mln=1... have low resolution, i.e. for target image of size 640 x
H
H
D
K.pod
{c Ce
Figure 2: Overview of NeurHal: See text for details.
480, we use a CNN with an effective stride of = and consequently the resulting correspondence maps (with 7 50%) are of size 160 x 120. Producing low resolution correspondence maps prevents NeurHal from predicting accurate correspondences. But as we show in the experiments, this low resolution is sufficient to hallucinate correspondences and have affirmative answer to both ques- tions: (i) can we derive a network architecture able to learn to hallucinate correspondences? and (ii) correspondence hallucination beneficial for absolute pose estimation? Thus, we leave the question of the accuracy hallucinated correspondences for future research. Additional details concerning the architecture are provided in Sec. C.I of the appendix.
3.4 TRAINING-TIME
Given a pair of partially overlapping images (Ig, Ir), set of keypoints with ground truth depths (Ps.., d ds........... as well as the ground truth relative camera pose (RTs, trs). the corresponding sum of NRE terms (Eq, MD can be minimized w.s. the parameters of the network that produces the correspondence maps. Thus, we train our network using stochastic gradient descent and early stopping by providing pairs of overlapping images along with the aforementioned ground truth information. Let also highlight that there is no distinction in the training process between the identifying, inpainting and outpainting tasks since the only thing our network outputs are correspondence maps. Moreover there is need for labeling keypoints with ground truth labels such "identify/visiblee" "inpain//occluded" or "outpaint/outside the field of view". Additional information concerning the training are provided in Sec. C.2 of the appendix.
3.5 TEST-TIME
At test-time, our network only requires a pair of partially overlapping images (Is, Ir) well as keypoints {Ps.. } n=l, N in Ig. and outputs a correspondence map Cr,, in the image plane of I, for each keypoint, regardless of its correspondent being visible, occluded or outside the field of view.
5

---

Published as a conference paper at ICLR 2022
Outpainted
Impairted
idendite
0.34
0.15
ScanNet:
0.02
0.06
0.01
In((
Megadepth
0.40
0.30
10
200
400
900
800
NRE
Argmax (px)
Figure 3: Evaluation of the ability of NeurHal to hallucinate correspondences on the test scenes of ScanNet and MegaDepth. (left) Histograms of the NRE (see Eq. [ for each task (identifying, outpainting, impainting), computed correspondence maps produced by NeurHal. The value in [So, the NRE of a uniform correspondence map. (right) Histograms of the errors between the argmax (mode) of correspondence map and the ground truth correspondent's location, for each task. The value Ex is the average error of a random prediction.
4 EXPERIMENTS
In these experiments, seek answer questions: "Is the proposed NeurHal approach presented in Sec. cec.bble hallucinating correspondences?" and 2) "In the context of absolute camera pose estimation, does the ability to hallucinate correspondences bring further robustness?".
4.1 EVALUATION OF THE ABILITY TO HALLUCINATE CORRESPONDENCES
We evaluate the ability of our network to hallucinate correspondences on four datasets: the indoor
datasets ScanNet (Dai et 2017) and NYU (Nathan Silberman Fergus 2012). and the outdoor datasets MegaDepth (Li& & Snavely 2018) and ETH-3D (Schöps et al. (017). For the indoor setting (outdoor setting, respectively). we train NeurHal on ScanNet (Megadepth, respectively) on the training scenes as described in Sec. 3.4 and evaluate it on the disjoint set of validation scenes. Thus, all the qualitative and quantitative results presented in this section cannot be ascribed to scene memorization. For each dataset, run predictions over 2, 500 source and target image pairs sampled from the test set, with overlaps between 2% and 80%. For every image pair, we also feed as input to NeurHal keypoints in the source image. These keypoints have known ground truth correspondents in the target image and labels (visible, occluded, outside the field of view) that we use to evaluate the ability of our network hallucinate correspondences. For more details on the settings of our experiment see Sec. C.2 For this experiment, we use 50%.
We report in Fig- two histograms computed over more than one million keypoints for each task we seek to validate: identification, inpainting, and outpainting. The first histogram Fig. 3 (left) obtained by evaluating for each correspondence map the NRE cost (Eq. [ at the ground truth correspondent's 's location. In order to draw conclusions, we also report the negative log-likelihood of uniform correspondence map (In [9c, (). We find that for each task and for both datasets, the predicted probability mass lies significantly below In [92, which demonstrates NeurHal's 's ability to perform identification, inpainting and outpainting. On ScanNet, we also observe that identification is simpler task than outpainting while inpainting is the hardest task: On average, the NRE cost of inpainted correspondents is higher than the average NRE cost of outpainted correspondents, which indicates the predicted correspondence maps are less peaked for inpainting than they are for outpainting. This corroborates what we empirically observed on qualitative results in Fig, ! and supports our analysis in Sec. 3.1 On Megadepth, outpainting and inpainting histograms have similar shape which does

---

Published as a conference paper at ICLR 2022
Method
66
15
LoFTR
2021
26 100 120 20 Cleance
S2D
Germann 2011
Distnner (pa)
Diseence [ps]
Dicennce [pc]
Neudtu
(4) Inpuinting
Outpainting S
(c) Impuinting
(d) Ourpainting M
Figure 4: Ability to hallucinate comparison against state-of-the-art local feature matching methods on ScanNet (S) and Megadepth (M). For each method, we report the percentage of keypoint's 's correspondents whose distance w.r.f. the ground truth location lower than z pixels, as a function of x, for (a-c) the inpainting task and (b-d) the outpainting task.
not reflect the previous statement, but we believe this is due the fact that inpainting labels are noisy for this dataset, as explained in Sec.
On the right histogram of Fig. we report the distribution of the distance between the argmax of a correspondence map and the ground truth correspondent's 's location. We also report the average error of a random prediction. We find the histogram mass lies significantly to the left of the random prediction average error, indicating our model is able to place modes correctly in the correspondence maps, regardless of the task at hand. On ScanNet, we observe that the inpainting and outpainting histograms are very similar, indicating the predicted argmax is equally good for both tasks. As mentioned above, the correspondence maps produced by NeurHal have low resolution (see Sec. 3.3 which explains why the "argmax error" is not closer zero pixel.
In Fig. we compare the hallucination performances of NeurHal against oate-off-theart local feature matching methods. Since all these local feature matching methods were designed and trained on pairs of images with significant overlap to perform only identification, they obtain poor inpainting results. Concerning the outpainting task, these methods seek to find a correspondent within the image boundaries, consequently they cannot outpaint correspondences and obtain very poor results.
In Fig. 5 we show several qualitative inpainting/outpaanting results on ScanNet and MegaDepth datasets. In the appendix. we also report qualitative results obtained on the NYU Depth dataset (Fig. 16 and on the ETH-3D dataset (Fig. 15}.
These results allow us to conclude that NeurHal is able to hallucinate correspondences with a strong generalization capacity. Additional experiments concerning the ability to hallucinate correspondences are provided in Sec. A as well as technical details regarding the evaluation protocol in Sec. C.3
4.2 APPLICATION TO ABSOLUTE CAMERA POSE ESTIMATION
In the previous experiment, we showed that our network is able to hallucinate correspondences. We now evaluate whether this ability helps improving the robustness of an absolute camera pose estimator. We run this evaluation on the test set of ScanNet over 2,500 source and target image pairs captured in scenes that were not used at training time. For each source/target image pair, we employ NeurHal to produce correspondence maps. As in the previous experiment, we use y = 50%. Given these correspondence maps and the depth map of the source image, we estimate the absolute camera pose between the target image and the source image using the method proposed in Germain et al. (202T).
In Fig. 6 we show the results of an ablation study conducted on ScanNet. In this study, we focus on the robustness of the camera pose estimate for various combinations of training data, i.e. we consider a pose is "correct" if the rotation error is lower than 20 degrees and the translation error is below 1.5 meters (see Sec. C.3) We find that training our network to perform the three tasks (identification, inpainting, and outpainting) produces the best results. In particular, we find that adding outpainting plays critical role in improving localization of low-overlap image pairs. We also find that learning to inpaint does not bring much improvement to the absolute camera pose estimation.
In Fig. we compare the results of NeurHal against state-of-the-art local feature matching methods. In low ooveraalap settings, very few keypoints' correspondents can be identified and many keypoints' correspondents have to be outpainted. In this case, find that NeurHal is able to estimate the camera pose correctly significantly more often than any other method, since NeurHal is the only method able to outpaint correspondences (see Fig. 4 For high-overlap image pairs, the ability to
7

---

Published as a conference paper at ICLR 2022
ScanNet
Megadepth
Source
Target
-ln
Source
Target
-In Cr
Figure 5: Ability to hallucinate Qualitative inpainting/outpainting results. To illustrate the ability of NeurHal to hallucinate correspondents, NeurHal on image pairs (captured in scenes that were not seen at training-timen (top row) outpainting examples, (bottom row) inpainting examples. In the source image, the red dot keypoint. In the target image and in the (negative-log) correspondence map, the red dot represents the ground truth keypoint's i correspondent. The dashed rectangles represent the borders of the target images. More results on the NYU and ETH-3D datasets can be found the appendix D.1
display correspondence maps predicted by
hallucinate is not useful since many keypoints' correspondents can be identified. In this case, we find that state-of-the-as local feature matching methods be slightly better than NeurHal. This is likely due to the fact that NeurHal outputs low resolution correspondences maps while the other methods output high resolution correspondences. The overall performance shows that NeurHal significantly outperforms all the competitors, which allows us to conclude that the ability of NeurHal to outpaint correspondences beneficial for absolute pose estimation. Technical details concerning the previous experiment well additional experiments concerning the application to absolute camera pose estimation are provided in Sec. B
LIMITATIONS
We identified the following limitations for our approach: (1) The previous experiments showed that NeurHal is able to inpaint correspondences but the inpainted correspondence maps are much less peaked compared the outpainted correspondence maps. This likely due to the fact that inpainting
correspondences is much more difficult than outpainting correspondences (see Sec 3.1. (ii) The proposed architecture outputs low resolution correspondence maps (see Sec. 3.3 egg. 160 x 120 for input images of size 640 480 and an amount of padding 7 50%. This is essentially due to the quadratic complexity of attention layers we use (see Sec. C.I of the appendix). (iti) Our approach able to outpaint correspondences but correspondence maps have finite size. Thus, in the case where keypoint's correspondent falls outside the correspondence map, the resulting correspondence map would be erroneous. We believe these three limitations are interesting future research directions.
8

---

Published as a conference paper at ICLR 2022
Training correspondences
Method
0.6
Identified
Inpainted
Outpainted
04
0
20
40
60
80
Visual Overlap (%))
Figure 6: Ablation study Impact of learning to hallucinate for absolute camera pose estima- tion. We compare the influence of adding inpainting and outpainting (y = 50%) tasks when training NeurHal. We report the percentage of camera poses being correctly estimated for image pairs having overlap between 2% and 2%, function of z, ScanNet (Dai et al. (017) with thresholds for translation and rotation errors of 7 = 1.5m and т, 20.0%. Learning to hallucinate correspondences (especially outpainting) significantly improves the amount of correctly estimated poses.
0.6
06
Methed
02
0.2
40
60
Visual Overlap
Visual Overlap (%))
(a) 1.5m, 20
(b) 1.0m, 15"
Figure 7: Absolute camera pose experiment. We compare the performance NeurHal against state-of-the-ar art local feature matching methods ScanNet (Dai et al.) 2017). The "identity" method consists in systematically predicting the identity pose. We report the percentage of camera poses being correctly estimated for pairs of images that have overlap between 2% and x%, function of x, for two rotation and translation eтror thresholds. See discussion in Sec.
6 CONCLUSION
To the best of our knowledge, this paper is the first attempt learn inpaint and outpaint cor- respondences. We proposed an analysis of this novel learning task, which has guided us towards employing an appropriate loss function and designing the architecture of our network. We experi- mentally demonstrated that our network indeed able to inpaint and outpaint correspondences on pairs of images captured in scenes that were not seen training-time time, in both indoor (ScanNet) and outdoor (Megadepth) settings. We also tested our network on other datasets (ETH3D and NYU) and discovered that our model has strong generalization ability. We then tried experimentally illustrate that hallucinating correspondences is not just fundamental Al problem but is also interesting from a practical point of view. We applied our network to an absolute camera pose estimation problem and found that hallucinating correspondences, especially outpainting correspondences, allowed to significantly outperform the state-of-the-as feature matching methods in terms of robustness of the resulting pose estimate. Beyond this absolute pose estimation application, this work points to new research directions such as integrating correspondence hallucination into Structure- cruuctureffrom.t wwwo--froooiiiie pipelines make them more robust when few images available.
Motion

---

Published as a conference paper at ICLR 2022
7 ETHICS STATEMENT
The method described this paper has the potential to greatly improve many computer vision-based industrial applications, especially those involving visual localization in GPS-denied cluttered environments. For example robotics or augmented reality applications could benefit from our algorithm better relocalize within their surroundings, which could lead more reliable and overall safer behaviours. If this was to be applied to autonomous driving or drone-based search and rescue, one could appreciate the positive societal impact of our method. On the other hand like many computer vision algorithms, it could be applied to improve robustness of malicious devices such as weaponized UAVs, or invade citizens privacy through environment re-identification. Thankfully as Al technology advances, discussions and regulations are brought forward by governments and public entities.
These ethical debates pave the way for brighter future and can only make us think NeurHal will more bring benefits than harms to society.
REPRODUCIBILITY
provide the NeurHal model architecture and weights the supplementary material. We also release simple evaluation script that generates qualitative results, and show in notebook the results obtained on an image pair captured indoors using a smartphone.
ACKNOWLEDGEMENT
The authors would like to thank Matthicu Vilain and Rémi Giraud for their insight on visual corre- spondence hallucination. This project has received funding from the Bosch Research Foundation (Bosch Forschungssttituung) This work was granted access to the HPC resources of IDRIS under the allocation 1682R made by GENCI.
REFERENCES
Vassileios Balntas, Edward Johns, Lilian Tang, and Krystian Mikolajezyk. PN-Net: Conjoined Triple Deep Network for Learning Local Image Descriptors. In arXiv Preprint, 2016a.
Vassileios Balntas, Edward Riba, Daniel Ponsa, and Krystian Mikolajczyk. Learning Local Feature Descriptors with Triplets and Shallow Convolutional Neural Networks. In British Machine Vision Conference, 2016b.
Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajezyk. HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors. In Conference on Computer Vision and Pattern Recognition, 2017.
Daniel Barath and Jiri Matas. Graph-Cut RANSAC. In Conference Computer Vision and Pattern Recognition, pp. 6733-6741, 2018.
Daniel Barath, Jiri Matas, and Jana Noskova. MAGSAC: Marginalizing Sample Consensus. In Conference Computer Vision and Patlern Recognition, 2019.
Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiri Matas. MAGSAC++, A Fast, Reliable and Accurate Robust Estimator. In Conference on Computer Vision and Pattern Recognition, pp. 1301-1309, 2020.
Andrew Blake and Andrew Zisserman. Visual Reconstruction. MIT press, 1987.
Eric Brachmann and Carsten Rother. Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses. In International Conference on Computer Vision, 2019.
Ruojin Cai, Bharath Hariharan, Noah Snavely, and Hadar Averbuch-Elor. Extreme Rotation Estima- tion using Dense Correlation Volumes. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
10

---

Published as a conference paper at ICLR 2022
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties Supervised Vision Transformers. In arXiv Preprint, 2021.
Kefan Chen, Noah Snavely, and Ameesh Makadia. Wide-baseline relative camera pose estimation with directional learning. In CVPR, 2021.
Christopher Choy, JunYoung Gwak, Silvio Savarese, and Manmohan Chandraker. Universal Corre- spondence Network. In Advances in Neural Information Processing Systems, 2016.
Christopher Choy, Junha Lee, René Ranftl, Jaesik Park, and Vladlen Koltun. ligh-Dimensional Convolutional Networks for Geometric Pattern Recognition. In Conference on Computer Vision and Pattern Recognition, pp. 11227-11236, 2020.
Ondrej Chum, Jiri Matas, and Josef Kittler. Locally Optimized RANSAC. In DAGM Symposium on Pattern Recognition, 2003.
Ondrej Chum, Tomas Werner, and Jiri Matas. Two-View Geometry Estimation Unaffected by Dominant Plane. In Conference on Computer Vision and Pattern Recognition, pp. 772-779, 2005.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the Relationship Between Self- Attention and Convolutional Layers. In arXiv Preprint, 2020.
Gabriela Csurka and Martin Humenberger. From Handcrafted to Deep Local Invariant Features. In Computing Research Repository, 2018.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proc. Computer Vision and Pattern Recognition (CVPR), JEEE, 2017.
Daniel Detone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperPoint: Self-Supervised Interest Point Detection and Description. In Conference on Computer Vision and Pattern Recognition, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. In arXiv Preprint, 2020.
Mihai Dusmanu, Ignacio Rocco, Tomás Pajdla, Mare Pollefeys, Josef Sivie, Akihiko Torii, and Torsten Sattler. D2-Net: A Trainable CNN for Joint Description and Detection of Local Features. In Conference on Computer Vision and Pattern Recognition, 2019.
Martin A. Fischler and Robert C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications Image Analysis and Automated Cartography. Association for Computing Machinery, 1981.
Steffen Gauglitz, Tobias Höllerer, and Matthew Turk. Evaluation of Interest Point Detectors and Feature Descriptors for Visual Tracking. International Journal of Computer Vision, 94:335-360, 2011.
Hugo Germain, Guillaume Bourmaud, and Vincent Lepetit. S2DNet: Learning Image Features for Accurate Sparse-to-Dent Matching. In European Conference Computer Vision, 2020.
Hugo Germain, Vincent Lepetit, and Guillaume Bourmaud. Neural Reprojection Error: Merging Feature Learning and Camera Pose Estimation. In Conference on Computer Vision and Pattern Recognition, 2021.
Albert Gordo, Jon Almazán, Jérôme Revaud, and Diane Larlus. Deep Image Retrieval: Learning Global Representations for Image Search. In European Conference on Computer Vision, 2016.
Linyi Jin, Shengyi Qian, Andrew Owens, and David E. Fouhey. Planar surface reconstruction from sparse views. ArXiv, 1bs444 2021.
11

---

Published as a conference paper at ICLR 2022
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franccois Fleuret. Transformers Are RNNs: Fast Autoregressive Transformers with Linear Attention. In International Conference on Machine Learning, 2020.
Xinghui Li, Kai Han, Shuda Li, and Victor Prisacariu. Resolutooott Correspondence Networks.
Advances in Neural Information Processing Systems, 33, 2020.
Z. Li and Noah Snavely. Megadepth: Learning Single- Depth Prediction from Internet Photos. In Conference on Computer Vision and Pattern Recognition, 2018.
Ily Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
D. G. Lowe. Distinetive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60(2), 2004.
Zixin Luo, Tianwei Shen, Lei Zhou, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan. ContextDesc: Local Descriptor Augmentation with Cross-Modality Context. In Conference on Computer Vision and Pattern Recognition, pp. 2522-2531, 2019.
Quan- Tuan Luong and Olivier D. Faugeras. The Fundamental Matrix: Theory, Algorithms, and Stability Analysis. International Journal of Computer Vision, 17(1):33-75, 1996.
Anastasiya Mishchuk, Dmytro Mishkin, Filip Radenović, and Jiri Matas. Working Hard to Know
Your Neighbor's Margins: Local Descriptor Learning Loss. In Advances in Neural Information Processing Systems, 2017.
Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learn- ing to Find Good Conrespondences. In Conference Computer Vision and Pattern Recognition, pp. (676,,6744, 2018.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. ECCV, 2012.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Devito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic Differentiation in Pytorch. In Advances in Neural Information Processing Systems, 2017.
Shengyi Qian, Linyi Jin, and David F. Fouhey. Associative3d: Volumetric reconstruction from sparse views. In ECCV, 2020.
Filip Radenović, Giorgos Tolias, and Ondrej Chum. CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples. In European Conference on Computer Vision, 2016.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-Alone Self-Attention In Vision Models. In Advances in Neural Information Processing Systems, 2019.
Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2D2: Reliable and Repeatable Detector and Descriptor. Advances in Neural Information Processing Systems, pp. 12405-12415, 2019.
Ignacio Rocco, M. Cimpoi, Relja Arandjelovié, Akihiko Torii, Tomás Pajdla, and Josef Sivic. Neighbourhood Consensus Networks. In Advances in Neural Information Processing Systems, 2018.
Ignacio Rocco, Relja Arandjelović, and Josef Sivie. Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions. In European Conference on Computer Vision, 2020.
Ehab Salahat and Murad Qasaimeh. Recent Advances in Features Extraction and Description Algorithms: A Comprehensive Survey. In 2017 IEEE International Conference on Industrial Technology (ICIT), pp. 1059-1063, 2017.
12

---

Published as a conference paper at ICLR 2022
Paul-Edouard Sarlin, C. Cadena, R. Siegwart, and M. Dymezyk. From Coarse to Fine: Robust Hier- archical Localization at Large Scale. In Conference on Computer Vision and Pattern Recognition, pp. 12716-2222,, 2019.
Paul-Edouard Sarlin, Daniel Detone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning Feature Matching with Graph Neural Networks. In Conference on Computer Vision and Pattem Recognition, 2020.
Torsten Sattler, W. Maddern, C. Toft, Akihiko Torii, L. Hammarstrand, E. Stenborg. D. Safari, M. Okutomi, Mare Pollefeys, Josef Sivie, F. Kahl, and Tomás Pajdla. Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions. In Conference Computer Vision and Pattern Recognition, 2018.
L., Schönberger and J.-M. Frahm. StrucureeFrom-MMotion Revisited. In Conference on Computer Vision and Pattern Recognition, 2016.
Thomas Schöps, Johannes L. Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Conference Computer Vision and Puttern Recognition (CVPR), 2017.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. In Computing Research Repository, 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Learning Local Feature Descriptors Using Convex Optimisation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36, 2014.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-Free Local Feature Matching with Transformers. In Conference on Computer Vision and Putlern Recognition, 2021.
Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, and Kwang Moo Yi. ACNe: Attentive Context Normalization for Robust Permutation Eqqivaarann Learning. In Conference Computer Vision and Pattern Recognition, pp. 11286-11295, 2020.
Christian Szegedy, V. Vanhoucke, S. loffe, Jon Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. In Conference Computer Vision and Pattern Recognition, pp. 2818-2826. 2016.
Philip HS Torr and Andrew Zisserman. MLESAC: A New Robust Estimator with Application to Estimating Image Geometry. Computer Vision and Image Understanding, 78(1):138-133 156, 2000.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv Preprint, 2017.
Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning feature descriptors using camera pose supervision. ArXiv, 2ass20004.13224, 2020.
Heng Yang. Wei Dong, Luca Carlone, and Vladlen Koltun. Self-supervised geometric perception. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14345- 14356, 2021.
Zhenpei Yang, Jeffrey Z. Pan, Linjie Luo, Xiaowei Zhou, Kristen Grauman, and Qixing Huang. Extreme relative pose estimation for rgb-d scans via scene completion. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4526-4535, 2019.
Zhenpei Yang, Siming Yan, and Qi-Xing Huang. Extreme relative pose network under hybrid representations. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
pp.
Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned Invariant Feature Transform. In European Conference on Computer Vision, 2016.
Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning Two-Vier Correspondences and Geometry Using Awaer-Aware Network. In International Conference on Computer Vision, 2019.
13

---

Published as a conference paper at ICLR 2022
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring Self-Attention for Image Recognition. In Conference on Computer Vision and Pattern Recognition, pp. 10073-100822 2020.
Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixé. Patch2pix: Epipolar-guided pixel-level corre- spondences. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4667-4676, 2021.
14

---

Published as a conference paper at ICLR 2022
APPENDIX
In the following pages, we present additional experiments and technical details about our visual correspondence hallucination method NeurHal. We present additional experiments on the ability to hallucinate in Sec. and and on camera pose estimation in Sec. B We describe technical details in Sec. Cadd provide additional qualitative results in Sec.
A
Additional experiments concerning the ability to hallucinate correspondences
15
A.
Impact of learning inpaint and outpaint
15
A.2
Ability hallucinate: NaurHal hallucination Homography- warping
16
A.3
Additional insights correspondence hallucination
16
A.4
Inpainted outpainted correspondence maps
16
A.5
Ability hallucinate: Neurllal vs. (Germain 2021
16
B
Additional experiments concerning the application to camera pose estimation
18
B.
Iniluence of the pose estimator: (Germain al. (202] (Chum et al. (003)
19
B.2
Impact of the value of
19
B. 3
Analysis of the impact of inpainting and outpainting
20
B. 4
Additional indoor pose estimation results
20
C
Technical details
22
Architecture details
................................
22
2
Datasets and Training details
................................
23
Evaluation Details
24
D Additional qualitative results
25
D ] Generalization to datasets
25
D.2 Qualitative correspondence hallucination results and failure cases
25
D.3 Qualitative camera pose estimation results
25
A ADDITIONAL EXPERIMENTS CONCERNING THE ABILITY TO HALLUCINATE CORRESPONDENCES
In this section we first present an additional ablation study on the ability to hallucinate, followed by additional insights model internal functioning.
A.1 IMPACT OF LEARNING TO INPAINT AND OUTPAINT
To supplement the study made in Sec. 4.1 we now aim at evaluating the impact of learning inpaint and outpaint specifically. To do so, we isolate keypoints with the identified, inpainted and outpainted labels in our ScanNet (Dai al. 2017) evaluation set.
In Fig. 8 we show the results of an ablation study on NeurHal's 's training setup. We report for the identification, inpainting and outpainting tasks two sets of cumulative histograms: 1) the NRE costs at ground truth keypoint correspondents' locations, and 2) distances between the argmax of the correspondence map and the ground truth location. NRE cost cumulative histograms, we also report the results from the uniform distribution, for models trained both with and without outpainting (y 0% and 7 = 50% respectively).
For the identification task (Fig. 8 (a)) find that all methods yield consistent performance. The left figure reveals that NeurHal predictions are significantly above the uniform distribution. indicating peaky maps and thus confident predictions. The right figure shows that the distance of the argmax location w.r.t. the ground truth is also robust (NeurHal predicts at 1/8th of the original resolution but the histogram is computed full resolution).
15

---

Published as a conference paper at ICLR 2022
For the inpainting task (Fig. (b)) draw similar conclusions. We find however that correspon- dence maps are overall less peaky and closer to their respective uniform distribution, which indicates that predictions are less confident. We also find that even though it was not trained to inpaint, the identification baseline is surprisingly able to inpaint correspondences as its performance is not far from the ddentfieation-inaannnin model.
Lastly for the outpainting task (Fig. 8 c)). we find that learning to outpaint gives a significant boost in performance both the NRE distribution and correspondents locations. We also find that jointly learning to inpaint and outpaint is beneficial to the quality of the outpainted cost maps, which implies that both objectives are complementary.
A.2 ABILITY TO HALLUCINATE: NEURHAL HALLUCINATION VS. HIOMGGRAPY-BASSE WARPING
To compare the ability to hallucinate of NeurHal against non learning- baaegg approach, we report in Fig-9tth the performance of homography-basee warping approaches that do no rely on correspondence hallucination.
We compare the performance of a) NeurHal trained to both identify and hallucinate correspon- dences against b) version of NeurHal trained without correspondence hallucination followed by a homography-barn based warping stage to hallucinate correspondences. We derive several baselines of b). We report the performance obtained using simple least-squares solver from all predicted correspondences, and RANSAC alternative. We also report the performance of using an oracle prior to the homography estimation stage to filter out outlier correspondence predictions. We lastly report the performance of estimating the homography using ground-truth identifiable correspondences inside RANSAC loop. For completeness we show the performance of RANSAC-based homogra- phy estimation for several inlier thresholds. In all cases, we find that performing correspondence hallucination using NeurHal significantly outperforms all homography- based alternatives that do not resort to ground-truth information.
This can be attributed the lack of sufficient visual overlap between image pairs that prevent from obtaining an accurate homography estimate, well as the llanar-assumption of this model.
Interestingly, can be seen that with perfect correspondences inpainted correspondences can be accurately recovered. On the other hand, outpainted correspondences do not seem to be robustly retrievable through simple homography estimation.
A.3 ADDITIONAL INSIGHTT ON CORRESPONDENCE IIALLUCINATION
While it is tempting to draw hypotheses regarding the internal functioning of NeurHal, we would like to highlight that this should be done with great care. Indeed, the sheer complexity of the operations run in the many attention layers of NeurHal prevents from interpreting the reasoning that leads to the outputted correspondence maps. From higher level however, we can reasonably speculate that model implicitly learns to jointly predict the depth of the source image and the relative camera pose warp the source keypoints and hallucinate their correspondents in the target image.
A.4 INPAINTED VS. OUTPAINTED CORRESPONDENCE MAPS
We can observe that the inpainted correspondence maps are less peaked than the outpainted corre- spondence maps (see Fig. 3p. We believe this is because outpainting correspondent essentially consists in transferring the features from location Ps.: in the source features to location Pt.: in the target features to obtain d,, (PP.... In the inpainting case, Pa,i is occluded by an object in the target image, and this object is (often) also visible the source image at location ps,ce Thus, to produce peaked correspondence maps for both ps and Ps.ooe the network has to output features such that d = DI, (p...) = de,oce which is more difficult than just de,s = Dr.ppa(Pes)
A.5 ABILITY TO HALLUCINATE: NEURHAL VS. (GERMAIN ET AL. 2021
The architecture proposed by Germain et al. 202T) is not able to predict outpainted correspondences. They do consider an extra category for un-matched keypoints but the probability for that category is
16

---

Published as a conference paper at ICLR 2022
0.5
0.5
10
50
100
150
NRE Cost
Argmax distance (px)
(a) Identified
0.5
100
NRE Cost
Argmax distance (px)
Inpainted
0.5
10
50
100
150
NRE Cost
Argmax distance (px)
(c) Outpainted
Training correspondences
Method
Identified
Inpainted
Outpainted
In((9)))--w%
Figure 8: Ability to hallucinate Ablation study on ScanNet. We compare the influence of adding inpainting and outpainting when training NeurHal. (left column) We report the percentage of keypoint's correspondents whose NRE cost is lower than x, function of x, for (a) identified (b) inpainted and (c) outpainted keypoints. (right column) We report the percentage of keypoint's s correspondents whose distance w.r.t the ground truth is lower than æ pixels, function of for the same categories.
17

---

Published as a conference paper at ICLR 2022
0.5
0.5
Distance (px)
Distance (px)
(a) Inpainting
(b) Outpainting
0.5
0
100
150
200
50
100
150
200
Distance (px)
Distance (px)
(c) Inpainting RANSAC
(d) Outpainting RANSAC
0.5
0.4
0.2
100
150
200
50
100
150
200
Distance (px)
Distance (px)
(e) Inpainting
RANSAC
(f) Outpainting GT + RANSAC
10pe
25pr
50pe
100pz
200ppr 400pz
600pa
Method
Hallucination
Warping
None
Homography LS Homography RANSAC 200pz
Homography - LS Oracle
Homography GT RANSAC 50ppz
Figure 9: Ability to hallucinate Homography-based warping: We compare the performance of 1) NeurHal trained both identify and hallucinate correspondences against a version of NeurHal trained without correspondence hallucination followed by a homography-baser warping stage to hallucinate correspondences and 3) a homography estimated from the ground truth (GT) identified correspondences. We report the performance predict (a) inpainted and (b) outpainted correspondents locations. For each method we report the percentage of keypoint's 's correspondents whose distance w.r.f. the ground truth location is lower than pixels, function of æ. We show in (c) and (d) the performance of RANSAC-base We show in (e) and (f) similar curves when using ground-trutt truth correspondences to estimate the homography. See text for more details.
homography estimation for various inlier thresholds.
18

---

Published as a conference paper at ICLR 2022
set to zero, ie. the network does not output any score for that category. Setting this probability to zero is due to the fact that (Germain et al. 202I) considers a classical Siamese CNN architecture that does not allow the features of both images communicate. (Germain et al. 202T) is what we called, in the introduction, a "pure pattern recognition approach". Moreover, even if (Germain) al. (02T)) were using a non-siamese architecture, their method would output single score for the category "un-matched keypoint" which would allow the network to detect when the correspondent is visible but would not be sufficient to outpaint the location of the correspondent.
B ADDITIONAL EXPERIMENTS CONCERNING THE APPLICATION TO CAMERA POSE ESTIMATION
In this section, we present additional experiments on correspondence hallucination for camera pose estimation. We begin with study the impact of the pose estimator in Sec. B.1 followed by study on the impact of the padding value y in Sec. B.2 Lastly, we present in Sec. B.4 additional results on indoor camera pose estimation.
B.1 INFLUENCE OF THE POSE ESTIMATOR: (EERMAIN ET AL.. 2021 VS. (CHUM ET AL.. 2003)
(Germain et al. 2021 provides a pose estimation framework which leverages dense keypoint matching uncertainties predict more accurate and robust camera poses. Compared to the standard pose estimator presented in (Chum et al. 2003) which relies on sparse 2D-to-3D correspondences, the method from (Germain al., 2021) preserves rich information in the form of dense loss maps that is particularly suited for ambiguous matches. For the problem of correspondence hallucination we find the loss maps of both outpainted and inpainted correspondences are usually unimodal but quite diffuse, and are thus particularly suited for this pose estimator.
To study the influence the pose estimator, we report in Fig. 10 the performance of NeurHal (Ger main et al. 202T) vs. NeurHal + (Chum al., 2003). To estimate the camera pose using the method presented in (Chum et al. 2003), we simply take the argmax of each correspondence map and treat sparse 2D correspondent in the query image. We also include the performance of NeurHal when trained without visual correspondence hallucination (i.e. trained using only identified ground truth correspondences.)
We find that the two methods trained without hallucination have poor performances for very low- overlap image pairs which underlines the importance of correspondence hallucination in such cases.
Concerning NeurHal trained with hallucination and using the pose estimator (Chum et al., 2003).
taking the argmax of very coarse correspondence map prevents the pose estimator from achieving good results.
NeurHal trained with hallucination and coupled with the pose estimator of Germain al. 2021 achieves the best results which shows that to obtain robust absolute camera estimates it is important to combine the ability hallucinate correspondences of NeurHal with the pose estimator from (Germain
et al.
2021
B.2
IMPACT OF THE VALUE OF
We report in Fig. the absolute camera pose estimation performance for varying values of y. We compute the percentage of camera poses being correctly estimated for ScanNet (Dai et al.] (017) test images pairs that have overlap between 2% and x% (as function of x:) for a translation threshold of 1.5m and a rotation threshold of 20.0".
We find that using only small percentage of outpainting such as 10% does not improve the performance which is most likely due to the small amount of added training keypoints. For higher y values however significant gains are visible, especially at small visual overlaps. This experiment demonstrates the benefit learning to outpaint correspondences beyond image borders, and broaden the extent of usable source keypoints to perform camera pose estimation.
19

---

Published as a conference paper at ICLR 2022
Method
Hallucination
Pose estimator
0.6
Chum et al.
2003
0.4
Chum et
2003
0.2
Germain al.
2021
Germain et al.
2021
0
0
20
40
60
80
Visual Overlap (%)
Figure 10: Influence the pose estimator: (Germain al. 2021) vs. (Chum et al. 2003): To study the influence of using the pose estimator proposed in (Germain et al.) (021) compared to using the pose estimator from (Chum et al., 2003). we report the performance of NeurHal with both estimators. We also include, for both estimators, the performance of NeurHal trained with identified correspondences only (i.e, without hallucination). We report the percentage of camera poses being correctly estimated for pairs of ScanNet (Dai et al. 2017) images that have overlap between 2% and 5% (as a function of z))
Method
06
50%
04
7 25%
7 10%
0%
Visual Overlap (96)
Figure 11: Impact of the value of y: For increasing values of 7, report the percentage of camera poses being correctly estimated for pairs ScanNet images that have an overlap between 2% and x% (as function of z), for 7, 1.5m and т,- 20.0%. We find that small value of 10% yields no benefit and even damages performance, while values of ? 25% and 50% bring significant improvements, especially small visual overlaps.
We report in Fig. 12 the camera field-of-view as function of the padding parameter. We find that 7 50% provides 130 and 711 of field-of-view average on ScanNet and Megadepth respectively, which is significantly wider than 7 = 0%.
B.3 ANALYSIS OF THE IMPACT OF INPAINTING AND OUTPAINTING
In Fig. we reported the percentage camera poses being correctly estimated for several values
of y. which demonstrates the benefits of outpainting with a large for camera pose estimation. In
Fig. 6 we also showed that learning to inpaint does not bring any significant improvement. We believe that outpainting improves the camera pose because outpainted correspondences are outside the field of view and thus complement the identified correspondences, and thus better constrain the camera pose estimate. On the contrary, inpainted correspondences are usually surrounded by identified correspondences, thus the information they provide redundant and does not allow to better constrain the camera pose estimate.
20

---

Published as a conference paper at ICLR 2022
Field of (*)
100
Dataset
[Arx]
[Ary]
[Ars]
[6A
|Af
50
ScanNet
29.21
38.72"
25.08
55.20*
0.00mm
Megadepth
4.73"
6.91
1.64
20.25'
376.60hmm
0
10
20
30
40
50
(%)
(a) Ficld-of-view w.F. 7
(b) Relative viewpoint statisties
Scartlet
0.08
Megalegtt
0.08
0.04
0.04
0.02
0.02
0.00
50
100
50
100
(c) Histogram of absolute relative angle norm |
Figure 12: Field-of-view function of y and relative viewpoint statistics: We report in (a) the average camera field-of-view as a function of y on ScanNet (Dai et al. (2017) and Megadepth (Li) & Snavely 2018) images. We find that y 50% enables significant amount of additional visual content to reproject within the image boundaries. We report in (b) the median absolute difference in rotation along the x, y and axis,
of the relative rotation, along with the difference in focal length on low-overlap image pairs for ScanNet (Dai et al.) (017)) and Megadepth (Li & Snavely 2018). We report in (c) the histogram of absolute relative angle norm on both datasets. We find ScanNet image pairs exhibit strong relative angular motion while Megadepth image pairs display predominantly zoom-ins and zoom-outs.
21

---

Published as a conference paper at ICLR 2022
Edeniity
R2102
SP-5G
1AFTR
DRCN
S2D
NeusHal
0.2
0.2
0.2
0.2
0.2
0.2
0.2
10
13
20
10
20
10
10
10
Threshold 15)
Threshold (*)
Threshold (*)
Threshold (*)
Threshold (*)
Threshold (*)
Threshold (*)
0.6
0.5
0.6
0.6
0.5
0.5
04
0.4
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
6.2
0.2
0.2
2
Threshald (m)
Threshald (m)
Threshald (m)
Threshold (m)
Threshold (m)
Threshald (m)
Threshald (m)
Identity
R2D2 (Revaud al. 2019)
SP+SG (Sarlin et al.
2020
LoFTR
(Sun et al. 2021
DRCNet (Li ct al. 2020)
S2D Germain et al. 2021
NeurHal
Figure 13: Camera pose estimation experiment- Worst cases: We report the performance of
NeurHal and state-of-the-art feature matching methods on ScanNet (Dai et al. (017) image pairs with visual overlaps between 2% and 5%. For every column, we subselect the 25% of images pairs with the worst predictions for given method. We find that in all cases, NeurHal strongly outperforms its competitors. On the contrary, on the worst NeurHal predictions state-of -otte--ffth-aattt methods achieve much lower performance, which is either on par or lower than the predictions obtained using the Identity.
0.4
0
0
10
Threshold (*)
Threshold (m)
Identity
R2D2 [Revaud et al. 2019)
SP+SG (Sarlin et al. 2020
LoFTR (Sun et al. 2021 NeurHal
DRCNet (Li ct al. 2020)
S2D Germain et al. 2002
Figure 14: Camera pose estimation experiment varying the threshold values: We report the
performance of NeurHal and state-of-the-art feature matching methods on ScanNet (Dai et al. 2017) image pairs with visual overlaps between 2% and 5%. For various angular and translation thresholds we report the percentage of correctly localized images. We find that in all cases, NeurHal strongly outperforms its competitors.
B.4 ADDITIONAL INDOOR POSE ESTIMATION RESULTS
In addition to the results presented in Fig. we report in Fig. 13 the performance NeurHal and state-of-the-arr feature matching methods on ScanNet (Dai et al. 2017) image pairs with visual overlaps between 2% and 5%. For every method, we subselect the 25% of images pairs with the worst predictions, and compare it with the performance of its competitors. We find that in all cases, NeurHal strongly outperforms its competitors. On the worst NeurHal predictions, state-of state-of-the-art the--rtt methods achieve much lower performance. For this category competitors are either on par or achieve a lower performance than the Identity predictions.
observe that all NeurHal
This figure highlights the fact that when NeurHal fails to correctly estimate the camera pose, all the competitors also fail since all the methods perform similarly to the "identity" method, i.e. the method that consists in systematically predicting the identity pose.
Fig. 14 shows that NeurHal is much more robust than state-of ciee--ffthheeeee the-art local feature matching methods for pairs of images with low overlap.
22

---

Published as a conference paper at ICLR 2022
C TECHNICAL DETAILS
C.I ARCHITECTURE DETAILS
NeurHal's architecture can be separated in two building blocks: the convolutional backbone and the multi-head attention block.
Convolutional backbone. The convolutional backbone consists of truncated Incep- tionv3 (Szegedy et al. 2016) model (up to Mixed-6a, 768-dimensional descriptors), modified as per Germain al. (2021) 1p to provide, the case of ScanNet (Dai al. 2017), , 1/8 output-to-input resolution ratio. To help with memory consumption we apply simple 2D convolutional layer to compress the descriptor size 384. In the case where 7 we subsequently pad Hy with the learned vector X, producing Hr.pud.
Positional encoding. After computing Hş and pyd with the convolutional backbone, positional encoding applied both dense feature maps. Similarly SuperGlue (Sarlin et al. 2020). we use 6-layer MLP of size (32, 64, 128, 256, 384), mapping positional meshgrid between (-1, 1) (centered around the image center) to higher dimensionalities. BatchNorm and ReLU layers are placed between every module. In our experiments, but found it did not make a difference in performance. After applying the positional encoding, sparse descriptors {ds, )
tried adding more positional encoding layers
bilincarly interpolated at (Ps.n).=I...) in Hg.
Self-attention. Following the positional encoding, single multi-head attention layer is applied on Hyd, with heads. consists of a standard dot- not-product attention (Vaswani et al., (017). coupled with gating mechanism. For given query Q. key K and value V, compute the attention as Attention Q, K, V) = softmax Q QK"*V where y = a (raa((K)) To mitigate the quadratic cost of the dot-product attention, also apply max-pooling operator keys and values with stride of 2, as we empirically found it had very little impact on performance. We also tried using Linear Transformer (e.g. LinFormer (Katharopoulos al.) (0200) architecture, but despite trying numerous variants we found it consistently damaged the convergence of the model.
Cross-attention. Using the same attention-layer design, we subsequently apply it once between (ds.n) } p=1.... and Hg. This layer allows for communication between the interpolated source descrip- tors which will be used to produce the final correspondence maps, and the original dense source image content. Then, we apply cross-attention layers between (ds,").=1...) and Hr.pad- We empirically found these layers to be most important, they allow for direct communication between the sparse source descriptors and the dense target feature maps, prior to the correspondence maps computation. After trying different values for ls and with memory consumption in mind, settled for & 4 in all our experiments.
Affine transformation. The affine transformation matrix Kç for the correspondence map Cr of resolution (Wr, Hr) is computed from the target image calibration matrix and downscaling factors 8 using:
0
T
0
W/
2
Kc
0
K7
+
0
0
7H4
1
0
0
We can determine if a point lies within the boundaries of C7 if its (x, v) coordinates are between (->W,, HHH and ((1 y)Wr, (1 +1++%) H)))
The model is implemented in PyTorch (Pazzke et al. 2017). For indoor sample with 2000 keypoints has an average throughput of 8.84 image/s on an NVIDIA RTX 3070 GPU. We report the number of parameters in our model in Table
Implementation.
23

---

Published as a conference paper at ICLR 2022
Layer
# of parameters
CNN
2.4 M
Positional Encoding Self-Attention
142 K
1.9 M
Cross-Attention
7.2 M
Total
11.7 M
Table 1: Number of parameters in NeurHal
C.2 DATASETS AND TRAINING DETAILS
ScanNet. The ScanNet Dai et al. 2017) dataset is large-scale indoor dataset containing monoc- ular RGB videos and dense depth images, along with ground truth absolute camera poses. As SuperGlue (Sarlin et al., 2020) and LoFTR (Sun et al. (021). we pre-compute the visual overlaps between all image pairs for both training and test scenes. For the training set we sample images with visual overlap between 2% and 50% from the ScanNet training scenes, which provides us with challenging images to handle. We assemble 6M image pairs and randomly subsample 200k pairs at every training epoch. For testing images, sample 2, 500 image pairs with overlaps between 2% and 80% from the ScanNet testing scenes, using several bins to ensure the sampling is close to being uniform. For both training and testing images, we sample keypoints in the source image along a regular grid with cell sizes of 16 pixels. We remove keypoints with invalid depth, as well as those where the local depth gradient too high, as the depth information might not be reliable. We mark keypoints falling outside the target image plane as being outpainted, and we automatically detect the keypoints to inpaint through a cyclic projection of the source keypoints to the target image and back. The remaining keypoints are labeled as identifiable. For all ScanNet experiments, NeurHal uses 1/8 output-to-inpe resolution ratio, with a target correspondence map maximum edge size of 80 pixels (when = 0%).
Megadepth. We use Megadepth (Li & Snavely 2018) to train and evaluate NeurHal on outdoor images. This dataset contains over one million imgges captured in touristic places, and split in 196 scenes. To train NeurHal and following Germain et al. (021) guidelines, we use the provided SIFT Lowe 2004)-based 3D reconstruction which was made with COLMAP (Schönberger & Frahm 2016). Because the sparse 3D point cloud comes from S1M, we find however that very little keypoints can be marked as inpainted. Indeed, 3D reconstruction is applied objects or people occluding the scene. To allow for wide variety of image pairs we use the sparse reconstruction to estimate the visual overlap and sample pairs with an overlap between 20% and 100%. We however find this overlap estimation to be quite unreliable, as only part of the scene is usually reconstructed. Since Megadepth (Li & Snavely 2018) images are of much higher resolution than ScanNet (Dai al.. 2017), we configure NeurHal to use a /116 putput-to-input resolution (with simple max-pooling layer in the CNN). We set the target correspondence map maximum edge size of 60 pixels (when 7 0%), to allow for space in memory when 7 50%.
Overlap estimation. For a given pair of images, we approximate the visual overlap by computing the covisibility ratio of keypoints for every image pair. For given source and target image pair, first compute the source-to-target and target-to-source covisibility ratios using ground truth depth data and camera poses. We then define the visual overlap as the minimum between both ratios. On Megadepth we find this overlap estimation to be fairly noisy, as depth is only partially known.
Optimizers and scheduling. On both datasets NeurHal is trained for a maximum of 40 epochs. We use initial learning rate of 10 with linear learning rate warm-up in 3 epochs from 0. of the initial learning rate. As Sun et al. (2021). we decay the learning rate by 0.5 every 8 epochs starting from the 8th epoch. We apply the linear scaling rule and use a batch size of 8 over 8 NVIDIA V100 GPUs We use the AdamW (Loshchilov & Hutter 2019) optimizer, with a weight decay of 0.I. In all training procedures, randomly initialize the model weights.
24

---

Published as a conference paper at ICLR 2022
C.3 EVALUATION DETAILS
Evaluation protocol. All baselines follow the same standard protocol in which we: 1) Compute 2D-2D correspondences between the reference image and the query image, 2) Lift these 2D-2D cor- respondences 2D-3D correspondences using the available 3D information for the reference image. 3) Estimate the camera pose given these 2D-3D correspondences by minimizing the Reprojection Error (RE), i.e. applying LO-RANSAC+PnP (Chum et al., 2003) followed by non- non-linear iterative refinement. This approach is widely used and leads to state-of-the-art results in visual localization benchmarks. We also include results for Germain et al. (202T) which we call S2D. For the evaluation of Fig. 4 we find the inpainted and outpainted correspondents for LoFTR (Sun et al., (021) and DRCNet (Li et al. 2020) by fetching the argmax 2D coordinates in the 4D matching confidence volume. For S2D and NeurHal, simply take the argmax in correspondence maps for the same set of keypoints.
Choice of threshold. We reported in Fig. 14 the performance of NeurHal, state-of-the-art feature matching methods and the identity pose, on ScanNet for several rotation and translation thresholds. We can see that arbitrarily choosing a threshold of T. = 1.5m and Tr = 20.0" sets a hard objective as the identity pose particularly poor.
(Chum et al. 2003)- -based pose estimator. For all (Chum et al., 2003) 200se-based bbased methods, we estimate the camera pose using the pycolmap python binding. We tune the RANSAC threshold for optimal performance, and mark all cases where less than 3 valid correspondences (i.e. with a valid depth value) failure cases (infinite pose error). The remaining parameters are left as default. We follow the evaluation instructions provided by each method, and use indoor weights for SP+SG (Sarlin et al., 2020) and the dual-softmax indoor weights for LoFTR (Sun et al., (021). In the case of NaurHal + (Chum et al. 2003), we simply read the argmax of the predicted correspondence maps to obtain explicit 2D-to-3D correspondences.
Gerrmin et al., 2021 2021>-based -based pose estimator. For both S2D (Germain al. 2021 and NeurHal we only use coarse models, which operate at either 1/8th or 1/16th of the original input resolution. We first retrain the S2D coarse model (fully-convolutional Inceptionv3 (Szegedy et al. 2016). up to Mixed-6e) on the same training set as our method, with the same target resolution of 80 pixels. We refer to this model as S2D. Given correspondence maps and the depth map of the source image, we estimate the camera pose between the target image and the source image using the method proposed in Germain et al. (2021). For both S2D and NeurHal we use the same set of regularly sampled source keypoints (see Sec. C.1, and we perform camera pose estimation first using P3P inside an MSAC (Torr& & Zisserman 2000) loop. We run P3P for a maximum of 5, 000 iterations over the top-20% correspondences. We then apply coarse GNC (Blake & Zisserman 1987) over all source keypoints with maa = 2.0 and Omin = 0.6. Let us highlight that in all the camera pose experiments, the performances of NeurHal are obtained by predicting only low resolution correspondence maps (see Sec. C.I.
D ADDITIONAL QUALITATIVE RESULTS
D.1 GENERALIZATION NEW DATASETS
So far we have demonstrated the ability of NeurHal hallucinate correspondences on unseen validation scenes from both ScanNet (Dai et al., 2017) and Megadepth (Li & Snavely 2018). In order to further demonstrate the generalization capacity of NeurHal, we report qualitative results obtained on the NYU Depth Dataset (Nathan Silberman & Fergus 2012) in Fig. 16 and on the ETH-3D (Schöps et al. 2017) dataset in Fig. [5] We use the set of indoor weights for NYU (i.e. NeurHal trained on ScanNet) and outdoor weights for ETH-3D (i.e. NeurHal trained on MegaDepth). We report the overlayed and upsampled coarse truncated loss map computed following Germain et al. (2021) on low-overlap image pairs. We find that NeurHal is able to robustly outpaint correspondences despite little visual overlaps and strong relative camera motions. These visuals demonstrate the strong generalization ability of NeurHal.
25

---

Published as a conference paper at ICLR 2022
D.2 QUALITATIVE CORRESPONDENCE HALLUCINATION RESULTS AND FAILURE CASES
To further demonstrate the ability of NeurHal to perform visual correspondence hallucination, we report in Fig. 17 and Fig. T8 qualitative results ScanNet (Dai al. 2017) and Megadepth (Li & Snavely, 2018) respectively on scenes that were not seen at training-timee In the target image and in the (negative log) correspondence map, the red dot represents the ground truth keypoint's correspondent. The dashed rectangles represent the borders of the target images.
Let us recall that NeurHal outputs probability distributions (a.k.a. correspondence maps) assuming the two input images are partially overlapping. It is essential to keep this assumption in mind when looking these qualitative results. For instance, concerning the example Fig. (b) (middle), is very difficult for our human visual system to be sure that the two images are actually overlapping, and consequently the network prediction seems to good to be true. However, if we assume that there an overlap, realize that it is actually possible to perform correspondence hallucination, by drawing out the two skirting boards, to correctly outpaint the correspondent.
In fact, this overlapping assumption has regularization effect in cases where the covisible image areas show no distinctive regions, and one image could be at an infinite translation of the other, e.g. Fig. 17 (b) (second to last).
In Fig. (d) and Fig. 18 (d) we show failure cases where the correspondence maps modes predicted by NeurHal are either partially completely off. We find that failure cases often correlate with strongly ambiguous image pairs, or images that have extremely limited visual overlap.
D.3 QUALITATIVE CAMERA POSE ESTIMATION RESULTS
We show in Fig. 19 qualitative results in camera pose estimation on low-overlap images from ScanNet (Dai et al. 7). for NeurHal and its three best-performing competitors. For every method we display the keypoints used as input to the camera pose estimator the source image, along with their reprojection at the estimated camera pose in the target image. For methods using the pose estimator from (Chum et al. 2003), the keypoints are those that have been successfully matched. When using the pose estimator of Germain et al. (2021). the keypoints prediction of the dense NRE maps. We color in keypoints based on their spatial 2D position in the source image. We find that NeurHal strongly benefits from its outpainting ability, in comparison with all other competitors which struggle to find both sufficient and reliable correspondences. We also report in Fig. faillure cases for NeurHal. We find that such cases correspond to image pairs exhibiting extremely limited visual overlap, strong camera pose rotations and overall significant ambiguities.
those involved in the
26

---

Published as a conference paper at ICLR 2022
Source
Target
Source
Target
Figure 15: Qualitative results on ETH3D dataset: We evaluate NeurHal on outdoor image pairs from the ETH-3D (Schöps et al. 2017) dataset and find it is able to outpaint correspondences despite low visual overlaps. We report pairs of source and target images and overlay the upsampled coarse loss map corresponding to the source detection (in red) the target image.
27

---

Published as a conference paper at ICLR 2022
Source
Target
Source
Target
Figure 16: Qualitative results on the NYU dataset: We evaluate NeurHal on indoor images from the NYU (Nahann Silberman & Fergus 2012) dataset and find it is able to outpaint correspondences despite low visual overlaps. We report pairs of source and target images and overlay the upsampled coarse loss map corresponding to the source detection (in red) the target image.
28

---

Published as a conference paper at ICLR 2022
Source
Target
-In
(a) Identification Inpainting Examples
Source
Target
-I Cr
(b) Outpainting Examples
Source
Target
-In Cr
Challenging Examples
Source
Target
In Cr
(d) Borderline Failure Cases
Figure 17: Additional qualitative ScanNet
Dai et 2017 examples. See for details.
29

---

Published as a conference paper at ICLR 2022
Source
Target
(a) Identification Inpainting Examples
Source
Target
-I Cr
(b) Outpainting Examples
Source
Target
-In
Challenging Examples
Source
Target
In Cr
(d) Borderline Failure Cases
Figure 18: Additional qualitative Megadepth
Li Snavely, 2018 examples. See text for details.
30

---

Published as a conference paper at ICLR 2022
SP+SG
LoFTR
DRCNet
NeurHal
Figure 19: Qualitative camera pose estimation results on low-overlap images from Scan- Net (Dai al., 2017): We show for every method keypoints used as input for the camera pose estimator in the source image (left image), along with their predicted reprojection in the target image (right image). We color-code keypoints based 2D spatial position in the source image. We also report for every pair and every method the camera pose estimation error in translation and rotation, colored in green when the pose is less than T, 0.5m and T, = 10.0", and red otherwise.
31

---

Published as a conference paper at ICLR 2022
SP+SG
LOFTR
DRCNet
NeurHal
Figure 20: NeurHal failure cases low-overlap images from ScanNet (Dai et al., 2017): We report cases where NeurHal fails to estimate a camera pose with an error less than T. = 0.5m and 7, 10.0%. We find these cases often correlate with extremely low covisibility coupled with strong camera rotations.
32


Reviews:
[{"review_id": "y1QImMpCSy", "reviewer": "Reviewer_uCMw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\nThe capability to predict the location of points that have just become occluded or exited the frame is useful in many tasks, such as perception for robots where such predictions could guide the robot’s next moves. Humans are good at this, but computer vision research has focused on the limited setting when the keypoints are visible in both images. Loss functions used for training networks for correspondence estimation only consider descriptors of visible keypoints. This is an important contribution of the paper and it goes beyond more qualitative work of predicting unseen objects based on context.\n\nThe approach builds upon the concept of Neural Reprojection Error (NRE) which was recently introduced by Germain et al. (2021). NRE does not require that the appearance of two image regions, that are hypothesized to be projections of the same 3D point, is similar. It can be viewed as an extension of the purely geometric reprojection error. Germain et al. did not consider occluded or out-of-bounds keypoints.\n\nI consider the unified treatment of visible, occluded and out-of-bounds keypoints an advantage.\n\nExperiments are thorough, findings are explained clearly, and limitations of the proposed method are pointed out. I consider the use of four publicly available datasets, two indoor and two outdoor, sufficient. Over one million keypoints are used in the experiments.\n\nGeneralization on additional datasets not used for training is also shown. This is important since it allows broader deployment of the algorithm.\n\nThe main limitation of NeurHal is low accuracy under favorable conditions, due to the low resolution of its output. (Improving this is left as a direction for future work.) \n\nThe appendices contain useful additional information, experiments and implementation details. \n\n*Weaknesses*\n\nIt is unclear to me what the network actually learns. I speculate that it learns to warp the source image to the target given the relative pose between the two cameras and the depth of the keypoint in the source image. The output of this process is a correspondence map, which suggests that multiple warpings are considered. This is not described clearly enough. (The fact that the visibility of keypoint in the target images does not need to be labeled is clear.)\n\nThe metrics used for evaluation are somewhat arbitrary, or not sufficiently justified. Figure 3, for example, compares the results to random chance, which is a very weak baseline. Camera pose estimation is considered correct when the rotation error is under 20 degrees and the translation error is under 1.5 m. The latter is uninformative without knowing the magnitude of the translation between the two cameras. 1.5 m may be a very small or a very large error depending on the input data.\n\n\n*Other Comments*\n\nI find the term hallucination in the title marginally acceptable, but I think that it is abused in the rest of the paper. For example in: “Local feature matching methods are only able to identify the correspondent’s location when it is visible, while humans can also hallucinate its location when it is occluded or outside the field of view through geometric reasoning.” Humans do not hallucinate, they predict or estimate. \n\nFootnote 1 on p. 3 is unnecessary. The equation it refers to holds for any images in general configuration sharing the same intrinsic parameters (K matrix). The latter constraint can be easily relaxed.\n\nSection 3.2: the calibration matrix K_c does not encode the boundaries of the image. Instead of a different matrix, what is needed are different boundary conditions for considering pixels to be within the image or not.\n\nSection A.2 that discusses additional related work should be moved to the main paper. It is very relevant to the problem at hand.\n\nFigure 8: precision on the y-axis is perplexing. The description suggests that this should be recall.\n\n", "summary_of_the_paper": "The paper proposes a method that accepts two partially overlapping images and a keypoint in one of them, and detects the location of the keypoint in the other image, regardless of whether it is visible, occluded, or outside the frame of the image. This is an interesting re-formulation of the correspondence estimation problem, the conventional formulation of which considers only keypoints that are visible in both images. Under the previous formulation, correspondence estimators are considered successful if they can declare that the correspondent of the keypoint is invisible in the target image. In addition to the capability to predict the location of invisible keypoints in isolation, the paper demonstrates that this capability is beneficial to camera pose estimation.", "summary_of_the_review": "The paper contains an important contribution as discussed above. The fact that predicting the locations of invisible points improves camera pose estimation provides further support that the method is useful in downstream tasks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tz6cced5D1B", "reviewer": "Reviewer_nobf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+The paper proposed an new problem: if and how neural network can predict unseen scene (keypoints) from a source-target image pair. Human is able to heuristically guess the location of outpainting correspondences from the geometry. For example, the keypoint displacement is relative camera pose and scene depth. This paper uses a deep model to accomplish this goal.\n+ The paper proposed a method that is based on correspondence map and neural reprojection error [Germain 2021] with sufficient details.\n+ NeurHal is tested on public datasets for precision of correspondences and camera pose estimation, showing better result than baselines.\n\nWeakness:\n- The proposed method has limited technique contribution. For example, the original NRE [Germain 2021]  is able to predict outpainting correspondences (it gives an extra category for un-matched keypoints). As a result, this paper adapts [Germain 2021] to solve this problem.\n- In figure 3, the baseline (uniform correspondence map) is too weak. In figure 4, I do not find the precision for identified keypoints. It is unfair for the other methods.\n- In camera pose estimation (section 4.2), the description of the baseline (correspondent to the first (light blue) method in figure 6) is not clear to me. Because some methods highly rely on good correspondences but others are not. So, it is hard to tell if the proposed method is able to improve the state-of-the-art camera pose estimation methods.   ", "summary_of_the_paper": "The paper proposed a deep method (NeurHal) to predict visible, occluded or out-of-view keypoint matching from source images to target images. In training, a correspondence map is obtained from ground truth of camera matrix/pose, and images. In testing, the model directly outputs three categories of matchings: identified, outputpainting and inpainting. As an application of NeurHal, the method is applied to camera pose estimation and tested on ScanNet and Megadepth. The experiments shows the method improves the estimation accuracy, particularly the outputpainting correspondences. ", "summary_of_the_review": "I tend to reject the paper because of the lack of contribution in technique and insufficient experiments. I would like to see \n1. the technique differences, compared with [Germain 2021] .\n2. improve the baselines, and compare with the state-of-the-art camera pose estimation methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VjkTIgQhQr8", "reviewer": "Reviewer_BNXs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1) This work provides an interesting research direction correspondence learning, in which occluded and out of FoV correspondence can also help large-baseline camera pose estimation.\n(2) Based on this insight: authors proposed a new model for hallucinating the correspondence based on attention blocks.\n(3) Extensive experiments on indoor and outdoor benchmarks demonstrated that proposed model is able to hallucinate correspondence and outperform prior SoTA on wide baseline setting in terms of camera poes estimation.\n(4) Author provided detailed supplementary material and trained model and source code for better supporting reproducibility.\n\nWeakness:\n(1) Authors admitted that the proposed method can only produce low resolution correspondence, which can have a negative impact on localization accuracy. \n(2) Is there any explanation why correspondence inpainting is harder for inpainting?\n(3) Did authors define how to compute Kc for transforming points from image plane to correspondence plane? It seems that I am not able to find it in the main manuscript.\n(4) It would be better if authors could provide more analysis and ablation study on the relationship between accuracy of camera pose estimation and distribution of hallucinated correspondence. And in which case, hallucinated correspondence will be more accurate or less accurate.\n(5) Authors should also cite several other related work for feature matching:\nLearning feature descriptors using camera pose supervision, ECCV 2020\nSelf-Supervised Geometric Perception, CVPR 2021\nPatch2Pix: Epipolar-Guided Pixel-Level Correspondences, CVPR 2021\n", "summary_of_the_paper": "In this manuscript, authors proposed a new problem of correspondence hallucination, in which for keypoint in the source image, its correspondence should be detected regardless if it is occluded or outside the field of view. In particular, the authors proposed a new model and training paradigm that learn to hallucinate correspondence by predicting its probability distribution of its location. Extensive experiments on both indoor and outdoor benchmarks demonstrated that proposed method  can help camera pose estimation and outperforms prior state of the art feature matching approaches.\n", "summary_of_the_review": "In summary, based on the strength and weakness I mentioned, I like this paper's idea for hallucinating correspondence so that we can obtain better accuracy for camera pose estimation, although it would be better to have more analysis and ablation study for the impact of correspondence hallucination on camera pose estimation accuracy.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper receives positive reviews. The authors provide additional results and justifications during the rebuttal phase. All reviewers find this paper interesting and the contributions are sufficient for this conference. The area chair agrees with the reviewers and recommends it be accepted for presentation.
Decision: Accept (Poster)