VISUAL CORRESPONDENCE HALLUCINATION
Year: 2022

Abstract:
Given a pair of partially overlapping source and target images and a keypoint in the source image, the keypoint’s correspondent in the target image can be either visible, occluded or outside the field of view. Local feature matching methods are only able to identify the correspondent’s location when it is visible, while humans can also hallucinate (i.e. predict) its location when it is occluded or outside the field of view through geometric reasoning. In this paper, we bridge this gap by training a network to output a peaked probability distribution over the correspondent’s location, regardless of this correspondent being visible, occluded, or outside the field of view. We experimentally demonstrate that this network is indeed able to hallucinate correspondences on pairs of images captured in scenes that were not seen at training-time. We also apply this network to an absolute camera pose estimation problem and find it is significantly more robust than state-of-the-art local feature matching-based competitors.

heading: 1 INTRODUCTION
text: Establishing correspondences between two partially overlapping images is a fundamental computer vision problem with many applications. For example, state-of-the-art methods for visual localization from an input image rely on keypoint matches between the input image and a reference image (Sattler et al., 2018; Sarlin et al., 2019; 2020; Revaud et al., 2019). However, these local feature matching methods will still fail when few keypoints are covisible, i.e. when many image locations in one image are outside the field of view or become occluded in the second image. These failures are to be expected since these methods are pure pattern recognition approaches that seek to identify correspondences, i.e. to find correspondences in covisible regions, and consider the non-covisible regions as noise. By contrast, humans explain the presence of these non-covisible regions through geometric reasoning and consequently are able to hallucinate (i.e. predict) correspondences at those locations. Geometric reasoning has already been used in computer vision for image matching, but usually as an a posteriori processing (Fischler & Bolles, 1981; Luong & Faugeras, 1996; Barath & Matas, 2018; Chum et al., 2003; 2005; Barath et al., 2019; 2020). These methods seek to remove outliers from the set of correspondences produced by a local feature matching approach using only limited geometric models such as epipolar geometry or planar assumptions.
Contributions. In this paper we tackle the problem of correspondence hallucination. In doing so we seek to answer two questions: (i) can we derive a network architecture able to learn to hallucinate correspondences? and (ii) is correspondence hallucination beneficial for absolute pose estimation? The answer to these questions is the main novelty of this paper. More precisely, we consider a network that takes as input a pair of partially overlapping source/target images and keypoints in the source image, and outputs for each keypoint a probability distribution over its correspondent’s location in the target image plane. We propose to train this network to both identify and hallucinate the keypoints’ correspondents. We call the resulting method NeurHal, for Neural Hallucinations. To the best of our knowledge, learning to hallucinate correspondences is a virgin territory, thus we first provide an analysis of the specific features of that novel learning task. This analysis guides us towards employing an appropriate loss function and designing the architecture of the network. After training the network, we experimentally demonstrate that it is indeed able to hallucinate correspondences on unseen pairs of images captured in novel scenes. We also apply this network to a camera pose estimation problem and find it is significantly more robust than state-of-the-art local feature matching-based competitors.

heading: 2 RELATED WORK
text: To the best of our knowledge, aiming at hallucinating visual correspondences has never been done but the related fields of local feature description and matching are immensely vast, and we focus here only on recent learning-based approaches.
Learning-based local feature description. Using deep neural networks to learn to compute local feature descriptors have shown to bring significant improvements in invariance to viewpoint and illumination changes compared to handcrafted methods (Csurka & Humenberger, 2018; Gauglitz et al., 2011; Salahat & Qasaimeh, 2017; Balntas et al., 2017). Most methods learn descriptors locally around pre-computed covisible interest regions in both images (Yi et al., 2016; Detone et al., 2018; Balntas et al., 2016a; Luo et al., 2019), using convolutional-based siamese architectures trained with a contrastive loss (Gordo et al., 2016; Schroff et al., 2015; Balntas et al., 2016b; Radenović et al., 2016; Mishchuk et al., 2017; Simonyan et al., 2014), or using pose (Wang et al., 2020; Zhou et al., 2021) or self (Yang et al., 2021) supervision. To further improve the performances, (Dusmanu et al., 2019; Revaud et al., 2019) propose to jointly learn to detect and describe keypoints in both images, while Germain et al. (2020) only detects in one image and densely matches descriptors in the other.
Learning-based local feature matching. All the methods described in the previous paragraph establish correspondences by comparing descriptors using a simple operation such as a dot product. Thus the combination of such a simple matching method with a siamese architecture inevitably produces outlier correspondences, especially in non-covisible regions. To reduce the amount of outliers, most approaches employ so-called Mutual Nearest Neighbor (MNN) filtering. However, it is possible to go beyond a simple MNN and learn to match descriptors. Learning-based matching methods (Zhang et al., 2019; Brachmann & Rother, 2019; Moo Yi et al., 2018; Sun et al., 2020; Choy et al., 2020; 2016) take as input local descriptors and/or putative correspondences, and learn to output correspondences probabilities. However, all these matching methods focus only on predicting correctly covisible correspondences.
Jointly learning local feature description and matching. Several methods have recently proposed to jointly learn to compute and match descriptors (Sarlin et al., 2020; Sun et al., 2021; Li et al., 2020; Rocco et al., 2018; 2020). All these methods use a siamese Convolutional Neural Network (CNN) to obtain dense local descriptors, but they significantly differ regarding the way they establish matches. They actually fall into two categories. The first category of methods (Li et al., 2020; Rocco et al., 2018; 2020) computes a 4D correlation tensor that essentially represents the scores of all the possible correspondences. This 4D correlation tensor is then used as input to a second network that learns to modify it using soft-MNN and 4D convolutions. Instead of summarizing all the information into a 4D correlation tensor, the second category of methods (Sarlin et al., 2020; Sun et al., 2021) rely on Transformers (Vaswani et al., 2017; Dosovitskiy et al., 2020; Ramachandran et al., 2019; Caron et al., 2021; Cordonnier et al., 2020; Zhao et al., 2020; Katharopoulos et al., 2020) to let the descriptors of both images communicate and adapt to each other. All these methods again focus on identifying correctly covisible correspondences and consider non-covisible correspondences as noise. While our architecture is closely related to the second category of methods as we also rely on Transformers, the motivation for using it is quite different since it is our goal of hallucinating correspondences that calls for a non-siamese architecture (see Sec.3).
Visual content hallucination. (Yang et al., 2019) proposes to hallucinate the content of RGB-D scans to perform relative pose estimation between two images. More recently (Chen et al., 2021) regresses distributions over relative camera poses for spherical images using joint processing of both images. The work of (Yang et al., 2020; Qian et al., 2020; Jin et al., 2021) shows that employing a hallucinate-then-match paradigm can be a reliable way of recovering 3D geometry or relative pose from sparsely sampled images. In this work, we focus on the problem of correspondence hallucination which unlike previously mentioned approaches does not aim at recovering explicit visual content or directly regressing a camera pose. Perhaps closest to our goal is Cai et al. (2021) that seeks to estimate a relative rotation between two non-overlapping images by learning to reason about “hidden” cues such as direction of shadows in outdoor scenes, parallel lines or vanishing points.

heading: 3 OUR APPROACH
text: Our goal is to train a network that takes as input a pair of partially overlapping source/target images and keypoints in the source image, and outputs for each keypoint a probability distribution over its correspondent’s location in the target image plane, regardless of this correspondent being visible, occluded, or outside the field of view. While the problem of learning to find the location of a visible correspondent received a lot of attention in the past few years (see Sec. 2), to the best of our knowledge, this paper is the first attempt of learning to find the location of a correspondent regardless of this correspondent being visible, occluded, or outside the field of view. Since this learning task is virgin territory, we first analyze its specific features below, before defining a loss function and a network architecture able to handle these features.

heading: 3.1 ANALYSIS OF THE PROBLEM
text: The task of finding the location of a correspondent regardless of this correspondent being visible, occluded, or outside the field of view actually leads to three different problems. Before stating those three problems, let us first recall the notion of correspondent as it is the keystone of our problem.
Correspondent. Given a keypoint pS ∈ R2 in the source image IS, its depth dS ∈ R+, and the relative camera pose RTS ∈ SO(3), tTS ∈ R3 between the coordinate systems of IS and the target image IT, the correspondent pT ∈ R2 of pS in the target image plane is obtained by warping pS: pT := ω (dS,pS, RTS, tTS) := KTπ ( dSRTSK −1 S pS + tTS ) , where KS and KT are the camera calibration matrices of source and target images and π (u) := [ux/uz,uy/uz, 1] T is the projection function. In a slight abuse of notation, we do not distinguish a homogeneous 2D vector from a non-homogeneous 2D vector. Let us highlight that the correspondent pT of pS may not be visible, i.e. it may be occluded or outside the field of view.
Identifying the correspondent. In the case where a network has to establish a correspondence between a keypoint pS in IS and its visible correspondent pT in IT, standard approaches, such as
comparing a local descriptor computed at pS in IS with local descriptors computed at detected keypoints in IT, are applicable to identify the correspondent pT.
Outpainting the correspondent. When pT is outside the field of view of IT, there is nothing to identify, i.e. neither can pT be detected as a keypoint nor can a local descriptor be computed at that location. Here the network first needs to identify correspondences in the region where IT overlaps with IS and realize that the correspondent pT is outside the field of view to eventually outpaint it (see Fig. 1). We call this operation "outpainting the correspondent" as the network needs to predict the location of pT outside the field of view of IT.
Inpainting the correspondent. When pT is occluded in IT, the problem is even more difficult since local features can be computed at that location but will not match the local descriptor computed at pS in IS. As in the outpainting case, the network needs to identify correspondences in the region where IT overlaps with IS and realize that the correspondent pT is occluded to eventually inpaint the correspondent pT (see Fig. 1). We call this operation "inpainting the correspondent" as the network needs to predict the location of pT behind the occluding object.
Let us now introduce a loss function and an architecture that are able to unify the identifying, inpainting and outpainting tasks.

heading: 3.2 LOSS FUNCTION
text: The distinction we made between the identifying, inpainting and outpainting tasks come from the fact that the source image IS and the target image IT are the projections of the same 3D environment from two different camera poses. In order to integrate this idea and obtain a unified correspondence learning task, we rely on the Neural Reprojection Error (NRE) introduced by (Germain et al., 2021). In order to properly present the NRE, we first recall the notion of correspondence map.
Correspondence map. Given IS, IT and a keypoint pS in the image plane of IS, the correspondence map CT of pS in the image plane of IT is a 2D tensor of size HC × WC such that CT (pT) := p (pT|pS, IS, IT) is the likelihood of pT being the correspondent of pS. The likelihood can only be evaluated for pT ∈ ΩCT where ΩCT is the set of all the pixel locations in CT. Here, we implicitly defined that the likelihood of pT falling outside the boundaries of CT is zero. In practice, a correspondence map CT is implemented as a neural network that takes as input pS, IS and IT, and outputs a softmaxed 2D tensor. A correspondence map CT may not have the same number of lines and columns than IT especially when the goal is to outpaint a correspondence. Thus, in the general case, to transform a 2D point from the image plane of IT to the correspondence plane of CT, we will need another affine transformation matrix KC. Let us highlight that this likelihood is obtained using the visual content of IS and IT only.
Neural Reprojection Error. The NRE (Germain et al., 2021) is a loss function that warps a keypoint pS into the image plane of IT and evaluates the negative log-likelihood at this location. In our context, the NRE can be written as:
NRE (pS, CT, RTS, tTS, dS) := − ln CT (xT) where xT = KCω (dS,pS, RTS, tTS) . (1)
In general, xT does not have integer coordinates and the notation ln CT (xT) corresponds to performing a bilinear interpolation after the logarithm. For more details concerning the derivation of the NRE, the reader is referred to Germain et al. (2021).
The NRE provides us with a framework to learn to identify, inpaint or outpaint the correspondent of pS in IT in a unified manner since Eq. (1) is differentiable w.r.t. CT and there is no assumption regarding covisibility. The main difficulty to overcome is the definition of a network architecture able to output a consistent CT being given only pS, IS and IT as inputs, i.e. the network must figure out whether the correspondent of pS in IT can be identified or has to be inpainted or outpainted.

heading: 3.3 NETWORK ARCHITECTURE
text: The analysis from Sec. 3.1 and the use of the NRE as a loss (Sec. 3.2) call for: • a non-siamese architecture to be able to link the information from IS with the information from IT
to outpaint or inpaint the correspondent if needed; • an architecture that outputs a matching score for all the possible locations in IT as well as locations beyond the field of view of IT as the network could decide to identify, inpaint or outpaint a correspondent at these locations.
To fulfill these requirements, we propose the following: Our network takes as input IS and IT as well as a set of keypoints {pS,n}n=1...N in the source image plane of IS. A siamese CNN backbone is applied to IS and IT to produce compact dense local descriptor maps HS and HT. In order to be able to outpaint correspondents in the target image plane, we pad HT with a learnable fixed vector λ. This padding step allows to initialize descriptors at locations outside the field of view of IT. We note γ the relative output-to-input correspondence map resolution ratio.
The dense descriptor maps HS and HT,pad, and the keypoints {pS,n}n=1...N are then used as inputs of a cross-attention-based backbone F with positional encoding. This part of the network outputs a feature vector dS,n for each keypoint pS,n and dense feature vectors DT,pad of the size of HT,pad. This cross-attention-based backbone allows the local descriptors HS and HT,pad to communicate with each other. Thus, during training, the network will be able to leverage this ability to communicate, to learn to hallucinate peaked inpainted and outpainted correspondence maps.
The correspondence map CT,n of pS,n in the image plane of IT is computed by applying a 1×1 convolution to DT,pad using dS,n as filter, followed by a 2D softmax.
An overview of our architecture, that we call NeurHal, is presented in Fig. 2. In practice, in order to keep the required amount of memory and the computational time reasonably low, the correspondence maps {CT,n}n=1...N have a low resolution, i.e. for a target image of size 640 ×
480, we use a CNN with an effective stride of s = 8 and consequently the resulting correspondence maps (with γ = 50%) are of size 160×120. Producing low resolution correspondence maps prevents NeurHal from predicting accurate correspondences. But as we show in the experiments, this low resolution is sufficient to hallucinate correspondences and have an affirmative answer to both questions: (i) can we derive a network architecture able to learn to hallucinate correspondences? and (ii) is correspondence hallucination beneficial for absolute pose estimation? Thus, we leave the question of the accuracy of hallucinated correspondences for future research. Additional details concerning the architecture are provided in Sec. C.1 of the appendix.

heading: 3.4 TRAINING-TIME
text: Given a pair of partially overlapping images (IS, IT), a set of keypoints with ground truth depths {pS,n, dS,n}n=1...N as well as the ground truth relative camera pose (RTS, tTS), the corresponding sum of NRE terms (Eq. 1) can be minimized w.r.t. the parameters of the network that produces the correspondence maps. Thus, we train our network using stochastic gradient descent and early stopping by providing pairs of overlapping images along with the aforementioned ground truth information. Let us also highlight that there is no distinction in the training process between the identifying, inpainting and outpainting tasks since the only thing our network outputs are correspondence maps. Moreover there is no need for labeling keypoints with ground truth labels such as "identify/visible", "inpaint/occluded" or "outpaint/outside the field of view". Additional information concerning the training are provided in Sec. C.2 of the appendix.

heading: 3.5 TEST-TIME
text: At test-time, our network only requires a pair of partially overlapping images (IS, IT) as well as keypoints {pS,n}n=1...N in IS, and outputs a correspondence map CT,n in the image plane of IT for each keypoint, regardless of its correspondent being visible, occluded or outside the field of view.

heading: 4 EXPERIMENTS
text: In these experiments, we seek to answer two questions: 1) "Is the proposed NeurHal approach presented in Sec. 3 capable of hallucinating correspondences?" and 2) "In the context of absolute camera pose estimation, does the ability to hallucinate correspondences bring further robustness?".

heading: 4.1 EVALUATION OF THE ABILITY TO HALLUCINATE CORRESPONDENCES
text: We evaluate the ability of our network to hallucinate correspondences on four datasets: the indoor datasets ScanNet (Dai et al., 2017) and NYU (Nathan Silberman & Fergus, 2012), and the outdoor datasets MegaDepth (Li & Snavely, 2018) and ETH-3D (Schöps et al., 2017). For the indoor setting (outdoor setting, respectively), we train NeurHal on ScanNet (Megadepth, respectively) on the training scenes as described in Sec. 3.4, and evaluate it on the disjoint set of validation scenes. Thus, all the qualitative and quantitative results presented in this section cannot be ascribed to scene memorization. For each dataset, we run predictions over 2, 500 source and target image pairs sampled from the test set, with overlaps between 2% and 80%. For every image pair, we also feed as input to NeurHal keypoints in the source image. These keypoints have known ground truth correspondents in the target image and labels (visible, occluded, outside the field of view) that we use to evaluate the ability of our network to hallucinate correspondences. For more details on the settings of our experiment see Sec. C.2. For this experiment, we use γ = 50%.
We report in Fig. 3 two histograms computed over more than one million keypoints for each task we seek to validate: identification, inpainting, and outpainting. The first histogram Fig. 3 (left) is obtained by evaluating for each correspondence map the NRE cost (Eq. 1) at the ground truth correspondent’s location. In order to draw conclusions, we also report the negative log-likelihood of a uniform correspondence map (ln |ΩCT |). We find that for each task and for both datasets, the predicted probability mass lies significantly below ln |ΩCT |, which demonstrates NeurHal’s ability to perform identification, inpainting and outpainting. On ScanNet, we also observe that identification is a simpler task than outpainting while inpainting is the hardest task: On average, the NRE cost of inpainted correspondents is higher than the average NRE cost of outpainted correspondents, which indicates the predicted correspondence maps are less peaked for inpainting than they are for outpainting. This corroborates what we empirically observed on qualitative results in Fig. 1, and supports our analysis in Sec. 3.1. On Megadepth, outpainting and inpainting histograms have a similar shape which does
not reflect the previous statement, but we believe this is due to the fact that inpainting labels are noisy for this dataset, as explained in Sec. C.2.
On the right histogram of Fig. 3, we report the distribution of the distance between the argmax of a correspondence map and the ground truth correspondent’s location. We also report the average error of a random prediction. We find the histogram mass lies significantly to the left of the random prediction average error, indicating our model is able to place modes correctly in the correspondence maps, regardless of the task at hand. On ScanNet, we observe that the inpainting and outpainting histograms are very similar, indicating the predicted argmax is equally good for both tasks. As mentioned above, the correspondence maps produced by NeurHal have a low resolution (see Sec. 3.3) which explains why the "argmax error" is not closer to zero pixel.
In Fig. 4, we compare the hallucination performances of NeurHal against state-of-the-art local feature matching methods. Since all these local feature matching methods were designed and trained on pairs of images with significant overlap to perform only identification, they obtain poor inpainting results. Concerning the outpainting task, these methods seek to find a correspondent within the image boundaries, consequently they cannot outpaint correspondences and obtain very poor results.
In Fig. 5 we show several qualitative inpainting/outpainting results on ScanNet and MegaDepth datasets. In the appendix, we also report qualitative results obtained on the NYU Depth dataset (Fig. 16) and on the ETH-3D dataset (Fig. 15).
These results allow us to conclude that NeurHal is able to hallucinate correspondences with a strong generalization capacity. Additional experiments concerning the ability to hallucinate correspondences are provided in Sec. A as well as technical details regarding the evaluation protocol in Sec. C.3.

heading: 4.2 APPLICATION TO ABSOLUTE CAMERA POSE ESTIMATION
text: In the previous experiment, we showed that our network is able to hallucinate correspondences. We now evaluate whether this ability helps improving the robustness of an absolute camera pose estimator. We run this evaluation on the test set of ScanNet over 2,500 source and target image pairs captured in scenes that were not used at training time. For each source/target image pair, we employ NeurHal to produce correspondence maps. As in the previous experiment, we use γ = 50%. Given these correspondence maps and the depth map of the source image, we estimate the absolute camera pose between the target image and the source image using the method proposed in Germain et al. (2021).
In Fig. 6, we show the results of an ablation study conducted on ScanNet. In this study, we focus on the robustness of the camera pose estimate for various combinations of training data, i.e. we consider a pose is "correct" if the rotation error is lower than 20 degrees and the translation error is below 1.5 meters (see Sec. C.3). We find that training our network to perform the three tasks (identification, inpainting, and outpainting) produces the best results. In particular, we find that adding outpainting plays a critical role in improving localization of low-overlap image pairs. We also find that learning to inpaint does not bring much improvement to the absolute camera pose estimation.
In Fig. 7, we compare the results of NeurHal against state-of-the-art local feature matching methods. In low-overlap settings, very few keypoints’ correspondents can be identified and many keypoints’ correspondents have to be outpainted. In this case, we find that NeurHal is able to estimate the camera pose correctly significantly more often than any other method, since NeurHal is the only method able to outpaint correspondences (see Fig. 4). For high-overlap image pairs, the ability to
hallucinate is not useful since many keypoints’ correspondents can be identified. In this case, we find that state-of-the-art local feature matching methods to be slightly better than NeurHal. This is likely due to the fact that NeurHal outputs low resolution correspondences maps while the other methods output high resolution correspondences. The overall performance shows that NeurHal significantly outperforms all the competitors, which allows us to conclude that the ability of NeurHal to outpaint correspondences is beneficial for absolute pose estimation. Technical details concerning the previous experiment as well as additional experiments concerning the application to absolute camera pose estimation are provided in Sec. B).

heading: 5 LIMITATIONS
text: We identified the following limitations for our approach: (i) - The previous experiments showed that NeurHal is able to inpaint correspondences but the inpainted correspondence maps are much less peaked compared to the outpainted correspondence maps. This is likely due to the fact that inpainting correspondences is much more difficult than outpainting correspondences (see Sec 3.1). (ii) - The proposed architecture outputs low resolution correspondence maps (see Sec. 3.3), e.g. 160× 120 for input images of size 640× 480 and an amount of padding γ = 50%. This is essentially due to the quadratic complexity of attention layers we use (see Sec. C.1 of the appendix). (iii) - Our approach is able to outpaint correspondences but our correspondence maps have a finite size. Thus, in the case where a keypoint’s correspondent falls outside the correspondence map, the resulting correspondence map would be erroneous. We believe these three limitations are interesting future research directions.

heading: 6 CONCLUSION
text: To the best of our knowledge, this paper is the first attempt to learn to inpaint and outpaint correspondences. We proposed an analysis of this novel learning task, which has guided us towards employing an appropriate loss function and designing the architecture of our network. We experimentally demonstrated that our network is indeed able to inpaint and outpaint correspondences on pairs of images captured in scenes that were not seen at training-time, in both indoor (ScanNet) and outdoor (Megadepth) settings. We also tested our network on other datasets (ETH3D and NYU) and discovered that our model has strong generalization ability. We then tried to experimentally illustrate that hallucinating correspondences is not just a fundamental AI problem but is also interesting from a practical point of view. We applied our network to an absolute camera pose estimation problem and found that hallucinating correspondences, especially outpainting correspondences, allowed to significantly outperform the state-of-the-art feature matching methods in terms of robustness of the resulting pose estimate. Beyond this absolute pose estimation application, this work points to new research directions such as integrating correspondence hallucination into Structure-from-Motion pipelines to make them more robust when few images are available.

heading: 7 ETHICS STATEMENT
text: The method described in this paper has the potential to greatly improve many computer vision-based industrial applications, especially those involving visual localization in GPS-denied or cluttered environments. For example robotics or augmented reality applications could benefit from our algorithm to better relocalize within their surroundings, which could lead to more reliable and overall safer behaviours. If this was to be applied to autonomous driving or drone-based search and rescue, one could appreciate the positive societal impact of our method. On the other hand like many computer vision algorithms, it could be applied to improve robustness of malicious devices such as weaponized UAVs, or invade citizens privacy through environment re-identification. Thankfully as AI technology advances, discussions and regulations are brought forward by governments and public entities.
These ethical debates pave the way for a brighter future and can only make us think NeurHal will more bring benefits than harms to society.

heading: 8 REPRODUCIBILITY
text: We provide the NeurHal model architecture and weights in the supplementary material. We also release a simple evaluation script that generates qualitative results, and show in a notebook the results obtained on an image pair captured indoors using a smartphone.

heading: ACKNOWLEDGEMENT
text: The authors would like to thank Matthieu Vilain and Rémi Giraud for their insight on visual correspondence hallucination. This project has received funding from the Bosch Research Foundation (Bosch Forschungsstiftung). This work was granted access to the HPC resources of IDRIS under the allocation 2021-AD011011682R1 made by GENCI.

heading: APPENDIX
text: In the following pages, we present additional experiments and technical details about our visual correspondence hallucination method NeurHal. We present additional experiments on the ability to hallucinate in Sec. A and on camera pose estimation in Sec. B. We describe technical details in Sec. C and provide additional qualitative results in Sec. D.

heading: A Additional experiments concerning the ability to hallucinate correspondences 15
text: A.1 Impact of learning to inpaint and outpaint . . . . . . . . . . . . . . . . . . . . . 15 A.2 Ability to hallucinate: NeurHal hallucination vs. Homography-based warping . . 16 A.3 Additional insights on correspondence hallucination . . . . . . . . . . . . . . . 16 A.4 Inpainted vs. outpainted correspondence maps . . . . . . . . . . . . . . . . . . 16 A.5 Ability to hallucinate: NeurHal vs. (Germain et al., 2021) . . . . . . . . . . . . 16

heading: B Additional experiments concerning the application to camera pose estimation 18
text: B.1 Influence of the pose estimator: (Germain et al., 2021) vs. (Chum et al., 2003) . . 19 B.2 Impact of the value of γ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B.3 Analysis of the impact of inpainting and outpainting . . . . . . . . . . . . . . . 20 B.4 Additional indoor pose estimation results . . . . . . . . . . . . . . . . . . . . . 20

heading: C Technical details 22
text: C.1 Architecture details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.2 Datasets and Training details . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C.3 Evaluation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

heading: D Additional qualitative results 25
text: D.1 Generalization to new datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 D.2 Qualitative correspondence hallucination results and failure cases . . . . . . . . 25 D.3 Qualitative camera pose estimation results . . . . . . . . . . . . . . . . . . . . 25

heading: A ADDITIONAL EXPERIMENTS CONCERNING THE ABILITY TO HALLUCINATE CORRESPONDENCES
text: In this section we first present an additional ablation study on the ability to hallucinate, followed by additional insights on our model internal functioning.
A.1 IMPACT OF LEARNING TO INPAINT AND OUTPAINT
To supplement the study made in Sec. 4.1, we now aim at evaluating the impact of learning to inpaint and outpaint specifically. To do so, we isolate keypoints with the identified, inpainted and outpainted labels in our ScanNet (Dai et al., 2017) evaluation set.
In Fig. 8, we show the results of an ablation study on NeurHal’s training setup. We report for the identification, inpainting and outpainting tasks two sets of cumulative histograms: 1) the NRE costs at ground truth keypoint correspondents’ locations, and 2) distances between the argmax of the correspondence map and the ground truth location. On NRE cost cumulative histograms, we also report the results from the uniform distribution, for models trained both with and without outpainting (γ = 0% and γ = 50% respectively).
For the identification task (Fig. 8 (a)) we find that all methods yield a consistent performance. The left figure reveals that NeurHal predictions are significantly above the uniform distribution, indicating peaky maps and thus confident predictions. The right figure shows that the distance of the argmax location w.r.t. the ground truth is also robust (NeurHal predicts at 1/8th of the original resolution but the histogram is computed at full resolution).
For the inpainting task (Fig. 8 (b)) we can draw similar conclusions. We find however that correspondence maps are overall less peaky and closer to their respective uniform distribution, which indicates that predictions are less confident. We also find that even though it was not trained to inpaint, the identification baseline is surprisingly able to inpaint correspondences as its performance is not far from the identification+inpainting model.
Lastly for the outpainting task (Fig. 8(c)), we find that learning to outpaint gives a significant boost in performance on both the NRE distribution and correspondents locations. We also find that jointly learning to inpaint and outpaint is beneficial to the quality of the outpainted cost maps, which implies that both objectives are complementary.

heading: A.2 ABILITY TO HALLUCINATE: NEURHAL HALLUCINATION VS. HOMOGRAPHY-BASED WARPING
text: To compare the ability to hallucinate of NeurHal against a non learning-based approach, we report in Fig. 9 the performance of homography-based warping approaches that do no rely on correspondence hallucination.
We compare the performance of a) NeurHal trained to both identify and hallucinate correspondences against b) a version of NeurHal trained without correspondence hallucination followed by a homography-based warping stage to hallucinate correspondences. We derive several baselines of b). We report the performance obtained using a simple least-squares solver from all predicted correspondences, and a RANSAC alternative. We also report the performance of using an oracle prior to the homography estimation stage to filter out outlier correspondence predictions. We lastly report the performance of estimating the homography using ground-truth identifiable correspondences inside a RANSAC loop. For completeness we show the performance of RANSAC-based homography estimation for several inlier thresholds. In all cases, we find that performing correspondence hallucination using NeurHal significantly outperforms all homography-based alternatives that do not resort to ground-truth information.
This can be attributed to the lack of sufficient visual overlap between image pairs that prevent from obtaining an accurate homography estimate, as well as the planar-assumption of this model.
Interestingly, it can be seen that with perfect correspondences inpainted correspondences can be accurately recovered. On the other hand, outpainted correspondences do not seem to be robustly retrievable through a simple homography estimation.

heading: A.3 ADDITIONAL INSIGHTS ON CORRESPONDENCE HALLUCINATION
text: While it is tempting to draw hypotheses regarding the internal functioning of NeurHal, we would like to highlight that this should be done with great care. Indeed, the sheer complexity of the operations run in the many attention layers of NeurHal prevents from interpreting the reasoning that leads to the outputted correspondence maps. From a higher level however, we can reasonably speculate that our model implicitly learns to jointly predict the depth of the source image and the relative camera pose to warp the source keypoints and hallucinate their correspondents in the target image.
A.4 INPAINTED VS. OUTPAINTED CORRESPONDENCE MAPS
We can observe that the inpainted correspondence maps are less peaked than the outpainted correspondence maps (see Fig. 3). We believe this is because outpainting a correspondent essentially consists in transferring the features from location ps,i in the source features to location pt,i in the target features to obtain ds,i = DT,pad(pt,i). In the inpainting case, ps,i is occluded by an object in the target image, and this object is (often) also visible in the source image at location ps,occ. Thus, to produce peaked correspondence maps for both ps and ps,occ , the network has to output features such that ds,i = DT,pad(pt,i) = ds,occ which is more difficult than just ds,i = DT,pad(pt,i)

heading: A.5 ABILITY TO HALLUCINATE: NEURHAL VS. (GERMAIN ET AL., 2021)
text: The architecture proposed by Germain et al. (2021) is not able to predict outpainted correspondences. They do consider an extra category for un-matched keypoints but the probability for that category is
set to zero, ie. the network does not output any score for that category. Setting this probability to zero is due to the fact that (Germain et al., 2021) considers a classical siamese CNN architecture that does not allow the features of both images to communicate. (Germain et al., 2021) is what we called, in the introduction, a "pure pattern recognition approach". Moreover, even if (Germain et al., 2021) were using a non-siamese architecture, their method would output a single score for the category "un-matched keypoint" which would allow the network to detect when the correspondent is not visible but would not be sufficient to outpaint the location of the correspondent.

heading: B ADDITIONAL EXPERIMENTS CONCERNING THE APPLICATION TO CAMERA POSE ESTIMATION
text: In this section, we present additional experiments on correspondence hallucination for camera pose estimation. We begin with a study on the impact of the pose estimator in Sec. B.1, followed by a study on the impact of the padding value γ in Sec. B.2. Lastly, we present in Sec. B.4 additional results on indoor camera pose estimation.
B.1 INFLUENCE OF THE POSE ESTIMATOR: (GERMAIN ET AL., 2021) VS. (CHUM ET AL., 2003)
(Germain et al., 2021) provides a pose estimation framework which leverages dense keypoint matching uncertainties to predict more accurate and robust camera poses. Compared to the standard pose estimator presented in (Chum et al., 2003) which relies on sparse 2D-to-3D correspondences, the method from (Germain et al., 2021) preserves rich information in the form of dense loss maps that is particularly suited for ambiguous matches. For the problem of correspondence hallucination we find the loss maps of both outpainted and inpainted correspondences are usually unimodal but quite diffuse, and are thus particularly suited for this pose estimator.
To study the influence of the pose estimator, we report in Fig. 10 the performance of NeurHal + (Germain et al., 2021) vs. NeurHal + (Chum et al., 2003). To estimate the camera pose using the method presented in (Chum et al., 2003), we simply take the argmax of each correspondence map and treat it as a sparse 2D correspondent in the query image. We also include the performance of NeurHal when trained without visual correspondence hallucination (i.e. trained using only identified ground truth correspondences.)
We find that the two methods trained without hallucination have poor performances for very lowoverlap image pairs which underlines the importance of correspondence hallucination in such cases.
Concerning NeurHal trained with hallucination and using the pose estimator (Chum et al., 2003), taking the argmax of a very coarse correspondence map prevents the pose estimator from achieving good results.
NeurHal trained with hallucination and coupled with the pose estimator of Germain et al. (2021) achieves the best results which shows that to obtain robust absolute camera estimates it is important to combine the ability to hallucinate correspondences of NeurHal with the pose estimator from (Germain et al., 2021).
B.2 IMPACT OF THE VALUE OF γ
We report in Fig. 11 the absolute camera pose estimation performance for varying values of γ. We compute the percentage of camera poses being correctly estimated for ScanNet (Dai et al., 2017) test images pairs that have an overlap between 2% and x% (as a function of x) for a translation threshold of 1.5m and a rotation threshold of 20.0◦.
We find that using only a small percentage of outpainting such as γ = 10% does not improve the performance which is most likely due to the small amount of added training keypoints. For higher γ values however significant gains are visible, especially at small visual overlaps. This experiment demonstrates the benefit of learning to outpaint correspondences beyond image borders, and broaden the extent of usable source keypoints to perform camera pose estimation.
We report in Fig. 12 the camera field-of-view as a function of the padding parameter. We find that γ = 50% provides 130◦ and 71◦ of field-of-view on average on ScanNet and Megadepth respectively, which is significantly wider than γ = 0%.

heading: B.3 ANALYSIS OF THE IMPACT OF INPAINTING AND OUTPAINTING
text: In Fig. 11 we reported the percentage of camera poses being correctly estimated for several values of γ, which demonstrates the benefits of outpainting with a large γ for camera pose estimation. In Fig. 6 we also showed that learning to inpaint does not bring any significant improvement. We believe that outpainting improves the camera pose because outpainted correspondences are outside the field of view and thus complement the identified correspondences, and thus better constrain the camera pose estimate. On the contrary, inpainted correspondences are usually surrounded by identified correspondences, thus the information they provide is redundant and does not allow to better constrain the camera pose estimate.

heading: B.4 ADDITIONAL INDOOR POSE ESTIMATION RESULTS
text: In addition to the results presented in Fig. 7, we report in Fig. 13 the performance of NeurHal and state-of-the-art feature matching methods on ScanNet (Dai et al., 2017) image pairs with visual overlaps between 2% and 5%. For every method, we subselect the 25% of images pairs with the worst predictions, and compare it with the performance of its competitors. We find that in all cases, NeurHal strongly outperforms its competitors. On the worst NeurHal predictions, state-of-the-art methods achieve a much lower performance. For this category we can observe that all NeurHal competitors are either on par or achieve a lower performance than the Identity predictions.
This figure highlights the fact that when NeurHal fails to correctly estimate the camera pose, all the competitors also fail since all the methods perform similarly to the "identity" method, i.e. the method that consists in systematically predicting the identity pose.
Fig. 14 shows that NeurHal is much more robust than state-of-the-art local feature matching methods for pairs of images with a low overlap.

heading: C TECHNICAL DETAILS

heading: C.1 ARCHITECTURE DETAILS
text: NeurHal’s architecture can be separated in two building blocks: the convolutional backbone and the multi-head attention block.
Convolutional backbone. The convolutional backbone consists of a truncated Inceptionv3 (Szegedy et al., 2016) model (up to Mixed-6a, 768-dimensional descriptors), modified as per Germain et al. (2021) to provide, in the case of ScanNet (Dai et al., 2017), a 1/8 output-to-input resolution ratio. To help with memory consumption we apply a simple 2D convolutional layer to compress the descriptor size to 384. In the case where γ > 0, we subsequently pad HT with the learned vector λ, producing HT,pad.
Positional encoding. After computing HS and HT,pad with the convolutional backbone, positional encoding is applied to both dense feature maps. Similarly to SuperGlue (Sarlin et al., 2020), we use a 6-layer MLP of size (32, 64, 128, 256, 384), mapping a positional meshgrid between (−1, 1) (centered around the image center) to higher dimensionalities. BatchNorm and ReLU layers are placed between every module. In our experiments, we tried adding more positional encoding layers but found it did not make a difference in performance. After applying the positional encoding, sparse descriptors {dS,n}n=1...N are bilinearly interpolated at {pS,n}n=1...N in HS.
Self-attention. Following the positional encoding, a single multi-head attention layer is applied on HT,pad, with 4 heads. It consists of a standard dot-product attention (Vaswani et al., 2017), coupled with a gating mechanism. For a given query Q, key K and value V , we compute the attention as Attention(Q,K, V ) = softmax(g ∗ QKT )V where g = σ(max(QK)). To mitigate the quadratic cost of the dot-product attention, we also apply a max-pooling operator on keys and values with a stride of 2, as we empirically found it had very little impact on performance. We also tried using a Linear Transformer (e.g. LinFormer (Katharopoulos et al., 2020)) architecture, but despite trying numerous variants we found it consistently damaged the convergence of the model.
Cross-attention. Using the same attention-layer design, we subsequently apply it once between {dS,n}n=1...N and HS. This layer allows for communication between the interpolated source descriptors which will be used to produce the final correspondence maps, and the original dense source image content. Then, we apply k cross-attention layers between {dS,n}n=1...N and HT,pad. We empirically found these layers to be most important, as they allow for direct communication between the sparse source descriptors and the dense target feature maps, prior to the correspondence maps computation. After trying different values for k and with memory consumption in mind, we settled for k = 4 in all our experiments.
Affine transformation. The affine transformation matrix KC for the correspondence map CT of resolution (WT, HT) is computed from the target image calibration matrix KT and downscaling factors s using:
KC = s 0 s−120 s s−12 0 0 1  KT + [0 0 γWT0 0 γHT 0 0 0 ] (2)
We can determine if a point lies within the boundaries of CT if its (x, y) coordinates are between (−γWT,−γHT) and ((1 + γ)WT, (1 + γ)HT).
Implementation. The model is implemented in PyTorch (Paszke et al., 2017). For an indoor sample with 2000 keypoints it has an average throughput of 8.84 image/s on an NVIDIA RTX 3070 GPU. We report the number of parameters in our model in Table 1.

heading: C.2 DATASETS AND TRAINING DETAILS
text: ScanNet. The ScanNet (Dai et al., 2017) dataset is a large-scale indoor dataset containing monocular RGB videos and dense depth images, along with ground truth absolute camera poses. As SuperGlue (Sarlin et al., 2020) and LoFTR (Sun et al., 2021), we pre-compute the visual overlaps between all image pairs for both training and test scenes. For the training set we sample images with a visual overlap between 2% and 50% from the ScanNet training scenes, which provides us with challenging images to handle. We assemble 6M image pairs and randomly subsample 200k pairs at every training epoch. For testing images, we sample 2, 500 image pairs with overlaps between 2% and 80% from the ScanNet testing scenes, using several bins to ensure the sampling is close to being uniform. For both training and testing images, we sample keypoints in the source image along a regular grid with cell sizes of 16 pixels. We remove keypoints with invalid depth, as well as those where the local depth gradient is too high, as the depth information might not be reliable. We mark keypoints falling outside the target image plane as being outpainted, and we automatically detect the keypoints to inpaint through a cyclic projection of the source keypoints to the target image and back. The remaining keypoints are labeled as identifiable. For all ScanNet experiments, NeurHal uses a 1/8 output-to-input resolution ratio, with a target correspondence map maximum edge size of 80 pixels (when γ = 0%).
Megadepth. We use Megadepth (Li & Snavely, 2018) to train and evaluate NeurHal on outdoor images. This dataset contains over one million images captured in touristic places, and split in 196 scenes. To train NeurHal and following Germain et al. (2021) guidelines, we use the provided SIFT (Lowe, 2004)-based 3D reconstruction which was made with COLMAP (Schönberger & Frahm, 2016). Because the sparse 3D point cloud comes from SfM, we find however that very little keypoints can be marked as inpainted. Indeed, no 3D reconstruction is applied to objects or people occluding the scene. To allow for a wide variety of image pairs we use the sparse reconstruction to estimate the visual overlap and sample pairs with an overlap between 20% and 100%. We however find this overlap estimation to be quite unreliable, as only part of the scene is usually reconstructed. Since Megadepth (Li & Snavely, 2018) images are of much higher resolution than ScanNet (Dai et al., 2017), we configure NeurHal to use a 1/16 output-to-input resolution (with a simple max-pooling layer in the CNN). We set the target correspondence map maximum edge size of 60 pixels (when γ = 0%), to allow for space in memory when γ = 50%.
Overlap estimation. For a given pair of images, we approximate the visual overlap by computing the covisibility ratio of keypoints for every image pair. For a given source and target image pair, we first compute the source-to-target and target-to-source covisibility ratios using ground truth depth data and camera poses. We then define the visual overlap as the minimum between both ratios. On Megadepth we find this overlap estimation to be fairly noisy, as depth is only partially known.
Optimizers and scheduling. On both datasets NeurHal is trained for a maximum of 40 epochs. We use an initial learning rate of 10−3, with a linear learning rate warm-up in 3 epochs from 0.1 of the initial learning rate. As Sun et al. (2021), we decay the learning rate by 0.5 every 8 epochs starting from the 8th epoch. We apply the linear scaling rule and use a batch size of 8 over 8 NVIDIA V100 GPUs. We use the AdamW (Loshchilov & Hutter, 2019) optimizer, with a weight decay of 0.1. In all training procedures, we randomly initialize the model weights.

heading: C.3 EVALUATION DETAILS
text: Evaluation protocol. All baselines follow the same standard protocol in which we: 1) Compute 2D-2D correspondences between the reference image and the query image, 2) Lift these 2D-2D correspondences to 2D-3D correspondences using the available 3D information for the reference image, 3) Estimate the camera pose given these 2D-3D correspondences by minimizing the Reprojection Error (RE), i.e. applying LO-RANSAC+PnP (Chum et al., 2003) followed by a non-linear iterative refinement. This approach is widely used and leads to state-of-the-art results in visual localization benchmarks. We also include results for Germain et al. (2021) which we call S2D. For the evaluation of Fig. 4, we find the inpainted and outpainted correspondents for LoFTR (Sun et al., 2021) and DRCNet (Li et al., 2020) by fetching the argmax 2D coordinates in the 4D matching confidence volume. For S2D and NeurHal, we simply take the argmax in correspondence maps for the same set of keypoints.
Choice of threshold. We reported in Fig. 14 the performance of NeurHal, state-of-the-art feature matching methods and the identity pose, on ScanNet for several rotation and translation thresholds. We can see that arbitrarily choosing a threshold of τt = 1.5m and τr = 20.0◦ sets a hard objective as the identity pose is particularly poor.
(Chum et al., 2003)-based pose estimator. For all (Chum et al., 2003)-based methods, we estimate the camera pose using the pycolmap python binding. We tune the RANSAC threshold for optimal performance, and mark all cases where less than 3 valid correspondences (i.e. with a valid depth value) as failure cases (infinite pose error). The remaining parameters are left as default. We follow the evaluation instructions provided by each method, and use indoor weights for SP+SG (Sarlin et al., 2020) and the dual-softmax indoor weights for LoFTR (Sun et al., 2021). In the case of NeurHal + (Chum et al., 2003), we simply read the argmax of the predicted correspondence maps to obtain explicit 2D-to-3D correspondences.
(Germain et al., 2021)-based pose estimator. For both S2D (Germain et al., 2021) and NeurHal we only use coarse models, which operate at either 1/8th or 1/16th of the original input resolution. We first retrain the S2D coarse model (fully-convolutional Inceptionv3 (Szegedy et al., 2016), up to Mixed-6e) on the same training set as our method, with the same target resolution of 80 pixels. We refer to this model as S2D. Given correspondence maps and the depth map of the source image, we estimate the camera pose between the target image and the source image using the method proposed in Germain et al. (2021). For both S2D and NeurHal we use the same set of regularly sampled source keypoints (see Sec. C.1), and we perform camera pose estimation first using P3P inside an MSAC (Torr & Zisserman, 2000) loop. We run P3P for a maximum of 5, 000 iterations over the top-20% correspondences. We then apply a coarse GNC (Blake & Zisserman, 1987) over all source keypoints with σmax = 2.0 and σmin = 0.6. Let us highlight that in all the camera pose experiments, the performances of NeurHal are obtained by predicting only low resolution correspondence maps (see Sec. C.1).

heading: D ADDITIONAL QUALITATIVE RESULTS

heading: D.1 GENERALIZATION TO NEW DATASETS
text: So far we have demonstrated the ability of NeurHal to hallucinate correspondences on unseen validation scenes from both ScanNet (Dai et al., 2017) and Megadepth (Li & Snavely, 2018). In order to further demonstrate the generalization capacity of NeurHal, we report qualitative results obtained on the NYU Depth Dataset (Nathan Silberman & Fergus, 2012) in Fig. 16 and on the ETH-3D (Schöps et al., 2017) dataset in Fig. 15. We use the set of indoor weights for NYU (i.e. NeurHal trained on ScanNet) and outdoor weights for ETH-3D (i.e. NeurHal trained on MegaDepth). We report the overlayed and upsampled coarse truncated loss map computed following Germain et al. (2021) on low-overlap image pairs. We find that NeurHal is able to robustly outpaint correspondences despite little visual overlaps and strong relative camera motions. These visuals demonstrate the strong generalization ability of NeurHal.

heading: D.2 QUALITATIVE CORRESPONDENCE HALLUCINATION RESULTS AND FAILURE CASES
text: To further demonstrate the ability of NeurHal to perform visual correspondence hallucination, we report in Fig. 17 and Fig. 18 qualitative results on ScanNet (Dai et al., 2017) and Megadepth (Li & Snavely, 2018) respectively on scenes that were not seen at training-time. In the target image and in the (negative log) correspondence map, the red dot represents the ground truth keypoint’s correspondent. The dashed rectangles represent the borders of the target images.
Let us recall that NeurHal outputs probability distributions (a.k.a. correspondence maps) assuming the two input images are partially overlapping. It is essential to keep this assumption in mind when looking at these qualitative results. For instance, concerning the example Fig. 17 (b) (middle), it is very difficult for our human visual system to be sure that the two images are actually overlapping, and consequently the network prediction seems to good to be true. However, if we assume that there is an overlap, we realize that it is actually possible to perform correspondence hallucination, by drawing out the two skirting boards, to correctly outpaint the correspondent.
In fact, this overlapping assumption has a regularization effect in cases where the covisible image areas show no distinctive regions, and one image could be at an infinite translation of the other, e.g. Fig. 17 (b) (second to last).
In Fig. 17 (d) and Fig. 18 (d) we show failure cases where the correspondence maps modes predicted by NeurHal are either partially or completely off. We find that failure cases often correlate with strongly ambiguous image pairs, or images that have extremely limited visual overlap.

heading: D.3 QUALITATIVE CAMERA POSE ESTIMATION RESULTS
text: We show in Fig. 19 qualitative results in camera pose estimation on low-overlap images from ScanNet (Dai et al., 2017), for NeurHal and its three best-performing competitors. For every method we display the keypoints used as input to the camera pose estimator in the source image, along with their reprojection at the estimated camera pose in the target image. For methods using the pose estimator from (Chum et al., 2003), the keypoints are those that have been successfully matched. When using the pose estimator of Germain et al. (2021), the keypoints are those involved in the prediction of the dense NRE maps. We color in keypoints based on their spatial 2D position in the source image. We find that NeurHal strongly benefits from its outpainting ability, in comparison with all other competitors which struggle to find both sufficient and reliable correspondences. We also report in Fig. 20 failure cases for NeurHal. We find that such cases correspond to image pairs exhibiting extremely limited visual overlap, strong camera pose rotations and overall significant ambiguities.

Review summaries:
The paper proposes a method that accepts two partially overlapping images and a keypoint in one of them, and detects the location of the keypoint in the other image, regardless of whether it is visible, occluded, or outside the frame of the image. This is an interesting re-formulation of the correspondence estimation problem, the conventional formulation of which considers only keypoints that are visible in both images. Under the previous formulation, correspondence estimators are considered successful if they can declare that the correspondent of the keypoint is invisible in the target image. In addition to the capability to predict the location of invisible keypoints in isolation, the paper demonstrates that this capability is beneficial to camera pose estimation.

The paper proposed a deep method (NeurHal) to predict visible, occluded or out-of-view keypoint matching from source images to target images. In training, a correspondence map is obtained from ground truth of camera matrix/pose, and images. In testing, the model directly outputs three categories of matchings: identified, outputpainting and inpainting. As an application of NeurHal, the method is applied to camera pose estimation and tested on ScanNet and Megadepth. The experiments shows the method improves the estimation accuracy, particularly the outputpainting correspondences. 

In this manuscript, authors proposed a new problem of correspondence hallucination, in which for keypoint in the source image, its correspondence should be detected regardless if it is occluded or outside the field of view. In particular, the authors proposed a new model and training paradigm that learn to hallucinate correspondence by predicting its probability distribution of its location. Extensive experiments on both indoor and outdoor benchmarks demonstrated that proposed method  can help camera pose estimation and outperforms prior state of the art feature matching approaches.