Visual Correspondence Hallucination

1 INTRODUCTION . Establishing correspondences between two partially overlapping images is a fundamental computer vision problem with many applications . For example , state-of-the-art methods for visual localization from an input image rely on keypoint matches between the input image and a reference image ( Sattler et al. , 2018 ; Sarlin et al. , 2019 ; 2020 ; Revaud et al. , 2019 ) . However , these local feature matching methods will still fail when few keypoints are covisible , i.e . when many image locations in one image are outside the field of view or become occluded in the second image . These failures are to be expected since these methods are pure pattern recognition approaches that seek to identify correspondences , i.e . to find correspondences in covisible regions , and consider the non-covisible regions as noise . By contrast , humans explain the presence of these non-covisible regions through geometric reasoning and consequently are able to hallucinate ( i.e . predict ) correspondences at those locations . Geometric reasoning has already been used in computer vision for image matching , but usually as an a posteriori processing ( Fischler & Bolles , 1981 ; Luong & Faugeras , 1996 ; Barath & Matas , 2018 ; Chum et al. , 2003 ; 2005 ; Barath et al. , 2019 ; 2020 ) . These methods seek to remove outliers from the set of correspondences produced by a local feature matching approach using only limited geometric models such as epipolar geometry or planar assumptions . Contributions . In this paper we tackle the problem of correspondence hallucination . In doing so we seek to answer two questions : ( i ) can we derive a network architecture able to learn to hallucinate correspondences ? and ( ii ) is correspondence hallucination beneficial for absolute pose estimation ? The answer to these questions is the main novelty of this paper . More precisely , we consider a network that takes as input a pair of partially overlapping source/target images and keypoints in the source image , and outputs for each keypoint a probability distribution over its correspondent ’ s location in the target image plane . We propose to train this network to both identify and hallucinate the keypoints ’ correspondents . We call the resulting method NeurHal , for Neural Hallucinations . To the best of our knowledge , learning to hallucinate correspondences is a virgin territory , thus we first provide an analysis of the specific features of that novel learning task . This analysis guides us towards employing an appropriate loss function and designing the architecture of the network . After training the network , we experimentally demonstrate that it is indeed able to hallucinate correspondences on unseen pairs of images captured in novel scenes . We also apply this network to a camera pose estimation problem and find it is significantly more robust than state-of-the-art local feature matching-based competitors . 2 RELATED WORK . To the best of our knowledge , aiming at hallucinating visual correspondences has never been done but the related fields of local feature description and matching are immensely vast , and we focus here only on recent learning-based approaches . Learning-based local feature description . Using deep neural networks to learn to compute local feature descriptors have shown to bring significant improvements in invariance to viewpoint and illumination changes compared to handcrafted methods ( Csurka & Humenberger , 2018 ; Gauglitz et al. , 2011 ; Salahat & Qasaimeh , 2017 ; Balntas et al. , 2017 ) . Most methods learn descriptors locally around pre-computed covisible interest regions in both images ( Yi et al. , 2016 ; Detone et al. , 2018 ; Balntas et al. , 2016a ; Luo et al. , 2019 ) , using convolutional-based siamese architectures trained with a contrastive loss ( Gordo et al. , 2016 ; Schroff et al. , 2015 ; Balntas et al. , 2016b ; Radenović et al. , 2016 ; Mishchuk et al. , 2017 ; Simonyan et al. , 2014 ) , or using pose ( Wang et al. , 2020 ; Zhou et al. , 2021 ) or self ( Yang et al. , 2021 ) supervision . To further improve the performances , ( Dusmanu et al. , 2019 ; Revaud et al. , 2019 ) propose to jointly learn to detect and describe keypoints in both images , while Germain et al . ( 2020 ) only detects in one image and densely matches descriptors in the other . Learning-based local feature matching . All the methods described in the previous paragraph establish correspondences by comparing descriptors using a simple operation such as a dot product . Thus the combination of such a simple matching method with a siamese architecture inevitably produces outlier correspondences , especially in non-covisible regions . To reduce the amount of outliers , most approaches employ so-called Mutual Nearest Neighbor ( MNN ) filtering . However , it is possible to go beyond a simple MNN and learn to match descriptors . Learning-based matching methods ( Zhang et al. , 2019 ; Brachmann & Rother , 2019 ; Moo Yi et al. , 2018 ; Sun et al. , 2020 ; Choy et al. , 2020 ; 2016 ) take as input local descriptors and/or putative correspondences , and learn to output correspondences probabilities . However , all these matching methods focus only on predicting correctly covisible correspondences . Jointly learning local feature description and matching . Several methods have recently proposed to jointly learn to compute and match descriptors ( Sarlin et al. , 2020 ; Sun et al. , 2021 ; Li et al. , 2020 ; Rocco et al. , 2018 ; 2020 ) . All these methods use a siamese Convolutional Neural Network ( CNN ) to obtain dense local descriptors , but they significantly differ regarding the way they establish matches . They actually fall into two categories . The first category of methods ( Li et al. , 2020 ; Rocco et al. , 2018 ; 2020 ) computes a 4D correlation tensor that essentially represents the scores of all the possible correspondences . This 4D correlation tensor is then used as input to a second network that learns to modify it using soft-MNN and 4D convolutions . Instead of summarizing all the information into a 4D correlation tensor , the second category of methods ( Sarlin et al. , 2020 ; Sun et al. , 2021 ) rely on Transformers ( Vaswani et al. , 2017 ; Dosovitskiy et al. , 2020 ; Ramachandran et al. , 2019 ; Caron et al. , 2021 ; Cordonnier et al. , 2020 ; Zhao et al. , 2020 ; Katharopoulos et al. , 2020 ) to let the descriptors of both images communicate and adapt to each other . All these methods again focus on identifying correctly covisible correspondences and consider non-covisible correspondences as noise . While our architecture is closely related to the second category of methods as we also rely on Transformers , the motivation for using it is quite different since it is our goal of hallucinating correspondences that calls for a non-siamese architecture ( see Sec.3 ) . Visual content hallucination . ( Yang et al. , 2019 ) proposes to hallucinate the content of RGB-D scans to perform relative pose estimation between two images . More recently ( Chen et al. , 2021 ) regresses distributions over relative camera poses for spherical images using joint processing of both images . The work of ( Yang et al. , 2020 ; Qian et al. , 2020 ; Jin et al. , 2021 ) shows that employing a hallucinate-then-match paradigm can be a reliable way of recovering 3D geometry or relative pose from sparsely sampled images . In this work , we focus on the problem of correspondence hallucination which unlike previously mentioned approaches does not aim at recovering explicit visual content or directly regressing a camera pose . Perhaps closest to our goal is Cai et al . ( 2021 ) that seeks to estimate a relative rotation between two non-overlapping images by learning to reason about “ hidden ” cues such as direction of shadows in outdoor scenes , parallel lines or vanishing points . 3 OUR APPROACH . Our goal is to train a network that takes as input a pair of partially overlapping source/target images and keypoints in the source image , and outputs for each keypoint a probability distribution over its correspondent ’ s location in the target image plane , regardless of this correspondent being visible , occluded , or outside the field of view . While the problem of learning to find the location of a visible correspondent received a lot of attention in the past few years ( see Sec . 2 ) , to the best of our knowledge , this paper is the first attempt of learning to find the location of a correspondent regardless of this correspondent being visible , occluded , or outside the field of view . Since this learning task is virgin territory , we first analyze its specific features below , before defining a loss function and a network architecture able to handle these features . 3.1 ANALYSIS OF THE PROBLEM . The task of finding the location of a correspondent regardless of this correspondent being visible , occluded , or outside the field of view actually leads to three different problems . Before stating those three problems , let us first recall the notion of correspondent as it is the keystone of our problem . Correspondent . Given a keypoint pS ∈ R2 in the source image IS , its depth dS ∈ R+ , and the relative camera pose RTS ∈ SO ( 3 ) , tTS ∈ R3 between the coordinate systems of IS and the target image IT , the correspondent pT ∈ R2 of pS in the target image plane is obtained by warping pS : pT : = ω ( dS , pS , RTS , tTS ) : = KTπ ( dSRTSK −1 S pS + tTS ) , where KS and KT are the camera calibration matrices of source and target images and π ( u ) : = [ ux/uz , uy/uz , 1 ] T is the projection function . In a slight abuse of notation , we do not distinguish a homogeneous 2D vector from a non-homogeneous 2D vector . Let us highlight that the correspondent pT of pS may not be visible , i.e . it may be occluded or outside the field of view . Identifying the correspondent . In the case where a network has to establish a correspondence between a keypoint pS in IS and its visible correspondent pT in IT , standard approaches , such as comparing a local descriptor computed at pS in IS with local descriptors computed at detected keypoints in IT , are applicable to identify the correspondent pT . Outpainting the correspondent . When pT is outside the field of view of IT , there is nothing to identify , i.e . neither can pT be detected as a keypoint nor can a local descriptor be computed at that location . Here the network first needs to identify correspondences in the region where IT overlaps with IS and realize that the correspondent pT is outside the field of view to eventually outpaint it ( see Fig . 1 ) . We call this operation `` outpainting the correspondent '' as the network needs to predict the location of pT outside the field of view of IT . Inpainting the correspondent . When pT is occluded in IT , the problem is even more difficult since local features can be computed at that location but will not match the local descriptor computed at pS in IS . As in the outpainting case , the network needs to identify correspondences in the region where IT overlaps with IS and realize that the correspondent pT is occluded to eventually inpaint the correspondent pT ( see Fig . 1 ) . We call this operation `` inpainting the correspondent '' as the network needs to predict the location of pT behind the occluding object . Let us now introduce a loss function and an architecture that are able to unify the identifying , inpainting and outpainting tasks .

Summary: In this manuscript, authors proposed a new problem of correspondence hallucination, in which for keypoint in the source image, its correspondence should be detected regardless if it is occluded or outside the field of view. In particular, the authors proposed a new model and training paradigm that learn to hallucinate correspondence by predicting its probability distribution of its location. Extensive experiments on both indoor and outdoor benchmarks demonstrated that proposed method  can help camera pose estimation and outperforms prior state of the art feature matching approaches.