Neural Outlier Rejection for Self-Supervised Keypoint Learning

Identifying salient points in images is a crucial component for visual odometry , Structure-from-Motion or SLAM algorithms . Recently , several learned keypoint methods have demonstrated compelling performance on challenging benchmarks . However , generating consistent and accurate training data for interest-point detection in natural images still remains challenging , especially for human annotators . We introduce IO-Net ( i.e . InlierOutlierNet ) , a novel proxy task for the self-supervision of keypoint detection , description and matching . By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework , we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching . Second , we introduce KeyPointNet , a keypoint-network architecture that is especially amenable to robust keypoint detection and description . We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task , and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution . Through extensive experiments and ablative analysis , we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.† 1 INTRODUCTION . Detecting interest points in RGB images and matching them across views is a fundamental capability of many robotic systems . Tasks such Simultaneous Localization and Mapping ( SLAM ) ( Cadena et al. , 2016 ) , Structure-from-Motion ( SfM ) ( Agarwal et al. , 2010 ) and object detection assume that salient keypoints can be detected and re-identified in a wide range of scenarios , which requires invariance properties to lighting effects , viewpoint changes , scale , time of day , etc . However , these tasks still mostly rely on handcrafted image features such as SIFT ( Lowe et al. , 1999 ) or ORB ( Rublee et al. , 2011 ) , which have been shown to be limited in performance when compared to learned alternatives ( Balntas et al. , 2017 ) . Deep learning methods have revolutionized many computer vision applications including 2D/3D object detection ( Lang et al. , 2019 ; Tian et al. , 2019 ) , semantic segmentation ( Li et al. , 2018 ; Kirillov et al. , 2019 ) , human pose estimation ( Sun et al. , 2019 ) , etc . However , most learning algorithms need supervision and rely on labels which are often expensive to acquire . Moreover , supervising interest point detection is unnatural , as a human annotator can not readily identify salient regions in images as well as key signatures or descriptors , which would allow their re-identification . Selfsupervised learning methods have gained in popularity recently , being used for tasks such as depth regression ( Guizilini et al. , 2019 ) , tracking ( Vondrick et al. , 2018 ) and representation learning ( Wang et al. , 2019 ; Kolesnikov et al. , 2019 ) . Following DeTone et al . ( 2018b ) and Christiansen et al . ( 2019 ) , we propose a self-supervised methodology for jointly training a keypoint detector as well as its associated descriptor . †Code : https : //github.com/TRI-ML/KP2D Our main contributions are : ( i ) We introduce IO-Net ( i.e . InlierOutlierNet ) , a novel proxy task for the self-supervision of keypoint detection , description and matching . By using a neurally-guided outlier-rejection scheme ( Brachmann & Rother , 2019 ) as an auxiliary task , we show that we are able to simultaneously self-supervise keypoint description and generate optimal inlier sets from possible corresponding point-pairs . While the keypoint network is fully self-supervised , the network is able to effectively learn distinguishable features for two-view matching , via the flow of gradients from consistently matched point-pairs . ( ii ) We introduce KeyPointNet , and propose two modifications to the keypoint-network architecture described in Christiansen et al . ( 2019 ) . First , we allow the keypoint location head to regress keypoint locations outside their corresponding cells , enabling keypoint matching near and across cell-boundaries . Second , by taking advantage of sub-pixel convolutions to interpolate the descriptor feature-maps to a higher resolution , we show that we are able to improve the fine-grained keypoint descriptor fidelity and performance especially as they retain more fine-grained detail for pixel-level metric learning in the self-supervised regime . Through extensive experiments and ablation studies , we show that the proposed architecture allows us to establish state-of-the-art performance for the task of self-supervised keypoint detection , description and matching . 2 RELATED WORK . The recent success of deep learning-based methods in many computer vision applications , especially feature descriptors , has motivated general research in the direction of image feature detection beyond handcrafted methods . Such state-of-the-art learned keypoint detectors and descriptors have recently demonstrated improved performance on challenging benchmarks ( DeTone et al. , 2018b ; Christiansen et al. , 2019 ; Sarlin et al. , 2019 ) . In TILDE ( Verdie et al. , 2015 ) , the authors introduced multiple piece-wise linear regression models to detect features under severe changes in weather and lighting conditions . To train the regressors , they generate pseudo ground truth interest points by using a Difference-of-Gaussian ( DoG ) detector ( Lowe , 2004 ) from an image sequence captured at different times of day and seasons . LIFT ( Yi et al. , 2016 ) is able to estimate features which are robust to significant viewpoint and illumination differences using an end-to-end learning pipeline consisting of three modules : interest point detection , orientation estimation and descriptor computation . In LF-Net ( Ono et al. , 2018 ) , the authors introduced an end-to-end differentiable network which estimates position , scale and orientation of features by jointly optimizing the detector and descriptor in a single module . Quad-networks ( Savinov et al. , 2017 ) introduced an unsupervised learning scheme for training a shallow 2-layer network to predict feature points . SuperPoint ( DeTone et al. , 2018b ) is a selfsupervised framework that is trained on whole images and is able to predict both interest points and descriptors . Its architecture shares most of the computation in the detection and description modules , making it fast enough for real-time operation , but it requires multiple stages of training which is not desirable in practice . Most recently , UnsuperPoint ( Christiansen et al. , 2019 ) presented a fast deeplearning based keypoint detector and descriptor which requires only one round of training in a selfsupervised manner . Inspired by SuperPoint , it also shares most of the computation in the detection and description modules , and uses a siamese network to learn descriptors . They employ simple homography adaptation along with non-spatial image augmentations to create the 2D synthetic views required to train their self-supervised keypoint estimation model , which is advantageous because it trivially solves data association between these views . In their work , Christiansen et al . ( 2019 ) predict keypoints that are evenly distributed within the cells and enforce that the predicted keypoint locations do not cross cell boundaries ( i.e . each cell predicts a keypoint inside it ) . We show that this leads to sub-optimal performance especially when stable keypoints appear near cell borders . Instead , our method explicitly handles the detection and association of keypoints across cell-boundaries , thereby improving the overall matching performance . In Self-Improving Visual Odometry ( DeTone et al. , 2018a ) , the authors first estimate 2D keypoints and descriptors for each image in a monocular sequence using a convolutional network , and then use a bundle adjustment method to classify the stability of those keypoints based on re-projection error , which serves as supervisory signal to retrain the model . Their method , however , is not fully differentiable , so it can not be trained in an end-to-end manner . Instead , we incorporate an end-to-end differentiable and neurally-guided outlierrejection mechanism ( IO-Net ) that explicitly generates an additional proxy supervisory signal for the matching input keypoint-pairs identified by our KeyPointNet architecture . This allows the keypoint descriptions to be further refined as a result of the outlier-rejection network predictions occurring during the two-view matching stage . 3 SELF-SUPERVISED KEYPOINT LEARNING . In this work , we aim to regress a function which takes as input an image and outputs keypoints , descriptors , and scores . Specifically , we define K : I → { p , f , s } , with input image I ∈ R3×H×W , and output keypoints p = { [ u , v ] } ∈ R2×N , descriptors f ∈ R256×N and keypoint scores s ∈ RN ; N represents the total number of keypoints extracted and it varies according to an input image resolution , as defined in the following sections . We note that throughout this paper we use p to refer to the set of keypoints extracted from an image , while p is used to refer to a single keypoint . Following the work of Christiansen et al . ( 2019 ) , we train the proposed learning framework in a self-supervised fashion by receiving as input a source image Is such that K ( Is ) = { ps , fs , ss } and a target image It such that K ( It ) = { pt , ft , st } . Images Is and It are related through a known homography transformation H which warps a pixel from the source image and maps it into the target image . We define p∗t = { [ u∗i , v∗i ] } = H ( ps ) , with i ∈ I - e.g . the corresponding locations of source keypoints ps after being warped into the target frame . Inspired by recent advances in Neural Guided Sample Consensus methods ( Brachmann & Rother , 2019 ) , we define a second function C which takes as input point-pairs along with associated weights according to a distance metric , and outputs the likelihood that each point-pair belongs to an inlier set of matches . Formally , we define C : { ps , p∗t , d ( fs , f∗t ) } ∈ R5×N → RN as a mapping which computes the probability that a point-pair belongs to an inlier set . We note that C is only used at training time to choose an optimal set of consistent inliers from possible corresponding point pairs and to encourage the gradient flow through consistent point-pairs . An overview of our method is presented in Figure 1 . We define the model K parametrized by θK as an encoder-decoder style network . The encoder consists of 4 VGG-style blocks stacked to reduce the resolution of the image H ×W to Hc×Wc = H/8×W/8 . This allows an efficient prediction for keypoint location and descriptors . In this low resolution embedding space , each pixel corresponds to an 8 × 8 cell in the original image . The decoder consists of 3 separate heads for the keypoints , descriptors and scores respectively . Thus for an image of input size H ×W , the total number of keypoints regressed is ( H ×W ) /64 , each with a corresponding score and descriptor . For every convolutional layer except the final one , batch normalization is applied with leakyReLU activation . A detailed description of our network architecture can be seen in Figure 2 . The IO-Net is a 1D CNN parametrized by θIO , for which we follow closely the structure from Brachmann & Rother ( 2019 ) with 4 default setting residual blocks and the original activation function for final layer is removed . A more detailed description of these networks can be found in the Appendix ( Tables 6 and 7 ) .

Summary: The paper is devoted to self-supervised learning of local features (both detectors and descriptors simultaneously). The problem is old yet not fully solved yet, because handcrafted SIFT is still winning the benchmarks. This work mostly follows and improves upon SuperPoint (DeTone et.al 2017) and the follow-up work UnsuperPoint (Christiansen et.al 2019) architecture and training scheme.