Neural Outlier Rejection for Self-Supervised Keypoint Learning

Identifying salient points in images is a crucial component for visual odometry , Structure-from-Motion or SLAM algorithms . Recently , several learned keypoint methods have demonstrated compelling performance on challenging benchmarks . However , generating consistent and accurate training data for interest-point detection in natural images still remains challenging , especially for human annotators . We introduce IO-Net ( i.e . InlierOutlierNet ) , a novel proxy task for the self-supervision of keypoint detection , description and matching . By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework , we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching . Second , we introduce KeyPointNet , a keypoint-network architecture that is especially amenable to robust keypoint detection and description . We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task , and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution . Through extensive experiments and ablative analysis , we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.† 1 INTRODUCTION . Detecting interest points in RGB images and matching them across views is a fundamental capability of many robotic systems . Tasks such Simultaneous Localization and Mapping ( SLAM ) ( Cadena et al. , 2016 ) , Structure-from-Motion ( SfM ) ( Agarwal et al. , 2010 ) and object detection assume that salient keypoints can be detected and re-identified in a wide range of scenarios , which requires invariance properties to lighting effects , viewpoint changes , scale , time of day , etc . However , these tasks still mostly rely on handcrafted image features such as SIFT ( Lowe et al. , 1999 ) or ORB ( Rublee et al. , 2011 ) , which have been shown to be limited in performance when compared to learned alternatives ( Balntas et al. , 2017 ) . Deep learning methods have revolutionized many computer vision applications including 2D/3D object detection ( Lang et al. , 2019 ; Tian et al. , 2019 ) , semantic segmentation ( Li et al. , 2018 ; Kirillov et al. , 2019 ) , human pose estimation ( Sun et al. , 2019 ) , etc . However , most learning algorithms need supervision and rely on labels which are often expensive to acquire . Moreover , supervising interest point detection is unnatural , as a human annotator can not readily identify salient regions in images as well as key signatures or descriptors , which would allow their re-identification . Selfsupervised learning methods have gained in popularity recently , being used for tasks such as depth regression ( Guizilini et al. , 2019 ) , tracking ( Vondrick et al. , 2018 ) and representation learning ( Wang et al. , 2019 ; Kolesnikov et al. , 2019 ) . Following DeTone et al . ( 2018b ) and Christiansen et al . ( 2019 ) , we propose a self-supervised methodology for jointly training a keypoint detector as well as its associated descriptor . †Code : https : //github.com/TRI-ML/KP2D Our main contributions are : ( i ) We introduce IO-Net ( i.e . InlierOutlierNet ) , a novel proxy task for the self-supervision of keypoint detection , description and matching . By using a neurally-guided outlier-rejection scheme ( Brachmann & Rother , 2019 ) as an auxiliary task , we show that we are able to simultaneously self-supervise keypoint description and generate optimal inlier sets from possible corresponding point-pairs . While the keypoint network is fully self-supervised , the network is able to effectively learn distinguishable features for two-view matching , via the flow of gradients from consistently matched point-pairs . ( ii ) We introduce KeyPointNet , and propose two modifications to the keypoint-network architecture described in Christiansen et al . ( 2019 ) . First , we allow the keypoint location head to regress keypoint locations outside their corresponding cells , enabling keypoint matching near and across cell-boundaries . Second , by taking advantage of sub-pixel convolutions to interpolate the descriptor feature-maps to a higher resolution , we show that we are able to improve the fine-grained keypoint descriptor fidelity and performance especially as they retain more fine-grained detail for pixel-level metric learning in the self-supervised regime . Through extensive experiments and ablation studies , we show that the proposed architecture allows us to establish state-of-the-art performance for the task of self-supervised keypoint detection , description and matching . 2 RELATED WORK . The recent success of deep learning-based methods in many computer vision applications , especially feature descriptors , has motivated general research in the direction of image feature detection beyond handcrafted methods . Such state-of-the-art learned keypoint detectors and descriptors have recently demonstrated improved performance on challenging benchmarks ( DeTone et al. , 2018b ; Christiansen et al. , 2019 ; Sarlin et al. , 2019 ) . In TILDE ( Verdie et al. , 2015 ) , the authors introduced multiple piece-wise linear regression models to detect features under severe changes in weather and lighting conditions . To train the regressors , they generate pseudo ground truth interest points by using a Difference-of-Gaussian ( DoG ) detector ( Lowe , 2004 ) from an image sequence captured at different times of day and seasons . LIFT ( Yi et al. , 2016 ) is able to estimate features which are robust to significant viewpoint and illumination differences using an end-to-end learning pipeline consisting of three modules : interest point detection , orientation estimation and descriptor computation . In LF-Net ( Ono et al. , 2018 ) , the authors introduced an end-to-end differentiable network which estimates position , scale and orientation of features by jointly optimizing the detector and descriptor in a single module . Quad-networks ( Savinov et al. , 2017 ) introduced an unsupervised learning scheme for training a shallow 2-layer network to predict feature points . SuperPoint ( DeTone et al. , 2018b ) is a selfsupervised framework that is trained on whole images and is able to predict both interest points and descriptors . Its architecture shares most of the computation in the detection and description modules , making it fast enough for real-time operation , but it requires multiple stages of training which is not desirable in practice . Most recently , UnsuperPoint ( Christiansen et al. , 2019 ) presented a fast deeplearning based keypoint detector and descriptor which requires only one round of training in a selfsupervised manner . Inspired by SuperPoint , it also shares most of the computation in the detection and description modules , and uses a siamese network to learn descriptors . They employ simple homography adaptation along with non-spatial image augmentations to create the 2D synthetic views required to train their self-supervised keypoint estimation model , which is advantageous because it trivially solves data association between these views . In their work , Christiansen et al . ( 2019 ) predict keypoints that are evenly distributed within the cells and enforce that the predicted keypoint locations do not cross cell boundaries ( i.e . each cell predicts a keypoint inside it ) . We show that this leads to sub-optimal performance especially when stable keypoints appear near cell borders . Instead , our method explicitly handles the detection and association of keypoints across cell-boundaries , thereby improving the overall matching performance . In Self-Improving Visual Odometry ( DeTone et al. , 2018a ) , the authors first estimate 2D keypoints and descriptors for each image in a monocular sequence using a convolutional network , and then use a bundle adjustment method to classify the stability of those keypoints based on re-projection error , which serves as supervisory signal to retrain the model . Their method , however , is not fully differentiable , so it can not be trained in an end-to-end manner . Instead , we incorporate an end-to-end differentiable and neurally-guided outlierrejection mechanism ( IO-Net ) that explicitly generates an additional proxy supervisory signal for the matching input keypoint-pairs identified by our KeyPointNet architecture . This allows the keypoint descriptions to be further refined as a result of the outlier-rejection network predictions occurring during the two-view matching stage . 3 SELF-SUPERVISED KEYPOINT LEARNING . In this work , we aim to regress a function which takes as input an image and outputs keypoints , descriptors , and scores . Specifically , we define K : I → { p , f , s } , with input image I ∈ R3×H×W , and output keypoints p = { [ u , v ] } ∈ R2×N , descriptors f ∈ R256×N and keypoint scores s ∈ RN ; N represents the total number of keypoints extracted and it varies according to an input image resolution , as defined in the following sections . We note that throughout this paper we use p to refer to the set of keypoints extracted from an image , while p is used to refer to a single keypoint . Following the work of Christiansen et al . ( 2019 ) , we train the proposed learning framework in a self-supervised fashion by receiving as input a source image Is such that K ( Is ) = { ps , fs , ss } and a target image It such that K ( It ) = { pt , ft , st } . Images Is and It are related through a known homography transformation H which warps a pixel from the source image and maps it into the target image . We define p∗t = { [ u∗i , v∗i ] } = H ( ps ) , with i ∈ I - e.g . the corresponding locations of source keypoints ps after being warped into the target frame . Inspired by recent advances in Neural Guided Sample Consensus methods ( Brachmann & Rother , 2019 ) , we define a second function C which takes as input point-pairs along with associated weights according to a distance metric , and outputs the likelihood that each point-pair belongs to an inlier set of matches . Formally , we define C : { ps , p∗t , d ( fs , f∗t ) } ∈ R5×N → RN as a mapping which computes the probability that a point-pair belongs to an inlier set . We note that C is only used at training time to choose an optimal set of consistent inliers from possible corresponding point pairs and to encourage the gradient flow through consistent point-pairs . An overview of our method is presented in Figure 1 . We define the model K parametrized by θK as an encoder-decoder style network . The encoder consists of 4 VGG-style blocks stacked to reduce the resolution of the image H ×W to Hc×Wc = H/8×W/8 . This allows an efficient prediction for keypoint location and descriptors . In this low resolution embedding space , each pixel corresponds to an 8 × 8 cell in the original image . The decoder consists of 3 separate heads for the keypoints , descriptors and scores respectively . Thus for an image of input size H ×W , the total number of keypoints regressed is ( H ×W ) /64 , each with a corresponding score and descriptor . For every convolutional layer except the final one , batch normalization is applied with leakyReLU activation . A detailed description of our network architecture can be seen in Figure 2 . The IO-Net is a 1D CNN parametrized by θIO , for which we follow closely the structure from Brachmann & Rother ( 2019 ) with 4 default setting residual blocks and the original activation function for final layer is removed . A more detailed description of these networks can be found in the Appendix ( Tables 6 and 7 ) .

Summary: The following work proposes several improvements over prior works in unsupervised/self-supervised keypoint-descriptor learning such as Christiansen et al. One improvement is the relaxation of the cell-boundaries for keypoint prediction -- specifically allowing keypoints anchored at the cell's center to be offset into neighboring cells. Another change was the introduction of an inlier-outlier classifier network to be used as a proxy loss for the keypoint position and descriptors. They found the inlier-outlier loss to improve homography accuracy at 1 and 3 pixel thresholds.