Neural Outlier Rejection for Self-Supervised Keypoint Learning
Authors: Jiexiong Tang
Hanme Kim
Vitor Guizilini
Sudeep Pillai
Rares Ambrus
Venue:  (2020)
Keywords: Self-Supervised Learning
Keypoint Detection
Outlier Rejection
Deep Learning

Abstract: Identifying salient points in images is a crucial component for visual odometry, Structure-from-Motion or SLAM algorithms. Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks.  However, generating consistent and accurate training data for interest-point detection in natural images still remains challenging, especially for human annotators. We introduce IO-Net (i.e. InlierOutlierNet), a novel proxy task for the self-supervision of keypoint detection, description and matching. By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework, we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching. Second, we introduce KeyPointNet, a keypoint-network architecture that is especially amenable to robust keypoint detection and description. We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task, and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution. Through extensive experiments and ablative analysis, we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.

Published as a conference paper at ICLR 2020
NEURAL OUTLIER REJECTION FOR SELF-SUPERVISER KEYPOINT LEARNING
Jiexiong Tang Hanme Kim
Vitor Guizilini'
Sudeep Pillai' Rareş Ambrus'
1 Toyota Research Institute (TRI)
*KTH Royal Institute of Technology
[ffrrst. last aast]ttri.gg.om
ABSTRACT
Identifying salient points in images crucial component for visual odometry, Stucture-from-Motion or SLAM algorithms. Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks. However, generating consistent and accurate training data for interest-point de- tection in natural images still remains challenging, especially for human anno- tators. We introduce 10-Net (i.e. InlierOuflierNet), novel proxy task for the self-supervision of keypoint detection, description and matching. By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework, we show that are able to simultane- ously self-supervise keypoint description and improve keypoint matching. Sec- ond, we introduce KeyPointNet, keypoint-networf architecture that is especially amenable robust keypoint detection and description. We design the network allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task, and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsam- ple the descriptor feature-maps to higher operating resolution. Through exten- sive experiments and ablative analysis, show that the proposed self-supervised keypoint learning method greatly improves the quality feature matching and homography estimation challenging benchmarks over the state-of trrt
INTRODUCTION
Detecting interest points in RGB images and matching them across views is a fundamental capa- bility of many robotic systems. Tasks such Simultancous Localization and Mapping (SLAM) (Ca-) dena et al. 2016). sume that salient keypoints be detected and re-identified in wide range of scenarios, which requires invariance properties to lighting effects, viewpoint changes, scale, time of day, etc. How- ever, these tasks still mostly rely handcrafted image features such SIFT (Lowe al. 1999) or ORB (Rublee et al., (20TT), which have been shown to be limited in performance when compared to learned alternatives (Balntas et al. 2017).
(SfM) (Agarwal et al.) 2010) and object detection as-
Deep learning methods have revolutionized many computer vision applications including 2D/3D object detection (Lang et al. 2019 Tian et al. 2019). semantic segmentation (Li al. 2018 Kirilov et al., 2019), human pose estimation (Sun et al. 2019, etc. However, most learning algorithms need supervision and rely on labels which are often expensive to acquire. Moreover, supervising interest point detection is unnatural, in images as well as key signatures or descriptors, which would allow their re-identification. Self- supervised learning methods have gained in popularity recently. being used for tasks such as depth regression (Guizilini et al.) 2019), tracking /Vondrick et al. (0188) and representation learning (Wang et al. 2019 Kolesnikov al. 2019). Following De Tone et al. (2018b) and Christiansen et al. 2019)) we propose a self-supervised methodology for jointly training a keypoint detector as well as its associated deseriptor.
|
human annotator cannot readily identify salient regions
(oode: http://gtthu..omm/RRMMMXXXX

---

Published as a conference paper at ICLR 2020
Our main contributions are: (i) We introduce 1O-Net (i.e. InlierOuttierNet), novel proxy task for the self-supervision of keypoint detection, description and matching. By using a neurally-guided auxiliary task, we show that able to simultaneously self-supervise keypoint description and generate optimal inlier sets from possible corresponding point-pairs. While the keypoint network is fully self-supervised, the net- work is able to effectively learn distinguishable features for two-view matching, via the flow of gradients from consistently matched point-pairs. (ii) We introduce KeyPointNet, and propose two modifications to the eeypoint-networt architecture described in Christiansen et al. (2019). First, we allow the keypoint location head to regress keypoint locations outside their corresponding cells, en- abling keypoint matching near and across sell-boundaries. Second, by taking advantage of sub-pixel convolutions to interpolate the descriptor feature-maps to a higher resolution, we show that we are able to improve the fine-grained keypoint descriptor fidelity and performance especially as they re- tain more fine-grained detail for pixel-level metric learning in the self-supervised regime. Through extensive experiments and ablation studies, we show that the proposed architecture allows us to tablish oftt-oo--hh-arrr performance for the task of self-supervised keypoint detection, description and matching.
outlier-rejection scheme (Brachmann & Rother 2019)
2 RELATED WORK
The recent success of deep learning-basee methods in many computer vision applications, espe- cially feature descriptors, has motivated general research in the direction of image feature detection beyond handcrafted methods. Such tffte-of-the-an art learned keypoint detectors and descriptors have recently demonstrated improved performance on challenging benchmarks (DeTTone et al.) 2018b Christiansen et al., 2019 Sarlin et al., 2019). In TILDE (Verdie et al. 2015), the authors introduced multiple wise linear regression models to detect features under severe changes in weather and lighting conditions. To train the regressors, they generate pseudo ground truth interest points by using a officrence-o--Gausiia (DoG) detector (Lowe) 2004) from an image sequence captured at different times of day and seasons. LIFT (Yi et al. (016) is able to estimate features which are robust to significant viewpoint and illumination differences using an end-to-end learning pipeline consisting of three modules: interest point detection, orientation estimation and descriptor compu- tation. In LF-Net (Ono al., 2018),, the authors introduced an end-to-end differentiable network which estimates position, scale and orientation of features by jointly optimizing the detector and descriptor in a single module.
Quad-networks (Savinov et al. 2017) introduced an unsupervised learning scheme for training a shallow 2-layer network to predict feature points. SuperPoint (DeTone al., 2b18b) self- supervised framework that is trained on whole images and is able to predict both interest points and descriptors. Its architecture shares most of the computation in the detection and description modules, making it fast enough for real-time operation, but it requires multiple stages of training which is not desirable in practice. Most recently, UnsuperPoint (Christiansen et al., 2019) presented a fast deep- learning based keypoint detector and descriptor which requires only one round of training in a self- supervised manner. Inspired by SuperPoint, it also shares most of the computation in the detection and description modules, and uses a siamese network to learn descriptors. They employ simple homography adaptation along with non-spatial image augmentations to create the 2D synthetic views required to train their self- self-supervised keypoint estimation model, which is advantageous because it trivially solves data association between these views. In their work, Christiansen et al. (2019) predict keypoints that are evenly distributed within the cells and enforce that the predicted keypoint locations do not cross cell boundaries (i.e. each cell predicts a keypoint inside it). We show that this leads to sub-optimal performance especially when stable keypoints appear near cell borders. Instead, our method explicitly handles the detection and association of keypoints across cell- cell-boundaries, thereby improving the overall matching performance. Self-Improving Visual Odometry (DeTonee) et al. (018a), the authors first estimate 2D keypoints and descriptors for each image in a monocular sequence using a convolutional network, and then use a bundle adjustment method to classify the stability of those keypoints based on re-projection error, which serves as supervisory signal to re- train the model. Their method, however, is not fully differentiable, so it cannot be trained in an end-to-end manner. Instead, we incorporate an end-to-ent differentiable and neurally-guide outlier- rejection mechanism (1O-Net) that explicitly generates an additional proxy supervisory signal for the matching input keypoint-pairs identified by our Key PoinoNtt architecture. This allows the keypoint
2

---

Published as a conference paper at ICLR 2020
Indirect Supervisory Signal
Lio
KeyPointNet
Point Paii-ePiis
Matching
I0-Net
Targe: Frame
X
Direct Supervisory Signal
KeyPointNet
Lkeypoint
Liiese
Lecore
Source
Figure I: Our proposed framework for self-supervised keypoint detector and descriptor learning us- ing KeyPointNet and 10-Net. The KeyPointNet is optimized in end-to-end differentiable manner by imposing an explicit loss on each of the 3 target outputs (score, location and descriptor). Addi- tionally, the IO-Net produces indirect supervisory signal to KeyPointNet targets by propagating gradients from the classification of matching input point-pairs.
descriptions be further refined result of the outlier-rejectoon network predictions occurring during the two-view matching stage.
3 SELF-SUPERVISED KEYPOINT LEARNING
In this work, aim regress function which takes input image and outputs keypoints, descriptors, and scores. Specifically, we define K: --> {p. f.s). with input image / E 33 and output keypoints p {[u, v]} E g2xN descriptors E 25 and keypoint scores s RN; N represents the total number of keypoints extracted and it varies according to an input image resolution, as defined in the following sections. We note that throughout this paper we use p to refer to the set of keypoints extracted from an image, while is used to refer to a single keypoint.
Following the work of Christiansen al.) (2019), we train the proposed learning framework self-supervised fashion by receiving as input source image I, such that K(,,) = (Px, (,, and a target image J, such that (T+,) {Pt. fe, st Images I, and I are related through a known homography transformation H which warps pixel from the source image and maps it into the target image. We define P = {[u]. e]]} = H(P.). ). with E I e.g. the corresponding locations of source keypoints ps after being warped into the target frame.
Inspired by recent advances in Neural Guided Sample Consensus methods (Brachmann Rother 2019). define second function C' which takes input point-paii along with associated weights according to a distance metric, and outputs the likelihood that each point-pair belongs to an inlier set of matches. Formally, computes the probability that point-pair belongs to an inlier set. We note that C' is only used at training time to choose an optimal set of consistent inliers from possible corresponding point pairs and to encourage the gradient flow through consistent point-pairs.
define C {Ps p d(f,, ) R - N mapping which
An overview of method is presented in Figure I an encoder-decoder style network. The encoder consists 4f VGG-style blocks stacked to reduce the resolution of the image keypoint location and descriptors. In this low resolution embedding space, each pixel corresponds an cell in the original image. The decoder consists of 3 separate heads for the keypoints, descriptors and scores respectively. Thus for an image of input size H x W, the total number of keypoints regressed is (H x W)/64 /64, each with a corresponding score and descriptor. For every convolutional layer except the final one, batch normalization is applied with SeakyReLU activation. detailed description of our network architecture can be seen in Figure 2 The IO-Net is 1D CNN parametrized by 910, for which follow closely the structure from Brachmann & Rother (2019) with 4 default setting residual blocks and the original activation function for final layer is removed. A more detailed description of these networks can be found in the Appendix (Tables 6 and
We define the model K parametrized by 0x as
W to He Wc H/8 x W/8. This allows an efficient prediction for
3

---

Published as a conference paper at ICLR 2020
Excode
Scew Head
Deurrper
Figure 2: The proposed KeyPointNel architecture leverages shared-encoder backbone with three output heads for the regression of keypoint locations (p), scores (s) and descriptions (f). To further improve keypoint description performance, KeyPointNet produces higher-resolution feature-maps for the keypoint descriptors via efficient sub-pixel convolutions at the keypoint descriptor head.
3.1 KRYPOINTNET: NEURAL KEYPOINT DETECTOR AND DESCRIPTOR LEARNING
Detector Learning. Following Christiansen et al. (2019). the keypoint head outputs a location relative the grid in which operates for each pixel in the encoder embedding: [u vf]- The corresponding imput image resolution coordinates [u,, vi] are computed taking into account the grid's position in the encoder embedding. We compute the corresponding keypoint location [u,, o²"] in the target frame after warping via the known homography H. For each warped keypoint, the closest corresponding keypoint in the target frame is associated based on Euclidean distance, We discard keypoint pairs for which the distance is larger than threshold Eve. The associated keypoints in the target frame are denoted by ps = {[u,, 8.].. We optimize keypoint locations using the following self- supervidee loss formulation, which enforces keypoint location consistency across different views of the same scene:
Lioc Σ llpi - -p.l22
As described earlier, the method of Christiansen et al. (2019) does not allow the predicted keypoint locations for each cell cross cell-boundaries. Instead, propose novel formulation which allows us to effectively aggregate keypoints across cell boundaries. Specifically, we map the relative cell coordinates [u/, v'll to input image coordinates via the following function:
(02 -
[u u,] = [row col. conentrr] [0%, u4]
2
S/ u (-1,1)
with 02 = 8, i.e. the cell size, and is ratio relative to the cell size. rowpenter colpenter are the center coordinates of each cell. By setting 0₁ larger than 1, we allow the network to predict keypoint locations across cell borders. Our formulation predicts keypoint locations with respect to the cell center, and allows the predicted keypoints to drift across cell boundaries. We illustrate this in Figure 3] where allow the network to predict keypoints outside the cell-boundary, thereby allowing the keypoints especially at the aoll-bonnaries to be matched effectively. In the ablation study (Section4433 we quantify the effect of this contribution and show that significantly improves the performance of our keypoint detector.
Descriptor Learning. As recently shown by Pillai al. (2019) and Guizilini al. (2019). subpixel convolutions via pixel-shuffle operations (Shi et al.) (0166) can greatly improve the quality of dense predictions, especially in the self-supervised regime. In this work, include fast upsampling step

---

Published as a conference paper at ICLR 2020
(a)
(b)
Figure 3: Cross-border detection illustration. We illustrate how warped point (in red) can be associated to multiple predicted points (in blue) based on a distance threshold (dashed circle). (a) The keypoint location head in Chhistiaasen et al. (2019) forces keypoint predictions in the same cell, causing convergence issues since these points can only be pulled up to the cell-boundary. (b) Instead, in our method, we design the network to predict the localization from the cell-center and allow keypoints to be outside the border for better matching and aggregation.
before regressing the descriptor, which promotes the capture of finer details higher resolution grid. The architectural diagram of the descriptor head is show in Figure 2 In the ablative analysis (Section 4.3) we show that the addition of this step greatly improves the quality of our descriptors.
We employ metric learning for training the descriptors. While the contrastive loss (Hadsell et al.) this task, we propose to use per-pixel triplet
2006) commonly used in the literature loss (Schroff et al., 2015) with nested hardest sample mining to train the descriptor. Recall that each keypoint p; E Pa in the source image has associated de- scriptor fi. an anchor descriptor, which obtain by sampling the appropriate location in the dense descriptor map f, as described De'Toee et al. (2018)). The associated descriptor fi++ in the target frame, positive descriptor, is obtained by sampling the appropriate location in the target descriptor map f. based on the warped keypoint position p, The nested triplet loss is therefore defined as:
described in (Tang et al. (2018)
Ldesc Σ ma [If,, 12 - [|f,, fr, -l, m))
which minimizes the distance between the anchor and positive descriptors, and maximizes the dis- tance between the anchor and negative fü, sample. We pick the negative sample which is the closest the descriptor space that is not a positive sample. Any sample other than the true match can be used as the negative pair for the anchor, but the hardest negative sample will contribute the most to the loss function, and thereby accelerating the metric learning. Here m denotes the distance margin enforcing how far dissimilar descriptors should be pushed away in the descriptor space.
Score Learning. The third head of the decoder responsible for outputting the score associated with each descriptor. At test time, this value will indicate the most reliable keypoints from which subset will be selected. Thus objective of Lecore is two-fold: (i) we want to ensure that feature- pairs have consistent scores, and (ii) the network should learn that good keypoints the ones with low feature point distance. Following Christiansen et al.] (2019) we achieve this objective by mini- mizing the squared distance hetween scores for each keypoint-pair, and minimizing maximizing the average score of keypoint-pair the distance between the paired keypoints is greater or less than the average distance respectively:
Σ (s, +()
(d(p:, - d) (s: 5;)3]
L Lecore =
(4)
2
Here, 8; and are the scores of the source and target frames respectively, and d is the average reprojection eиrоr of associated points in the current frame. d = Σt d(pr))) with d being the feature distance in 2D Euclidean space and I. being the total number of feature pairs.
5

---

Published as a conference paper at ICLR 2020
3.2 IO-NET: NEURAL OUTLIER REJECTION AS AN AUXILIARY TASK
Keypoint and descriptor learning is a task which is tightly coupled with outlier rejection. In this work, we propose
the latter as a proxy task supervise the former. Specifically, we associate keypoints from the source and target images based on descriptor distance: (Ps P (fg. ()) In addition, only keypoints with the lowest K predicted scores are used for training. Similar to the hardest sample mining, this approach accelerates the converging rate and encourages the generation of richer supervisory signal from the outlier rejection loss. To disambiguate the earlier association of keypoint pairs based on reprojected distance defined in Section [.1 we denote the distance metric by x and specify that we refer to Euclidean distance descriptor space. The resulting keypoint pairs along with the computed distance are passed through our proposed IO-Net which outputs the probability that each pair an inlier or outlier. Formally, we define the loss at this step as:
LIo Σ} sign(lpp Poll2 - Eun))
(5)
where is the output of the IO-Net, while cuv is the same Euclidean distance threshold used in Section 3 Different from a normal classifier, we also back propagate the gradients back to the input optimize both the location and descriptor for
sample, Lee, {Ps, , ) thus allowing these associated point-pairs in an end-to-end differentiable manner.
The outlier rejection task is related the neural network based RANSAC (Brachmann & Rother 2019) in terms of the final goal. In our case, since the ground truth homography transform H is known, the random sampling and consensus steps are not required. Intuitively, this can be seen special case where only one hypothesis is sampled. i.c. the ground truth. Therefore, the task is sim- plified to directly classifying the outliers from the input point-pairs. Moreover, a second difference with respect to existing neural RANSAC methods arises from the way the outlier network is used. Specifically, we use the outlier network to explicitly generate an additional proxy supervisory signal for the input point-pairs, as opposed to rejecting outliers.
The final training objective we optimize is defined as:
C = aboo + BLdtse + Miscoree + Lio
(6)
where {a, 3,A] are weights balancing different losses.
4 EXPERIMENTAL RESULTS
4.1 DATASETS
We train our method using the COCO dataset (Lin et al. 2014), specifically the 2017 version which contains 118k training images. Note that we solely use the images, without any training labels, as our method is completely self-supervised. Training on COCO allows us to compare against SuperPoint (DeTone et al., 20188)) and UnsuperPoint (Christiansen et al. 2019), which use the same data for training. We evaluate our method on image sequences from the HPatches dataset (Baintas) al. (017). which contains 57 illumination and 59 viewpoint sequences. Each sequence consists of a reference image and 5 target images with varying photometric and geometric changes for a total of 580 image pairs. In Table and Table we report results averaged over the whole dataset. And for fair comparison, we evaluate results generated without applying Non-Maxima Suppression (NMS).
To evaluate our method and compare with the state-of-the-arr, we follow the same procedure as described in (DeTone et al.) 2018b; Christiansen et al. 2019) and report the following metrics: Repeatability, Localization Error, Matching Score (M.Ccore) and Homography Accuracy. For the Homography accuracy we use thresholds of 1, 3 and 5 pixels respectively (denoted as 1 Cor-3 and Cor-5 in Table 3) The details of the definition of these metrics can be found in the appendix.
4.2 IMPLEMENTATION DETAILS
We implement networks in PyTorch (Pazke et al. 2017) and the ADAM (Kingma & Ba 2014) optimizer. We set the learning rate to 10 and train for 50 epochs with batch size
of 8, halving the learning rate once after 40 epochs of training. The weights of both networks

---

Published as a conference paper at ICLR 2020
Method
Repeat.
Loc.
Cor-I
Cor-3
Cor-5 f
M.Score t
VO- Baseline
0.633
1.044
0.503
0.796
0,868
0.491
V1- Cross
0.689
0.935
0.491
0.805
0,874
0.537
V2 CrossUpsampling
0.686
0.918
0,579
0.866
0,916
0.544
V3 IO-Net
0.685
0.885
0.602
0.836
0.886
0.520
V4 Proposed
0.686
0.890
0.591
0.867
0.912
0.544
Table I: Ablative comparison for 5 different configurations where VO: Baseline, VI: VO + Cross- border detection, V2: VI + Descriptor up-sampling, V3: V2 + Lio, and finally the proposed method V4: V3 + se In general, the results indicate that for most metrics, the proposed method is within reasonable margin of the pert-performing model variant, while achieving strong generalization per- formance across all performance metries including repeatability, localization error, homography accuracy and matching score.
randomly initialized. We set the weights for the total training loss as defined Equation 6 = 2, and À = 1. These weights are selected to balance the scales of different terms. Wet in order avoid border effects while maintaining distributed keypoints over image, described in Section 3. The triplet loss margin m is set to 0.2. The relaxation criteria e for negative sample
1,
set 01 =
mining is set to 8. When training the outlier rejection network described in Section 3.2 K = 300, i.c. we choose the lowest 300 scoring pairs to train on.
we set
We perform the same types of homography adaptation operations as Tone et al. (2018b): crop, translation, scale, rotation, and symmetric perspective transform. After cropping the image with 0.7 (relative to the original image resolution), the amplitudes for other transforms are sampled uni- formly from pre-defined range: scale [0.8, 1.2]. rotation [0, 7. and perspective [0, 0.2). Follow- ing Christiansen et al. (2019), we then apply non-spatial augmentation separately on the source and target frames to allow the network to learn illumination invariance. We add random per-pixel Gaus- sian noise with magnitude 0.02 (for image intensity normalized to [0, 1)) and Gaussian blur with kernel sizes [1, 3, 5] together with color augmentation in brightness J0.5, 1.5), contrast [0.5, 1.5], satu- ration [0.8, 1.2] and hue [-0.2, 0.2). In addition, we randomly shuffle the color channels and convert color image to gray with probability 0.5.
4.3 ABLATIVE STUDY
In this section, evaluate five different variants of our method. All experiments described in this section are performed on images of resolution 240x320. We first define V0-V2 as (i) VO: baseline version with cross border detection and descriptor upsampling disabled; (ii) VI: VO with cross border detection enabled; (iii) V2: V1 with descriptor upsampling enabled. These three variants are trained without neural outlier rejection, while the other two variants (iv) V3: V2 with descriptor trained using only Lio and without Laesee and finally (v) V4 - proposed: V3 together with Lieese loss. The evaluation of these methods is shown in Table ! We notice that by avoiding the border effect described in Section 3.1] VI achieves an obvious improvement in Repeatablity as well as the Matching Score. Adding the descriptor upsampling step improves the matching performance greatly without degrading the Repeatability, as can be seen by the numbers reported under V2. Importantly, even though V/ is trained without the descriptor loss Laese defined in Section [.1] we note further improvements in matching performance. This validates our hypothesis that the proxy task of inlier-outlier prediction can generate supervision for the original task of keypoint and descriptor learning. Finally, by adding the triplet loss, model reported under V4 Proposed achieves good performance which is within error- mrroimmaginne of the berf-performing model variant, while achieving strong generalization performance across all performance metrics including repeatability, localization error, homography accuracy and matching score.
To quantify our runtime performance, we evaluated our model on desktop with Nvidia Titan Xp GPU on images of 240x320 resolution. We recorded 174.5 FPS and 194.9 FPS when running our model with and without the descriptor upsampling step.
4.4 PERFORMANCE EVALUATION
In this section, we compare the performance of our method with the of well with traditional methods on images resolutions 240 x 320 and 480 x 640 respectively. For the results obtained using traditional features well as for L.F-Net (Ono et al. 2018) and SuperPoint (DeToone)
7

---

Published as a conference paper at ICLR 2020
Repeat.
1
Loc. Error
Method
240x320
480x640
240x320
480x640
ORB
0.532
0.525
1.429
1.430
SURF
0,491
0.468
1.150
1.244
BRISK
0.566
0.505
1.077
1.207
SIFT
0.451
0.421
0.855
1.011
[F-Net(indoor) (Ono et al.
2018
0.486
0.467
1,341
1,385
L.- IF-Net(outdoor) (Ono a
2018
0.538
0.523
1.084
1.183
SuperPoint Toee et al. UnsuperPoint (Christtansen el al.
2018b
0.631
0.593
1.109
1.212
2019)
0.645
0.612
0.832
0.991
Proposed
0.686
0.684
0.890
0.970
Table 2: Our proposed method outperforms all the listed traditional and learned feature detectors in repeatability (higher better). For localization error (lower better), UnsuperPoint performs better in lower resolution images while our method performs better for higher resolutions.
240x320,
300 points Cor-5
480 x 640,
1000 points Cor-5
Method
Cor-1
Cor-3
M.Score
Cor-1
Cor-3
M.Score
ORB SURF
0.131
0.422
0.540
0.218
0.286
0.607
0.71
0.204
0.397
0.702
0.762
0.255
0.421
0.745
0.812
0,230
BRISK
0.414
0.767
0.826
0.258
0.300
0.653
0.746
0.211
SIFT
0.622
0.845
0.878
0.304
0.602
0.833
0.876
0.265
LF--Ne((idooor)
0.183
0.628
0.779
0.326
0.231
0.679
0.803
0.287
LF-Net(outdoor)
0.347
0.728
0.831
0.296
0.400
0.745
0.834
0.241
SuperPoint
0.491
0.833
0.893
0.318
0.509
0.834
0,900
0,281
UnsuperPoint 0.579
0.855
0.903
0.424
0.493
0.843
0.905
0.383
Proposed
0.591
0.867
0.912
0,544
0,564
0.851
0.907
0,510
Table 3: Homography estimation accuracy with 3 different pixel distance thresholds (i.e. Cor-1,3 and 5) and matching performance comparison. As shown, our proposed method outperforms all the listed traditional and learning based methods except the one case where SIFT performs the best.
et al. 2018b) report the same numbers computed by (Christiansen al. 2019). During testing, keypoints are extracted in each view keeping the top 300 points for the lower resolution and 1000 points for the higher resolution from the score map. The evaluation of keypoint detection is shown in Table For Repeatibility, our method notably outperforms other methods and is not significantly affected when evaluated with different image resolutions. For the Localization Error, UnsuperPoint performs better in lower resolution image while our method performs better for higher resolution.
The homography estimation and matching performance results are shown in Table 3 self-supervised learning methods provide keypoints with higher matching score and better homog- raphy estimation for the Cor-3 and Cor-5 metrics, as compared traditional handcrafted features (e.g. SIFT). For the more stringent threshold Cor- SIFT performs the best, however, our method outperforms all other learning based methods. As shown in Table our best performing model for this metric is trained using only supervision from the outlier rejection network, without the triplet loss. This indicates that, even though fully self-supervised, this auxiliary task can generate high quality supervisory signals for descriptor training. We show additional qualitative and qualitative results of our method in the appendix.
In general,
5 CONCLUSION
In this paper, we proposed a new learning scheme for training a keypoint detector and associated descriptor in self-supervised fashion. Different with existing methods, we used proxy network to generate an extra supervisory signal from task tightly connected to keypoint extraction: out- lier rejection. We show that even without explicit keypoint descriptor loss in the IO-Net, the supervisory signal from the auxiliary task can be effectively propagated back to the keypoint net- work to generate distinguishable descriptors. Using the combination of the proposed method as well as the improved network structure, we achieve competitive results in the homography estimation benchmark.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle adjustment in the large. In European conference on computer vision, pp. 29 42. Springer, 2010.
Vassilcios Baintas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A bench- mark and evaluation of handcrafted and learned local descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5173-5182, 2017.
Eric Brachmann and Carsten Rother. Neural-guided ransac: Learning where sample model hy- potheses, arXiv preprint wwXivv:905.041.. 2019.
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, lan Reid, and John Leonard. Past, present, and future of simultaneous localization and mapping: Toward the
age. IEEE Transactions on robotics, 32(6): (16):1309-1333) 2016.
Peter Hviid Christiansen, Mikkel Fly Kragh, Yury Brodskiy, and Henrik Karstoft. Unsuperpoint: End-to-end unsupervised interest point detector and descriptor. arXiv preprint arXiv: 0000 2019.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Self-improving visual odometry daniel. arXiv preprint arXiv:1812.03225. 2018a.
Daniel DeTooee, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest. point detection and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 224-236, 2018b.
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, and Adrien Gaidon. Packnet-sfm: 3d packing for self-supervised monocular depth estimation. arXiv preprint
2019.
Raia Hadsell, Sumit Chopra, and LeCun Yann. Dimensionality reduction by learning an invariant mapping. In Proc. CVPR, June 2006.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,6980 2014.
Alexander Kirillov, Ross Girshick. Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6399-6408, 2019.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. arXiv preprint arXiv:1901.09005. 2019.
Alex Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Osear Beijbom. Pointpil- lars: Fast encoders for object detection from point clouds. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12697-12705, 2019.
Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, and Adrien Gaidon. Learning to fuse things and stuff. arXiv preprint 1 192, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
David G. Lowe. Distinctive image features from scale-invariant keypoints. Int. Comput. Vision, 6012):91-110, November 2004. ISSN 0920-5691. doi: 110.///BB.LL..0000964.00.00 URL
www.S/0...0002226.0000 0111//66.9991..1. 99615.99.
David G Lowe et al. Object recognition from local scale-invariant features. In iccv, volume 99, pp- 1150-1157, 1999.
Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. LF-Net: Learning local features from images. In NIPS, 2018.

---

Published as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-PS-WR W, 2017.
Sudeep Pillai, Rareş Ambruş, and Adrien Gaidon. Superdepth: Self-supervised, super-resolved monocular depth estimation. In 2019 International Conference on Robotics and Automation (ICRA). pp. 9250-9256. IEEE, 2019.
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R Bradski. Orb: An efficient alternative to sift or surf. In ICCV, volume 11, pp. 2. Citeseer, 2011.
Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From coarse fine: Robust hierarchical localization at large scale. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Nikolay Savinov, Akihito Seki, Lubor Ladicky, Torsten Sattler, and Mare Pollefeys. Quad-networks: Unsupervised learning to rank for interest point detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proc. CVPR, June 2015.
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1874-1883, 2016.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. arXiv preprint arXiv: 090.0092222 2019.
Jiexiong Tang. John Folkesson, and Patric Jensfelt. Geometric correspondence network for camera motion estimation. IEEE Robotics and Automation Letters, ((2):0010-1017 2018.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object
detection. arXiv preprint arXiv; 190.009904003555 2019.
Yannick Verdie, Kwang Yi, Pascal Fua, and Vincent Lepetit. Tilde: A temporally invariant learned detector. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Track- ing emerges by colorizing videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 391-9994-- 2018.
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2566-2576, 2019.
Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned Invariant Feature Transform, In ECCV, 2016.
10

---

Published as a conference paper at ICLR 2020
A HOMOGRAPHY ESTIMATION EVALUATION METRIC
We evaluated our results using the same metrics as De'Tone et al. (2018b). The Repeatability, Local- ization Error and Matching Score are generated with correctness distance threshold of 3. All the metrics are evaluated from both view points for each image pair.
Repeatability. The repeatability is the ratio of correctly associated points after warping into the target frame. The association is performed by selecting the closest in-view point and comparing the distance with the correctness distance threshold.
Localization Error. The localization error is calculated by averaging the distance between warped points and their associated points.
Matching Score (M.Score). The matching score is the success rate of retrieving correctly associated points through nearest neighbour matching using descriptors.
Homography Accuracy. The homography accuracy is the success rate of correctly estimating the homographies. The mean distance between four corners of the image planes and the warped im- age planes using the estimated and the groundtruth homography matrices are compared with dis- tances 1, 3, 5. To estimate the homography, we perform reciprocal descriptor matching, and we use openCV's confidence threshold 0.9995 and error threshold
method with RANSAC, maximum number of 5000 iterations,
B DETAILED QUALITATIVE AND QUANTITATIVE ANALYSIS ON HPATCHES
To capture the variance induced by the RANSAC component during evaluation we perform addi- tional experiments summarized in Table @hblee/hher where each entry reports the mean and standard deviation across runs with varying RANSAC seeds. We notice better homography performance on the illumination subset than the viewpoint subset. This is be expected as the viewpoint subset contains image pairs with extreme rotation which are problematic for our method which is fully convolutional.
HPatches subset
Repeat,
Loc. 4
Cor-1 f
Cor-3 f
Cor-5 f
M.Score t
All
0.686
0.890
0.595:10.012
0.912-10.055 0.87110.005
0.544
Illumination 0.678
0.826
0.753-10.014
0.972+40.004
0.984410.004
0.614
Viewpoint
0.693
0.953
0.494=0.015
0.801
0.857=0.008
0.479
Table 4: Detailed analysis of the performance of our method: we evaluate on different splits of the HPatches dataset (llummination, Viewpoint and complete dataset) on images of resolution 320x240. To capture the variance in the evaluation of the homography (Cor-1,3 and 5) due to RANSAC, we perform 10 evaluation runs with different random seeds and report the mean and standard deviation.
We also evaluate our method as well as SIFT and ORB on the graffiti, bark and boat sequences of the HPatches dataset and summarize our results in Table 5 again reporting averaged results over 10 runs. We note that our method consistently outperforms ORB. Our method performs worse than SIFT (which is more robust to extreme rotations) on the bark and boat sequences, but we obtain better results the graffiti sequence.
Figure 4 denotes examples successful matching under strong illumination, rotation and perspec- tive transformation. Additionally, we also show matches pairs of images from the challenging graffiti, bark and boat sequences of HPatches in Figures 5 6) and 7 Specifically, the top row each figure shows our results, while the bottom images (1,2) of each sequence, while the right sub-figure shows images (1,6). We note that on im- ages (1,2) our results are comparable to SIFT, while images (1,6) get fewer matches. Despite the extreme perspective change, we report that our method is able to successfully match features on images (1,6) of the boat sequence.
shows SIFT. The left sub-figure
each row shows
11

---

Published as a conference paper at ICLR 2020
HPatches subset
Repeat.
Loc. 4
Cor-1 f
Cor-3 f
Cor-5 1
M.Score
Bark
0.419
1.734
0.200=0.000
0.200=000000
0.200 0.00
0.088
Ours
Boat
0.588
1.357
0.260=0.092
0.380=0.006
0,400
0.223
Graffiti
0.670
1.039
0.340=0.092
0.720=:0.09
0.780=:0.060
0,336
Bark
0.620
1.259
0.000.10.000
0.060-10.092
0.180.10.060
0.110
ORB
Boat
0.812
1.127
0.000=0.000
0.420= 108
0.560::0.120
0,226
Graffiti
0.722
1.164
0.080=0.098
0.340=#0.092
0,400:000000
0.210
Bark
0.470
1,252
0.600-10.000
0.960=10.080
0.960:10.080
0.292
SIFT
Boat
0,527
0.928
0.560=0.080
0.980=10.060
1.000=000000
0.351
Graffiti
0.515
1.277
0.340=0.092
0.560=0..100
0.660::0.092
0.237
Table 5: Detailed analysis of the performance of our method on the Bark, Boal and Graffitti se-
quences of the HPatches dataset. To capture the variance in the evaluation of the homography (Cor-1,3 and 5) due to RANSAC, we perform 10 evaluation runs with different random seeds and report the mean and standard deviation.
(a) Qualitative results of our method for illumination cases.
(b) Qualitative results our method rotation cases.
FOREUA AWAMAAAAA AEE
(c) Qualitative results of method for perspective
Figure 4: Qualitative results of our method on images pairs of the HPatches dataset.
12

---

Published as a conference paper at ICLR 2020
(a) Ours
(b) SIFT
Figure 5: Qualitative results of our method vs SIFT on the "graffiti" subset of images of the HPatches dataset.
(a) Ours
(b) SIFT
Figure 6: Qualitative results of our method vs SIFT on the "bark" subset of images of the HPatches dataset.
(a) Ours
(b) SIFT
Figure 7: Qualitative results of our method vs SIFT on the "boat" subset of images of the HPatches dataset.
13

---

Published as a conference paper at ICLR 2020
C
ARCHITECTURE DIAGRAM
Layer Description
Output Tensor Dim.
Input RGB image
3xHxW
Encoder
#1
Conv2d BatchNorm + LReLU
3
32 W
#2
Conv2d BatchNorm LReLU Dropout
32xHxw
#3
Max. Pooling (x 1/2)
32xH/2x W/2
#4
Conv2d BatchNorm + LReLU
3
64xH/2xW/2
Conv2d BatchNorm + LRcLU Dropout
64xH/2xW W/2
#6
Max. Pooling (x 1/2)
3
64xH/4xW14
#7
Conv2d BatchNorm LRcLU
3
228xHIXWWA4
Conv2d BatchNorm + LReLU Dropout
x
#9
Max. Pooling (x 1/2)
3
#10
Conv2d BatchNorm + LReLU
3
256x W/88
#11
Conv2d BatchNorm + LReLU Dropout
256x
Score Head
#12
Conv2d BatchNorm + Dropout (#11)
256xH/8x W/8
#13
Conv2d Sigmoid
3
1xH/8xW/8
Location Head
#14
Conv2d BatchNorm + Dropout (#11)
3
256xH/8 W/8
#15
Conv2d Tan. Hyperbolic
2xH/8xWI8
Descriptor Head
#16
Conv2d BatchNorm + Dropout (#11)
256x W/8
#17
Conv2d BatchNorm
3
512xH/8 W/8
#18
Pixel Shuffle (x 2)
3
128x W/8
#19
Conv2d BatchNorm (#8 @ #18)
256xH/4 W/4
#20
Conv2d
3 256xH/4x W//4
Table 6: KeyPointNet diagram, composed of an encoder followed by three decoder heads. The network receives as input an RGB image and returns scores, locations and descriptors. Numbers in parenthesis indicate input layers, @ denotes feature concatenation, and we used 0.2 dropout values.
Layer Description
K
Output Tensor Dim.
#0
Input KeyPoints
5xN
Convld ReLU
128
#2
ResidualBlock
128 X
#3
ResidualBlock (#2 @ #1)
128xN
#4
ResidualBlock (#3 @ #2)
128xN
#5
ResidualBlock (#4 @ #3)
128xN
#6
Convld
1xN
ResidualBlock
Convld InstNorm BatchNorm + ReLU Convld 1d + InstNorm + BatchNorm + ReLU
128xN
128 X
residual blocks. The network receives as input a series of 5-dimensional vector consists of keypoint pair and descriptor distance, outputs binary outier-outlier classification. Numbers in parenthesis indicate input layers, and @ denotes feature addition.
Table 1O-Net diagram, composed
14


Reviews:
[{"review_id": "rklyawTEYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is devoted to self-supervised learning of local features (both detectors and descriptors simultaneously). The problem is old yet not fully solved yet, because handcrafted SIFT is still winning the benchmarks. This work mostly follows and improves upon SuperPoint (DeTone et.al 2017) and the follow-up work UnsuperPoint (Christiansen et.al 2019) architecture and training scheme.\n\nThe claimed contributions are following:\n  - use the recently published Neural Guided RANSAC as additional auxilary loss provider\n  - allowing the \"cells\" to predict keypoint location outside the cell while learning\n  - special procedure for improving descriptors interpolation\n\n\nThe experiments are performed on HSequences dataset (wrongly called \"HPatches\", as HPatches dataset is literally image patches, not full images), showing noticable improvement over the state of the art.\n\n\nStrong points:\n\n - Method is sound, paper is mostly well written and results are good (may be too good, see questions).\n \n \n \n Questions:\n \n  1) Regarding descriptor interpolation, which is claimed as contribution. It is not clear for me, how different it is compared to SuperPoint one, which also do descriptor upsampling, so that network output is H x W x [256], i.e. full resolution. Could you please clarify the differences to it? Also, in Figure 2 it is not clear, how one can do \"feature concatenation\" for blocks with different spatial resolution.\n\n  2) Why the IONet is used only for training?  Wouldn`t it better to actually learn everything end-to-end, which is already done in paper and evaluate? \n  \n  3) How is association in training (e.g. on Fig.3) done, if multiple cells in img2 returns keypoint close to the same keypoint in img1? \n\n  4) HSequences consists of two subsets: Illumination and Viewpoint. Could you please report results per subset instead of per whole dataset? Could you also please specifically report results for the following image sequences: graffity, bark, boat, especially for 1-6 pairs and visualize matches (same way as in Figure 4-6)?\n  The reason that I am asking these, is results looks like too well and I suspect overfitting to a points, which are suitable for estimation (small) homography, not general-purpose points.\n  \n  5) Could you please explain in more details, how did you do homography estimation precision benchmark? Specifically, was Lowe`s second nearest neighbor ratio used for filtering out wrong matches? If not, could you please repeat this experiments with it, at least for SIFT matches?\n  \n  \nSmall comments:\n    - list of contributions in abstract is inconsistent with 3rd paragraph in Introduction, which also lists contributions.\n    \n\n***\nOverall I like the work, but there are unclear moments to me. \n\n\n****\nAfter rebuttal comments. While this paper may appear not \"sexy\" I think it is quite valuable for the local features learning community: both for the main contributions, and small details and tricks evaluated inside. \nI am happy to increase my score to strong accept.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJeKkwvk5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The following work proposes several improvements over prior works in unsupervised/self-supervised keypoint-descriptor learning such as Christiansen et al. One improvement is the relaxation of the cell-boundaries for keypoint prediction -- specifically allowing keypoints anchored at the cell's center to be offset into neighboring cells. Another change was the introduction of an inlier-outlier classifier network to be used as a proxy loss for the keypoint position and descriptors. They found the inlier-outlier loss to improve homography accuracy at 1 and 3 pixel thresholds.\n\nStrengths:\n-The ablation study seems complete\n-Clear improvements over state of the art methods\n\nWeaknesses/improvements:\n-The description of the evaluation procedure was a bit vague. Is RANSAC being used to find correspondences? If so, perhaps error bars are necessary to account for variance across multiple runs?\n-Make it more clear in the related works about how the proposed method relates to Unsuperpoint. My understanding is that the proposed work is a somewhat incremental improvement over Unsuperpoint.\n-Section 3.3 (Score learning) was a bit difficult to follow. I find it better to start by stating the high level goal of the loss function before going into the formulation.\n-Captions for Tables 2 and 3 are lacking. At the very least, mention what the numbers being compared are.\n\nOverall, I think the improvements are a bit incremental, but the experiments seem to support the claim that they are beneficial. I had some concerns about the clarity of the paper, and would be willing to raise my rating if addressed.\n\nPost Rebuttal:\nThe authors have adequately addressed my concerns regarding clarity. I have updated my rating to weak accept in agreement with the other reviews.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkluihCpYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# UPDATE following rebuttal\n\nScore increased to 6 due to architecture details in supplemental.\n\n\n# Contributions\n\nThe paper contributes a self-supervised method of jointly learning 2D keypoint locations, descriptors, and scores given an input RGB image. The paper builds on previous work, adding:\n\n* A more expressive keypoint location regression, which allows each 8x8 pixel region to vote for a keypoint location outside its boundary\n\n* An upsampling step, similar to a U-net, to allow descriptors to be regressed with more detailed information\n\n* An additional proxy task for the total loss, based on outlier rejection.\n\nThe authors train on COCO by manually distorting images to generate pairs with known homography, and show competitive results for keypoint detection and homography estimation tasks.\n\nDecision: Weak reject. I would give this a 5 if the website allowed me to. A more detailed explanation of the neural network architecture, along with minor fixes described below, would make me increase my rating.\n\n\nI feel the additions to existing pipelines are well motivated but insufficiently explained. In particular, the explanation of the neural network architecture along with figures 1 and 2 leaves many details unclear to me. Phrases like \"a 1D CNN ... with 4 default setting residual blocks\" is to me insufficient - residual networks have many details such as Resnet V1 or V2 style (ie is there a path right through the network which doesn't hit any activation functions), what kind of normalization is applied, number of channels in each block, how to do skips between different spatial resolutions, etc. The upsampling step for the descriptor head, which is claimed as a novel contribution, is not fully explained - \"fast upsampling\" implies (correctly) there are many variants of upsampling with different tradeoffs, but from the text I am unsure whether this is nearest neighbour upsampling, a ConvTranspose, etc. Similarly, \"VGG style block\" leaves some details unclear - whether the resolution downsampling is with a strided convolution / pooling / etc. Lots of the details are implied to be in other previous papers, but I feel that the paper would be hugely improved by exact architectural details.\n\nThere are various minor notational discrepancies in the paper - for example the outlier rejection is various defined as \"InlierOuterNet (IONet)\" and \"The Inlier-Outlier model \\emph{IO}\", which also seems to be the same as the function $C$ defined a paragraph above. Perhaps it is common in this part of the literature, but to me an encoder decoder network is more likely to either be an autoencoder, or for the decoder to output something in the same modality (eg in machine translation). To say that some VGG blocks are an encoder, and the heads which produce keypoint locations / score / descriptor is a decoder, implies all neural networks could be described as an encoder/decoder.\n\nThe two figures showing the architecture are very different in design, which is not in itself a problem but the relationship between them could be clearer. I feel that the 'matching' box in figure 1 is misleading because it implies that matching only happens for the IONet, but the loss function for location described in Eq 1 also requires matching keypoints between the image pair. I'm also unclear on the division between direct and indirect supervisory signal - all the 4 loss components have a clear purpose, but it's not obvious what this partitioning means. \"Indirect\" only appears in this figure and the caption - perhaps.\n\nThe term \"Anchor\" appears only once with no reference, below equation 3 - I appreciate this is an existing term in this subfield, but given that the start of section 3 goes as far as explicitly defining what it means to produce 2D keypoints for an image, I feel defining this term would make the descriptor loss much clearer. \n\nOne of the main contributions, that of allowing locations to regress outside their 8x8 area, sounds like a good idea but I feel that Figure 3 does not adequately show the benefit. In both a) and b), the blue estimates appear to be roughly as good as each other - clearly from the ablation a large benefit is gained from this innovation but perhaps a better illustrative example could be made here?\n\nOn a more positive note, I feel the components of the loss function are in general very clearly motivated and defined, and the description of training & data augmentation hyperparameters appears complete. If the description of the architecture could be improved that would result in a paper very amenable to reproduction.\n\nThe experiments are well explained, and the ablation of the various proposed  components is good. I feel table 1 would be improved with error bars - given that the bold best score is not exclusively next to V4, but in many cases the difference between V4 and the best is ~1%, error bars from different training runs might make clearer that V4 is overall the best configuration.\n\nIn the conclusion - \"even without an explicit loss\" - what is the difference between the loss functions used in this work, and an explicit loss?\n\n\nMinor corrections:\n\nThe euclidean distance between descriptors is various notated as $d$ (section 3), $x$ (above equation 5) and $d$ again (below equation 5).\n\nTypos: \"normalzation\" -> \"normalization\", \"funcion\" -> \"function\", \"tripled\" -> \"triplet\".\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a solid (if somewhat incremental) improvement on an interesting and well-studied problem. I suggest accepting it.
Decision: Accept (Poster)