Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The following work proposes several improvements over prior works in unsupervised/self-supervised keypoint-descriptor learning such as Christiansen et al. One improvement is the relaxation of the cell-boundaries for keypoint prediction -- specifically allowing keypoints anchored at the cell's center to be offset into neighboring cells. Another change was the introduction of an inlier-outlier classifier network to be used as a proxy loss for the keypoint position and descriptors. They found the inlier-outlier loss to improve homography accuracy at 1 and 3 pixel thresholds.

Strengths:
-The ablation study seems complete
-Clear improvements over state of the art methods

Weaknesses/improvements:
-The description of the evaluation procedure was a bit vague. Is RANSAC being used to find correspondences? If so, perhaps error bars are necessary to account for variance across multiple runs?
-Make it more clear in the related works about how the proposed method relates to Unsuperpoint. My understanding is that the proposed work is a somewhat incremental improvement over Unsuperpoint.
-Section 3.3 (Score learning) was a bit difficult to follow. I find it better to start by stating the high level goal of the loss function before going into the formulation.
-Captions for Tables 2 and 3 are lacking. At the very least, mention what the numbers being compared are.

Overall, I think the improvements are a bit incremental, but the experiments seem to support the claim that they are beneficial. I had some concerns about the clarity of the paper, and would be willing to raise my rating if addressed.

Post Rebuttal:
The authors have adequately addressed my concerns regarding clarity. I have updated my rating to weak accept in agreement with the other reviews.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: # UPDATE following rebuttal

Score increased to 6 due to architecture details in supplemental.


# Contributions

The paper contributes a self-supervised method of jointly learning 2D keypoint locations, descriptors, and scores given an input RGB image. The paper builds on previous work, adding:

* A more expressive keypoint location regression, which allows each 8x8 pixel region to vote for a keypoint location outside its boundary

* An upsampling step, similar to a U-net, to allow descriptors to be regressed with more detailed information

* An additional proxy task for the total loss, based on outlier rejection.

The authors train on COCO by manually distorting images to generate pairs with known homography, and show competitive results for keypoint detection and homography estimation tasks.

Decision: Weak reject. I would give this a 5 if the website allowed me to. A more detailed explanation of the neural network architecture, along with minor fixes described below, would make me increase my rating.


I feel the additions to existing pipelines are well motivated but insufficiently explained. In particular, the explanation of the neural network architecture along with figures 1 and 2 leaves many details unclear to me. Phrases like "a 1D CNN ... with 4 default setting residual blocks" is to me insufficient - residual networks have many details such as Resnet V1 or V2 style (ie is there a path right through the network which doesn't hit any activation functions), what kind of normalization is applied, number of channels in each block, how to do skips between different spatial resolutions, etc. The upsampling step for the descriptor head, which is claimed as a novel contribution, is not fully explained - "fast upsampling" implies (correctly) there are many variants of upsampling with different tradeoffs, but from the text I am unsure whether this is nearest neighbour upsampling, a ConvTranspose, etc. Similarly, "VGG style block" leaves some details unclear - whether the resolution downsampling is with a strided convolution / pooling / etc. Lots of the details are implied to be in other previous papers, but I feel that the paper would be hugely improved by exact architectural details.

There are various minor notational discrepancies in the paper - for example the outlier rejection is various defined as "InlierOuterNet (IONet)" and "The Inlier-Outlier model \emph{IO}", which also seems to be the same as the function $C$ defined a paragraph above. Perhaps it is common in this part of the literature, but to me an encoder decoder network is more likely to either be an autoencoder, or for the decoder to output something in the same modality (eg in machine translation). To say that some VGG blocks are an encoder, and the heads which produce keypoint locations / score / descriptor is a decoder, implies all neural networks could be described as an encoder/decoder.

The two figures showing the architecture are very different in design, which is not in itself a problem but the relationship between them could be clearer. I feel that the 'matching' box in figure 1 is misleading because it implies that matching only happens for the IONet, but the loss function for location described in Eq 1 also requires matching keypoints between the image pair. I'm also unclear on the division between direct and indirect supervisory signal - all the 4 loss components have a clear purpose, but it's not obvious what this partitioning means. "Indirect" only appears in this figure and the caption - perhaps.

The term "Anchor" appears only once with no reference, below equation 3 - I appreciate this is an existing term in this subfield, but given that the start of section 3 goes as far as explicitly defining what it means to produce 2D keypoints for an image, I feel defining this term would make the descriptor loss much clearer. 

One of the main contributions, that of allowing locations to regress outside their 8x8 area, sounds like a good idea but I feel that Figure 3 does not adequately show the benefit. In both a) and b), the blue estimates appear to be roughly as good as each other - clearly from the ablation a large benefit is gained from this innovation but perhaps a better illustrative example could be made here?

On a more positive note, I feel the components of the loss function are in general very clearly motivated and defined, and the description of training & data augmentation hyperparameters appears complete. If the description of the architecture could be improved that would result in a paper very amenable to reproduction.

The experiments are well explained, and the ablation of the various proposed  components is good. I feel table 1 would be improved with error bars - given that the bold best score is not exclusively next to V4, but in many cases the difference between V4 and the best is ~1%, error bars from different training runs might make clearer that V4 is overall the best configuration.

In the conclusion - "even without an explicit loss" - what is the difference between the loss functions used in this work, and an explicit loss?


Minor corrections:

The euclidean distance between descriptors is various notated as $d$ (section 3), $x$ (above equation 5) and $d$ again (below equation 5).

Typos: "normalzation" -> "normalization", "funcion" -> "function", "tripled" -> "triplet".


review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper is devoted to self-supervised learning of local features (both detectors and descriptors simultaneously). The problem is old yet not fully solved yet, because handcrafted SIFT is still winning the benchmarks. This work mostly follows and improves upon SuperPoint (DeTone et.al 2017) and the follow-up work UnsuperPoint (Christiansen et.al 2019) architecture and training scheme.

The claimed contributions are following:
  - use the recently published Neural Guided RANSAC as additional auxilary loss provider
  - allowing the "cells" to predict keypoint location outside the cell while learning
  - special procedure for improving descriptors interpolation


The experiments are performed on HSequences dataset (wrongly called "HPatches", as HPatches dataset is literally image patches, not full images), showing noticable improvement over the state of the art.


Strong points:

 - Method is sound, paper is mostly well written and results are good (may be too good, see questions).
 
 
 
 Questions:
 
  1) Regarding descriptor interpolation, which is claimed as contribution. It is not clear for me, how different it is compared to SuperPoint one, which also do descriptor upsampling, so that network output is H x W x [256], i.e. full resolution. Could you please clarify the differences to it? Also, in Figure 2 it is not clear, how one can do "feature concatenation" for blocks with different spatial resolution.

  2) Why the IONet is used only for training?  Wouldn`t it better to actually learn everything end-to-end, which is already done in paper and evaluate? 
  
  3) How is association in training (e.g. on Fig.3) done, if multiple cells in img2 returns keypoint close to the same keypoint in img1? 

  4) HSequences consists of two subsets: Illumination and Viewpoint. Could you please report results per subset instead of per whole dataset? Could you also please specifically report results for the following image sequences: graffity, bark, boat, especially for 1-6 pairs and visualize matches (same way as in Figure 4-6)?
  The reason that I am asking these, is results looks like too well and I suspect overfitting to a points, which are suitable for estimation (small) homography, not general-purpose points.
  
  5) Could you please explain in more details, how did you do homography estimation precision benchmark? Specifically, was Lowe`s second nearest neighbor ratio used for filtering out wrong matches? If not, could you please repeat this experiments with it, at least for SIFT matches?
  
  
Small comments:
    - list of contributions in abstract is inconsistent with 3rd paragraph in Introduction, which also lists contributions.
    

***
Overall I like the work, but there are unclear moments to me. 


****
After rebuttal comments. While this paper may appear not "sexy" I think it is quite valuable for the local features learning community: both for the main contributions, and small details and tricks evaluated inside. 
I am happy to increase my score to strong accept.
review_title: Official Blind Review #3

Meta-review: This paper proposes a solid (if somewhat incremental) improvement on an interesting and well-studied problem. I suggest accepting it.
Decision: Accept (Poster)