Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a learnt local keypoint descriptor for deformable objects. It is trained in a supervised fashion using the HardNet network and loss on the SIFT keypoints of a synthetic dataset (succinctly presented in the paper, based on existing SfM datasets). The novelty is the addition of a spatial transformer network (STN) to normalize the patches, inline with recent local descriptor learning approaches. As far as I understand the main conceptual difference with existing works is that the (STN) is not only affine but uses a TPS on a log-polar grid. My understanding of the paper is that it argues this part is crucial for matching deformable objects.
main_review: Novelty/contributions/ablation:
While the proposed approach makes sense, I do not really see any novelty/important difference compared to standard learned keypoint descriptors except (i) that the deformation learned by the transformer network includes a TPS deformation on a log-polar grid (and not only an affine one) (ii) the learning dataset seem to be an "in house" dataset (on which I would appreciate much more details and which I would like the authors to confirm they will release with their code) with a HardNet approach. It is unclear to me what is the influence of each on the results. There is only an embryo of ablation study (Table 3, two numbers on an unspecified handpicked small subset of the data) where the full transformer network is ablated. I would like to also see **an ablation of only the TPS part, and a TPS on a standard regular grid**, to also have access to the corresponding results on the full dataset, and to have some analysis of the effect of the training dataset (e.g. same data generation process but without the non-rigid deformations, size). More generally, I would also like to see ablation that replace different parts of the proposed pipeline (loss, network archi, training data) with those of a recent learned feature descriptor.

Comparisons to other approaches:
My main concern is that the results are reported on two recent (2019) datasets that have been used by very few other works (5 and 2 citations), one thus has to rely on the paper itself to run relevant baselines. While there is a clear effort in this direction in Table 2 and 4, there are some baselines that seem important to me and would be worth adding a few: SIFT, LIFT, superpoint, D2-Net, R2D2 and maybe other recent learnt descriptors. Note that a single number on R2D2 is reported in 4.2, but I think this is not really sufficient and it should be compared with other methods (even if it makes the table more complicated because they can be used either with SIFT keypoints or with their own keypoints) Conversely, I would like to see the performance of the proposed descriptor on standard keypoint matching benchmarks (potentially for downstream tasks such as calibration). While I understand the advantage of the proposed keypoints is to handle deformable objects, they should also be able to handle rigid object, and showing they give competitive results on standard benchmark would reassure me that the results reported in the paper are indeed impressive and not only due to comparison bias.

In conclusion, there are too many things in this paper I am not confident about and that I would like to see added to recommend acceptance. I see little demonstrated important technical and conceptual novelty and I don't think improving SoA on seldom used datasets is itself sufficient for a publication.

Minor: some writing issue, e.g. last sentence of the first paragraph "After all, we live in a non-rigid world, where beyond rigid surfaces, it is also populated ...'

limitations_and_societal_impact: .discussion of limitations and failure cases missing
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to learn deformation-aware local feature descriptors by rectifying the local region with carefully designed spatial transformer networks (STNs). Instead of using 3D training data, the paper proposes to learn from deformed 2D samples. The feature extraction network utilizes two STNs to rectify the local region and HardNet to extract the final feature. Among the two STNs, the second STN is specially designed with the thin-plate-spline transform and a polar grid so that the resulting region flexibly deform from a circular shape. The experimental results show that better or competitive results with much less processing time.
main_review: +- Not highly novel, but practically beneficial: Using spatial transformers and deep learning to learn robust features is not exactly new. I think that the main contributions are in designing it for general deformation, and also in the detailed design (a simple but specially designed deformation module with two STNs) of the proposed method. On the other hand, the method has many practical benefits: no requirement for 3D training data, fast processing time, and good performance.

-- Missing reference: There is a highly related work that uses STN for learning local features [Ono et al., NeurIPS 2018], which is missing in the current paper. This has to be mentioned and compared with the proposed method.

-- Why do we need two STNs?: I said that one of the main contributions is the module with two STNs. But why do we need this? Can't we just use the second STN only? What is the exact role of the first (plain) STN? The reason for such a design is not well explained in the paper. I strongly suggest providing (or analyzing) this reason, which can be crucial in lifting the value of the paper.

+- Using only 2D data: Although I mentioned that requiring only 2D data is a benefit, there is an issue to discuss here. I see that the proposed network can also be trained with RGB-D data with no problem. (In this case, moreover, we might also add some more supervisions, say, something like deformation supervision.) What happens if we do that? For example, the proposed method and GeoBit are competitive in performance in the experiments. Do things get any different with RGB-D data?

[After rebuttal]
Most of my concern has been addressed in the authors' response. I believe that this is a sound contribution in the field. I raise my score to 7.
limitations_and_societal_impact: Limitations are described in the final section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a new deep learning pipeline for the computation of local image features that are robust to non-rigid deformations. The proposed method is end-to-end, and works by simulating non-rigid transformations on a synthetic dataset to jointly learn how to rectify the deformed patches, while extracting discriminative and invariant features at the same time. Comparisons with several state of the art pipelines on multiple datasets confirm the effectiveness of the proposed approach.
main_review: I enjoyed reading the paper, which is well written, fluent, and easily accessible for someone not fully familiar with this family of methods (such as this reviewer). The method is described well and the computational modeling is sound. The underlying idea of explicitly parametrizing deformations via a learnable TPS model is straightforward and well executed, and I believe there is value in this simplicity, given the good results it enables in several practical cases.

Worth mentioning is the extensive experimental evaluation, which includes several recent datasets comprised of real-world deformable objects (despite the training being based on synthetic examples) undergoing diverse picture-space transformations (e.g. illumination, viewpoint, object deformation), as well as comparisons with several baselines and recent learning-based methods. Hyperparameter sensitivity, ablation, and runtime studies are also included. Two real-world applications are analyzed, demonstrating competitive results if compared to other recent methods.

As a minor comment, I found the results reported in Table 3 (Ablation) a bit difficult to interpret: if the NRW component is key to learning the unwrap, why is the improvement so mild over vanilla PTN (+0.04 mean matching accuracy)? The paper refers to this as significant improvement, but more comments on this being a true gain would be useful.

Overall, I believe this is a solid paper that deserves to be communicated. At the same time, I have some reservations on this being a computer vision paper first and foremost, with machine learning just being a convenient ingredient to compose the full pipeline. In other words, I think the key contribution is in the vision area, while it might be not very significant for ML. Therefore I am not 100% confident NeurIPS is the right venue for it, but I won't oppose acceptance based solely on this doubt.
limitations_and_societal_impact: The limitations are briefly discussed in the conclusions, and failure cases are included in the supplementary material.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes DEAL, an RGB patch descriptor that has a learned robustness against non-rigid deformations. Given a SIFT keypoint, surrounding ResNet features are used to estimate a non-rigid thin-plate spline deformation of the patch. The RGB patch, after being rectified by the thin-plate spline, is passed to a CNN that regresses a 128-dim descriptor. The network is trained via a hard triplet loss. The training data is simulated and uses groundtruth correspondences/matches, but results are shown on both real and synthetic data. Experiments compare to a number of classic and learned baselines. DEAL outperforms other methods in the main experiment and shows best or second-best performance on a couple of applications.
main_review: The proposed method is a novel descriptor that tackles non-rigid deformations, which is a very desirable goal. The experiments on real data show promising results, especially considering that the descriptor was trained on only synthetic data. My main concern is a better attribution of where the improvement comes from: data or architecture (thin plate splines)? I find the current experiments unsatisfying in this regard, the next paragraph goes into details.

The "Contribution of the TPS warper" ablation study is interesting. My understanding is that "(i)"/PTN is a pipeline that is the same (up to maybe number of conv layers etc.) as Fig. 3 in Ebel et al.'s Log-Polar descriptor [12]: sample the original image with a circular polar grid around the keypoint and then feed that patch through a CNN to get the descriptor. Table 2 shows an MMA of 0.49 for Log-Polar on the Simulation dataset, while DEAL has 0.80. The ablation study shows 0.79 for DEAL (i.e. with NRW) and 0.75 for the PTN ablation. The ablation uses a specific scene from the Simulation dataset, which explains the differing performance of DEAL. I interpret the discrepancy between PTN ablation (0.75) and Log-Polar (0.49) to mean that most of the improvement that DEAL shows over Log-Polar is due to training data (from 0.49 to 0.75) rather than architecture (from 0.75 to 0.79). Is that correct? If so, I believe that further baseline experiments for Table 2 would be necessary that train the two learning baselines (TFeat and Log-Polar, lines 218-219) with the proposed *non-rigid* simulated dataset (lines 229-239) instead of their original datasets, which seem to be *rigid*.

I have a few bigger questions about the paper, but which I do not expect to sway the acceptance decision:

- Limitations should be discussed more, including qualitative examples.
- What is the intuition behind using the first spatial transformer? What would happen if it isn't used and Y = X? Such an ablation experiment should be added.
- The sensibility analysis in Table 1 shows that increasing the margin to 0.75 and increasing the STN output to 5 x 5 improves over the baseline. Why is only the latter change adapted (lines 266-267)? The margin change gives 50% as much improvement as changing the STN output does after all.
- The STN is meant to normalize out some spatial transforms around the keypoint. But how can the localization network then determine the right (desirable) rectifying transformation for the original patch around the keypoint if the localization network doesn't have access to the uncorrected spatial layout of the original patch? I.e. the STN removes some information about the spatial layout of the patch, then the TPS transform is estimated from that reduced information, but then the TPS tranform is applied to the original patch. Does such a setup make sense?
- Is there a reason that the DeepDeform dataset [4] was not used for evaluation? Is it not suited for the task?

There are a number of smaller clarifications that I suggest a revised version should take into account:

- What is the size of the receptive field of the ResNet features after the sixth convolutional block (line 134)?
- In lines 156-158, it would be good to mention shortly that the parameters theta get *regressed* by the spatial transformer from X. It is not fully clear whether theta is the same globally for all H/s x W/s pixels or whether it differs between them.
- The MLP from line 178 is not convolutional, so are the 5 x 5 128-dim feature vectors stacked before passing them to the MLP?
- Does T_theta in line 159 refer to the STN in Fig. 2? If so, the notation for T_theta(G_I) in Fig. 2 should be changed to avoid confusion.
- It would also be good to add the missing notations to Fig. 2, e.g. the TPS parameter regressor S (line 152), M_theta and M_I (line 159), the warped features Y (line 161).
- What does the "TPS" in the green box in Fig. 2 stand for? Doesn't the TPS regressor S regress theta_tps directly? 
- I don't understand the sentence in lines 186-188. Why is the regular (not the TPS-transformed) polar grid applied to the image (not just the patch)? What are the keypoint's attributes, where do they come from, have they been mentioned before? Why are they needed to get an identity transform? What is the difference between scale and size?
- The sentence in lines 196-198 is a bit ambiguous and could be split into two shorter sentences, and the sentence in lines 198-199 could be pulled in front. "which" could refer to the loss or to HardNet. It is not obvious that HardNet comes from [22], only that the triplet loss is used from [22].
- A short explanation of what needed to change for adapting to R2D2 and ASLFeat (lines 282-283) in the appendix would be good for completeness.

Update after rebuttal ---

I initially had a similar difficulty in understanding the usages of two STNs as reviewer fuQg, which made me question the rationale behind the method design and potential issues with it. Given the authors' clarifications, I corrected my understanding (the first STN uses a regular, straight grid) and am now satisfied with the design. Similarly, I was concerned about baselines (maybe training data is the crucial difference), which the new experiments in the rebuttal show to not be the case. Reviewer dECi brought up some more important ablations, where I find the results provided in the authors' response convincing. I find the remaining points mentioned by the reviewers to be addressed well by the authors' response.

However, there is one part that is not fully convincing me, namely comparison to existing work, as mentioned by reviewer dECi. The lack of non-rigid datasets in the literature suitable for evaluation is unfortunate, but the authors make an effort to introduce their own dataset and to evaluate on the couple of datasets that are suitable. The performance on rigid datasets is competitive, but not state-of-the-art, which I find good enough, given that the submission has to relax assumptions to be able to handle non-rigid deformations.

I will change my rating from 5 to 7.
limitations_and_societal_impact: Limitations are mentioned in one sentence (lines 371-372) but not discussed to give a good sense of them, and I am unable to locate the limitations that are supposed to be in the appendix. Qualitative results could be added.

Potential negative societal impact is not discussed. Better tracking of clothed people (which have non-rigid deformations) through surveillance might be one point.
review_title: null

Meta-review: This paper proposes an approach to learn deformation-aware local feature descriptors using spatial transformer networks (STNs).  The reviewers found the paper clear, the method practically useful and the experimental comparison to be comprehensive with the results quite strong.    I agree with this assessment.  I also appreciate the robust and healthy discussion between the reviewers and the authors that I think helped point in directions that make the paper better.
Decision: Accept (Poster)