Extracting Deformation-Aware Local Features by Learning to Deform
Authors: Guilherme Potje
Renato Martins
Felipe C. Chamone
Erickson Rangel Nascimento
Venue: NeurIPS 2021 Poster (2021)
Keywords: Local descriptors
Non-rigid matching
Deformation
Spatial Transformers

Abstract: Despite the advances in extracting local features achieved by handcrafted and learning-based descriptors, they are still limited by the lack of invariance to non-rigid transformations. In this paper, we present a new approach to compute features from still images that are robust to non-rigid deformations to circumvent the problem of matching deformable surfaces and objects. Our deformation-aware local descriptor, named DEAL, leverages a polar sampling and a spatial transformer warping to provide invariance to rotation, scale, and image deformations. We train the model architecture end-to-end by applying isometric non-rigid deformations to objects in a simulated environment as guidance to provide highly discriminative local features. The experiments show that our method outperforms state-of-the-art handcrafted, learning-based image, and RGB-D descriptors in different datasets with both real and realistic synthetic deformable objects in still images. The source code and trained model of the descriptor are publicly available at https://www.verlab.dcc.ufmg.br/descriptors/neurips2021.

Extracting Deformation-Aware Local Features by Learning to Deform
Guilherme Potje
Renato Martins
Universidade Federal de Minas Gerais
Université Bourgogne Franche- Coméé
Felipe Cadar Universidade Federal de Minas Gerais
Erickson R. Nascimento Universidade Federal de Minas Gerais
Abstract
Despite the advances in extracting local features achieved by handerafted and learning-based descriptors, they are still limited by the lack of invariance to non- rigid transformations. this paper, we present new approach to compute features from still images that are robust to non-rigid deformations to circumvent the problem of matching deformable surfaces and objects. Our deformation-awee local descriptor, named DEAL, leverages a polar sampling and a spatial transformer warping to provide invariance to rotation, scale, and image deformations. We train the model architecture end-to-end by applying isometric non-rigid deformations to objects simulated environment as guidance to provide highly discriminative local features. The experiments show that our method outperforms kate-offtheeart handcrafted, learning-base image, and RGB-D descriptors in different datasets with both real and realistic synthetic deformable objects in still images. The source code and trained model the descriptor are publicly available at https: //www. verlab ab. dcc. wffg.ffg....... br / http///cripporseeerrpeeeeee
Introduction
High-quality matching of images taken from cameras in different poses and conditions remains one of the key challenges several tasks. Tracking points as camera or an object interest is moving is crucial for finding and reconstructing objects and scenes [29, 30, 31], camera localization and mapping [18, 36], registration [43, 37], and image retrieval [26, 39], to name a few tasks. Extracting invariant and compact representations to describe the vicinity of pixel has been one of the key forces in enabling high-quality matching on these tasks. Despite the importance of invariance to varying illumination, viewpoint, and the distance the object of interest, real-world scenes impose additional challenges. After all, we live in a non-rigid world populated by objects that may have different shapes over time due to deformations. Over the past few decades, many descriptors have been proposed [19, 3, 6, 35, 42, 47, 44, 40, 23, 13]. These approaches can be roughly grouped based the type of input information, such intensity and/or depth images, and be further divided into handcrafted and learning- based methods. While local learning- approaches, such as Log-Polar [13], hold state of the regarding affine transformations and local illumination changes, recent handcrafted techniques achieve better performance in the presence of non-rigid transformations by using depth data [25] or by modeling the deformations from image intensity information, which inevitably leads to an increase of the computational cost to compute the description [38]. Since depth data is not available many contexts and applications, and low processing time is highly
Department of Computer Science, Universidade Federal de Minas Gerais, Brazil. VIBOT EMR CNRS 6000, ImViA, Université Bourgogne Franche-Comté France. Corresponding author's e-mail: guipot ufmg.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

recommended for low-level processing such as the estimation of local features, devising a descriptor that correctly and reliably establishes invariant features from corresponding points is of central importance to high-quality matching of images in real-world scenarios.
In this paper, we present new end-to-end method extract local features from still images that are robust to non-rigid deformations. Conversely to recent approaches that exploit a single network branch to extract features from local patches or depth data, our approach is composed of additional guidance branch component (spatial transformer non-rigid warper) that jointly learns context about the global deformation affecting the image.
Figure/ 1 depicts the rationale behind our method. For learning the contextual deformation infor- mation, we provide several views of deformed objects by applying isometric non-rigid trans- formations to the objects in a simulated envi- ronment guidance to extract highly discrim- inative local deformation-aware features. Un- like non-rigid aware deformations descriptors like GeoBit [25], our approach does not require depth data to produce descriptors robust to non-
Teanue extractor
warper net
Seature eMEraCICE
deformed patches
descriptors
Descrppoo Value
rigid deformations and it is notably faster than DaLI [38] while holding the best performance.
-0.1
Although many descriptors still disregard non- rigid transformations, in recent years, that as- sumption has been dispelled, partly due to the increasing popularity in the use of RGB-D sen- sors to capture properties that characterize 3D objects such as geodesic distances. On the flip side, RGB-D sensors are still less prevalent than RGB cameras. In our nonrigid-world with flexible humans, animals, tissues, and leaves where high-resolution RGB cameras are ubiq- uitous, the ability to create discriminative and deeformationninvariii invariant descriptors using a single picture plays central role in achieving high- quality matching in real-world scenarios. this work, we demonstrate that our end-to-end architec- ture outperforms state-of-thh--ar handcrafted and dearning-based image and RGB-D DD descriptors in matching scores on different benchmark datasets of real deformable objects, as well as with realistic application scenarios on content- based object retrieval and tracking.
-0.7
0
16
32
54 80
96
112
128
Descriptor Indexx
Figure I: Learning to deform and extract fea- tures. Our network learns meaningful deforma- tions during training that lead to improved match- ing performance, Notice that the descriptor's 's signatures (Descriptor Value plot) of corresponding keypoints, with and without a local deformation (red and blue curves), highly correlated when extracted with the pro- posed model.
shown in the experiments.
2 Related work
Handerafted and learning-based feature extraction. Trraditionall,, image descriptors are aimed to providing rich and invariant representations of objects observed from different viewing conditions. Two notable handcrafted approaches to describe keypoints still images are SIFT [19] and ORB [35]. These methods use local gradients and dominant orientation estimation to extract visual features with scale and rotation invariance. Different extensions of these deseriptors have been proposed to images in some particular manifolds such as BRISKS [14] and SphORB [50] for spherical images. Recently the advent of convolutional neural networks boosted the development of several end-to-end learned descriptors [47, 44, 40, 23, 13, 20, 32). These descriptors often outperform handcrafted counterpart on classical image matching benchmarks such UBC Phototour [5] and HPatches [2]. Yet, the vast majority of existing local descriptors are approximately invariant to affine image transformations, often disregarding images of deformable surfaces, which the main feature of our descriptor.
Deformation-awarc reconstruction and matching. Although most works on local descriptors are designed focusing on affine image transformations, some works considered non-rigid surfaces and deformable objects. In DeepDeform [4], for instance, the authors propose to estimate non-rigid point-wise RGB-D correspondences by data-driven approach that is then used by a non-rigid 3D reconstruction framework. Although their method works well for fast motions and non-rigid surface

---

tracking, it has heavy computational burden for establishing correspondences for typical amounts of keypoints from commonly used detectors. Also, the method does not generate a descriptor that can be used for other related vision tasks. Simo-Serra et al. proposed the Dal.I descriptor [38], where image patches are converted to 3D surface in order to encode features robust to non-rigid deformations and illumination changes. Despite achieving high matching performance, DaLI suffers from high computational and storage requirements. Recently, DeformNet [31] was designed to infer non-rigid shapes from still images. The network architecture is composed of a 2D detection branch and a depth branch. The detection and depth branches are subsequently merged by the shape branch, which uses a pinhole camera model to re- re-project the shape to 3D. In the same context, GeoBit [25] takes advantage of geodesics from objects surface to compute
comeeric-invariian features. The method uses RGB-D data combined with the Heatflow [7] method to encode visual features in a binary string that are invariant to deformations. However, GeoBit requires accurate depth information and only works with RGB-D images. Our method, in its turn, only requires monocular images, is faster to compute, and provides a better matching performance, as demonstrated in our experiments.
Spatial attention mechanisms and matching. Spatial transformer networks (STNs), introduced by the seminal work of Jaderherg al. [16], allowed differentiable warping operations be used directly with existing network architectures. The core idea is the learning of an attention mechanism (named Localization network) that is parameterized to represent the image transformation of interest, e.g., affine, homography or a thin-plate-spline warp. They have been applied to several tasks including image classification [16], semantic alignment [34], geometric matching [33] and local descriptors [13]. LF-Net [27]. a detect-and-descriie method, first locates interest points using a fully convolutional network, which are then cropped in patches with a STN layer considering an affine transformation encoded by keypoint attributes. The cropped patches are subsequently used to compute local descriptors by the description network. Spatial attention mechanisms have also been extended into more generic settings via deformable kernels [9], to learn kernel offsets in network layers operators to define deformable convolution networks. More recently, the end-to-end joint detector and descriptor ASLFeat [21] used deformable convolutions to increase the network's kss expressiveness, which shares some concepts with our method. However, the deformable kernels employed by ASLFeat considers high-level feature maps, and it does not model deformations explicitly. Its strategy results in earlier layers not being aware of low level image deformations, usually present in geometric frasmtomatioos.. Moreover, the method is not robust to rotation changes. Our method leverages two carefully designed spatial transformer networks to sample and guide the model to provide invariance to deformations and, to the best of our knowledge, is the first end-to learning method to explicitly tackle non- non-rigid deformations with a carefully designed spatial attention mechanism.
3 Deformation-aware network architecture
Our REformation-Awaare Local descriptor (DEAL) has been designed to jointly learn to undeform and extract discriminative and invariant features from local regions. The proposed architecture is able to handle deformations and perspective distortions of sampled patches in the vicinity of the keypoint. It also attenuates noise in keypoint attributes which improves the local description performance and robustness to image transformations. We designed our network as hybrid approach, in the sense that it first extracts dense features and the final output is a local descriptor for each keypoint, unlike both trends of describing local patches like HardNet [23] or extracting dense features jointly with keypoints such as SuperPoint [10] and R2D2 [32]. Our approach combines the best of these two trends: i) the robustness to occlusion, strong illumination, and perspective changes of local patch-based descriptors that work directly with low-level image information; and ii) the capability of dense methods in encoding local semanties from the image to disambiguate hard pairs. Additionally, our model decouples the problem of learning distinctive descriptors robust to deformations in two complementary tasks, i.e., first estimating a warp with spatial attention mechanism and then describing the sampled local patches using the estimated non- non-rigid warp. Figure 2 outlines schematic representation of the model.
3.1 Mid-level feature extraction
For the extraction of mid-level image features that encode local deformation information, we employ the ResNet-34 [15] and truncate it at the sixth convolutional block. The motivations to process
3

---

Image + Keypoints
w
Non-rigid Warper
TPS Regressor:
ResNet-34 Head
STN:
STN: Ω
X. Poo
i Otps
G1
Resample
To (9r)
(Undeform)
Go
Undeformed Patches
128-d descriptors
Hard Triplet Loss:
(32,32)
+ d(H( d(H
HardNet
positive
hard
negative
Figure 2: Proposed formulation for computing descriptors of deforming objects. The non-rigid warper component undeforms local image patches by applying a learned deformation for each keypoint. Higher level image information such as appearance and shape are encoded globally by ResNet feature tensor X. Then, the two carefully designed spatial transformers and the TPS regressor network decode the information locally, by estimating the warp parameters Osps used to rectify patches from the original image. The green arrows imply keypoint attribute information. The rectified patches are fed to HardNet that extracts discriminant descriptors. The full network model is trained end-to-end by optimizing the hard triplet loss.
features from ResNet instead of directly sampling local patches the image are twofold: First, the mid-level feature maps provided by ResNet features allow the network to learn to encode local deformations from contextual cues such local texture statistics, illumination and viewpoint changes; Secondly, the receptive field of our local feature descriptor is not tied to a fixed support region size and sampling pattern like traditional patch-basee local descriptors.
Strong perspective and deformation changes drastically influence the optimal support region sizes for the same keypoint different images; thus, present architecture that is aware of these local deformations by design, since the dense feature map allows such information to flow to the non-rigid warper module. The ResNet-34 head outputs a feature map X E "f/8x W/8x 128 by performing convolutions until the spatial resolution achieves 1 1/8 of the original input resolution. The feature map X is used as input for the the non-rigid warper component, which computes the Thin-Plate Splines (TPS) parameters with the spatial transformer model non-rigid warp.
3.2 Non-rigid warper
The Non-Rigid Warper (NRW) is composed by two spatial transformers and Q in addition to the TPS parameter estimator S. After obtaining the mid-level feature tensor of the entire image X. for each keypoint, W samples a local x 5 x 128 tensor centered at the coordinate of the keypoint, which are flattened and forwarded to S, to estimate the local deformation parameters of the keypoint. Afterward, Ω initializes the polar transformation G, with the keypoint attributes, and finally, patches re-sampled from the original image and propagated HardNet descriptor extractor.
Feature sampler. We designed Spatial Transformer Network (STN) Li to sample local patch of mid-level features Y from the ResNet feature map X at the position of each keypoint in an end-to-end differentiable manner. For each keypoint, given 2D affine transformation function TK. where K is an affine matrix obtained from the downscaled keypoint position (to match the spatially downscaled spatial feature map X by a factor of 8), and the identity mesh grid M,, generate transformed mesh grid coordinates as: MK
Tk (M,). Once the meshgrid is transformed, for each pixel's
spatial coordinate p. p E 2 in M (1-) Vi [L, [1, Mj [1, W]. we bilinearly interpolate values in

---

X for each channel e to obtain the warped features Y per keypoint:
p Σ Σ X (pp(mmm) 1 m P/1) x n P/e)
(1)
where p' is the decimal part of the pixel coordinate p [16] and
x is the non-negative value of x.
The mesh grid .MK_ E _xx5x2 encodes local patch of 2D spatial coordinates centered at the keypoint. MM is used to interpolate the feature map X to sample the local mid-level features that encode local keypoint deformations. The local feature maps Y are flattened and forwarded to the localization network that regresses the parameters of the local deformations.
Localization network. We model the non-rigid object deformations with Thin-Plate Splines (TPS) [11]. TPS warps representing 2D coordinate mappings R2 + IR² are often used to model non-rigid deformations. They produce differentiable smooth interpolations, which can be easily coupled in neural networks using attention mechanisms. The TPS formulation elegantly enables one to model the affine component the transformation through an affine matrix component E R2x3, and non-affine component encoded by the weight coefficients wx E I separately representing distortions. Given a 2D point E R., weight coefficients and control points Ck2 E R2 both in homogencous coordinates, the corresponding mapping q' can be computed with:
q' = Aq
Σ k=1
P(
cs W
where p(r) = y2 log is the TPS radial basis function. To estimate the coefficient weights of the TPS, employ regressor network S, implemented as Multi-layer Perceptron (MLP) that takes as input the interpolated feature map Y of each keypoint to compute the TPS parameters O1ps. that composed by the affine parameters A and the weight coefficients Ws. We use 64 control points in our implementation, which provides a good trade-off between non-rigid modeling capacity and computational cost according our experiments. The estimated 0tp in conjunction with the identity polar grid Gj containing all control points C₁ (initialized using the Polar Transformer) are used to sample a rectified image patch.
Spatial transformer network sampler. According Ebel et al. [13], polar representations of patch around keypoints increase the robustness of the descriptor extractor to image transformations, while also achieving rotation equivariance. Considering these findings, we adopt this representation in the sampled rectified patches. For that, a fixed regular polar transformation G1 is computed in the second Spatial Transformer Network Ω parameterized by the keypoints' attributes. The attributes are obtained from keypoint detector such as SIFT, and encode the keypoint's location (its coordinates in the image), its canonical orientation (used to achieve image in-plane rotation invariance), and its size (an estimate of the support region around the keypoint). We use those attributes to generate an identity polar transformation map 97 IR 3232x2x2 for each keypoint, encoding the point's initial position, rotation and size. The identity grid Gr is transformed to the warped grid Go using the estimated Otpe- which samples the final rectified patches by the second STN Ω. Finally, the patches are forwarded to the feature extraction network that computes a distinctive and compact descriptor for the rectified polar patch. We employed HardNet [23] as the backbone network in our implementation.
3.3 Local descriptor extraction and loss
Our architecture can trained end-to-end since all employed operations are differentiable, including the non-rigid warper component. The HardNet H(-) takes as input 32 x 32 grayscale patches rectified by the non-rigid warper, and outputs a 128-dimensional I.2 normalized feature vector for each keypoint. Then, the hardest in-batch triplet loss [23] is computed on mini-batch of pairs of feature vectors, enforcing distinctiveness of the extracted descriptors.
Let A E N and B E NDD be a matrix of N vertically stacked D-dimensional feature vectors of corresponding patches extracted by HardNet. Assuming that the descriptors are L2 normalized, we can calculate the matrix of distances DNXN = 2(1 VABT). The hardest negative example h; for each row D', D' = D + NINxN, where is constant 2 2 needed to supress the corresponding
5

---

pairs from the diagonal of the distance matrix, is computed as 0/²² min(D,). The hardest in in-batch margin ranking loss is then calculated as:
CH (0)
max(0, 1 84 (f
where j is the margin, 54 Hp) Hy''Il, is the distance between the corresponding patches,
and 5, = H()) - Hh)ll is the distance to the hardest negative sample in the batch.
Experiments and results
We evaluate our descriptor in different publicly available datasets containing deformable objects in diverse viewing conditions such as illumination, viewpoint, and deformation. For that, we have selected the two datasets recently proposed by GeoBit [25] and DeSurT [45]. They contain color images of 1l deforming real-world objects, where keypoints are independently detected with SIFT and ground-truth correspondences are done following the protocol of [17].
Baselines and metries. For the descriptors based on pre-detected keypoints, we use SIFT keypoints that are detected independently for each frame and shared among all methods, and we detect at most 2, 048 keypoints. Since this paper tackles the problem of matching images of deformable objects. we compare our results against two infformmationiivarrina
local descriptors: DaLI [38] and GeoBit [25]. Aside from these descriptors, we also include two the well-known local binary descriptors for 2D images ORB [35] and FREAK [1]: dooting-point gradient based descriptor: DAISY [42]; one local descriptor that combines texture and shape: BRAND [24]; and three learning methods, TFeat [44], Log-Polar [13), which is an improvement of HardNet [23]. and SOSNet [41].
We also demonstrate that method is able surpass recently proposed detect-and-deseribe methods. These methods detect their own 2,048 keypoints that are optimized for their descriptors. Since the -eeet-a techniques do not share the same keypoints, their evaluated scores can be also impacted by the repeatability of their keypoints, and directly comparing them to the local descriptors that use the same set of SIFT keypoints is not straightforward. We have included those methods in the experiments for a more comprehensive evaluation of recent methods. We considered in the comparisons R2D2 [32], ASLFeat [21], D2-Net [12], LF-Net [27]. and LIFT [48].
The performance assessment is done with the matching score (MS) metric defined by Miko- lajezyk al. [22]. Considering a pair of images i and j. the metric is defined as: MS #correct/min( # keypoints, keypoints;), where #correct is the number of correct keypoint matches at a pixels threshold (used in all experiments in this paper) obtained by matching the descriptors using nearest neighbor search, and #keypoints is the number of detected keypoints on image i and j. We also compute the Mean Matching Accuracy (MMA @ 3 pixels) as M MMA A = #correct/ #matc/ambleee where #malhhable is the number of points that can be matched under the threshold.
4.1 Implementation details
Training dataset. Although most of the reported results are from real-world images, our network was trained with simulated data only. For that we developed a simulation physics engine to generate plausible non-rigid deformations (isometric transformations) of surfaces (please check the supple- mentary material for additional details of the simulation engine). The dataset is composed of 20, 20,000 (960 720 resolution) image pairs with ground-truth correspondences. In order to generate realistic deformed images in the simulation engine, we performed texture mapping of the surfaces with real images extracted from large-scale variation such as intensity, global position, number of light sources, directional lighting, and color changes to enforce realistic non-linear illumination conditions in the simulation. The correspondences between frames are generated by first independently detecting up to 2,048 SIFT keypoints for each frame and then corresponding them using the simulation data under a threshold of pixels. Each image pair contains, on average, approximately 1K ground-truth correspondences, summing up to total of about 20M keypoint correspondences. Independently detecting the keypoints in the frames key step to improve generalization, since it considers repeatability properties of keypoint detectors.
Ceocuurr-foo--Motiin datasets [46]. We also added illumination

---

Network and training setup. We implement our network using Py PyTorch [28] and optimize it via Adam with initial learning rate of 5e 5, scaling it by 0.9 every 3, 800 steps. The network is trained end-to-end, setting the TPS regressor's weights of the last layer to zero, resulting in an identity warp at the beginning of training. We used a batch size of image pairs containing up to 128 keypoint correspondences for each pair in our setup. The keypoint correspondences are randomly sampled from a uniform distribution with fixed seed during training, and we train the network for 10 epochs. The model is trained using a siamese scheme, where descriptors are extracted for the first and second set of corresponding keypoints (positive examples) using two networks with shared weights. The negative examples are calculated using hard mining strategy in the batch as described by Mishchuk et al. [23] and used in the triplet loss. We also apply Average Pooling in the angle-axis of the polar patches at the end of HardNet feature maps to achieve rotation invariance. Our network implementation has 3.7M trainable parameters and takes about 5.5 hours to train on a GeForce GTX 1080 Ti GPU.
Sensitivity analysis. To evaluate the sensitivity and the influence of hyperparameters in our network, we performed the following experiments: (i) margin ranking parameter sensitivity in the triplet loss; (ii) use of anchor swap in the triplet loss [44]: (iii) larger support region for the non-rigid warper (NRW) component; and (iv) use of dropout in the fully connected (FC) layers. Table I shows the
Table I: Effect of hyperparame-
ters in the proposed network.
MMA average achieved by testing different hyperparameters on the Bag sequence [25], which we used as validation set and removed it from the benchmark experiments. The training steps and initialization are kept the same for all tested variations, and only one tested hyperparameter is changed while fixing all others for the sake of reproducibilityy and consistency. The baseline model uses the margin jr = 0.5, no anchor swap, STN output of 3 x 3 and Dropout with probability p= = 0.1 in the FC layers. We can observe that using the margin 0. 75 and STN output of 5 x 5 increases the performance individually, We tested both changes simultancously but it resulted in worse results than the individual changes. Thus, we update the final model (used in all experiments) to use STN output of 5 x 5 and keep other parameters from baseline unchanged, since they decrease the performance. We did not include larger STN outputs since we noticed marginal performance gains, while the model increases its computational requirements.
Hyperparameter
MMA
Baseline
0.603
Margin 0.25
0.592
Margin 0.75
0.608
Anchor Swp.
0.592
STN out. 5 x 5
0.613
No Dropout FC
0.601
4.2 Results on real image sequences of deformable objects
Table shows the matching scores for all descriptors in our experiments. For the sake of a fair comparison, the keypoints are shared among all local descriptors in this experiment, including rotation and scale attributes of SIFT keypoints, with exception of methods that detect their own keypoints jointly with the descriptors. In this case, is also impacting the performance of MS and MMA. One can see that our method outperforms all local descriptors that use SIFT keypoints, including GeoBit, that uses additional depth information to extract deformation- informattion-invariant features. GeoBit, as expected, achieved the second-bes accuracy overall, followed by the learning-bast based methods. DaLI performs the best among the hand-crafted descriptors, due to its robustness against deformations. BRAND, for its turn, provides the worst scores overall because of its strong assumptions of geometric rigidity of the object both image and 3D space. L.-NNet performs best among the joint detection and description methods. We also re-trained two of the learning- non-rigid training dataset can improve the matching of deformable surfaces for existing methods. The re-trained TFeat"* on our training data did not improve the MS, and slightly decreased the MMA. On the other hand, the re-trained Log-Pot Polar"* yielded an increase from 0.57 to 0.65 MMA (8 p.p difference), but it is still 10 p.p. under the performance of our deformation-ave aware descriptor scores. Additional details about the re-trained methods are available in our supplementary material. Figure 3 shows a matching example from Kinect 1 dataset (Shirt I sequence), depicting a strong deformation. It is noticeable that our method improved the matching, especially in the most deformed regions (red dashed square). We also evaluated our descriptor on HPatches [2], a dataset of planar scenes
worth recalling that the repeatability of the detector
methods, "'Fett* and Log-Polarr", to verify if the proposed
'The training data and source code are available at www. verlab. dec. ufng. br/descriptors/ neurips2021.
7

---

Table Comparison with state-of-the-art descriptors. The methods marked with are computed on RGB-D images. The mark ** indicates that we re-trained the networks using our proposed non-rigid dataset, and 1 indicates that the method detect own 2,048 keypoints. Best in bold and second-best underlined. The mean was calculated with full-precision values before rounding. These results indicate that the proposed deseriptor exhibit improved robustness against deformations in all considered datasets.
Datasets: 833 pairs total MS MMA @ 3 pixels
Descriptors
Mean
Kinect [25]
Kinect [25]
DeSurT [45]
Simulation [25]
BRAND* [24]
0.17 0.34
0.22 / 0.49
0.14 / 0.33
0.04/ 0.09
0.16 0.34
R2D2 [32]
0.17 0.36
0.25 / 0.59
0.14 / 0.32
0.06 / 0.16
0.17 0.39
ORB [35]
0.19/ / 0.38
0.25 / 0.55
0.18 / 0.40
0.14/0.30
0.20 0.43
SOSNet [41]
0.17 0.34
0.25 / 0.55
0.17 0.38
0.26 / 0.57
0.22 0.44
DAISY [42]
0.23 / 0.47
0.29 / 0.62
0.16 / 0.37
0.19 / 0.39
0.22 / 0.48
FREAK [1]
0.24
0.33 0.72
0.16 0.38
0.15
0.23/0.51
DaLI [38]
0.25
0.35 / 0.76
0.21 / 0.48
0.10
0.25
TFeat"
0.48
0.28/ 0.62
0.20 / 0.46
0.28/ 0.61
0.25
TFeat [44]
0.25/0.55 0.50
0.28 0.61
0.21 / 0.48
0.29 0.63
ASLFeatt [21]
0.31 / 0.58
0.39 / 0.69
0.28 0.53
0.19
0.31 /.56
Log-Polar [13]
0.28 / 0.58
0.30 0.65
0.23 / 0.54
0.22 / 0.49
0.26 / 0.57
D2-Nert [12]
0.20 0.50
0.23 0.82
0.14 0.47
0.11 / 0.30
0.17
LF-Net [27]
/
0.51 0.44
0.28 0.77
0.21 / 0.74
0.36 0.59
LIFTt [48]
0.09 / 0.57
2.16/0660
0.08 / 0.52
0.13 / 0.73
0.12 0.62
Log-Polar"
0.29 0.60
0.31 / 0.69
0.24/ 0.56
0.33 / 0.72
0.29 0.65
GeoBit [25]
0.31 0.65
0.35 / 0..7
0.20 / 0.47
0.32 / 0.71
0.30 / 0.66
Ours (DEAL)
0.33/ / 0.68
0.38 / 0.85
0.27 / 0.63
0.36 0.80
0.34 0.75
a) DaLI - 126 inliers
b) GeoBit 179 inliers
DEAL 208 inliers
Figure 3: Matching example of deformed shirt. Correct matches are drawn as green, and wrong ones as red lines. Our descriptor better handles the strong deformation (highlighted in the zoomed boxes), among the deformation-awan competitors.
affected by either illumination or viewpoint rigid changes (homography). Our descriptor presented competitive performance in this dataset, even though it does not contain non-rigid deformations. These additional experiments can be checked in our supplementary material.
Rotation and scale invariance. To evaluate the robustness of our descriptor to rotation and scale, we select sequences with strong rotation and scale transformations, where the camera suffers in ml-paane rotations from 0 to 180 with 10 degrees steps for rotation tests. For the scale tests, the camera was moved backward in the direction, producing downscaling of the object. From Figure 4, one notice that our method holds the best invariance image in-plane rotations, and its very close to GeoBit in the scale sequences, although our method only requires monocular images.
4.3 Ablation studies and processing time
To evaluate the contribution of the components of our architecture and support our implementation decisions, we performed the ablation analysis of different parts the proposed method.
Contribution of the TPS warper. We consider four different setups: (i) the use of the TPS on standard regular grid patch, (ii) training the proposed architecture on a new dataset con- taining only rigid planar objects, (ii) using the second STN network l with fixed parameters

---

06
2.5
Dell
GooBit -x. Logetoler
Logaoolar
94
-+ IFeat DEAL
Matching Score
Matching
0.
1.2
10
70
130
1.50
Rotation (degrees)
Scale
Figure Rotation and scale invariance. Matching score curves obtained by matching SIFT keypoints showing results for rotation and scale for each target frame relative to the reference using the Simulation dataset. This experiment evaluates the robustness the descriptors to both deformation, scale, rotation, and illumination in the most challenging sequence.
at the identity (Fixed STN polar sampler- akin to Log-Polar); and (iv) with the NRW compo- nent for modeling the deformations. In this analysis, we use the scale simulation sequences, that were purposefully chosen because they exhibit mostly strong deformations across multiple im- age pairs. The MMA achieved by each configuration can be seen Table 3 (best in bold). All variations were trained until convergence with the same training procedure and HardNet initialization.
The ablation tests show that the polar sampler can improve matching accuracy compared to the Cartesian counterpart, and the non-rigid dataset also helps to improve accuracy. Most importantly, the NRW module alone can improve the accuracy of the descriptors by 4 p.p. when keypoints are affected by deformations, important gain for tasks that require high matching accuracies such as deformable surface registration and non-rigid reconstruction.
Table 3; Ablation.
Method
MMA
Cartesian NRW
0.68
Planar Data NRW
0.75
Fixed Polar STN
0.75
Polar NRW
0.79
Use of pre-trained ResNet features. We also verified if transfer learning from pre-trained ResNet model ImageNet is feasible for the task of patch rectification. However, the training of the network did not converge well when using the pre-trained weights. We argue that the non-rigid warper module is focused on learning warping parameters, which is a considerably distinct task compared to classification. Thus, we initialize the weights of the entire network and train from scratch.
Processing time. We executed the three deformation-invarree DEAL) on set of 250 descriptors (from 640 480 resolution images), running on a Intel (R) Core (TM) i7-7700 CPU @ 3.60 GHz and GTX 1080 Ti GPU. Our descriptor was significantly faster than DaLI and GeoBit. While our method (GPU) spent 0.03 seconds to compute the descriptors, DaLI (CPU-only) and GeoBit (CPU-only) spent 112.95 and 33.72 seconds, respectively.
descriptors (DaLi, GeoBit, and
4.4 Applications
1.00
Object Setrieval with SET Seyooints Arc Lu
Object Retrieval. To further demonstrate the effectiveness of our descriptor in potential real- world applications, we performed experiments in two complementary real-world applications: object retrieval and tracking of deformable ob- jects (more results can be found in the supple- mentary material).
0.30
0.75
0.0
DELF
Geedit Log Fobo
Foot DE31
In the retrieval application, we used a Bag-of- Visual-Wai Wordds approach [8]. For each descrip- tor, we first construct visual dictionary used to compute a global descriptor for each image. Given a query image, descriptor and use K -Narest Neighbor search to obtain the top K closest objects. The metric
Dell
-30
Figure 5: Object retrieval. Our descriptor outper- forms descriptors with < 10, including the sfate-of-ther the-t-rrrtt retrieval descriptor DELF.
compute the global

---

Table 4: Tracking average LPIPS and RANSAC inliers rate. Approaches with are computed on their own detected keypoints, while uses RGB-D images. Best in bold and second-bes underlined.
DaLI
DELF
GeoBit"
Log-Polar
TFeat
DEAL
Inliers RANSAC t
0.32
0.25
0.46
0.36
0.32
0.46
LPIPS +
0,44
0.33
0.23
0.24
0.24
0.24
used is the retrieval aceuracy, i.e., the number of correct objects retrieved in the top K images. Figure 5 shows the results with the best performing descriptors according to the matching experi- ments. Our method achieved the best accuracy with K S 10. In this experiment, we also consider DELF [39], the state-of-the thrttrr-t descriptor designed and trained for image retrieval. Since DELF does not allow the description on commonly adopted keypoint detectors, we perform its evaluation using its own detected keypoints. The results indicate that our deformation aware network was able to provide good results different but related task.
Non-rigid tracking results. This task consisted of tracking a region-of-interee of tem- plate image over time using the correspondences from the computed descriptors. We approx- imate the deformation using TPS warping, which was combined with a RANSAC filtering to remove outlier correspondences for all descriptors, following the protocol proposed in [43]. Table 4 shows the average values of the Perceptual Patch Similarity distance (LPIPS) [49] and the number of inliers found by RANSAC for the main competi- tors. We also included DELF using their own detected keypoints on all sequences. As can be seen, even using RGB data only, our descriptor showed similar results GeoBit. Representative qualitative results are shown Figure for the sequences Shirt3 and kanagawa_rd rot. While the corre- spondences using our descriptor (last row) allowed the tracking of frames even in ex- treme conditions of rotation and deforma- tion, DaLI was particularly affected by the object viewing changes.
Temolate:
Shirt3 (Kinect1)
kanagawa, rot (Simulation)
frami
framm
Trams
frame
framme
Current
iagas
Devitl
IngyPibe
SUN55
DEAL
Conclusions
Figure 6: Tracking results. Qualitative tracking results for two sequences. Please notice the performance of the proposed approach (last row) for these challenging frames.
This paper presents a new local descriptor robust to non-rigid deformations, scale, and rotation transforms. The local features are extracted by deformation-awar architecture trained in an end-to-end manner following defor- mation guidance strategy. Besides providing highly discriminative and invariant feature vectors, our approach requires only still images and is time-efficient. We demonstrate the effectiveness and robustness of our descriptor by extensive experiments comprising the comparison with state-of-the-ar descriptors, ablation studies, and its use on two real applications.
limitation of our work is that its performance compromised when dealing with surfaces having sharp deformations, discontinuities, reflections, and poor keypoint detector repeatability. These conditions also hamper the competitors. A more tailored discussion on the limitations of our method, including visual results and examples of failure cases of all methods can be checked in our supplementary material. Since matching keypoints across images fundamental step in many applications, our proposed method could be potentially used in surveillance contexts, for instance, recognizing and tracking humans from images. this circumstance, the misuse of such technology could negatively affect people's 's privacy and mobility rights.
10

---

Acknowledgments and Disclosure of Funding
The authors would like thank CAPES (#88881. (18888L.10226/20100 120236//0016-01), CNPq, FAPEMIG, and Petrobras for funding different parts of this work. R. Martins was also supported by the French National Research Agency through grants ANR MOBIDEEP (ANR-17-CE3--011) ANR CLARA (ANR-18. CE33-0004) and by the French Conseil Régional de Bourgogne-Franche-Cooe Franche-Comté.
References
[1] A. Alahi, R. Ortiz, and P. Vandergheynst. Freak: Fast retina keypoint. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 510-517. Ieee, 2012.
[2] V. Balntas, K. Lenc, A. Vedaldi, and K. Mikolajczyk. HPatches: benchmark and evaluation of handcrafted and learned local descriptors. In CVPR, 2017.
[3] H. Bay, A. Ess, T. Tuytelaars, and L. V. Gool. Speeded-up robust features (surf). Computer Vision and Image Understanding, 110(3):346 - 359, 2008.
[4] A. Božič, M. Zollhöfer, C. Theobalt, and M. Nießner. DeepDeform: Learning non non-right ron-rigid rgb-d reconstruction with semi-supervised data. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[5] M. Brown and D. G. Lowe. Automatic panoramic image stitching using invariant features. International journal of computer vision, 74(1):59-73 2007.
[6] M. Calonder, V. Lepetit, M. Ozuysal, T. Trzcinski, C. Strecha, and P. Fua. Brief: Computing a local binary descriptor very fast. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7):1281-1298, 2012.
[7] K. Crane, C. Weischedel, and M. Wardetzky. Geodesics in heat: A new approach to computing distance based on heat flow. ACM Trans. Graph. (TOG), 32(5):152:1-152:11 oct 2013.
[8] G. Csurka, C. Dance, L. Fan, Willamowski, and C. Bray. Visual categorization with bags of keypoints. In Workshop on statistical learning in computer vision, ECCV, volume 1, pages 1-2. Prague, 2004.
[9] J. Dai, H. Qi, Y. Xiong, Y. Li G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks. In IEEE International Conference on Computer Vision (ICCV), 2017.
[10] D. DeTone, T. Malisiewicz, and A. Rabinovich. Superpoint: supersupe interest point detection and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 224-236, 2018
[11] G. Donato and S. Belongie. Approximate thin plate spline mappings. In European conference on computer vision, pages 21-31. Springer. 2002.
[12] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, Sivic, Torii, and T. Sattler. D2-net: A trainable enn for joint description and detection of local features. In Proceedings of the ieeetcvf conference on computer vision and pattern recognition, pages 8092-8101, 2019.
[13] Ebel, A. Mishchuk, K. M. Yi, P. Fua, and E. Trulls. Beyond cartesian representations for local descriptors. In Proceedings of the IEEE International Conference Computer Vision, pages 253-262, 2019.
[14] H. Guan and W. A. Smith. Brisks: Binary features for spherical images on a geodesic grid. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4516 4524, 2017.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
[16] M. Jaderberg, K. Simonyan, A. Zisserman, et Spatial transformer networks. Advances in neural information processing systems, 28:2017-2025, 2015.

---

[17] Y. Jin, D. Mishkin, A. Mishchuk, J. Matas, P. Fua, K. M. Yi, and E. Trulls. Image matching across wide baselines: From paper to practice. International Journal of Computer Vision, pages 1-31, 2020.
[18] S. Kumar, A. Cherian, Y. Dai, and H. Li. Scalable dense non-rigid muucture-frommmee A grassmannian perspective. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
[19] D. G. Lowe. Distinetive image features from ocale-invariant keypoints. International Journal of Computer Vision, pages 91-110, 2004.
[20] Z. Luo, T. Shen, L. Zhou, S. Zhu, R. Zhang, Y. Yao, T. Fang, and L.. Quan. Geodesc: Learning local descriptors by integrating geometry constraints. In Proceedings of the European conference on computer vision (ECCV), pages 168-183, 2018.
[21] Z. Luo, L. Zhou, X. Bai, H. Chen, J. Zhang, Y. Yao, S. Li, T. Fang, and L. Quan. Aslfeat: Learning local features of accurate shape and localization. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 6589-6598, 2020.
[22] K. Mikolajczyk. T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and L. Van Gool. A comparison of affine region detectors. International journal of computer vision, 65(1-2):43-72, 2005.
[23] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. In Advances in Neural Information Processing Systems, pages 4826-4837, 2017.
[24] E. R. Nascimento, G. L. Oliveira, M. F. M. Campos, A. W. Vieira, and W. R. Schwartz. BRAND: A robust appearance and depth descriptor for RGB-D images. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pages 1720-1726. IEEE, 2012.
[25] E. R. Nascimento, G. Potje, R. Martins, F. Cadar, M. F. M. Campos, and R. Bajesy. GEOBIT: geodesic-based binary descriptor invariant to non-rigid deformations for RGB-D images. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10003-10011, 2019.
[26] H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han. Large-scale image retrieval with attentive deep local features. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 3476-3485, 2017.
[27] Y. Ono, E. Trulls, P. Fua, and K. M. Yi. Lf-net: learning local features from images. In Advances in neural information processing systems, pages 6234 6244, 2018.
[28) A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, 7. Lin, A. Desmaison,
L. Antiga, and A. Lerer. Automatic differentiation in pytorch. In NeurIPS Workshops, 2017.
[29] J. Pilet, V. Lepetit, and P. Fua. Fast non-rigid surface detection, registration and realistic augmentation. International Journal of Computer Vision, 76(2):109-123 122, 2008.
[30] G. A. Potje, G. Resende, M. F. M. Campos, and E. R. Nascimento. Towards efficient 3D model estimation methodology for aerial and ground images. Machine Vision and Applications, 28(8):937-52,, 2017.
[31] A. Pumarola, A. Agudo, L. Porzi, A. Sanfeliu, V. Lepetit, and F. Moreno-Noguer. Geometry- Aware Network for Non-Rigid Shape Prediction from a Single View. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[32] Revaud, C. De Souza, M. Humenberger, and P. Weinzaepfel. R2D2: Reliable and repeatable detector and descriptor. In Advances in Neural Information Processing Systems, volume 32, pages -1415, 2019.
[33] I. Rocco, R. Arandjelovic, and Sivie. Convolutional neural network architecture for geometric matching. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
12

---

[34] I. Rocco, R. Arandjelović, and J. Sivic. End-to-end weakly- semantic alignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[35] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. ORB: an efficient alternative to SIFT or SURF, In Proc. ICCV, Barcelona, 2011.
[36] P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymezyk. From coarse fine: Robust hierarchical localization at large scale. In Proc. of IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2019.
[37] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2020.
[38] E. Simo-Serra, C. Torras, and Moreno-Noguer. DaL.: deformation and light invariant descriptor. International Journal of Computer Vision, 115(2), 2015.
[39] M. Teichmann, A. Araujo, M. Zhu, and J. Sim. Detect-to-retrieve: Efficient regional aggregation for image search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5109-5118 5119,,111220111 2019.
[40] Y. Tian, B. Fan, and F. Wu. L2-net: Deep learning of discriminative patch descriptor in euclidean space. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 661-669, 2017.
[41] Y. Tian, X. Yu, B. Fan, F. Wu, H. Heijnen, and V. Balntas. Sosnet: Second order similarity regularization for local descriptor learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11016-1 1005,, 2019.
[42] E. Tola, V. Lepetit, and Fua, Daisy: An efficient dense descriptor applied to wide baseline stereo. IEEE Transactions Pattern Analysis and Machine Intelligence, 32(5):815-88) 5300, 2010.
[43] Q.-H. Tran, T.-J. Chin, G. Carneiro, M. S. Brown, and D. Suter. In defence of ransac for outlier rejection in deformable registration. In ECCV, 2012.
[44] D. P. Vassileios Balntas, Edgar Riba and K. Mikolajezyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016
[45] T. Wang, H. Ling, C. Lang, S. Feng, and X. Hou. Deformable surface tracking by graph matching. In IEEE International Conference on Computer Vision, 2019.
[46] K. Wilson and N. Snavely. Robust global translations with Idsfm. In European Conference on Computer Vision, pages 61-75. Springer, 2014.
[47] K. M. Yi, E. Trulls, V. Lepetit, and P. Fua. LIFT: Learned invariant feature transform. In ECCV, 2016.
[48] K. M. Yi, F. Trulls, V. Lepetit, and P. Fua. Lift: Learned invariant feature transform. In European Conference on Computer Vision, pages 467-483. Springer, 2016.
[49] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[50] Q. Zhao, W. Feng, L. Wan, and J. Zhang. Sphorb: A fast and robust binary feature on the sphere. International journal of computer vision, 133(2):443-159, 2015.
13


Reviews:
[{"review_id": "ejsKNQmW3cO", "reviewer": "Reviewer_dECi", "summary": "This paper presents a learnt local keypoint descriptor for deformable objects. It is trained in a supervised fashion using the HardNet network and loss on the SIFT keypoints of a synthetic dataset (succinctly presented in the paper, based on existing SfM datasets). The novelty is the addition of a spatial transformer network (STN) to normalize the patches, inline with recent local descriptor learning approaches. As far as I understand the main conceptual difference with existing works is that the (STN) is not only affine but uses a TPS on a log-polar grid. My understanding of the paper is that it argues this part is crucial for matching deformable objects.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty/contributions/ablation:\nWhile the proposed approach makes sense, I do not really see any novelty/important difference compared to standard learned keypoint descriptors except (i) that the deformation learned by the transformer network includes a TPS deformation on a log-polar grid (and not only an affine one) (ii) the learning dataset seem to be an \"in house\" dataset (on which I would appreciate much more details and which I would like the authors to confirm they will release with their code) with a HardNet approach. It is unclear to me what is the influence of each on the results. There is only an embryo of ablation study (Table 3, two numbers on an unspecified handpicked small subset of the data) where the full transformer network is ablated. I would like to also see **an ablation of only the TPS part, and a TPS on a standard regular grid**, to also have access to the corresponding results on the full dataset, and to have some analysis of the effect of the training dataset (e.g. same data generation process but without the non-rigid deformations, size). More generally, I would also like to see ablation that replace different parts of the proposed pipeline (loss, network archi, training data) with those of a recent learned feature descriptor.\n\nComparisons to other approaches:\nMy main concern is that the results are reported on two recent (2019) datasets that have been used by very few other works (5 and 2 citations), one thus has to rely on the paper itself to run relevant baselines. While there is a clear effort in this direction in Table 2 and 4, there are some baselines that seem important to me and would be worth adding a few: SIFT, LIFT, superpoint, D2-Net, R2D2 and maybe other recent learnt descriptors. Note that a single number on R2D2 is reported in 4.2, but I think this is not really sufficient and it should be compared with other methods (even if it makes the table more complicated because they can be used either with SIFT keypoints or with their own keypoints) Conversely, I would like to see the performance of the proposed descriptor on standard keypoint matching benchmarks (potentially for downstream tasks such as calibration). While I understand the advantage of the proposed keypoints is to handle deformable objects, they should also be able to handle rigid object, and showing they give competitive results on standard benchmark would reassure me that the results reported in the paper are indeed impressive and not only due to comparison bias.\n\nIn conclusion, there are too many things in this paper I am not confident about and that I would like to see added to recommend acceptance. I see little demonstrated important technical and conceptual novelty and I don't think improving SoA on seldom used datasets is itself sufficient for a publication.\n\nMinor: some writing issue, e.g. last sentence of the first paragraph \"After all, we live in a non-rigid world, where beyond rigid surfaces, it is also populated ...'\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": ".discussion of limitations and failure cases missing", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "DKIvGMOhVh", "reviewer": "Reviewer_1fX6", "summary": "The paper introduces a new deep learning pipeline for the computation of local image features that are robust to non-rigid deformations. The proposed method is end-to-end, and works by simulating non-rigid transformations on a synthetic dataset to jointly learn how to rectify the deformed patches, while extracting discriminative and invariant features at the same time. Comparisons with several state of the art pipelines on multiple datasets confirm the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading the paper, which is well written, fluent, and easily accessible for someone not fully familiar with this family of methods (such as this reviewer). The method is described well and the computational modeling is sound. The underlying idea of explicitly parametrizing deformations via a learnable TPS model is straightforward and well executed, and I believe there is value in this simplicity, given the good results it enables in several practical cases.\n\nWorth mentioning is the extensive experimental evaluation, which includes several recent datasets comprised of real-world deformable objects (despite the training being based on synthetic examples) undergoing diverse picture-space transformations (e.g. illumination, viewpoint, object deformation), as well as comparisons with several baselines and recent learning-based methods. Hyperparameter sensitivity, ablation, and runtime studies are also included. Two real-world applications are analyzed, demonstrating competitive results if compared to other recent methods.\n\nAs a minor comment, I found the results reported in Table 3 (Ablation) a bit difficult to interpret: if the NRW component is key to learning the unwrap, why is the improvement so mild over vanilla PTN (+0.04 mean matching accuracy)? The paper refers to this as significant improvement, but more comments on this being a true gain would be useful.\n\nOverall, I believe this is a solid paper that deserves to be communicated. At the same time, I have some reservations on this being a computer vision paper first and foremost, with machine learning just being a convenient ingredient to compose the full pipeline. In other words, I think the key contribution is in the vision area, while it might be not very significant for ML. Therefore I am not 100% confident NeurIPS is the right venue for it, but I won't oppose acceptance based solely on this doubt.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are briefly discussed in the conclusions, and failure cases are included in the supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "D6RnOQbbWnF", "reviewer": "Reviewer_fuQg", "summary": "This paper proposes to learn deformation-aware local feature descriptors by rectifying the local region with carefully designed spatial transformer networks (STNs). Instead of using 3D training data, the paper proposes to learn from deformed 2D samples. The feature extraction network utilizes two STNs to rectify the local region and HardNet to extract the final feature. Among the two STNs, the second STN is specially designed with the thin-plate-spline transform and a polar grid so that the resulting region flexibly deform from a circular shape. The experimental results show that better or competitive results with much less processing time.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+- Not highly novel, but practically beneficial: Using spatial transformers and deep learning to learn robust features is not exactly new. I think that the main contributions are in designing it for general deformation, and also in the detailed design (a simple but specially designed deformation module with two STNs) of the proposed method. On the other hand, the method has many practical benefits: no requirement for 3D training data, fast processing time, and good performance.\n\n-- Missing reference: There is a highly related work that uses STN for learning local features [Ono et al., NeurIPS 2018], which is missing in the current paper. This has to be mentioned and compared with the proposed method.\n\n-- Why do we need two STNs?: I said that one of the main contributions is the module with two STNs. But why do we need this? Can't we just use the second STN only? What is the exact role of the first (plain) STN? The reason for such a design is not well explained in the paper. I strongly suggest providing (or analyzing) this reason, which can be crucial in lifting the value of the paper.\n\n+- Using only 2D data: Although I mentioned that requiring only 2D data is a benefit, there is an issue to discuss here. I see that the proposed network can also be trained with RGB-D data with no problem. (In this case, moreover, we might also add some more supervisions, say, something like deformation supervision.) What happens if we do that? For example, the proposed method and GeoBit are competitive in performance in the experiments. Do things get any different with RGB-D data?\n\n[After rebuttal]\nMost of my concern has been addressed in the authors' response. I believe that this is a sound contribution in the field. I raise my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are described in the final section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1GT4gUcewne", "reviewer": "Reviewer_P78p", "summary": "The paper proposes DEAL, an RGB patch descriptor that has a learned robustness against non-rigid deformations. Given a SIFT keypoint, surrounding ResNet features are used to estimate a non-rigid thin-plate spline deformation of the patch. The RGB patch, after being rectified by the thin-plate spline, is passed to a CNN that regresses a 128-dim descriptor. The network is trained via a hard triplet loss. The training data is simulated and uses groundtruth correspondences/matches, but results are shown on both real and synthetic data. Experiments compare to a number of classic and learned baselines. DEAL outperforms other methods in the main experiment and shows best or second-best performance on a couple of applications.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is a novel descriptor that tackles non-rigid deformations, which is a very desirable goal. The experiments on real data show promising results, especially considering that the descriptor was trained on only synthetic data. My main concern is a better attribution of where the improvement comes from: data or architecture (thin plate splines)? I find the current experiments unsatisfying in this regard, the next paragraph goes into details.\n\nThe \"Contribution of the TPS warper\" ablation study is interesting. My understanding is that \"(i)\"/PTN is a pipeline that is the same (up to maybe number of conv layers etc.) as Fig. 3 in Ebel et al.'s Log-Polar descriptor [12]: sample the original image with a circular polar grid around the keypoint and then feed that patch through a CNN to get the descriptor. Table 2 shows an MMA of 0.49 for Log-Polar on the Simulation dataset, while DEAL has 0.80. The ablation study shows 0.79 for DEAL (i.e. with NRW) and 0.75 for the PTN ablation. The ablation uses a specific scene from the Simulation dataset, which explains the differing performance of DEAL. I interpret the discrepancy between PTN ablation (0.75) and Log-Polar (0.49) to mean that most of the improvement that DEAL shows over Log-Polar is due to training data (from 0.49 to 0.75) rather than architecture (from 0.75 to 0.79). Is that correct? If so, I believe that further baseline experiments for Table 2 would be necessary that train the two learning baselines (TFeat and Log-Polar, lines 218-219) with the proposed *non-rigid* simulated dataset (lines 229-239) instead of their original datasets, which seem to be *rigid*.\n\nI have a few bigger questions about the paper, but which I do not expect to sway the acceptance decision:\n\n- Limitations should be discussed more, including qualitative examples.\n- What is the intuition behind using the first spatial transformer? What would happen if it isn't used and Y = X? Such an ablation experiment should be added.\n- The sensibility analysis in Table 1 shows that increasing the margin to 0.75 and increasing the STN output to 5 x 5 improves over the baseline. Why is only the latter change adapted (lines 266-267)? The margin change gives 50% as much improvement as changing the STN output does after all.\n- The STN is meant to normalize out some spatial transforms around the keypoint. But how can the localization network then determine the right (desirable) rectifying transformation for the original patch around the keypoint if the localization network doesn't have access to the uncorrected spatial layout of the original patch? I.e. the STN removes some information about the spatial layout of the patch, then the TPS transform is estimated from that reduced information, but then the TPS tranform is applied to the original patch. Does such a setup make sense?\n- Is there a reason that the DeepDeform dataset [4] was not used for evaluation? Is it not suited for the task?\n\nThere are a number of smaller clarifications that I suggest a revised version should take into account:\n\n- What is the size of the receptive field of the ResNet features after the sixth convolutional block (line 134)?\n- In lines 156-158, it would be good to mention shortly that the parameters theta get *regressed* by the spatial transformer from X. It is not fully clear whether theta is the same globally for all H/s x W/s pixels or whether it differs between them.\n- The MLP from line 178 is not convolutional, so are the 5 x 5 128-dim feature vectors stacked before passing them to the MLP?\n- Does T_theta in line 159 refer to the STN in Fig. 2? If so, the notation for T_theta(G_I) in Fig. 2 should be changed to avoid confusion.\n- It would also be good to add the missing notations to Fig. 2, e.g. the TPS parameter regressor S (line 152), M_theta and M_I (line 159), the warped features Y (line 161).\n- What does the \"TPS\" in the green box in Fig. 2 stand for? Doesn't the TPS regressor S regress theta_tps directly? \n- I don't understand the sentence in lines 186-188. Why is the regular (not the TPS-transformed) polar grid applied to the image (not just the patch)? What are the keypoint's attributes, where do they come from, have they been mentioned before? Why are they needed to get an identity transform? What is the difference between scale and size?\n- The sentence in lines 196-198 is a bit ambiguous and could be split into two shorter sentences, and the sentence in lines 198-199 could be pulled in front. \"which\" could refer to the loss or to HardNet. It is not obvious that HardNet comes from [22], only that the triplet loss is used from [22].\n- A short explanation of what needed to change for adapting to R2D2 and ASLFeat (lines 282-283) in the appendix would be good for completeness.\n\nUpdate after rebuttal ---\n\nI initially had a similar difficulty in understanding the usages of two STNs as reviewer fuQg, which made me question the rationale behind the method design and potential issues with it. Given the authors' clarifications, I corrected my understanding (the first STN uses a regular, straight grid) and am now satisfied with the design. Similarly, I was concerned about baselines (maybe training data is the crucial difference), which the new experiments in the rebuttal show to not be the case. Reviewer dECi brought up some more important ablations, where I find the results provided in the authors' response convincing. I find the remaining points mentioned by the reviewers to be addressed well by the authors' response.\n\nHowever, there is one part that is not fully convincing me, namely comparison to existing work, as mentioned by reviewer dECi. The lack of non-rigid datasets in the literature suitable for evaluation is unfortunate, but the authors make an effort to introduce their own dataset and to evaluate on the couple of datasets that are suitable. The performance on rigid datasets is competitive, but not state-of-the-art, which I find good enough, given that the submission has to relax assumptions to be able to handle non-rigid deformations.\n\nI will change my rating from 5 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are mentioned in one sentence (lines 371-372) but not discussed to give a good sense of them, and I am unable to locate the limitations that are supposed to be in the appendix. Qualitative results could be added.\n\nPotential negative societal impact is not discussed. Better tracking of clothed people (which have non-rigid deformations) through surveillance might be one point.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to learn deformation-aware local feature descriptors using spatial transformer networks (STNs).  The reviewers found the paper clear, the method practically useful and the experimental comparison to be comprehensive with the results quite strong.    I agree with this assessment.  I also appreciate the robust and healthy discussion between the reviewers and the authors that I think helped point in directions that make the paper better.
Decision: Accept (Poster)