Extracting Deformation-Aware Local Features by Learning to Deform

Despite the advances in extracting local features achieved by handcrafted and learning-based descriptors , they are still limited by the lack of invariance to nonrigid transformations . In this paper , we present a new approach to compute features from still images that are robust to non-rigid deformations to circumvent the problem of matching deformable surfaces and objects . Our deformation-aware local descriptor , named DEAL , leverages a polar sampling and a spatial transformer warping to provide invariance to rotation , scale , and image deformations . We train the model architecture end-to-end by applying isometric non-rigid deformations to objects in a simulated environment as guidance to provide highly discriminative local features . The experiments show that our method outperforms state-of-the-art handcrafted , learning-based image , and RGB-D descriptors in different datasets with both real and realistic synthetic deformable objects in still images . The source code and trained model of the descriptor are publicly available at https : //www.verlab.dcc.ufmg.br/descriptors/neurips2021 . 1 Introduction . High-quality matching of images taken from cameras in different poses and conditions remains one of the key challenges in several tasks . Tracking points as a camera or an object of interest is moving is crucial for finding and reconstructing objects and scenes [ 29 , 30 , 31 ] , camera localization and mapping [ 18 , 36 ] , registration [ 43 , 37 ] , and image retrieval [ 26 , 39 ] , to name a few tasks . Extracting invariant and compact representations to describe the vicinity of a pixel has been one of the key forces in enabling high-quality matching on these tasks . Despite the importance of invariance to varying illumination , viewpoint , and the distance to the object of interest , real-world scenes impose additional challenges . After all , we live in a non-rigid world populated by objects that may have different shapes over time due to deformations . Over the past few decades , many descriptors have been proposed [ 19 , 3 , 6 , 35 , 42 , 47 , 44 , 40 , 23 , 13 ] . These approaches can be roughly grouped based on the type of input information , such as intensity and/or depth images , and can be further divided into handcrafted and learning-based methods . While local learning-based approaches , such as Log-Polar [ 13 ] , hold state of the art regarding affine transformations and local illumination changes , recent handcrafted techniques achieve better performance in the presence of non-rigid transformations by using depth data [ 25 ] or by modeling the deformations from image intensity information , which inevitably leads to an increase of the computational cost to compute the description [ 38 ] . Since depth data is not available in many contexts and applications , and low processing time is highly ∗Department of Computer Science , Universidade Federal de Minas Gerais , Brazil . VIBOT EMR CNRS 6000 , ImViA , Université Bourgogne Franche-Comté , France . Corresponding author ’ s e-mail : guipotje @ dcc.ufmg.br 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . recommended for low-level processing such as the estimation of local features , devising a descriptor that correctly and reliably establishes invariant features from corresponding points is of central importance to high-quality matching of images in real-world scenarios . In this paper , we present a new end-to-end method to extract local features from still images that are robust to non-rigid deformations . Conversely to recent approaches that exploit a single network branch to extract features from local patches or depth data , our approach is composed of an additional guidance branch component ( spatial transformer non-rigid warper ) that jointly learns context about the global deformation affecting the image . Figure 1 depicts the rationale behind our method . For learning the contextual deformation information , we provide several views of deformed objects by applying isometric non-rigid transformations to the objects in a simulated environment as guidance to extract highly discriminative local deformation-aware features . Unlike non-rigid aware deformations descriptors like GeoBit [ 25 ] , our approach does not require depth data to produce descriptors robust to nonrigid deformations and it is notably faster than DaLI [ 38 ] while holding the best performance . Although many descriptors still disregard nonrigid transformations , in recent years , that assumption has been dispelled , partly due to the increasing popularity in the use of RGB-D sensors to capture properties that characterize 3D objects such as geodesic distances . On the flip side , RGB-D sensors are still less prevalent than RGB cameras . In our nonrigid-world with flexible humans , animals , tissues , and leaves where high-resolution RGB cameras are ubiquitous , the ability to create discriminative and deformation-invariant descriptors using a single picture plays a central role in achieving high- quality matching in real-world scenarios . In this work , we demonstrate that our end-to-end architecture outperforms state-of-the-art handcrafted and learning-based image and RGB-D descriptors in matching scores on different benchmark datasets of real deformable objects , as well as with realistic application scenarios on content-based object retrieval and tracking . 2 Related work . Handcrafted and learning-based feature extraction . Traditionally , image descriptors are aimed to providing rich and invariant representations of objects observed from different viewing conditions . Two notable handcrafted approaches to describe keypoints on still images are SIFT [ 19 ] and ORB [ 35 ] . These methods use local gradients and dominant orientation estimation to extract visual features with scale and rotation invariance . Different extensions of these descriptors have been proposed to images in some particular manifolds such as BRISKS [ 14 ] and SphORB [ 50 ] for spherical images . Recently the advent of convolutional neural networks boosted the development of several end-to-end learned descriptors [ 47 , 44 , 40 , 23 , 13 , 20 , 32 ] . These descriptors often outperform handcrafted counterpart on classical image matching benchmarks such as UBC Phototour [ 5 ] and HPatches [ 2 ] . Yet , the vast majority of existing local descriptors are approximately invariant to affine image transformations , often disregarding images of deformable surfaces , which are the main feature of our descriptor . Deformation-aware reconstruction and matching . Although most works on local descriptors are designed focusing on affine image transformations , some works considered non-rigid surfaces and deformable objects . In DeepDeform [ 4 ] , for instance , the authors propose to estimate non-rigid point-wise RGB-D correspondences by a data-driven approach that is then used by a non-rigid 3D reconstruction framework . Although their method works well for fast motions and non-rigid surface tracking , it has a heavy computational burden for establishing correspondences for typical amounts of keypoints from commonly used detectors . Also , the method does not generate a descriptor that can be used for other related vision tasks . Simo-Serra et al . proposed the DaLI descriptor [ 38 ] , where image patches are converted to a 3D surface in order to encode features robust to non-rigid deformations and illumination changes . Despite achieving high matching performance , DaLI suffers from high computational and storage requirements . Recently , DeformNet [ 31 ] was designed to infer non-rigid shapes from still images . The network architecture is composed of a 2D detection branch and a depth branch . The detection and depth branches are subsequently merged by the shape branch , which uses a pinhole camera model to re-project the shape to 3D . In the same context , GeoBit [ 25 ] takes advantage of geodesics from objects surface to compute isometric-invariant features . The method uses RGB-D data combined with the Heatflow [ 7 ] method to encode visual features in a binary string that are invariant to deformations . However , GeoBit requires accurate depth information and only works with RGB-D images . Our method , in its turn , only requires monocular images , is faster to compute , and provides a better matching performance , as demonstrated in our experiments . Spatial attention mechanisms and matching . Spatial transformer networks ( STNs ) , introduced by the seminal work of Jaderberg et al . [ 16 ] , allowed differentiable warping operations to be used directly with existing network architectures . The core idea is the learning of an attention mechanism ( named Localization network ) that is parameterized to represent the image transformation of interest , e.g. , affine , homography or a thin-plate-spline warp . They have been applied to several tasks including image classification [ 16 ] , semantic alignment [ 34 ] , geometric matching [ 33 ] and local descriptors [ 13 ] . LF-Net [ 27 ] , a detect-and-describe method , first locates interest points using a fully convolutional network , which are then cropped in patches with a STN layer considering an affine transformation encoded by keypoint attributes . The cropped patches are subsequently used to compute local descriptors by the description network . Spatial attention mechanisms have also been extended into more generic settings via deformable kernels [ 9 ] , to learn kernel offsets in network layers operators to define deformable convolution networks . More recently , the end-to-end joint detector and descriptor ASLFeat [ 21 ] used deformable convolutions to increase the network ’ s expressiveness , which shares some concepts with our method . However , the deformable kernels employed by ASLFeat considers high-level feature maps , and it does not model deformations explicitly . Its strategy results in earlier layers not being aware of low level image deformations , usually present in geometric transformations . Moreover , the method is not robust to rotation changes . Our method leverages two carefully designed spatial transformer networks to sample and guide the model to provide invariance to deformations and , to the best of our knowledge , is the first end-to-end learning method to explicitly tackle non-rigid deformations with a carefully designed spatial attention mechanism . 3 Deformation-aware network architecture . Our DEformation-Aware Local descriptor ( DEAL ) has been designed to jointly learn to undeform and extract discriminative and invariant features from local regions . The proposed architecture is able to handle deformations and perspective distortions of sampled patches in the vicinity of the keypoint . It also attenuates noise in keypoint attributes which improves the local description performance and robustness to image transformations . We designed our network as a hybrid approach , in the sense that it first extracts dense features and the final output is a local descriptor for each keypoint , unlike both trends of describing local patches like HardNet [ 23 ] or extracting dense features jointly with keypoints such as SuperPoint [ 10 ] and R2D2 [ 32 ] . Our approach combines the best of these two trends : i ) the robustness to occlusion , strong illumination , and perspective changes of local patch-based descriptors that work directly with low-level image information ; and ii ) the capability of dense methods in encoding local semantics from the image to disambiguate hard pairs . Additionally , our model decouples the problem of learning distinctive descriptors robust to deformations in two complementary tasks , i.e. , first estimating a warp with a spatial attention mechanism and then describing the sampled local patches using the estimated non-rigid warp . Figure 2 outlines a schematic representation of the model . 3.1 Mid-level feature extraction . For the extraction of mid-level image features that encode local deformation information , we employ the ResNet-34 [ 15 ] and truncate it at the sixth convolutional block . The motivations to process features from ResNet instead of directly sampling local patches over the image are twofold : First , the mid-level feature maps provided by ResNet features allow the network to learn to encode local deformations from contextual cues such as local texture statistics , illumination and viewpoint changes ; Secondly , the receptive field of our local feature descriptor is not tied to a fixed support region size and sampling pattern like traditional patch-based local descriptors . Strong perspective and deformation changes drastically influence the optimal support region sizes for the same keypoint on different images ; thus , we present an architecture that is aware of these local deformations by design , since the dense feature map allows such information to flow to the non-rigid warper module . The ResNet-34 head outputs a feature map X ∈ RH/8×W/8×128 by performing convolutions until the spatial resolution achieves 1/8 of the original input resolution . The feature map X is used as input for the the non-rigid warper component , which computes the Thin-Plate Splines ( TPS ) parameters with the spatial transformer to model a non-rigid warp .

Summary: This paper proposes to learn deformation-aware local feature descriptors by rectifying the local region with carefully designed spatial transformer networks (STNs). Instead of using 3D training data, the paper proposes to learn from deformed 2D samples. The feature extraction network utilizes two STNs to rectify the local region and HardNet to extract the final feature. Among the two STNs, the second STN is specially designed with the thin-plate-spline transform and a polar grid so that the resulting region flexibly deform from a circular shape. The experimental results show that better or competitive results with much less processing time.