Extracting Deformation-Aware Local Features by Learning to Deform

Despite the advances in extracting local features achieved by handcrafted and learning-based descriptors , they are still limited by the lack of invariance to nonrigid transformations . In this paper , we present a new approach to compute features from still images that are robust to non-rigid deformations to circumvent the problem of matching deformable surfaces and objects . Our deformation-aware local descriptor , named DEAL , leverages a polar sampling and a spatial transformer warping to provide invariance to rotation , scale , and image deformations . We train the model architecture end-to-end by applying isometric non-rigid deformations to objects in a simulated environment as guidance to provide highly discriminative local features . The experiments show that our method outperforms state-of-the-art handcrafted , learning-based image , and RGB-D descriptors in different datasets with both real and realistic synthetic deformable objects in still images . The source code and trained model of the descriptor are publicly available at https : //www.verlab.dcc.ufmg.br/descriptors/neurips2021 . 1 Introduction . High-quality matching of images taken from cameras in different poses and conditions remains one of the key challenges in several tasks . Tracking points as a camera or an object of interest is moving is crucial for finding and reconstructing objects and scenes [ 29 , 30 , 31 ] , camera localization and mapping [ 18 , 36 ] , registration [ 43 , 37 ] , and image retrieval [ 26 , 39 ] , to name a few tasks . Extracting invariant and compact representations to describe the vicinity of a pixel has been one of the key forces in enabling high-quality matching on these tasks . Despite the importance of invariance to varying illumination , viewpoint , and the distance to the object of interest , real-world scenes impose additional challenges . After all , we live in a non-rigid world populated by objects that may have different shapes over time due to deformations . Over the past few decades , many descriptors have been proposed [ 19 , 3 , 6 , 35 , 42 , 47 , 44 , 40 , 23 , 13 ] . These approaches can be roughly grouped based on the type of input information , such as intensity and/or depth images , and can be further divided into handcrafted and learning-based methods . While local learning-based approaches , such as Log-Polar [ 13 ] , hold state of the art regarding affine transformations and local illumination changes , recent handcrafted techniques achieve better performance in the presence of non-rigid transformations by using depth data [ 25 ] or by modeling the deformations from image intensity information , which inevitably leads to an increase of the computational cost to compute the description [ 38 ] . Since depth data is not available in many contexts and applications , and low processing time is highly ∗Department of Computer Science , Universidade Federal de Minas Gerais , Brazil . VIBOT EMR CNRS 6000 , ImViA , Université Bourgogne Franche-Comté , France . Corresponding author ’ s e-mail : guipotje @ dcc.ufmg.br 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . recommended for low-level processing such as the estimation of local features , devising a descriptor that correctly and reliably establishes invariant features from corresponding points is of central importance to high-quality matching of images in real-world scenarios . In this paper , we present a new end-to-end method to extract local features from still images that are robust to non-rigid deformations . Conversely to recent approaches that exploit a single network branch to extract features from local patches or depth data , our approach is composed of an additional guidance branch component ( spatial transformer non-rigid warper ) that jointly learns context about the global deformation affecting the image . Figure 1 depicts the rationale behind our method . For learning the contextual deformation information , we provide several views of deformed objects by applying isometric non-rigid transformations to the objects in a simulated environment as guidance to extract highly discriminative local deformation-aware features . Unlike non-rigid aware deformations descriptors like GeoBit [ 25 ] , our approach does not require depth data to produce descriptors robust to nonrigid deformations and it is notably faster than DaLI [ 38 ] while holding the best performance . Although many descriptors still disregard nonrigid transformations , in recent years , that assumption has been dispelled , partly due to the increasing popularity in the use of RGB-D sensors to capture properties that characterize 3D objects such as geodesic distances . On the flip side , RGB-D sensors are still less prevalent than RGB cameras . In our nonrigid-world with flexible humans , animals , tissues , and leaves where high-resolution RGB cameras are ubiquitous , the ability to create discriminative and deformation-invariant descriptors using a single picture plays a central role in achieving high- quality matching in real-world scenarios . In this work , we demonstrate that our end-to-end architecture outperforms state-of-the-art handcrafted and learning-based image and RGB-D descriptors in matching scores on different benchmark datasets of real deformable objects , as well as with realistic application scenarios on content-based object retrieval and tracking . 2 Related work . Handcrafted and learning-based feature extraction . Traditionally , image descriptors are aimed to providing rich and invariant representations of objects observed from different viewing conditions . Two notable handcrafted approaches to describe keypoints on still images are SIFT [ 19 ] and ORB [ 35 ] . These methods use local gradients and dominant orientation estimation to extract visual features with scale and rotation invariance . Different extensions of these descriptors have been proposed to images in some particular manifolds such as BRISKS [ 14 ] and SphORB [ 50 ] for spherical images . Recently the advent of convolutional neural networks boosted the development of several end-to-end learned descriptors [ 47 , 44 , 40 , 23 , 13 , 20 , 32 ] . These descriptors often outperform handcrafted counterpart on classical image matching benchmarks such as UBC Phototour [ 5 ] and HPatches [ 2 ] . Yet , the vast majority of existing local descriptors are approximately invariant to affine image transformations , often disregarding images of deformable surfaces , which are the main feature of our descriptor . Deformation-aware reconstruction and matching . Although most works on local descriptors are designed focusing on affine image transformations , some works considered non-rigid surfaces and deformable objects . In DeepDeform [ 4 ] , for instance , the authors propose to estimate non-rigid point-wise RGB-D correspondences by a data-driven approach that is then used by a non-rigid 3D reconstruction framework . Although their method works well for fast motions and non-rigid surface tracking , it has a heavy computational burden for establishing correspondences for typical amounts of keypoints from commonly used detectors . Also , the method does not generate a descriptor that can be used for other related vision tasks . Simo-Serra et al . proposed the DaLI descriptor [ 38 ] , where image patches are converted to a 3D surface in order to encode features robust to non-rigid deformations and illumination changes . Despite achieving high matching performance , DaLI suffers from high computational and storage requirements . Recently , DeformNet [ 31 ] was designed to infer non-rigid shapes from still images . The network architecture is composed of a 2D detection branch and a depth branch . The detection and depth branches are subsequently merged by the shape branch , which uses a pinhole camera model to re-project the shape to 3D . In the same context , GeoBit [ 25 ] takes advantage of geodesics from objects surface to compute isometric-invariant features . The method uses RGB-D data combined with the Heatflow [ 7 ] method to encode visual features in a binary string that are invariant to deformations . However , GeoBit requires accurate depth information and only works with RGB-D images . Our method , in its turn , only requires monocular images , is faster to compute , and provides a better matching performance , as demonstrated in our experiments . Spatial attention mechanisms and matching . Spatial transformer networks ( STNs ) , introduced by the seminal work of Jaderberg et al . [ 16 ] , allowed differentiable warping operations to be used directly with existing network architectures . The core idea is the learning of an attention mechanism ( named Localization network ) that is parameterized to represent the image transformation of interest , e.g. , affine , homography or a thin-plate-spline warp . They have been applied to several tasks including image classification [ 16 ] , semantic alignment [ 34 ] , geometric matching [ 33 ] and local descriptors [ 13 ] . LF-Net [ 27 ] , a detect-and-describe method , first locates interest points using a fully convolutional network , which are then cropped in patches with a STN layer considering an affine transformation encoded by keypoint attributes . The cropped patches are subsequently used to compute local descriptors by the description network . Spatial attention mechanisms have also been extended into more generic settings via deformable kernels [ 9 ] , to learn kernel offsets in network layers operators to define deformable convolution networks . More recently , the end-to-end joint detector and descriptor ASLFeat [ 21 ] used deformable convolutions to increase the network ’ s expressiveness , which shares some concepts with our method . However , the deformable kernels employed by ASLFeat considers high-level feature maps , and it does not model deformations explicitly . Its strategy results in earlier layers not being aware of low level image deformations , usually present in geometric transformations . Moreover , the method is not robust to rotation changes . Our method leverages two carefully designed spatial transformer networks to sample and guide the model to provide invariance to deformations and , to the best of our knowledge , is the first end-to-end learning method to explicitly tackle non-rigid deformations with a carefully designed spatial attention mechanism . 3 Deformation-aware network architecture . Our DEformation-Aware Local descriptor ( DEAL ) has been designed to jointly learn to undeform and extract discriminative and invariant features from local regions . The proposed architecture is able to handle deformations and perspective distortions of sampled patches in the vicinity of the keypoint . It also attenuates noise in keypoint attributes which improves the local description performance and robustness to image transformations . We designed our network as a hybrid approach , in the sense that it first extracts dense features and the final output is a local descriptor for each keypoint , unlike both trends of describing local patches like HardNet [ 23 ] or extracting dense features jointly with keypoints such as SuperPoint [ 10 ] and R2D2 [ 32 ] . Our approach combines the best of these two trends : i ) the robustness to occlusion , strong illumination , and perspective changes of local patch-based descriptors that work directly with low-level image information ; and ii ) the capability of dense methods in encoding local semantics from the image to disambiguate hard pairs . Additionally , our model decouples the problem of learning distinctive descriptors robust to deformations in two complementary tasks , i.e. , first estimating a warp with a spatial attention mechanism and then describing the sampled local patches using the estimated non-rigid warp . Figure 2 outlines a schematic representation of the model . 3.1 Mid-level feature extraction . For the extraction of mid-level image features that encode local deformation information , we employ the ResNet-34 [ 15 ] and truncate it at the sixth convolutional block . The motivations to process features from ResNet instead of directly sampling local patches over the image are twofold : First , the mid-level feature maps provided by ResNet features allow the network to learn to encode local deformations from contextual cues such as local texture statistics , illumination and viewpoint changes ; Secondly , the receptive field of our local feature descriptor is not tied to a fixed support region size and sampling pattern like traditional patch-based local descriptors . Strong perspective and deformation changes drastically influence the optimal support region sizes for the same keypoint on different images ; thus , we present an architecture that is aware of these local deformations by design , since the dense feature map allows such information to flow to the non-rigid warper module . The ResNet-34 head outputs a feature map X ∈ RH/8×W/8×128 by performing convolutions until the spatial resolution achieves 1/8 of the original input resolution . The feature map X is used as input for the the non-rigid warper component , which computes the Thin-Plate Splines ( TPS ) parameters with the spatial transformer to model a non-rigid warp .

Summary: The paper proposes DEAL, an RGB patch descriptor that has a learned robustness against non-rigid deformations. Given a SIFT keypoint, surrounding ResNet features are used to estimate a non-rigid thin-plate spline deformation of the patch. The RGB patch, after being rectified by the thin-plate spline, is passed to a CNN that regresses a 128-dim descriptor. The network is trained via a hard triplet loss. The training data is simulated and uses groundtruth correspondences/matches, but results are shown on both real and synthetic data. Experiments compare to a number of classic and learned baselines. DEAL outperforms other methods in the main experiment and shows best or second-best performance on a couple of applications.