# Learning Local Descriptor for Comparing Renders with Real Images
###### Abstract
The inspection elements that interest us are industrial parts such as supports, clamps, etc. that have rigid and regular structures compared to natural everyday objects that have more nonrigid geometries. Because of this, the 2D features can be attractive choices for comparing a shape in a real image with that in the render [2;3]. However, the 2D features do not capture view-point dependent information since it is possible to obtain, for example, same contours from different viewpoints due to object symmetries. Comparing edges in render with those in a real image can also be misleading since edges might result from textures, shadows, deformations, surrounding objects, etc. Such features are hence not informative and discriminative since they capture only simple geometric characteristics. In contrast, \(2\frac{1}{2}D\) features, which are features computed from image patches centered at interesting points, capture more detailed information that is revealed due to viewpoint and are robust to the problems faced by 2D features as described before [4]. Local features, engineered as well as learned, have been extensively studied for comparing images, finding local correspondences and detecting objects given a template [5; 6; 7]. The common denominator among the studied features has been comparison between only real images. Consequently, these features capture local texture information, important in real images, in addition to local geometric information. Consideration of texture information is problematic when the CAD models lack material specifications restricting renders to plain renders. We are thus interested in a feature extractor that can produce descriptors to compare textureless patches from plain renders with textured patches from real images. In other words, we want our feature extractor to be as much as possible invariant to texture and to work even in absence of any texture in the compared images. Additionally, we also want to ignore deformable elements such as wires and plastic caps in real image patches when extracting descriptors since we do not have them in our simple CAD models; we are restricted to low poly models that use simple primitives to convey complex shapes. Given the success of learned features in comparing real image patches, we are interested in training a neural network that can extract features from real and rendered image patches that satisfy the requirements described before. Due to the unique nature of our problem, we face two main challenges:  
* Lack of a data set of corresponding real and rendered image patches
* Domain shift due to the patches coming from different domains  
Since ours is the first work, to the best of our knowledge, that tries to learn a local descriptor for comparing renders of simplistic CADs of mechanical assemblies and real inspection images, we are unable to use available data sets of real image patches, like in [8]. We create our own data set using <100 real-rendered image pairs, with an initial coarse registration and a manual fine registration. We then crop the images around interest points detected by the FAST detector [1]. This method for producing a patch-pair data set is our first contribution. The images we used were taken by an industrial camera mounted on a robotic platform that inspected an assembly from a set of pre-planned positions during scheduled inspections. The time for inspection and the time on the imaging platform were both limited, because of which a limited number of real images with approximate viewpoint information were available for us to work with. Since we also produce one render for each real image, this also limited the number of renders. Our second contribution is a two-step approach for training a neural-network feature descriptor. Because our patches come from two different domains (real and rendered) and are limited in number, we can not train a neural-network descriptor from scratch.