# Learning Local Descriptor for Comparing Renders with Real Images
###### Abstract  
We present a method to train a deep-network-based feature descriptor to calculate discriminative local descriptions from renders and corresponding real images with similar geometry. We are interested in using such descriptors for automatic industrial visual inspection whereby the inspection camera has been coarsely localized with respect to a relatively large mechanical assembly and presence of certain components needs to be checked compared to the reference computer-aided design model (CAD). We aim to perform the task by comparing the real inspection image with the render of textureless 3D CAD using the learned descriptors. The descriptor was trained to capture geometric features while staying invariant to image domain. Patch pairs for training the descriptor were extracted in a semisupervised manner from a small data set of 100 pairs of real images and corresponding renders that were manually finely registered starting from a relatively coarse localization of the inspection camera. Due to the small size of the training data set, the descriptor network was initialized with weights from classification training on ImageNet. A two-step training is proposed for addressing the problem of domain adaptation. The first, \"bootstrapping\", is a classification training to obtain good initial weights for second training step, triplet-loss training, that provides weights for extracting the discriminative features comparable using \(l_{2}\) distance. The descriptor was tested for comparing renders and real images through two approaches: finding local correspondences between the images through nearest neighbor matching and transforming the images into Bag of Visual Words (BoVW) histograms. We observed that learning a robust cross-domain descriptor is feasible, even with a small data set, and such features might be of interest for CAD-based inspection of mechanical assemblies, and related applications such as tracking or finely registered augmented reality. To the best of our knowledge, this is the first work that reports learning local descriptors for comparing renders with real inspection images. deep convolutional neural networks; learned feature descriptor; 3D CAD; nonphotorealistic rendering +
Footnote †: journal: Applied  
0000-0001-1000-1000-1000-1000-1000-1000-1000-10000-1000-1000-1000-1000-10000-1000-10000  
on the CAD is present or absent in the real assembly given its inspection image. We have CAD models that lack material related information (color and texture) and so we work with simple renders that reveal geometry of the CAD, specifically, changes in normals and depths of faces of the CAD model. Our CAD models also lack deformable elements such as wires and plastic caps that are present in the real assemblies, and occlude the elements we want to check. In tackling this challenging setting, we assume that the inspection camera is calibrated and coarsely localized with respect to the inspected assembly, hence providing us approximate camera viewpoint (pose + intrinsics) for each inspection image. For this, we rely on an in-house developed localization module based on 2D/3D alignment. The CAD can thus be rendered with a viewpoint similar to the real image. In order to check whether certain components, henceforth inspection elements, are mounted on the assembly correctly i.e., at expected positions and with expected orientations as specified in the CAD, we propose to compare learned local features extracted from real images with those from corresponding renders at interest points such as FAST corners [1]. This is an object detection approach using learned features to check whether an element is present in the real image the way it is in the CAD render. We posit that such learned features are more informative and hence more discriminative than simple 2D features such as contours and edges. We want to use such features because the images we want to inspect are challenging, while our reference CAD models are simple and incomplete, as described before.