# Learning Local Descriptor for Comparing Renders with Real Images
## 3 Methodology
### Creation of Patch-Pair Data Set  
We intend to train a descriptor. We are inspired by the approaches in [6; 7; 10] and we formulate a similar one with adaptations to address our particular problem. For this, we need a data set of corresponding patches that come from renders and real images. Since there isn't an existing one to the best of our knowledge, we create such a data set by first producing for each real image a simple render using the available viewpoint, and CADs of inspection elements and those of assemblies they are mounted on. Second, we finely register each render with its corresponding real image by manually selecting a set of corresponding points between the two images and estimating a homography that registers the render with the real image. This is necessary because of the inaccuracy in our in-house localization module, as a result of which the renders are not perfectly registered with the real images. Finally, to create the patch-pair data set, we crop each pair of real-rendered images around same points that are detected by the FAST detector in the render. The patch-pairs are created only using locations that are deemed interesting in the render since we want patches that capture interesting geometry in the plain CADs and their corresponding manifestations in real images, not patches that are found interesting in real images by themselves.  
We also produce patches by cropping renders and real images around most salient points detected by the FAST detector exclusively in the real images, to produce 'texture' patches. The real ones of these patches contribute to a reservoir of texture patches when training a descriptor by minimizing triplet loss and are always labeled as negatives in the triplets (see Section 3.3.2). Our entire data set generation procedure is illustrated in Figure 2.  
Figure 1: Simple shading for fast rendering based on fragment normal \(n\) and fragment depth \(d\). \(L(\theta)\) is the component of fragment’s intensity in render due orientation \(\theta\) of \(n\) relative to render camera’s optical axis, and \(G(\theta)\) that due to its depth. Final intensity in render, \(I\), is an affine combination of the two, with manually chosen \(\alpha\), which is the same for all fragments. \(I\) is also varied based on whether the fragment belongs to CAD of inspection element or that of assembly on which it is mounted.  
There are 2 main challenges in training with the patch-pair data set produced as described in Section 3.2:  
* Small number of patch pairs due to less than 100 image pairs available for producing the training data set, and need for large patch sizes for discriminative context, due to simplistic CADs
* Domain shift due to patches coming from synthetic and real domains  
These two problems demanded a two-step approach, much like the one in [13], with our modifications in the \"bootstrapping\" stage. The two-step method is in contrast with related works like [6,7,10] that directly minimize a discriminative or metric loss on matching and nonmatching patches from a data set. We also tried this approach but this lead to no convergence, possibly because of the two reasons mentioned before. However, in face recognition literature, a similar two-step method was reported, although with different strategies for \"bootstrapping\" and without identifying the stage explicitly as meant for addressing domain-shift [13]. Our entire training process will be detailed in the following two Sections 3.3.1 and 3.3.2 and is summarized in Figure 3.