# Learning Local Descriptor for Comparing Renders with Real Images
## 3 Methodology
### Two-Step Training for Feature Extractor
#### 3.3.2 Triplet Loss Training  
The weights learned from bootstrapping were retained except for the last 2-way linear+softmax layer. We denote the retained architecture with \(\phi\). It maps an input patch \(P\in\mathbb{R}^{M}\) to \(\mathbb{R}^{D}\); \(\phi(P)\in\mathbb{R}^{D}\). To \(\phi\), we appended an \(l_{2}\) normalization layer and a linear layer \(W^{\prime}\in\mathbb{R}^{L\times D},L<<D\), which together implement \(e_{P}=W^{\prime}\phi(P)/||\phi(P)||_{2}\). \(W^{\prime}\) is an affine transformation layer without any bias, since bias would be canceled when minimizing triplet loss detailed in Equation (2). The thus augmented \(\phi\) is replicated three times to create a triplet network [13; 29] and triplet loss [13] is minimized for batches of triplets (anchor (a), positive (p), negative (n)) produced by a batch server. In each triplet, an anchor is a negative or a positive example. The weights are shared among all the towers in the triplet network, so they receive the same weight updates during training. Figure 3: Proposed two-step method for training a deep descriptor using a data set of patches created as described in Figure 2. The first stage (left of the vertical divider), bootstrapping, produces weights such that distinction between domains is removed and the second stage (right of the divider), triplet-loss training, trains an embedding layer on top of the weights learned previously that produces discriminative features. The rationale behind triplet-loss minimization is to learn weights such that distances between matching patches is minimized while at the same time, that between nonmatching patches is maximized. \[E(W^{\prime})=\sum_{(a,p,m)\in T}max\{0,\alpha-d_{an}+d_{ap}\} \tag{2}\]  
Here, \(d_{an}=||e_{a}-e_{n}||_{2}\), \(d_{ap}=||e_{a}-e_{p}||_{2}\), and \(T\) is the set of triplets in each training mini-batch. Embeddings extracted from patches through a forward pass are denoted by \(e_{x}\). \(\alpha>0\) is the margin parameter that forces a desired minimum difference between average distances between matching and nonmatching patches. The method and notations followed in this section are similar to those in [13]. During triplet-loss training, only the \"embedding layer\" \(W^{\prime}\) is updated, the rest of the layers are frozen. Thus, it is only \(W^{\prime}\) that learns a discriminative projection of features extracted by \(\phi\) (Steps 3 and 4 in Figure 3), learned during bootstrapping. The architecture \(\phi\) together with the \(l_{2}\) normalization and \(W^{\prime}\) layers is the learned cross-domain feature extractor. The extracted features are of dimension \(D\), with \(D=512\) when patches are of size \(128\times 128\) and \(D=1024\) when they are of size \(224\times 224\). Of the \(D^{\prime}s\) we experimented with, these were the best values. During triplet-loss training, from each batch, we pick \"hard\" triplets, i.e., triplets for which \(d_{an}<d_{ap}\) so that the network can learn from minimizing the triplet loss computed from them. To perform hard-negative mining, we filter a batch of triplets produced by the triplet batch server by passing them through the network once before training for getting their embeddings from which three pairwise distances are computed. We also perform \"in triplet hard-negative mining\", introduced by [10], by swapping anchor and positive patches if \(d_{an}>d_{pn}\). This improves triplet-loss training by making triplets even harder. We use the Adam optimizer with an initial learning rate of 0.005, \(\beta_{1}\) 0.9, \(\beta_{2}\) 0.999 and \(\epsilon\) 10\({}^{-08}\). Regularization was performed via dropout added before the embedding layer. When picking negative real patches for a geometrically interesting patch from a render, batch server has a choice of picking either a geometrically interesting nonmatching real patch or a real texture patch, of which there are thousands in a \"reservoir\". We under-sample the texture patches with ratio of 3:7 to address class imbalance [30].