# Visible Spectrum and Infra-Red Image Matching: A New Method
## 3 Proposed Method
### Regression Modeling Using Corresponding Descriptors  
We trained a regression model on corresponding MN-SIFT descriptors of the train set. To understand the training process, let I\({}_{R}\) and I\({}_{T}\) be two images of the same scene. These images depict the same scene contents in VS and IR bands, respectively. Feature points are detected on I\({}_{R}\) and I\({}_{T}\) images and the feature point locations (pixels) are stored as \((x_{r},y_{r})\) and \((x_{t},y_{t})\), respectively, where \(r=1,2,3,...,u\), \(t=1,2,3,...,v\). \(u\) and \(v\) represent total number of feature points detected on I\({}_{R}\) and I\({}_{T}\) images, respectively. The feature points of I\({}_{R}\) are projected onto I\({}_{T}\) with a homography \(K\), which acts as ground truth data between I\({}_{R}\) and I\({}_{T}\). This homography is known in advance between every VS-IR images of the training set according to Reference [18; 34]. We use a projection error of 2 pixels to identify corresponding feature points between I\({}_{R}\) and I\({}_{T}\). Figure 3 shows detected and corresponding feature points as blue '+' and green 'o' between VS-LWIR images of a MSD scene. Then MN-SIFT descriptors are computed for corresponding feature points. Such descriptors are referred to as corresponding/correct descriptors. Let \(R_{1}\) be a descriptor of \(I_{R}\) image and let its corresponding descriptor match is \(T_{1}\) in \(I_{T}\) image:  
\[R_{1} = [r_{1_{1}},\ r_{1_{2}},\ r_{1_{3}},\ \cdots,\ r_{1_{m}}] \tag{9}\] \[T_{1} = [t_{1_{1}},\ t_{1_{2}},\ t_{1_{3}},\ \cdots,\ t_{1_{m}}], \tag{10}\]  
Figure 2: An illustration of \(4\times 4\) locations bins of Modified Normalized-Scale Invariant Feature Transform (MN-SIFT) [17]. where \(m=128\) represents the length of \(R_{1}\) and \(T_{1}\) MN-SIFT descriptors. Let \(f(R_{1},\Theta_{1})\) be a model function that gives an error \(e_{1}\) when it is subtracted from the first element of \(T_{1}\):  
\[e_{1_{1}} = t_{1_{1}}-f(R_{1},\Theta_{1}), \tag{11}\]  
where \(\Theta_{1}\) are the parameters of \(f(R_{1},\Theta_{1})\) which are required to be learnt to minimize the \(e_{1}^{2}\) error i.e., square of error between the first element of \(T_{1}\) and the model function \(f(R_{1},\Theta_{1})\). To learn \(\Theta_{1}\), projection error equal to or less than 2 pixels is used to identify \(n\) corresponding MN-SIFT descriptors between \(I_{R}\) and \(I_{T}\) images. Then corresponding descriptors are stored as \(R\) and \(T\) matrices where \(R_{i}\) and \(T_{i}\) are corresponding MN-SIFT descriptors, which are stored as the \(i\)th row of \(R\) and \(T\), where \(i=1,2,3,...,n\). With \(n\) corresponding descriptors, \(n\) errors are obtained as:  
\[e_{1_{1}} = t_{1_{1}}-f(R_{1},\Theta_{1}) \tag{12}\] \[e_{1_{2}} = t_{2_{1}}-f(R_{2},\Theta_{1})\] (13) \[e_{1_{3}} = t_{3_{1}}-f(R_{3},\Theta_{1})\] (14) \[\vdots = \vdots\] \[e_{1_{n}} = t_{n_{1}}-f(R_{n},\Theta_{1}) \tag{15}\]  
Here \(\Theta_{1}\) is learnt with an objective to minimize \(S_{1}=\sum_{i=1}^{n}e_{1i}^{2}\) i.e., sum of squared error. Similarly, \(\Theta_{2}\), \(\Theta_{3}\),..., \(\Theta_{m}\) are learnt with the same objective to minimize \(S_{2},\ S_{3},...,S_{m}\), respectively, where  
\[S_{1} = \sum_{i=1}^{n}[\ t_{i_{1}}-f(R_{i},\Theta_{1})\ ]^{2} \tag{16}\] \[T_{1} = [t_{1_{1}},\ t_{1_{2}},\ t_{1_{3}},\ \cdots,\ t_{1_{m}}], \tag{10}\]0 (17) \[T_{1} = [t_{1_{1}},\ t_{1_{2}},\ t_{1_{3}},\ \cdots,\ t_{1_{m}}], \tag{10}\]1 (18) \[T_{1} = [t_{1_{1}},\ t_{1_{2}},\ t_{1_{3}},\ \cdots,\ t_{1_{m}}], \tag{10}\]2 \[T_{1} = [t_{1_{1}},\ t_{1_{2}},\ t_{1_{3}},\ \cdots,\ t_{1_{m}}], \tag{10}\]3  
The regression modeling, as explained above, was based on a single image pair, that is, \(I_{R}\) and \(I_{T}\). In the case of a dataset, for instance RGB-NIR and MSD datasets, each dataset is randomly divided into two disjoint sets. One set for training and the other one for testing.