# Remote Sensing Multimodal Image Matching Based on Structure Feature and Learnable Matching Network
## 3 Template Matching Based on PCA-LMN
### Orientation Estimation and Enhancing  
A significant challenge in matching remotely sensed multimodal images lies in the severe distortion of grayscale relationships among the images. As shown in Figure 4, the grayscale change between infrared image (a) and visible light image (b) is considerably significant; however, the structural orientation, which is present in images (c) and (d), is considerably more stable than the gray part of the image. Accordingly, the structure orientation based on log-Gabor is employed by some methods [7; 9] for matching multimodal images. These methods exhibit a significant advantage in terms of matching performance over template-matching algorithms directly based on the image grayscale. Figure 2: Example of template matching with the multimodal image. However, the structure orientation estimated with the log-Gabor filter is sensitive to noise distortion. The orientation map is estimated via log-Gabor filters. Note that the orientation map is disturbed by noise distortion. To improve the noise adaptiveness, the PCA is employed to enhance the structure orientation estimated using log-Gabor. The PCA is typically used to calculate the dominant vectors of a given dataset that can reduce the noise component in signals and estimate the local dominant orientation. For each pixel, the PCA can be applied to the local gradient vectors to obtain their local dominant direction. In general, the PCA can be implemented in two ways: eigenvalue decomposition (EVD) of the data covariance matrix and singular value decomposition  
Figure 4: Multimodal images and their orientation maps: (**a**) infrared image; (**b**) visible light image; (**c**) orientation map of infrared image; and (**d**) orientation map of visible light image. Short yellow lines indicate the structure orientation direction, which is also the image edge direction. Figure 3: Flowchart of PCA–LMN template-matching algorithm. (SVD) of the data matrix. In this work, because of the superiorities in flexibility and robustness [39], SVD is employed to calculate the PCA. Given the original image (I) with its horizontal derivative image (\(a\)) and vertical derivative image (\(b\)), an \(N\times 2\) local gradient matrix is constructed for each pixel:  
\[G= \begin{bmatrix}\mathrm{A}^{T}&\mathrm{B}^{T}\end{bmatrix}  \tag{1}\]  
where \(N\) is determined by the size of the local window of the PCA calculation. For example, if the size of the window for each pixel is \(3\times 3\), then the value of \(N\) is 9. The vectors of the local derivatives, i.e., A and B, can be calculated using the following expression. \[\mathrm{A=}\{a_{1}a_{2},\ldots,a_{N}\},\ \mathrm{B=}\{b_{1}b_{2},\ldots,b_{N}\} \tag{2}\]  
where \(a_{1,2,\ldots,N}\) can be calculated according to Equation (1) and \(b_{1,2,\ldots,N}\) can be calculated according to Equation (2). The dominant orientation can be estimated by determining a unit vector, \(u\), perpendicular to the local gradient vectors (Figure 5). This can be formulated as the following minimization problem. \[\left\|u^{T}G\right\|=\sum_{i=1}^{N}\left[a_{i},b_{i}\right]^{T}u \tag{3}\]  
This can be solved by applying SVD to the local gradient matrix, \(G\). \[G=USV^{T} \tag{4}\]  
where \(U\) is an \(N\times N\) orthogonal matrix; \(S\) is an \(N\times N\) matrix; \(V= \begin{bmatrix}v&u\end{bmatrix} \) is a \(2\times 2\) orthogonal matrix; and \(v\) indicates the local dominant orientation of the local gradient vectors. The PCA-enhanced structure orientation can be calculated according to the following equation. \[\varphi=\frac{1}{2}\angle(u_{1},u_{2}) \tag{5}\]  
where \(\varphi\in[0^{\circ},180^{\circ})\), \(u_{1}\) is the element in the first row of \(u\) and \(u_{1}\) is the element in the second row of \(u\). Note that the structure orientation is orthogonal to the orientation of the gradient vector. The structure orientation enhanced by PCA is more robust against noise than the structure orientation directly estimated with log-Gabor filters.