# STN-Homography: Direct Estimation of Homography Parameters for Image Pairs
## 3 STN-Homography  
### Architecture of STN-Homography  
Figure 2 depicts our STN-Homography architecture which was used to predict the normalized homography matrix \(\overline{H}_{ba}\).  
Our regression model outputs eight regression values which correspond to the first eight elements of the flattened\(\overline{H}_{ba}\), while the last element of \(\overline{H}_{ba}\) is equal unity. The architecture of our regression model is similar to the VGG Net [23]. We used eight convolutional layers with a maximum pooling layer (\(2\times 2\), stride 2) after every two convolutions. The eight convolutional layers all used batch normalization and Relu activation, and have the following number of filters per layer: 64, 64, 64, 64, 128, 128, 128, and 128. The output deep features of the last convolutional layer are followed by a global average pooling layer and by two fully connected layers. The first fully connected layer has 1024 units and the second fully connected layer has eight units. A dropout with a probability of 0.5 is applied after the first fully connected layer. The input to our regression model is a two-channel grayscale image with a size of \(128\times 128\times 2\) pixels. In other words, the two input grayscale images of patch_a and patch_b, which are related by homography, are stacked in a channel-wise manner and are input into the network.  
The grid generator uses a regularly spaced grid \(G_{o}\) with a height \(H_{o}\) and width \(W_{o}\) with pixel normalized coordinates \(u_{i},v_{j}(u_{i}\in[-1,1],v_{j}\in[-1,1])\), where \(i\in H_{o}\) and \(j\in W_{o}\), in conjunction with the predicted normalized homography values \(\overline{H}_{ba}\), to produce one regular grid \(G_{i}\) of normalized coordinates \(u^{{}^{\prime}}_{i^{\prime}},v^{{}^{\prime}}_{j}\) for the inputting single-channel grayscale image patch_a, as shown in Equation (2). The height \(H_{o}\) and width \(W_{o}\) can be chosen freely, and in our experiments, we choose \(H_{o}=h\) and \(W_{o}=w\), where \(h\) and \(w\) denote the height and width of input patch_a.  
\[\begin{pmatrix}u^{{}^{\prime}}_{i}\\ v^{{}^{\prime}}_{j}\\ 1.\end{pmatrix}=\overline{H}_{ba}\begin{pmatrix}u_{i}\\ v_{j}\\ 1.\end{pmatrix}=\begin{bmatrix}h_{11}&h_{12}&h_{13}\\ h_{21}&h_{22}&h_{23}\\ h_{31}&h_{32}&1.\end{pmatrix}\begin{pmatrix}u_{i}\\ v_{j}\\ 1.\end{pmatrix} \tag{2}\]  
The regular grid \(G_{i}\) produced by the grid generator is used to sample the values of the input grayscale patch_a at the corresponding normalized coordinates \(u^{{}^{\prime}}_{i^{\prime}},v^{{}^{\prime}}_{j}\) (note that the normalized coordinated should be converted to the nonnormalized coordinates first). Naturally, these points will not always perfectly align with the integer pixel coordinate values in patch_a. Thus, we use bilinear sampling that extracts the value at a given float coordinate based on the use of bilinear interpolation of the values of the nearest integer coordinate neighbors. The output of the bilinear sampling is patch_a_warp, which will be used to compute the L1 pixelwise photometric loss with the input patch_a_t. The bilinear sampling is differentiable. Hence, it is possible to propagate the L1 loss to the regression model by using standard backpropagation techniques.  
Figure 2: Architecture of STN-Homography.  
As shown in Figure 2, we use two losses in the training of STN-homography. The first is the L2 loss between the regression output \(\overline{H}_{ba}\) and ground true \(\overline{H}_{ba}^{*}\), and the second is the L1 pixelwise photometric loss between the output patch_a_warp of the spatial transformer and the ground truth patch_a_t. The same L1 loss is also used in [25]. However, [25] is based on four-point homography while we directly estimate the \(3\times 3\) homography matrix. The entire network is differentiable and can be trained with standard backpropagation techniques.