# Cross-Modal Image Registration via Rasterized Parameter Prediction for Object Tracking
## 3 Methodology
### Registration Parameter Prediction  
The rasterized registration parameter prediction schema is provided in Algorithm 1. After feature extraction, a group of homography matrices is predicted for the feature pyramids in each layer, as illustrated in the top right part of Figure 1. We first predict a coarse-grained global homography matrix \(H_{g}^{0}\) in the \(L_{0}\) layer, which is divided into an \(N_{1}\times N_{1}\) raster for the \(L_{1}\) layer. Next, we predict a group of fine-grained local homography matrices \(H_{z}^{1},z\in[1,N_{1}\times N_{1}]\) in the \(L_{1}\) layer. Iteratively, we then predict a more fine-grained group of local homography matrices \(H_{z}^{2},z\in[1,N_{2}\times N_{2}]\) in the \(L_{2}\) layer. ```
0:\([F_{r}^{l}],[F_{m}^{l}],l\in[0,L-1]\)
0:\([H_{i}],i\in[1,N_{L-1}^{2}]\)
0:\(L,[N_{i}],l\in[1,L-1]\)
1: Predict a base corner offset for the 0-th layer of feature map in FPN: \(\Delta X^{0}=\textit{OffsetRegressor}(CV(F_{r}^{0},F_{m}^{0}))\)
2: Predict a group of residual offsets for the \(l\)-th layer of feature map in FPN: for the \(i\)-th raster in this layer local residual offsets prediction: \(\Delta X_{i}^{l}=\textit{OffsetRegressor}(CV(F_{r}^{l}[i],F_{m}^{l}[i]))\) endfor \(\Delta X^{l}=[\Delta X_{i}^{l}],i\in[1,N_{l}^{2}]\)
3: Feature corner offsets fusion: for the \(i\)-th local residual offset of \((L-1)\)-th layer feature map in FPN \(\Delta X_{i}=\sum_{l=0}^{L-1}\Delta X^{l}[i]\) endfor \(\Delta X=[\Delta X_{i}],i\in[1,N_{L-1}^{2}]\)
4: Homography matrix group resolution: \(H=DLT(Y,X+\Delta X)=[DLT(Y,X+\Delta X_{i})]=[H_{i}],i\in[1,N_{L-1}^{2}]\)
```  
**Algorithm 1** The registration parameter prediction algorithm  
Specifically, as illustrated in Figure 2, assuming that the corner coordinate set of the reference image \(I_{r}\) and moving image \(I_{m}\) are \(X\) and \(Y\), respectively, the matching response of dual-way features are fed into some fully-connected layers to predict the offsets \(\Delta X\) of the four corner points between the moving image \(I_{m}\) and reference image \(I_{r}\). Then, the feature point pair \(<Y,X+\Delta X>\) is obtained and used to resolve homography matrices \(H\) using the Direct Linear Transform [51]. Multi-modal fusion: To fuse the features of the reference image \(I_{r}\) and moving image \(I_{m}\), the Cost Volume [27] is computed for stereo matching of each pixel feature vector between the two input images, as follows:  
\[CV(\mathbf{F_{A}},\mathbf{F_{B}})=\bigg{[}\frac{<\mathbf{F_{A}}(x_{a}), \mathbf{F_{B}}(x_{b})>}{\|\mathbf{F_{A}}(x_{a})\|\cdot\|\mathbf{F_{B}}(x_{b}) \|}\bigg{]}, \tag{1}\]  
where \(\mathbf{F_{A}},\mathbf{F_{B}}\in\mathbb{R}^{C\times H\times W}\) is the feature map of the reference image and moving image, respectively, in one layer of the feature pyramid, and \(x_{a},x_{b}\in\mathbb{R}^{2}\) represents the pixelcoordinates of images \(\mathbf{F_{A}}\) and \(\mathbf{F_{B}}\), respectively. After multi-modal fusion, we obtain a matching response \(\mathbf{H}\in\mathbb{R}^{H\times W\times HW}\). Multi-scale prediction: To enhance the matching effect of high-resolution images, we propose a Rasterized Feature Pyramid Network (RFPN) for multi-scale feature construction and further estimate the registration parameters from a coarse-grained resolution to a fine-grained resolution in a top-down manner. Specifically, we first use the feature pyramid network to predict the base offset \(\Delta X^{0}\) of the top layer, which is then used to predict a residual offset \(\Delta X^{l}\) for the lower layers. next, we sum up all the prediction offsets in each layer as the final offset \(\Delta X=\sum_{L-1}^{l=0}\Delta X^{l}\), where \(L\) is the number of layers. Rasterized homography estimation: As multi-modal CV computation is an outer product operation in the spatial dimension, it requires significant computing and storage resources; moreover, its efficiency is very low on low-level feature maps with large spatial dimensions.