# Cross-Modal Image Registration via Rasterized Parameter Prediction for Object Tracking
###### Abstract  
Object tracking requires heterogeneous images that are well registered in advance, with cross-modal image registration used to transform images of the same scene generated by different sensors into the same coordinate system. Infrared and visible light sensors are the most widely used in environmental perception; however, misaligned pixel coordinates in cross-modal images remain a challenge in practical applications of the object tracking task. Traditional feature-based approaches can only be applied in single-mode scenarios, and cannot be well extended to cross-modal scenarios. Recent deep learning technology employs neural networks with large parameter scales for prediction of feature points for image registration. However, supervised learning methods require numerous manually aligned images for model training, leading to the scalability and adaptivity problems. The Unsupervised Deep Homography Network (UDHN) applies Mean Absolute Error (MAE) metrics for cost function computation without labelled images; however, it is currently inapplicable for cross-modal image registration. In this paper, we propose aligning infrared and visible images using a rasterized parameter prediction algorithm with similarity measurement evaluation. Specifically, we use Cost Volume (CV) to predict registration parameters from coarse-grained to fine-grained layers with a raster constraint for multimodal feature fusion. In addition, motivated by the utilization of mutual information in contrastive learning, we apply a cross-modal similarity measurement algorithm for semi-supervised image registration. Our proposed method achieves state-of-the-art performance on the MS-COCO and FLIR datasets. image registration; infrared image; visible light image; rasterized parameter prediction; similarity measurement; contrastive learning +
Footnote †: journal: Medical Image Registration via Rasterized Parameter Prediction via Rasterized Parameter Prediction for Object Tracking  
Footnote †: journal: Medical Image Registration via Rasterizedimage pairs based on mutual relation of image features. Although these algorithms achieve excellent results in single-mode scenarios [18; 19; 20], they cannot be well extended to multimodal scenarios, as the appearance and texture of images in different modalities are quite disparate. Deep learning-based methods design diverse neural networks to predict the offsets of pixel points and use labelled data to train neural networks for homography prediction, such as the DenseFuse [21], FusionGAN [22], SDNet [23], and other models [24], which have been reported to have better adaptability, fault tolerance, and noise resistance for image registration study [25]. These supervised learning methods require a large number of manually aligned images as labels for model training, which is time-consuming and labor-intensive. [PERSON] et al. (2018) [26] proposed an Unsupervised Deep Homography Network (UDHN) which applies Mean Absolute Error (MAE) metrics for cost function computation without labelled images; however, it is not yet applicable for multimodal image registration. To solve this problem, we first propose a rasterized registration parameter prediction method for image registration that uses Cost Volume (CV) [27] to fuse multimodal features and estimate registration parameters from coarse-grained to fine-grained layers. CV is a commonly used multi-scale feature matching method in computer vision, and is often used to find corresponding points between two images. To address the boundary discontinuity problem in rasterized parameter prediction, we use a position-dependent weight matrix as a raster constraint to make the inside pixels more significant and the intersection points more equal. Additionally, motivated by the utilization of mutual information in contrastive learning, we propose a cross-modal similarity measurement algorithm for semi-supervised image registration.