## 1 Introduction  
Unmanned aerial vehicle (UAV) navigation is generally achieved by means of global navigation satellite system (GNSS) [1] or the fusion of GNSS with inertial navigation system (INS [2, 3]). As is known, the GNSS signal might be blocked, cheated, or even shut down [4, 5]. Under these circumstances, standalone INS generally cannot fulfill the navigation necessity, especially for the micro electro mechanical systems (MEMS) INS, which is widely used in the small- or medium-size UAVs [6]. If the UAVs fly at a proper altitude and have a good view of the ground, a vision-based approach can be used to enhance the performances of the MEMS INS, and an acceptable navigation accuracy might be achieved to ensure the reliable flying of UAV in GNSS denied circumstances [7]. Literatures [3, 8] discussed the image matching-based methods for UAV navigation. The position of the UAV can be estimated according to the results produced by matching the real-time image to the referenced image. However, it is still challenging to reliably and accurately match the real-time aerial image and the referenced image when they are captured by different types of sensors at different times, which are defined as multi-sensor temporal images. The distortions between the multi-sensor temporal images, as shown in Figure 1, include nonlinear grayscale variation, severe noise distortion, and even the changes in local ground features with time, which will severely degrade the matching performance [9, 10]. The matching algorithms developed in [8, 11] are very computationally expensive. To solve the problem, the visual odometer based on optical flow [12] combined with particle filter is adopted to restrict the search area of the matching and eliminate the incorrect results [11]. However, optical flow is not drift-free and can be easily interrupted by motion blur or light changes. Consequently, the position solution is still unstable if both the image matching and the visual odometer produce unreliable input. Deep learning-based image matching method is employed in [3] for UAV localization; however, the precision of the image matching algorithm is very low to estimate the precise heading of the UAV, which means that the heading estimation from the image matching cannot be used to correct the INS heading. Therefore, only INS position can be corrected by the image matching results. Finally, these image matching methods [3, 8, 11] are developed mainly for the visible light images, which means their visual positioning methods only work in daylight. In the last decade, vision-based simultaneous localization and mapping (VSLAM) has been extensively studied [11, 13]. However, pure vision-based SLAM is usually not a robust solution for navigation. The main drawback of VSLAM is that the vision tracker could be easily interrupted by light change, motion blurring, or occlusion of moving objects. For solving this problem, vision and inertial fusion-based SLAM (VISLAM) has been proposed [14, 15]. VISLAM is more robust than VSLAM since the inertial module would not be affected by light change, motion blurring, or occlusion, which can consistently provide the motion states to the navigation system [16, 17, 18]. Meanwhile, the vision system can provide environmental perception and states correction, which is supplemental to the internal module. Nevertheless, the VISLAM suffers from the position drift problem. Therefore, closed-loop optimization is employed for correcting the drift [15, 19]. However, the long path closed-loop optimization is very expensive in both computation and memory consumption. In addition, closed-loop path is not always available for UAVs navigation, for example, for single-way flying of UAVs. Although VISLAM has been widely and  
Figure 1: Cross-modal image pair and their enlarged subareas. Image (**a**) shows the visible light image and the infrared image captured in a same scene. Image (**b**) shows the enlarged subareas which are captured and enlarged from the rectangle areas in Image (**a**).