###### Abstract  
Due to the differences in radiation and geometric characteristics of optical and synthetic aperture radar (SAR) images, there is still a huge challenge for accurate matching. In this paper, we propose a patch-matching network (PM-Net) to improve the matching performance of optical and SAR images. First, a multi-level keypoints detector (MKD) with fused high-level and low-level features is presented to extract more robust keypoints from optical and SAR images. Second, we use a two-channel network structure to improve the image patch matching performance. Benefiting from this design, the proposed method can directly learn the similarity between optical and SAR image patches without manually designing features and descriptors. Finally, the MKD and two-channel net-work are trained separately on GL3D and QXS-SAROPT data sets, and the PM-Net is tested on multiple pairs of optical and SAR images. The experimental results demonstrate that the proposed method outperforms four advanced image matching networks on qualitative and quantitative assessments. The quantitative experiment results show that using our method correct matching points numbers are increased by more than 1.15 times, the value of F1-measure is raised by an average of 7.4% and the root mean squared error (RMSE) is reduced by more than 15.3%. The advantages of MKD and the two-channel network are also verified through ablation experiments. remote sensing; feature learning; patch matching; two-channel network; optical and SAR image +
Footnote †: journal: Applied  
turned to deep learning with the development of artificial intelligence techniques. Deep learning methods are applied to optical and SAR image matching, with the aim of learning high-level semantic features for robust matching. For optical and SAR image matching, the existing methods based on deep learning mainly adopt a two-branch framework in which the convolutional neural network (CNN) acts as both the feature extractor and feature descriptor [10]. The two-branch networks can be divided into Siamese networks and pseudo-Siamese networks. [PERSON] et al. [11] use the Siamese network to learn homogeneous features from optical and SAR images by using dilated convolution to obtain a bigger receptive field, and generate image patch matching similarity based on deep features using dot product computation. A similar architecture was applied in [12]; the differences being that it connected the output feature pairs using a convolution operation. However, Siamese networks using shared parameters to reduce the training cost cannot easily describe the differences between the different modalities for optical and SAR image matching tasks [10]. As a result, pseudo-Siamese networks are increasingly being used for cross-modality remote sensing image matching [13; 14]. Although pseudo-Siamese networks can help capture significant differences between cross-modality images, there still exists similar semantic information, such as structure and shape, between optical and SAR images that the pseudo-Siamese networks cannot extract effectively. This local information can also have an impact on optical and SAR image matching performance. In this study, we focus on learning high-level semantic information while preserving low-level local features to achieve accurate patch matching of optical and SAR images. An end-to-end matching network architecture of optical and SAR images is proposed. It mainly contains multi-level keypoints detection and patch matching. The contribution of this paper can be summarized as follows:  
1. We present a multi-level keypoints detector, called MKD, which detects and fuses low-level and high-level feature maps by joint peak values to obtain richer image information for more robust keypoints. 2. We propose a two-channel patch matching network for optical and SAR images to improve the patch matching performance. This trained Siamese-type network can directly determine the similarity between optical and SAR image patches without manually designed features and descriptors.