## 3 Method
### Multi-Level Keypoints Detector  
Considering the image differences between optical and SAR images, extracting only local or high-level features may cause matching problems. We present MKD, which fuses the detection results of higher-level feature maps while preserving lower-level features through a pyramidal feature hierarchy of convolutional neural networks. The detection of multi-layer feature maps is combined.  
MKD scores the keypoints regarding both spatial and channel-wise responses at first. To further enhance the correlation with the actual distribution of all responses along the  
Figure 1: The pipeline of PM-Net. The proposed method has two stages: multi-level keypoints detector and patch feature learning.  
channel [27], the peak value is used as the basis for final keypoints measurement. The illustration of the peak value can be seen in Figure 2.  
The feature map of network output under the feature space \(\mathbb{R}^{H\times W\times C}\) is \(F\), and for each location \((i,j)\) in \(\mathbf{F}^{c}(c=1,2,\ldots,C)\), the peak value of channel-wise is obtained by:  
\[\alpha_{ij}^{c}=softmax(F_{ij}^{c}-\frac{1}{C}\sum_{t}F_{ij}^{t}) \tag{1}\]  
where softmax activates the peak value to a positive value. The corresponding peak value of local space is obtained by:  
\[\beta_{ij}^{c}=softmax(F_{ij}^{c}-\frac{1}{|N(i,j)|}\sum_{(I,J)\in N(i,j)}F_{ IJ}^{c}) \tag{2}\]  
where \(N(i,j)\) is the neighboring pixels within 3 \(\times\) 3 around \((i,j)\). The final keypoints detection score is combined as:  
\[s_{ij}^{c}=\max_{I}(\alpha_{ij}^{c}\beta_{ij}^{c}) \tag{3}\]  
The detected score is used as the weighting term of hardest-contrastive loss function (Section 3.4.1).  
To better preserve low-level features, such as corner points and edges, MKD combines the detections from multiple feature levels to achieve multi-level integration. Specifically, using network convolution to obtain different levels of feature maps \(\{\mathbf{F}^{(1)},\mathbf{F}^{(2)},\ldots,\mathbf{F}^{(l)}\}\), we use the aforementioned detection method at each feature map separately and obtain the corresponding score map \(\{\mathbf{s}^{(1)},\mathbf{s}^{(2)},\ldots,\mathbf{s}^{(l)}\}\). Next, it is up-sampled and reduced to the same spatial resolution as the input image. Finally, the weighted combination is combined as:  
\[\mathbf{s}=\frac{1}{\sum_{l}w_{l}}\sum_{l}w_{l}\;s^{(l)} \tag{4}\]  
where \(w_{l}=1\), 2, 3,which can balance the information from low-level and abstracted feature maps [27], and \(s^{(l)}\) is the score maps.  
The lightweight L2-Net [37] is chosen for the MKD backbone network to reduce the amount of network computation. Meanwhile, to improve the ability of adapting to geometric changes, the last three convolutional layers of L2-Net (conv7, conv8 and conv9) are replaced with deformable convolutional networks [38; 39]. To extract information from different feature layers, three layers, conv2, conv4, and deform_conv9, are selected to perform above detection as well as the weighted combination.  
Figure 2: Illustration of peak value, reprinted from Ref. [36]. 2018, [PERSON]. (**Left**): different response curves can lead to same ranking. (**Right**): peak value of the response curve can be evaluated as the area above the most-peaked response curve.