Interpretable Matching of Optical-SAR Image via Dynamically Conditioned Diffusion Models

Authors: shuiping  gou
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 13:29:22

Abstract: Driven by the complementary information fusion of optical and synthetic aperture radar (SAR) images, the optical-SAR image matching has drawn much attention. However, the significant radiometric differences between them imposes great challenges on accurate matching. Most existing approaches convert SAR and optical images into a shared feature space to perform the matching, but these methods often fail to achieve the robust matching since the feature spaces are unknown and uninterpretable. Motivated by the interpretable latent space of diffusion models, this paper formulates an optical-SAR image translation and matching framework via a dynamically conditioned diffusion model (DCDM) to achieve the interpretable and robust optical-SAR cross-modal image matching. Specifically, in the denoising process, to filter out outlier matching regions, a gated dynamic sparse cross-attention module is proposed to facilitate efficient and effective long-range interactions of multi-grained features between the cross-modal data. In addition, a spatial position consistency constraint is designed to promote the cross-attention features to perceive the spatial corresponding relation in different modalities, improving the matching precision. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods in terms of both the matching accuracy and the interpretability.

Main paper: 
 introduction: 
 With the rapid development of remote sensing technology, different image sensors are constantly emerging and provide rich image data for the earth observation. Owing to the complementary property between different modality images, the multi-source information integration is widely explored. Especially, the synthetic aperture radar (SAR) and optical sensing images have been increasingly explored and applied to mapping [45] , object detection [36] , and etc. Therefore, effectively integrating and exploiting optical and SAR images is the focus, in which the optical-SAR image matching is the core issues.However, due to their different imaging mechanisms, there exist remarkable geometric differences and nonlinear radiometric variations between optical and SAR images. As a result, the optical and SAR image matching still remains challenging.To solve the task, researchers devote themselves to the challenge and propose various algorithms. In the early stages, traditional image matching methods are widely used, including regionbased and feature-based methods. Region-based methods use similarity metrics, such as normalized mutual information (NMI) [2] and normalized cross-correlation (NCC) [20] , to find the correspondence between the template and the reference images. The kind of approaches only leverage the global pixel intensity information within a window to calculate the similarity of the corresponding region, which is sensitive to the image intensity differences and the noise. To this end, feature-based image matching methods are developed, which aims to extract keypoints of each image and find matching points. The representative algorithm is the scale-invariant feature transform (SIFT) [23] , which detect keypoints in different scales by utilizing its description in terms of the scale, the gradient magnitude, and direction. Based on the SIFT algorithm, some variants, such as BFSIFT [38] , AAGSIFT [37] , and RIFT [21] , have been derived. In addition, to overcome the nonlinear radiometric differences in SAR and optical images, structural similarity-based descriptors have been proposed, such as histogram of oriented phase coherence (HOPC) [43] , and channel feature of oriented gradients (CFOG) [42] . However, the potential of hand-designed features for improving the matching performance is very limited considering the remarkable differences between SAR and optical images.Thanks to the considerable achievements made by the convolutional neural networks (CNNs) in computation vision tasks, and the launch of the publicly available remote sensing data [31, 34, 41] , deep-learning-based cross-modal image matching approaches have constantly emerged [16, [24] [25] [26] 44] . Existing deep-learningbased optical-SAR image matching methods includes two kinds: the Siamese network-based feature mapping, and the generative adversarial network(GAN)-based cross-modality translation. The Siamese network-based methods [24] [25] [26] 44] exploit the effective feature extraction ability of CNN to map the multi-modal images into the common feature spaces, and perform similarity metrics on the mapped features spaces. However, the mapping feature spaces are not visual, and lack of interpretability. Furthermore, these methods fail to extract useful features for the matching when there are more textureless regions in the images. In contrast, the GAN-based approaches [13, 28] attempt to translate the SAR or optical image from one modality to the other by the adversarial learning between generators and discriminators. But, GAN trains the model through the mutual game between the generator and discriminator, which easily causes the model to fall into a local minimum, generating unstable translations on the cross-modal images with large differences. Moreover, GAN-based matching methods are not end-toend.Recently, the diffusion model [17] has received considerable attention on generative models, which has a diffusion process to gradually add Gaussian noise to the data and a denoising process to learn to remove it. Driven by its stable training manner and highquality generation results, the diffusion models have been explored to the optical-SAR image translation. Bai et al. [5] utilizes a conditional diffusion model to efficiently translate SAR images into optical images, and Shi et al. [32] proposes self-attention and long skip connections in denoising networks to enhance feature extraction, which demonstrates the potential of diffusion models in translating SAR images. However, existing diffusion model-based optical-SAR translation methods only focus on generating cross-modal images for enhancing human visual perception, and have not yet explored the downstream cross-modal image matching task. In fact, using the generated image for matching requires elaborate design of matching methods, since there still exist different attributes between the generated remote sensing images and the real images, which deteriorates the matching performance and the speed.To address the above problem, this paper formulates the optical-SAR cross-modal image matching as a dynamically conditioned diffusion model (DCDM), which aims to learn the posterior distribution of regions with dense correspondences. Specifically, the optical template and SAR search image pairs are taken as conditions to respectively provide the content for the better generation and the texture details for the accurate matching. Moreover, a gated dynamic sparse cross-attention (GDSC) module is designed to dynamically inject reliable conditional information into the generative network and accelerate denoising process. On this basis, to enhance the perception of matching positions, we introduce the spatial position consistency constraint. In the matching, to reduce the computational effort, the latent features of the generated SAR are directly matched with features of the search SAR, instead of decoding them into images and then matching them. The contributions of this paper are summarized as follows:• We propose an end-to-end cross-modal image matching framework, dynamically conditioned diffusion model (DCDM). It not only translates cross-modal images, but also completes the pixel-level matching in the latent space. • A gated dynamic sparse cross-attention module is present to perform the controlled and efficient cross-interaction between the template and the search, aiming to filter out the outlier matching regions while improving the computational efficiency. • A spatial position consistency constraint is designed to enhance the detail perception of the cross-attention to generate more accurate cross-modal features for matching. Experimental results on two matching datasets quantitatively and qualitatively demonstrates the effectiveness and the interpretability of the proposed approach. 
learning-based multimodal remote sensing image matching: 
 In learning-based multimodal image matching models often leverage intensive interactions between two modalities to capture effective matching features. The studies [7, 14] apply the cross attention to perform long-range interactions of cross-modal features, thus capturing features suitable for matching. Other methods [9, 26] slide the template features on the reference features pixel by pixel to calculate the similarity heatmap, which is computationally intensive, especially for large images. Fang et al. [12] leverages U-Net [30] to extract high-resolution features, and use the Fast Fourier Transform (FFT) to implement the NCC similarity metric in the frequency domain to accelerate, which has been widely used. Mu et al. [26] proposes a two-stage feature extraction network to achieve precise localization from coarse to fine, which designs a suppression network and a triple loss to suppress false matching position. Zhang et al. [44] presents to fuse low-level fine-grained localization features with high-level semantic features to enhance feature discrimination. Michele et al. [15] extracts features from full-size and half-size images, and then fuses these features to construct pixel-level features for matching. Despite the advances in matching accuracy, these methods still face challenges of the uninterpretable feature spaces, the indiscriminate matching for textureless regions, and the large amounts of computation requirements. In contrast, we propose the gated dynamical sparse attention under the latent diffusion paradigm to efficiently extract cross-modal features with consistent representation, achieving interpretable and robust matching.The emergence of the denoising diffusion probabilistic model (DDPM) [17] has led to the widespread use of diffusion models in computer vision [4] , natural language processing [3] , interdisciplinary applications [1] , and audio processing [18] , which outperforms the current GAN-based generative models in image synthesis [27] . DDPM is a parametric Markov chain that incrementally adds noise to the data during forward diffusion until the original signal is completely corrupted, and then reconstructs the signal during reverse diffusion. The denoising diffusion implicit model (DDIM) [33] is evolved from DDPM, which introduces non-Markov chain diffusion process. This innovation reduces the number of steps required in the inference process. Furthermore, to train denoising diffusion models on limited computational resources while maintaining their quality and flexibility, the latent diffusion model (LDM) [29] applies DDPM to the latent space extracted by the powerful pre-trained autoencoders. Compared to other diffusion models, LDM significantly reduces computational requirements, and achieves efficient cross-modal generation. Therefore, our approach adopts LDM as the base framework to develop the optical-SAR matching algorithm.Over the past few years, the transformers have been exploded in the computer vision community [6, 10] . Contrary to the convolution operation that extract local features, transformers exploit selfattention mechanisms to capture long-distance dependencie, and have global receptive fields [35] . However, such a property comes at the cost of having a high computational complexity and a large memory footprint. To mitigate this problem, the sparse attention [8] has been proposed, in which each query focuses on only a small number of key-value pairs instead of all key-value pairs. Several hand-crafted sparse patterns have been proposed, such as limiting attention in localized windows [22] , expanding the windows [39] , and etc. Recently, a novel dynamic sparse attention, Biformer, [46] has been proposed, whose two-layer routing architecture performs the dynamic computational allocation through information awareness, effectively reducing the computational complexity.All of the above approaches focus on designing sparse self-attention by reducing the number of key-value tokens. In fact, for the crossattention in remote sensing image matching task, filtering queries related to the template is also important, since invalid queries bring interferences. Hence, this paper presents a gated dynamic sparse cross-attention by dynamically selecting both effective queries and keys for efficient computation and matching. 
the proposed method: 
 To achieve the cross-modal optical-SAR image matching, we formulate a dynamically conditioned diffusion model to translate the optical templates into SAR templates, and perform the matching. The pipeline of the proposed method is illustrated in Figure 1 . Firstly, we continuously add Gaussion noise to the ground-truth SAR template in each diffusion step, and then generate the SAR template via training a U-Net-based denoising network. To generate more realistic SAR images, the corresponding optical template is adopted as a condition to provide the scene content. Simultaneously, a taskoriented condition, the search SAR image, is introduced to provide the texture details for the accurate matching. Afterwards, apply the gated dynamic sparse cross-attention module and the spatial position consistency constraint to achieve the effective and efficient cross-modal feature interaction and aggregation. Finally, FFTbased NCC [12] is adopted to perform the matching between the generated SAR template and the search SAR in the latent space.The optical-SAR cross-modal image matching task aims to find the corresponding spatial position of the optical image in the SAR image. To reduce the influence of modality differences, this paper proposes a dynamically conditioned diffusion model (DCDM) to formulate the generative process to translate optical images. However, using noise only to generate SAR image is intractable, since remote sensing images contains rich targets. Fortunately, optical images contain object information, which can provide the content information. Thus, the optical template 𝑇 𝑂 is treated as the condition to guide the generation of the real SAR template. In addition, the texture details is important for the matching. Considering that the generated SAR template is matched with the search SAR, we further take the search SAR image as another condition to supplement the texture details.However, denoising in pixel space is time-consuming and resourceshortcoming. Inspired by the LDM [29] , we utilize an auto-encoder [11] to learn a latent space of the perceptual compression of SAR images, and perform diffusion process in latent space to reduce the computational complexity. Specifically, the encoder is respectively applied to the real SAR template image 𝑇 𝑆 and the search SAR image patches 𝑆 𝑆 to obtain their emdeddings 𝑥 0 , and 𝑐 𝑠𝑎𝑟 . Correspondingly, the optical template 𝑇 𝑂 is encoded to 𝑐 𝑜𝑝𝑡 by two convolutional layers. Notable, to speed up the perceptual compression, the search SAR images are splitted into smaller patches 𝑆 𝑆 to input the encoder, and stitched after the encoding. In the forward diffusion process, continuously add Gaussian noise into 𝑥 0 in each diffusion step to obtain 𝑥 𝑡 . In the reverse diffusion, denoise 𝑥 𝑡 given conditions 𝑐 𝑜𝑝𝑡 and 𝑐 𝑠𝑎𝑟 , which is expressed as:EQUATIONThe SAROPT-conditioned latent diffusion model not only increases the denoising speed, but also make the matching to be endto-end, instead of constructing the SAR template and then extracting features for the matching.Despite denoising in the latent space, it is still requires a large computations in conditional features interaction. Particularly, the search image is relatively larger. In addition, conditional remote sensing images contains a large number of textureless regions, resulting in tedious similarity calculation, and even a negative effect. Therefore, we present a gated dynamic sparse cross-attention module to dynamically select effective regions to perform the crossmodal conditional interaction, shown in Figure 2 . It firstly exploits the coarse-grained routing block (CRB) to efficiently calculate the sparse cross-attention map, and then utilizes the attention calibration block (ACB) to smooth the cross-attention map, and finally leverages an attention gate (AG) to control the preference between the conditional information interaction and the denoising generation. The details are elaborated on the following.Coarse-Grained Routing Block. Recently, Biformer [46] presents the sparse self-attention to save both computation and memory. Benefiting from the idea, we downsample the feature map in the U-Net encoder to obtain cross attention on coarse-grained feature map, and then filter out redundant similar feature regions. Afterwards, the fine-grained cross attention is performed on the remaining feature regions, thus reducing the computational overhead.As shown in Figure 3 , the template feature maps 𝑋 , and the search image feature maps 𝑌 are respectively divided into 𝑠 and 𝑆 feature patches as 𝑋 ∈ R 𝑠 ×𝑃 ×𝐶 and 𝑌 ∈ R 𝑆 ×𝑃 ×𝐶 . Afterwards, the query 𝑄, key 𝐾, and value 𝑉 , are calculated through the linear projection 𝑊 𝑞 , 𝑊 𝑘 , 𝑊 𝑣 : Then, find patches in the 𝐾 that should be attended for each given patch in the 𝑄. Specifically, we obtain the coarse-grained features of 𝑄 and 𝐾 in the encoder, denoted as 𝑄 𝑐 ∈ R 𝑠 ×𝐶 and 𝐾 𝑐 ∈ R 𝑆 ×𝐶 . Herein, the average of each feature patch is adopted to integrate its information. Thus, the importance of feature patches of projected features 𝐾 of 𝑌 can be represented by the coarse-grained cross-attention map 𝑀𝑎𝑝 𝑌 , formulated as:𝑄 = 𝑋𝑊 𝑞 , 𝐾 = 𝑌𝑊 𝑘 , 𝑉 = 𝑌𝑊 𝑘 (2)𝑀𝑎𝑝 𝑌 = 𝑏𝑚𝑚 ( 𝑄 𝑐 , 𝐾 𝑐 𝑇 ) √ 𝐶 (3)Notably, in the decoder stage, we directly input the cross-attention map of the previous layer into the next layers, instead of recalculating the coarse-grained cross-attention map of next layer highresolution features. It not only avoids duplicate calculations, but also exploits the cross-attention map obtained by rich semantics. Besides, we calculate the importance of each feature patch in the template features 𝑋 . Different from the calculation of 𝑌 , we define a score to measure the important feature patches of 𝑋 . It is known that patches with rich matching information tend to have higher variance values. Hence, the variance of each patch is adopted as a metric of information importance. But, some outlier feature patches inevitably obtain high variances, which causes negative effects. To avoid selecting patches with high variances but side effects, we further exploit two learnable weights, 𝑀 𝑐 and 𝑀 𝑠 , to dynamically interact with each feature patch of 𝑋 in the spatial and channel dimensions, respectively. Thus, a learnable importance score is obtained. Finally, use the learnable score to refine the variance score. The process is expressed as follows:𝑠𝑐𝑜𝑟𝑒 𝑣 = 1 𝐶 • 𝑃 𝐶 ∑ 𝑗=1 𝑃 ∑ 𝑖=1 ( 𝑋 𝑖,𝑗 - 1 𝑃 𝑃 ∑ 𝑖 𝑋 𝑖,𝑗 ) 2 𝑠𝑐𝑜𝑟𝑒 𝑙 = Sigmoid ( Mean (𝑋 𝑀 𝑐 ) + Mean ( 𝑋 𝑇 𝑀 𝑠))𝑠𝑐𝑜𝑟𝑒 𝑋 = 𝑠𝑐𝑜𝑟𝑒 𝑣 × 𝑠𝑐𝑜𝑟𝑒 𝑙 (4)where 𝑠𝑐𝑜𝑟𝑒 𝑣 is the variance measured importance, 𝑠𝑐𝑜𝑟𝑒 𝑙 is the learned importance, and 𝑠𝑐𝑜𝑟𝑒 𝑋 ∈ R 1×𝑠 is the refined importance metric. 𝑋 ∈ R 𝑠 ×𝑃 ×𝐶 represents template features. 𝑀 𝑐 ∈ R 𝐶 ×𝑀 and 𝑀 𝑠 ∈ R 𝑃 ×𝑀 denote learnable weights of linear layers (1D-Conv), and 𝑀 is the specified dimension. In experiments, we set 𝑀 = 𝐶.Based on the feature patch importance 𝑀𝑎𝑝 𝑌 ∈ R 𝑠 ×𝑆 and 𝑠𝑐𝑜𝑟𝑒 𝑋 ∈ R 1×𝑠 of 𝑌 and 𝑋 , the feature patches are selected. In experiments, we set a hyperparameter 𝛾 (0∼1) to control the number of selected tokens. Assuming that the number of feature patches of a feature map is 𝑁 , the top 𝑁 × 𝛾 tokens, i.e., 𝑁 𝛾 tokens are selected. Moreover, 𝛾 is in inverse proportion to the feature resolution to ensure smaller computation and progressive refinement in the decoder. The token selection process is formulated as:EQUATIONwhere 'topkIndex' is used to obtain the indexes of top 𝐾 values in a feature map. 'gather' is used to select feature patches based on the index. 𝐼 𝑥 ∈ R 𝑠 𝛾 is the indexes of the top 𝑠 𝛾 = 𝑠 × 𝛾 feature patches in 𝑋 . 𝑀𝑎𝑝 𝑦 represents the selected row tokens of 𝑀𝑎𝑝 𝑌 .𝐼 𝑦 ∈ R 𝑠 𝛾 ×𝑆 𝛾 is indexes of 𝑆 𝛾 patches in 𝑌 that should be attended for each patch in the sparsed 𝑋 . Based on the coarse-grained index matrix 𝐼 𝑥 and 𝐼 𝑦 , the fine-grained projection features of the query, key, and value are obtained:EQUATIONThen, the sparse fine-grained cross-attention map 𝑀𝑎𝑝 𝑓 ∈ R 𝑠 𝛾 𝑃 ×𝑆 𝛾 𝑃 (𝑆 𝛾 𝑃 = 𝑆 𝛾 × 𝑃) is calculated using the 𝑄 𝑓 ∈ R 𝑠 𝛾 ×𝑃 ×𝐶 , 𝐾 𝑓 ∈ R 𝑠 𝛾 ×𝑆 𝛾 𝑃 ×𝐶 , and 𝑉 𝑓 ∈ R 𝑠 𝛾 ×𝑆 𝛾 𝑃 ×𝐶 :EQUATION)where 𝑏𝑚𝑚 is the batch matrix multiplication, the 𝑇 denotes the transpose operation. Attention Calibration Block. All the selected coarse-grained feature patches have similar semantics, but the corresponding multiple fine-grained features contain different information, which lead to the fact that the fine-grained features in the same coarse-grained feature patch focus on inconsistent positions after cross-attention. To overcome the problem, we use a calibration module to locally smooth the cross-attention map, as shown in Figure 4 .Due to the fact that the cross-attention map represents the matching position of template features 𝑄 𝑓 to the search features 𝐾 𝑓 , the distribution of highlighted values in the cross-attention map is diagonal sparse and discrete, shown in Figure 5 . Therefore, each row of the cross-attention map 𝑀𝑎𝑝 𝑓 is firstly circularly shifted to achieve spatial alignment attention map 𝑀𝑎𝑝 𝑓 according to 𝐼 𝑥 and 𝐼 𝑦 . Then, three convolution operations 𝐾 1 ∈ R 1×𝐶 , 𝐾 2 ∈ R 1×𝐶 , and 𝐾 3 ∈ R 𝐶 ×1 are performed on the spatially aligned attention map for smoothing, expressed as:𝑀𝑎𝑝 𝑓 𝑎 = Align ( 𝑀𝑎𝑝 𝑓 ) 𝐴 𝑤 = Sigmoid ( Up(AvgPool(𝑀𝑎𝑝 𝑓 𝑎 ) * 𝐾 1 ))EQUATIONwhere AvgPool(•) is average pooling with pooling size 2×2, and its step size is 2. UP(•) is a bilinear interpolation.After obtaining the smoother attention map, the cross-attention map is reset to the origin spatial position by cyclic shifting to calculate the attention features, represented as:EQUATIONwhere 𝑋 𝑎0 denotes the features padded the unselected feature regions with 0. It has the same size as the original feature 𝑋 . As a note, the calibration module is lightweight, since it performed on 2-D cross-attention map. Attention Gate. The remote sensing images often contain plain areas, which is textureless. Hence, using the kind of images as conditions is ineffective and even cause a negative effect on the denoising generation. In the situation, the denoising network should focus on using the input itself to autonomously generate, instead of relying on the information in conditions. Considering that, we design an attention gate to control the inflow of conditional information, as shown in Figure 4 .In detail, the smaller the variance of the spatially aligned crossattention map, the better the attention. Hence, the variance is calculated to measure the attention effect. Moreover, conditional features at different steps have different perceptual abilities, the time step 𝑡 is also introduced to control the usage of conditions in 𝑡 𝑡ℎ step. Given the spatially aligned and smoothed cross-attention map 𝑀𝑎𝑝 𝑓 𝑎_𝑠𝑚 ∈ R 𝐼 ×𝐽 (𝐼 = 𝑠 𝛾 𝑃, 𝐽 = 𝑆 𝛾 𝑃), the step time embedding 𝑡 𝑒𝑚𝑏 ∈ R 𝐶/2 , the linear encoding layers 𝐿 1 ∈ R 1×𝐶/2 and 𝐿 2 ∈ R 𝐶 ×𝐶 , the attention gate is expressed as:EQUATIONThere exists great scattering characteristic differences in cross-modal images, the random matching position, and multiple similar matching, which leads to incorrect attention. Therefore, the paper further design a spatial position consistency constraint to constrain the correspondence relationships in the cross-attention map. We use the cross-entropy loss to constrain the predicted heatmap p and ground-truth attentional map 𝑝, express as: The ground-truth attention map 𝑝 is generated as shown in Figure 5 . Assuming that the template image (the red box) (128×128) is matched on the position of (x=40, y=40) in the search image (256×256), the ground-truth matching position in the 32× downsampling feature map is a decimal. In general, we regard the attention value of the matching region is 1, while the non-matching region's is 0. Hence, we divide 1 into different weights according to the ratio of each pixel of the template (4×4) to each pixel of the search (8×8), resulting in 16×64 ground truth matches values to constrain the attention map.EQUATIONThe optimization of the proposed dynamically conditioned diffusion model involves two parts: the denoising network, and the spatial position consistency constraint. To optimize the denoising network, the 𝐿2 loss and the SSIM [40] loss are adopted, expressed as:EQUATIONIn summary, the total loss 𝐿 is the sum of the denoising loss 𝐿 𝑑𝑒𝑛 and the spatial position consistency constraint loss 𝐿 𝑠𝑝𝑐𝑐 (refer to Equation ( 11) ):EQUATION) 
datasets: 
 Sentinel Implementation details: The method is implemented based on the PyTorch framework, and run on Nvidia Geforce RTX4090 GPU and Core i7-12700KF CPU. The feature stride of the autoencoder is 4. The feature strides of each feature layer in the denoising U-Net are {1, 2, 4, 8}. The gated dynamical sparse cross-attention module is used at the end of the feature layer with stride of {2, 4, 8}. The sparsity parameter 𝛾 is fixed to 1 when feature stride is 8, since this layer features have rich semantics and few tokens. To train the denoising U-Net, we set the minimum time step 𝑡 to 0.001. The model is trained using the AdamW optimizer for 30k iterations. The decay rate ranges from 5e-5 to 5e-6.In experiments, the Root Mean Square Error (RMSE) and Correct Matching Rate (CMR) are used as evaluation metrics. The RMSE measures the average Euclidean distance between the prediction and the ground-truth. The CMR denotes the correct matching rate when the RMSE is less than a given threshold 𝑇 , denoted as CMR(T). In the heterogenous reomote sensing image matching task, the matching error is less than or equal to 5 pixels is regarded as the successful matching. Therefore, we choose CMR (T=5) and RMSE (T=5) to evaluate the proposed approach. To further evaluate the overall matching performance, the average mean squared error of all samples, RMSE (all), is also adopted.Quantitative comparison. To evaluate the performance of the proposed method, we compare it with state-of-the-arts on SEN1-2 and OSdataset. The compared approaches include traditional methods, NCC [20] , NMI [2] , CFOG [42] , RIFT [21] , and deep-learningbased methods, PSiam [19] , VSMatch [9] , SCMatch [26] , OSMNet [44] , MARU-Net [15] . Table 1 shows the matching performance of state-of-the-arts on SEN1-2 dataset. It is seen that the proposed method achieves 93.02% on CMR(T=5), performing best. Compared to the state-ofthe-art approach OSMNet, our method improves 1.34%. The deeplearning based method, VSMatch, achieves the best RMSE(T=5) of 1.3156, which has a slight improvement of 0.034 compared to our Compared to the overall performance on the SEN1-2 dataset, the OSdataset has a drop, since images in the OSdataset have higher spatial resolution, containing noises. Table 2 compares the matching results on the OSdataset. It is obvious that the region-based methods, NCC and NMI, perform poorly and take more time. The NCC only obtains 10.83% on CMR(T=5), and 2.8915 RMSE(T=5). Compared to region-based methods, the feature-based methods achieve a large improvement, where RIFT has 75.83% on CMR(T=5). The learning-based algorithms, Psiam, SCMNet and VSmatch are under 80% on the CMR(T=5). Whereas, the SCMNet achieves the best result on RMSE(T=5), reaching 1.318. Moreover, the three methods have quite long inference times compared to other learningbased and feature-based methods, since they slide the template by pixel over the search image and feed into the matching model to compute the similarity of matches. On the contrary, OSMnet, MARU-Net and our method take very short matching time. Furthermore, our method achieves 84.91% on CMR(T=5), having a 1.34% gains compare to the state-of-the art approach MARU-Net. Qualitative comparison of similarity maps. Figure 6 qualitatively shows the similarity maps produced by different methods on the two samples. In the similarity map, the higher the response value, the brighter the color. The location corresponding to the peak value is the best matching position. The red boxes in the first column 'Ground truth' denote the ground-truth matching areas. The red dots in other approaches represent the ground-truth offset coordinates in the matching map. The first-row sample shows a pair of optical-SAR image in the rural farmland. The image has little texture and multiple similar regions, which makes optical-SAR matching be challenging. The comparison methods obtain similar responses in parallel linear directions, which is difficult to produce focused similarity maps. Especially, there exists a large highlight regions of similarity map obtained by NMI. VSMatch fails to correctly focus on, and the focused peak of MARU-Net is relatively unremarkable. Whereas the proposed method has low response values in non-matched regions and has a relatively prominent single peak.The second-row sample depicts an urban area with dense geometrically similar buildings. The NMI and VSMatch are shifted from the correct matching point, where the VSMatch shows an undesired neighbouring bimodal characteristic. The peaks of the proposed method and MARU-Net correspond to the matched regions. But the response values of the similarity map achieved by our method are smoother in the non-matched region, which is due to the fact that generating the same modality features are more distinguishable. Qualitative comparison of matching features. Figure 7 visualizes the two samples' template feature maps (Template 𝑂 ′ ) and search feature maps (Search 𝑆 ′ ) obtained by the most representative state-of-the-art method MARU-Net and our approach. The first sample is a rural farmland area with less texture, and the second sample is a complex urban road area. Compare the feature maps and similarity maps produced by the two methods, it is observed that both the template and search feature maps of MAR-Net have very sparse texture features (especially for the first textureless regions) and inconsistent visualization, even though it attempts to map them into a shared feature space. These unreliable features lead to high correlation in non-matching regions. In contrast, the template and search feature maps generated by our method maintains all the texture information of the original image. This is due to the fact that our method focuses on generating feature maps of the same modality in latent space, which is more interpretable, and ensures that the similarity maps are single-peaked. To verify the effect of each proposed components and the parameters on the matching performance, we conduct ablation experiments on the SEN1-2 dataset. The effect of key components. Based on the LDM, we proposed three key components to improve the cross-modal remote sensing image matching performances. To verify the effectiveness of each proposed component, we conduct ablation experiments on the coarse-grained routing block (CRB), the attention calibration block (ACB), and the attention gate (AG). For the baseline and our approach, the denoising sampling step 𝑡 is set to 5, and the cross-attention module is added to feature layers with strides of {2, 4, 8}. For our method, the sparsity parameter, 𝛾, is set to (1, 3/4, 3/8). As shown in Table 3 , the baseline only achieves 84.57% on the CMR(T=5). Compared to the cross-attention in the baseline, the proposed CRB outperforms the base attention by 2.15% on the CMR(T=5), reaching 86.72%. The attention calibration module obtains a remarkable gains of 4.51% on CMR(T=5), reaching 91.23%. The attention gate module improves 2.59% on CMR(T=5), achieving 89.31%. The proposed three modules achieve remarkable improvements on the CMR(T=5) and the RMSE(T=5). Although introducing three modules, the inference time is not significantly increased, since the proposed modules are lightweight.The effect of the number of GDSCs and sample steps. We investigate the effect of the number of GDSCs on the matching accuracy.In the denoising U-Net, the gated dynamical sparse cross-attention (GDSC) module are respectively introduced into the three feature layers corresponding to feature strides of {2, 4, 8}. The sparsity parameters 𝛾 in the three feature layers are set as 1, 3/4, and 3/8, respectively. Figure 8 illustrates the effect of the number of the GDSC module. It is found that the CMR(T=5) improves with the increase of the number of GDSCs, which is due to the fact that a larger number of attention layers enable the network to capture fine-grained information. In addition, it is observed that the denoising sample steps influence the matching performance. The higher the number of sample steps, the higher the value of CMR(T=5). Nevertheless, there are a few increases when the sample steps greater than 5. For instance, given the added feature layers of GDSC as feature strides of {2, 4, 8}, the CMR (T=5) only increases by 0.5% when the sampling steps increase from 5 to 10 (the green line in Figure 8 ), when the GDSC is introduced into features layers . To balance the trade-off between the speed and the performance, the sample steps is set as 5 in experiments.The effect of the sparsity parameter 𝛾. We further analyze the effect of the sparsity parameter 𝛾 on the matching accuracy and time.As shown in Table 4 , the model performs best when the sparsity parameters 𝛾=(1, 3/4, 3/8), where the CMR reaches to 93.02% and the RMSE is 1.3496. Compared to the best performance, the CMR(T=5) slightly decreases by 1.27% and the RMSE(T=5) increases by 0.0525 when 𝛾=(1, 1, 1/2), since larger 𝛾 tends to introduce more error messages and greater matching delays. The CMR(T=5) decreases by 4.59% when the sparse parameters 𝛾=(1, 1/2, 1/4), and even decreased by 10.49% when 𝛾=(1, 1/4, 1/8). This is due to the fact that too smaller 𝛾 causes insufficient cross-modal information interaction, which make the model be difficult to converge and thus degrading the performance. 
conclusion: 
 The paper proposes a dynamically conditioned diffusion model to achieve the interpretable and robust optical-SAR cross-modal image matching. Specifically, the gated dynamic sparse cross-attention module is used to guide the diffusion model to capture information from conditions through the efficient long-range cross-modal interactions, and thus filtering out outlier matching regions. In addition, the spatial position consistency constraint promotes the crossattention features to perceive the spatial corresponding relation in different modalities, and improves the matching accuracy. Experimental results on two datasets show that the proposed method outperforms state-of-the-art approaches in terms of the matching accuracy and the interpretability. The study provides an exploration for future researches on the cross-modal image matching or registration under the diffusion models. 
 
 Appendix:  


Decision: In this paper, the authors propose a dynamic conditional diffusion model for optical-SAR image matching. It introduces a gated dynamic sparse cross-attention module to inject reliable conditional information and a spatial position consistency constraint to enhance the perception of matching positions. The experimental results prove the effectiveness of the proposed method on two datasets. The paper is overall well written, but some grammatical errors need to be corrected. Its novelty is good. The authors are expected to analyze its generalization as well as the model complexity, and add more comparison results with recent SOTA works published in 2023 and 2024. 

After rebuttal, this paper receives 3 positive reviews and 1 negative review, so I recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper formulates the optical-SAR cross modal image matching as a dynamically conditioned diffusion model (DCDM) to learn the posterior distribution of regions with dense correspondences. Firstly, the Gaussian noise is continuously added to the ground-truth SAR template in each diffusion step, and the SAR template is generated by training a U-Net-based denoising network. To generate more realistic SAR images, the corresponding optical template is adopted as a condition to provide the scene content. Additionally, the search SAR image is introduced to provide the texture details for the accurate matching. The gated dynamic sparse cross-attention module and the spatial position consistency constraint are applied to achieve the effective and efficient cross-modal feature interaction and aggregation. Finally, FFT-based NCC is adopted to perform matching between the generated SAR template and the search SAR in the latent space. Experimental results also show that the proposed method outperforms state-of-the-art methods.
strength_weakness: strength_weakness: 1. The introduction of the gated dynamic sparse cross-attention module enhances the efficiency and effectiveness of feature interaction between cross-modal data. This module helps filter out outlier matching regions, which is crucial for accurate matching.
2. The paper introduces a spatial position consistency constraint that promotes better perception of spatial correspondences across different modalities, thereby improving the matching precision.
3. This paper proposes an end-to-end cross-modal image matching framework, which not only translates cross-modal images, but also completes the pixel-level matching in the latent space.
limitations: limitations: 1. While the paper claims that the proposed method reduces computational complexity by operating in the latent space, the overall framework, including the diffusion process and the gated dynamic sparse cross-attention, may still be computationally intensive, especially for large datasets. I hope the authors can provide an analysis of the model complexity and the number of parameters.
2. The authors should carefully review the entire manuscript to improve language accuracy and avoid grammatical errors such as "The study provide" and similar issues.
3. For Figure 1, the authors should make the "Divide" and other labels clearer. Additionally, it would be best to provide some relevant legends in Figure 3.
paper_summary: paper_summary: This paper introduces a dynamically conditioned diffusion model for optical-SAR image matching. Their approach presents a gated dynamic sparse cross-attention module to inject reliable conditional information and a spatial position consistency constraint to enhance the perception of matching positions. Experimental results demonstrate the effectiveness of the proposed approach on two datasets.
strength_weakness: strength_weakness: 1. The writing is clear with clear figures.
2. The proposed two modules are demonstrated concisely and easy to follow.
limitations: limitations: 1. While the testing time is provided for comparison, the paper lacks discussion on memory usage and network parameters of the proposed method, which should be included for comprehensive analysis.
2. The paper introduces an interpretable and robust matching method, yet it fails to provide theoretical or experimental justification for its interpretability and robustness.
3. The comparison in the paper only includes one IEEE JSTRAR paper from 2023. It would be beneficial to include comparisons with recent state-of-the-art (SOTA) methods published in top journals or conferences in 2023 or 2024, if available.
4. Tables 1 and 2 indicate that the proposed method does not achieve state-of-the-art experimental results.
paper_summary: paper_summary: In this manuscript, a dynamic conditional diffusion model is proposed, based on real interpretable and robust optical-SAR cross-modal image matching, mainly gated dynamic sparse cross-attention module and spatial position consistency constraint, and experiments on two data sets are carried out to prove the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The manuscriptdescribes the proposed method clearly, and the language is fluent and professional.
2.  The method is novel, based on diffusion model design, and cross attention is introduced into the proposed method to promote the interaction of multi-granularity features between cross-modal data.
3. Considering that the plain area in the remote sensing image is texture-free, Attention Gate is introduced to control the inflow of conditional information.
4.  In the experimental part, there are many methods of comparison, and explore the influence of super-parameters on the model.
limitations: limitations: 1. Although a large number of methods are compared in quantitative experiments, 
only one method has been used in the past two years, and the other methods are relatively old. In addition, there are few methods of visual comparison. 
2. From tables 1 and 2, there is little difference in performance between the proposed method and the most advanced method, and it does not seem to be very convincing. 
3. In Table 3, why does it take less time overall when modules are added to Base? 
4.  In the The effect of key components of Section 4.4, the manuscript only describes the experimental results and lacks the explanation of the mechanism of the components.
paper_summary: paper_summary: The paper introduces a novel method for matching optical and Synthetic Aperture Radar (SAR) images using a Dynamically Conditioned Diffusion Model (DCDM). This approach is designed to address the significant challenge of matching these images due to their different imaging characteristics. The method utilizes a gated dynamic sparse cross-attention module to facilitate efficient interaction of features across modalities and a spatial position consistency constraint to enhance matching accuracy. Experimental results on two datasets show that the proposed method outperforms state-of-the-art approaches in terms of the matching accuracy and the interpretability.
strength_weakness: strength_weakness: 1. Novel approach: The paper proposes a unique approach using diffusion models for cross-modal image matching, which is a relatively unexplored area in image processing.

2. Experimental validation: The proposed approach is evaluated on two challenging datasets, SEN1-2 and OSdataset, and demonstrates superior performance compared to existing techniques in terms of Correct Matching Rate.

3. Paper writing: The paper provides a detailed explanation of the construction of different modules, making it easy for researchers in the field to understand the work.
limitations: limitations: 1. It appears that the method does not perform as well on the RMSE metric when compared to the best-performing methods in Table 2,  and could you provide a brief analysis of the reasons?

2. About the stated criterion ‘In the task of matching heterogeneous remote sensing images, a matching error of less than or equal to 5 pixels is considered a successful match’ In Evaluation metrics section. What is the basis for this matching error criterion? Can you provide relevant literature support or examples of the application of similar criteria in previous studies?

3. Lack of generalization analysis: The paper does not provide a generalization analysis of the proposed approach on other datasets, which would provide insights into the adaptability of the network to different scenarios.

4. The integration of various strategies like Dynamically Conditioned Diffusion Models, gated dynamic sparse cross-attention, and spatial position consistency constraints, while innovative, introduces considerable complexity. This blend of multiple advanced components may complicate model interpretation, implementation, and optimization. The lack of clarity in how these components interact could lead to potential inefficiencies or diminished performance gains due to overlapping features.