## 3 Methods and Materials
### Multi-Scale Complementary Features Interaction  
These methods [28,52] use cross-attention to fuse features of different modalities to learn complementary information between them. By improving the multi head attention mechanism, the features of the auxiliary modality are considered as the key and value, while the target modality is used as the query, thus enabling the interaction between the target and auxiliary features. Inspired by these methods, we introduce an improved dual cross-attention block (DCAB), which simultaneously processes both auxiliary and target modalities. The auxiliary branch's query \(\mathcal{Q}_{aux}\) interacts with the target branch's key \(\text{K}_{tar}\) and value \(V_{tar}\), while the target branch's query \(\text{Q}_{tar}\) also interacts with the auxiliary branch's key \(\text{K}_{aux}\) and value \(V_{aux}\). Unlike previous work, our proposed DCAB facilitates mutual interaction between the features of different modalities, enabling the learning of complementary information across modalities and allowing for the simultaneous reconstruction of images for both modalities, as shown in Figure 5. In addition, DCAB is used as the basic building block of MMFIM, i.e., \(\text{MMFIM}=(\text{DCAB}_{1},\text{DCAB}_{2},\cdots,\text{DCAB}_{k})\), where \(k=1,2,\cdots,\text{K}\) and K is set to 3 in this paper. As shown in Figure 2, the input of MMFIM comes from the output of the decoder part in the dual-branch learning architecture, i.e., \(I^{i}_{aux}\in\mathbb{R}^{\text{H}\cdot 2^{-1}\times\text{W}\cdot 2^{-1} \times\text{C}/2^{-1}}\) and \(I^{i}_{tar}\in\mathbb{R}^{\text{H}\cdot 2^{-1}\times\text{W}\cdot 2^{-1} \times\text{C}/2^{-1}}\), where \(i=1,2,3\). First, the input feature maps \(I^{i}_{aux}\) and \(I^{i}_{tar}\) are normalized and then, unlike traditional cross-attention with its use of linear projection before each attention block, we use deep convolution to generate query matrices \(\mathcal{Q}_{aux}\) and \(\mathcal{Q}_{tar}\), keyword matrices \(\text{K}_{aux}\) and \(\text{K}_{tar}\), and value matrices \(\text{V}_{aux}\) and \(\text{V}_{tar}\) for 2D normalized feature mapping mapping. The process can be expressed as:  
\[\mathcal{Q}^{i}_{aux}=I^{i}_{aux}\mathcal{W}^{q}_{aux},\text{ K}^{i}_{aux}=I^{i}_{aux}\mathcal{W}^{k}_{aux},\text{ V}^{i}_{aux}=I^{i}_{aux}\mathcal{W}^{v}_{aux} \tag{4}\]  
\[\mathcal{Q}^{i}_{tar}=I^{i}_{tar}\mathcal{W}^{q}_{tar},\text{ K}^{i}_{tar}=I^{i}_{tar}\mathcal{W}^{k}_{tar},\text{ V}^{i}_{tar}=I^{i}_{tar}\mathcal{W}^{v}_{tar} \tag{5}\]  
Figure 4: Detailed structure of the three types of attention (channel, spatial, and gate attention). where \(\text{W}_{aux/tar}^{q}\), \(\text{W}_{aux/tar}^{k}\), \(\text{W}_{aux/tar}^{v}\) are the weight matrices of the convolution operations used to generate query, key, and value matrices for the auxiliary and target modalities, respectively. Different from the traditional cross-attention, in the proposed DCAB, one branch generates a query matrix \(\text{Q}_{aux/tar}\) and uses \(\text{Q}_{aux/tar}\) to query the \(\text{K}_{tar/aux}\) and \(\text{K}_{tar/aux}\) of the other branch through attention. This method ensures that the features of different branches can effectively interact and fuse, thereby capturing the complementary information between multi-modalities. The process can be expressed as:  
\[I_{aux}^{sx}=softmax\left(\frac{\text{Q}_{aux}\text{K}_{tar}^{\text{T}}}{ \sqrt{d}}\right)\text{V}_{tar} \tag{6}\]  
\[I_{tar}^{sx}=softmax\left(\frac{\text{Q}_{tar}\text{K}_{aux}^{\text{T}}}{ \sqrt{d}}\right)\text{V}_{aux} \tag{7}\]  
where \(I_{aux}^{sx}\), \(I_{tar}^{sx}\) are the features of the auxiliary modality and the target modality after cross-attention, and \(d\) is the embedding dimension of the attention head. In the proposed DCAB, we also have two trainable parameters, \(\alpha\) and \(\beta\), where the model can dynamically adjust the attention weights to optimize the weight configuration during training, i.e., \(\alpha I_{aux}^{sx}\) and \(\beta I_{tar}^{sx}\).