# Cross-Modal Adaptive Interaction Network for RGB-D Saliency Detection
## 3 Methodology
### Modal Integration Network
#### 3.3.1 Cross-Modal Feature Integration Module  
Because RGB data focus on the expression of color and depth data focus on showing the spatial information of objects, they cannot be simply integrated. Therefore, we propose the CMF module to integrate and enhance cross-modal features. As shown in Figure 4, \(1\times 1\) convolutions are applied to the two inputs, respectively, reducing the number of channels to half of the original to speed up the model. To reduce the redundancy of single-modal features and learn the complementarity of the two modalities, we first use a cross-refinement enhancement strategy, employing attention mechanisms to automatically select and cross-refine important image features. Specifically, combining the unique characteristics of the two modalities, we apply channel attention and spatial attention to them, respectively, to highlight the characteristic responses of salient regions. This process can be described as  
\[f_{out}^{r}=f_{in}^{r}\bigotimes\sigma(conv1(GAP(f_{in}^{r}))+conv1(GMP(f_{in}^{ r}))) \tag{1}\]  
\[f_{out}^{d}=f_{in}^{d}\bigotimes\sigma\Big{(}conv2\Big{(}concat\Big{(}GAP \Big{(}f_{in}^{d}\Big{)},GAP\Big{(}f_{in}^{d}\Big{)},dim=channel\Big{)}\Big{)} \Big{)} \tag{2}\]  
Figure 4: This figure depicts the structure of the CMF module, “CH” denotes the channel attention mechanism, “SP” denotes the spatial attention mechanism, “\(\times\)” and “\(+\)” denote element-wise multiplication and addition, respectively, and “C” denotes feature concatenation. Among them, \(f_{in}^{r}\) and \(f_{in}^{d}\) represent the input features of the RGB branch and the depth branch after 1 \(\times\) 1 convolution, respectively. Global Average Pooling is represented by GAP, and Global Maximum Pooling is represented by GMP. _convi_ (i = 1, 2) represents the convolution layers, \(\bigotimes\) represents element-wise multiplication, \(\sigma\) represents the ReLU activation function, _concat_ represents the concatenation operation in the channel dimension, and \(f_{out}^{r}\) and \(f_{out}^{d}\) represent the RGB and depth features after attention selection (i.e., \(f_{r}^{i}\) and \(f_{d}^{i}\)), respectively. Inspired by the success of the self-attention mechanism [42], as shown in Figure 5, we designed two submodules, DA and RA, to cross-enhance and refine modal features from a cross-modal perspective. Taking DA as an example, since the depth image can provide spatial and positional information for the RGB image, the features of the depth image are used to generate a weight for the RGB image and multiply it to achieve the purpose of enhancing the features of the RGB image. Specifically, first, use 1 \(\times\) 1 convolution to map \(f_{d}^{i}\) to \(W_{q}\in R^{C_{1}\times(H\times W)}\) and \(W_{k}\in R^{C_{1}\times(H\times W)}\) and map \(f_{r}^{i}\) to \(W_{v}\in R^{C\times(H\times W)}\). The number of channels, height, and width of the input feature map are represented by C, H, and W. The value of C1 is set to 1/6 of C to speed up the model operation. The results of these mappings are used to calculate the enhanced features. The attention weight matrix \(W_{a}\) is obtained by multiplying the inputs \(W_{k}\) and \(W_{k}\) and then applying the \(softmax\) function. The size of this matrix is (HW) \(\times\) (HW), which corresponds to the attention weights of each position in the input feature map. Next, the attention weight matrix \(W_{a}\) is multiplied by the projected \(W_{v}\) matrix to obtain the enhanced feature vector \(f_{da}\). Finally, the enhanced feature vector \(f_{da}\) is transformed back to the same shape as the original input feature map, that is, C \(\times\) H \(\times\) W. The process can be described as  
\[W_{a}=softmax\Big{(}W_{q}^{T}\bigotimes W_{k}\Big{)} \tag{3}\]  
\[f_{da}=W_{v}\bigotimes W_{a} \tag{4}\]  
The other modules, RA and DA, are complementary and are designed to use RGB image features to generate spatial weights that provide rich texture and color information for depth image features.