# Multimodal Network for Object Detection Using Channel Adjustment and Multi-Scale Attention
## 3 Methodology
### Multi-Scale Activated Attention Mechanism  
The multi-scale activated attention mechanism improves the model's ability to capture both local and global contexts by applying dilated convolutions at multiple scales within the attention framework. This enables the effective extraction of multi-scale features from input images through adjusted receptive fields.  
To implement this mechanism, we perform convolutions on feature maps \(x_{1}^{\prime}\) and \(x_{2}^{\prime}\) from channel-exchanged modalities, generating queries (\(q\)), keys (\(k\)), and values (\(v\)) for each. A standard convolution generates \(q\), while dilated convolutions with rates \(r=1,2\), and \(3\) produce \(k\) and \(v\). This retains channel dimensions and results in three sets of \(q\), \(k\), and \(v\) per modality.  
\[q_{i,r}=\mathrm{Conv}(x_{i}^{\prime}),\quad k_{i,r}=\mathrm{DilateConv}_{r}(x_{ i}^{\prime}),\quad v_{i,r}=\mathrm{DilateConv}_{r}(x_{i}^{\prime}) \tag{6}\]  
where \(q_{i,r}\), \(k_{i,r}\), and \(v_{i,r}\) represent the query, key, and value at scale \(r\) for each modality \(i\) (with \(r=1,2,3\)). Here, \(\mathrm{Conv}(\cdot)\) denotes a standard convolution operation, while \(\mathrm{DilateConv}_{r}(\cdot)\) signifies a dilated convolution with dilation rate \(r\).  
For each scale \(r\), the attention score \(\alpha_{i,r}\) is computed via a dot product between the queries and keys, followed by a scaling operation:  
\[\alpha_{i,r}=\mathrm{Softmax}\left(\frac{q_{i,r}k_{i,r}^{\top}}{\sqrt{d}}\right) \tag{7}\]  
\[\mathrm{head}_{i,r}=\alpha_{i,r}v_{i,r} \tag{8}\]  
where \(d\) represents a scaling factor, typically defined as \(C/h\), denoting the channel dimension per attention head. The attention score \(\alpha_{i,r}\) is then applied to the corresponding value \(v_{i,r}\), yielding the output of each attention head.  
The attention heads from different scales \(r=1,2,3\) are concatenated along the channel dimension to form a multi-scale attention feature representation for each modality:  
\[x_{i,\mathrm{concat}}=\mathrm{Concat}(\mathrm{head}_{i,1},\mathrm{head}_{i,2}, \mathrm{head}_{i,3}) \tag{9}\]  
Next, the concatenated feature maps from both modalities are processed in two ways: both are subjected to element-wise summation to produce \(x_{\mathrm{sum}}\) and element-wise multiplication to generate \(x_{\mathrm{mul}}\). A spatial attention mechanism is then applied to \(x_{\mathrm{sum}}\) to enhance spatial dependencies:  
\[x_{\mathrm{spatial}}=\sigma\big{(}\mathrm{Conv}\big{(}\mathrm{Concat}\big{(} \mathrm{F}_{\mathrm{avg}}(x_{\mathrm{sum}}),\mathrm{F}_{\mathrm{max}}(x_{ \mathrm{sum}})\big{)}\big{)}\big{)}\odot x_{\mathrm{sum}} \tag{10}\]  
where \(\sigma\) denotes the activation function, \(\mathrm{Conv}(\cdot)\) represents a convolutional layer, and \(\mathrm{F}_{\mathrm{avg}}\) and \(\mathrm{F}_{\mathrm{max}}\) refer to the average and maximum pooling operations, respectively.  
The spatially enhanced representation \(x_{\mathrm{spatial}}\) undergoes channel self-attention to capture inter-channel dependencies, resulting in the final fused representation \(x_{\mathrm{fused}}\):  
\[x_{\mathrm{fused}}=\sigma\Big{(}W_{\mathrm{sa}}\Big{(}F_{\mathrm{avg}}(x_{ \mathrm{spatial}}\odot x_{\mathrm{mul}})+F_{\mathrm{max}}(x_{\mathrm{spatial}} \odot x_{\mathrm{mul}})\Big{)}\Big{)} \tag{11}\]  
where \(W_{\mathrm{sa}}\) are learnable parameters; the symbol \(\odot\) denotes element-wise multiplication.  
The final fused representation \(x_{\mathrm{fused}}\) integrates multi-scale, spatial, and channel-level information. This enriched representation then undergoes five additional processing stages before entering the Feature Pyramid Network (FPN), ensuring a seamless transition and coherence throughout the architecture.