# Research on Negative Road Obstacle Detection Based on Multimodal Feature Enhancement and Fusion
## 3 The Proposed Network
### Feature Enhancement Module  
In order to enhance the expressive power of the encoder to extract features, this paper designs a feature enhancement module between the encoder and the decoder, as shown in Figure 2. This module combines the ASPP and multi-layer perceptron (MLP) structures to enhance and supplement the multi-level features extracted by the encoder from RGB and depth images, considering that, as the depth of the convolutional network increases, the degree of feature abstraction gradually increases, and the spatial resolution gradually decreases. We embed feature enhancement modules in the four stages of the decoder to fully enrich the feature representation capabilities of the encoder. The deep features processed by the feature enhancement module are fused with the shallow features through skip connections and finally upsampled to a resolution consistent with the input image.  
In the four stages of the encoder, the features of RGB and depth modality of each scale are processed by the ASPP module and the MLP, respectively. The ASPP module performs multi-scale feature extraction using atrous (dilated) convolution, enabling it to preserve local detail information while expanding the receptive field to capture global context. By using multiple dilation rates, the ASPP effectively balances the trade-off between local precision and global representation, ensuring that features at various spatial scales are integrated seamlessly. On the other hand, the MLP enhances the semantic expressiveness of the features through nonlinear transformations, which improve the network's ability to capture high-level abstractions and interdependencies between features. As shown in Equations (1) and (2), the multi-scale spatial features extracted by ASPP and the deep semantic features generated by the MLP are spliced in the channel dimension to generate a composite feature map with both local details and global semantics. Subsequently, the spliced features are upsampled using bilinear interpolation to recover the spatial resolution to a size close to that of the input image, while keeping the spatial structure intact.  
\[\hat{F}_{i}^{r}=Up(ASPP(F_{i}^{r})\oplus MLP(F_{i}^{r})) \tag{1}\]  
\[\hat{F}_{i}^{d}=Up(ASPP(F_{i}^{d})\oplus MLP(F_{i}^{d})) \tag{2}\]  
\(Up()\) is an upsampling operation that scales all features to the same resolution, and \(F_{i}\) denotes the feature map obtained from the encoder.  
As shown in Equation (3), the number of channels increases after splicing the RGB and depth modal features, and the spatial resolution remains unchanged. Ultimately, the feature representation is further optimized by linear fusion to generate a multimodal fused feature output.  
Figure 2: The feature extraction module.  
\[F_{fusion}=W\cdot\Phi(\mathop{\oplus}\limits_{i=1}^{4}(\hat{F}_{i}^{r},\hat{F}_{i}^ {d})) \tag{3}\]  
\(W\) denotes the weight matrix of the linear fusion layer, which is responsible for learning the optimal combination of features from different scales. The splicing operation is performed along the channel dimension, where features from multiple scales are concatenated to form a comprehensive feature representation. With this formula, we splice all scales of features \(F\) on the channel dimension and fuse them through the linear layer \(W\) to generate the final fused features \(F_{fusion}\). The linear layer not only integrates features effectively by learning their correlations but also balances the contributions of different scales through its trainable weights. This process not only ensures high-resolution restoration but also makes full use of the multimodal information and semantic details to improve the model's multi-scale and complex semantic understanding of multi-scale and complex semantic features.