# Multimodal Natural Disaster Scene Recognition with Integrated Large Model and Mamba
## 3 Methods
### Mambu Multimodal Disaster Recognition Network (Mamba-MDRNet)
#### 3.3.1 Feature Extraction Module  
1. Image Processing Module (EfficientNet) This module is primarily designed to extract feature information from images. In our experiments, we explored several visual models for feature extraction, using Efficient  
Figure 2: Generated by Gemini1.5 using prompts. Net as a representative example to demonstrate how the module captures features from the images. Specifically, we leveraged the pre-trained EfficientNet-B0 network, utilizing the weights trained on the ImageNet dataset. The input to EfficientNet is image data \(I\in\mathbb{R}^{H\times W\times C}\), where \(H\) and \(W\) represent the height and width of the image, and \(C\) denotes the number of channels. In the experiments, we set the default image dimensions as \(H=224\), \(W=224\), and \(C=3\). EfficientNet-B0 generates feature maps through convolution operations, which can be defined as follows:  
\[X_{I}=Conv(I)\in\mathbb{R}^{N\times D} \tag{5}\]  
Here, \(N\) represents the number of feature maps, and \(D\) is the embedding dimension. EfficientNet's network architecture uses a compound scaling method to expand the model's depth, width, and resolution in tandem, enhancing its representational capacity while maintaining computational efficiency. During this process, convolutions extract local features from the image and embed them as high-dimensional feature vectors. After the convolution operation, the feature map \(X_{I}\) is passed through a global average pooling (GAP) layer for feature aggregation, resulting in a global embedding vector \(H_{I}\):  
\[H_{I}=\text{GAP}(X_{I})\in\mathbb{R}^{D_{E}} \tag{6}\]  
where \(D_{E}=1280\) denotes the final embedding dimension of EfficientNet-B0. This embedding vector represents the global semantic information of the input image. Subsequently, \(H_{I}\) is further mapped to the output space through the network's fully connected layer, with the class probabilities defined as  
\[O_{I}=W_{\text{cls}}\cdot H_{I}^{T}+b_{\text{cls}},\quad O_{I}\in\mathbb{R}^{ n_{\text{class}}} \tag{7}\]  
where the weight matrix \(W_{\text{cls}}\in\mathbb{R}^{11\times 1280}\), the bias vector \(b_{\text{cls}}\in\mathbb{R}^{11}\), and \(n_{\text{class}}=11\). Here, \(O_{I}\) represents the output results for the classification task. In the designed image feature extraction module, we employ EfficientNet-B0, modifying only its final layer to output predictions for 11 classes. 2. Text Processing Module (TextCNN) The text processing module utilizes a CNN for feature extraction. The input text is first represented through an embedding layer. Let \(V\) denote the vocabulary size of the input text and \(d_{emb}\) represent the embedding dimension. The embedded text can be expressed as \[X_{T}=\text{Embedding}(T)\in\mathbb{R}^{L\times d_{emb}}\] (8) where \(L\) is the maximum length of the input text. During the experiments, the embedding dimension \(d_{emb}\) was set to 100. Next, embedded text features \(X_{T}\) are processed through multiple 1D convolutional kernels of varying sizes to extract local features. The output for each convolutional layer is as follows:  
\[C_{i}=\text{ReLU}(\text{Conv}\text{1D}(X_{T},k_{i}))\in\mathbb{R}^{L\times n_{ i}} \tag{9}\]  
Global features for each kernel are obtained through max-pooling:  
\[P_{i}=\text{MaxPool}(C_{i})\in\mathbb{R}^{n_{i}} \tag{10}\]  
In the designed model, the kernel sizes of the convolutional layers are set to \(k_{i}\in\{3,4,5\}\), and the number of convolutional kernels is defined as \(n_{i}=150\). The outputs from all convolutional layers are concatenated to form the final text feature representation:  
\[H_{T}=\text{Concat}(P_{1},P_{2},P_{3})\in\mathbb{R}^{\mathbb{D}^{n_{i}}} \tag{11}\]  
To prevent overfitting, \(H_{T}\) undergoes dropout regularization before being fed into the fully connected layer:  
\[H_{T}=\text{Dropout}(H_{T},p=0.1) \tag{12}\]  
where the dropout rate \(p\) is set to 0.1 in our experiments.