# Multimodal Natural Disaster Scene Recognition with Integrated Large Model and Mamba
## 1 Introduction  
The sudden and uncontrollable nature of natural disasters makes them a significant obstacle to human development [1]. According to the global disaster database EM-DAT, maintained by KU Leuven University in Belgium, 366 major natural disasters occurred worldwide in 2023, resulting in 82,151 fatalities, approximately 79.39 million injuries, and economic losses totaling USD 159.8 billion. These events not only pose immediate threats to human life and cause significant economic damage but also trigger secondary disasters that can devastate ecosystems and create long-term public health risks. Therefore, the timely and accurate identification of disaster scenarios is crucial in preventing secondary disasters and aiding post-disaster relief efforts. However, obtaining such information during emergencies is labor-intensive and costly, often requiring manual data processing and expert evaluation [2]. To alleviate this, computer vision techniques have been applied to satellite images, synthetic aperture radar, and other remote sensing data [3, 4, 5]. Unfortunately, these approaches remain costly to deploy and lack robustness in time-critical situations. Large-scale pre-trained models offer an effective solution. Through pre-training and fine-tuning paradigms, large language models (LLMs) [6] excel in a variety of tasks, such as sentiment analysis [7], image recognition [8], text completion [9], and cross-lingual translation [10]. In 2023, Google DeepMind introduced Gemini1.5, a high-performance multimodalmodel series that incorporates state-of-the-art innovations in sparse and dense scaling, training infrastructure, and distillation, achieving breakthroughs in efficiency, reasoning, planning, multilingual processing, function calling, and long-context performance [11]. Gemini1.5 also surpasses contemporary LLMs in recall and reasoning capabilities. However, the high computational costs associated with training GPT-4-level models make it nearly impossible for small enterprises to train such models directly for their business needs. In contrast, leveraging pre-trained large models can significantly reduce the operational costs. Additionally, due to the inherent limitations in the generative capabilities of chatbot models, they face challenges in rapidly recognizing images from disaster scenarios. Traditional visual models, such as ResNet and EfficientNet, remain superior for tasks like image classification and visual perception [12]. Therefore, utilizing pre-trained LLMs to enhance disaster prediction performance appears to be a more practical and cost-effective approach. In traditional supervised training paradigms, vision models learn visual representations solely from pixel data and labels, neglecting information from other modalities. In this paper, we propose a novel learning framework, the Mamba Multimodal Disaster Recognition Network (Mamba-MDRNet), inspired by the GPT4 Image model by [PERSON] et al. [13]. Our approach leverages knowledge from pre-trained large models to enhance the multimodal learning capabilities of conventional vision models, thereby improving the performance in perception tasks such as image classification. Specifically, we first utilize the multimodal understanding of pre-trained LLMs through dialog to generate detailed, high-quality descriptions and predicted probabilities for each image in the dataset. These probabilities sum to one across all classes. We then process the descriptive text using a convolutional neural network (CNN) [14] to extract text feature vectors, and the predicted probabilities are transformed into probabilistic feature vectors using a multi-layer perceptron (MLP) [15]. During training, in addition to supervision from ground-truth labels, we employ a unified attention mechanism to fuse multimodal data--images, text, and probabilities--enhancing the model's ability to capture relationships across modalities. This enables the transfer of the image understanding knowledge from LLMs to traditional visual perception models.