# A Deep Learning-Based Crop Disease Diagnosis Method Using Multimodal Mixup Augmentation
## 1 Introduction  
Crop diseases pose significant challenges, as they contribute directly to reduction in both crop yield and quality. Hence, the early detection, treatment, and prevention of crop diseases play a vital role in safeguarding farmers' interests. There are several methods for diagnosing crop diseases, including diagnostic kits [1; 2; 3], genetic analysis [4; 5], and the visual observation of symptoms [6]. Symptoms manifested on leaves, stems, and fruits serve as visual indicators for identifying diseased crops. However, the direct identification and diagnosis of crop diseases by human eyes face limitations due to the diverse types and varying progression of the diseases observed in different crops. Consequently, numerous automatic crop disease diagnosis technologies employing computer vision and deep learning have been developed.  
Deep learning has demonstrated excellent performance compared to other image processing methods in automated disease diagnosis. [PERSON] et al. achieved a 99.53% accuracy rate in classifying 58 crop diseases using VGG networks [7], and [PERSON] et al. employed the Resnet for classifying crop diseases collected via mobile devices [8]. In addition, many studies have been conducted to compare the performance based on model architectures or to improve the performance by modifying the model structures [9; 10; 11]. [PERSON] et al. [12] and [PERSON] et al. [13] corrected low-quality crop images in real environments and used data augmentation methods that simulate real-world conditions to train deep learning models for crop disease diagnosis that perform well in actual environments.  
The increase in adoption of smart farms, combined with recent advancements in IoT technology, has facilitated the collection of a broad spectrum of time-series data including temperature, humidity, and CO\({}_{2}\) concentration. Not all environmental data refer to time-series data, but in this study, we assume that environmental data refer to time-series data. [PERSON] et al. utilized the KNN (K-Nearest Neighbor) classifier and the ensemble method to predict crop diseases based on measured temperature data [14]. [PERSON] et al. designed a system equipped with sensors for the real-time measurement of temperature and humidity, employing a decision tree algorithm to predict the suitability of the environment for crop cultivation [15].  
Previous studies have focused mostly on utilizing either image data or environmental data alone. Currently, few studies predict crop diseases using multimodal data, which comprise diverse types of information with distinct characteristics. Furthermore, there is a lack of research on data augmentation techniques to compensate for insufficient data in methods utilizing time-series data.  
To address these issues, we propose a deep learning model that utilizes both crop images and environmental data, including temperature, humidity, and dew point, to predict crop diseases. The assumption that image data and environmental data for crops can play a complementary role in disease prediction is inherent in this approach. For crop leaves affected by powdery mildew, if the image quality is poor or the size of the powdery mildew spots is too small, a deep learning model might incorrectly classify the crop as healthy. However, if environmental variables such as temperature and humidity are also provided, it could help make a more accurate diagnosis. We will experimentally confirm that this assumption is correct in Section 5.  
We conduct a comparative analysis of disease prediction performance for the one utilizing either image data or environmental data alone, and for utilizing multimodal data. We also introduce a multimodal mixup augmentation method, which enables the simultaneous application of crop images and environmental data and experimentally demonstrates its effectiveness in improving prediction performance.