# A Deep Learning-Based Crop Disease Diagnosis Method Using Multimodal Mixup Augmentation
## 5 Results  
### Comparative Analysis of Crop Disease Diagnosis Performance  
Table 2 presents the classification performance of crop disease diagnosis for the testing dataset with respect to data modality. We conducted the same experiment three times, only varying the initial training conditions (e.g., random seed). Furthermore, we present the average \(F1\) score, along with its estimation under a 95% confidence interval, in Table 2.  
We expected that multimodal data would demonstrate a wider range of features compared to unimodal data, such as environmental or image data. As expected, the best classification performance was achieved when using multimodal data (e.g., average \(F1\) score of Resnet 50 is 90.37%). However, we also noticed that using only environmental data had a noticeable impact on the occurrence of crop diseases, as a performance of \(F1\) score 80.84% was observed. Additionally, we conducted iterative experiments by changing CNN models to show that the proposed method was not limited to a particular model. And, we confirmed once again that the performance was the best when environmental data and image data are used together. The models used were DenseNet 121 [30], Xception [31], MobileNetV3 [32], and EfficientNetV2-Small [33]. To improve image classification performance, these models reference the output of all previous layers (DenseNet 121), change the layer structure to increase accuracy while reducing computational cost (Xception, MobileNetV3), or systematically optimize the model's structure (EfficientNetV2-Small). Since these models have different architectures, they can extract different features and make different judgments, even from the same images. Our experiments with these different architectures demonstrate that the proposed method is robust across various CNN models. In Section 1, we assumed that image data and environmental data play a complementary role in predicting crop disease diagnosis. Here, we will confirm this assumption through two case studies of crop disease prediction results based on different data modalities. Figure 4 shows an image of a paprika leaf infected with intermediate-stage powdery mildew along with the associated environmental data. When only environmental data were used, the model accurately predicted the crop type and disease. However, when only images were used, the model correctly identified the crop type but incorrectly classified it as healthy. In fact, since the disease was in its early stage, the powdery mildew spores were barely visible in the images. When both image and environmental data were used, the model correctly identified both the crop type and the disease. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**CNN Model** & **Environmental Data** & **Image** & **Multimodal** \\ \hline Resnet 50 & & 87.79 \(\pm\) 0.70 & \(\textbf{90.37}\pm\textbf{0.59}\) \\ DenseNet 121 & & 90.60 \(\pm\) 0.46 & \(\textbf{92.02}\pm\textbf{0.74}\) \\ Xception & & 88.07 \(\pm\) 0.89 & \(\textbf{90.68}\pm\textbf{0.33}\) \\ MobileNet V3 & (LSTM) & 89.27 \(\pm\) 0.73 & \(\textbf{90.56}\pm\textbf{0.46}\) \\ EfficientNet V2-Small & & 90.34 \(\pm\) 0.53 & \(\textbf{92.15}\pm\textbf{1.18}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Crop disease diagnosis \(F1\) score (%) for the testing dataset by data modality. Figure 5 shows an image of chili in the early stage of macronutrient element N (nitrogen) deficiency, along with its environmental data. When only environmental data were used, the deep learning model misclassified the crop type and disease. However, when using only image data, the model accurately identified both the crop type and disease. When using both image and environmental data, the model also accurately identified the crop type and disease. These two cases show that while a deep learning model using a single modality might misclassify crop diseases, it predicted accurately when using multimodal data. This implies that multimodal data can complement each other and enhance the accuracy of crop disease diagnosis.