# A Deep Learning-Based Crop Disease Diagnosis Method Using Multimodal Mixup Augmentation
## 5 Results
### Comparative Analysis of Crop Disease Diagnosis Performance
Table 3 shows the result from crop disease diagnosis with respect to mixup augmentation. Using multimodal mixup augmentation resulted in the best performance, regardless of CNN model types. This implies that the proposed multimodal mixup augmentation method shows generalized performance not limited to specific model architectures. Figure 4: Crop disease prediction result by data modality—Case I: When using image data, crop type was predicted correctly, but disease type was not. However, both crop and disease type were correctly predicted when using environmental data and multi-modal data respectively. Figure 5: Crop disease prediction result by data modality—Case II: When using environmental data, both crop and disease type were misclassified. However, both crop and disease type were correctly predicted when using image data and multi-modal data, respectively. In Figures 6 and 7, the confusion matrix illustrates the predictions of a deep learning model trained with different data modality. In the confusion matrix, a square box drawn with bold blue lines means that the labels in the box belong to the same crops but have normal status or different disease severity. For example, in the box located at the top left corner, there are two labels for normal and powdery mildew with intermediate severity for tomato. The intensity of the box color represents the number of data. The darker blue, the more data it represents, and the lighter one, the fewer data it represents. The \(F1\) score for crop disease classification performance is 89.5% when only image data are used, which is 8.6%p higher than when only environmental data are used. On the other hand, when the type of crop is correctly predicted, there are quite a few cases where the severity of disease is also correctly predicted. From the facts above, we conclude that environmental data alone do not have a significant impact on crop type classification, but they provide sufficient information to classify crop diseases. When only image data are used, most of the types of crops are well predicted. However, there are quite a few cases where, even though the type of crop is correctly predicted, the type and severity of the disease were incorrectly predicted. Consequently, environmental data and image data seem to be able to play complementary roles in deep learning model learning. In fact, using multimodal data, as shown in Table 2, resulted in more than 2%p performance improvement compared to using unimodal data. Figures 8 and 9 show the confusion matrix for two mixup augmentation methods, that is, image-only mixup and multimodal mixup. Incorporating both environmental and image data notably enhances performance in both cases compared to using data from a single modality. In particular, when the mixup augmentation method is applied to both image and \begin{table}
\begin{tabular}{c c c c} \hline \hline
**CNN Model** & **Without Mixup** & **Image-Only Mixup** & **Multimodal Mixup** \\ \hline Resnet 50 & 90.37 \(\pm\) 0.59 & 90.93 \(\pm\) 0.34 & **92.26 \(\pm\) 0.79** \\ DenseNet 121 & 92.02 \(\pm\) 0.74 & 92.08 \(\pm\) 0.42 & **92.66 \(\pm\) 0.39** \\ Xception & 90.68 \(\pm\) 0.33 & 90.97 \(\pm\) 0.90 & **91.95 \(\pm\) 0.56** \\ MobileNet V3 & 90.56 \(\pm\) 0.46 & 91.17 \(\pm\) 0.80 & **91.94 \(\pm\) 0.64** \\ EfficientNet V2-Small & 92.15 \(\pm\) 1.18 & 90.85 \(\pm\) 0.46 & **93.22 \(\pm\) 0.61** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Crop disease diagnosis \(F1\) score (%) for the testing dataset by data augmentation method. Figure 6: Confusion matrix by data modality: environmental data (\(F1\) score 80.9%). environmental data, the recall score improved by 2.7%p compared to the image-only mixup method. The increase in the recall score indicates improved disease diagnosis performance when crops are diseased. The LSTM module demonstrates resistance to overfitting, and its generalization capability is enhanced through the multimodal mixup augmentation. Figure 8: Confusion matrix by augmentation method: image-only mixup (\(F1\) score 90.9%).