# A Deep Learning-Based Crop Disease Diagnosis Method Using Multimodal Mixup Augmentation
## 3 Materials and Methods
### Multimodal Deep Learning  
The deep learning model architecture used in this study is illustrated in Figure 2. This architecture includes an LSTM (long short-term memory) module for processing environmental data and a CNN (convolutional neural network) for image data. For the CNN model, we used a pretrained Resnet 50 [27] from the ImageNet dataset [28], which is one of the most commonly used networks for image-based deep learning research. After passing through the CNN, crop images are embedded into low-dimensional features with a dimension of 25. Environmental data include temperature, humidity, and dew point. These data are concatenated and passed through an LSTM module, which are then mapped to a  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Crop** & **Disease** & **Severity** & \begin{tabular}{c} **The Number of** \\ **Samples** \\ \end{tabular} \\ \hline \hline \multirow{2}{*}{Strawberry} & \multicolumn{1}{c}{Normal} & \multicolumn{1}{c}{810} \\ \cline{2-4}  & \multicolumn{1}{c}{Normal} & \multicolumn{1}{c}{143} \\ \cline{2-4}  & \multicolumn{1}{c}{Powdery Mildew} & \multicolumn{1}{c}{Intermediate} & 189 \\ \hline \multirow{6}{*}{Paprika} & \multicolumn{1}{c}{Normal} & \multicolumn{1}{c}{1177} \\ \cline{2-4}  & \multicolumn{1}{c}{Powdery Mildew} & \begin{tabular}{c} Early \\ Intermediate \\ Terminal \\ \end{tabular} & \
\begin{tabular}{c} 154 \\ 111 \\ 42 \\ \end{tabular} \\ \cline{2-4}  & \multicolumn{1}{c}{Ca} & \multicolumn{1}{c}{166} \\  & N & \multicolumn{1}{c}{Early} &
\begin{tabular}{c} 142 \\ 156 \\ 153 \\ \end{tabular} \\ \cline{2-4}  & \multicolumn{1}{c}{K} & \multicolumn{1}{c}{153} \\ \hline \hline \multicolumn{2}{c}{Cucumber} & \multicolumn{1}{c}{Normal} & \multicolumn{1}{c}{917} \\ \hline \hline \multirow{6}{*}{Chili} & \multicolumn{1}{c}{Normal} & \multicolumn{1}{c}{69} \\ \cline{2-4}  & \multicolumn{1}{c}{Anthracnose} & \multicolumn{1}{c}{Intermediate} & 99 \\ \cline{2-4}  & \multicolumn{1}{c}{N} & \multicolumn{1}{c}{} & 148 \\ \cline{2-4}  & \multicolumn{1}{c}{P} & \multicolumn{1}{c}{Early} & 159 \\ \cline{2-4}  & \multicolumn{1}{c}{K} & \multicolumn{1}{c}{157} \\ \hline \hline \multirow{6}{*}{Grape} & \multicolumn{1}{c}{Normal} & \multicolumn{1}{c}{828} \\ \cline{2-4}  & \multicolumn{1}{c}{Anthracnose} & \multicolumn{1}{c}{Early} & 40 \\ \cline{2-4}  & \multicolumn{1}{c}{Intermediate} & 12 \\ \cline{2-4}  & \multicolumn{1}{c}{Powdery Mildew} & \multicolumn{1}{c}{Early} & 13 \\ \cline{2-4}  & \multicolumn{1}{c}{Intermediate} & 29 \\ \cline{2-4}  & \multicolumn{1}{c}{Sunscald} & \multicolumn{1}{c}{Early} & 18 \\ \cline{2-4}  & \multicolumn{1}{c}{Intermediate} & 14 \\ \cline{2-4}  & \multicolumn{1}{c}{Corky Core} & \multicolumn{1}{c}{Early} & 21 \\ \cline{2-4}  & \multicolumn{1}{c}{Total} & \multicolumn{1}{c}{5767} \\ \hline \hline \end{tabular}
\end{table}
Table 1: Class types and the number of samples utilized for crop disease classification. N, P, K, and Ca denote deficiencies in the macronutrient elements nitrogen, phosphorus, potassium, and calcium, respectively.  
feature with a dimension of 200. Next, they are reduced to a low-dimensional feature with a dimension of 25, similar to the image data, by passing through two fc (fully connected) layers. The low-dimensional features from each modality are then concatenated to create a single dimensional feature, which is passed through a final fc layer to generate scores for 25 classes. To prevent overfitting, dropout layers were added at the output of the CNN and at the final layer of the LSTM.