# A Deep Learning-Based Crop Disease Diagnosis Method Using Multimodal Mixup Augmentation
## 3 Materials and Methods
### Multimodal Mixup Augmentation  
Equation (1) describes the newly generated data by unimodal mixup augmentation. \(x_{1}\) and \(x_{2}\) represent randomly sampled image data, and \(y_{1}\) and \(y_{2}\) represent the corresponding labels. \(\lambda\) refers to the weight used when mixing two data, and it is sampled from a beta distribution with parameters \(\alpha\) and \(\beta\). That is, \(\lambda\)\(\sim\)Beta (\(\alpha\), \(\beta\)) for a positive constant \(\alpha\). For convenience, we used both \(\alpha\) and \(\beta\) values of 1.0. This implies that \(\lambda\) is sampled from a uniform distribution over the interval [0, 1]. Then, the augmented data \(\overline{x}\) and \(\overline{y}\) can be generated by linearly interpolating each raw data and label.  
\[\overline{x}=\lambda\mathbf{x}_{1}+(1-\lambda)\mathbf{x}_{2}\ \text{\emph{and}}\ \overline{y}=\lambda\mathbf{y}_{1}+(1-\lambda)\mathbf{y}_{2} \tag{1}\]  
We can also infer that the change in the value of lambda affects the intensity and impact of data augmentation. As \(\lambda\) approaches 0.5, the augmentation becomes stronger, while approaching 0 or 1 makes the augmentation weaker. Also, as the \(\alpha\) value decreases, the mixup augmentation effect decreases.  
To apply multimodal mixup augmentation properly, we added the augmented environmental data \(\overline{z}\) as Equation (2). \(z_{1}\) and \(z_{2}\) represent environmental data, with the corresponding labels \(y_{1}\) and \(y_{2}\).  
\[\overline{z}=\lambda\mathbf{z}_{1}+(1-\lambda)\mathbf{z}_{2} \tag{2}\]  
If the data are multimodal, it is desirable to generate augmented data by linearly combining all modality data. For example, if mixup is performed only for the image modality as in the conventional method, the synthesized labels may play the role of noisy labels for the sequence model that takes environmental data as input, potentially leading to unfavorable learning outcomes.  
To linearly interpolate each sampled data, it is necessary to first align their data sizes. The size of the image data sampled was equal; however, the length of the environmental data varied. In general, this can be achieved through resampling, interpolation, or other methods to match the predetermined size. In this study, a crop or padding strategy was used for the convenience of implementation. When the length of the environmental data was excessively long, we cropped them--fixing the size at 320--whereas when they were too short, we applied zero-padding to ensure it reached a length of 320.  
Figure 3 shows an example of a proposed mixup augmentation for multimodal data. The top images represent crop images, while the bottom graphs represent environmental  
Figure 2: The deep learning model architecture for diagnosing crop diseases by analyzing crop image and environmental data.  
data normalized with min-max normalization. Figure 3a represents Paprika-Deficiency (P)-Early, (b) shows Paprika-Normal, and (c) illustrates the results of mixup augmentation. The multimodal data from (a) and (b) were linearly interpolated with a weight of 0.5 each, resulting in synthesized data (c). The synthesized data have labels of 0.5 \(\times\) Paprika-Deficiency (P)-Early' and 0.5 \(\times\) Paprika-Normal'.