# Crop Disease Diagnosis with Deep Learning-Based Image Captioning and Object Detection
## 3 Materials and Methods
### Data Preprocessing  
#### 3.2.1 Data Preprocessing for Image Captioning  
In order for the Image Captioning model to generate detailed and accurate diagnostic sentences in images of disease-infected crops, appropriate diagnostic sentences which comprehensively take into account crop types, disease types, disease damage levels, and disease symptoms must be produced. To this end, websites of both AI-Hub and the National Crop Pest Management System were referred to in order to identify types of crops, diseases, disease damage, and disease symptoms, and a crop disease diagnosis sentence necessary for model training could be produced. The process of producing diagnostic sentences for each image, used in model training, is as follows. Referring to Metadata in AI-Hub  
AI-Hub also provides metadata about additional information of the crop images, as shown in Table 1.  
The metadata of the image is provided as a file in JSON form, as shown in Figure 3. Among the various categories that make up the metadata, the type of disease was identified in the 'disease' part, the type of disease-infected crops in the 'crop' part, and the degree of disease damage, such as early, mid, late stage, in the 'risk' part. Referring to Disease Keywords Provided by National Crop Pest Management System provides various information to prevent diseases of crops, such as the type of disease of crops and the timing of disease outbreaks [25]. Particularly, some information about a specific disease can be found there, which contains the types of crops that can be infected with the disease and various symptoms which can be identified from the infected crops. In the information describing the symptoms, the features that appear prominently as symptoms of the disease were identified, and these features were regarded as keywords for symptoms of the disease. For Example, if 'Anthracose', one of the names of diseases which can infect crops, is searched, several crops that \begin{table}
\begin{tabular}{c c c} \hline
**Item** & **Type** & **Code** & **Example of Code** \\ \hline Disease & Integer & 0, 1,..., 20 & 14 \\ Crop & Integer & 0, 1,..., 10 & 9 \\ Area & Integer & 0, 1,..., 7 & 3 \\ Grow & Integer & 11, 12, 13 & 12 \\ Risk & Integer & 0, 1, 2, 3 & 2 \\ Bounding Box Points & Dictionary & \{xtl, ytl, xbr, ybr\} &
\begin{tabular}{c} [\"xtl\": 100, \"ytl\": 200, \\ \"xbr\": 1100, \"ytr\": 1200] \\ \end{tabular} \\ \hline \end{tabular}
\end{table} 
Table 1: Examples of Metadata Information Provided by AI-Hub. Figure 3: Metadata in JSON form provided to each image. can be infected with the disease, such as peppers, are introduced. Among them, \"round spots\", \"dark or yellowish-brown spores\", and \"dry twisting\" can be identified as keywords for the symptoms of the disease. Using All the Information Gathered to Produce Crop Disease Diagnostic Sentences  
First of all, disease symptom keywords were allocated according to the type of disease and the degree of damage previously identified. As a result of comparing the images of various crops and the symptom keywords of the disease in which the crop was infected according to the degree of disease damage, it was confirmed that the number of keywords appearing in the crop image increased as the disease damage intensified. Reflecting this point, more symptom keywords were assigned to disease diagnosis sentences as the degree of disease damage in the crop image increased. For example, keywords according to the degree of disease damage of pepper anthracrose are as follows. * Early Stage: \"round spots\"
* Middle Stage: \"round spots\" + \"dark or yellowish-brown spores\"
* Late Stage: \"round spots\" + \"dark or yellowish brown spores\" + \"dry twisting\"  
Next, a disease diagnosis sentence including both symptoms and types of disease was produced using keywords allocated according to the degree of disease damage. For example, the disease diagnosis sentence for pepper anthracose is as follows. * Early Stage: A round spot appears on the pepper, which is suspected to be a red pepper anthracose.