# Crop Disease Diagnosis with Deep Learning-Based Image Captioning and Object Detection
## 3 Materials and Methods
### Data Preprocessing
#### 3.2.1 Data Preprocessing for Image Captioning
* Middle Stage: The pepper has yellowish brown spores and round spots, so it is suspected to be a pepper anthracose. * Late Stage: It is suspected that it is a pepper anthracose, as it appears to have circular spots, yellow-brown spores, and dry twist on the pepper. Sentence Augmentation for Model Training  
Finally, two text augmentation techniques, 'word order change' and 'back translation', were used to produce additional synonymous sentences to enable the model to learn sentences in various contexts. Word order change is a technique that changes the structure of a sentence by changing the order of sentence components. In Korean, the overall meaning of the sentence is generally maintained even if the order of the sentence components is changed. Using the characteristics of Korean, which has relatively flexible word order [26], various forms of sentences were produced by changing the positions of various sentence components that make up the sentence. Next, back translation is a technique that uses a machine translator to translate a sentence into another language and, then, translates it back into the language before it was translated to change the form of the sentence [27]. Many IT companies, such as Google and Naver, are providing machine translation services. As a result of directly comparing the quality of various machine translation services, the machine translation service called \"Papago\" provided by Naver tended to be more natural, in terms of foreign to Korean translation, than other services. Therefore, using Naver's 'Papago', an original diagnostic sentence was translated from Korean to English, translated from English to Japanese, and then translated from Japanese to Korean again to produce a new sentence. For model training, basically 5 disease diagnosis sentences were produced per image using disease symptom keywords, and then, word order change and back translation techniques were applied to the 5 basically produced sentences, respectively, to produce a total of 10 new sentences. In addition, 5 additional sentences were produced by applying back translation techniques to the existing 5 sentences produced by applying word order change. In this way, a total of 20 candidate sentences were produced according to the disease and symptoms of one crop, and 5 non-overlapping sentences were assigned out of 20 candidate sentences per image. In the same way, sentences for normal crops and non-crops were generated and assigned to each image. The number of images used for model training was 123,913. As a result, a total of 619,565 sentences were produced for 123,913 images. On the other hand, the number of images used for model testing was 303,collected from sample images provided by AI-hub's \"Facility Crop Disease Diagnostic Image\" dataset and \"Outdoor Crop Disease Diagnostic Image\" dataset, along with non-crop images collected by crawling websites and taking them in person. Thus, a total of 1515 sentences were produced for 303 images. The overall preprocessing step of Image Captioning is shown in the Figure 4.