<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Synthetic Data for Machine Learning on Embedded Systems in Precision Agriculture*

Olahiyi Bayonle Alao¹, Kristian Rother¹, and Stefan Henkler¹

Hamm-Lippstadt University of Applied Sciences, 59557 Lippstadt, Germany
olaniyi-bayonle.alao@stud.hshl.de
{kristian.rother,stefan.henkler}@hshl.de

**Abstract.** Embedded systems are used in precision agriculture for data collection via sensors and for the control of actuators such as sprayers based on machine learning models. For plant classification and monitoring, it is easier to collect data of healthy plants than it is to collect data of plants that are infected by various diseases, because they are simply more common. Sufficient data are therefore often lacking for the accurate detection of diseased plants. In this paper, we outline an approach for the generation of synthetic data of infected plants that can be used to train a machine learning model for the classification of sugar beets. We use image augmentation techniques to build a pipeline that can automatically overlay diseased areas on healthy areas of leaf images.

**Keywords:** Embedded Systems · Machine Learning · Precision Agriculture · Synthetic Data

## 1 Motivation

Agriculture performs a crucial role in food security and the economic infrastructure of a country [15]. However, the agricultural sector requires a considerable amount of land and water and it also poses a substantial threat to the biological variety and variability of life on earth [5, 7]. Additionally, in order to achieve better environmental protection of water and soils, there is a need for the reduction of fertilisers and pesticides in crop production [13].

Sugar beets and sugar cane are the two vital plants that supply the raw materials needed for the commercial production of sugar [2, 4]. However, diseases like cercospora Leaf Spot (CLS), beet yellow virus, ramularia leaf spot, rhizoctonia root and crown rot, and powdery mildew inhibit the growth of sugar beet plants during cultivation [14, 2].

Conventional methods of disease detection and classification entail visual inspections by experts and laboratory tests of plant leaves or soil samples [23].

<footnote>* This research is supported by a grant from the Ministry of Economic Affairs, Industry, Climate Action and Energy of the State of North Rhine-Westphalia (MWIDE) as part of the 5G-Landwirtschaft-ML project in the context of the program 5G.NRW (01.05.2022 – 31.12.2024, grant number 005-2108-0039).</footnote>
However, they have proven to be tedious, ineffective and time-consuming [11]. Hence, there is a need for the automation of the detection of diseases in sugar beets by applying precision agriculture techniques.

Precision agriculture uses technology to optimise agricultural production through the use of site-specific management. In crop production, this aims at crop-specific nutrient deficiencies and disease monitoring, application of fertiliser or pesticides, prediction of yields and automated counting of crops. Precision agriculture reduces harmful outputs into the environment by applying only the specific amount of fertiliser or pesticides needed by each plant [1]. However, the most significant limiting factor in precision agriculture is the interpretation of collected data from sensors [12] and the availability of adequate datasets.

Machine learning methods have been used in different studies to classify plant diseases using features like texture, type and colour of plant leaf images [2, 20]. Nevertheless, there is a lack of diversity in the quality of plant leaf image datasets. Two limiting factors in precision agriculture are the interpretation of collected data from sensors [12] and the availability of adequate datasets.

Because farmers try to keep their plants healthy, one of the challenges when building adequate datasets for the use of machine learning techniques in precision agriculture is the lack of images of non-healthy plants that are infected by various diseases. To solve this problem, we propose a process for generating synthetic datasets of diseased plants by combining images from different datasets.

In the next Section we give an overview of the related work. In Section 3, we present our approach of synthesizing non-healthy plant images. An evaluation of our approach is presented in Section 4. We conclude our paper with a summary and outlook in Section 5.

## 2 Related Work

Different approaches for synthetic image generation for ML have been proposed in the agricultural domain. Ward et al. [22] proposed a method of synthetically generating images of the Arabidopsis plant described in their research to augment existing natural plant datasets. Their synthetic image dataset generation begins with manually tracing a randomly chosen Arabidopsis leaf image in Blender to produce a 3D mesh of an inspiration leaf based on the original leaf image. Then, more 3D leaf images are created by randomly scaling to model leaves of different shapes and sizes. Next, the synthetically generated plant leaves are circularly arranged close to each other at a similar height. Finally, random background, camera and lightning are added to the generated plant leaves, which are then rendered as a 2D synthetic image and corresponding segmentation mask. Their proposed framework success is evidenced by its average accuracy of 90% in leaf segmentation.

Da Silva et al. [18] proposed three methods for generating synthetic defoliation images used in training CNN-based models to estimate soybean leaf defoliation. Their proposed methods remove leaf-belonging pixels in different ways to simulate actual defoliation in a pre-processed image and returns a new defoiled
image along with its level of defoliation. The first method simulates defoliation using random sizes of polygons formed in random pixels in the leaf area. The second method makes circles with random radii to remove leaf-belonging pixels in the leaf region. Then, secondary circles with different radii are generated in the circumference of the main circle. The third method is similar to the second method except that the secondary circles are generated within the main circle. They were able to generate 10,000 synthetic defoilation images with each method. Their models were trained only with the synthetically generated images and evaluated using natural images.

Datasets are also synthetically generated in domains other than precision agriculture. For example, Björklund et al. [3] proposed a framework for generating challenging synthetic license plate images to avoid collecting and annotating the thousands of images required to train a CNN. Likewise, Silvano et al. [19] described a combination of techniques to generate large random synthetic images of license plates to supplement their small volumes of available authentic images for training deep learning-based automated license plate recognition systems.

In the field of biomedical image analysis, Svoboda and Ulman [21] generated synthetic static and time-lapse image sequences of fully 3D fluorescence microscopy images showing the motion of objects of various sizes. Han et al. [8] generated synthetic multi-sequence brain Magnetic Resonance images using Generative Adversarial Networks. Prokopenko et al. [16] investigated approaches for generating synthetic Computed Tomography images from actual Magnetic Resonance Imaging data using GANs to enable single-modality radiotherapy planning in clinical oncology.

## 3 Solution

Our approach for the creation of synthetic data uses images of healthy sugar beet seedlings from one dataset and combines those with the diseased areas of corn leaves from another dataset to create synthetic images of diseased sugar beets. We segment the leaf areas of the diseased and healthy plants from their background and then concatenate the extracted diseased areas intelligently on the healthy leaf areas. First, we propose segmenting foreground and background leaf areas by converting the images to their respective Hue Saturation values (HSV) to find the leaves' upper and lower bound values. We then propose using thresholding algorithms to create binary masks of the leaf areas and the background and finally use boolean operations to create the new image. Furthermore, we propose bringing the infected areas of the diseased leaf to the foreground and then masking out healthy areas to become the background. Figure 1 showcases a visual representation of the proposed synthetic dataset creation pipeline. The extracted disease areas on the left side is concatenated with segmented healthy leaf areas of sugar beets to create synthetic disease images. We used Python 3 and OpenCV 2 to implement the synthetic dataset generation. The sugar beet disease detection models were trained using the generated synthetic images and the fast.ai [9] library.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Overview of the proposed approach for generating synthetic data.</figcaption></figure>

## 3.1 Datasets

We used the publicly available PlantVillage [10] and Plant seedlings [6] datasets to generate the synthetic data. The Plant Seedlings dataset contains images of approximately 960 unique plants belonging to 12 species at several growth stages. The dataset was captured at the Aarhus University Flakkebjerg Research station in collaboration with the University of Southern Denmark and Aarhus University. In contrast, the PlantVillage dataset consists of 54,323 images divided into 38 diseased and healthy plants based on 14 different plant species. All images are taken as a single leaf on a solid background labelled only by a class name. For creating the synthetic dataset, we used 463 images of sugar beets from the plant seedlings dataset as a healthy leaf dataset and 513 images of foreground segmented corn leaves infected with grey leaf spot (GLS) in the PlantVillage dataset as the diseased dataset. Corn leaves infected with GLS (caused by cercospora zeae-maydis and cercospora zeina) were used since it was the only disease in the PlantVillage dataset sharing the same genus ( cercospora ) of CLS which is a common disease for sugar beets caused by cercospora beticola.

## 3.2 Process Overview

The overall process of creating the synthetic dataset consists of four steps. First, the images from different datasets are prepared. Second, the images are segmented. Third, the images are prepared for concatenation and lastly they are concatenated to produce the synthetic images.
**Image Preparation** Since the used datasets consist of images of different sizes, resizing the images to the same size needs to be done to ensure uniformity in the synthetic dataset. The activity diagram in figure 2 shows the complete control flow of operations in preparing the images for image segmentation.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Activity diagram for image preparation.</figcaption></figure>

The images were resized to 256 x 256 pixels based on research conducted by Rukundo [17] who found the size works well for training deep learning models. The resized image is passed on to the image segmentation block.

**Image Segmentation** The leaf areas in images of the diseased and healthy datasets were segmented from their respective backgrounds to reduce the computational complexity of manipulating images with non-uniform backgrounds. Figure 3 shows the flow of actions in segmenting the leaf areas from the background.

First, the resized image is colour transformed from blue-green-red (BGR) into HSV colour space. Using the HSV information of the image, the values of the lower and upper boundary representing the green leaf and disease areas in the image were found by splitting and plotting random pixels in the image into their respective hue and saturation values.

A mask of the leaf area was created with the found lower and upper boundary HSV value using OpenCV `inRange()` function. Then, the masked image was used to create a segmented image from the original image using the `bitwise_AND` operator. Finally, the segmented image was passed on to the next block of actions for further image processing.

**Preparation for Concatenation** In this section, the actions depicted in figure 4 were carried out. The main goal of the actions performed in this section is to rotate the leaf or disease areas in the images vertically. Rotating the leaf/disease
<figure><img src="image_4.png" /><figcaption>Fig. 3. Activity diagram showing the flow of actions for image segmentation.</figcaption></figure>

area in the vertical direction allows for a consistent positioning of the disease/leaf areas since they have different orientations in their respective datasets. In addition, having the disease and leaf areas in the same direction ensures that a more significant part of the segmented disease area lands on the sugar beet leaf area during image concatenation.

As a first step, the segmented image was converted to grayscale. Converting the image to grayscale reduces the dimension of the image to an 8-bit single-channel image, which allows easier manipulations than BGR images, which hold the information of all three channels. Equation 1 is used for the colour to grayscale algorithm implementation in OpenCV.

$$Y = 0.299R + 0.587G + 0.114B \quad (1)$$

Furthermore, the methods `cv2.ADAPTIVE_THRESHOLD_MEAN_C` and `cv2.THRESH_BINARY_INV` available in OpenCV were used to create an adaptive threshold binary image of the segmented input image. Binarizing the image enabled us to find the curve joining all the continuous areas along the external boundary of the leaf area since they have the same intensity and colour, i.e. white. Finding the contours of the object of interest in the image gives access to the object's width, height, length, and orientation angle with respect to the whole image.

The values of the contours of the object of interest were then used to calculate the object's width, height, length and orientation angle by drawing a minimum area rectangle around the found leaf area.

Non vertically oriented leaf areas were vertically rotated using wrap affine transformation on the transformation matrix $M$ gotten from the OpenCV function `cv2.getRotationMatrix2D()`. Rotation in OpenCV is counterclockwise ( i.e. positive degrees specify counterclockwise rotation while negative degrees specify clockwise rotation). The transformation matrix $M$ is defined by equation 2
<figure><img src="image_5.png" /><figcaption>Fig. 4. Activity diagram showing the flow of actions for preparing segmented images for concatenation.</figcaption></figure>

$$M = \begin{bmatrix} \alpha & \beta(1-\alpha).c_x - \beta.c_y \\ -\beta\alpha\beta.c_x + (1-\alpha).c_y \end{bmatrix} \quad (2)$$

where $\alpha = scale$.cos $\theta$, $\beta = scale$.sin $\theta$, $c_x$ and $c_y$ are the x and y coordinates of the centre of the image and scale is the isotropic scale factor.

Finally, the vertically rotated image was passed on to the next block for the concatenation operation discussed in the next section.

**Concatenation** Before concatenating the segmented disease and healthy leaf image, there needs to be a confirmation that the images are of the same dimensions (i.e. 256 x 256). Inequality in the size of the images will lead to an error. Figure 6 shows the activity diagram of operations for this section.

A region of interest (ROI) was created on the rotated sugar beet image using the total rows and columns of the disease image. The created ROI has the same size as the disease image. Using the disease image, disease areas only corresponding to the leaf regions were extracted using bitwise and operation as shown in figure 5. The bitwise addition decides the pixels to be displayed using an AND operation on each pixel of the images (i.e. a bitwise AND operation is true if and only if both pixels are greater than zero). Next, a mask of the disease was created by converting the diseased image to grayscale and then thresholding the grayscale image using thresh_binary. Then an inverse of the mask was created using the bitwise not operation. The inverse mask was used to create areas where the disease will fall on in the leaf area. Then, this region of disease on the sugar
beet leaf was blacked out using the inverse mask and *bitwise operation* as shown in figure 5a. Finally, the blackout ROI on the leaf and extracted disease region were added together.

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** (a) Blacked out leaf area for addition of disease (b) Region of disease area which will be overlayed on the leaf area in (a).</figcaption></figure>

## 3.3 Synthetic Dataset

To create the synthetic dataset, every image in the 513 diseased leaf collection was overlaid on the 463 sugar beet images. This resulted in a dataset of 237 520 synthetic images. The generated images are labelled according to their corresponding number in the original image path to easily find the original images used to form the synthetic image. For example, the image with the name `cglsls_48SB_248.png` was generated from the disease image number 48 in corn's grey leaf spot image folder and the sugar beet image number 248.

The pseudo-code for implementing the algorithm for the generation of the synthetic dataset is highlighted in listing 1.1. The function begins by looping through each disease in the disease dataset path. For each disease, disease areas are segmented and rotated. The function continues by looping through the healthy leaf images which are concatenated with the segmented disease area.

**Listing 1.1.** Pseudo code for generating the synthetic dataset.

def createSyntheticImages():
    for disease in diseasePath:
        open disease
        segment disease
        find orientation
<figure><img src="image_7.png" /><figcaption>Fig. 6. Activity diagram showing the flow of actions for concatenating disease are with leaf area.</figcaption></figure>
rotate disease
**for** leaf **in** leafPath:
    **open** leaf
    resize image
    segment leaf area
    find orientation
    rotate leaf
    add disease **and** leaf area
    save image
return 'done'

## 4 Evaluation

For the evaluation, we created three subsets from the synthetic dataset. Subset 1 contains 985 randomly chosen synthetic images and all healthy sugar beet images. Subset 2 consists of 463 synthetic images and 417 healthy sugar beet images. Subset 3 is comprised of 453 healthy sugar beet plant images that were not used in the synthetic data generation and 1000 synthetically generated images from 10 random sugar beets and 100 random disease plant images.

The background of the healthy sugar beet images was segmented across all subsets to avoid learning only to make predictions based on the background. Image augmentation techniques like random flipping and clipping were used during model training. The pre-trained versions of Alexnet, ResNet-34 and ResNet-152 were used for model training and five epochs of transfer learning were used for Subset 2 while ten epochs of transfer learning were used for Subset 1 and Subset 3. Table 1 shows the average training time per epoch and accuracy of each architecture trained on the three subsets.

<table><caption>**Table 1.** Accuracies and training time of CNN models trained on three different subsets of data.</caption><thead><tr><th>Dataset</th><th>Architecture</th><th>Training time per epoch</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="3">Subset 1 (1448 images)</td><td>ResNet-34</td><td>5 s</td><td>93.05%</td></tr><tr><td>AlexNet</td><td>3 s</td><td>90.27%</td></tr><tr><td>ResNet-152</td><td>31 s</td><td>97.91%</td></tr><tr><td rowspan="3">Subset 2 (880 images)</td><td>ResNet-34</td><td>6 s</td><td>93.75%</td></tr><tr><td>AlexNet</td><td>4 s</td><td>94.88%</td></tr><tr><td>ResNet-152</td><td>18 s</td><td>91.47%</td></tr><tr><td rowspan="3">Subset 3 (1453 images)</td><td>ResNet-34</td><td>5 s</td><td>98.62%</td></tr><tr><td>AlexNet</td><td>4 s</td><td>93.10%</td></tr><tr><td>ResNet-152</td><td>15 s</td><td>98.27%</td></tr></tbody></table>
# 5 Summary and Outlook

In this paper, we outlined a process for the creation of synthetic data of diseased sugar beet seedlings from available datasets of healthy sugar beet seedlings and diseased corn leaves. We evaluated the created data by using it to train different models to differentiate between healthy and diseased sugar beet seedlings. This research is an initial technical feasibility study and can be improved upon in several ways.

On the data level, collecting healthy and diseased images of the target plant would decrease the need for our process. However, it would be interesting to see if a model trained on both synthetic and real images would outperform a model that is only trained on real images.

On a technical level, the process can be improved by having a smoother transfer of the images. This could be achieved by using styleGANs to treat a diseased plant as a different style of a healthy plant.

Another approach for creating the synthetic data is to use image generation from text like DALLE-2 with strings like "Photo of a sugar beet plant with cercospora leaf spot".

Lastly, if one only wants to know if a plant is diseased or not but not which disease the plant has, the issue of no available diseased images could be sidestepped by using techniques that don't require this data such as outlier detection with autoencoders or one-class classifiers.

# References

1. Auernhammer, H.: Precision farming—the environmental challenge. Computers and electronics in agriculture **30**(1-3), 31–43 (2001)

2. Barreto, A., Paulus, S., Varrelmann, M., Mahlein, A.K.: Hyperspectral imaging of symptoms induced by rhizoctonia solani in sugar beet: Comparison of input data and different machine learning algorithms. Journal of Plant Diseases and Protection **127**(4), 441–451 (2020)

3. Björklund, T., Fiandrotti, A., Annarumma, M., Francini, G., Magli, E.: Robust license plate recognition using neural networks trained on synthetic images. Pattern Recognition **93**, 134–146 (2019)

4. Draycott, A.P.: Sugar beet (2008)

5. (Food, F., of the United Nations), A.O.: The state of food and agriculture: paying farmers for environmental services (2007)

6. Giselsson, T.M., Jørgensen, R.N., Jensen, P.K., Dyrmann, M., Midtiby, H.S.: A public image database for benchmark of plant seedling classification algorithms. arXiv preprint arXiv:1711.05458 (2017)

7. Green, R.E., Cornell, S.J., Scharlemann, J.P., Balmford, A.: Farming and the fate of wild nature. science **307**(5709), 550–555 (2005)

8. Han, C., Hayashi, H., Rundo, L., Araki, R., Shimoda, W., Muramatsu, S., Furukawa, Y., Mauri, G., Nakayama, H.: Gan-based synthetic brain mr image generation. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018). pp. 734–738. IEEE (2018)
9. Howard, J., Gugger, S.: Fastai: A layered api for deep learning. Information **11**(2) (2020). https://doi.org/10.3390/info11020108, https://www.mdpi.com/2078-2489/11/2/108

10. Hughes, D., Salathé, M., et al.: An open access repository of images on plant health to enable the development of mobile disease diagnostics. arXiv preprint arXiv:1511.08060 (2015)

11. Lu, J., Tan, L., Jiang, H.: Review on convolutional neural network (cnn) applied to plant leaf disease classification. Agriculture **11**(8), 707 (2021)

12. Ondoua, R.N.: Precision agriculture advances and limitations

13. Otero, N., Vitoria, L., Soler, A., Canals, A.: Fertiliser characterisation: major, trace and rare earth elements. Applied geochemistry **20**(8), 1473-1488 (2005)

14. Ozguven, M.M., Adem, K.: Automatic detection and classification of leaf spot disease in sugar beet using deep learning algorithms. Physica A: statistical mechanics and its applications **535**, 122537 (2019)

15. Pawlak, K., Kołodziejczak, M.: The role of agriculture in ensuring food security in developing countries: Considerations in the context of the problem of sustainable food production. Sustainability **12**(13), 5488 (2020)

16. Prokopenko, D., Stadelmann, J.V., Schulz, H., Renisch, S., Dylov, D.V.: Unpaired synthetic image generation in radiology using gans. In: Workshop on Artificial Intelligence in Radiation Therapy. pp. 94–101. Springer (2019)

17. Rukundo, O.: Effects of image size on deep learning. arXiv preprint arXiv:2101.11508 (2021)

18. da Silva, L.A., Bressan, P.O., Gonçalves, D.N., Freitas, D.M., Machado, B.B., Gonçalves, W.N.: Estimating soybean leaf defoliation using convolutional neural networks and synthetic images. Computers and electronics in agriculture **156**, 360–368 (2019)

19. Silvano, G., Ribeiro, V., Greati, V., Bezerra, A., Silva, I., Endo, P.T., Lynn, T.: Synthetic image generation for training deep learning-based automated license plate recognition systems on the brazilian mercosur standard. Design Automation for Embedded Systems **25**(2), 113–133 (2021)

20. Sujatha, R., Chatterjee, J.M., Jhanjhi, N., Brohi, S.N.: Performance of deep learning vs machine learning in plant leaf disease detection. Microprocessors and Microsystems **80**, 103615 (2021)

21. Svoboda, D., Ulman, V.: Mitogen: a framework for generating 3d synthetic time-lapse sequences of cell populations in fluorescence microscopy. IEEE transactions on medical imaging **36**(1), 310–321 (2016)

22. Ward, D., Moghadam, P., Hudson, N.: Deep leaf segmentation using synthetic data. arXiv preprint arXiv:1807.10931 (2018)

23. Yang, R., Tian, H., Kan, J.: Classification of sugar beets based on hyperspectral and extreme learning machine methods. Applied Engineering in Agriculture **34**(6), 891–897 (2018)