# Automatic Monitoring System for Seed Germination Test Based on Deep Learning

Authors: Qi Peng; Lifen Tu; Yunyun Wu; Zhenyu Yu; Gerui Tang; Wei Song
Journal: Journal of Electrical and Computer Engineering
 (2022)

---

## Abstract

Germination test is an irreplaceable step in seed selection and breeding. The current traditional germination test method must rely on experienced professional technicians to repeatedly classify and count the germination status of seeds and count the germination rate at different moments during the whole test process (usually takes 2 to 10 days). Currently, only the German seed germination detection system (Germination Scanalyzer) can solve this problem, but it is so expensive that it has not been practically promoted. In order to improve breeding efficiency, an automatic monitoring system for seed germination tests based on deep learning was designed. It includes a modified germination thermostat, connected with a three-dimensional movable camera bin with built-in camera; a multifunctional software system capable of online, offline, and sentinel mode monitoring; a dense distributed small target detection algorithm (DDST-CenterNet) for seed germination monitoring systems. The system test results show that the seed germination test automatic monitoring system is low cost, does not depend on the seed background, light, camera model, and other usage environments, and has high scalability. The DDST-CenterNet algorithm proposed in this paper can still maintain high accuracy and good stability in the process of seed target detection and classification as the number and density of seeds increase, which is suitable for a special application scenario of seed germination test. In addition, the algorithm has high computational efficiency and can give detection results at a frame rate of not less than 10fps, which can be used in practical applications.


---

## Body

## 1. Introduction

Seed selection is an important part of breeding, and the quality of seeds is an important factor in ensuring effective farming, and the evaluation of seed quality is a crucial aspect. Seed quality evaluation is usually performed intuitively by experienced professional technicians, which is based on subjective experience and time-consuming. In recent years, researchers have proposed machine vision-based methods for monitoring seed quality [1–3], which combine characteristics such as color, shape, and texture of seeds to quickly and automatically obtain relevant parameters and facilitate seed grading by seed producers. However, whether it is an intuitive judgment by a technician or an automatic classification by a computer, the grading results of seeds must be finally verified by the test, namely, the standard germination test [4]. A complete germination test usually takes about 2 to 10 days for different types of seeds and requires experienced staff to manually count the number of seeds that have germinated at certain intervals and to repeat the test when the difference between the results of different samples is not within a certain range, which requires more effort from professional technicians.It is due to the irreplaceable nature of germination tests and the fact that traditional methods must rely on experienced professional technicians to classify the germination status of seeds that an automated system for monitoring seed germination tests with high accuracy, stability, and independence from species type has become an urgent need.At present, the automated monitoring systems for germination tests mentioned in academic research are mainly based on traditional image analysis methods. Khoenkaw et al. [5] proposed a method based on hue, saturation, and value (HSV) color model to classify germination status of seeds according to color differences between seeds and roots, which was successfully applied to germination monitoring of rice seeds. Ducournau et al. [6] designed an automatic monitoring system for the whole process of sunflower seed germination based on color, edge, and other information under controllable lighting, temperature, and humidity conditions, which can be used as the basis for the design of environmental parameters in the breeding process. However, these two methods are based on traditional image analysis, which mainly determines the germination status according to the color characteristics of the seed shell and bud, so they are only effective for specified kinds of seeds and need to be performed under specific experimental environments. They also have high requirements for image acquisition equipment, which are not universal. In addition, a small number of automated products for seed germination tests have been produced. The German-made seed germination detection system [7] (Germination Scanalyzer) is a relatively advanced automated equipment for germination test, which requires seeds to be placed on specific blue or gray filter paper, relying on seed weight to locate the center of mass, relying on the color characteristics of seeds, buds, and background to determine the length of germination, and use this as a basis for germination status determination. The stability and accuracy of this method have reached the practical requirements; however, its use conditions are limited, need specific color filter paper, and have certain requirements for the color of seeds and buds, which usually can only be used for specific seeds. In addition, the domestic Wanshen seed automatic counting instrument [8] can assist in the automation of the seed germination test. This instrument has no requirement for seed color and mainly relies on shape fitting to achieve automatic seed counting, and the accuracy can also reach the practical requirements. However, the seeds need to be placed on a specific white light plate, and only the total number of seeds can be automatically obtained, and the germination status of the seeds cannot be automatically judged.Artificial intelligence technology has been performing well in online monitoring system applications, and in the early stage, the use of pattern recognition technology to achieve automatic classification has achieved greater success [9]. In recent years, deep learning methods based on convolutional neural networks [10] have improved the performance of all aspects of online monitoring and identification applications to a new level, with better performance than traditional methods in terms of environmental adaptability and detection accuracy. With the successful application of deep learning algorithms in the field of agriculture [11–13], new ideas for automatic monitoring methods of seed germination tests have been born. However, as a special research object in agriculture, relatively little attention has been paid to online monitoring of seed germination tests. So far, only a few groups have reported their work. Thuy Nguyen et al. [14] first used U-Net [15] to segment the original rice seeds, then combined distance and threshold transformations to obtain single seed regions, and finally used the ResNet-101 [16] classification model to distinguish rice seeds into germinating and nongerminating categories. This method can achieve automatic classification of seed germination status under specific hardware environment in the laboratory, which cannot be effectively detected when the seed culture environment and image acquisition system are changed. In addition, the SeedGerm system [17], introduced by Prof. Zhou et al., covers seed germination tests, germination timing images, generalized image processing, real-time training, and machine learning-based phenotypic trait analysis for different crop types such as wheat, barley, maize, tomato, pepper, and rape based on economical hardware and open source software design, finally generating reliable germination trait analysis datasets for quantitative analysis. Due to the construction of a huge dataset and a complex learning model, the method can achieve automatic monitoring of seed germination status of many different species and is the most advanced device for automatic monitoring of seed germination tests that the author has inquired about so far. The above is almost all the reports on automatic seed germination test monitoring methods or products that we could find in recent years. Although good results have been achieved, all of these methods require customized hardware equipment and have a large impact on the test results when the seed culture environment, image acquisition system, lighting, etc., are changed, and even fail to give correct test results.This paper designs a deep learning-based automatic seed germination test monitoring system. Compared with the existing research results, the main contributions of this paper are as follows: (1) a self-made lightweight hardware device is convenient for users to obtain images of seed germination process for target identification in real time using noncustomized image acquisition devices, and it is inexpensive and easy to promote. (2) A complete software system allows both offline image analysis of seed germination and real-time online seed germination test analysis of video, including the total number of seeds, the number of germinated and ungerminated seeds, and the germination rate. The interface is user-friendly and can also be operated skillfully by people in the field of agricultural research, which can improve breeding efficiency and productivity to a greater extent. (3) Although there are many deep learning-based target detection algorithms, they cannot identify small targets with dense distribution or have low detection accuracy. In this paper, the CenterNet [18] target detection algorithm is improved and a dense distributed small target detection algorithm (DDST-CenterNet) for seed germination monitoring is proposed. For the detection of densely distributed small targets in Petri dishes, the detection results of the DDST-CenterNet algorithm are significantly improved compared to some of the currently popular deep learning-based target detection algorithms [19, 20].The rest of the paper is organized as follows: Section2 describes the seed germination test hardware and software system. Subsequently, the algorithm implementation is presented in Section 3, and the experiments and comparisons are presented in Section 4. Finally, Section 5 concludes the paper.


## 2. Composition of Automatic Monitoring System for Seed Germination Test

The system contains hardware and software parts, which can independently complete the seed germination test, and can collect seed images in real time during germination test. The deep learning algorithm is used to automatically analyze and identify the number of germinated and ungerminated seeds in the current Petri dish, calculate the real-time seed germination rate, and guide the breeding work.


### 2.1. Hardware System

The hardware system consists of a seed germination thermostat and a three-dimensional movable camera compartment. The thermostat was purchased as a finished product and has been modified to be suitable for automatic monitoring of germination tests. The main modification was to cut holes on the top of the thermostat for the convenience of placing the camera, as shown in Figure1(a). Since the opening is relatively sharp, it is easy to cause collision when installing the camera. In order to protect the camera lens, a protective cover for the top of the thermostat was designed according to the size of the top opening, as shown in Figure 1(b), and printed out for use with a 3D printer. The long cylinder part has a slightly smaller diameter and is installed outside the thermostat, while the short cylinder part has a slightly larger diameter and is installed inside the thermostat. The bottom of the cylinder of the two parts has a spiral, and the two parts are connected together to form a closed sleeve to avoid collision between the camera lens and the open part of the thermostat.Figure 1
The modified seed germination thermostat. (a) The seed germination thermostat. (b) The top protective cover.
(a)(b)The 3D movable camera compartment was designed and made independently. The purpose of this device is to make the camera fixed on the top of the seed germination box so that the camera can detect the germination of the seeds through the round hole above the seed germination box. The device design diagram is shown in Figures2(a)–2(d). According to the actual requirements of automatic monitoring system for seed germination test, the three-dimensional movable camera bin has the following functions:(1)
As shown in ① in Figure2(a), there are three screw slots on the top of the box, any two of which are fixed with the slot above the 7-shaped part by two screws, so as to achieve the effect of allowing the 7-shaped part to be adjusted back and forth, left and right within a certain range. It is convenient for the camera to adjust the position forward and backward, left and right within a certain range. The purpose of this design is to solve the problem that when using different cameras, because of the difference in the length and width of different camera models, the camera viewpoint and the upper hole of the seed germination incubator are not aligned, resulting in a failure to take effective pictures. The purpose of the three screw slots on the box is to prevent the 7-shaped parts from rotating left and right by connecting any two of them to the 7-shaped parts, and to increase the range of movement by designing three slots instead of two.(2)
As shown in ② in Figure2(a), two slots in different directions below the 7-shaped part are fixed to the camera by screws so that the camera can be adjusted up and down within a certain range. The purpose of this design is to solve the problem that when using different types of cameras, due to the height difference of different cameras, the camera does not have the right distance to observe the seeds, resulting in the failure to take effective pictures. There are two slots under the parts in different directions so that the camera can be rotated 90° to change perspective.(3)
As shown in ③ in Figure2(a), the five L-shaped parts are aligned with the holes in the seed germination thermostat and fixed with screws. The purpose of this design is to make the box and the seed germination thermostat firmly fixed together.(4)
As shown in ④ in Figure2(a), there are two hinges connecting the main body and the door. The purpose of this design is to open the box easily when the camera compartment needs to be adjusted.(5)
As shown in ⑤ in Figure2(a), there are openings in the panels around the box, and these openings are for heat dissipation. The purpose of this design is as follows: the upper openings of the seed germination thermostat were originally used for heat dissipation and to water vapor distribution, and the camera would fog up as a result, so this heat dissipation design is required.(6)
As shown in ⑥ in Figure2(b), the top of the box is used to place the portable computer. The purpose of this design is that the camera can be adjusted more conveniently and quickly.(7)
As shown in ⑦ in Figure2(b), a pull handle is added to the outside of the door body, and the purpose of this design is that the door body can be opened and closed easily.(8)
As shown in ⑧ in Figure2(c), a strip-mounted pad is added above the box body, which is higher than the screw on the top. The purpose of this design is to prevent the screws from contacting the computer when the portable computer is placed above the box body. The gap between the strips can also improve the heat dissipation of the computer.(9)
As shown in ⑨ in Figure2(c), a round hole is opened at the top of the box. The purpose of this design is to allow the signal cable and power cable to pass through the box when the portable computer is connected to the camera.(10)
As shown in ⑩ in Figure2(d), the fixed position of the box and the seed germination thermostat is moved inward. The purpose of this design is to avoid the fixed parts of the door body at the corresponding position on the thermostat and not to affect the normal opening and closing of the thermostat door body.Figure 2
Three-dimensional movable camera bin. (a) The front. (b) The side. (c) The top. (d) The bottom.
(a)(b)(c)(d)We connect the self-made camera bin to the modified thermostat, and install a common type of camera to form a complete set of seed germination test environments and a three-dimensional adjustable image acquisition system. The front, side, and back finished photographs of the system appearance are shown in Figure3(a)–3(c), respectively.Figure 3
Hardware system for automatic monitoring of seed germination tests. (a) The front. (b) The side. (c) The back.
(a)(b)(c)


### 2.2. Software System

The interface of this software is designed in PyQt5, and the functional module algorithm is written in python language. The main function of the software system is to realize the automatic monitoring of the seed germination test, that is, to identify the germination status of seeds in the thermostat for germination test, and count them separately to calculate the current germination rate. This software system has a total of three working modes as follows:(1)
Real-time online monitoring mode: as shown in Figure4(a), the portable computer runs the software, connects the camera, and displays the current image acquired by the camera in real time on the left side of the image display area. By clicking the Recognition button, the core algorithm of seed germination rate analysis can be invoked to analyze the germination status of the frame, obtain all seed areas in real time, and classify the germination status of each seed. The germinated and ungerminated seeds are marked with red and blue boxes, respectively, making it easy for professionals to review the test results. The seed germination monitoring results are displayed in real time on the right side of the image display area, and its upper left corner shows the detection frame rate. The number and germination rate of germinated and ungerminated seeds are displayed separately and saved to the background, which is convenient for later viewing of historical data. In order to obtain better video effects and improve the germination status recognition accuracy, the video acquired by the camera can be adjusted in real-time post, such as individually enhancing or weakening the values of R, G, and B channels, changing exposure time, brightness, contrast, inversion, histogram equalization, filtering, etc. The operation method is shown in the function area on the left side of the software interface.(2)
Offline recognition mode: as shown in Figure4(b), the offline mode is mainly used to analyze a single frame image or video that has been collected. Click Open the image or Open the video to display the current image or video on the left side of the image display area, and click the Recognition button to call the core algorithm of seed germination rate analysis of the system to analyze the germination status of the frame image or video. The detection result is displayed on the right side of the image display area, and the number of germinated and ungerminated seeds and germination rate are displayed on the upper left of the interface. The automatic monitoring algorithm of seed germination test based on deep learning used in this system has good robustness. It is not limited to the camera used in the hardware system designed in this paper to obtain images, offline seed images, or videos obtained by any common devices, such as mobile phone cameras, different models of ordinary cameras, and industrial cameras, can be used for seed germination status analysis.(3)
Unsupervised mode: this mode is based on the real-time online monitoring mode with the addition of a clock function, that is, in this mode, every hour interval, the camera automatically collects a current frame and recognizes it, obtains the number of seeds that have germinated and not germinated in the current frame, calculates the current germination rate, and stores these three data together with the current time in the database, which is convenient for later analysis of the whole process parameters of the germination test.Figure 4
System software interface. (a) Real-time online monitoring mode and (b) offline recognition mode.
(a)(b)


## 2.1. Hardware System

The hardware system consists of a seed germination thermostat and a three-dimensional movable camera compartment. The thermostat was purchased as a finished product and has been modified to be suitable for automatic monitoring of germination tests. The main modification was to cut holes on the top of the thermostat for the convenience of placing the camera, as shown in Figure1(a). Since the opening is relatively sharp, it is easy to cause collision when installing the camera. In order to protect the camera lens, a protective cover for the top of the thermostat was designed according to the size of the top opening, as shown in Figure 1(b), and printed out for use with a 3D printer. The long cylinder part has a slightly smaller diameter and is installed outside the thermostat, while the short cylinder part has a slightly larger diameter and is installed inside the thermostat. The bottom of the cylinder of the two parts has a spiral, and the two parts are connected together to form a closed sleeve to avoid collision between the camera lens and the open part of the thermostat.Figure 1
The modified seed germination thermostat. (a) The seed germination thermostat. (b) The top protective cover.
(a)(b)The 3D movable camera compartment was designed and made independently. The purpose of this device is to make the camera fixed on the top of the seed germination box so that the camera can detect the germination of the seeds through the round hole above the seed germination box. The device design diagram is shown in Figures2(a)–2(d). According to the actual requirements of automatic monitoring system for seed germination test, the three-dimensional movable camera bin has the following functions:(1)
As shown in ① in Figure2(a), there are three screw slots on the top of the box, any two of which are fixed with the slot above the 7-shaped part by two screws, so as to achieve the effect of allowing the 7-shaped part to be adjusted back and forth, left and right within a certain range. It is convenient for the camera to adjust the position forward and backward, left and right within a certain range. The purpose of this design is to solve the problem that when using different cameras, because of the difference in the length and width of different camera models, the camera viewpoint and the upper hole of the seed germination incubator are not aligned, resulting in a failure to take effective pictures. The purpose of the three screw slots on the box is to prevent the 7-shaped parts from rotating left and right by connecting any two of them to the 7-shaped parts, and to increase the range of movement by designing three slots instead of two.(2)
As shown in ② in Figure2(a), two slots in different directions below the 7-shaped part are fixed to the camera by screws so that the camera can be adjusted up and down within a certain range. The purpose of this design is to solve the problem that when using different types of cameras, due to the height difference of different cameras, the camera does not have the right distance to observe the seeds, resulting in the failure to take effective pictures. There are two slots under the parts in different directions so that the camera can be rotated 90° to change perspective.(3)
As shown in ③ in Figure2(a), the five L-shaped parts are aligned with the holes in the seed germination thermostat and fixed with screws. The purpose of this design is to make the box and the seed germination thermostat firmly fixed together.(4)
As shown in ④ in Figure2(a), there are two hinges connecting the main body and the door. The purpose of this design is to open the box easily when the camera compartment needs to be adjusted.(5)
As shown in ⑤ in Figure2(a), there are openings in the panels around the box, and these openings are for heat dissipation. The purpose of this design is as follows: the upper openings of the seed germination thermostat were originally used for heat dissipation and to water vapor distribution, and the camera would fog up as a result, so this heat dissipation design is required.(6)
As shown in ⑥ in Figure2(b), the top of the box is used to place the portable computer. The purpose of this design is that the camera can be adjusted more conveniently and quickly.(7)
As shown in ⑦ in Figure2(b), a pull handle is added to the outside of the door body, and the purpose of this design is that the door body can be opened and closed easily.(8)
As shown in ⑧ in Figure2(c), a strip-mounted pad is added above the box body, which is higher than the screw on the top. The purpose of this design is to prevent the screws from contacting the computer when the portable computer is placed above the box body. The gap between the strips can also improve the heat dissipation of the computer.(9)
As shown in ⑨ in Figure2(c), a round hole is opened at the top of the box. The purpose of this design is to allow the signal cable and power cable to pass through the box when the portable computer is connected to the camera.(10)
As shown in ⑩ in Figure2(d), the fixed position of the box and the seed germination thermostat is moved inward. The purpose of this design is to avoid the fixed parts of the door body at the corresponding position on the thermostat and not to affect the normal opening and closing of the thermostat door body.Figure 2
Three-dimensional movable camera bin. (a) The front. (b) The side. (c) The top. (d) The bottom.
(a)(b)(c)(d)We connect the self-made camera bin to the modified thermostat, and install a common type of camera to form a complete set of seed germination test environments and a three-dimensional adjustable image acquisition system. The front, side, and back finished photographs of the system appearance are shown in Figure3(a)–3(c), respectively.Figure 3
Hardware system for automatic monitoring of seed germination tests. (a) The front. (b) The side. (c) The back.
(a)(b)(c)


## 2.2. Software System

The interface of this software is designed in PyQt5, and the functional module algorithm is written in python language. The main function of the software system is to realize the automatic monitoring of the seed germination test, that is, to identify the germination status of seeds in the thermostat for germination test, and count them separately to calculate the current germination rate. This software system has a total of three working modes as follows:(1)
Real-time online monitoring mode: as shown in Figure4(a), the portable computer runs the software, connects the camera, and displays the current image acquired by the camera in real time on the left side of the image display area. By clicking the Recognition button, the core algorithm of seed germination rate analysis can be invoked to analyze the germination status of the frame, obtain all seed areas in real time, and classify the germination status of each seed. The germinated and ungerminated seeds are marked with red and blue boxes, respectively, making it easy for professionals to review the test results. The seed germination monitoring results are displayed in real time on the right side of the image display area, and its upper left corner shows the detection frame rate. The number and germination rate of germinated and ungerminated seeds are displayed separately and saved to the background, which is convenient for later viewing of historical data. In order to obtain better video effects and improve the germination status recognition accuracy, the video acquired by the camera can be adjusted in real-time post, such as individually enhancing or weakening the values of R, G, and B channels, changing exposure time, brightness, contrast, inversion, histogram equalization, filtering, etc. The operation method is shown in the function area on the left side of the software interface.(2)
Offline recognition mode: as shown in Figure4(b), the offline mode is mainly used to analyze a single frame image or video that has been collected. Click Open the image or Open the video to display the current image or video on the left side of the image display area, and click the Recognition button to call the core algorithm of seed germination rate analysis of the system to analyze the germination status of the frame image or video. The detection result is displayed on the right side of the image display area, and the number of germinated and ungerminated seeds and germination rate are displayed on the upper left of the interface. The automatic monitoring algorithm of seed germination test based on deep learning used in this system has good robustness. It is not limited to the camera used in the hardware system designed in this paper to obtain images, offline seed images, or videos obtained by any common devices, such as mobile phone cameras, different models of ordinary cameras, and industrial cameras, can be used for seed germination status analysis.(3)
Unsupervised mode: this mode is based on the real-time online monitoring mode with the addition of a clock function, that is, in this mode, every hour interval, the camera automatically collects a current frame and recognizes it, obtains the number of seeds that have germinated and not germinated in the current frame, calculates the current germination rate, and stores these three data together with the current time in the database, which is convenient for later analysis of the whole process parameters of the germination test.Figure 4
System software interface. (a) Real-time online monitoring mode and (b) offline recognition mode.
(a)(b)


## 3. The Core Algorithm for Seed Germination Rate Analysis

During the traditional seed germination test, it is necessary to manually count the total number of tested seeds in the Petri dish or germination bed. At intervals, it is also necessary to observe the change of the germination status of the tested seeds, count the number of seeds that have germinated and have not germinated, and calculate and record the germination rate of the seeds at the current moment. This process is labor-intensive, and the manual judgment of the germination status of seeds is easily influenced by subjective factors and errors. In this system, target detection method based on deep convolutional neural network is used to judge the seed count and germination status, which has high detection accuracy and good environmental adaptability, and can adapt to various influences of background noise, brightness change, and illumination inequality.Currently, the mainstream target detection algorithms are divided into two categories: one is based on the two-stage method; that is, a series of sample candidate boxes are generated first, and then, the samples are classified by convolutional neural networks, such as RCNN [19], Fast RCNN [20], and Faster RCNN [21]. The other category is based on one-stage object detection framework, which completes detection and regression tasks at the same time, such as YOLO [22] and SSD [23]. These algorithms require a series of anchor boxes to determine the position and size of the target. Two-stage task has higher accuracy but slower speed. One stage can achieve real-time performance at the expense of accuracy. Most importantly, the objects to be detected in this paper are the seeds in the germination test process, with a large number of targets, dense distribution, and very small targets (rape seeds), so the test results of the above algorithms are not good. Therefore, in order to solve the above problems, a target detection architecture based on center point, namely, CenterNet neural network, was adopted and optimized, and a dense distributed small target detection algorithm (DDST-CenterNet) suitable for seed germination monitoring system was proposed. It can avoid enumerating a large number of candidate boxes and achieve a more simplified and fast detection with higher accuracy. The optimization is mainly carried out in two aspects.


### 3.1. Target Bounding Box Optimization Method

The original CenterNet algorithm does not perform nonmaximum suppression (NMS: nonmaximum suppression) postprocessing after decoding. When the target is a small target, it has little impact, but when the target is too large, it will also generate partial frame overlap. In the seed germination test application in this paper, because there are multiple sizes of Petri dishes, the focal length of the camera will be adjusted according to the size of the Petri dishes, resulting in the size of the target changing in each test. In practice, it is found that the seed, as the detection target, belongs to the typical small target detection range, but it also generates the frame overlap problem.In addition, CenterNet, as a new anchor-free end-to-end target detection algorithm, uses only one point to represent the center of the bounding box of the object and then obtains the object size, orientation, pose, and other attributes directly from the image feature regression around the center location, turning the target detection problem into a standard key point estimation problem. Compared with other target detection algorithms, CenterNet has better detection effects for the densely distributed small target problem in the seed germination test. However, the color features of the seeds' shells are significant in the seed germination images, but the color features of the buds are very similar to the commonly used white filter paper background, which can easily cause misclassification. As shown in Figure5(a), the young buds in the red box grow out from the front with a more obvious break in the seed shell, which is easier to identify. The blue boxes are completely ungerminated seeds, and the seeds in the yellow boxes have actually germinated, but the buds grow out from the bottom without damaging the appearance of the seed shell. The color of the buds is very similar to the background, which is easy to cause misjudgment. In Figure 5(b), the seeds marked by the yellow box have short buds, a situation that we also consider as a germinated target, but in the actual detection process, it also occurs that both germinated and ungerminated targets are detected at the same time.Figure 5
Germinated and ungerminated seeds. (a) Poorly characterized buds. (b) Short buds.
(a)(b)In practical applications, it is found that when the CenterNet algorithm is used to identify the germination state of seeds, the buds with indistinct features and the short buds are prone to generate overlapping boxes; that is, they will be recognized as two targets of germination and ungerminated at the same time, resulting in more overlapping boxes. The test was conducted in Figure5(b), and the results of identification results are shown in Figure 6: the identification results were 20 germinated seeds and 9 ungerminated seeds. However, after manual counting, the actual seeds are only 25, including 20 germinated seeds and 5 ungerminated seeds; that is to say, 4 seeds were counted repeatedly. As shown in Figure 6(b), the identified target categories and the central coordinates of the bounding boxes are displayed. It is found that four pairs of box center points appeared to be overlapping or very close to each other, which is obviously a misjudgment.Figure 6
Original CenterNet algorithm recognition results. (a) Identified targets. (b) Score and center position (with overlapping boxes).
(a)(b)In order to solve the overlapping box problem, the CenterNet algorithm was improved as follows:(1) For the overlapping box problem of the same category targets, the method of NMS after decoding is used to eliminate them. The commonly used NMS algorithm is divided into three steps: first, sort the scores of all boxes and select the highest score and its corresponding box, and then traverse the remaining boxes. If the IOU [24] with the current highest score box is greater than a certain threshold (the threshold we selected is 0.3), the box will be deleted. Finally, continue to select the one with the highest score from the unprocessed box and repeat the above process. However, we found that this method is easy to lose part of the real target area in the densely distributed seed images. As shown in Figure 7, the adjacent seeds marked with red and blue boxes both belong to germinated seeds, but the boxes have a high degree of overlap, which makes it easy to lose some real targets when NMS is carried out.Figure 7
Densely distributed seed image.To overcome this problem, we improved the NMS by considering not only the intersection and union between two bounding boxes but also increasing the center point distance and aspect ratio between two bounding boxes, that is, replacing IOU with CIOU [25]. It is found that this scheme can effectively solve the problem of overlapping boxes of similar targets, and at the same time prevent the problem of losing connected similar targets with dense distribution.(2) For the overlapping box problem of different categories of targets, a multitarget center point distance suppression (CPDS) algorithm is designed to solve the problem that targets with insignificant features are detected as multiple sprouting states at the same time, resulting in overlapping boxes. Since seed germination tests require that seeds cannot overlap and neighboring seeds need to have a certain spacing, there must be a target that is a false detection target when the detected target center points are too close. Therefore, we designed the CPDS optimization method based on this theory. When the two target centers are too close, the correction is made to eliminate the redundant target.Suppose thatN targets are detected in a frame, and xi1,yi1 denotes the coordinate of the upper left corner of the i-th target bounding box, and xi2,yi2 denotes the coordinates of the lower right corner of the i-th target bounding box. Then, the threshold value Tc is obtained by the following equation:(1)Tc=12×N∑i=iNxi2−xi12+yi2−yi12.UsingFxi,Fyi to denote the center position coordinates of the i-th i=1∼N germinated seed, and Fxj,Fyj to denote the center position coordinates of the j-th jj=1∼N∩j≠i germinated seed. Then, when the positions of Fxi,Fyi and Fxj,Fyj satisfy (2), both bounding boxes are kept; otherwise, the lower scoring one is excluded and the higher scoring one is kept.(2)Fxi−Fxj2+Fyi−Fyj2>Tc.Figure6 was tested again with the improved algorithm, and the results are shown in Figure 8: the identification results are 20 germinated seeds and 5 ungerminated seeds, which is consistent with the manual counting result and effectively solves the overlapping box problem.Figure 8
Improved CenterNet algorithm recognition results. (a) Identified target. (b) Score and center position (without overlapping boxes).
(a)(b)


### 3.2. Loss Function

The loss function of the original CenterNet algorithm contains three components, which are the loss of the heat map, marked asLk; the loss of the width and height of the bounding box, marked as Lsize; and the loss of the center point coordinate offset, marked as Loff. The loss predicted by the width and height will be relatively larger, and the loss is multiplied by a coefficient (0.1 in the original paper), which constitutes the total loss, as shown in the following equation:(3)Ldet=Lk+0.1Lsize+Loff.The object detected in this paper is the seeds, and the object of classification is two states of germination and nongermination. After theoretical analysis and extensive experimental verification, the position of the center point of the seed as the target has less effect on the result, so we corrected the coefficients in the original loss function, and the revised loss function is as follows:(4)Lseed=2.2Lk+0.2Lsize+Loff.Keep other parameters unchanged, such as training samples and number of steps, change only the loss function, train a new model, and test it on the samples to be detected. First, the images used in Figure8 are detected, and the results are shown in Figure 9: the identification results are 20 germinated seeds and 5 ungerminated seeds, which is consistent with the manual counting results and also consistent with the result before changing the loss function. However, by comparing Figures 8 and9, we find that the target detection result score of the new model has been significantly improved.Figure 9
Model test results after changing the loss function. (a) Identified targets. (b) Score and center position (score improved significantly).
(a)(b)Seed germination status classification is a typical dense small target detection scene. The denser the seed target, the more difficult it is to detect. In order to further verify the advantages of the new model, we increase the detection difficulty again by increasing the density of seeds and detect the same images with the original model and the improved model, respectively. The results are shown in Figure10, where 10(a) and 10(b) are the detection results of the original model, and 10(c) and 10(d) are the detection results of the new model. The original model detected 34 germinated targets and 13 ungerminated targets, while the new model detected 31 germinated targets and 18 ungerminated targets. After manual confirmation, there are actually 31 germinated targets and 18 ungerminated targets; that is, the new model has no false detection, while the original model has partial false detections and one missed target.Figure 10
Test results for dense small targets. (a) The target identified by the original model. (b) The score and center position of the original model. (c) The target identified by the new model. (d) The score and center position of the new model.
(a)(b)(c)(d)In addition, the improved algorithm is better adapted to the environment. The Petri dish in Figure10 was placed in a more complex background environment and tested with the original model and the new model, respectively. The results are shown in Figure 11, where 11(a) and 11(b) are the detection results of the original model, and 11(c) and 11(d) are the detection results of the new model. The original model detected 36 germinated targets and 12 ungerminated targets, while the new model detected 31 germinated targets and 18 ungerminated targets. After manual confirmation, there are actually 31 germinated targets and 18 ungerminated targets; that is, the new model has no false detection, while the original model has some false detection, and some insignificant overlapping boxes are also detected. When we carefully analyze the data in Figure 11(b), no obvious overlapping boxes similar to those evident in Figure 6(b) appear due to the inclusion of the overlapping box elimination algorithm. However, because the targets in the field of view in this example are too many and too dense, the parameter Tc obtained by formula (1) is small. Therefore, when the position offset of some overlapping boxes is large, they cannot be removed, and some insignificant overlapping boxes in this example appear.Figure 11
Test results for dense small targets in complex backgrounds. (a) The target identified by the original model. (b) The score and center position of the original model. (c) The target identified by the new model. (d) The score and center position of the new model.
(a)(b)(c)(d)Based on extensive experimental analysis, it is found that the new loss function is more suitable for this application scenario, which not only significantly improves the score of the target but also improves the detection accuracy and reduces the false detection rate. In addition, the new model has better environmental adaptability. For the same Petri dish, changing different backgrounds results in a slight change in the score of the target, but the detected germinated and ungerminated seeds are stable and unchanged, with better robustness than the original model.


## 3.1. Target Bounding Box Optimization Method

The original CenterNet algorithm does not perform nonmaximum suppression (NMS: nonmaximum suppression) postprocessing after decoding. When the target is a small target, it has little impact, but when the target is too large, it will also generate partial frame overlap. In the seed germination test application in this paper, because there are multiple sizes of Petri dishes, the focal length of the camera will be adjusted according to the size of the Petri dishes, resulting in the size of the target changing in each test. In practice, it is found that the seed, as the detection target, belongs to the typical small target detection range, but it also generates the frame overlap problem.In addition, CenterNet, as a new anchor-free end-to-end target detection algorithm, uses only one point to represent the center of the bounding box of the object and then obtains the object size, orientation, pose, and other attributes directly from the image feature regression around the center location, turning the target detection problem into a standard key point estimation problem. Compared with other target detection algorithms, CenterNet has better detection effects for the densely distributed small target problem in the seed germination test. However, the color features of the seeds' shells are significant in the seed germination images, but the color features of the buds are very similar to the commonly used white filter paper background, which can easily cause misclassification. As shown in Figure5(a), the young buds in the red box grow out from the front with a more obvious break in the seed shell, which is easier to identify. The blue boxes are completely ungerminated seeds, and the seeds in the yellow boxes have actually germinated, but the buds grow out from the bottom without damaging the appearance of the seed shell. The color of the buds is very similar to the background, which is easy to cause misjudgment. In Figure 5(b), the seeds marked by the yellow box have short buds, a situation that we also consider as a germinated target, but in the actual detection process, it also occurs that both germinated and ungerminated targets are detected at the same time.Figure 5
Germinated and ungerminated seeds. (a) Poorly characterized buds. (b) Short buds.
(a)(b)In practical applications, it is found that when the CenterNet algorithm is used to identify the germination state of seeds, the buds with indistinct features and the short buds are prone to generate overlapping boxes; that is, they will be recognized as two targets of germination and ungerminated at the same time, resulting in more overlapping boxes. The test was conducted in Figure5(b), and the results of identification results are shown in Figure 6: the identification results were 20 germinated seeds and 9 ungerminated seeds. However, after manual counting, the actual seeds are only 25, including 20 germinated seeds and 5 ungerminated seeds; that is to say, 4 seeds were counted repeatedly. As shown in Figure 6(b), the identified target categories and the central coordinates of the bounding boxes are displayed. It is found that four pairs of box center points appeared to be overlapping or very close to each other, which is obviously a misjudgment.Figure 6
Original CenterNet algorithm recognition results. (a) Identified targets. (b) Score and center position (with overlapping boxes).
(a)(b)In order to solve the overlapping box problem, the CenterNet algorithm was improved as follows:(1) For the overlapping box problem of the same category targets, the method of NMS after decoding is used to eliminate them. The commonly used NMS algorithm is divided into three steps: first, sort the scores of all boxes and select the highest score and its corresponding box, and then traverse the remaining boxes. If the IOU [24] with the current highest score box is greater than a certain threshold (the threshold we selected is 0.3), the box will be deleted. Finally, continue to select the one with the highest score from the unprocessed box and repeat the above process. However, we found that this method is easy to lose part of the real target area in the densely distributed seed images. As shown in Figure 7, the adjacent seeds marked with red and blue boxes both belong to germinated seeds, but the boxes have a high degree of overlap, which makes it easy to lose some real targets when NMS is carried out.Figure 7
Densely distributed seed image.To overcome this problem, we improved the NMS by considering not only the intersection and union between two bounding boxes but also increasing the center point distance and aspect ratio between two bounding boxes, that is, replacing IOU with CIOU [25]. It is found that this scheme can effectively solve the problem of overlapping boxes of similar targets, and at the same time prevent the problem of losing connected similar targets with dense distribution.(2) For the overlapping box problem of different categories of targets, a multitarget center point distance suppression (CPDS) algorithm is designed to solve the problem that targets with insignificant features are detected as multiple sprouting states at the same time, resulting in overlapping boxes. Since seed germination tests require that seeds cannot overlap and neighboring seeds need to have a certain spacing, there must be a target that is a false detection target when the detected target center points are too close. Therefore, we designed the CPDS optimization method based on this theory. When the two target centers are too close, the correction is made to eliminate the redundant target.Suppose thatN targets are detected in a frame, and xi1,yi1 denotes the coordinate of the upper left corner of the i-th target bounding box, and xi2,yi2 denotes the coordinates of the lower right corner of the i-th target bounding box. Then, the threshold value Tc is obtained by the following equation:(1)Tc=12×N∑i=iNxi2−xi12+yi2−yi12.UsingFxi,Fyi to denote the center position coordinates of the i-th i=1∼N germinated seed, and Fxj,Fyj to denote the center position coordinates of the j-th jj=1∼N∩j≠i germinated seed. Then, when the positions of Fxi,Fyi and Fxj,Fyj satisfy (2), both bounding boxes are kept; otherwise, the lower scoring one is excluded and the higher scoring one is kept.(2)Fxi−Fxj2+Fyi−Fyj2>Tc.Figure6 was tested again with the improved algorithm, and the results are shown in Figure 8: the identification results are 20 germinated seeds and 5 ungerminated seeds, which is consistent with the manual counting result and effectively solves the overlapping box problem.Figure 8
Improved CenterNet algorithm recognition results. (a) Identified target. (b) Score and center position (without overlapping boxes).
(a)(b)


## 3.2. Loss Function

The loss function of the original CenterNet algorithm contains three components, which are the loss of the heat map, marked asLk; the loss of the width and height of the bounding box, marked as Lsize; and the loss of the center point coordinate offset, marked as Loff. The loss predicted by the width and height will be relatively larger, and the loss is multiplied by a coefficient (0.1 in the original paper), which constitutes the total loss, as shown in the following equation:(3)Ldet=Lk+0.1Lsize+Loff.The object detected in this paper is the seeds, and the object of classification is two states of germination and nongermination. After theoretical analysis and extensive experimental verification, the position of the center point of the seed as the target has less effect on the result, so we corrected the coefficients in the original loss function, and the revised loss function is as follows:(4)Lseed=2.2Lk+0.2Lsize+Loff.Keep other parameters unchanged, such as training samples and number of steps, change only the loss function, train a new model, and test it on the samples to be detected. First, the images used in Figure8 are detected, and the results are shown in Figure 9: the identification results are 20 germinated seeds and 5 ungerminated seeds, which is consistent with the manual counting results and also consistent with the result before changing the loss function. However, by comparing Figures 8 and9, we find that the target detection result score of the new model has been significantly improved.Figure 9
Model test results after changing the loss function. (a) Identified targets. (b) Score and center position (score improved significantly).
(a)(b)Seed germination status classification is a typical dense small target detection scene. The denser the seed target, the more difficult it is to detect. In order to further verify the advantages of the new model, we increase the detection difficulty again by increasing the density of seeds and detect the same images with the original model and the improved model, respectively. The results are shown in Figure10, where 10(a) and 10(b) are the detection results of the original model, and 10(c) and 10(d) are the detection results of the new model. The original model detected 34 germinated targets and 13 ungerminated targets, while the new model detected 31 germinated targets and 18 ungerminated targets. After manual confirmation, there are actually 31 germinated targets and 18 ungerminated targets; that is, the new model has no false detection, while the original model has partial false detections and one missed target.Figure 10
Test results for dense small targets. (a) The target identified by the original model. (b) The score and center position of the original model. (c) The target identified by the new model. (d) The score and center position of the new model.
(a)(b)(c)(d)In addition, the improved algorithm is better adapted to the environment. The Petri dish in Figure10 was placed in a more complex background environment and tested with the original model and the new model, respectively. The results are shown in Figure 11, where 11(a) and 11(b) are the detection results of the original model, and 11(c) and 11(d) are the detection results of the new model. The original model detected 36 germinated targets and 12 ungerminated targets, while the new model detected 31 germinated targets and 18 ungerminated targets. After manual confirmation, there are actually 31 germinated targets and 18 ungerminated targets; that is, the new model has no false detection, while the original model has some false detection, and some insignificant overlapping boxes are also detected. When we carefully analyze the data in Figure 11(b), no obvious overlapping boxes similar to those evident in Figure 6(b) appear due to the inclusion of the overlapping box elimination algorithm. However, because the targets in the field of view in this example are too many and too dense, the parameter Tc obtained by formula (1) is small. Therefore, when the position offset of some overlapping boxes is large, they cannot be removed, and some insignificant overlapping boxes in this example appear.Figure 11
Test results for dense small targets in complex backgrounds. (a) The target identified by the original model. (b) The score and center position of the original model. (c) The target identified by the new model. (d) The score and center position of the new model.
(a)(b)(c)(d)Based on extensive experimental analysis, it is found that the new loss function is more suitable for this application scenario, which not only significantly improves the score of the target but also improves the detection accuracy and reduces the false detection rate. In addition, the new model has better environmental adaptability. For the same Petri dish, changing different backgrounds results in a slight change in the score of the target, but the detected germinated and ungerminated seeds are stable and unchanged, with better robustness than the original model.


## 4. Analysis of System Test Results

In order to verify the effectiveness of the densely distributed small target detection algorithm proposed in this paper for the seed germination monitoring system, a series of comparative experiments were conducted. Since there is no seed germination test image in the public dataset, all training and testing samples are collected and labeled by ourselves. The number of labeled samples is 3868. In addition, after enhancing and rotating the samples, the training samples are expanded to 12 times the original dataset.In order to demonstrate the advantages of the detection algorithm in terms of detection accuracy and speed, the algorithm in this paper is compared with three current mainstream target detection algorithms, namely, the CenterNet algorithm, the Faster RCNN algorithm, and the SSD algorithm. All algorithms were trained on this dataset with the same number of training steps and tested with the same images. In an image to be detected, the number of correctly detected germinated seeds is denoted byTp, the number of incorrectly detected germinated seeds (i.e., the seeds that were not actually germinated but were identified as germinated) is denoted by Fp, the number of correctly detected ungerminated seeds is denoted by TN, the number of incorrectly detected ungerminated seeds (i.e., the seeds that were actually germinated but were identified as ungerminated) is denoted by FN, and the number of undetected seeds, that is, missed seeds, is denoted by TD. In order to quantitatively evaluate the effectiveness of each algorithm, the accuracy ACC of these four algorithms is calculated separately, and the formula is shown in equation (5), where a larger ACC indicates a better recognition effect.(5)ACC=TP+TNTP+FP+FN+TN+TD.A large number of experiments have found that the above four algorithms have better detection effects on a small number of targets, but with the increase of the number of seeds, the Faster RCNN algorithm and the SSD algorithm basically cannot detect the targets. The CenterNet algorithm and the DDST-CenterNet algorithm proposed in this paper can still detect the target. However, when the seeds are particularly dense, the accuracy of the CenterNet algorithm decreases, while the DDST-CenterNet algorithm still gives good results, as shown in Figure12.Figure 12
Detection results of different algorithms at different levels of density. (a) The detection results of Faster RCNN. (b) The detection results of SSD. (c) The detection results of CenterNet. (d) The detection results of DDST-CenterNet proposed in this paper.
(a)(b)(c)(d)Therefore, we divided the test subjects into three groups: the number of seeds in the Petri dish is less than 10, the number of seeds is between 10 and 40, and the number of seeds is more than 40, for a total of three environments, which were tested. The test results are shown in Table1.Table 1
Comparison of the accuracy of various algorithms.
Target detection algorithmAccuracy/%Less than 10 seedsBetween 10 and 40 seedsMore than 40 seedsFaster RCNN98.6478.1012.32SSD98.1255.310CenterNet99.6495.2585.36DDST-CenterNet99.6497.2396.57From the results in Table1, we can find that: for a small number of target detection, all four algorithms perform well because there are only two types of detection results with more significant features in this system. However, as the density of the targets increased, the accuracy rate showed a decrease. The accuracy of SSD and Faster RCNN algorithms decreases rapidly or even fails to detect the target completely. CenterNet performs better in the test set where the number of seeds does not exceed 40, but as the density increases further, the number of false detection and missed detection increased, leading to a more significant decrease in accuracy. The accuracy of DDST-CenterNet algorithm proposed in this paper is not significantly decreased, and the detection results have better stability.The DDST-CenterNet algorithm is highly scalable and can adapt to a variety of different environments. In this experiment, different models of mobile phones and industrial cameras were used to obtain the dataset. Petri dishes were placed in different background environments, and the system was able to acquire the total number of seeds and the germination status of each seed with high accuracy. It is only used for online monitoring of rape seeds since only samples of different varieties of rape seeds have been labeled and learned so far. However, the system's hardware and software, and algorithms, are scalable to collect image samples from seeds of other species during germination and annotate them to learn new classification models, which can also be used for online monitoring of other seed germination tests.In addition, in terms of computational cost, the time for the algorithm to process one frame is mainly related to the hardware configuration of the processor. In the process of this experiment, the system configuration is windows 10; CPU is Intel 10th-generation i3-10100 with default frequency of 3.6 GHz, 4 cores, and 8 threads; memory capacity is 32G; GPU is Nvidia’s 2080Ti, and this algorithm mainly uses GPU for reference inference. The online recognition system shown in Figure4(a) uses an industrial camera with resolution of 640 × 480 and an average frame rate of 15 fps. The image resolution involved in Figure 12 is 800 × 800, and the average frame rate is 13 fps. As the resolution increases and the seed density increases, the recognition rate decreases, but does not show a significant decrease. Therefore, it is fully adequate for online monitoring of slowly germinating seeds.


## 5. Conclusion

This paper constructs a system for automatic analysis of germination rate during germination tests, including the design of hardware devices, system software, and deep learning-based algorithms for seed germination status detection, and the main contributions of this paper are as follows:(1)
This system can replace the current experimental means of manual repeated counting as the main operation method, which reduces the operational difficulty of seed germination test staff to a greater extent and improves the work efficiency.(2)
The DDST-CenterNet algorithm used in this system detects seeds with different germination status. The effect of DDST-CenterNet is significantly improved compared with the original algorithm in seed germination monitoring.(3)
The system has good scalability; the background, lighting, camera type, and other requirements of the seeds do not need to be customized; the seeds can be placed in different culture environments, using a variety of commonly used image acquisition equipment, or even images obtained from mobile phones, can be used as input; and the system can give a high accuracy rate of results, easy to promote.Although better results were achieved, there is a lack of datasets on seed germination status classification because the deep learning-based target detection algorithm requires a large number of labeled training samples to obtain a prediction model. Therefore, this system combined with the detection needs of our university and only constructed different varieties of rape seed germination classification datasets, so it cannot be used for online monitoring of other seed germination tests such as rice and corn for the time being. This research group will later seek cooperation with other agronomy research teams to obtain more species germination test image samples to extend the application of this system.



---

*Source: 4678316-2022-10-06.xml*