# A Quadratic Traversal Algorithm of Shortest Weeding Path Planning for Agricultural Mobile Robots in Cornfield

Authors: Le Zhang; Rui Li; Zhiqiang Li; Yuyao Meng; Jinxin Liang; Leiyang Fu; Xiu Jin; Shaowen Li
Journal: Journal of Robotics
 (2021)

---

## Abstract

In order to improve the weeding efficiency and protect farm crops, accurate and fast weeds removal guidance to agricultural mobile robots is an utmost important topic. Based on this motivation, we propose a time-efficient quadratic traversal algorithm for the removal guidance of weeds around the recognized corn in the field. To recognize the weeds and corns, a Faster R-CNN neural network is implemented in real-time recognition. Then, an ultra-green characterization (EXG) hyperparameter is used for grayscale image processing. An improved OTSU (IOTSU) algorithm is proposed to accurately generate and optimize the binary image. Compared to the traditional OTSU algorithm, the improved OTSU algorithm effectively shortens the search speed of the algorithm and reduces the calculation processing time by compressing the range of the search grayscale range. Finally, based on the contour of the target plants and the Canny edge detection operator, the shortest weeding path guidance can be calculated by the proposed quadratic traversal algorithm. The experimental results proved that our search success rate can reach 90.0% on the testing date. This result ensured the accurate selection of the target 2D coordinates in the pixel coordinate system. Transforming the target 2D coordinate point in the pixel coordinate system into the 3D coordinate point in the camera coordinate system as well as using a depth camera can achieve multitarget depth ranging and path planning for an optimized weeding path.


---

## Body

## 1. Introduction

In order to achieve the green and pollution-free growth of the whole life cycle of field crops and the sustainable development of agriculture, many scientific researchers focus on the field of fully automatic weeding by weeding robots [1–4]. The emergence and use of agricultural mobile robots [5–9] not only can replace humans to complete dull and repetitive agricultural work but also can efficiently and continuously work in different outdoor environments. The use of robotic weeding techniques can also improve production efficiency and effectively liberate the human’s hands. Therefore, under the natural growth environment conditions, accurate and rapid identification and removal of weeds in field crops play an important role in achieving intelligent field management [10–14].Until now, many researchers have carried out specific research studies on the removal of weeds in field crops. Maruyama and Naruse developed a small weeding robot for rice fields [15]. They proposed an approach of moving multiple robots around a field to prevent weed seeds from sprouting. But the small weeding robot can only be used to prevent the seeds germination of weeds in rice fields. Nan et al. proposed a machine-vision-based method to locate crops by providing real-time positional information of crop plants for a mechanical intrarow weeding robot [16]. This method is only used to remove weeds in a certain area. Zhang et al. proposed a navigation method for weeding robots based on the smallest univalue segment assimilating nucleus (SUSAN) corner and improved sequential clustering algorithm [17] and does not involve the removal of weeds in paddy fields. Malavaz et al. developed a general and robust approach for autonomous robot navigation inside a crop by using light detection and ranging (LiDAR) data [18]. The approach can only detect the distribution of weeds by LiDAR and cannot perform ranging. Gokul et al. developed a trainable, automatic robot which helps to remove unwanted weed on agricultural fields by using a gesture to control based on three-axis robotic arm to do the necessary work [19]. This study only introduces the design of the robot and does not involve the specific identification and location of weeds. Chechliński et al. solved the bottleneck problem in terms of the deployment of deep network models on low-cost computers, the bottleneck problem above, which has laid a research foundation for further transplantation into agricultural robot tasks [20]. The core of this study is the study of deep network models, rather than solving the problem of how to remove weeds in the field. Kanagasingham et al. attempted to integrate GNSS, compass, and machine vision into a rice field weeding robot to achieve fully autonomous navigation for the weeding operation [21]. A novel crop row detection algorithm was developed to extract the four immediate rows spanned by a camera installed at the front of the robot. The algorithm cannot accurately locate weeds in rice row.


## 2. Related Works

For robotic path planning, visual detection methods and other sensors-based detection methods are widely implemented. Choi et al. found that the guidance line extracted from an image of a rice row had precisely guided a robot for weed control in paddy fields and proposed a new guidance line extraction algorithm to improve the navigation accuracy of the weeding robots in paddy fields [22]. Hossain and Ferdous et al. developed a new algorithm based on the bacterial foraging optimization (BFO) technique [23]. They also explored the application of BFO in terms of the mobile robot navigation to determine the shortest feasible path to move from any current position to the target position in an unknown environment with moving obstacles. Contrerascruz et al. proposed an evolutionary approach to solve the mobile robot path planning [24]. The proposed approach combines the artificial bee colony algorithm as a local search procedure and the evolutionary programming algorithm to refine the feasible path detected by a set of local procedures. The methods they proposed above are mainly used for robotic path planning, and most of them did not consider that the removal of weeds also requires path planning.Andújar et al. proposed a method for estimating the volume of weeds by using a depth camera [25]. Reconstructing a 3D point cloud of corn crops infested with weeds in the field and using the Kinect device to estimate volume can determine the crop status and achieve accurate estimates of weed height. Bakhshipour and Jafari used the morphological characteristics of crops to evaluate the application of the support vector machine (SVM) and artificial neural network in weed detection [26]. Through experimental comparison, it was concluded that the method based on SVM can be better used for weed detection. Xu et al. developed a real-time weed positioning and variable speed herbicide spraying (VRHS) system for row crops [27]. They proposed an improved particle swarm optimization (IPSO) algorithm for wild cornfield weed images segmentation, which optimizes the traditional particle swarm optimization algorithm to meet the real-time data processing needs of field management. The abovementioned researchers have realized the detection and segmentation of weeds through traditional machine learning methods. However, they failed to provide a way to accurately locate weeds.In order to reduce route length and operation time, Ya et al. proposed a new path-planning algorithm for static weeding [28]. At the same time, to demonstrate the feasibility and improve the implementation of laser weeding, a prototype robot was built and equipped with machine vision and gimbal mounted laser pointers camera. Liu et al. designed an on-site imaging spectrometer system to distinguish crop and weed targets [29]. The use of a limited number of spectral bands can achieve multicategory distinction between weeds or between crops and weeds. In general, some of the algorithms or specific systems developed above have achieved effective experimental results, but none of them have qualitatively analysed and measured the distance between crops and weeds as well as between weeds and weeds. In addition, the path planning guidance of the weed removal for protecting target crops was missing from above research.To solve the above problems as well as provide an efficient and accurate weeds removing guidance, this study proposes an efficient quadratic traversal algorithm for the field weeding robot. Through the combination of deep learning technology and traditional algorithms, a depth camera is used to accurately measure the distance of corn and weeds. And then, a shortest weeding path around the crops is further planned for an efficient weeding process. The proposed method provides a better way to assist intelligent agricultural mobile robots to precisely perform weeding operations [30–33]. The implementation of our method can facilitate the intelligent weeding robot to better perform precise weeding operations and improve the robotic working efficiency.


## 3. System Framework

Figure1 shows an overview of the system framework of agricultural mobile robots for cornfield weeding. The detailed function introduction of the proposed system is as follows.Figure 1
Overview of the system framework. (1) Depth camera is used to obtain real-time image information of cornfield. (2) Data preprocessing is used to complete the preparation of corn and weed images. (3) Improved OTSU algorithm is used to obtain the optimal segmentation threshold for the corn and weed grayscale images. (4) Edge detection based on Canny is used for the contour extraction of corn and weed targets. (5) Quadratic traversal algorithm is used to extract the corresponding 2D coordinate information. (6) Depth ranging and shortest path planning is used to show the experimental effect of our proposed method.The depth camera is used to obtain real-time images from the video stream as RGB color images. It is further used to achieve multitarget depth ranging and path planning for an optimized weeding path. The data preprocessing mainly includes target recognition and processing of grayscale image. Among them, target recognition is used for corn and weed images target recognition and automatic cutting, and the processing of grayscale image is based on the EXG method to achieve grayscale image under the RGB color space. On the basis of ensuring the performance of the algorithm, the improved OTSU algorithm can effectively reduce the calculation cost by compressing the range of the search grayscale range. The edge detection based on Canny can realize the contour extraction of corn and weed targets. The quadratic traversal algorithm includes two parts: the first traversal and the second traversal. The first traversal is used to extract the specified area in the contour edge image. The second traversal is used to determine the corresponding 2D coordinate information extracted on the keyframe colorful image from the video stream. Depth ranging and shortest path planning can verify the feasibility of our proposed method and achieve the experimental goals.


## 4. Image Processing Method and Quadratic Traversal Algorithm

### 4.1. Data Preprocessing Method

Under natural conditions in the field, in order to achieve automatic recognition of corn and weed, a Faster R-CNN deep neural network based on the VGG-16 feature extraction network [34–36] is trained on the collected corn and weed image data, so as to obtain a deep network model for automatically identifying corn and weed targets.The depth camera (RealSense D435i) needs to be turned on to obtain RGB images and depth information images aligned with it, and then, keyframe images (640×480) need to be extracted from the video stream as an RGB image. The RGB image is compressed into a size of 500×400 and imported into a deep network model for target recognition. The calculated corn-weed target recognition result is further used to facilitate subsequent image automatic cutting.Before grayscale image processing, we need to obtain the number of corn and weed target images in database A and database B. Then, record the original lengthL and width W of the target image after cutting. After that, it is zoomed out to a target size. Here, we use 640×480 pixels. Finally, the zoomed image is cut (the center of the cut is 500×400 pixels). The purpose of it is to retain the main information of the target image. At this point, the preparation of grayscale image processing data is completed.By observing the cropped images of corn and weed targets, it is not difficult to find the color difference among the corn-weed targets and the soil background, which is easy to distinguish and identify by traditional algorithms. According to the value of the target image in the three color channels ofR, G, and B, these are obviously different. In this study, by studying the linear combination of the characteristics of the three color components of R, G, and B, the green area of the target image can be better extracted. In order to select the best method for extracting the green area of the target image, the EXG method, the GMR method for the RGB color space, and the Cg method for the YCrCb color space [37–39] are implemented. The detailed comparison and analysis are performed in Section 6.1. An ultra-green characterization (EXG) hyperparameter is used for grayscale images. The specific formula is as follows:(1)GrayEXG=2g−r−b.We can obtain the maximum (maxVal) and minimum (maxVal) values in the gray array, and the formula (2) can be used to convert the grayEXG array into an grayEXG array for subsequent optimal segmentation threshold selection.(2)GrayEXG=Gray−minvalMaxVal−minval×255.


### 4.2. Improved OTSU Algorithm

The OTSU algorithm, which can automatically calculate the segmentation threshold, has been widely used in the field of agricultural image processing [40, 41]. Through the leveraging of the OTSU features, we propose an improved OTSU (IOTSU) algorithm. On the basis of ensuring the performance of the algorithm, the IOTSU algorithm effectively improves the search speed and reduces the calculation processing time by compressing the range of the search grayscale range. The following steps specifically summarize the detailed contents of the IOTSU algorithm:(a)
For an imported grayscale image, equations (3) and (4) are used to complete the parameter definition and initialization:(3)ω1=N1M×N,(4)ω2=N2M×N,whereω1 is the ratio of the total number of pixels of in the foreground target image for the total number of pixels in the entire image, ω2 is the ratio of the total number of pixels in the background image for the total number of pixels in the entire image; N1 indicates the total number of pixels in the image whose grayscale value is less than the foreground and background segmentation threshold T, N2 indicates the total number of pixels in the image whose grayscale value is more than the foreground and background segmentation threshold T; and M×N is the total number of pixels in the entire image.(b)
The relationship expressions (5) and (6) can be obtained from equations (3) and (4) and the inherent relationship of the parameters(5)N1+N2=M×N,(6)ω1+ω2=1,(c)
μ is the average gray level of all pixels of the input grayscale image. Then, the average gray level μ1 of the pixels of the foreground target image and the average gray level μ2 of the pixels of the background image can be calculated by(7)μ1=1ω1×∑i=0Tiωi,μ2=1ω2×∑i=T+1L−1iωi.In addition, there is a linear correlation amongμ, μ1, and μ2. The linear relationship expression is given in equation (8):(8)μ=ω1×μ1+ω2×μ2.(d)
According to the equation in the above steps, a maximum variance between clusters can be obtained by equation (9). By putting equation (8) into equation (9) and simplifying it, the corresponding equivalent equation is given as equation (10):(9)G=ω1μ1−μ2+ω2μ2−μ2,(10)G=ω1×ω2μ1−μ22.(e)
Traverse the compressed grayscale interval to obtain the segmentation thresholdT when the variance between classes is maximum. T is the optimal segmentation threshold. First, we need to obtain the average gray level μ in step (c). Second, obtain the minimum gray value gmin and the maximum grayscale value gmax in the grayscale interval of the grayscale image. Finally, in the grayscale interval (gmin, gmax), the golden section points on the left and right sides of the average gray level are used to as the compressed grayscale interval (0.382μ+0.618gmin, 0.382μ+0.618gmax).(f)
The equivalent equation (10) is used to traverse the grayscale interval (0.382μ+0.618gmin, 0.382μ+0.618gmax). Obtain the segmentation threshold T of the foreground target image and the background image that maximized the variance between the classes. According to the obtained segmentation threshold T, use equation (11) to generate the binary image for the imported grayscale image.(11)Pi,j=mIi,j>T0Ii,j≤T,wherem as a substitute parameter represents the maximum value in the grayscale interval, and the value is 255; Ii,j is the grayscale value of the pixels i,j of the grayscale image, and Pi,j is the binary image generated by equation (11).(g)
For the generated binary image, we first perform an area threshold filtering operation. The purpose of it is to remove the background image that wrongly divided into the foreground target image. Then, the Gaussian filtering is performed to remove the noise information in the binary image. Finally, morphological operations are performed to smooth the binary image for obtaining an optimized binary image, which is used for subsequent image edge detection.


### 4.3. Edge Detection Algorithm Based on Canny

Complete and effective edge contour image information plays an important role in studying the characteristics of corn and weed targets in the field. It is also convenient for the corn and weed targets to accurately select the 2D coordinate points in the subsequent study. Generally, in order to extract more complete and effective edge contour image information as much as possible, an appropriate edge detection algorithm can be selected by studying the extraction effect of the edge detection operator [42]. In this study, the binary image after optimization of the corn and weed targets is taken as the research object. We first compare two kinds of second-order edge detection operators: Canny operator and Laplacian operator, and then, we compare three kinds of first-order edge detection operators: Sobel operator, Roberts operator, and Prewitt operator.


### 4.4. Quadratic Traversal Algorithm

In order to realize an accurate selection of the 2D coordinate points of the corn and weed targets in the field crop image, the edge contour image of the corn and weed targets is taken as the research object. In this study, a quadratic traversal algorithm is proposed for selecting target 2D coordinate points in the pixel coordinate system, and the corresponding traversal search box is designed. The algorithm’ main implementation steps are as follows:S1. Define a row step size ispi pixels, a column step size is qj pixels, and a traversal search box size is mi×nj. Calculate the number of row direction traversal search boxes fRi and the number of column direction traversal search boxes fCj in the target contour edge image of size M×N. Here, the value of fRi is the number of search boxes walked in each row when the traversal search box advances in the row direction; the value of fCj is the number of search boxes walked in each column when the traversal search box advances in the column direction.S2. The traversal search box is used to traverse the edge contour image of corn and weed objects according to the priority row traversal method. For the traversal search box, store the number of pixels that meet the set conditions in the database C in sequence. Where, the setting conditions are that the values ofR, G, and B are all more than 250. At the same time, record the corresponding serial number fSi of the traversal search box.S3. Obtain the serial numberfSi corresponding to the traversal search box with the largest number of pixels that meets the set conditions in the database C. It should be noted that the serial number fSi is a positive integer counting from one.S4. Calculate the position information of the traversal search box on the edge contour image of the target by using the corresponding serial numberfSi of the traversal search box. Here, fLi and fWj are the natural numbers counting from zero, which are used to represent the row and column position information of the traversal search box on the edge contour image of the target. The specific calculation equation is(12)fLi=fSi−1fRi,fWj=fSi−1−fRi×fLi.S5. Calculate the 2D coordinate points information of the upper left endsfLi_min,fWj_min and lower right ends fLi_max,fWj_max of the traversal search box by using the row fLi and fWj column position information. The specific calculation equation is(13)fLi_min=0+pi×fWj,fWj_min=0+qj×fLi,fLi_max=mi+pi×fWj,fWj_max=nj+qj×fLi.S6. According to the 2D coordinate points information of the upper left endsfLi_min,fWj_min and lower right ends fLi_max,fWj_max of the traversal search box, the specified area (mi×nj) on the edge contour image of the target can be implemented.S7. Select the pixel coordinatesfi,fj that meet the set conditions in the traversal search box. The pixel coordinates fi,fj are fixed as close as possible to the center of the traversal search box of the second traversal and then back to the corresponding 2D coordinate points information fx,fy on the keyframe image. The specific calculation equation is(14)fx=fi+∑i=1nfLi_min+p×L+P×xminM,fy=fj+∑jnfWj_min+q×W+Q×yminN.In order to further describe our core work for the quadratic traversal algorithm in detail, as shown in Figure2, taking corn in the image of field crops as an example, the detailed explanation of each step of the quadratic traversal algorithm is shown.Figure 2
Detailed explanation diagram of each step of the quadratic traversal algorithm.Among them, Figure2(a) shows the preparation processing of the data preprocessing part, and the corresponding key information is marked on the image. Here, the lower ends of Figure 2(a) are the corn-weed target recognition result image, and the upper ends of Figure 2(a) are the processed result of cutting the corn target from the target recognition result image. Figure 2(b) introduces the core steps of the quadratic traversal algorithm and annotates the corresponding location information. Here, the lower ends of Figure 2(b) are enlarged images for a local area with a size of mi×nj in the traversal search box. The pixel coordinates fi,fj are located in the x1−y1 coordinate system, and the upper left ends fLi_min,fWj_min and lower right ends fLi_max,fWj_max of the traversal search box are located in the x2−y2 or x3−y3 coordinate system. The upper ends of Figure 2(b) are reserved for the edge contour image of the corn target with the main information area. Figure 2(c) demonstrates the result of using equation (16) to calculate the pixel coordinates fi,fj back to the corresponding 2D coordinate information fx,fy on the keyframe image. The 2D coordinate information fx,fy is located in the x4−y4 coordinate system.


### 4.5. Depth Ranging and Shortest Path Planning

The accurate selection of the 2D coordinate point of the corn and weed targets is obtained in the cropped image. Then, the target distance measurement and the shortest weeding path planning work are required. There are generally four coordinate systems in the field of computer vision. They are the pixel coordinate system, imaging coordinate system, camera coordinate system, and world coordinate system. Generally, the depth camera ranging research work is implemented in the camera coordinate system. Its core content is to transform the target 2D coordinate point in the pixel coordinate system into the 3D coordinate point in the camera coordinate system.However, the 2D coordinate point informationfx,fy is the coordinate in the pixel coordinate system, and the corresponding 3D coordinate point information X,Y,Z in the camera coordinate system needs to be generated through coordinate system conversion. At this time, the acquisition of camera internal parameters needs to be completed by the depth camera calibration [43, 44]. camera_cx,camera_cy represents the principal point coordinates in the imaging coordinate system, which is used to realize the conversion between the pixel coordinate system and the imaging coordinate system. camera_fx and camera_fy represents the focal length of the depth camera, which is used to realize the conversion between the imaging coordinate system and the camera coordinate system.Next, it is necessary to obtain depth informationd for the 2D coordinate point fx,fy of the target in the depth image aligned with the colored image and obtain the ratio of the depth pixel to the real unit depth_scale. In the end, the conversion from the pixel coordinate system to the camera coordinate system can be directly completed by equation (17). So that, the target 2D coordinate point in the pixel coordinate system can be transformed into the 3D coordinate point in the camera coordinate system. By using the distance formula between two points in the 3D space, the distance between the corn target and the weed target as well as the distance between the weed target and the weed target can be implemented.(15)X=fy−camera_cx×Zcamera_fx,Y=fx−camera_cy×Zcamera_fy,Z=d×depth_scale.Based on realizing the multiobject depth ranging research works, we take the corn crop target as the starting position in the shortest weeding path planning research. Using the Dijkstra algorithm for the shortest weeding path planning can achieve excellent experimental result [45]. Figure 3 shows a detailed diagram of the Dijkstra algorithm.Figure 3
Schematic diagram of the corresponding shortest weeding path calculated manually. (a) The results of using the manual measurement method to carry out distance data statistics. (b) The manual calculation process of the shortest path planning algorithm Dijkstra.
(a)(b)


## 4.1. Data Preprocessing Method

Under natural conditions in the field, in order to achieve automatic recognition of corn and weed, a Faster R-CNN deep neural network based on the VGG-16 feature extraction network [34–36] is trained on the collected corn and weed image data, so as to obtain a deep network model for automatically identifying corn and weed targets.The depth camera (RealSense D435i) needs to be turned on to obtain RGB images and depth information images aligned with it, and then, keyframe images (640×480) need to be extracted from the video stream as an RGB image. The RGB image is compressed into a size of 500×400 and imported into a deep network model for target recognition. The calculated corn-weed target recognition result is further used to facilitate subsequent image automatic cutting.Before grayscale image processing, we need to obtain the number of corn and weed target images in database A and database B. Then, record the original lengthL and width W of the target image after cutting. After that, it is zoomed out to a target size. Here, we use 640×480 pixels. Finally, the zoomed image is cut (the center of the cut is 500×400 pixels). The purpose of it is to retain the main information of the target image. At this point, the preparation of grayscale image processing data is completed.By observing the cropped images of corn and weed targets, it is not difficult to find the color difference among the corn-weed targets and the soil background, which is easy to distinguish and identify by traditional algorithms. According to the value of the target image in the three color channels ofR, G, and B, these are obviously different. In this study, by studying the linear combination of the characteristics of the three color components of R, G, and B, the green area of the target image can be better extracted. In order to select the best method for extracting the green area of the target image, the EXG method, the GMR method for the RGB color space, and the Cg method for the YCrCb color space [37–39] are implemented. The detailed comparison and analysis are performed in Section 6.1. An ultra-green characterization (EXG) hyperparameter is used for grayscale images. The specific formula is as follows:(1)GrayEXG=2g−r−b.We can obtain the maximum (maxVal) and minimum (maxVal) values in the gray array, and the formula (2) can be used to convert the grayEXG array into an grayEXG array for subsequent optimal segmentation threshold selection.(2)GrayEXG=Gray−minvalMaxVal−minval×255.


## 4.2. Improved OTSU Algorithm

The OTSU algorithm, which can automatically calculate the segmentation threshold, has been widely used in the field of agricultural image processing [40, 41]. Through the leveraging of the OTSU features, we propose an improved OTSU (IOTSU) algorithm. On the basis of ensuring the performance of the algorithm, the IOTSU algorithm effectively improves the search speed and reduces the calculation processing time by compressing the range of the search grayscale range. The following steps specifically summarize the detailed contents of the IOTSU algorithm:(a)
For an imported grayscale image, equations (3) and (4) are used to complete the parameter definition and initialization:(3)ω1=N1M×N,(4)ω2=N2M×N,whereω1 is the ratio of the total number of pixels of in the foreground target image for the total number of pixels in the entire image, ω2 is the ratio of the total number of pixels in the background image for the total number of pixels in the entire image; N1 indicates the total number of pixels in the image whose grayscale value is less than the foreground and background segmentation threshold T, N2 indicates the total number of pixels in the image whose grayscale value is more than the foreground and background segmentation threshold T; and M×N is the total number of pixels in the entire image.(b)
The relationship expressions (5) and (6) can be obtained from equations (3) and (4) and the inherent relationship of the parameters(5)N1+N2=M×N,(6)ω1+ω2=1,(c)
μ is the average gray level of all pixels of the input grayscale image. Then, the average gray level μ1 of the pixels of the foreground target image and the average gray level μ2 of the pixels of the background image can be calculated by(7)μ1=1ω1×∑i=0Tiωi,μ2=1ω2×∑i=T+1L−1iωi.In addition, there is a linear correlation amongμ, μ1, and μ2. The linear relationship expression is given in equation (8):(8)μ=ω1×μ1+ω2×μ2.(d)
According to the equation in the above steps, a maximum variance between clusters can be obtained by equation (9). By putting equation (8) into equation (9) and simplifying it, the corresponding equivalent equation is given as equation (10):(9)G=ω1μ1−μ2+ω2μ2−μ2,(10)G=ω1×ω2μ1−μ22.(e)
Traverse the compressed grayscale interval to obtain the segmentation thresholdT when the variance between classes is maximum. T is the optimal segmentation threshold. First, we need to obtain the average gray level μ in step (c). Second, obtain the minimum gray value gmin and the maximum grayscale value gmax in the grayscale interval of the grayscale image. Finally, in the grayscale interval (gmin, gmax), the golden section points on the left and right sides of the average gray level are used to as the compressed grayscale interval (0.382μ+0.618gmin, 0.382μ+0.618gmax).(f)
The equivalent equation (10) is used to traverse the grayscale interval (0.382μ+0.618gmin, 0.382μ+0.618gmax). Obtain the segmentation threshold T of the foreground target image and the background image that maximized the variance between the classes. According to the obtained segmentation threshold T, use equation (11) to generate the binary image for the imported grayscale image.(11)Pi,j=mIi,j>T0Ii,j≤T,wherem as a substitute parameter represents the maximum value in the grayscale interval, and the value is 255; Ii,j is the grayscale value of the pixels i,j of the grayscale image, and Pi,j is the binary image generated by equation (11).(g)
For the generated binary image, we first perform an area threshold filtering operation. The purpose of it is to remove the background image that wrongly divided into the foreground target image. Then, the Gaussian filtering is performed to remove the noise information in the binary image. Finally, morphological operations are performed to smooth the binary image for obtaining an optimized binary image, which is used for subsequent image edge detection.


## 4.3. Edge Detection Algorithm Based on Canny

Complete and effective edge contour image information plays an important role in studying the characteristics of corn and weed targets in the field. It is also convenient for the corn and weed targets to accurately select the 2D coordinate points in the subsequent study. Generally, in order to extract more complete and effective edge contour image information as much as possible, an appropriate edge detection algorithm can be selected by studying the extraction effect of the edge detection operator [42]. In this study, the binary image after optimization of the corn and weed targets is taken as the research object. We first compare two kinds of second-order edge detection operators: Canny operator and Laplacian operator, and then, we compare three kinds of first-order edge detection operators: Sobel operator, Roberts operator, and Prewitt operator.


## 4.4. Quadratic Traversal Algorithm

In order to realize an accurate selection of the 2D coordinate points of the corn and weed targets in the field crop image, the edge contour image of the corn and weed targets is taken as the research object. In this study, a quadratic traversal algorithm is proposed for selecting target 2D coordinate points in the pixel coordinate system, and the corresponding traversal search box is designed. The algorithm’ main implementation steps are as follows:S1. Define a row step size ispi pixels, a column step size is qj pixels, and a traversal search box size is mi×nj. Calculate the number of row direction traversal search boxes fRi and the number of column direction traversal search boxes fCj in the target contour edge image of size M×N. Here, the value of fRi is the number of search boxes walked in each row when the traversal search box advances in the row direction; the value of fCj is the number of search boxes walked in each column when the traversal search box advances in the column direction.S2. The traversal search box is used to traverse the edge contour image of corn and weed objects according to the priority row traversal method. For the traversal search box, store the number of pixels that meet the set conditions in the database C in sequence. Where, the setting conditions are that the values ofR, G, and B are all more than 250. At the same time, record the corresponding serial number fSi of the traversal search box.S3. Obtain the serial numberfSi corresponding to the traversal search box with the largest number of pixels that meets the set conditions in the database C. It should be noted that the serial number fSi is a positive integer counting from one.S4. Calculate the position information of the traversal search box on the edge contour image of the target by using the corresponding serial numberfSi of the traversal search box. Here, fLi and fWj are the natural numbers counting from zero, which are used to represent the row and column position information of the traversal search box on the edge contour image of the target. The specific calculation equation is(12)fLi=fSi−1fRi,fWj=fSi−1−fRi×fLi.S5. Calculate the 2D coordinate points information of the upper left endsfLi_min,fWj_min and lower right ends fLi_max,fWj_max of the traversal search box by using the row fLi and fWj column position information. The specific calculation equation is(13)fLi_min=0+pi×fWj,fWj_min=0+qj×fLi,fLi_max=mi+pi×fWj,fWj_max=nj+qj×fLi.S6. According to the 2D coordinate points information of the upper left endsfLi_min,fWj_min and lower right ends fLi_max,fWj_max of the traversal search box, the specified area (mi×nj) on the edge contour image of the target can be implemented.S7. Select the pixel coordinatesfi,fj that meet the set conditions in the traversal search box. The pixel coordinates fi,fj are fixed as close as possible to the center of the traversal search box of the second traversal and then back to the corresponding 2D coordinate points information fx,fy on the keyframe image. The specific calculation equation is(14)fx=fi+∑i=1nfLi_min+p×L+P×xminM,fy=fj+∑jnfWj_min+q×W+Q×yminN.In order to further describe our core work for the quadratic traversal algorithm in detail, as shown in Figure2, taking corn in the image of field crops as an example, the detailed explanation of each step of the quadratic traversal algorithm is shown.Figure 2
Detailed explanation diagram of each step of the quadratic traversal algorithm.Among them, Figure2(a) shows the preparation processing of the data preprocessing part, and the corresponding key information is marked on the image. Here, the lower ends of Figure 2(a) are the corn-weed target recognition result image, and the upper ends of Figure 2(a) are the processed result of cutting the corn target from the target recognition result image. Figure 2(b) introduces the core steps of the quadratic traversal algorithm and annotates the corresponding location information. Here, the lower ends of Figure 2(b) are enlarged images for a local area with a size of mi×nj in the traversal search box. The pixel coordinates fi,fj are located in the x1−y1 coordinate system, and the upper left ends fLi_min,fWj_min and lower right ends fLi_max,fWj_max of the traversal search box are located in the x2−y2 or x3−y3 coordinate system. The upper ends of Figure 2(b) are reserved for the edge contour image of the corn target with the main information area. Figure 2(c) demonstrates the result of using equation (16) to calculate the pixel coordinates fi,fj back to the corresponding 2D coordinate information fx,fy on the keyframe image. The 2D coordinate information fx,fy is located in the x4−y4 coordinate system.


## 4.5. Depth Ranging and Shortest Path Planning

The accurate selection of the 2D coordinate point of the corn and weed targets is obtained in the cropped image. Then, the target distance measurement and the shortest weeding path planning work are required. There are generally four coordinate systems in the field of computer vision. They are the pixel coordinate system, imaging coordinate system, camera coordinate system, and world coordinate system. Generally, the depth camera ranging research work is implemented in the camera coordinate system. Its core content is to transform the target 2D coordinate point in the pixel coordinate system into the 3D coordinate point in the camera coordinate system.However, the 2D coordinate point informationfx,fy is the coordinate in the pixel coordinate system, and the corresponding 3D coordinate point information X,Y,Z in the camera coordinate system needs to be generated through coordinate system conversion. At this time, the acquisition of camera internal parameters needs to be completed by the depth camera calibration [43, 44]. camera_cx,camera_cy represents the principal point coordinates in the imaging coordinate system, which is used to realize the conversion between the pixel coordinate system and the imaging coordinate system. camera_fx and camera_fy represents the focal length of the depth camera, which is used to realize the conversion between the imaging coordinate system and the camera coordinate system.Next, it is necessary to obtain depth informationd for the 2D coordinate point fx,fy of the target in the depth image aligned with the colored image and obtain the ratio of the depth pixel to the real unit depth_scale. In the end, the conversion from the pixel coordinate system to the camera coordinate system can be directly completed by equation (17). So that, the target 2D coordinate point in the pixel coordinate system can be transformed into the 3D coordinate point in the camera coordinate system. By using the distance formula between two points in the 3D space, the distance between the corn target and the weed target as well as the distance between the weed target and the weed target can be implemented.(15)X=fy−camera_cx×Zcamera_fx,Y=fx−camera_cy×Zcamera_fy,Z=d×depth_scale.Based on realizing the multiobject depth ranging research works, we take the corn crop target as the starting position in the shortest weeding path planning research. Using the Dijkstra algorithm for the shortest weeding path planning can achieve excellent experimental result [45]. Figure 3 shows a detailed diagram of the Dijkstra algorithm.Figure 3
Schematic diagram of the corresponding shortest weeding path calculated manually. (a) The results of using the manual measurement method to carry out distance data statistics. (b) The manual calculation process of the shortest path planning algorithm Dijkstra.
(a)(b)


## 5. Experimental Setup

### 5.1. Cornfield Mobile Robotics Platform

GPS is the abbreviation of the global positioning system. It is an omnidirectional, all-weather, all-time, high-precision satellite navigation system that can provide global users with low-cost, high-precision three-dimensional position, speed, and precise timing navigation information. Lidar (VLP-16) is responsible for constructing a real-time 2D or 3D navigation map of the cornfield at close range and providing real-time 3D point cloud information around it, which can further provide precise navigation information for the cornfield mobile robotics platform. The depth camera as an RGB-D camera has a pair of left-eye and right-eye stereo infrared cameras, infrared dot-matrix laser emitters, and RGB cameras [46, 47]. The size is 90 mm × 25 mm × 25 mm, suitable for indoor and outdoor environments. The depth camera is based on the triangulation method for binocular stereo distance measurement, in which a pair of stereo infrared cameras is used to collect depth information of the target, and an infrared dot-matrix laser emitter is used to project certain structural features of light on the target in the visual scene. RGB camera is used to collect color image data and can achieve color image video stream and depth image video stream alignment. The maximum distance can be up to 10 meters. It is widely used in research fields such as drones, robots, and AR/VR.Universal robots (UR5), as a collaborative robotic arm, have six rotary joints (degrees of freedom) and can perform automated tasks with a maximum load of 5 kg. Its effective working radius is up to 850 mm. Robotic mobile base (Husky A200) is used as the mobile carrier of the cornfield mobile robotics platform. It uses four-wheel drive to work. The maximum payload is 75 Kg, and the maximum speed can reach 1 m/s. As a high-performance processing unit, the workstation is essentially an industrial personal computer. On the one hand, it is used to deploy the algorithm program we designed. On the other hand, it is used to communicate with the abovementioned key devices.On a workstation with Windows 10 operating system, run the supporting software development program (Intel RealSense SDK 2.0) based on RealSense D435i and compile and generate image acquisition and target ranging software again. The image acquisition and target ranging software includes the driver of the RealSense D435i depth camera, which will allow the depth camera to collect image depth information and RGB information at a rate of not less than 20 frames per second (fps). At the same time, the pixels of the collected image are processed to 640× 480. Its purpose is to measure the distance between corn and weeds and between weeds and weeds based on the currently acquired image depth information and then plan the shortest weeding path. Figure 4 shows specific details.Figure 4
Schematic design of the overall structure with the weeding robot for corn-weed detection. GPS is responsible for providing outdoor navigation information for agricultural mobile robots. Lidar is responsible for further providing precise navigation information for robotic mobile base. Depth camera is responsible for multitarget depth ranging and shortest path planning. Universal robots are responsible for carrying weeding actuators for multitask weeding operations. Robotic mobile base is responsible for autonomous movement in the cornfield as a mobile carrier. Workstation is responsible for deploying the algorithm program and communicating with the abovementioned key devices.


### 5.2. Data Collection and Preprocessing

The corn field data of this experiment were collected in the agricultural experimental field, called “Nong Cui Yuan,” of Anhui Agricultural University. According to the time of seeding and growth of corn, our data collection days are from May 1 to May 4 in 2019. In order to ensure a clear image collection, our collection time depended on a strong visibility which was 9 : 00 AM to 12 : 00AM and 2 : 00PM to 5 : 00 PM. In the collection process, images of corn and weeds under natural conditions were collected in the three directions of head-up, top-view, and 45 degree squint, and the collection steps were strictly followed. The image data acquisition equipment used a high-definition digital camera (Canon EOS 6D Mark II camera). A total of 3906 corns with weed images were collected. All images are in JPEG format with resolution of 5472×3648 pixels. Some of the corn and weed images collected are shown in Figure 5. Figures 5(a)–5(c) show different numbers of corn and weeds in the agricultural experimental field, respectively.Figure 5
Corn and weed images under natural conditions were collected in the agricultural experimental field. (a) Single plant of corn and single weed images. (b) Single plant of corn and multiple weeds images. (c) Multiple corn and multiple weeds images.
(a)(b)(c)To improve the computational time efficiency as well as to preserve the useful information in the images, it is better to compress and reduce the resolution to 500×400 according to our experience. In this dataset, 3500 samples are randomly set as training data, and the remaining 406 samples are testing data. The ratio is 8 : 2 according to the ratio of the training set and the verification set. The dataset was manually labeled, and the corn and weed in the images are labeled by using the minimum circumscribed rectangle method.


## 5.1. Cornfield Mobile Robotics Platform

GPS is the abbreviation of the global positioning system. It is an omnidirectional, all-weather, all-time, high-precision satellite navigation system that can provide global users with low-cost, high-precision three-dimensional position, speed, and precise timing navigation information. Lidar (VLP-16) is responsible for constructing a real-time 2D or 3D navigation map of the cornfield at close range and providing real-time 3D point cloud information around it, which can further provide precise navigation information for the cornfield mobile robotics platform. The depth camera as an RGB-D camera has a pair of left-eye and right-eye stereo infrared cameras, infrared dot-matrix laser emitters, and RGB cameras [46, 47]. The size is 90 mm × 25 mm × 25 mm, suitable for indoor and outdoor environments. The depth camera is based on the triangulation method for binocular stereo distance measurement, in which a pair of stereo infrared cameras is used to collect depth information of the target, and an infrared dot-matrix laser emitter is used to project certain structural features of light on the target in the visual scene. RGB camera is used to collect color image data and can achieve color image video stream and depth image video stream alignment. The maximum distance can be up to 10 meters. It is widely used in research fields such as drones, robots, and AR/VR.Universal robots (UR5), as a collaborative robotic arm, have six rotary joints (degrees of freedom) and can perform automated tasks with a maximum load of 5 kg. Its effective working radius is up to 850 mm. Robotic mobile base (Husky A200) is used as the mobile carrier of the cornfield mobile robotics platform. It uses four-wheel drive to work. The maximum payload is 75 Kg, and the maximum speed can reach 1 m/s. As a high-performance processing unit, the workstation is essentially an industrial personal computer. On the one hand, it is used to deploy the algorithm program we designed. On the other hand, it is used to communicate with the abovementioned key devices.On a workstation with Windows 10 operating system, run the supporting software development program (Intel RealSense SDK 2.0) based on RealSense D435i and compile and generate image acquisition and target ranging software again. The image acquisition and target ranging software includes the driver of the RealSense D435i depth camera, which will allow the depth camera to collect image depth information and RGB information at a rate of not less than 20 frames per second (fps). At the same time, the pixels of the collected image are processed to 640× 480. Its purpose is to measure the distance between corn and weeds and between weeds and weeds based on the currently acquired image depth information and then plan the shortest weeding path. Figure 4 shows specific details.Figure 4
Schematic design of the overall structure with the weeding robot for corn-weed detection. GPS is responsible for providing outdoor navigation information for agricultural mobile robots. Lidar is responsible for further providing precise navigation information for robotic mobile base. Depth camera is responsible for multitarget depth ranging and shortest path planning. Universal robots are responsible for carrying weeding actuators for multitask weeding operations. Robotic mobile base is responsible for autonomous movement in the cornfield as a mobile carrier. Workstation is responsible for deploying the algorithm program and communicating with the abovementioned key devices.


## 5.2. Data Collection and Preprocessing

The corn field data of this experiment were collected in the agricultural experimental field, called “Nong Cui Yuan,” of Anhui Agricultural University. According to the time of seeding and growth of corn, our data collection days are from May 1 to May 4 in 2019. In order to ensure a clear image collection, our collection time depended on a strong visibility which was 9 : 00 AM to 12 : 00AM and 2 : 00PM to 5 : 00 PM. In the collection process, images of corn and weeds under natural conditions were collected in the three directions of head-up, top-view, and 45 degree squint, and the collection steps were strictly followed. The image data acquisition equipment used a high-definition digital camera (Canon EOS 6D Mark II camera). A total of 3906 corns with weed images were collected. All images are in JPEG format with resolution of 5472×3648 pixels. Some of the corn and weed images collected are shown in Figure 5. Figures 5(a)–5(c) show different numbers of corn and weeds in the agricultural experimental field, respectively.Figure 5
Corn and weed images under natural conditions were collected in the agricultural experimental field. (a) Single plant of corn and single weed images. (b) Single plant of corn and multiple weeds images. (c) Multiple corn and multiple weeds images.
(a)(b)(c)To improve the computational time efficiency as well as to preserve the useful information in the images, it is better to compress and reduce the resolution to 500×400 according to our experience. In this dataset, 3500 samples are randomly set as training data, and the remaining 406 samples are testing data. The ratio is 8 : 2 according to the ratio of the training set and the verification set. The dataset was manually labeled, and the corn and weed in the images are labeled by using the minimum circumscribed rectangle method.


## 6. Experimental Results and Evaluation

### 6.1. Results of Data Preprocessing

In data preprocessing, Figure6 shows the result of corn and weed recognition as well as automatic weed removing for the Section 4.1.Figure 6
Schematic diagram of corn and weed recognition as well as automatic weed removing. (a) Recognition results of corn and weed. (b) and (c) Details of corn recognition results. (d) and (e) Details of weed recognition results.Figure7 shows the effect of grayscale image corresponding to green area extraction methods for the Section 4.1. Among them, the first column of Figure 7 is the grayscale image processing results of the EXG method, the second column of Figure 7 is the grayscale image processing results of the GMR method, and the third column of Figure 7 is the grayscale image processing results of the Cg method.Figure 7
Experimental results of grayscale image processing.
(a)(b)(c)(d)


### 6.2. Evaluation of the Extraction Methods for Greenness Detection

To better perform grayscale image processing, we select the EXG method for extracting the green area of the target image and use it to extract the green area of the corn and weed targets in the field crop image at the same time. In this study, the target of maize and weeds after shearing is taken as the research object. The only change of the grayscale generation method is used to generate multiple sets of binary image experimental data. We compared the green area extraction effect of the EXG method and GMR method in the RGB color space as well as the Cg method in the YCrCb color space. The detailed experimental results are shown in Figure8.Figure 8
The effect of extraction methods on the unoptimized OTSU algorithm. (a) and (b) Results of the corn green area extraction image. (c) and (d) Results of the weed green area extraction image.
(a)(b)(c)(d)In this study, we introduced a standard image as a benchmark (Figure8). The traditional OTSU algorithm which only changes the grayscale image generation method generates multiple sets of binary image experimental data for the comparison. Use the three indicators of ratio, variance, and standard deviation to measure the pros and cons of the EXG method, GMR method, and Cg method. Where, the ratio is defined as the ratio of the sum of the number of pixels in the black area to the sum of the number of pixels in the black area in the standard image after the binary image is generated by the unoptimized OTSU algorithm. The equation (16) is the equation for calculating variance. The detailed comparison results are shown in Table 1.(16)σ2=1N×∑i=1NAi−A¯2,where Ai is the i ratio, A¯ is the average value of the ratio, and N is the number of experimental images participating in the comparison.Table 1
The comparison results of the three measurement indicators on the extraction methods.
Color indicesFormulaRatioσ2σEXG2g−r−b1.0430.0010.031GMRg−r0.8770.0250.160Cg0.4g−0.3r−0.1b1.0020.0020.044Combining the results of Figure8 and Table 1, it can be seen that the average ratio of the Cg method in the YCrCb color space is the smallest. Due to its obvious error in the extraction of the green area of the weed target, a large area is classified in a wrong way. And it also results in an average ratio abnormality. The EXG method in the RGB color space can effectively suppress the interference of environmental factors such as soil background, dry grass, and shadow. It is more prominent for the green area of the image of extracting corn and weed targets. It is suitable for processing field crop images under natural conditions, which can obtain an ideal grayscale image effect.


### 6.3. Results of Edge Detection

By comparing the experimental results of edge detection, we can find that the edge detection algorithm based on the Canny operator has a perfect effect [48]. The specific effect is shown in Figure 9.Figure 9
Experimental results of five classical edge detection operators. (a) Corn edge contour extraction image. (b) Weed edge contour extraction image.
(a)(b)


### 6.4. Evaluation of the Quadratic Traversal Algorithm

A quadratic traversal algorithm is proposed for selecting the target 2D coordinate points. By taking a single plant of corn and weeds in the keyframe image as the research goal, a series of image processing and optimization operations are used to obtain the key corn and weed target edge contour image. The quadratic traversal algorithm is mainly based on the target edge contour image as a research basis. Through the combination of the first traversal and the second traversal, it is possible to accurately select the 2D coordinate points’ information of the target on the keyframe image.The detailed process is shown in Figure10. First, we need to cut out the target recognition results of the keyframe image and the data preparation after a series of zooming processes. Second, the EXG method in the RGB color space is used to complete the grayscale processing of the image. For the grayscale image, the IOTSU algorithm can be used to generate and optimize the binary image, and then, the Canny operator’s edge detection algorithm can be used to perform binary image processing to achieve target edge contour extraction. Finally, in the target edge contour image, the first traversal of the quadratic traversal algorithm uses the traversal search box to search the pixel area that meets the set conditions as well as cut and store it locally for the second traversal. In the quadratic traversal algorithm, the second traversal takes the output of the first traversal as the research content, also uses the other traversal search box to search the pixel area that meets the set conditions, and selects the pixel fixed as close as possible to the center position. Besides, according to the pixel point using equation (14), the pixel point can be calculated back to the corresponding position of the target in the keyframe image, so as to realize the research work of selecting 2D coordinate points.Figure 10
Schematic diagram of the quadratic traversal algorithm for selecting the target 2D coordinate points. (a) The process of selecting 2D coordinate points of corn. (b) The process of selecting 2D coordinate points of weed.For the design of the traversal search box in the first traversal method, this study is based on the size of the target edge contour image (500 × 400) processed by the edge detection, and the traversal search box can traverse the target contour edge image as much as possible. Therefore, the traversal search boxes with sizes of 50 × 40, 50 × 100, and 100 × 100 were designed to select the appropriate traversal search box size. We take thirty corn and weed target contour images as the total sample number, and the total number of successful samples and failed samples of each traversal search box were counted, respectively. Table2 presents the performance of various traversal search box sizes. The success rate of the traversal search box with a size of 100 × 100 can reach 90.0%, which can meet the actual requirements of the experiment.Table 2
The success rate of the three traversal search boxes.
Traversal boxExperimental sampleSuccess sampleFailed sampleSuccess rate (%)50 × 403022873.350 × 1003026486.6100 × 1003027390.0


### 6.5. Evaluation of Depth Ranging and Shortest Path Planning

In order to verify the feasibility of our proposed research method, we conducted a systematic test in the agricultural experimental field, called “Nong Cui Yuan,” of Anhui Agricultural University. On the one hand, we started by the keyframe image in the colored image video stream captured in real-time by the depth camera. Through the method proposed in this study, the 2D coordinate points of the corn-weed targets in the field crop image can be accurately selected. Finally, multitarget depth ranging and shortest weeding path planning can be achieved, and the system test has achieved favorable experimental results.Figure11 mainly shows the results of multitarget ranging and the shortest weeding path planning results in different scenarios. The results in Figure 11 are highlighted for better results observation. Among them, Figure 11(a) is the result of multitarget ranging and the shortest path planning of a single plant of corn and multiple weeds under 45 degree squint angles. Figure 11(b) is the result of multitarget ranging and the shortest path planning of a single plant of corn and multiple weeds under top-view angles. As a complex and special scenario, Figures 11(c) and 11(d) are the results of multitarget ranging and the shortest path planning of multiple corn and multiple weeds. By further observing the results of Figure 11, we can find that the first column in Figures 11(a)∼11(d) are the original images obtained by the depth camera. The second column in Figures 11(a)∼11(d) are the truthful shortest weeding path planning results images obtained by manual measurement. Staring from the field corn target, each step is marked with the corresponding number in the direction shown by the arrow. In Figures 11(a)∼11(d), the third column in the figure shows the result of multitarget ranging. The upper left corner of this column of the figure shows the result of the depth camera ranging, and the corresponding 2D coordinate points of the target are marked. The experimental effect of our proposed algorithm is the fourth column in Figures 11(a)∼11(d). The extraction results of the shortest weeding path are shown in Figures 11(a)∼11(d) in the fifth column in Figure 11.Figure 11
Multitarget depth ranging and shortest path planning in different scenarios.
(a)(b)(c)(d)On the other hand, take a single plant of corn and multiple weeds as examples in a scenario. Figure12 shows the detailed processing flow. We collected distance data by the manual measurement method in the agricultural experimental field. The distance between the corn target and the weed target as well as the distance between the weed target and the weed target in the corresponding field of view of the depth camera are recorded, respectively. Here, the unit of the distance data is also in meters, and the corresponding distance data statistics between the targets are shown in Figure 3. At the same time, according to the recorded distance data statistics, combined with the shortest path planning algorithm Dijkstra’s ideas, we manually calculate the corresponding shortest weeding path. The specific calculation process is shown in Figure 3. By observing and comparing the extraction results of the shortest weeding path C1⟶W2⟶W4⟶W5⟶W3⟶W6⟶W1 in Figure 12(a) and the result of artificially calculating the corresponding shortest weeding path C1⟶W2⟶W4⟶W5⟶W3⟶W6⟶W1 in Figure 13(b), it is not difficult to find that the shortest weeding path of the two methods can be consistent. Thus, it further illustrates the feasibility and accuracy of our research method.Figure 12
Schematic diagram of the target depth ranging and the shortest path planning. (a) Keyframe image of depth camera; (b) multitarget ranging result image; (c) shortest weeding path planning result image; and (d) the shortest weeding path extracts the rendered image.Figure 13
The results of segmentation using different algorithms. (a) and (b) Optimized results of corn binary image. (c) and (d) Optimized results of weed binary image.
(a)(b)(c)(d)


## 6.1. Results of Data Preprocessing

In data preprocessing, Figure6 shows the result of corn and weed recognition as well as automatic weed removing for the Section 4.1.Figure 6
Schematic diagram of corn and weed recognition as well as automatic weed removing. (a) Recognition results of corn and weed. (b) and (c) Details of corn recognition results. (d) and (e) Details of weed recognition results.Figure7 shows the effect of grayscale image corresponding to green area extraction methods for the Section 4.1. Among them, the first column of Figure 7 is the grayscale image processing results of the EXG method, the second column of Figure 7 is the grayscale image processing results of the GMR method, and the third column of Figure 7 is the grayscale image processing results of the Cg method.Figure 7
Experimental results of grayscale image processing.
(a)(b)(c)(d)


## 6.2. Evaluation of the Extraction Methods for Greenness Detection

To better perform grayscale image processing, we select the EXG method for extracting the green area of the target image and use it to extract the green area of the corn and weed targets in the field crop image at the same time. In this study, the target of maize and weeds after shearing is taken as the research object. The only change of the grayscale generation method is used to generate multiple sets of binary image experimental data. We compared the green area extraction effect of the EXG method and GMR method in the RGB color space as well as the Cg method in the YCrCb color space. The detailed experimental results are shown in Figure8.Figure 8
The effect of extraction methods on the unoptimized OTSU algorithm. (a) and (b) Results of the corn green area extraction image. (c) and (d) Results of the weed green area extraction image.
(a)(b)(c)(d)In this study, we introduced a standard image as a benchmark (Figure8). The traditional OTSU algorithm which only changes the grayscale image generation method generates multiple sets of binary image experimental data for the comparison. Use the three indicators of ratio, variance, and standard deviation to measure the pros and cons of the EXG method, GMR method, and Cg method. Where, the ratio is defined as the ratio of the sum of the number of pixels in the black area to the sum of the number of pixels in the black area in the standard image after the binary image is generated by the unoptimized OTSU algorithm. The equation (16) is the equation for calculating variance. The detailed comparison results are shown in Table 1.(16)σ2=1N×∑i=1NAi−A¯2,where Ai is the i ratio, A¯ is the average value of the ratio, and N is the number of experimental images participating in the comparison.Table 1
The comparison results of the three measurement indicators on the extraction methods.
Color indicesFormulaRatioσ2σEXG2g−r−b1.0430.0010.031GMRg−r0.8770.0250.160Cg0.4g−0.3r−0.1b1.0020.0020.044Combining the results of Figure8 and Table 1, it can be seen that the average ratio of the Cg method in the YCrCb color space is the smallest. Due to its obvious error in the extraction of the green area of the weed target, a large area is classified in a wrong way. And it also results in an average ratio abnormality. The EXG method in the RGB color space can effectively suppress the interference of environmental factors such as soil background, dry grass, and shadow. It is more prominent for the green area of the image of extracting corn and weed targets. It is suitable for processing field crop images under natural conditions, which can obtain an ideal grayscale image effect.


## 6.3. Results of Edge Detection

By comparing the experimental results of edge detection, we can find that the edge detection algorithm based on the Canny operator has a perfect effect [48]. The specific effect is shown in Figure 9.Figure 9
Experimental results of five classical edge detection operators. (a) Corn edge contour extraction image. (b) Weed edge contour extraction image.
(a)(b)


## 6.4. Evaluation of the Quadratic Traversal Algorithm

A quadratic traversal algorithm is proposed for selecting the target 2D coordinate points. By taking a single plant of corn and weeds in the keyframe image as the research goal, a series of image processing and optimization operations are used to obtain the key corn and weed target edge contour image. The quadratic traversal algorithm is mainly based on the target edge contour image as a research basis. Through the combination of the first traversal and the second traversal, it is possible to accurately select the 2D coordinate points’ information of the target on the keyframe image.The detailed process is shown in Figure10. First, we need to cut out the target recognition results of the keyframe image and the data preparation after a series of zooming processes. Second, the EXG method in the RGB color space is used to complete the grayscale processing of the image. For the grayscale image, the IOTSU algorithm can be used to generate and optimize the binary image, and then, the Canny operator’s edge detection algorithm can be used to perform binary image processing to achieve target edge contour extraction. Finally, in the target edge contour image, the first traversal of the quadratic traversal algorithm uses the traversal search box to search the pixel area that meets the set conditions as well as cut and store it locally for the second traversal. In the quadratic traversal algorithm, the second traversal takes the output of the first traversal as the research content, also uses the other traversal search box to search the pixel area that meets the set conditions, and selects the pixel fixed as close as possible to the center position. Besides, according to the pixel point using equation (14), the pixel point can be calculated back to the corresponding position of the target in the keyframe image, so as to realize the research work of selecting 2D coordinate points.Figure 10
Schematic diagram of the quadratic traversal algorithm for selecting the target 2D coordinate points. (a) The process of selecting 2D coordinate points of corn. (b) The process of selecting 2D coordinate points of weed.For the design of the traversal search box in the first traversal method, this study is based on the size of the target edge contour image (500 × 400) processed by the edge detection, and the traversal search box can traverse the target contour edge image as much as possible. Therefore, the traversal search boxes with sizes of 50 × 40, 50 × 100, and 100 × 100 were designed to select the appropriate traversal search box size. We take thirty corn and weed target contour images as the total sample number, and the total number of successful samples and failed samples of each traversal search box were counted, respectively. Table2 presents the performance of various traversal search box sizes. The success rate of the traversal search box with a size of 100 × 100 can reach 90.0%, which can meet the actual requirements of the experiment.Table 2
The success rate of the three traversal search boxes.
Traversal boxExperimental sampleSuccess sampleFailed sampleSuccess rate (%)50 × 403022873.350 × 1003026486.6100 × 1003027390.0


## 6.5. Evaluation of Depth Ranging and Shortest Path Planning

In order to verify the feasibility of our proposed research method, we conducted a systematic test in the agricultural experimental field, called “Nong Cui Yuan,” of Anhui Agricultural University. On the one hand, we started by the keyframe image in the colored image video stream captured in real-time by the depth camera. Through the method proposed in this study, the 2D coordinate points of the corn-weed targets in the field crop image can be accurately selected. Finally, multitarget depth ranging and shortest weeding path planning can be achieved, and the system test has achieved favorable experimental results.Figure11 mainly shows the results of multitarget ranging and the shortest weeding path planning results in different scenarios. The results in Figure 11 are highlighted for better results observation. Among them, Figure 11(a) is the result of multitarget ranging and the shortest path planning of a single plant of corn and multiple weeds under 45 degree squint angles. Figure 11(b) is the result of multitarget ranging and the shortest path planning of a single plant of corn and multiple weeds under top-view angles. As a complex and special scenario, Figures 11(c) and 11(d) are the results of multitarget ranging and the shortest path planning of multiple corn and multiple weeds. By further observing the results of Figure 11, we can find that the first column in Figures 11(a)∼11(d) are the original images obtained by the depth camera. The second column in Figures 11(a)∼11(d) are the truthful shortest weeding path planning results images obtained by manual measurement. Staring from the field corn target, each step is marked with the corresponding number in the direction shown by the arrow. In Figures 11(a)∼11(d), the third column in the figure shows the result of multitarget ranging. The upper left corner of this column of the figure shows the result of the depth camera ranging, and the corresponding 2D coordinate points of the target are marked. The experimental effect of our proposed algorithm is the fourth column in Figures 11(a)∼11(d). The extraction results of the shortest weeding path are shown in Figures 11(a)∼11(d) in the fifth column in Figure 11.Figure 11
Multitarget depth ranging and shortest path planning in different scenarios.
(a)(b)(c)(d)On the other hand, take a single plant of corn and multiple weeds as examples in a scenario. Figure12 shows the detailed processing flow. We collected distance data by the manual measurement method in the agricultural experimental field. The distance between the corn target and the weed target as well as the distance between the weed target and the weed target in the corresponding field of view of the depth camera are recorded, respectively. Here, the unit of the distance data is also in meters, and the corresponding distance data statistics between the targets are shown in Figure 3. At the same time, according to the recorded distance data statistics, combined with the shortest path planning algorithm Dijkstra’s ideas, we manually calculate the corresponding shortest weeding path. The specific calculation process is shown in Figure 3. By observing and comparing the extraction results of the shortest weeding path C1⟶W2⟶W4⟶W5⟶W3⟶W6⟶W1 in Figure 12(a) and the result of artificially calculating the corresponding shortest weeding path C1⟶W2⟶W4⟶W5⟶W3⟶W6⟶W1 in Figure 13(b), it is not difficult to find that the shortest weeding path of the two methods can be consistent. Thus, it further illustrates the feasibility and accuracy of our research method.Figure 12
Schematic diagram of the target depth ranging and the shortest path planning. (a) Keyframe image of depth camera; (b) multitarget ranging result image; (c) shortest weeding path planning result image; and (d) the shortest weeding path extracts the rendered image.Figure 13
The results of segmentation using different algorithms. (a) and (b) Optimized results of corn binary image. (c) and (d) Optimized results of weed binary image.
(a)(b)(c)(d)


## 7. Comparison to Other Works

### 7.1. Performance Comparison of Image Segmentation

On the basis of obtaining an ideal grayscale image, the next research is to perform binary image generation and optimization on the grayscale image. In this study, the PSO algorithm [49, 50], traditional OTSU algorithm [51], and the proposed IOTSU algorithm were used to segment the corn and weed targets in the field crop image, respectively, to generate a binary image. Figure 9 shows the original image of the corn and weed target as an example to study the image segmentation effect of the PSO algorithm, traditional OTSU algorithm, and IOTSU algorithm. It is obvious that the segmentation effect of the IOTSU algorithm is better than the traditional OTSU algorithm and the PSO algorithm which loses a lot of important information.In order to further evaluate the performance of the segmentation algorithm, the three aspects of segmentation threshold, running time, and diversity rate are evaluated, respectively, where the diversity rate can be calculated by equation (17). Table 3 lists the performance of the three segmentation algorithms, where the segmentation threshold and running time are average values.(17)DR=PT−OTOT×100%,where PT is the average segmentation threshold of the segmentation algorithm, and OT represents the average segmentation threshold of the reference algorithm OTSU.Table 3
Comparison of performance parameters using different segmentation algorithms.
AlgorithmOriginal image (pixels × pixels)Cut image (pixels × pixels)Threshold valueRunning time (s)Diversity rate (%)PSO640 × 480500 × 400130.30.15416.1OTSU640 × 480500 × 400112.30.5070IOTSU640 × 480500 × 400112.30.2250As shown in Table3, although the PSO algorithm can meet the needs of real-time processing, compared with the traditional OTSU algorithm and the IOTSU algorithm, it has a faster processing speed, but its segmentation performance is poor and the difference rate is large. Moreover, it is unable to find the best segmentation threshold. Although the traditional OTSU algorithm can find the optimal segmentation threshold, the high cost of time cannot meet the real-time requirements because it needs to traverse the entire grayscale space to find the optimal threshold. Compared to the traditional OTSU algorithm, the IOTSU algorithm can ensure the segmentation performance and can be processed at a faster speed under the condition that the segmentation threshold is consistent. At the same time, the processing time of the PSO algorithm is close, so it can meet the needs of real-time image processing.


### 7.2. Performance Comparison of Path Planning

In order to further evaluate the performance of the quadratic traversal algorithm, the two aspects of running time and diversity rate are evaluated, respectively, where the diversity rate can be calculated by equation (18). Table 4 lists the performance of the three different algorithms.(18)DR=Ed−MdMd×100%,where Ed is the sum of the algorithm’s estimated distances of multiple weeds, and Md is the sum of the true values of the multitarget weeds distance manually measured.Table 4
Comparison of performance parameters using different algorithms.
AlgorithmsNumber of targetsRunning time (s)Diversity rate (%)Genetic algorithm [52]74.2018.3Path planning algorithm [28]78.6332.4Quadratic traversal algorithm79.103.90As shown in Table4, although the genetic algorithm has a faster processing speed, compared with the path planning algorithm and the quadratic traversal algorithm, it must be processed when the position of the target is known. Therefore, the genetic algorithm cannot meet the real-time processing requirements of agricultural weeding robots. At the same time, we can find that the running time of the path planning algorithm and quadratic traversal algorithm is close. In order to further illustrate the superiority of our proposed quadratic traversal algorithm, we compared the diversity rate of these three algorithms on the premise of the same number of targets. It is not difficult to find that our proposed method has a lower diversity rate than the genetic algorithm and path planning algorithm. Therefore, the various performances of our proposed method can basically meet the real-time processing requirements of agricultural weeding robots.


## 7.1. Performance Comparison of Image Segmentation

On the basis of obtaining an ideal grayscale image, the next research is to perform binary image generation and optimization on the grayscale image. In this study, the PSO algorithm [49, 50], traditional OTSU algorithm [51], and the proposed IOTSU algorithm were used to segment the corn and weed targets in the field crop image, respectively, to generate a binary image. Figure 9 shows the original image of the corn and weed target as an example to study the image segmentation effect of the PSO algorithm, traditional OTSU algorithm, and IOTSU algorithm. It is obvious that the segmentation effect of the IOTSU algorithm is better than the traditional OTSU algorithm and the PSO algorithm which loses a lot of important information.In order to further evaluate the performance of the segmentation algorithm, the three aspects of segmentation threshold, running time, and diversity rate are evaluated, respectively, where the diversity rate can be calculated by equation (17). Table 3 lists the performance of the three segmentation algorithms, where the segmentation threshold and running time are average values.(17)DR=PT−OTOT×100%,where PT is the average segmentation threshold of the segmentation algorithm, and OT represents the average segmentation threshold of the reference algorithm OTSU.Table 3
Comparison of performance parameters using different segmentation algorithms.
AlgorithmOriginal image (pixels × pixels)Cut image (pixels × pixels)Threshold valueRunning time (s)Diversity rate (%)PSO640 × 480500 × 400130.30.15416.1OTSU640 × 480500 × 400112.30.5070IOTSU640 × 480500 × 400112.30.2250As shown in Table3, although the PSO algorithm can meet the needs of real-time processing, compared with the traditional OTSU algorithm and the IOTSU algorithm, it has a faster processing speed, but its segmentation performance is poor and the difference rate is large. Moreover, it is unable to find the best segmentation threshold. Although the traditional OTSU algorithm can find the optimal segmentation threshold, the high cost of time cannot meet the real-time requirements because it needs to traverse the entire grayscale space to find the optimal threshold. Compared to the traditional OTSU algorithm, the IOTSU algorithm can ensure the segmentation performance and can be processed at a faster speed under the condition that the segmentation threshold is consistent. At the same time, the processing time of the PSO algorithm is close, so it can meet the needs of real-time image processing.


## 7.2. Performance Comparison of Path Planning

In order to further evaluate the performance of the quadratic traversal algorithm, the two aspects of running time and diversity rate are evaluated, respectively, where the diversity rate can be calculated by equation (18). Table 4 lists the performance of the three different algorithms.(18)DR=Ed−MdMd×100%,where Ed is the sum of the algorithm’s estimated distances of multiple weeds, and Md is the sum of the true values of the multitarget weeds distance manually measured.Table 4
Comparison of performance parameters using different algorithms.
AlgorithmsNumber of targetsRunning time (s)Diversity rate (%)Genetic algorithm [52]74.2018.3Path planning algorithm [28]78.6332.4Quadratic traversal algorithm79.103.90As shown in Table4, although the genetic algorithm has a faster processing speed, compared with the path planning algorithm and the quadratic traversal algorithm, it must be processed when the position of the target is known. Therefore, the genetic algorithm cannot meet the real-time processing requirements of agricultural weeding robots. At the same time, we can find that the running time of the path planning algorithm and quadratic traversal algorithm is close. In order to further illustrate the superiority of our proposed quadratic traversal algorithm, we compared the diversity rate of these three algorithms on the premise of the same number of targets. It is not difficult to find that our proposed method has a lower diversity rate than the genetic algorithm and path planning algorithm. Therefore, the various performances of our proposed method can basically meet the real-time processing requirements of agricultural weeding robots.


## 8. Conclusions

In this study, the task of weed one-time removal operations in cornfields is studied. Major innovations and contributions of this study are as follows:(1)
The Faster R-CNN deep network model based on the VGG-16 feature extraction network is used to realize real-time target recognition and complete automatic cut classification of targets. By returning the predicted parameter information of the border regression and the color of the prediction border, the target category in the image can be accurately determined. And we realized the data connection between deep learning and traditional algorithms.(2)
Using our improved OTSU algorithm can achieve the generation and optimization of binary images. Compared with the traditional OTSU algorithm, the algorithm compresses the range of the search grayscale interval. The search speed of the algorithm is effectively improved, and the proposed path planning calculation is time efficient. It meets the demand of the real-time data processing requirements, which allows that our method can be further applied to the mobile agricultural weeding robot in the field.(3)
A quadratic traversal algorithm is proposed for selecting the target 2D coordinate point. The corresponding traversal search box is designed; the search success rate of the traversal search box with a size of 100 × 100 can reach 90.0% of the testing date effect. By transforming the target 2D coordinate point in the pixel coordinate system into the 3D coordinate point in the camera coordinate system, the used depth camera can achieve the multitarget depth ranging and the shortest weeding path planning as well as avoid using complex point cloud information. This way effectively saves computing resources and avoids redundant information.(4)
The application and implementation of our methods in this research can assist intelligent weeding robots to carry out precise weeding operations and improve the efficiency of them. Meantime, it has an important practical significance for promoting the application of intelligent weeding robots to the filed.


## 9. Future Works

Our future works includes two aspects: (1) quantitative analysis of power consumption for the robot. (2) Consider the influence of outdoor dynamic environmental factors.(1)
Our optimized weeding path planning has the potential to reduce power consumption for the robot. In order to prove this qualitative conclusion, we will include a quantitative analysis and carry out quantitative analysis experiments in our future works.(2)
Because the depth camera will be affected by different environmental factors in the outdoor, we will design corresponding experiments based on various environmental conditions in our future works. At the same time, we will fuse multiple sensors to develop more robust algorithms and overcome the weakness of the current used depth camera.



---

*Source: 6633139-2021-02-20.xml*