# A Visual Recognition and Path Planning Method for Intelligent Fruit-Picking Robots

Authors: Hongli Li
Journal: Scientific Programming
 (2022)

---

## Abstract

With the rapid development of economy and the increasing improvement of agricultural production level, people’s demand for fruits is also increasing year by year. China is the largest fruit production and consumption country in the world. According to relevant statistics reported for China, by the end of 2019, the total amount of various fruits sold had reached about 270 million tons, with apples accounting for 48% of the global output and pears accounting for 69% of the national total output. However, China’s fruit picking is still dominated by manual picking process, which takes a lot of manpower and time to complete, therefore resulting in low fruit picking efficiency. Some farmers are unable to complete fruit picking in a short time, resulting in a large number of fruits unable to be listed, resulting in huge losses. To solve this problem, this paper focuses on the visual recognition and path planning for intelligent fruit-picking robot. Using robot to complete fruit picking is the best way at present. This paper establishes a picking robot recognition and positioning system based on stereo vision, which is used to identify and locate the fruits planted in the orchard area. The coordinate error of the target point of the intelligent fruit-picking robot coordinate system is less than 10 mm, which has high accuracy. Then, the path of intelligent fruit-picking robot is planned based on visual feedback algorithm and biological stimulation neural network. Our empirical evaluations suggest that the proposed robot walks in the planting park in the shape of “zigzag” and realizes full-coverage path planning after 6 turns. The results show the efficiency of the intelligent method.


---

## Body

## 1. Introduction

The most important type of agricultural robot is the picking robot, which has great development potentials. However, there is no accurate definition of fruit and vegetable harvesting robot. Generally, the picking robot only completes the picking of vegetables and fruits and realizes the basic work of crop picking, packaging, and transportation by programming. It has strong automatic mechanical and perception abilities. It is an interdisciplinary frontier discipline, which involves electronics, machinery, information, agriculture, computer science, biology, and intelligent technology including artificial intelligence and machine learning techniques. At the same time, we should master the knowledge of mechanical structure, sensors, visual image technology, and other related fields in the context of decision making and implementation.At present, agricultural production is also developing towards diversification, scale, and accuracy, and the cost of agricultural labor continues to increase, resulting in labor shortage and increased costs. Robot technology has become the main way to solve agricultural problems, in particular, when picking fruits for packaging and exporting. Through improving the automation of orchard fruits and vegetables, the automation of farmers’ orchard planting and picking can be improved. At the same time, it can accurately identify and plan the picking path and greatly improve the efficiency of fruit and vegetable picking, which is of great value to promote the development of agricultural services and their modernization (i.e., intelligent agriculture).The innovations of this paper in studying the visual recognition and path planning for intelligent fruit-picking robot are as follows: (1) establish a picking robot recognition system based on stereo vision, which uses a binocular stereo vision technology to identify and locate the fruit in the orchard. Furthermore, the method subsequently uses a fuzzy-based, two-dimensional entropy algorithm to identify the obstacles in the orchard, which can accurately identify and locate the specific positions of the fruit in the orchard—and it is a key part of the fruit picking. (2) Because the characteristics of the orchard plant planting are parallel rows and columns, therefore a biological stimulation neural network method is used to plan the optimal path of the global coverage of the intelligent fruit picking robot. The robot takes the shape of “zigzag” when picking fruits so that the picking robot can plan the optimal safe path in a short time and shorten the fruits’ picking time. The following are the major contributions of this research.(i)
Establish a picking robot recognition and positioning system based on stereo vision.(ii)
Use fuzzy two-dimensional entropy algorithm to identify the obstacles in the orchard.(iii)
The biological stimulation neural network method is used to plan the optimal path of the global coverage of the intelligent fruit-picking robot.The rest of the paper is organized as follows. In Section2, we offer an overview of the related work. Section 3 is about the recognition and positioning system of picking robot based on stereo vision. In Section 4, trajectory planning of picking robot based on visual feedback is discussed. Section 5 illustrates the results of visual recognition and path planning of intelligent fruit picking robot. Finally, Section 6 concludes this paper along with future work.


## 2. Related Work

In the 1960s, Zhang and Chaisattapagon proposed to use robots for the purpose of picking fruits. Since then, fruit-picking robots have become increasingly mature with the development of decades [1]. It takes tens of seconds to pick successfully at first, but now it can be completed within 10 seconds [2]. The way of picking fruits is to separate fruit trees from fruits by mechanical vibration. This way has high picking efficiency. However, it will damage the fruits and it is difficult to ensure the quality of fruits [3]. In recent years, computer technology and electronic information technology have developed rapidly, especially artificial intelligence, machine learning, and image processing. Fruit picking robots are more intelligent [4]. Barnett et al. developed a fruit-harvesting auxiliary equipment based on Orsi ecological picking mobile platform [5]. This equipment is widely used in small- and medium-sized orchards and can assist in picking all kinds of fruits. It cannot be completed automatically in actual operation. Special personnel must lift the platform and mobile equipment. Moreover, similar devices should be operated by a computer application that has more control over the fruit-picking task and can guide them more intelligently through machine learning methods.Wang et al. developed a shaking apple-picking robot [6], which surrounds the whole fruit tree from both sides and then vibrates the trunk area of the fruit tree with high frequency. The fruit falls from the fruit tree to the V-shaped collection device, and then the collection device is closed to complete the fruit picking. Chen et al. developed a negative-pressure clamping apple-picking robot [7]. The main body of the picking machine on this robot is a six-degree-of-freedom industrial robot, which can move the arm freely in a large area and install a micro camera at the end of the robot arm to collect the fruit position [8]. Feng et al. have developed a conceptual prototype of an apple-picking robot with a degree of freedom of 6 to automatically pick fruit in the orchard. The average cycle of picking fruit is 5–9 s [9]. The research time of picking robots in China is relatively short, and now the vast majority of picking robots cannot be applied in the market. Su et al. established the mobile system of apple-picking robot based on Voyager IIA and ARM11 + Windows CE and explored the visual obstacle detection technology in Apple Park. Furthermore, the proposed approach puts forward the path planning method of mobile robot based on biological stimulation neural network and uses a BP neural network to complete the navigation control algorithm. However, this technology cannot run in an embedded computer system [10, 11].Li et al. deeply investigated and studied the forward kinematics and inverse kinematics of the mechanical arm and suggested an approach that completes the trajectory interpolation algorithm of the picking arm, which uses LQR for zoning control according to the basic constraints of trajectory planning, and completes the fuzzy trajectory tracking control. Based on the HOO stability control algorithm, Li et al. analyzed the control mode of the stable picking arm so as to effectively suppress the “shaking” problem of the apple-picking robot arm. However, the problem of arm motion planning and obstacle avoidance has not been deeply explored in the existing literature [12, 13]. Furthermore, machine learning-based methods are also not investigated in terms of fruit picking robots (location and position) and their accuracies.


## 3. Recognition and Positioning System of Picking Robot Based on Stereo Vision

### 3.1. Building an Independent Picking System

This paper constructs an intelligent fruit picking robot system, which consists of vision system [14], industrial robot, computer, and end effector. The vision system carried on the robot adopts the stereo vision product Bumblee2 manufactured by PGR company. The baseline length is 120 mm. The left and right lenses are perpendicular to the baseline and parallel to the optical axis. The lens adopts 1/3-inch Sony icx204 color CCD lens, which can be automatically synchronized, with a focal length of 3.8 mm. It connects the computer and IEEE-1394 to transmit data. It can search and locate the picking target in a certain area and is applied in the visual navigation of the rear mobile fruit picking robot. The end effector is manufactured by German Schunk company, and the clamping force is guaranteed to be in a constant state. Finally, the system forms an open-loop visual control system. The control quantity of the robot controller is calculated based on the stereo vision information. It runs outside the control cycle, and the calculation times are only once, which is shown in Figure 1.Figure 1
Open-loop visual control system.


### 3.2. Identification and Location of Fruits

During fruit-picking process, the end effector of the intelligent fruit-picking robot determines the working position according to the fruits’ target. Nowadays, various common methods of fruit picking are pure air suction picking, finger grasping picking, combined air suction, finger picking, and so on. [15–17]. In this paper, the electric gripper is selected to clamp the fruit, and then the fruit is picked by twisting the fruit handle. Therefore, the fruit center of gravity of the band actuator should be perpendicular to the working center of the gravity so that the fruit could be positioned according to the fruit center of gravity [18, 19].The reliability and accuracy of fruit positioning in binocular stereo vision technology are disturbed by various factors, including various uncontrollable factors. If the real shape and ambient light lead to random distributions of invalid matching points during stereo vision matching, then it is difficult to obtain depth information. This can potentially be resulting in failure to complete fruit picking. In this paper, the depth of difficult matching points is set to 0.385 m through the stereo vision module [20]. Furthermore, approximately 70% of the fruits are spherical, which can avoid the above problems of stereo vision technology. The three-dimensional position information of different points on the target is used to construct the fruit ball model, and then the position of the fruit center of gravity is calculated by the least square method. The following is the calculation formula of the fruit ball model:(1)x−a2+y−b2+z−c2=R2.If the coordinates of the spherical center in the camera coordinate system are (a, b, c), formula (1) can be converted to the following formula:(2)2ax+2by+2cz+R2−a2+b2+c2=x2+y2+z2.If there aren points, then they are expressed in the matrix form. The above formula can be illustrated as follows:(3)2x12x2...2xn2y12y2...2yn2z12z2...2zn11...1abcR2−a2+b2+c2=x12+y12+z12x22+y22+z22...xn2+yn2+zn2T.The above formula is simplified as(4)Am=B.m is calculated by the least-squares method:(5)m=ATA−1ATB.Through the above calculation, the spherical center coordinates are obtained.In this paper, eight different points on the target are selected to calculate the target position, and the shape of the fruit is described as a circle. Based on Hough transform, the center coordinates (i, j) and radius R of different fruits on the image plane can be obtained. This method can accurately identify the center of the fruit and simply obtain the circumscribed rectangle of the outer circle of the fruit. The coordinates of the upper left point and the lower right point of the rectangle are (xmin, ymin), (xmax, ymax), respectively. Finally, the (x, y) points in the rectangular area are randomly selected. The entire process of fruit recognition program is shown in Figure 2.Figure 2
Flowchart of fruit identification and positioning.


### 3.3. Obstacle Recognition Based on Fuzzy Two-Dimensional Entropy

Different illumination and imaging resolution will interfere with the accuracy of target detection, resulting in the integration of pixels such as image boundary, edge, and region, showing an intermediate transition phenomenon, so the image has strong fuzziness in the natural environment. Image processing itself has the disadvantage of uncertain information. At the same time, there are many fuzzy information after image processing. Therefore, the fuzzy processing algorithm can be selected when dealing with the problem of identifying obstacles in the natural environment.Assume that the size of a gray image isM × N. The gray value of (m, n) pixel coordinates is represented by Xmn, L is the gray level series, and a is any fuzzy set A, μA (Xmn) is the fuzzy membership function of set a, that is, the degree of fuzzy set a to which the gray value of Xmn pixels on the image belongs.The background or target on the image is represented by setA, which is used to segment the gray image threshold; that is, select a threshold to divide the image into two parts: target and background image. Assuming A1 as the target and A2 as the background, the following formula represents the image blur line H:(6)H=−PA1logPA1−PA2logPA2.The above formulaP(A1) represents the probability of occurrence of A1 fuzzy event, P(A2) represents the probability of occurrence of A2 fuzzy event, and P(A1) + P(A2) = 1. If P(A1) + P(A2) = 0.5, the value of di h is the highest; that is, di h is the highest at the same time.(7)PA1=1MN∑m=1M∑n=1NμA1xmn,PA2=1MN∑m=1M∑n=1NμA2xmn.Upper formulaμA1(xmn) represents the fuzzy membership function of the detection target, μA2(xmn) is the background fuzzy membership function, and μA1(xmn) + μA2(xmn) = 1.


## 3.1. Building an Independent Picking System

This paper constructs an intelligent fruit picking robot system, which consists of vision system [14], industrial robot, computer, and end effector. The vision system carried on the robot adopts the stereo vision product Bumblee2 manufactured by PGR company. The baseline length is 120 mm. The left and right lenses are perpendicular to the baseline and parallel to the optical axis. The lens adopts 1/3-inch Sony icx204 color CCD lens, which can be automatically synchronized, with a focal length of 3.8 mm. It connects the computer and IEEE-1394 to transmit data. It can search and locate the picking target in a certain area and is applied in the visual navigation of the rear mobile fruit picking robot. The end effector is manufactured by German Schunk company, and the clamping force is guaranteed to be in a constant state. Finally, the system forms an open-loop visual control system. The control quantity of the robot controller is calculated based on the stereo vision information. It runs outside the control cycle, and the calculation times are only once, which is shown in Figure 1.Figure 1
Open-loop visual control system.


## 3.2. Identification and Location of Fruits

During fruit-picking process, the end effector of the intelligent fruit-picking robot determines the working position according to the fruits’ target. Nowadays, various common methods of fruit picking are pure air suction picking, finger grasping picking, combined air suction, finger picking, and so on. [15–17]. In this paper, the electric gripper is selected to clamp the fruit, and then the fruit is picked by twisting the fruit handle. Therefore, the fruit center of gravity of the band actuator should be perpendicular to the working center of the gravity so that the fruit could be positioned according to the fruit center of gravity [18, 19].The reliability and accuracy of fruit positioning in binocular stereo vision technology are disturbed by various factors, including various uncontrollable factors. If the real shape and ambient light lead to random distributions of invalid matching points during stereo vision matching, then it is difficult to obtain depth information. This can potentially be resulting in failure to complete fruit picking. In this paper, the depth of difficult matching points is set to 0.385 m through the stereo vision module [20]. Furthermore, approximately 70% of the fruits are spherical, which can avoid the above problems of stereo vision technology. The three-dimensional position information of different points on the target is used to construct the fruit ball model, and then the position of the fruit center of gravity is calculated by the least square method. The following is the calculation formula of the fruit ball model:(1)x−a2+y−b2+z−c2=R2.If the coordinates of the spherical center in the camera coordinate system are (a, b, c), formula (1) can be converted to the following formula:(2)2ax+2by+2cz+R2−a2+b2+c2=x2+y2+z2.If there aren points, then they are expressed in the matrix form. The above formula can be illustrated as follows:(3)2x12x2...2xn2y12y2...2yn2z12z2...2zn11...1abcR2−a2+b2+c2=x12+y12+z12x22+y22+z22...xn2+yn2+zn2T.The above formula is simplified as(4)Am=B.m is calculated by the least-squares method:(5)m=ATA−1ATB.Through the above calculation, the spherical center coordinates are obtained.In this paper, eight different points on the target are selected to calculate the target position, and the shape of the fruit is described as a circle. Based on Hough transform, the center coordinates (i, j) and radius R of different fruits on the image plane can be obtained. This method can accurately identify the center of the fruit and simply obtain the circumscribed rectangle of the outer circle of the fruit. The coordinates of the upper left point and the lower right point of the rectangle are (xmin, ymin), (xmax, ymax), respectively. Finally, the (x, y) points in the rectangular area are randomly selected. The entire process of fruit recognition program is shown in Figure 2.Figure 2
Flowchart of fruit identification and positioning.


## 3.3. Obstacle Recognition Based on Fuzzy Two-Dimensional Entropy

Different illumination and imaging resolution will interfere with the accuracy of target detection, resulting in the integration of pixels such as image boundary, edge, and region, showing an intermediate transition phenomenon, so the image has strong fuzziness in the natural environment. Image processing itself has the disadvantage of uncertain information. At the same time, there are many fuzzy information after image processing. Therefore, the fuzzy processing algorithm can be selected when dealing with the problem of identifying obstacles in the natural environment.Assume that the size of a gray image isM × N. The gray value of (m, n) pixel coordinates is represented by Xmn, L is the gray level series, and a is any fuzzy set A, μA (Xmn) is the fuzzy membership function of set a, that is, the degree of fuzzy set a to which the gray value of Xmn pixels on the image belongs.The background or target on the image is represented by setA, which is used to segment the gray image threshold; that is, select a threshold to divide the image into two parts: target and background image. Assuming A1 as the target and A2 as the background, the following formula represents the image blur line H:(6)H=−PA1logPA1−PA2logPA2.The above formulaP(A1) represents the probability of occurrence of A1 fuzzy event, P(A2) represents the probability of occurrence of A2 fuzzy event, and P(A1) + P(A2) = 1. If P(A1) + P(A2) = 0.5, the value of di h is the highest; that is, di h is the highest at the same time.(7)PA1=1MN∑m=1M∑n=1NμA1xmn,PA2=1MN∑m=1M∑n=1NμA2xmn.Upper formulaμA1(xmn) represents the fuzzy membership function of the detection target, μA2(xmn) is the background fuzzy membership function, and μA1(xmn) + μA2(xmn) = 1.


## 4. Trajectory Planning of Picking Robot Based on Visual Feedback

### 4.1. Global Path of Picking Robot

Suppose the end effector of the fruit robot wants to pickn fruits, and its coordinates can be expressed as {g1, g2, …, gn}, where g1 represents the initial departure and returns to gi after picking. It is assumed that the distance between any two points including the starting point and the target point is represented by d(gi, gj). Then, the picking path planning problem is described as follows: Let x = {1, 2, …, n} be a subset of search integers, where X is the corresponding number of n fruits:(8)TdC=dgc1,gc2+∑n=2n−1dgci,gci+1+dgcn,gcj.In this paper, genetic algorithm is used to calculate the minimum value.


### 4.2. Mathematical Model of Genetic Algorithm

Genetic algorithm (GA) refers to taking the solution set of a problem as a population and improving the quality of the solution through genetic operations such as crossover, selection, and mutation. American scholars first proposed genetic algorithm, which has the advantages of strong universality, simple calculation process, and strong robustness. It can quickly deal with nonlinear and complex problems that traditional search algorithms cannot deal with. The other two characteristics of this genetic algorithm are global search and parallelism. That is, the following is the calculation expression of genetic algorithm:(9)SGA=C,E,P0,M,Φ,Γ,T.In the above formula,C represents the individual coding method, E denotes the individual fitness evaluation function, and P0 represents the initial population. Similarly, M represents the population size, Φ denotes the selection operator, Γ stands for the crossover operator, and T represents the termination condition of the genetic operation. The basic flowchart of the genetic algorithm is shown in Figure 3.Figure 3
Flowchart of genetic algorithm.


### 4.3. Fruit-Picking Sequence Planning Based on Genetic Algorithm

This paper uses binocular stereo vision to recognize 9 different positions of fruits on the tree, and the positions are transformed into robotic positions, which are shown in Table1 below (in terms of x-, y-, and z-axes). This should be noted that in the number column; 1 represents the initial position of the robot, and the remaining nine values are numbered as fruit picking positions [21]. Furthermore, the value of M is 20, the value of qc is 0.9, and the value of qm is 0.1. We can substitute these predefined parameters and values into the above formula in order to calculate the best path, as given by C = {1, 8, 6, 10, 4, 9, 3, 5, 2, 7, 1}.Table 1
Fruit coordinate position.
NumberX (mm)Y (mm)Z (mm)1723.121.94272.852945.28565.71246.3231218.37821.76284.6441114.02663.27525.1651176.55848.75289.756849.22656.74536.4971113.12469.32199.248945.28533.02525.4691164.08545.37480.02101162.27702.64626.93Figure4 shows the picking path planning of intelligent fruit-picking robot; l is its shortest path, and the length is 2846 mm.Figure 4
Optimal picking sequence diagram of fruit intelligent picking robot.The basic picking path of the fruit intelligent picking robot is to arrange the coordinate origin of the fruit robot first, then pick the fruit from far to near, and then return to the initial position. The picking path based on this method is given byC' = {1, 2, 6, 8, 7, 9, 4, 10, 3, 1}. After calculation, the path is computed as given by L∗ = 3202.5 mm. Therefore, L is greater than L∗, which shows that the genetic algorithm has certain significance for the path planning of the fruit robot. Moreover, this illustrates that it can improve the operation efficiency of the fruit picking robot, and the amount of work is also lower than that of other methods.


### 4.4. Path Planning Based on Biological Stimulation Neural Network

#### 4.4.1. Neural Network Dynamic Model of Biological Stimulation

Based on the biological stimulation neural network, the connection relationship between neurons and external inputs in the dynamic environment is defined, and the targets and obstacles corresponding to the lowest end are distributed at the top of the neural activity state diagram. The neuron activities that are easy to be constrained by kinematics are used to spread so as to attract the robot in the global space. Obstacles in the neural activity state diagram can only be partially avoided by the robot [22]. The dynamic neuron activity state diagram based on neural network, combined with the incomplete kinematic constraints of the robot, can reasonably plan the robot path for autonomous obstacle avoidance [23].In 1952, Huxley and Hodgkin first used circuit elements to build a small dendritic cell membrane calculation model based on biological nervous system. The following is the expression of dynamic voltageVm passing through the cell membrane:(10)CmdVmdt=−Ep+Vmgp+ENa+VmgNa−Ek+Vmgk.In the above formula,Cm represents the cell membrane capacitance, Ek represents the potassium ion saturation potential on the cell membrane, ENa represents the sodium ion saturation potential in the cell membrane, and Ep represents the neutral leakage current saturation potential; gk represents potassium ion conductivity, gNa represents sodium ion conductivity, and gp represents neutral channel conductivity.


#### 4.4.2. Motion Control

The intelligent fruit-picking robot should complete the steering control and obstacle avoidance control when picking fruit in the planting park. Based on the analysis and derivation of error mechanics, the path planning control of the rear driving wheel of the robot is obtained. It is assumed that the reference speed of the robot isVr and the angular speed is determined by ωr which both robot speeds are calculated by the following formulas [24]:(11)v=c1eD+vrcoseθ,ω=ωr+c2vreL+c3vdsineθ.In the above equations,C1, C2, and C3 represent positive gain parameters, eD represents driving direction, eL represents new direction, and eθ indicates steering angle and direction error.


#### 4.4.3. Path Planning

Based on the biological stimulation neural network method, the intelligent fruit-picking robot needs to walk in the botanical garden with complex environment under the condition of no control. Here, it is necessary to use biological stimulation neural network in the static plantation, and the number of static obstacles in each row of the park is less. The walking route of the intelligent fruit picking robot is “Zhi,” and it traverses each row of plants planted until it stops at the position of the picking robot. The number of discrete topological tissue neurons in the biological stimulation neural network architecture is 32 × 32, assuming that the upper boundsB and D of external stimulation are both 1, ωij parameters in the latest connection weight μ. The value of μ is 0.7, the value of r0 is 2, the calculated value of C is 0.35, and the value of neuron external input parameter E is 100. In Figure 5 below, two groups of obstacles with different sizes are selected, and the intelligent fruit-picking robot moves upward from the initial position of point P(1, 1). The external input Ii of obstacle neurons in static environment is represented by E, and the values of other neurons are 0. If the intelligent fruit picking robot passes through any point, and the reset result of the external input of the neuron at this point is 0, it indicates that the picking work in the cost area has been completed.Figure 5
Full-coverage path planning.In Figure5 above, the intelligent fruit picking robot starts picking from the bottom to the top and then walks from the top to the bottom in a zigzag shape between planting lines. The steering angle values of the intelligent fruit picking robot are listed in Table 2. The results in the table show that the intelligent fruit robot has to undergo six turns after encountering two groups of obstacles during the picking period and to realize the full-coverage path planning [25].Table 2
Robot motion angle.
The difference between thex-coordinatesThe difference between the coordinatesθ0−150111−14103112


## 4.1. Global Path of Picking Robot

Suppose the end effector of the fruit robot wants to pickn fruits, and its coordinates can be expressed as {g1, g2, …, gn}, where g1 represents the initial departure and returns to gi after picking. It is assumed that the distance between any two points including the starting point and the target point is represented by d(gi, gj). Then, the picking path planning problem is described as follows: Let x = {1, 2, …, n} be a subset of search integers, where X is the corresponding number of n fruits:(8)TdC=dgc1,gc2+∑n=2n−1dgci,gci+1+dgcn,gcj.In this paper, genetic algorithm is used to calculate the minimum value.


## 4.2. Mathematical Model of Genetic Algorithm

Genetic algorithm (GA) refers to taking the solution set of a problem as a population and improving the quality of the solution through genetic operations such as crossover, selection, and mutation. American scholars first proposed genetic algorithm, which has the advantages of strong universality, simple calculation process, and strong robustness. It can quickly deal with nonlinear and complex problems that traditional search algorithms cannot deal with. The other two characteristics of this genetic algorithm are global search and parallelism. That is, the following is the calculation expression of genetic algorithm:(9)SGA=C,E,P0,M,Φ,Γ,T.In the above formula,C represents the individual coding method, E denotes the individual fitness evaluation function, and P0 represents the initial population. Similarly, M represents the population size, Φ denotes the selection operator, Γ stands for the crossover operator, and T represents the termination condition of the genetic operation. The basic flowchart of the genetic algorithm is shown in Figure 3.Figure 3
Flowchart of genetic algorithm.


## 4.3. Fruit-Picking Sequence Planning Based on Genetic Algorithm

This paper uses binocular stereo vision to recognize 9 different positions of fruits on the tree, and the positions are transformed into robotic positions, which are shown in Table1 below (in terms of x-, y-, and z-axes). This should be noted that in the number column; 1 represents the initial position of the robot, and the remaining nine values are numbered as fruit picking positions [21]. Furthermore, the value of M is 20, the value of qc is 0.9, and the value of qm is 0.1. We can substitute these predefined parameters and values into the above formula in order to calculate the best path, as given by C = {1, 8, 6, 10, 4, 9, 3, 5, 2, 7, 1}.Table 1
Fruit coordinate position.
NumberX (mm)Y (mm)Z (mm)1723.121.94272.852945.28565.71246.3231218.37821.76284.6441114.02663.27525.1651176.55848.75289.756849.22656.74536.4971113.12469.32199.248945.28533.02525.4691164.08545.37480.02101162.27702.64626.93Figure4 shows the picking path planning of intelligent fruit-picking robot; l is its shortest path, and the length is 2846 mm.Figure 4
Optimal picking sequence diagram of fruit intelligent picking robot.The basic picking path of the fruit intelligent picking robot is to arrange the coordinate origin of the fruit robot first, then pick the fruit from far to near, and then return to the initial position. The picking path based on this method is given byC' = {1, 2, 6, 8, 7, 9, 4, 10, 3, 1}. After calculation, the path is computed as given by L∗ = 3202.5 mm. Therefore, L is greater than L∗, which shows that the genetic algorithm has certain significance for the path planning of the fruit robot. Moreover, this illustrates that it can improve the operation efficiency of the fruit picking robot, and the amount of work is also lower than that of other methods.


## 4.4. Path Planning Based on Biological Stimulation Neural Network

### 4.4.1. Neural Network Dynamic Model of Biological Stimulation

Based on the biological stimulation neural network, the connection relationship between neurons and external inputs in the dynamic environment is defined, and the targets and obstacles corresponding to the lowest end are distributed at the top of the neural activity state diagram. The neuron activities that are easy to be constrained by kinematics are used to spread so as to attract the robot in the global space. Obstacles in the neural activity state diagram can only be partially avoided by the robot [22]. The dynamic neuron activity state diagram based on neural network, combined with the incomplete kinematic constraints of the robot, can reasonably plan the robot path for autonomous obstacle avoidance [23].In 1952, Huxley and Hodgkin first used circuit elements to build a small dendritic cell membrane calculation model based on biological nervous system. The following is the expression of dynamic voltageVm passing through the cell membrane:(10)CmdVmdt=−Ep+Vmgp+ENa+VmgNa−Ek+Vmgk.In the above formula,Cm represents the cell membrane capacitance, Ek represents the potassium ion saturation potential on the cell membrane, ENa represents the sodium ion saturation potential in the cell membrane, and Ep represents the neutral leakage current saturation potential; gk represents potassium ion conductivity, gNa represents sodium ion conductivity, and gp represents neutral channel conductivity.


### 4.4.2. Motion Control

The intelligent fruit-picking robot should complete the steering control and obstacle avoidance control when picking fruit in the planting park. Based on the analysis and derivation of error mechanics, the path planning control of the rear driving wheel of the robot is obtained. It is assumed that the reference speed of the robot isVr and the angular speed is determined by ωr which both robot speeds are calculated by the following formulas [24]:(11)v=c1eD+vrcoseθ,ω=ωr+c2vreL+c3vdsineθ.In the above equations,C1, C2, and C3 represent positive gain parameters, eD represents driving direction, eL represents new direction, and eθ indicates steering angle and direction error.


### 4.4.3. Path Planning

Based on the biological stimulation neural network method, the intelligent fruit-picking robot needs to walk in the botanical garden with complex environment under the condition of no control. Here, it is necessary to use biological stimulation neural network in the static plantation, and the number of static obstacles in each row of the park is less. The walking route of the intelligent fruit picking robot is “Zhi,” and it traverses each row of plants planted until it stops at the position of the picking robot. The number of discrete topological tissue neurons in the biological stimulation neural network architecture is 32 × 32, assuming that the upper boundsB and D of external stimulation are both 1, ωij parameters in the latest connection weight μ. The value of μ is 0.7, the value of r0 is 2, the calculated value of C is 0.35, and the value of neuron external input parameter E is 100. In Figure 5 below, two groups of obstacles with different sizes are selected, and the intelligent fruit-picking robot moves upward from the initial position of point P(1, 1). The external input Ii of obstacle neurons in static environment is represented by E, and the values of other neurons are 0. If the intelligent fruit picking robot passes through any point, and the reset result of the external input of the neuron at this point is 0, it indicates that the picking work in the cost area has been completed.Figure 5
Full-coverage path planning.In Figure5 above, the intelligent fruit picking robot starts picking from the bottom to the top and then walks from the top to the bottom in a zigzag shape between planting lines. The steering angle values of the intelligent fruit picking robot are listed in Table 2. The results in the table show that the intelligent fruit robot has to undergo six turns after encountering two groups of obstacles during the picking period and to realize the full-coverage path planning [25].Table 2
Robot motion angle.
The difference between thex-coordinatesThe difference between the coordinatesθ0−150111−14103112


## 4.4.1. Neural Network Dynamic Model of Biological Stimulation

Based on the biological stimulation neural network, the connection relationship between neurons and external inputs in the dynamic environment is defined, and the targets and obstacles corresponding to the lowest end are distributed at the top of the neural activity state diagram. The neuron activities that are easy to be constrained by kinematics are used to spread so as to attract the robot in the global space. Obstacles in the neural activity state diagram can only be partially avoided by the robot [22]. The dynamic neuron activity state diagram based on neural network, combined with the incomplete kinematic constraints of the robot, can reasonably plan the robot path for autonomous obstacle avoidance [23].In 1952, Huxley and Hodgkin first used circuit elements to build a small dendritic cell membrane calculation model based on biological nervous system. The following is the expression of dynamic voltageVm passing through the cell membrane:(10)CmdVmdt=−Ep+Vmgp+ENa+VmgNa−Ek+Vmgk.In the above formula,Cm represents the cell membrane capacitance, Ek represents the potassium ion saturation potential on the cell membrane, ENa represents the sodium ion saturation potential in the cell membrane, and Ep represents the neutral leakage current saturation potential; gk represents potassium ion conductivity, gNa represents sodium ion conductivity, and gp represents neutral channel conductivity.


## 4.4.2. Motion Control

The intelligent fruit-picking robot should complete the steering control and obstacle avoidance control when picking fruit in the planting park. Based on the analysis and derivation of error mechanics, the path planning control of the rear driving wheel of the robot is obtained. It is assumed that the reference speed of the robot isVr and the angular speed is determined by ωr which both robot speeds are calculated by the following formulas [24]:(11)v=c1eD+vrcoseθ,ω=ωr+c2vreL+c3vdsineθ.In the above equations,C1, C2, and C3 represent positive gain parameters, eD represents driving direction, eL represents new direction, and eθ indicates steering angle and direction error.


## 4.4.3. Path Planning

Based on the biological stimulation neural network method, the intelligent fruit-picking robot needs to walk in the botanical garden with complex environment under the condition of no control. Here, it is necessary to use biological stimulation neural network in the static plantation, and the number of static obstacles in each row of the park is less. The walking route of the intelligent fruit picking robot is “Zhi,” and it traverses each row of plants planted until it stops at the position of the picking robot. The number of discrete topological tissue neurons in the biological stimulation neural network architecture is 32 × 32, assuming that the upper boundsB and D of external stimulation are both 1, ωij parameters in the latest connection weight μ. The value of μ is 0.7, the value of r0 is 2, the calculated value of C is 0.35, and the value of neuron external input parameter E is 100. In Figure 5 below, two groups of obstacles with different sizes are selected, and the intelligent fruit-picking robot moves upward from the initial position of point P(1, 1). The external input Ii of obstacle neurons in static environment is represented by E, and the values of other neurons are 0. If the intelligent fruit picking robot passes through any point, and the reset result of the external input of the neuron at this point is 0, it indicates that the picking work in the cost area has been completed.Figure 5
Full-coverage path planning.In Figure5 above, the intelligent fruit picking robot starts picking from the bottom to the top and then walks from the top to the bottom in a zigzag shape between planting lines. The steering angle values of the intelligent fruit picking robot are listed in Table 2. The results in the table show that the intelligent fruit robot has to undergo six turns after encountering two groups of obstacles during the picking period and to realize the full-coverage path planning [25].Table 2
Robot motion angle.
The difference between thex-coordinatesThe difference between the coordinatesθ0−150111−14103112


## 5. Results of Visual Recognition and Path Planning of Intelligent Fruit-Picking Robot

### 5.1. Visual Recognition Analysis of Intelligent Fruit-Picking Robot

In this paper, the following translation matrix can be obtained by recording the end motion distance of the fruit intelligent robot with camera:(12)PxPyPzT=−401.602720.326343.479.The calibrated coordinates are obtained in the robot coordinate system, and then the end position is taught at the target position, and the robot base coordinates are read by the teaching pendant [26]. Based on the binocular stereo vision module, the fruit image is collected and stored. Next, the image coordinates of the calibration object center are marked through the system drawing tool, and then the two-dimensional coordinates (u, v) of the calibration object center are substituted into the stereo vision module to establish the three-dimensional library function and calculate the setting value of the camera coordinate system. In this experiment, ten calibration points were selected, which are shown in Table 3, and the mapping matrix T between the camera coordinates and the robot is obtained through using the least square method, as explained in earlier sections.Table 3
Calibration point camera and robot coordinates.
NumberCamera coordinate systemRobot coordinate systemXc (mm)Yc (mm)Zc (mm)Xr (mm)Yr (mm)Zr (mm)112.69−14.621397.54985.46786.33375.732−203.12−98.421364.35936.65978.32451.1438.26−86.741382.11983.52769.43443.264−0.5210.751535.451126.43785.92352.255271.02−64.171565.261156.45520.48425.326386.99−146.461594.211195.35415.72506.247−34.51−331.241394.71979.43806.92688.738272.69−319.521469.171052.61508.73668.799384.18158.261365.27952.47398.37202.8710232.1656.341433.261024.15558.37302.86The mapping matrix of the intelligent fruit robot in the camera coordinate system is obtained by programming and calculating with MATLAB software, as follows:(13)T=0.01820.00410.9923−401.62−0.99230.02950.0498720.62−0.0009−0.97540.0127343.4290001.The mapping matrix is obtained after the verification of this experiment. The position of the teaching target point at the end of the fruit intelligent robot can be obtained by using the above calibration method, and the corresponding three-dimensional coordinate measurement value can be obtained. However, there is an error between the calculated value of the target point in the intelligent fruit robot coordinate system and the actual measurement value. This error is called calibration error, which is shown in Figure6 below.Figure 6
Test results of 10 calibrated points.According to the experimental results, the maximum calibration error of different components on the target position coordinates of the fruit intelligent robot coordinate system is less than 10 mm. However, the error belongs to the comprehensive error, which also includes the error in the measurement process of the stereo vision module. The following are the two main factors that interfere with the calibration accuracy:(i)
When the detection target is in the coordinate system of the fruit intelligent robot, teach the end center of the robot to the center of the target and then read the position of the robot, that is, the position of the target robot. At this time, it is difficult to coincide the target center and the end center, which interferes with the measurement accuracy and leads to considerable and nontrivial errors.(ii)
Camera 3D reconstruction and intelligent fruit-picking robot have high accuracy. However, when calibrating the system, this error will lead to gain in the calibration system and interfere with the accuracy of the calibration system. Therefore, after several groups of the measurement data, the error interference can be reduced by using the least-squares method.


### 5.2. Visual Path Planning Analysis of Intelligent Fruit-Picking Robot

Due to the complexity of the park itself and the unpredictability of obstacles, it is necessary to combine with genetic algorithm and use the maximum fuzzy two-dimensional entropy to detect different types of obstacles in the orchard. The mathematical model of the genetic algorithm is established, and the fruit picking sequence is formulated according to the global picking path of intelligent fruit robot. The best picking path is calculated asC = {1, 8, 6, 10, 4, 9, 3, 5, 2, 7, 1}, of which the shortest picking path length is 2846 mm. Then, the path is planned based on the biological stimulation neural network dynamic model to realize the full path planning. The results show that the intelligent fruit robot has to go through 6 special projects to realize the full coverage of the orchard path.When planning the full coverage path in the orchard planting area based on biological stimulation neural network, the optimal path and suboptimal path of the intelligent fruit picking robot from the starting point to the key location can be accurately obtained. When the fixed obstacles are known or there are no obstacles, then the intelligent fruit picking robot can fully cover the operation area, and its path shape is similar to the word “Zhi.” This method is simple and efficient. During the path planning, it is necessary to establish a discrete topology map, which can reduce the analysis of the geometric parameters of the park, and the number of columns on the topology map can be determined only according to the number of rows in the park. Experiments show that this path planning method will not be affected by environmental obstacles. It only takes 50 ms to complete the path planning operation, and the biological stimulation neural network model does not need to learn, nor will it cause “deadlock,” so it has strong universal applicability.


## 5.1. Visual Recognition Analysis of Intelligent Fruit-Picking Robot

In this paper, the following translation matrix can be obtained by recording the end motion distance of the fruit intelligent robot with camera:(12)PxPyPzT=−401.602720.326343.479.The calibrated coordinates are obtained in the robot coordinate system, and then the end position is taught at the target position, and the robot base coordinates are read by the teaching pendant [26]. Based on the binocular stereo vision module, the fruit image is collected and stored. Next, the image coordinates of the calibration object center are marked through the system drawing tool, and then the two-dimensional coordinates (u, v) of the calibration object center are substituted into the stereo vision module to establish the three-dimensional library function and calculate the setting value of the camera coordinate system. In this experiment, ten calibration points were selected, which are shown in Table 3, and the mapping matrix T between the camera coordinates and the robot is obtained through using the least square method, as explained in earlier sections.Table 3
Calibration point camera and robot coordinates.
NumberCamera coordinate systemRobot coordinate systemXc (mm)Yc (mm)Zc (mm)Xr (mm)Yr (mm)Zr (mm)112.69−14.621397.54985.46786.33375.732−203.12−98.421364.35936.65978.32451.1438.26−86.741382.11983.52769.43443.264−0.5210.751535.451126.43785.92352.255271.02−64.171565.261156.45520.48425.326386.99−146.461594.211195.35415.72506.247−34.51−331.241394.71979.43806.92688.738272.69−319.521469.171052.61508.73668.799384.18158.261365.27952.47398.37202.8710232.1656.341433.261024.15558.37302.86The mapping matrix of the intelligent fruit robot in the camera coordinate system is obtained by programming and calculating with MATLAB software, as follows:(13)T=0.01820.00410.9923−401.62−0.99230.02950.0498720.62−0.0009−0.97540.0127343.4290001.The mapping matrix is obtained after the verification of this experiment. The position of the teaching target point at the end of the fruit intelligent robot can be obtained by using the above calibration method, and the corresponding three-dimensional coordinate measurement value can be obtained. However, there is an error between the calculated value of the target point in the intelligent fruit robot coordinate system and the actual measurement value. This error is called calibration error, which is shown in Figure6 below.Figure 6
Test results of 10 calibrated points.According to the experimental results, the maximum calibration error of different components on the target position coordinates of the fruit intelligent robot coordinate system is less than 10 mm. However, the error belongs to the comprehensive error, which also includes the error in the measurement process of the stereo vision module. The following are the two main factors that interfere with the calibration accuracy:(i)
When the detection target is in the coordinate system of the fruit intelligent robot, teach the end center of the robot to the center of the target and then read the position of the robot, that is, the position of the target robot. At this time, it is difficult to coincide the target center and the end center, which interferes with the measurement accuracy and leads to considerable and nontrivial errors.(ii)
Camera 3D reconstruction and intelligent fruit-picking robot have high accuracy. However, when calibrating the system, this error will lead to gain in the calibration system and interfere with the accuracy of the calibration system. Therefore, after several groups of the measurement data, the error interference can be reduced by using the least-squares method.


## 5.2. Visual Path Planning Analysis of Intelligent Fruit-Picking Robot

Due to the complexity of the park itself and the unpredictability of obstacles, it is necessary to combine with genetic algorithm and use the maximum fuzzy two-dimensional entropy to detect different types of obstacles in the orchard. The mathematical model of the genetic algorithm is established, and the fruit picking sequence is formulated according to the global picking path of intelligent fruit robot. The best picking path is calculated asC = {1, 8, 6, 10, 4, 9, 3, 5, 2, 7, 1}, of which the shortest picking path length is 2846 mm. Then, the path is planned based on the biological stimulation neural network dynamic model to realize the full path planning. The results show that the intelligent fruit robot has to go through 6 special projects to realize the full coverage of the orchard path.When planning the full coverage path in the orchard planting area based on biological stimulation neural network, the optimal path and suboptimal path of the intelligent fruit picking robot from the starting point to the key location can be accurately obtained. When the fixed obstacles are known or there are no obstacles, then the intelligent fruit picking robot can fully cover the operation area, and its path shape is similar to the word “Zhi.” This method is simple and efficient. During the path planning, it is necessary to establish a discrete topology map, which can reduce the analysis of the geometric parameters of the park, and the number of columns on the topology map can be determined only according to the number of rows in the park. Experiments show that this path planning method will not be affected by environmental obstacles. It only takes 50 ms to complete the path planning operation, and the biological stimulation neural network model does not need to learn, nor will it cause “deadlock,” so it has strong universal applicability.


## 6. Conclusions and Future Work

At present, the problems of agriculture, rural areas, and farmers’ income have attracted the attention of the state. China continues to adjust the agricultural industrial structure. The most obvious change is to expand the area of planting fruit trees. Fruit trees have become the second largest crop after grain lag. However, China’s fruit and vegetable industry also faces many problems in the development process. The production technology is backward, the automation level is raised, the labor efficiency is low, and the fruit is not picked in time, which has a direct impact on the income of farmers. Aiming at this problem, this paper focuses on the visual recognition and path planning of fruit intelligent picking robot, establishes the recognition and positioning system of fruit picking robot based on three-dimensional four wonders, constructs the fruit autonomous picking system, uses binocular stereo vision technology to locate and identify the fruit, and uses fuzzy two-dimensional entropy to identify the obstacles. Then, the trajectory planning of the picking robot based on visual feedback is constructed, and the picking order is reasonably planned by using genetic algorithm. The best path is the best pathC = {1, 8, 6, 10, 4, 9, 3, 5, 2, 7, 1}.We evaluated and showed that the visual recognition system and path planning method of the intelligent fruit picking robot, established in this paper, can accurately identify the fruit position in the park and complete the picking task. In practice, the application of the robot can reduce the labor of fruit farmers and improve the efficiency of fruit picking. This takes trivial time to complete the path planning operation, and the biological stimulation neural network model does not need to learn, nor will it cause “deadlock,” so it has strong universal applicability. In the future, we will integrate machine learning-based intelligent algorithms for decisions making of the fruit picking robots. Moreover, we will use more advanced methods to further improve the efficiency and accuracy of the robots in order to increase the farmer’s profit and decrease the labor efforts.



---

*Source: 1297274-2022-04-14.xml*