# A Point Cloud-Based Feature Recognition and Path Planning Method

Authors: Changhong Chen; Shaofeng Wang; Shunzhou Huang
Journal: Shock and Vibration
 (2022)

---

## Abstract

Based on the research background of in situ automatic ultrasonic phased array inspection of irregular porous castings, due to the limited in situ inspection stations, complex shape of irregular porous castings, and extreme multireflection structural features, it is necessary to identify the positioning inspection features among multiple features to be inspected and plan the optimal inspection path. This research is interested in the porous location recognition and its detection path planning of irregular porous castings. For this, a point cloud-based multifeature contour recognition and location algorithm were proposed to simultaneously extract and locate the hole feature and cylindrical feature from an irregular porous castings. Furthermore, a detection path planning method was put forward to search the shortest robot’s detection path based on the above acquired features by visual recognition and positioning technology. First, through the calibration of the industrial robot tool coordinate system and the internal and external parameters of the camera, the “EyeinHand” hand-eye conversion relationship was established. Second, the robot vision system collects the point cloud information of the area to be inspected and performs point cloud splicing, the accuracy of the original data, the removal of noise such as invalid points, outliers and internal noise points, on this basis, the boundary curve of the hole to be inspected was extracted, the cylindrical equation was fitted, its geometric center was calculated, and the central coordinates and axis direction of the contour of the hole to be inspected were obtained. Finally, all the detection paths were traversed through the multibranch tree to obtain the optimal detection path of the detection points of multiple targets. The experimental results show that the positioning accuracy of the feature of the hole to be inspected by the vision system is 0.107 mm, the aperture extraction accuracy is 0.002 mm, the cylinder fitting accuracy is 0.04 mm, and the calculation accuracy of the angle between the two axes is within 0.4. When the number of features to be inspected is different, the average moving distance can be saved by 10.7% by using the end effector after path optimization. The feasibility of in situ automatic ultrasonic phased array detection for irregular porous castings using by visual positioning is verified.


---

## Body

## 1. Introduction

Irregular porous castings are widely used in active military and naval weapons and equipment. As a key component of this type of large-scale weapons equipment, with the improvement of the requirements of battlefield indexes such as multiple bursts, high bore pressure, and long distance, its structural features are large and heavy loads, multiple deep and narrow cylindrical cavities, and adjacent distribution of barrel assembly holes. If the structural strength of the adjacent area of the multideep and narrow column cavity is weak, and there are missing casting defects in the critical stress area, then with the omission of inspection defects, under the cyclic ignition loads of up to 3000 tons, the defects are constantly expanding. Inevitably, there will be failure problems such as large deformation or instant fracture. In order to control the internal quality of irregular porous castings more effectively, the manufacturing process requires flaw detection of irregular porous castings after the process of “parts processing.” At this time, the irregular porous casting has not been processed and cannot be reassembled or removed from the machine tool. Otherwise, machining accuracy will be affected, and the irregular porous castings can only be inspected online. Therefore, the detection station is limited, and the shape of irregular porous casting parts is complex, and there are more extreme multireflective structure features. Aiming at the abovementioned detection problems, it is an effective solution to use in situ automated ultrasonic phased array detection.For most application scenarios of industrial robots, it is necessary to predict the position and pose of the target object to complete the corresponding task. Xu Chunguang’s team [1, 2] developed a robotic ultrasonic inspection system, which requires the calibration of the fixture coordinate system of the workpiece to be inspected and the inspection coordinate system of the dual robotic arms. Furthermore, the digital model is used to locate the ultrasonic testing reference point of the regular shape workpiece, and the automatic control of the ultrasonic testing of the robotic arm is finally realized. Zhao et al. [3] used tooling fixtures to fix the workpiece to be inspected, and based on the mathematical representation method of the workpiece geometry and model, realized the positioning of the workpiece to be inspected datum point on the CNC machine tool. Abdallah et al. [4] used the information in the CAD model of the part to perform offline programming to detect defects in aviation machinery parts and check whether all subcomponents were installed correctly. Božek et al. [5] offered a new method of analytic geometry to calculate the trajectories of mechatronic systems and CAD/CAM. The method can accurately calculate the trajectory for the centrally symmetric conic sections and, in some cases, arbitrary differentiable planar curves. Obviously, the use of robotic arms in this type of technology has become the mainstream of automatic ultrasonic testing, but the abovementioned technology cannot realize the testing of irregular porous castings. The key reason is that its actual casting shape does not correspond to its digital model, so it is impossible to accurately locate its testing reference point by precision tooling. At present, in the industrial applications of robots, the working path of most robot is planned through manual teaching or offline programming. However, this method strictly limits the starting and ending poses of the target object. The preplanned path can only be repeated mechanically, and it is necessary to teach different working objects separately to get the corresponding teaching files. The highly structured working environment seriously restricts the efficiency, flexibility, and intelligence of the robot, which cannot meet the requirements of a flexible production system [6]. Machine vision is widely used in industrial production because of its fast, stable, and noncontact characteristics [7]. With the help of visual guidance, assist the robot to locate the target object. This method can effectively improve the working efficiency and automation level of the production line.DeAraujo and Lins [8] used the binocular vision inspection method to detect the features to be inspected, and the spatial coordinates of the reference points to be inspected in the machine coordinate system are output, so as to realize the accurate spatial positioning of the reference points of irregular features to be inspected, the proposed system presented, on average, absolute deviations of (0.46 ± 0.06 mm), (0.11 ± 0.05 mm), and (0.25 ± 0.09 mm) in X, Y, and Z axes. Mineo et al. [9] developed a 6DOF robotic arm-assisted ultrasonic phased array inspection system, which uses machine vision inspection technology to build a global ultrasonic inspection reference field for the workpiece to be inspected, and realizes effectively locate and detect the characteristics of the workpiece to be inspected. However, the disadvantage is that the construction cost of this global ultrasonic testing system is quite expensive. Khan et al. [10] used the depth vision sensor to pick up the datum of the workpiece to be inspected, and based on this, the ultrasonic inspection path is planned to realize the guidance of automatic ultrasonic inspection, and the image acquisition method produces a UT tool path with an accuracy of 0.43 mm. Sun et al. [11] calculated the local plane normal vector determined by the point cloud method, and then fitted the reference point with B-spline curve equation, and finally, the elevation of the reference surface and the inclination angle of the radial point were calculated according to the fitted curve equation, so as to obtain the position and normal characteristics of the key points of the point cloud. Yang et al. [12] propose a novel 3-D path extraction method of weld seams based on a stereo-structured light sensor, which could be well adapt to different types of welds and different groove sizes. Meanwhile, the position information and pose information of weld seam are established to serve 3-D path teaching of a welding robot. The maximum path extraction error of V-type but joint is less than 0.7 mm. Liu [13] in order to realize the rapid measurement for large dimension workpieces in industrial fields, a new measuring system based on stereovision was proposed and implemented. A binocular structure light scanner was constructed to acquire the surface detail information of the workpieces. Experimental results indicate that the new measuring system is suitable for large workpiece measurement with an accuracy of 0.112 mm/3 m. It can be seen that using machine vision technology to pick up the special-shaped porous castings with variable curvature casting surface characteristics to be inspected, such as the space coordinates of the reference points to be inspected, normal, and inspection sequence planning, so as to drive the ultrasonic transducer. It is a good choice to perform ultrasonic testing of the robotic arm. Kuric et al. [14] present pose repeatability measurement of the industrial robot Fanuc LR Mate 200iC. By evaluating results of pose repeatability, it can be said that the repeatability of the robot at the selected area of its workspace corresponds to the repeatability value stated by the manufacturer (±0.02 mm). Pollak [15] describes the measurement of unidirectional pose accuracy and repeatability of a collaborative robot. The objective of the measurements is to investigate and evaluate unidirectional accuracy of the six-axis collaborative robot UR5 of the company Universal Robots. On the basis of the measurement, they verified technical specifications of unidirectional pose accuracy and repeatability of the robotic arm UR5specified by its producer. Ren [16] proposed a general pose testing method of industrial robot. This testing method is illustrated with a KUKA KR210 R2700 industrial robot with a drilling end effector. The pose accuracy and repeatability of the robot are 0.3 mm and 0.06 mm, which are in accordance with the nominal index. And the coordinate deviation of the tool center point (TCP) from direct measurement is less than 0.015 mm. It can be seen that using a robot to drive an ultrasonic transducer for ultrasonic nondestructive testing has the characteristics of good pose repeatability and positioning accuracy.The global positioning of the ultrasonic inspection of irregular porous castings is mainly reflected in the determination of the spatial coordinates and normal vectors of several reference points to be inspected and the planning of ultrasonic scanning sequence. To solve this problem, Graumann [17] introduced an automatic ultrasonic examination trajectory planning algorithm, which uses the depth camera to obtain the point cloud of the three-dimensional patient surface, and set the parameter path function to generate single or multiple parallel scanning trajectories of a larger target surface. Then, the spline curve is generated by preparing the path, so that the ultrasonic probe can scan smoothly. Wu and Huang [18] obtained the depth data of the tissue surface through Kinect, converted it to the coordinate system of the detection equipment, automatically planned the scanning path, and guides the ultrasonic probe to automatically scan. Shahzad et al. [19] put forward machine vision and path planning technology for shortest path calculation to improve the production efficiency of industrial robots and successfully realized the precise path optimization in the bottle manufacturing industry. It can be seen that by traversing all the reference points to be inspected, it is possible to realize the inspection and scanning sequence planning of multiple inspection surfaces of the irregular porous castings, thus realizing the global path planning of the ultrasonic inspection of the robot arm of the irregular porous castings.According to the above research, planning the automatic detection process through manual teaching or off-line programming can only be repeated mechanically, and different working objects need to be taught separately to obtain the corresponding teaching files, which severely restricts the working efficiency and flexibility of the robot. At present, there are some problems in the field of ultrasonic automatic detection, such as insufficient detection accuracy, expensive equipment, and less application. Therefore, this paper constructs a vision-guided industrial robot ultrasonic detection system. First, aiming at the difficult positioning of the complex structure of irregular porous castings, a multifeature profile recognition and positioning algorithm based on point cloud are proposed to achieve the accurate acquisition of the poses of the feature points to be inspected. Second, a multifeature detection path optimization method is presented. Based on the vision inspection technology, the feature position of the irregular porous casting is identified, and the inspection path of the robot is generated, which solves the problem of inefficiency caused by unreasonable inspection paths. Finally, experimental tests and results are provided, and the performance of this method is illustrated in the following section.


## 2. Materials and Methods

### 2.1. Visual Positioning Algorithm

In order to solve the difficult problem of detecting the complex structure of special-shaped porous castings, this paper proposes a multifeature surface mathematical representation algorithm based on point clouds. This method can identify the inner cylindrical surface and outer cylindrical surface of irregular castings in the point cloud data and accurately extract the hole position, aperture parameters, and mathematical characteristics of the cylindrical surface. This method uses both the contour extraction method and the cylindrical fitting method, which can extract multiple features of irregular porous castings more conveniently and directly, and is more convenient than the traditional single feature extraction method. The original algorithm makes the edge extraction effect better by adjusting the threshold and improves the accuracy of cylinder fitting.The point cloud data collected by the RVCXmini depth camera have no obvious geometric distribution characteristics, and the measurement points are randomly generated, showing a disordered state. Because of the disordered and random distribution of the point cloud, most of the measurement points are distributed around the feature to be detected. The measuring points at the boundary of the hole may be distributed on the inner diameter and surrounding plane of the hole, and only a few points are just at the boundary of the hole. According to the distribution of measuring points on the hole boundary, it can be judged whether it is a boundary point or not by the geometric distribution ofk neighborhood points [20]. If a certain point is a boundary point, the distribution of its k neighborhood points will be tilted to one side (as shown in Figure 1(a)). If a certain point is not a boundary point, the k neighborhood points are more evenly around the point (as shown in Figure 1(b)).Figure 1
(a) Boundary point. (b) Not a boundary point.
(a)(b)Therefore, for the scattered point cloud at the boundary of the hole, it is necessary to perform boundary point extraction preprocessing [21, 22], including point cloud splicing, original data accuracy inspection, invalid points removal, outliers and internal noise points, and other noise removal. Research [23, 24] proposes deep learning methods to remove noise and preserve its clear features to filter point clouds automatically and robustly and achieve automatic prediction of normals. Research [25] proposed a robust normal estimation method for point clouds using a low-rank matrix approximation algorithm and provide a new filtering method for point cloud data to smooth the position data to fit the estimated normals. At present, the deep learning method for point cloud preprocessing is very common. However, in this study, the traditional filtering method is still used for point cloud preprocessing. Because the characteristics of special-shaped porous castings are relatively obvious, this research is carried out for this workpiece. In order to reduce the influence of discrete point cloud on segmentation, based on the principle of statistical filtering, the discrete noise point cloud is removed according to the degree of discrete point cloud [26]. For any point px,y,z in the point cloud, define a point with px,y,z as the center, R is the radius of the sphere VRpx,y,z, As shown in Figure 2, the black solid point in the figure indicates the point cloud. If pixi,yi,zi falls within the sphere VRpx,y,z, then the point cloud pixi,yi,zi is considered to belong to the current point cloud px,y,z. The number of points pixi,yi,zi falling in the sphere VRpx,y,z is recorded as Drp, and Drp is used to measure the dispersity of point cloud, which is defined as(1)di=x−xi2+y−yi2+z−zi2,i=1,2,…,n,Drp=numdi<R.Figure 2
Statistical filtering.In formula (1), di represents the distance between px,y,z and pix,y,z, and num represents the number of distances less than R. If Drp is less than the threshold Drmin, the point px,y,z is detected as a discrete point, otherwise it is a valid point.Since the average distance from one point to the adjacent point in the point cloud diagram is approximately in accordance with Gaussian distribution. Calculate the averageμd and standard deviation σd of the distance between px,y,z and all point clouds pixi,yi,zii=1,2,…,n in the neighborhood of R:(2)μd=∑i=1ndi/n,σd=1/n∑i=1ndi−μd2.According to Chebyshev’s inequalityPX−μ≥ε≤σ2/ε2, when ε=5σ, the misjudgment rate is the smallest. Therefore, R in formula (1) is set to 5σd, and Drmin  = 50.As shown in Figure3, the number of point clouds before filtering is 424355, and the number of point clouds after filtering is 219066. In addition, the discrete points and the background points can be eliminated through filtering, see Figure 3(b).Figure 3
Point cloud image. (a) Before filtering; (b) after filtering.According to the characteristics of flat hole, the contour is linearly approximated byα shapes algorithm [27]. For a plane point cloud of arbitrary shape, if a circle with a radius of α is rolled around it, if the rolling circle radius α is small enough, every point in the point cloud is a boundary point, as shown in Figure 4; If α is properly increased to a certain degree, it will only roll on the boundary point, and its rolling trajectory is the boundary of point cloud.Figure 4
Alpha shapes algorithm.The specific steps of the algorithm are in(1)
For any pointP, the radius of the rolling circle is α, and all points within 2α from the point P are searched in the point cloud, which is recorded as the point set Q;(2)
As shown in Figure5, select any point p1x1,y1 in Q. According to the coordinates of these two points and α, the center coordinates are calculated:The center of the circlep2 and p3 are the coordinates of the center of the circle when the two points p and p1 are passed and the radius is α. The coordinate calculation formula is shown in formula (3):(3)x2=x+12x1−x−H∗y1−y,y2=y+12y1−y−H∗x−x1,x3=x+12x2−x−H∗y2−y,y3=y+12y2−y−H∗x−x2.Among them(4)H=α2S2−14,S=x−x12y−y12.(3)
After removing the pointp1 from the Q point, calculate the distances from the remaining points to the points p2 and p3 respectively. If the distance from all points to p2 or p3 is greater than α, it indicates that p is a boundary point.(4)
If the distances between the remaining points to the pointsp2 or p3 are not all greater than α, all points in the traversal point set Q are rotated as points p1. If there is a point that satisfies conditions (2) and (3), it means that the point is a boundary point, and the judgment of this point is terminated to judge the next point. If there is no such point as p1 among adjacent points in Q, it means that point p is a nonboundary point.Figure 5
Center coordinates.In the algorithm, the radiusα controls the fineness of the boundary. The radius of the radius α directly affects the accuracy of extracting boundary point. Through many experiments, the proper radius α value is set to estimate the characteristic boundary of the irregular porous castings. Specific steps are as follows:(1)
Input the point cloud and set the initialα.(2)
Set the initial point of the boundary. According to the properties of the convex hull, any extreme point of the convex hull in the point set is the boundary point of the point set, and it is set as the initial point of the boundary.(3)
Obtain the boundary point set of the entire point cloud. Determine whether the boundary point set contains boundary initial points and set the threshold of the number of boundary initial points included.(4)
Updateα and judge whether α reaches the termination condition.The abovementionedα shapes boundary extraction method is used to extract the boundary of the hole features of special-shaped porous castings, and the obtained hole boundary by setting the α value to 1 mm is shown in Figure 6.Figure 6
Special-shaped porous casting’s boundary.According to the cylindrical surface characteristics of the special-shaped porous castings, as shown in Figure7, P is any point on the cylindrical surface, P0x0,y0,z0 is a point on the cylinder axis, a,b,c is the cylinder axis vector, and r0 is the radius of the cylinder bottom circle. It can be concluded that the distance from a point on the cylindrical surface to its axis is always equal to the radius r0, namely: x−x02+y−y02+z−z02−ax−x0+by−y0+cz−z02=r02.Figure 7
Geometric characteristics of the cylindrical surface.By calculating these seven parameters, one cylinder can be uniquely determined [28]. The cylinder fitting method used in this paper is described in detail as follows:(1)
Determine the initial value of the parameters of the cylinder. Search for several adjacent points at any point on the cylinder, fit these points into a plane, and unify the normal vector of the plane into the unit normal vector of the point. Each point on the cylinder is processed to obtain the unit normal vector of each point. The unit normal vector of each point is regarded as a point, and these points are fitted into a plane to get the plane normal vector, that is, the initial valuesa0, b0, and c0 of the cylinder axial quantity. In this step, principal component analysis method is used to solve the problem. After the axis is obtained, coordinate transformation is carried out on the cylinder, so that the axial quantity a0,b0,c0 of the cylinder is transformed into a vector parallel to the Z axis, and the (x, y) coordinate of a point on the cylinder is a plane circle. Use them fit the center of the circle, namely x1,y1,z1 and radius r.(2)
Establish the error equation.(5)f=x−x12+y−y12+z−z12−ax−x1+by−y1+cz−z12−r2,f=x−x12+y−y12+z−z12−ax−x1+by−y1+cz−z12−r22r.Linearizef:(6)f=f0+∂f∂x1x1+∂f∂y1y1+∂f∂z1z1+∂f∂aa+∂f∂bb+∂f∂cc+∂f∂rr.The error equation is(7)V=f0+∂f∂x1x1+∂f∂y1y1+∂f∂z1z1+∂f∂aa+∂f∂bb+∂f∂cc+∂f∂rr,among them(8)f0=x−x102+y−y102+z−z102−ax−x10+by−y10+cz−z102−r022r0.The error equation is written in matrix formAX=Y, the least square solution of AX=0 is solved by the least square method, and the eigenvector corresponding to the minimum eigenvalue of AT∗A is solved. This is a process of cyclic iteration. The initial value substituted in each iteration is equal to the previous initial value plus the modified X value. When the X value is small enough to meet the required accuracy, the iteration will exit.The cylindrical feature of special-shaped porous castings is extracted by using the abovementioned cylindrical feature fitting method, and the extracted cylindrical feature is shown in Figure8. The point cloud after the above processing can represent the main features to be inspected of special-shaped porous castings, which lays a foundation for the subsequent experiments.Figure 8
Cylindrical surface fitting.


### 2.2. Inspection Path Planning

The location of features to be detected and the extraction of the detection paths by machine vision lay the foundation for automatic detection. Because there is more than one feature to be inspected, it is inevitable that there will be many features to be inspected in the working range. In order to improve detection efficiency, this paper studies and designs a robot path optimization algorithm suitable for multifeatures to be detected. Through this path optimization algorithm, when multiple targets to be detected are detected, the path can make the end of the robot move the shortest distance among multiple targets can be found, thus improving the efficiency of automatic detection of ultrasonic phased array.Assume that there are 3 features to be detected, in which pointO represents the initial point when the end effector starts working, and A, B, and C represent the position of the feature to be detected. Starting from the initial point O, the end effector can select features to be inspected from A, B, and C for detection. Considering that the area to be inspected has three characteristics to be inspected, the schematic diagram of any detection path is shown in Figure 9. Assuming that there are N < 8 features to be inspected in the visual area, the total number of all features to be inspected is N! alternative paths. Taking 3 features to be inspected in the area as an example, there are 6 different inspection paths in total, and a path planning decision tree as shown in Figure 10 can be built.Figure 9
Detection path.Figure 10
Path planning decision tree.The optimal path planning algorithm is designed according to the path planning decision tree. By traversing all paths, we can get the shortest path during the process of the end effector movement. The formula for calculating the distance of the segment is as follows:(9)d1=xa−xo2−ya−yo2,d2=xb−xa2−yb−ya2,d3=xc−xb2−yc−yb2.The formula for calculating the total distance of the path is(10)d=∑i=13di.This path is an alternative one. To get the optimal path, traverse all feasible paths in the path planning decision tree shown in Figure8 and select the path with the shortest distance as the optimal path.


## 2.1. Visual Positioning Algorithm

In order to solve the difficult problem of detecting the complex structure of special-shaped porous castings, this paper proposes a multifeature surface mathematical representation algorithm based on point clouds. This method can identify the inner cylindrical surface and outer cylindrical surface of irregular castings in the point cloud data and accurately extract the hole position, aperture parameters, and mathematical characteristics of the cylindrical surface. This method uses both the contour extraction method and the cylindrical fitting method, which can extract multiple features of irregular porous castings more conveniently and directly, and is more convenient than the traditional single feature extraction method. The original algorithm makes the edge extraction effect better by adjusting the threshold and improves the accuracy of cylinder fitting.The point cloud data collected by the RVCXmini depth camera have no obvious geometric distribution characteristics, and the measurement points are randomly generated, showing a disordered state. Because of the disordered and random distribution of the point cloud, most of the measurement points are distributed around the feature to be detected. The measuring points at the boundary of the hole may be distributed on the inner diameter and surrounding plane of the hole, and only a few points are just at the boundary of the hole. According to the distribution of measuring points on the hole boundary, it can be judged whether it is a boundary point or not by the geometric distribution ofk neighborhood points [20]. If a certain point is a boundary point, the distribution of its k neighborhood points will be tilted to one side (as shown in Figure 1(a)). If a certain point is not a boundary point, the k neighborhood points are more evenly around the point (as shown in Figure 1(b)).Figure 1
(a) Boundary point. (b) Not a boundary point.
(a)(b)Therefore, for the scattered point cloud at the boundary of the hole, it is necessary to perform boundary point extraction preprocessing [21, 22], including point cloud splicing, original data accuracy inspection, invalid points removal, outliers and internal noise points, and other noise removal. Research [23, 24] proposes deep learning methods to remove noise and preserve its clear features to filter point clouds automatically and robustly and achieve automatic prediction of normals. Research [25] proposed a robust normal estimation method for point clouds using a low-rank matrix approximation algorithm and provide a new filtering method for point cloud data to smooth the position data to fit the estimated normals. At present, the deep learning method for point cloud preprocessing is very common. However, in this study, the traditional filtering method is still used for point cloud preprocessing. Because the characteristics of special-shaped porous castings are relatively obvious, this research is carried out for this workpiece. In order to reduce the influence of discrete point cloud on segmentation, based on the principle of statistical filtering, the discrete noise point cloud is removed according to the degree of discrete point cloud [26]. For any point px,y,z in the point cloud, define a point with px,y,z as the center, R is the radius of the sphere VRpx,y,z, As shown in Figure 2, the black solid point in the figure indicates the point cloud. If pixi,yi,zi falls within the sphere VRpx,y,z, then the point cloud pixi,yi,zi is considered to belong to the current point cloud px,y,z. The number of points pixi,yi,zi falling in the sphere VRpx,y,z is recorded as Drp, and Drp is used to measure the dispersity of point cloud, which is defined as(1)di=x−xi2+y−yi2+z−zi2,i=1,2,…,n,Drp=numdi<R.Figure 2
Statistical filtering.In formula (1), di represents the distance between px,y,z and pix,y,z, and num represents the number of distances less than R. If Drp is less than the threshold Drmin, the point px,y,z is detected as a discrete point, otherwise it is a valid point.Since the average distance from one point to the adjacent point in the point cloud diagram is approximately in accordance with Gaussian distribution. Calculate the averageμd and standard deviation σd of the distance between px,y,z and all point clouds pixi,yi,zii=1,2,…,n in the neighborhood of R:(2)μd=∑i=1ndi/n,σd=1/n∑i=1ndi−μd2.According to Chebyshev’s inequalityPX−μ≥ε≤σ2/ε2, when ε=5σ, the misjudgment rate is the smallest. Therefore, R in formula (1) is set to 5σd, and Drmin  = 50.As shown in Figure3, the number of point clouds before filtering is 424355, and the number of point clouds after filtering is 219066. In addition, the discrete points and the background points can be eliminated through filtering, see Figure 3(b).Figure 3
Point cloud image. (a) Before filtering; (b) after filtering.According to the characteristics of flat hole, the contour is linearly approximated byα shapes algorithm [27]. For a plane point cloud of arbitrary shape, if a circle with a radius of α is rolled around it, if the rolling circle radius α is small enough, every point in the point cloud is a boundary point, as shown in Figure 4; If α is properly increased to a certain degree, it will only roll on the boundary point, and its rolling trajectory is the boundary of point cloud.Figure 4
Alpha shapes algorithm.The specific steps of the algorithm are in(1)
For any pointP, the radius of the rolling circle is α, and all points within 2α from the point P are searched in the point cloud, which is recorded as the point set Q;(2)
As shown in Figure5, select any point p1x1,y1 in Q. According to the coordinates of these two points and α, the center coordinates are calculated:The center of the circlep2 and p3 are the coordinates of the center of the circle when the two points p and p1 are passed and the radius is α. The coordinate calculation formula is shown in formula (3):(3)x2=x+12x1−x−H∗y1−y,y2=y+12y1−y−H∗x−x1,x3=x+12x2−x−H∗y2−y,y3=y+12y2−y−H∗x−x2.Among them(4)H=α2S2−14,S=x−x12y−y12.(3)
After removing the pointp1 from the Q point, calculate the distances from the remaining points to the points p2 and p3 respectively. If the distance from all points to p2 or p3 is greater than α, it indicates that p is a boundary point.(4)
If the distances between the remaining points to the pointsp2 or p3 are not all greater than α, all points in the traversal point set Q are rotated as points p1. If there is a point that satisfies conditions (2) and (3), it means that the point is a boundary point, and the judgment of this point is terminated to judge the next point. If there is no such point as p1 among adjacent points in Q, it means that point p is a nonboundary point.Figure 5
Center coordinates.In the algorithm, the radiusα controls the fineness of the boundary. The radius of the radius α directly affects the accuracy of extracting boundary point. Through many experiments, the proper radius α value is set to estimate the characteristic boundary of the irregular porous castings. Specific steps are as follows:(1)
Input the point cloud and set the initialα.(2)
Set the initial point of the boundary. According to the properties of the convex hull, any extreme point of the convex hull in the point set is the boundary point of the point set, and it is set as the initial point of the boundary.(3)
Obtain the boundary point set of the entire point cloud. Determine whether the boundary point set contains boundary initial points and set the threshold of the number of boundary initial points included.(4)
Updateα and judge whether α reaches the termination condition.The abovementionedα shapes boundary extraction method is used to extract the boundary of the hole features of special-shaped porous castings, and the obtained hole boundary by setting the α value to 1 mm is shown in Figure 6.Figure 6
Special-shaped porous casting’s boundary.According to the cylindrical surface characteristics of the special-shaped porous castings, as shown in Figure7, P is any point on the cylindrical surface, P0x0,y0,z0 is a point on the cylinder axis, a,b,c is the cylinder axis vector, and r0 is the radius of the cylinder bottom circle. It can be concluded that the distance from a point on the cylindrical surface to its axis is always equal to the radius r0, namely: x−x02+y−y02+z−z02−ax−x0+by−y0+cz−z02=r02.Figure 7
Geometric characteristics of the cylindrical surface.By calculating these seven parameters, one cylinder can be uniquely determined [28]. The cylinder fitting method used in this paper is described in detail as follows:(1)
Determine the initial value of the parameters of the cylinder. Search for several adjacent points at any point on the cylinder, fit these points into a plane, and unify the normal vector of the plane into the unit normal vector of the point. Each point on the cylinder is processed to obtain the unit normal vector of each point. The unit normal vector of each point is regarded as a point, and these points are fitted into a plane to get the plane normal vector, that is, the initial valuesa0, b0, and c0 of the cylinder axial quantity. In this step, principal component analysis method is used to solve the problem. After the axis is obtained, coordinate transformation is carried out on the cylinder, so that the axial quantity a0,b0,c0 of the cylinder is transformed into a vector parallel to the Z axis, and the (x, y) coordinate of a point on the cylinder is a plane circle. Use them fit the center of the circle, namely x1,y1,z1 and radius r.(2)
Establish the error equation.(5)f=x−x12+y−y12+z−z12−ax−x1+by−y1+cz−z12−r2,f=x−x12+y−y12+z−z12−ax−x1+by−y1+cz−z12−r22r.Linearizef:(6)f=f0+∂f∂x1x1+∂f∂y1y1+∂f∂z1z1+∂f∂aa+∂f∂bb+∂f∂cc+∂f∂rr.The error equation is(7)V=f0+∂f∂x1x1+∂f∂y1y1+∂f∂z1z1+∂f∂aa+∂f∂bb+∂f∂cc+∂f∂rr,among them(8)f0=x−x102+y−y102+z−z102−ax−x10+by−y10+cz−z102−r022r0.The error equation is written in matrix formAX=Y, the least square solution of AX=0 is solved by the least square method, and the eigenvector corresponding to the minimum eigenvalue of AT∗A is solved. This is a process of cyclic iteration. The initial value substituted in each iteration is equal to the previous initial value plus the modified X value. When the X value is small enough to meet the required accuracy, the iteration will exit.The cylindrical feature of special-shaped porous castings is extracted by using the abovementioned cylindrical feature fitting method, and the extracted cylindrical feature is shown in Figure8. The point cloud after the above processing can represent the main features to be inspected of special-shaped porous castings, which lays a foundation for the subsequent experiments.Figure 8
Cylindrical surface fitting.


## 2.2. Inspection Path Planning

The location of features to be detected and the extraction of the detection paths by machine vision lay the foundation for automatic detection. Because there is more than one feature to be inspected, it is inevitable that there will be many features to be inspected in the working range. In order to improve detection efficiency, this paper studies and designs a robot path optimization algorithm suitable for multifeatures to be detected. Through this path optimization algorithm, when multiple targets to be detected are detected, the path can make the end of the robot move the shortest distance among multiple targets can be found, thus improving the efficiency of automatic detection of ultrasonic phased array.Assume that there are 3 features to be detected, in which pointO represents the initial point when the end effector starts working, and A, B, and C represent the position of the feature to be detected. Starting from the initial point O, the end effector can select features to be inspected from A, B, and C for detection. Considering that the area to be inspected has three characteristics to be inspected, the schematic diagram of any detection path is shown in Figure 9. Assuming that there are N < 8 features to be inspected in the visual area, the total number of all features to be inspected is N! alternative paths. Taking 3 features to be inspected in the area as an example, there are 6 different inspection paths in total, and a path planning decision tree as shown in Figure 10 can be built.Figure 9
Detection path.Figure 10
Path planning decision tree.The optimal path planning algorithm is designed according to the path planning decision tree. By traversing all paths, we can get the shortest path during the process of the end effector movement. The formula for calculating the distance of the segment is as follows:(9)d1=xa−xo2−ya−yo2,d2=xb−xa2−yb−ya2,d3=xc−xb2−yc−yb2.The formula for calculating the total distance of the path is(10)d=∑i=13di.This path is an alternative one. To get the optimal path, traverse all feasible paths in the path planning decision tree shown in Figure8 and select the path with the shortest distance as the optimal path.


## 3. Experimental System

### 3.1. System Components

The vision-guided ultrasonic inspection system for industrial robots is mainly composed of KUKA KR20 R1810 industrial robot, M2M ultrasonic phased array detector, PC and RVCXmini depth camera, probe, fixed bracket, and other parts. Ultrasonic phased array detector is used for ultrasonic nondestructive testing of special-shaped porous castings, PC is used for image processing and path planning, probes are used for system calibration, and the attitude of probes and ultrasonic phased array probes can be interchanged. The RVCXmini depth camera is composed of two 2D industrial cameras and a structured light sensor. The structure of the system is shown in Figure11.Figure 11
System architecture.The algorithm development environment adopts 64-bit Windows 10 operating system, and the RVCXmini depth camera driver, Matlab, RVC Manager image collection toolbox, PCL point cloud processing toolbox and robot network communication toolbox are installed, and the Visual Studio 2015 software development framework is adopted. Develop directly with C++, call OpenCV, and PCL computer vision libraries and directly call function libraries to process images. The software flow chart is shown in Figure12.Figure 12
Software module flowchart.


### 3.2. System Calibration

Figure13 shows the experimental set-up for establishing the hand-target-eye spatial attitude transformation. The RVCXmini depth camera is installed at the end of the robot arm, and the pose of the target object in the robot tool coordinate system is further obtained by using the method of joint calibration of the robot hand-eye relationship and the tool coordinate system. According to the positioning result, the robot arm drives the end tool to move to the target position pose. By comparing the pose data of the robot arm tool end with the pose data of the robot arm tool end detected by the vision system, the robustness and accuracy of the algorithm are verified.Figure 13
Experimental set-up for establishing the hand-target-eye space pose conversion relationship.First, the internal parameters of the camera are calibrated using Zhang Zhengyou’s checkerboard calibration method [29] to take 15 to 20 checkerboard target images at different positions and angles. The principal point coordinates u0,v0 of the camera focal length f and the distortion coefficients k1, k2, p1, p2 can be obtained by Zhang Zhengyou’s calibration algorithm. Thereby, the internal parameter transformation matrix Mc between the image coordinate system and the camera coordinate system is obtained.Second, carry out EyeinHand hand-eye calibration [30, 31], fix the RVCXmini depth camera to the end flange of the robot and fix the checkerboard in the working area, so that the RVCXmini depth camera can follow robot's movement. The camera takes 15 to 20 two-dimensional images of checkerboard targets and point cloud images at different positions and angles and records the posture of the robot during each shooting time so as to calibrate the hand-eye transformation matrix Mu between the camera and the end of the robot.Finally, the tool coordinate system is calibrated, that is, the transformation matrix from the tool coordinate system to the robot flange coordinate system is determined, there is no attitude change between the center point of the ultrasonic phased array probe wafer and the center point of the robot flange end, but only the translation in position, so the transformation matrixMt of the tool coordinate system is established.As shown in Figure13, Cv is define as a certain point in the pixel coordinate system, and Cc is defined as the corresponding pixel point in the camera coordinate. Establish the relationship between them Cv∗Mc=Cc, define Cu as the coordinates of the robot coordinate system, the external parameter equation between the camera coordinate system and the robot coordinate system is Cc∗Mu=Cu, define Ce as the coordinate of the center point of the ultrasonic phased array wafer, and establish the coordinate relation expression according to the tool coordinate system as Cu∗Mt=Ce. When a certain point on the image Cv is identified, Ce=Mc∗Mu∗Mt∗Cv can be deduced according to the above formula.As a result, the hand-target-eye spatial attitude transformation relation is established. According to existing research [32–34], the camera is used to measure unidirectional pose accuracy and repeatability of industrial robots. The basic principle is to use a structured light sensor to indirectly measure the absolute posture of the robot by measuring 4 fixed reference points and to compare the real posture with the command posture to test the accuracy and repeatability of the posture. By evaluating results of pose repeatability, it can be said that the repeatability of the robot at the selected area of its workspace corresponds to the repeatability value stated by the manufacturer (±0.04 mm).


### 3.3. Experimental Project

Figure14 shows the experimental device for visually inspecting and identifying the characteristics for irregular porous castings. Make sure that the features to be inspected of irregular porous castings are within the camera's field of view, scan through the RVCXmini depth camera to identify the target object of the irregular porous castings to be inspected, and obtain the pose of the target object in the camera coordinate system. Using the measurement algorithm proposed in this paper, the 4 groups of features to be inspected for irregular porous castings are repeatedly measured for 9 times, and the error range of the calculation algorithm is analyzed according to the measurement results, so as to verify the stability of the algorithm.Figure 14
Visual inspection and identification of features.Figure15 shows the experimental setup of path optimization for multifeature detection. Obtain multiple features to be detected of special-shaped porous castings through visual inspection and feature recognition, by calculating the detection path distance between multiple features, select the shortest path during the movement of the ultrasonic probe at the end of the robotic arm. Calculate the distance of detection path, select the shortest path according to different numbers of features to be detected, and then carry out the corresponding ultrasonic phased array nondestructive testing operation.Figure 15
Multifeature detection path optimization.


## 3.1. System Components

The vision-guided ultrasonic inspection system for industrial robots is mainly composed of KUKA KR20 R1810 industrial robot, M2M ultrasonic phased array detector, PC and RVCXmini depth camera, probe, fixed bracket, and other parts. Ultrasonic phased array detector is used for ultrasonic nondestructive testing of special-shaped porous castings, PC is used for image processing and path planning, probes are used for system calibration, and the attitude of probes and ultrasonic phased array probes can be interchanged. The RVCXmini depth camera is composed of two 2D industrial cameras and a structured light sensor. The structure of the system is shown in Figure11.Figure 11
System architecture.The algorithm development environment adopts 64-bit Windows 10 operating system, and the RVCXmini depth camera driver, Matlab, RVC Manager image collection toolbox, PCL point cloud processing toolbox and robot network communication toolbox are installed, and the Visual Studio 2015 software development framework is adopted. Develop directly with C++, call OpenCV, and PCL computer vision libraries and directly call function libraries to process images. The software flow chart is shown in Figure12.Figure 12
Software module flowchart.


## 3.2. System Calibration

Figure13 shows the experimental set-up for establishing the hand-target-eye spatial attitude transformation. The RVCXmini depth camera is installed at the end of the robot arm, and the pose of the target object in the robot tool coordinate system is further obtained by using the method of joint calibration of the robot hand-eye relationship and the tool coordinate system. According to the positioning result, the robot arm drives the end tool to move to the target position pose. By comparing the pose data of the robot arm tool end with the pose data of the robot arm tool end detected by the vision system, the robustness and accuracy of the algorithm are verified.Figure 13
Experimental set-up for establishing the hand-target-eye space pose conversion relationship.First, the internal parameters of the camera are calibrated using Zhang Zhengyou’s checkerboard calibration method [29] to take 15 to 20 checkerboard target images at different positions and angles. The principal point coordinates u0,v0 of the camera focal length f and the distortion coefficients k1, k2, p1, p2 can be obtained by Zhang Zhengyou’s calibration algorithm. Thereby, the internal parameter transformation matrix Mc between the image coordinate system and the camera coordinate system is obtained.Second, carry out EyeinHand hand-eye calibration [30, 31], fix the RVCXmini depth camera to the end flange of the robot and fix the checkerboard in the working area, so that the RVCXmini depth camera can follow robot's movement. The camera takes 15 to 20 two-dimensional images of checkerboard targets and point cloud images at different positions and angles and records the posture of the robot during each shooting time so as to calibrate the hand-eye transformation matrix Mu between the camera and the end of the robot.Finally, the tool coordinate system is calibrated, that is, the transformation matrix from the tool coordinate system to the robot flange coordinate system is determined, there is no attitude change between the center point of the ultrasonic phased array probe wafer and the center point of the robot flange end, but only the translation in position, so the transformation matrixMt of the tool coordinate system is established.As shown in Figure13, Cv is define as a certain point in the pixel coordinate system, and Cc is defined as the corresponding pixel point in the camera coordinate. Establish the relationship between them Cv∗Mc=Cc, define Cu as the coordinates of the robot coordinate system, the external parameter equation between the camera coordinate system and the robot coordinate system is Cc∗Mu=Cu, define Ce as the coordinate of the center point of the ultrasonic phased array wafer, and establish the coordinate relation expression according to the tool coordinate system as Cu∗Mt=Ce. When a certain point on the image Cv is identified, Ce=Mc∗Mu∗Mt∗Cv can be deduced according to the above formula.As a result, the hand-target-eye spatial attitude transformation relation is established. According to existing research [32–34], the camera is used to measure unidirectional pose accuracy and repeatability of industrial robots. The basic principle is to use a structured light sensor to indirectly measure the absolute posture of the robot by measuring 4 fixed reference points and to compare the real posture with the command posture to test the accuracy and repeatability of the posture. By evaluating results of pose repeatability, it can be said that the repeatability of the robot at the selected area of its workspace corresponds to the repeatability value stated by the manufacturer (±0.04 mm).


## 3.3. Experimental Project

Figure14 shows the experimental device for visually inspecting and identifying the characteristics for irregular porous castings. Make sure that the features to be inspected of irregular porous castings are within the camera's field of view, scan through the RVCXmini depth camera to identify the target object of the irregular porous castings to be inspected, and obtain the pose of the target object in the camera coordinate system. Using the measurement algorithm proposed in this paper, the 4 groups of features to be inspected for irregular porous castings are repeatedly measured for 9 times, and the error range of the calculation algorithm is analyzed according to the measurement results, so as to verify the stability of the algorithm.Figure 14
Visual inspection and identification of features.Figure15 shows the experimental setup of path optimization for multifeature detection. Obtain multiple features to be detected of special-shaped porous castings through visual inspection and feature recognition, by calculating the detection path distance between multiple features, select the shortest path during the movement of the ultrasonic probe at the end of the robotic arm. Calculate the distance of detection path, select the shortest path according to different numbers of features to be detected, and then carry out the corresponding ultrasonic phased array nondestructive testing operation.Figure 15
Multifeature detection path optimization.


## 4. Results and Discussion

### 4.1. Visual Positioning Experiment

Experiments are carried out on the extraction effect of feature contour to be detected, and the effectiveness of the algorithm is verified by identifying the feature contour of the irregular porous casting under different backgrounds. Through many experiments, the recognition accuracy of the number of features to be detected within the shooting angle range is counted. Figure16(a) is the scattered point cloud data of the multitarget specimen measured by the structured light scanner, which contains 920688 points, and the detection results obtained are shown in Figure 16(b). The boundary point extraction algorithm proposed in this paper was used to extract boundary points in the scattered point cloud data. The number of extracted boundary points extracted is 37392; 9 holes features in this feature have been successfully extracted. It can be seen from Figure 16 that this method in this paper can effectively extract the hole boundary features in the scattered point cloud and meet the recognition requirements for multiple features to be inspected.Figure 16
(a) Scattered point cloud data. (b) Boundary point.Using RVCXmini depth camera to measure 9 holes with theoretical diameterd = 8 mm on multitarget specimens (as shown in Figure 16), the scattered point cloud data of specimens were obtained. The method proposed in this paper was used to process the data, and the parameters of the hole are obtained. The theoretical diameter is compared with the extraction result of this method, and the deviation of the extracted hole parameters by this method was obtained. The extraction results of the theoretical diameter and the parameters in this paper are shown in Figure 17.Figure 17
Deviation of the hole parameter extraction.When the machine vision system identifies the feature to be inspected, the central coordinates of the feature to be inspected in the image coordinate system are obtained, and the corresponding world coordinate system coordinates are solved as the calibration value. Through the forward kinematics of the robot, the robot is controlled to move to this point, and the center coordinates of the end effector at this time are obtained, which is used as the theoretical value of positioning. By comparing the deviation between the calibration value and the theoretical value of the center of the end effector in the world coordinate system, the positioning accuracy of the system can be judged. The experimental results are shown in Table1.Table 1
Positioning accuracy.
The calibration valueThe theoretical value1−223.529, −1292.96, 681.043−223.43, −1292.76, 681.042−222.737, −1371.17, 679.277−222.77, −1371.22, 679.283−226.886, −1331.39, 680.347−226.87, −1331.35, 680.374−148.652, −1375.95, 681.782−148.65, −1375.94, 681.785−153.801, −1336.63, 682.373−153.80, −1336.65, 682.356−207.535, −1388.82, 682.547−207.55, −1388.83, 682.547−233.099, −1331.19, 675.31−233.10, −1331.07, 675.428−159.567, −1340.06, 674.643−159.67, −1340.16, 674.659−161.729, −1300.75, 675.522−161.58, −1300.53, 675.52In order to verify the accuracy of the method to fit the cylindrical size, the RVCXmini depth camera was used to measure 5 theoretical cylindrical models (as shown in Figure8) on the special-shaped porous castings, and the scattered point cloud data of the test piece was obtained. The fitting results were compared with the standard data of cylinder measurement. Table 2 shows the statistical data of 5 groups of fitting data.Table 2
Fit the cylindrical size.
Calibration cylindrical modelsTheoretical cylindrical models12345x−47.5874−47.9466−47.6714−47.2678−47.3197−47.6108y28.400628.700828.618128.946628.707328.2363z434.9910434.4420434.3143434.2687434.0594434.6591dx59.450559.461459.489859.376559.862559.6430dy−158.6750−158.6250−158.8620−158.4440−158.3250−158.7892dz−985.5390−985.5480−985.1580−985.1930−985.4530−985.2741r156.735156.733156.692156.716156.697156.734From the extraction results and deviation in Table1 and Figure 17, it can be concluded that the average value of the extraction deviation of the holes on the test piece by the method proposed in this paper is 0.107 mm, and the maximum value is 0.266 mm; the average extraction deviation of hole diameter is 0.002 mm, and the maximum value is 0.02 mm. From the analysis of extraction results and extraction deviation accuracy of hole position by this method is high, which can reach 0.107; from the analysis of Table 2, it is found that the average error of fitting the cylinder is less than 0.04 mm, and the calculation accuracy of the included angle between the two axes is less than 0.4°, which meets the fitting accuracy requirements. Comparing the data obtained by the algorithm proposed in this paper with the data of other scholars, research [35] proposes a contour extraction method, in which the position accuracy of the extraction hole was 0.1 mm, and the maximum error of the circles radius was less than 0.05 mm. Research [36] proposed a cylindrical fitting method to extract the diameter at breast diameter of bamboo, and the fitting accuracy was 0.4 cm. The algorithm proposed in this paper has met the practical requirements in extracting aperture, hole location, and cylinder fitting.


### 4.2. Path Planning Experiment

The experiment of path optimization for multifeature detection is carried out. As shown in Figure9, the coordinates of the initial point O were (0, 0, 0), and the coordinates of the three feature center points A, B, and C to be inspected were (250, 200, 0), (600, 150, 0), (300, 450, 0) (unit: mm), According to the path distance calculation formula, it can be obtained that the shortest distance dmin of the 6 harvesting paths was 1127.1 mm, and the maximum distance dmax was 1210.9 mm. The shortest distance path corresponds to the first path A ⟶ B ⟶ C and the sixth path C ⟶ B ⟶ A in the decision tree, and the sequence coordinates of the shortest path were (250, 200 ,0), (600, 150, 0), (300, 450, 0). In the process of path optimization, when the number of features of the detection area is different, the photos of the detection area are grouped to get the position coordinates and optimize the path. The data are shown in Table 3.Table 3
Detection efficiency.
The number of featuresdmin (mm)dmax (mm)Efficiency (%)31127.11210.96.941933.52184.711.552274.32638.613.8The length of the detection path was used to evaluate detection efficiency. By analyzing the data in the table, the number of features to be inspected was 3, 4, and 5. Different detection trajectories are selected in each case, and the shortest path distance obtained by the path optimization algorithm was 6.9%, 11.5%, and 13.8% smaller than the longest path distance, respectively. The average value of the maximum efficiency improvement in the three cases was 10.7%. Considering that in the process of sequential detection, the actual detection path of the machine before path optimization may be any one in Figure10. Therefore, in these three cases, the detection efficiency is improved. It can be seen that when the path optimization algorithm proposed in this paper is used to sequentially detect multiple features, the moving distance of the end effector in the shortest detection path is shorter than that of other detection paths.


## 4.1. Visual Positioning Experiment

Experiments are carried out on the extraction effect of feature contour to be detected, and the effectiveness of the algorithm is verified by identifying the feature contour of the irregular porous casting under different backgrounds. Through many experiments, the recognition accuracy of the number of features to be detected within the shooting angle range is counted. Figure16(a) is the scattered point cloud data of the multitarget specimen measured by the structured light scanner, which contains 920688 points, and the detection results obtained are shown in Figure 16(b). The boundary point extraction algorithm proposed in this paper was used to extract boundary points in the scattered point cloud data. The number of extracted boundary points extracted is 37392; 9 holes features in this feature have been successfully extracted. It can be seen from Figure 16 that this method in this paper can effectively extract the hole boundary features in the scattered point cloud and meet the recognition requirements for multiple features to be inspected.Figure 16
(a) Scattered point cloud data. (b) Boundary point.Using RVCXmini depth camera to measure 9 holes with theoretical diameterd = 8 mm on multitarget specimens (as shown in Figure 16), the scattered point cloud data of specimens were obtained. The method proposed in this paper was used to process the data, and the parameters of the hole are obtained. The theoretical diameter is compared with the extraction result of this method, and the deviation of the extracted hole parameters by this method was obtained. The extraction results of the theoretical diameter and the parameters in this paper are shown in Figure 17.Figure 17
Deviation of the hole parameter extraction.When the machine vision system identifies the feature to be inspected, the central coordinates of the feature to be inspected in the image coordinate system are obtained, and the corresponding world coordinate system coordinates are solved as the calibration value. Through the forward kinematics of the robot, the robot is controlled to move to this point, and the center coordinates of the end effector at this time are obtained, which is used as the theoretical value of positioning. By comparing the deviation between the calibration value and the theoretical value of the center of the end effector in the world coordinate system, the positioning accuracy of the system can be judged. The experimental results are shown in Table1.Table 1
Positioning accuracy.
The calibration valueThe theoretical value1−223.529, −1292.96, 681.043−223.43, −1292.76, 681.042−222.737, −1371.17, 679.277−222.77, −1371.22, 679.283−226.886, −1331.39, 680.347−226.87, −1331.35, 680.374−148.652, −1375.95, 681.782−148.65, −1375.94, 681.785−153.801, −1336.63, 682.373−153.80, −1336.65, 682.356−207.535, −1388.82, 682.547−207.55, −1388.83, 682.547−233.099, −1331.19, 675.31−233.10, −1331.07, 675.428−159.567, −1340.06, 674.643−159.67, −1340.16, 674.659−161.729, −1300.75, 675.522−161.58, −1300.53, 675.52In order to verify the accuracy of the method to fit the cylindrical size, the RVCXmini depth camera was used to measure 5 theoretical cylindrical models (as shown in Figure8) on the special-shaped porous castings, and the scattered point cloud data of the test piece was obtained. The fitting results were compared with the standard data of cylinder measurement. Table 2 shows the statistical data of 5 groups of fitting data.Table 2
Fit the cylindrical size.
Calibration cylindrical modelsTheoretical cylindrical models12345x−47.5874−47.9466−47.6714−47.2678−47.3197−47.6108y28.400628.700828.618128.946628.707328.2363z434.9910434.4420434.3143434.2687434.0594434.6591dx59.450559.461459.489859.376559.862559.6430dy−158.6750−158.6250−158.8620−158.4440−158.3250−158.7892dz−985.5390−985.5480−985.1580−985.1930−985.4530−985.2741r156.735156.733156.692156.716156.697156.734From the extraction results and deviation in Table1 and Figure 17, it can be concluded that the average value of the extraction deviation of the holes on the test piece by the method proposed in this paper is 0.107 mm, and the maximum value is 0.266 mm; the average extraction deviation of hole diameter is 0.002 mm, and the maximum value is 0.02 mm. From the analysis of extraction results and extraction deviation accuracy of hole position by this method is high, which can reach 0.107; from the analysis of Table 2, it is found that the average error of fitting the cylinder is less than 0.04 mm, and the calculation accuracy of the included angle between the two axes is less than 0.4°, which meets the fitting accuracy requirements. Comparing the data obtained by the algorithm proposed in this paper with the data of other scholars, research [35] proposes a contour extraction method, in which the position accuracy of the extraction hole was 0.1 mm, and the maximum error of the circles radius was less than 0.05 mm. Research [36] proposed a cylindrical fitting method to extract the diameter at breast diameter of bamboo, and the fitting accuracy was 0.4 cm. The algorithm proposed in this paper has met the practical requirements in extracting aperture, hole location, and cylinder fitting.


## 4.2. Path Planning Experiment

The experiment of path optimization for multifeature detection is carried out. As shown in Figure9, the coordinates of the initial point O were (0, 0, 0), and the coordinates of the three feature center points A, B, and C to be inspected were (250, 200, 0), (600, 150, 0), (300, 450, 0) (unit: mm), According to the path distance calculation formula, it can be obtained that the shortest distance dmin of the 6 harvesting paths was 1127.1 mm, and the maximum distance dmax was 1210.9 mm. The shortest distance path corresponds to the first path A ⟶ B ⟶ C and the sixth path C ⟶ B ⟶ A in the decision tree, and the sequence coordinates of the shortest path were (250, 200 ,0), (600, 150, 0), (300, 450, 0). In the process of path optimization, when the number of features of the detection area is different, the photos of the detection area are grouped to get the position coordinates and optimize the path. The data are shown in Table 3.Table 3
Detection efficiency.
The number of featuresdmin (mm)dmax (mm)Efficiency (%)31127.11210.96.941933.52184.711.552274.32638.613.8The length of the detection path was used to evaluate detection efficiency. By analyzing the data in the table, the number of features to be inspected was 3, 4, and 5. Different detection trajectories are selected in each case, and the shortest path distance obtained by the path optimization algorithm was 6.9%, 11.5%, and 13.8% smaller than the longest path distance, respectively. The average value of the maximum efficiency improvement in the three cases was 10.7%. Considering that in the process of sequential detection, the actual detection path of the machine before path optimization may be any one in Figure10. Therefore, in these three cases, the detection efficiency is improved. It can be seen that when the path optimization algorithm proposed in this paper is used to sequentially detect multiple features, the moving distance of the end effector in the shortest detection path is shorter than that of other detection paths.


## 5. Conclusions

In this paper, special-shaped porous castings were taken as the research object, and a method of pore location parameter extraction based on scattered point clouds was proposed. This method was used to process the scattered point cloud data of special-shaped porous castings. The radius center search method based on distance range and scale was used to skillfully avoid the interference of noise points. Finally, the scanned cylindrical point cloud was subjected to three-dimensional visualization. So that it was convenient to intuitively reflect the state of the cylindrical model of the special-shaped porous casting. The test results show that this method can accurately extract all the hole features in the scattered point cloud, the extraction accuracy of hole position is high, the extraction accuracy can reach 0.107 mm, the average extraction deviation of hole diameter is 0.002 mm, the fitting diameter accuracy of the cylinder processed by this method is less than 0.04 mm, and the calculation accuracy of the included angle between the two axes is less than 0.4°, which meets the precision requirements of fitting cylindrical point cloud.From the point of view of coordinate transformation, the point cloud data of the robot tool end were fitted into the three-dimensional space, and the optimal path is used for detection in this process, which improves the detection efficiency and accuracy, and the measured coordinates can be transformed to the robot world coordinate system. Experiments show that it can effectively maintain real-time accuracy of the center point of the robot end tool and meet the needs of rapid calibration of the robot end tool parameters in actual industrial production. Combining with the high-precision and noncontact characteristics of structured light, the rapid, high-precision and noncontact global three-dimensional reconstruction of large-scale structures is realized, which provides a positioning method basis for the damage detection of irregular porous castings based on vision, and improves the detection efficiency by using the automatic detection of the optimal path.The research content of this article has been applied to the actual production of military ship weapons and equipment. The irregular porous castings are unprocessed and cannot be reassembled or disassembled from the machine tool. In situ automatic ultrasonic phased array inspection will be carried out. The feature extraction algorithm of this research can accurately identify the features of deep irregular porous castings, and the robot can accurately reach the location of the feature to be inspected. The path optimization algorithm can provide a shorter detection path when performing ultrasonic automatic detection and realize the moving distance of the end effector in the shortest detection path is shorter than that of other detection paths. It can directly solve the technical problems that large-scale military castings cannot be inspected effectively on-site at present, can guide the inspectors to carry out scientific inspections, reasonably evaluate the manufacturing quality and ensure its battlefield index, and has important military benefits. At the same time, it is very helpful to promote the ultrasonic automatic nondestructive testing of deep special-shaped porous castings and has great economic and social benefits.



---

*Source: 1050038-2022-02-27.xml*