# Online Measuring and Size Sorting for Perillae Based on Machine Vision

Authors: Bo Zhao; Ye Wang; Jun Fu; Rongqiang Zhao; Yashuo Li; Xin Dong; Chengxu Lv; Hanlu Jiang
Journal: Journal of Sensors
 (2020)

---

## Abstract

Perillae has attracted an increasing interest of study due to its wide usage for medicine and food. Estimating quality and maturity of a perillae requires the information with respect to its size. At present, measuring and sorting the size of perillae mainly depend on manual work, which is limited by low efficiency and unsatisfied accuracy. To address this issue, in this study, we develop an approach based on the machine vision (MV) technique for online measuring and size sorting. The geometrical model and the corresponding mathematical model are built for perillae and imaging, respectively. Based on the built models, the measuring and size sorting method is proposed, including image binarization, key point determination, information matching, and parameter estimation. Experimental results demonstrate that the average time consumption for a captured image, the average measuring error, the variance of measuring error, and the overall sorting accuracy are 204.175 ms, 1.48 mm, 0.07 mm, and 93%, respectively, implying the feasibility and satisfied accuracy of the proposed approach.


---

## Body

## 1. Introduction

Perillae is widely distributed throughout Asia, and it has attracted an increasing interest in the field of medicine and pharmacy. Perillae has been applied for bacteriostatic, detoxifying, antitussive, and phlegm for thousands of years in traditional Chinese medicine. Recently, more attentions have been paid to the study with respect to perillae, including the perillae herba ethanolic [1], the luteolin [2], and the rosmarinic acid [3]. Besides, perillae is also a popular food in countries such as China, Japan, and Korea. The size of perillae indicates its quality and maturity that closely relates to its medical and edible value. Hence, it is of great significance to measure and sort the size of perillae in advance. At present, measuring and sorting the size of perillae mainly depend on manual work, which is of low efficiency and unsatisfied accuracy. Therefore, it is valuable to develop an approach for accurate online measurement and sort of perillae.Machine vision (MV) is the technique to provide imaging-based automatic inspection [4–6], and it has been widely used in the fields of industry, such as object detection and robot guidance [7, 8]. For the applications in agriculture, previous work has explored into various crops [9, 10]. In [11], a compact machine vision system based on hyperspectral imaging and machine learning is presented to detect aflatoxin in chili pepper. In [12], a hierarchical grading method is applied for real-time defect detection and size sorting of potatoes. In [13], a study is conducted to predict ripening quality in mangoes using RGB images, for which the hierarchical clustering method is employed to classify the ripening period into five stages based on quality parameters. In [14], the authors combine the MV and the support vector machine to develop an intelligent system for sorting of peeled pistachio kernels and shells. In [15], Sabliov et al. develop an MV-based method to measure volume and surface area of ellipsoidal agricultural products by regarding the objects as the sum of superimposed elementary frustums of right circular cones. In [16], Yao et al. develop real-time detection instrumentation for aflatoxin-contaminated corn using a narrow-band fluorescence index. In [17], Pedreschi et al. present an inexpensive computer vision system for measuring the color of a highly heterogeneous food material such as potato chips. In [18], Huang et al. propose an approach for identification of defect pleurotus geesteranus based on computer vision. In [19], Sun et al. develop a machine vision system and the dynamic weighing system for the measurement of egg external physical characteristic and weight, such that the nondestructive method for online estimation of egg freshness is achieved. Additionally, a set of studies have been conducted on measurement, cabbage [20], flower mushroom [21], cherry [22], litchi [23], and other agricultural products [24–29].However, the automatic online detection of perillae has not been explored till now. The geometry model and the corresponding mathematical model of perillae used for automatic detection have not been built yet. The size sorting of perillae still requires manual work. To address this issue, in this study, we develop a novel approach for online measuring and size sorting of perillaes. A charge-coupled device (CCD) camera is employed to acquire the images of perillaes in the lighting box, and the MV method is proposed for image processing. The contributions of this study can be summarized as follows.(1)
We build the general geometry model of perillae and the corresponding mathematical model for image processing
(2)
We propose the MV-based method for accurate online measuring and size sorting of perillae under the proposed models
(3)
We develop the practical system for our theoretical method, the feasibility and accuracy of which are verified by the experimental resultsThe remainder of this paper is organized as follows. Section2 introduces the proposed geometry model and mathematical model. Section 3 provides the MV-based online measuring method. Section 4 presents the experimental results and analysis. Section 5 concludes this paper.


## 2. Geometry and Mathematical Models

### 2.1. Geometry Model

Figure1 shows a typical perillae for medical and food usage. The notation Lu is used to denote the length of a perillae, i.e., the maximum size of perillae leaf. We mainly consider the length between PA and PB as it is the useful part of perillae, regardless of the medical or food usage. The length between PB and PC, i.e., the length of perillae, is not taken into the measurement.Figure 1
Geometry model of perillae wherePA, PB, and PC represent the apex of perillae, the separation point of leaf and petiole, and the endpoint of petiole, respectively.Figure2 presents the sorting principle for the size of perillae. In this study, the sizes are divided into four grades according to Lu. As shown in Figure 2, the first grade is expressed as “Spec M” with the size of 7cm≤Lu<8.5cm, denoting the smallest perillae. The second grade, “Spec L(s),” and the third grade, “Spec L(b),” are determined by 8.5cm≤Lu<9.5cm and 9.5cm≤Lu<10.5cm, respectively. The fourth grade, expressed as “Spec 2M,” is used to represent the largest size, given by 10.5cm≤Lu<12cm.Figure 2
Classification of perillae sizes.


### 2.2. Mathematical Model

Next, we provide the mathematical model for imaging of perillae. The camera used in this study was calibrated before it is employed for imaging. As shown in Figure3, we captured 20 chessboard images with different poses and then utilized the calibration tool in the OpenCV to calibrate the camera.Figure 3
Classification of perillae sizes.Letu,v, and w denote the horizontal coordinate, the vertical coordinate, and the position perpendicular to the chessboard in the camera coordinate, respectively, and let x,y, and z denote the corresponding coordinates in the world coordinate. The calibration results demonstrate the distortion error, and the reprojection error can be ignored. Assume Mx,y,z is a pixel in the world coordinate system, and this pixel is then projected into the camera coordinate system, expressed as mu,v. Let G be the intrinsic matrix of the used camera. Based on the used calibration tool, the transformation between the world coordinate system and the camera coordinate system can be expressed as
(1)uvw=Gxyz.The matrixG is obtained through the calibration, and the result is
(2)G=1αx01βx01αy1βy001=11345.53801606.253011345.5381470.283001.By combining the geometrical model and (2), we can rewrite the size of perillae as follows:
(3)Lu=pΑ,x−pΒ,x2+pΑ,y−pΒ,y2=qA,x−qB,xαx+qA,y−qB,yαy2,where pA,x, pA,y, pB,x, and pB,y are the coordinates of points PA and PB in the world coordinate system and qA,x, qA,y, qB,x, and qB,y are the coordinates of points PA and PB in the image coordinate system. Obviously, the required size can be calculated out directly if parameter z and the coordinates of PA and PB are known. Hence, we introduce the proposed method to determine this information in the next section.


## 2.1. Geometry Model

Figure1 shows a typical perillae for medical and food usage. The notation Lu is used to denote the length of a perillae, i.e., the maximum size of perillae leaf. We mainly consider the length between PA and PB as it is the useful part of perillae, regardless of the medical or food usage. The length between PB and PC, i.e., the length of perillae, is not taken into the measurement.Figure 1
Geometry model of perillae wherePA, PB, and PC represent the apex of perillae, the separation point of leaf and petiole, and the endpoint of petiole, respectively.Figure2 presents the sorting principle for the size of perillae. In this study, the sizes are divided into four grades according to Lu. As shown in Figure 2, the first grade is expressed as “Spec M” with the size of 7cm≤Lu<8.5cm, denoting the smallest perillae. The second grade, “Spec L(s),” and the third grade, “Spec L(b),” are determined by 8.5cm≤Lu<9.5cm and 9.5cm≤Lu<10.5cm, respectively. The fourth grade, expressed as “Spec 2M,” is used to represent the largest size, given by 10.5cm≤Lu<12cm.Figure 2
Classification of perillae sizes.


## 2.2. Mathematical Model

Next, we provide the mathematical model for imaging of perillae. The camera used in this study was calibrated before it is employed for imaging. As shown in Figure3, we captured 20 chessboard images with different poses and then utilized the calibration tool in the OpenCV to calibrate the camera.Figure 3
Classification of perillae sizes.Letu,v, and w denote the horizontal coordinate, the vertical coordinate, and the position perpendicular to the chessboard in the camera coordinate, respectively, and let x,y, and z denote the corresponding coordinates in the world coordinate. The calibration results demonstrate the distortion error, and the reprojection error can be ignored. Assume Mx,y,z is a pixel in the world coordinate system, and this pixel is then projected into the camera coordinate system, expressed as mu,v. Let G be the intrinsic matrix of the used camera. Based on the used calibration tool, the transformation between the world coordinate system and the camera coordinate system can be expressed as
(1)uvw=Gxyz.The matrixG is obtained through the calibration, and the result is
(2)G=1αx01βx01αy1βy001=11345.53801606.253011345.5381470.283001.By combining the geometrical model and (2), we can rewrite the size of perillae as follows:
(3)Lu=pΑ,x−pΒ,x2+pΑ,y−pΒ,y2=qA,x−qB,xαx+qA,y−qB,yαy2,where pA,x, pA,y, pB,x, and pB,y are the coordinates of points PA and PB in the world coordinate system and qA,x, qA,y, qB,x, and qB,y are the coordinates of points PA and PB in the image coordinate system. Obviously, the required size can be calculated out directly if parameter z and the coordinates of PA and PB are known. Hence, we introduce the proposed method to determine this information in the next section.


## 3. Method for Measuring Perillae

In this section, we provide the proposed method for measuring and size sorting of perillae based on the MV and the proposed models. The method is composed of the following steps, binarizing the original image, determining the positions ofPA and PB, matching information, and estimating parameter z.


### 3.1. Image Binarization

Figure4(a) presents the original image of perillaes. The image should be binarized before it is used for measuring. The binarization principle is given by
(4)jx,y=255,380≤y≤690,hb<180,hr<220,0,otherwise,where jx,y is the value of the pixel x,y in the binarized image. The depth of the image is 8 bits, and therefore, the value of white pixels is set to 255. As indicated in (4), a pixel is determined to be white as long as 380≤y≤690, hb<180, and hr<220, where hb and hr are the values of the pixel in the blue channel and the red channel, respectively. The first condition, 380≤y≤690, ensures that the background is set to be black, and only the area containing perillaes is considered. The second and the third conditions are utilized to determine whether a pixel belongs to the area of perillae. The binarized image is shown as Figure 4(b).Figure 4
Image binarization.
(a)
(b)


### 3.2. Search forPA

We first derive the contour from the binarized image. The strategy is to set the thresholds for the largest and the smallest areas of the perillaes. Thus, the areas that satisfy the constraint of thresholds are selected, and their contours are regarded as the contours of perillaes. This process can be expressed as(5)cjri=1,6000<Ai<30000,0,otherwise,1≤i≤Nwhere N denotes the total number of contours in the binarized image, Ai denotes the area of the ith contours, and cjri is the judging result. Then, the minimum enclosing circle and the convex hull of each perillae can be obtained. These perillaes are labelled as 1, 2, 3, and 4 from left to right based on the following judgement:
(6)pjrj=1,1≤cj,x≤320,2,321≤cj,x≤640,3,641≤cj,x≤960,4,961≤cj,x≤1280,1≤j≤Num1,where pjrj is the judgement for each position of perillae, cj,x denotes the horizontal coordinate of the center of the jth enclosing circle, and “Num1” denotes the amount of perillae in the captured image. The results are displayed in Figure 5(b) and are saved as “Pos1.”Figure 5
Determination ofPA.
(a)
(b)Finally, we utilize the following strategy to find the endpoints of thelth contours, written as p11,l and p12,l:
(7)p11,l=argmincpl,kabscpl,k,x−ccl,x2+cpl,k,y−ccl,y2−rl,1≤l≤Num1,1≤k≤ml,(8)p12,l=argmincpl,kcpl,k,x−p11,l,x2+cpl,k,y−p11,l,y2,1≤l≤Num1,1≤k≤ml−1,where cpl,k denotes the kth pixel in the lth convex hull, cpl,k,x and cpl,k,y are the coordinates of cpl,k, ccl,x and ccl,y are the coordinates of the center of the lth enclosing circle, rl represents the radius of the lth enclosing circle, and ml is the amount of pixels of the lth convex hull. In fact, p11,l can be regarded as the point furthest away from the center of the enclosing circle, and p12,l is the point furthest away from p11,l. These two points are generally PA and PC. The data of “Num1,” “Pos1,” p11, and p12 are saved as “Info1.”


### 3.3. Search forPB

To acquire the position ofPB, we conduct one close operation, four erode operations, and four dilate operations in the OpenCV for the binarized image, successively. The results are presented in Figure 6(a). Then, we derive the treated contours using the strategy given by (7) and (8). The results are shown in Figure 6(b) and are saved as “Pos2.” Finally, the minimum enclosing circle and the convex hull of the derived contours are obtained, shown as Figure 6(c). Similar to the procedure to search for PA and PC, two endpoints, p21 and p22, are selected as the alternative points of PB. The other one is defined as PD. The data of “Pos2,” p21, and p22 are saved into “Info2.”Figure 6
Determination ofPB.
(a)
(b)
(c)


### 3.4. Information Matching

Next, we should recognizePA and PB from p11, p12, p21, and p22. For this purpose, we first compute the Euclidean distance from p21 to p11, p21 to p12, p22 to p11, and p22 to p12, respectively. Then, we compare the four distances and find out the two points that lead to the largest distance. Assuming the two points are p12 and p22, we can draw the conclusion that PC is p12, as p12 is the alternative point of PA and PC, and the distance from PC to PD is always larger than that of PA to PB. Repeat the procedure for all “Num1” contours of perillae, such that the position of PA and PB of each perillae can be obtained. We summarize all four possibilities in Table 1.Table 1
Information matching,PA and PB to p11, p12, p21, and p22.
Points leading to the largest distance
PA
PB
p11 and p21
p12
p22
p11 and p22
p12
p21
p12 and p21
p11
p22
p12 and p22
p11
p21


### 3.5. Scale Factor Estimation and Size Computation

As indicated in (8), parameter z is the factor controlling the scale of imaging. Hence, we utilize the following scale method to estimate z. As shown in Figure 7, the original plane is z=c2. We use A2B2 to represent the manual measurement and use A0B0 to represent the corresponding measurement in the camera coordinate system. Then, we manually set z to c1 to acquire the measurement A1B1 by using (8). Based on these measurements, we have
(9)A0B0A1B1=fc1,(10)A0B0A2B2=fc2.Figure 7
Pinhole imaging model.Therefore,z can be calculated by
(11)z=c2=A2B2A1B1.To eliminate the measurement error, we repeat the above process for 20 times, and the average value is used for the experiments, computed by(12)z=c2=∑i=120c2i20,where c2i denotes the calculated value in the ith trial.The proposed approach is summarized in Figure8.Figure 8
Outline of the proposed approach.


## 3.1. Image Binarization

Figure4(a) presents the original image of perillaes. The image should be binarized before it is used for measuring. The binarization principle is given by
(4)jx,y=255,380≤y≤690,hb<180,hr<220,0,otherwise,where jx,y is the value of the pixel x,y in the binarized image. The depth of the image is 8 bits, and therefore, the value of white pixels is set to 255. As indicated in (4), a pixel is determined to be white as long as 380≤y≤690, hb<180, and hr<220, where hb and hr are the values of the pixel in the blue channel and the red channel, respectively. The first condition, 380≤y≤690, ensures that the background is set to be black, and only the area containing perillaes is considered. The second and the third conditions are utilized to determine whether a pixel belongs to the area of perillae. The binarized image is shown as Figure 4(b).Figure 4
Image binarization.
(a)
(b)


## 3.2. Search forPA

We first derive the contour from the binarized image. The strategy is to set the thresholds for the largest and the smallest areas of the perillaes. Thus, the areas that satisfy the constraint of thresholds are selected, and their contours are regarded as the contours of perillaes. This process can be expressed as(5)cjri=1,6000<Ai<30000,0,otherwise,1≤i≤Nwhere N denotes the total number of contours in the binarized image, Ai denotes the area of the ith contours, and cjri is the judging result. Then, the minimum enclosing circle and the convex hull of each perillae can be obtained. These perillaes are labelled as 1, 2, 3, and 4 from left to right based on the following judgement:
(6)pjrj=1,1≤cj,x≤320,2,321≤cj,x≤640,3,641≤cj,x≤960,4,961≤cj,x≤1280,1≤j≤Num1,where pjrj is the judgement for each position of perillae, cj,x denotes the horizontal coordinate of the center of the jth enclosing circle, and “Num1” denotes the amount of perillae in the captured image. The results are displayed in Figure 5(b) and are saved as “Pos1.”Figure 5
Determination ofPA.
(a)
(b)Finally, we utilize the following strategy to find the endpoints of thelth contours, written as p11,l and p12,l:
(7)p11,l=argmincpl,kabscpl,k,x−ccl,x2+cpl,k,y−ccl,y2−rl,1≤l≤Num1,1≤k≤ml,(8)p12,l=argmincpl,kcpl,k,x−p11,l,x2+cpl,k,y−p11,l,y2,1≤l≤Num1,1≤k≤ml−1,where cpl,k denotes the kth pixel in the lth convex hull, cpl,k,x and cpl,k,y are the coordinates of cpl,k, ccl,x and ccl,y are the coordinates of the center of the lth enclosing circle, rl represents the radius of the lth enclosing circle, and ml is the amount of pixels of the lth convex hull. In fact, p11,l can be regarded as the point furthest away from the center of the enclosing circle, and p12,l is the point furthest away from p11,l. These two points are generally PA and PC. The data of “Num1,” “Pos1,” p11, and p12 are saved as “Info1.”


## 3.3. Search forPB

To acquire the position ofPB, we conduct one close operation, four erode operations, and four dilate operations in the OpenCV for the binarized image, successively. The results are presented in Figure 6(a). Then, we derive the treated contours using the strategy given by (7) and (8). The results are shown in Figure 6(b) and are saved as “Pos2.” Finally, the minimum enclosing circle and the convex hull of the derived contours are obtained, shown as Figure 6(c). Similar to the procedure to search for PA and PC, two endpoints, p21 and p22, are selected as the alternative points of PB. The other one is defined as PD. The data of “Pos2,” p21, and p22 are saved into “Info2.”Figure 6
Determination ofPB.
(a)
(b)
(c)


## 3.4. Information Matching

Next, we should recognizePA and PB from p11, p12, p21, and p22. For this purpose, we first compute the Euclidean distance from p21 to p11, p21 to p12, p22 to p11, and p22 to p12, respectively. Then, we compare the four distances and find out the two points that lead to the largest distance. Assuming the two points are p12 and p22, we can draw the conclusion that PC is p12, as p12 is the alternative point of PA and PC, and the distance from PC to PD is always larger than that of PA to PB. Repeat the procedure for all “Num1” contours of perillae, such that the position of PA and PB of each perillae can be obtained. We summarize all four possibilities in Table 1.Table 1
Information matching,PA and PB to p11, p12, p21, and p22.
Points leading to the largest distance
PA
PB
p11 and p21
p12
p22
p11 and p22
p12
p21
p12 and p21
p11
p22
p12 and p22
p11
p21


## 3.5. Scale Factor Estimation and Size Computation

As indicated in (8), parameter z is the factor controlling the scale of imaging. Hence, we utilize the following scale method to estimate z. As shown in Figure 7, the original plane is z=c2. We use A2B2 to represent the manual measurement and use A0B0 to represent the corresponding measurement in the camera coordinate system. Then, we manually set z to c1 to acquire the measurement A1B1 by using (8). Based on these measurements, we have
(9)A0B0A1B1=fc1,(10)A0B0A2B2=fc2.Figure 7
Pinhole imaging model.Therefore,z can be calculated by
(11)z=c2=A2B2A1B1.To eliminate the measurement error, we repeat the above process for 20 times, and the average value is used for the experiments, computed by(12)z=c2=∑i=120c2i20,where c2i denotes the calculated value in the ith trial.The proposed approach is summarized in Figure8.Figure 8
Outline of the proposed approach.


## 4. Experimental Results

Our experimental system is shown in Figure9, composed of a lighting box and a personal computer (PC). The lighting box contains four light-emitting diode (LED) light located as a ring on the top of the box and a CCD camera in the center of the box. The experimental setup is described as follows. Before conducting the experiments, we manually set c1=50cm and obtain z=68.765cm by using the strategy introduced in Subsection 3.5. This value of z is used for all experiments in this study. First, we randomly chose 100 perillaes and manually measured their sizes. Then, we used the developed system to measure these perillaes.Figure 9
Experimental system.The results by both manual measurement and automatic online measurement are presented in Figure10. We computed the measurement error for each perillae, and the results are provided in Figure 11. Obviously, the results obtained by online measurement are close to those obtained by manual measurement. As provided in Table 2, among the 100 perillaes, the maximum measuring error (MAME) is 3.66 mm, while the minimum measuring error (MIME) is 0.02 mm. Most of the errors are lower than 3 mm. The overall average measuring error (OAME) and the variance of measuring error (VME) are 1.47 mm and 0.07 mm, respectively.Figure 10
Measuring result.Figure 11
Measuring error for each perillae.Table 2
Overall measuring and sorting results.
ATCCI (ms)
MAME (mm)
MIME (mm)
OAME (mm)
VME (mm)
OSA (%)
Results
204.175
3.66
0.02
1.47
0.07
93


## 5. Conclusions

We developed an MV-based approach for automatic measuring and size sorting of perillae. We first built the geometric model for perillae and the mathematical model for imaging. Based on the models, the measuring and size sorting method was proposed, including image binarization, key point determination, information matching, and parameter estimation. We employed the CCD camera for imaging and the OpenCV tools for image processing. Experimental results have verified the feasibility of our system and its high accuracy of measuring and size sorting. By using 100 perillaes for experiments, the MAME and the MIME are 3.66 mm and 0.02 mm, respectively. Most of the errors are lower than 3 mm. The OAME and the VME are 1.47 mm and 0.07 mm, respectively. Further study could address on the following issues. First, miniaturization for the developed system requires future work to make the proposed approach more particle. Second, utilizing the RGB image or hyperspectral image to detect the maturity of perillaes is also valuable for further study.



---

*Source: 3125708-2020-01-13.xml*