<figure><img src="image_1.png" /></figure>

HAL
open science

# Three-Dimensional Reconstruction and Characteristics Computation of Corn Ears Based on Machine Vision

Jianjun Du, Xinyu Guo, Chuanyu Wang, Sheng Wu, Boxiang Xiao

► To cite this version:

Jianjun Du, Xinyu Guo, Chuanyu Wang, Sheng Wu, Boxiang Xiao. Three-Dimensional Reconstruction and Characteristics Computation of Corn Ears Based on Machine Vision. 7th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Sep 2013, Beijing, China. pp.290-300, 10.1007/978-3-642-54341-8_31. hal-01220839

HAL Id: hal-01220839

https://inria.hal.science/hal-01220839

Submitted on 27 Oct 2015

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Three-dimensional Reconstruction and
Characteristics Computation of Corn Ears
Based on Machine Vision

Jianjun Du<sup>a</sup>, Xinyu Guo<sup>b,*</sup>, Chuanyu Wang<sup>c</sup>, Sheng Wu<sup>d</sup>, Boxiang Xiao<sup>c</sup>

Beijing Research Center for Information Technology in Agriculture, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China;

<sup>a</sup>dujj@nercita.org.cn, <sup>b,*</sup>corresponding author:guoxy@nercita.org.cn,

<sup>c</sup>wangcy@nercita.org.cn, <sup>d</sup>wus@nercita.org.cn, <sup>e</sup>xiaobx@nercita.org.cn

**Abstract.** Three-dimensional shape descriptors of corn ears are important traits in corn breeding, genetic and genomics research, however it is difficult to accurately and consistently measure 3D features of corn ears by hand or traditional tools. This study presents a 3D modeling method based on machine vision to reconstruct the 3D model of corn ears for quantitative feature computation and analysis. Firstly, a simple machine vision system is designed to capture images of corn ears from different angles of view. The corn ears in these images are then registered in the uniform coordinate system using a rapid process pipeline which consists of image processing, object detection, distortion correction and registration in pixel level etc. After the registration, the point sets in edge contours and center skeletons of corn ears are used to reconstruct the surface model based on resample and interpolation techniques. The experimental results demonstrate that the presented method can not only build realistic 3D models of corn ears for visualization, also be used to accurately compute geometric characteristics.

**Keywords:** corn ear, reconstruction, modeling, machine vision

# 1 Introduction

Three-dimensional shape descriptors of corn ears are important traits for corn breeding, genetic and genomics research. However, it is difficult to measure 3D shape descriptors by manual operations or traditional tools. Machine vision (MI), which includes methods for acquiring, processing, analyzing, and understanding images, has been applied to automatic inspection, process control, and robot guidance in industry [1]. In agriculture applications, considerable research was reported using machine vision and image processing to identify features of agricultural products [2]. Hugue et al. described an approach for automatic assessment of crop and weed area in images of cereal crops using a tractor mounted camera [3]. Zhang et al. implemented fruits classification using computer vision and a multiclass support vector machine, and achieved classification accuracy of 88.2% [4]. Ni et al. designed a prototype machine vision system for inspecting corn kernels with random orientation, and respectively obtained classification rates of 91% and 94% for whole and broken kernels [5]. In
general, the detection system using machine vision only utilizes two-dimensional information of detected objects, and it is difficult to directly obtain the three-dimensional shape measurements from images.

Recently, a measurement method for 3D geometric features of corn ears based on image processing was developed [6]. Moreover, analysis models of corn ears are also established to analyze corn threshing process using the Discrete Element Method [7, 8]. The above methods assumed the corn ear as segmented truncated cones, thus the computation accuracy was limited owing to the deficiencies of the individual differences in its transverse profiles. The objective of this study is to develop a three-dimensional reconstruction method of corn ears based on machine vision for computing and visualizing the geometric characteristics of corn ears. This method mainly consists of five steps: image acquisition, image analysis (object detection, distortion correction), skeleton registration in pixel level, surface reconstruction, 3D characteristics computation and visualization.

# 2 Experiments and Methods

## 2.1 Image Acquisition

A traditional vision system is designed as a controllable device to capture consecutive images of corn ears from different profiles, as shown in Fig. 1(a). This system consists of digital camera, turn-plane, needle, stepping motor and LED light source etc. A simple turn-plane with a steel needle is used to fix the corn ear, and driven to rotate by stepping motor. At each specified rotate angle, a picture is captured by the digital camera and stored as JEPG format once the turn-plane stays stable. The size of corn ear in each image is related with the position of cameras, the distance between camera and corn ear, the offset orientation of ears and the inner parameters of the camera. It is difficult to guarantee the consistency of the rotate axis and the center axis of corn ear in the real application, therefore the corn ear in the images captured from different rotate angles will have different heights. Fig. 1(b) shows the schematic diagram of this machine vision system. We take the height of corn ear in two images with 180 degrees as $H1$ and $H3$, so the standard height of corn ear can be calculated as $H = (H1+H3)/2$. In the next step, the corn ear will be respectively extracted from serial images, and scaled to the same height according to the standard height.

The number of images is determined by the included angle between each two adjacent imaging positions. In order to accurately reconstruct the three-dimensional skeleton of corn ear, at least two orthogonal images are prerequisite in the image sets, and two images with 180 included angles are also integrant. Therein, the orthogonal images with 90 degrees will be used to determine the center axis of corn ears in three-dimensional space, and the two images with 180 degrees are used to compute the standard height of corn ear. Thus, at least four images need to be captured for each corn ear. Fig. 1(c) shows four images among which each image has included angle of 90 degrees with its adjacent images.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Image acquirement based on machine vision system</figcaption></figure>

In the image sequence, each image of corn ears only provides the local information of the entire corn ear. Due to the position, dip angle and shape differences of corn ears, the distortion of corn ears in different images manifests as inconsistencies of length, width and texture etc. To build the three-dimensional model of corn ears, these images with local information must be registered to the uniform coordinate system. In this study, the center axis of corn ear and the captured angle of images are used to determine the coordinate system of corn ears. Therefore, three continuous processes are implemented to register these images. Firstly, the valid region of corn ears in these images is extracted by threshold, contour extraction and classification techniques. The OBB of the corn ear in each image is then calculated and rotated to parallel to the (Cartesian) coordinate axes, and further zoomed to the standard height with the same proportion in length and width orientation of images. Finally, the contours of corn ears are extracted and split into three types of point sets. Combined with the captured angle of images, these point sets can be registered in the uniform coordinate system.

## 2.2 Object Extraction

The captured image generally contains corn ear and some background units, such as steel needle, turn-plane etc. In order to robustly extract the objects of interest, a simple method is used to clean the background information. A calibrated image is firstly obtained before placing corn ears, and then the subtraction result between this image and the image captured after placing corn ears will generate a cleaned image which only contains the corn ear. Furthermore, the cleaned image is converted into gray-level image for image segmentation and object extraction.

Since the image only contains the corn ear, Otsu's method [9] is used to calculate the threshold which converts the gray level image into the binary image. This method can automatically generate the optimum threshold by minimizing the intra-class variance or maximizing inter-class variance. Due to the color and shape varieties of corn ears, the islands or holes in binary images are difficult to be fully cleaned. Therefore, the contour of corn ear is more effective to represent the valid region of corn ear. Moreover, in order to obtain the smoothed edges of corn ears in binary images, iterative morphology operations with increasing radius of structure element will be used to clean the image until the image only contain the object of interest. Therein, Open operations are used to remove the islands, and Close operation to fill the holes. The maximum radius of structure element is taken as 10 using trial and error procedure to avoid the excessive smoothness. In most cases, the radius with 5 can effectively smooth the edges and fill holes in binary images. After each iterative
morphology operation, the object number is counted using the connected-component labeling algorithm [10].

The edge contour of corn ears is determined by contour extraction and classification. A simple method is also used to rapidly extract the contours in the binary image. The binary image is firstly eroded one pixel, and then subtracted by this image. Valid pixels in the resulted image are then linked into closed contours according to their connectivity. Among all the contours, the contour with maximum pixels is taken as the final contour of corn ear. This contour is further filled with uniform label value using region growing method which utilizes the contour center as the growing point. Valid regions of corn ears in binary images were represented as B<sub>11</sub>, B<sub>12</sub>, ..., B<sub>1N</sub>. Accordingly, regions of corn ears in RGB images can also be obtained according to the corresponding relation of pixel coordinates between the binary image and captured image, and expressed as S<sub>11</sub>, S<sub>12</sub>, ..., S<sub>1N</sub>.

## 2.3 Distortion Correction

As mentioned above, the distortion of corn ears in images must be firstly corrected. The rotation and scaling of corn ears are implemented to uniform the corn ear to the standard height and the same proportion. To rotated corn ears in images to parallel to the (Cartesian) coordinate axes, the oriented bounding box (OBB) is used to determine the center axis of corn ears. OBB as a two-dimensional rectangle containing the object is the most simple and appropriate shape descriptor for the valid regions of corn ears. Because of the inconsistency between the rotate axis and center axis of corn ear, the major axis of OBB always offset to the rotate axis to some extent. The OBB of corn ear will be rotated to be aligned with the axes of the Cartesian coordinate system in the image. The offset angle between the major axis of corn ear and vertical orientation can be computed according to the vertices position of OBB, and is simultaneously used to rotate the binary and color images with the same degree.

The heights of OBBs in serial images are inconsistent, so an image resample method based on a central differencing scheme in combination with a nearest neighbor interpolation [11] is used to scale OBBs to the standard height. During the distortion correction, the valid regions of corn ears are rotated and scaled for generating new serial binary images B21, B22 ... B2N and RGB images S21, S22 ... S2N. Therein, binary images are used to build the skeleton of corn ear, and RGB images to provide the rendering texture.

<figure><img src="image_4.png" /></figure>
<figure><img src="image_5.png" /></figure>

Since the reconstruction results are susceptible to the edge of input images, we test and verify the availability of edge maintain. For images of different ears, we use the above image analysis methods to obtain the contours of corn ears. Fig. 2 shows the process to extract and correct objects from a given input image. Fig. 2(a) computes the OBB of corn ear, and obtain the angle between the rotate axis and the longitude axis of OBB is 0.038(2.162 degree). Therefore, the corn ear is rotated to vertical orientation as Fig. 2(b). In Fig. 1(b), the interval angles between the center axis and rotate axis of four images with 90 intervals are respectively 2.16, -1.28, 2.15 and 3.96 degrees. These corn ears are rotated to parallel with vertical orientation, and the heights of corn ear are respectively 594, 576, 564 and 591 pixels. Corn ears are further zoomed to the standard height with scales (1.02, 0.99, 0.97 and 1.02). The pixel resolution in this vision system has been measured as 0.02887cm/pixel, therefore the maximum height difference of corn ears come up to 30 pixels (0.87cm, about 1.5 seed height). Fig. 2(d) and (e) shows the rotated and scaled corn ears. Compared with the manual measurement, the standard height of corn ear is consistent within 0.5% relative deviation. Accordingly, the contours of corn ears can accurately represent the edges of ears, and RGB region surrounded by the contours can also be used for the texture image.

## 2.4 Skeleton Registration

The surface model of corn ear originates from ear contours of serial captured images. After the distortion correction of corn ears, a registration method in pixel-level is used to reset the contour pixels in three-dimensional space. This method firstly built the three-dimensional skeleton of corn ear according to two orthogonal images (with 90 degree interval). And then all pixels in contours will be registered according to the center skeleton of corn ears.

In Fig. 2 (e), the contour pixels of corn ears are extracted and stored to an initial point set container. These pixels in the contours are firstly split into three kinds of point sets: left point sets (LPS), right point sets (RPS) and the center point sets (CPS). The split process is described as follows: For each initial point set, the two extreme points along Z axis are retrieved and respectively set as the Pzmin and Pzmax. In each pixel position in Z axis between Pzmin and Pzmax, the pixels are collected into a sub point set which are represented as Pixels (z). Therefore, two pixels with longest distance can be extracted from each Pixel (z) as the left and right contour pixels, and pushed respectively into the LPS and RPS. The center pixel is calculated according to the corresponding left and right contour pixels, and pushed into CPS. Once the above processes are executed at each position between P(zmin) and P(zmax), LPS, RPS and CPS will be filled with these feature pixels in the same number.

In each image, CPS can be taken as the two-dimensional skeleton of corn ear. Therefore, we use two orthogonal images to determine the three-dimensional skeleton of the corn ear throughout coordinate exchange. The global coordination of corn ears is defined according to two orthogonal images, and the first image is assumes in X-Z
plane (its orthogonal image locates in Y-Z plane). The origin point is set in the intersection point between the short edge and the longer center axis of OBB. The skeleton of corn ears is determined as follows: the coordinates of all pixels in the first image is defined as (X, 0, Z), and the coordinates of pixels in its corresponding orthogonal image is defined as (0, Y, Z). Therefore, the points in the three-dimensional skeleton of the ear can be represented as (X, Y, Z). The global coordination of corn ears is determined according to the OBBs and their camera angles. Each input image provides a longitudinal profile. In each valid position along Z axis, the transverse profile of corn ears consists of contour pixels from serial images, as shown in Fig. 3. The point O is assumed as the skeleton point of the corn ear. We assume images C1 and C2 as two orthogonal images. Therein, C1 has two edge points (C11, C12), and C2 is (C21, C22). The three-dimensional center of corn ears in Z coordinate is determined as follows: the X value in image C2 was modified as X value of the center point O, and the Y value in image C1 was also modified as the Y value of the center point O. Likewise, the coordination of points in the other images will be determined by the rotation angle of the images. After center point sets in two orthogonal images are brought into coincidence, point sets of the skeleton can be generated.

<figure><img src="image_6.png" /><figcaption>Fig. 3. A transverse profile of corn ears generated from serial images</figcaption></figure>

## 2.5 Surface Reconstruction

The surface contour points can be classified into longitudinal or transverse profile points. The contour points in the longitudinal profile are corresponding to the contour obtained from one image, and the points in the same transverse profile possess the same Z coordinate. Each image only provides two contour points for each transverse profile, as shown in Fig. 3. Once the number of serial images is enough to describe the edge features of corn ears in pixel-level, the elaborate three-dimensional surface model can be generated. However, it is low effective and unnecessary to capture lots of images to reconstruction the surface model of corn ears. Therefore, to fit the three-dimensional model based on several images becomes an essential method for improving the efficiency of modeling. Here, B-spline curve is used to interpolate and resample the surface points of corn ears.
The B-spline curve is typically specified in terms of n+1 control points $(P_0, P_1, ..., P_n)$ a knot vector $U = \{u_0, u_1, ..., u_m\}$, and a degree $p$, as shown in the following:

$$C(u) = \sum_{i=0}^{n} N_{i,p}(u)P_i \qquad (1)$$

Where, $N_{i,p}(u)$ is B-spline basis function of degree $p$, and $m = n + p + 1$. To wrap control points or knots can construct closed B-spline curve. The construction procedure: (1) Add a new control point $P_{n+1} = P_n$; (2) Find an appropriate knot sequence of $n+1$ knots $u_0, u_1, ..., u_n$; (3) Add $p+2$ knots and wrap around the first $p+2$ knots: $u_{n+1} = u_0, \quad u_{n+2} = u_1, \quad \dots, u_{n+p} = u_{p-1}, u_{n+p+1} = u_p, u_{n+p+2} = u_{p+1}$. (4) The closed curve with $C^{p-1}$ continuity has $n+p+2$ knots, and $C(u_0) = C(u_{n+1})$.

In each position of center skeleton of corn ear, a closed B-spline curve based on several edge points from images is generated to fit the transverse contour, and is sampled according to the assigned number of points. Since the points in the longitudinal profile with pixel-based resolution are much more than ones in the transverse profile, an equidistant resample technique is utilized to reduce the points in each longitudinal profile, as shown in Fig. 4(a) and (b). Fig. 4(c) is the surface model with 2500 triangles using resample rate 24 (i.e. take one point every 24 points) in the longitudinal profile; and Fig. 4(d) is the surface model with 5900 triangles using resample rate 10. Obviously, different resample rates of contour points will produce surface models with different detail characteristics. The point interpolation and resample techniques can regulate the number of points in each transverse and longitudinal profile. The polygonal surface model of corn ears can further be generated by linking the corresponding points between the adjacent contours.

<figure><img src="image_7.png" /></figure>
(a)

(b)

(c)

(d)

<caption>Fig. 4. Surface reconstruction of corn ears (a) Skeletons of corn ears; (b) Contour points after interpolation and resample; (c) Triangle meshes of 2500; (d) Triangle meshes of 5900</caption>

## 3 Results and Discussion

The length and width indices are the most fundamental descriptors of size. For each image, the oriented bounding box can be considered to truly represent the size of corn ear in the corresponding angle of view. However, the serial images of the same corn ear will result in different sizes which vary within a larger-scope. As described in section 1, the length and width of corn ears can be computed according to serial images. In this study, the length and width of 168 corn ears were respectively measured and computed by hands and by the presented method to verify the computation effectiveness based on machine vision. Since the measured results by hand may be little different due to subjective judgment for a given corn ear, we only check whether the computation result falls into the measuring range by hand. The comparison of results shows the presented method can accurately obtain the size (height and width) of corn ears, even though the center axis of corn ear has an obvious angle deviation with the rotation axis.

The reconstructed model of corn ears consists of sets of triangles which are collected together to form the surface of corn ear. This surface model can be used to calculate the 3D shape parameters, such as volume, surface area, shape index etc. Generally, these descriptors of corn ears are difficult to be measured by hand and tools. Although the volume can be measured using draining method, the experimental results always lead to large deviations due to surface concave-convex, water permeability, the thickness and tightness of wrappings etc. From the geometric modeling perspective, if the width and height of corn ears is enough accurate, the volume and surface area can also be accurately computed. The experiments have shown that the computed length and width can represent the true sizes of corn ears, thus other indices computed using three-dimensional model can be deemed to locate in high confidence level.

Based on three-dimensional models of corn ears, both volume and surface area are calculated using numerical integration. Other shape features of interest can also be identified and each feature is quantified by one or more indices. For example, the non-sphericity index (NSI) can express how closely the shape of the corn ear resembles a sphere [12]. The NSI is defined as:

$$NSI = 1 - (18\pi)^{\frac{1}{3}} \frac{V^{\frac{2}{3}}}{S} \qquad (2)$$

Where V is the volume and S is the surface area.

The number of resample points in the surface model is an important parameter to balance the efficiency and accuracy of three-dimensional modeling. We use the volume variance to determine the optimum resample rate. The source model in pixel-level resolutions can be taken as the standard model which provides the most accurate description for surface shape of corn ears. In different resample level, the three-dimensional model of corn ears can be generated and used to compute their surface
area, volume and NSI. The computed error between the current model and the standard model is computed as:

$$Val_{error} = \frac{Val_{standard} - Val_{current}}{Val_{standard}} \qquad (3)$$

Where, $Val_{standard}$ is the computed value (the surface area, volume or NSI) of the standard model, and $Val_{current}$ is the value of the current model in the designated resample rate. Fig. 5 shows the relationship between the number of triangles in the current model and computation errors. The volume error is the most relatively insensitive to the number change of triangles. Meanwhile, the surface area varies significantly with the decrease of triangles. For the models with 2500, 5900 and 29100 triangles, the errors of the volume, NSI and surface area are respectively (0.05%, 1.75% and 3.50%), (0.49%, 3.44% and 7.08%) and (1.93%, 3.75% and 8.56%). Generally, NSI can be taken as an effective shape descriptor of corn ears to determine the triangle number in the three-dimensional modeling. Therefore, the resample rate 10 (i.e. reserve one per ten points) can dynamically improve the modeling efficiency, and the shape error is controlled within 5%.

<figure><img src="image_8.png" /><figcaption>Fig. 5. The error analysis of surface area, volume and NSI</figcaption></figure>

There-dimensional models of corn ears can not only be used for the shape characteristic calculation, but for visualization analysis. Three-dimensional models of six corn ears are respectively reconstructed using the presented method, and then serial two-dimensional images are projected onto the surfaces of three-dimensional models to obtain visualization results, as shown in Fig. 6.
<figure><img src="image_9.png" /><figcaption>Fig. 6. Three-dimensional visualization of corn ears</figcaption></figure>

## 4 Conclusions

To measure the surface, volume and shape descriptors of corn ears is very difficult by hand or traditional tools, therefore we presented an automatic reconstruction method of corn ears based on machine vision for features computation and visualization. This method can automatically build the three-dimensional models of corn ears based on serial image of corn ears, thus it is suitable for breeders to study the geometrical, morphological and visual symptoms characteristics in quantitative and qualitative analysis. A successive processing pipeline has been developed to extract the effective corn regions, adjust the imaging distortion to unify the region sizes, classify the pixels in contours and skeletons of corn ears into different point sets, and register all points in pixel-level to the uniform coordinate system. The advantages of the proposed method contain: a simple system based on machine vision is designed to automatically capture image sequences of corn ears from different angles of view; the distortion correction of valid regions provides more precise computation results of sizes and 3D shape descriptors; and the registration and resample in pixel level can accurately reconstruct three-dimensional corn ears and improve the modeling efficiency.

## Acknowledgment

The research work was supported by Special Fund for Agro-scientific Research in the Public Interest under Grant No. 201203026, National Science & Technology Pillar Program under Grant No. 2012BAD35B01, and Special Fund for S&T Innovation of Beijing Academy of Agriculture and Forestry Sciences under Grant No. KJCX201204007.
References

1. Linda G. Shapiro, George C. Stockman. Computer Vision [M]. Prentice Hall, 2001.

2. Costa C., Antonucci F., Pallottino F. et al. Shape Analysis of Agricultural Products: a Review of Recent Research Advances and Potential Application to Computer Vision [J]. Food and Bioprocess Technology, 2011, 4(5): 673-692.

3. Hague T., Tillett N. D., Wheeler H. Automated Crop and Weed Monitoring in Widely Spaced Cereals [J]. Precision Agriculture, 2006, 7(1): 21-32.

4. Zhang, Y., Wu, L. Classification of Fruits Using Computer Vision and a Multiclass Support Vector Machine [J]. Sensors (Basel), 2012, 12(9):12489-505.

5. Ni B., Paulsen M. R., Liao K. et al. Design of an Automated Corn Kernel Inspection System for Machine Vision [J]. Transactions of the ASAE, 1997, 40(2):401-497.

6. Ma Q., Jiang J. T., Zhu D. H. et al. Rapid Measurement for 3D Geometric Features of Maize Ear based on Image Processing [J]. Transactions of the Chinese Society of Agricultural Engineering, 2012, 28(supp.2): 208-212.

7. Yu Y. J., Zhou H. L., Fu H. et al., Modeling Method of Corn Ears based on Particles Agglomerate [J], Transactions of the Chinese Society of Agricultural Engineering, 2012, 28(8): 167-174.

8. Fu H., Lv Y., Li Y. S. et al. Analysis of Corn Threshing Process based DEM [J]. Journal of Jilin University (Engineering and Technology Edition), 2012, 42(4): 997-1002.

9. Otsu N., A Threshold Selection Method from Gray-Level Histograms [J]. IEEE Transactions on Systems, Man and Cybernetics, 1979, 9(1): 62-66.

10. Wu K. S., Otoo E., Suzuki K., Optimizing Two-Pass Connected-Component Labeling Algorithms [J]. Pattern Analysis and Applications, 2009, 12(2): 117-135.

11. Parker J., Kenyon R. V., Troxel D. E. Comparison of Interpolating Methods for Image Resampling [J]. IEEE Trans Med Imaging, 1983, 2(1): 31-39.

12. Lauric A., Miller E. L., Baharogluet M. I. et al. 3D Shape Analysis of Intracranial Aneurysms Using the Writhe Number as a Discriminant for Rupture [J]. Annals of Biomedical Engineering, 2011, 39(5): 1457-1469.