# Brown Rice Germ Integrity Identification Based on Deep Learning Network

Authors: Shuofeng Li; Bing Li; Jin Li; Bin Liu
Journal: Journal of Food Quality
 (2022)

---

## Abstract

For the quality inspection of brown rice, the segmentation of connected brown rice and the identification of germ integrity are very important. However, there is no better traditional algorithm to achieve better segmentation and recognition results. This paper improves the brown rice (BR) segmentation algorithm based on background skeleton. The candidate matching points are obtained by the background skeleton method, and the optimal matching points are found by the ant colony algorithm. Experimental results show that the proposed segmentation algorithm achieves 96% accuracy, indicating that it can effectively suppress the interference from the endosperm surface. After segmentation is complete, identification of embryo integrity is performed. Firstly, a convolutional neural network (CNN) is built to identify the germ direction; then, the germ direction is normalized; finally, an improved Inception-v3 network is built to identify the germ integrity. On the basis of the Inception-v3 network, additional branches are added to improve the detection accuracy of small objects. In addition, mutual-channel loss and mlpconv are added to enable the model to better approximate the abstraction of the latent space. The experimental results show that the comprehensive recognition accuracy of the proposed algorithm is as high as 94.83%, which is significantly higher than the current mainstream recognition algorithms.


---

## Body

## 1. Introduction

Brown rice (BR), which is high in nutrients, is currently very popular; as a result, ensuring the quality of BR production is critical [1]. During production and processing, BR germ integrity is primarily manually and subjectively determined, which is a low-efficiency process, and the BR germ integrity cannot be quantitatively analyzed [2]. The BR quality is determined by how well the germ portion can be preserved. The amount of germ preserved in each BR grain has a direct impact on BR processing quality. The less the germ is preserved, the greater the loss of nutrients in BR is [3]. Therefore, it is necessary to not only preserve the germ of BR but also maximally increase the preserved extent of the germ of each BR grain during processing. Thus, it is necessary to examine the BR germ integrity during processing.Machine vision is being increasingly applied in agriculture. An increasing number of studies have focused on the use of machine vision to examine rice quality. Machine vision can improve rice quality testing efficiency while also overcoming the effects of subjective factors, reducing test errors, and increasing test speed and accuracy. In recent years, Courtois et al. [4] proposed the use of image analysis techniques to assess the breakage of and cracks in parboiled rice grains. Lin et al. [5] proposed a method for splitting touching rice grains based on their contour curvature. Using a support vector machine, Sun et al. [6] evaluated and analyzed the chalkiness of connected rice grains. Payman et al. [7] developed a rice quality index test system based on expert vision.In this study, BR germ integrity is comprehensively examined by image segmentation and identification. Wang et al. [8] improved the watershed algorithm to address the problem of oversegmentation of targets in image segmentation involving small connected targets. Jia et al. [9] inhibited the effects of noise on segmentation by bitmap cutting and region merging. Some researchers used various edge detection operators in the segmentation and extraction of images with targets and backgrounds that have significantly different features [10–13]. In a fully convolutional network (FCN), fully connected layers (in a conventional network) are treated as convolutional layers, and deconvolutional layers are connected. An FCN can generate a pixel segmentation map by upsampling [14, 15].Another method of detection is to use a target detection algorithm to directly identify and classify the targets in the original image, rather than using an image segmentation algorithm to extract single targets. Progress has been made since Krizhevsky et al. [16] proposed AlexNet [17]. At present, in computer vision tasks, convolutional neural networks are used to segment and classify targets [18–20]. In 2012, a major improvement was made to ImageNet [21–23]. Various deep learning methods, represented by convolutional neural networks (CNNs), have been used extensively well in terms of numerous vision tasks, including object detection. The single shot multibox detector (SSD) [23–25] performs exceedingly well in terms of detection accuracy and speed. In computer vision tasks, the use of multiresolution processing data will improve the overall level of image segmentation and classification [26, 27]. The SSD was proposed in 2016 to address the problem of you-only-look-once (YOLO) algorithms in target positioning having insufficient detection accuracy [28–30]. The main idea of the SSD is the dense and uniform sampling of an image at various locations. The SSD was established on the concept of anchor points in faster region-CNNs (R-CNNs) [31, 32]. In the SSD, various prior frames are used to predict the proportion and length-to-width ratio (LWR) of the boundary frame of an object during sampling. The features are then extracted, classified, and regressed using a CNN. There is only one step in the entire procedure. In order to detect targets, the SSD employs a pyramidical feature layer group. The shallow feature map has a smaller receptive field, so it can predict a small target, while the deep feature map has a larger receptive field, so it can predict a large target. In the SSD, however, feature layers of various scales are unrelated to one another, and feature complementarity between feature layers is lacking. As a result, the SSD is relatively inefficient in detecting general and small targets. In August 2020, Li et al. [33] investigated a deep learning based algorithm for marine target detection and segmentation with a 32.7% improvement in detection accuracy. In July 2021, Krishnamoorthy et al. [34] used the Inception ResNetV2 model to identify infectious organisms in rice plant leaves with a detection accuracy of 95.6%. In August 2021, Yang et al. [35] proposed a temporal feature-based segmentation (TFBS) model for accurate crop mapping using time-series SAR images. In October 2021, Lan et al. [36] proposed two improved recognition models MobileNetV2-UNet and FFB- BiSeNetV2 for the identification of rice weeds with an accuracy of 93%. In the same month, Wang et al. [37] proposed a small target detection algorithm based on Mask R-CNN model with a detection accuracy of 66.5%. In December 2021, Guo et al. [38] proposed an image segmentation and deep learning based method to automatically identify rice grains and calculate rice fructification rate with good results. In the same month, Wang and Qiu [39] used an unsupervised classification method for fast localization of small targets to improve detection efficiency.In the process of processing, the quality testing of brown rice is of great significance for retaining nutrients and reducing food waste. Among them, the segmentation of connected BR and the detection of germ integrity are extremely important. However, the above algorithms are unable to effectively identify all the rice grains in an image and have difficulties with the connected rice grains in particular.The main contributions of this study are summarized as follows:(1)
The BR germ integrity is classified by image segmentation combined with image identification. Single rice grains are extracted from connected rice grains using the proposed segmentation algorithm. Then, the BR germ integrity is classified based on an improved Inception-v3 model.(2)
The background skeleton-based segmentation algorithm is improved [40]. The region growing algorithm and ant colony algorithm [41–44] are integrated into reference-point matching to improve the matching accuracy for skeleton endpoints.(3)
Two neural network (NN) models are used to identify BR germ integrity. First, the visual geometry group (VGG) model is used to identify and classify the directions of the BR germs. Then, the Inception-v3 model is structurally improved to increase the identification accuracy.


## 2. Proposed Work

### 2.1. Preprocessing

There are two steps to preprocessing a BR image. (1) After binarization, small, scattered white areas are often generated in an image due to the miscellaneous items on the platforms where the husks, bran, and rice are placed during production and processing as well as noise from some image processing procedures. These small areas may subsequently affect the identification of the effective connected areas and thus need to be eliminated. (2) The training set is image preprocessed to enhance the features of the germ portion in the images in order to improve subsequent training performance for the germ integrity of single rice grains.


#### 2.1.1. Image Preprocessing

In the first step of image preprocessing, conventional spatial and frequency domain filters cannot simply and effectively eliminate the small areas, whereas a morphological filter can easily eliminate them. To eliminate the small areas, a structural element with a suitable dimension can be used to erode the original image. After erosion, these small areas can be eliminated. An erosion operation, on the other hand, enlarges the voids and separates the previously connected regions. A rice area can also be reduced by an erosion operation. To eliminate the negative effects of an erosion operation, a dilation operation can be subsequently performed. The combination consisting of first an erosion operation and a subsequent dilation operation is referred to as an opening operation. The “。” in the following equation is the operator of an opening operation and signifies that a structural element is first used to erode the image and then dilate the results of the erosion operation. We have(1)X∘B=X⊖B⊕B.A morphological filter is used to remove the small white area in the following left image. A 5 × 5 square is used as a structural element. The comparison image shows that the small white area is removed. The results are shown in Figure1.Figure 1
Comparison of images before and after morphological filtering. (a) The original image. (b) The image processed by morphological filter.
(a)(b)


#### 2.1.2. Image Segmentation Preprocessing

In the second step of image segmentation preprocessing, the training set is subjected to top-hat transformation to enhance the germ features in the image. This step involves reducing the grayscale levels of the dark regions and increasing the grayscale levels of the bright regions using the following equation to increase the contrast:(2)h=f−f∘b.Figure2(a) shows the original image. Figure 2(b) shows the result of politeness transformation. As a result, after the contrast stretching process, there is a sharper contrast between the germs and other parts of rice. This is beneficial to the subsequent germ integrity identification training.Figure 2
Image comparison after segmented transformation. (a) The original image. (b) The image processed by top-hat transformation.
(a)(b)


### 2.2. Image Segmentation

To segment small targets, this paper uses an improved image segmentation algorithm based on background skeleton features. In addition, the endpoint matching process in this algorithm is improved. To facilitate the analysis of BR grain features, it is necessary to separate the BR and the background based on image preprocessing. The background in the image is black, and its grayscale level differs relatively significantly from that of the BR in the image. To segment the BR grains, this study first uses a threshold segmentation algorithm. The image is segmented using the threshold segmentation algorithm based on the grayscale level. This method is applied to distinguish the BR foreground image from background image and then converts the grayscale image into a binary image. The segmentation equation is shown as follows:(3)gx,y=1,mx,y>N,0,mx,y≤N,where mx,y is the grayscale image, gx,y is the binary image, and N is the segmentation threshold (N is calculated using a classic and highly robust algorithm—the maximum between-class variance (Otsu) algorithm). A threshold is calculated using the Otsu algorithm primarily on the basis of maximum between-class difference between grayscale levels on the BR and the background. Based on the grayscale probability distribution, the Otsu algorithm for digital images is represented by the following equations:(4)w0=∑i=0zPi,w1=∑i=z+1L−1Pi,(5)μ0=∑i=0ziPiw0,μ1=∑i=z+1L−1iPiw1,(6)μ=w0μ0+w1μ1,(7)σ2=w0μ0−μ2+w1μ1−μ2,where z is the set grayscale segmentation threshold (grayscale levels are classified into classes A and B), Pi is the probability of each gray level, w0 and w1 are the probability sums of grayscale levels for classes A and B, respectively, the probability measures of classes A and B grayscale levels are μ0 and μ1, respectively, and σ2 is the between-class variance. The optimum segmentation threshold T that leads to a maximum σ2 is calculated by traversing z=0,255.The segmentation threshold for the grayscale BR image is calculated using the Otsu algorithm. Grayscale levels greater and less than the segmentation threshold in the image are set to 1 and 0, respectively. Thus, a binary BR image is obtained. Figure3 is a binary BR image obtained by processing a grayscale BR image.Figure 3
Binary image of brown rice.Background segmentation is followed by region segmentation. Based on the similarities between the spatial regions, in the region segmentation process, the regions are merged according to the similarity criteria. Below, the region growing algorithm is primarily used to segment the regions in the binary BR image to complete the marking and characterization of the regions.In region marking, the same connected components in an image are assigned the same mark, and different connected regions are differentiated with different marks. A binary image is used to mark the connected regions. The background is labeled with the same label (generally 0). Finally, the image is transformed into a mark matrix.The main algorithm used in region marking is region growing, which was originally used in grayscale image region segmentation. The idea of this algorithm is as follows. A seed pixel is set or found in the region that needs to be segmented. The region is expanded with this seed pixel as its initial point. Subsequently, the pixels within the neighborhood of the seed pixel are examined to determine their similarities to the seed pixel. The pixels that meet the similarity criteria are merged into the seed region. After that, until no more pixels meet the similarity criteria, the newly formed region is used as a seed for the similarity judgment and region merging steps. Finally, the merged region is one connected region that is independent.After subjecting the binary BR image to the background segmentation and region marking procedures, it is necessary to further segment different regions by marking their geometric properties. A method for describing the region features, which is involved in the extraction of the shape features from the binary BR image, is described by the following. The BR features include the perimeter, area, direction of the long axis, and center of gravity. The perimeter of a BR grain is the length of the boundary of the region. On a two-dimensional plane, the perimeter of a region can be approximately calculated by counting the number of pixels of its boundary. Sometimes, to obtain a more accurate representation, pixels are treated as points, and the bend and linear boundaries of a right angle are distinguished by chain-code representations. When the chain-code value is an odd number (i.e., bend boundary), the length is denoted by2. When the chain-code value is an even number (i.e., linear boundary), the length is denoted by 1. Thus, the perimeter p is represented by the following:(8)p=Ne+2No,where Ne is the number of chain codes with even-number values and No is the number of chain codes with odd-number values.The area of a region is its basic property. For the regionR in the BR image, its area is calculated based on the number of pixels contained within it. The area of the region R is represented by the following:(9)SR=∑x,y∈R1.The above equation calculates the cumulative number of pixels in all the regions plus 1.The position of a target region in the image is characterized by its center of gravity. In the binary BR image, all the pixels within the region have a value of 1 and thus follow a uniform distribution. Letxi,yi be the coordinates of the position of a pixel in the image. The equation for calculating the center of gravity is as follows:(10)x¯=1SR∑xi,yi∈Rxi,y¯=1SR∑xi,yi∈Ryi,where x¯ and y¯ are the coordinates of the position of the center of gravity and SR is the area of the region.During BR identification correction, the direction of each BR grain plays an important role and is the main feature parameter for extraction correction. Under normal circumstances, for a long and narrow region, the direction of the long side of its minimum bounding rectangle is selected as the direction of its long axis.(11)μpq=∑∑Rx−x¯py−y¯qfx,y,θ=12arctan2μ11μ20−μ02,where μpq is the central moment of the region, fx,y is the pixel value of the point x,y in the region, and θ is the angle of the direction of the long axis.The connected regions in the binary BR image are marked using the region growing algorithm, as shown in Figure4. In this paper, the connected regions of the image after binarization are numbered, and each number is at the center of each connected domain.Figure 4
Rice area marker image.The connected rice grains are segmented after the BR image has been subjected to the background threshold segmentation and region segmentation procedures. To segment the connected rice grains in the image, it is necessary to first determine the positions of the connected regions. The connected regions in the BR image are marked and described during the region segmentation procedure. Two of them were selected as examples for analysis. The original image is shown in Figure5(a). First, the image is binarized, as shown in Figure 5(b). Then, according to the calculation, the minimum convex hull (MCH) (i.e., minimum convex polygon) of each connected domain image is obtained. The MCH image is shown in Figure 5(c). Furthermore, the MCH region’s area is calculated using the region marking method. Equations (12) and (13) can be used to determine the connected regions.(12)fi=1,Si<0.85andAi>H,0,else,(13)solidity=areaconvex,where i∈1,m, fi is applied to judge the region connection condition (fi=1 means that the current region is a connected region; fi=0 means the current region is a nonconnected region), Ai is the area of the region, convex is the area of the MCH in the region, S is the ratio of the area of the region to the area of the MCH, and H is the area threshold (H is set based on the area of a single rice grain in the binary image; in this study, H is set to 2,400, which is a little bit smaller than the area of a single rice grain in the binary image).Figure 5
Schematic diagram of feature segmentation endpoints. (a) Original image. (b) Binary image. (c) MCH. (d) Background image. (e) Skeleton image. (f) Skeleton endpoints.
(a)(b)(c)(d)(e)(f)After determining the connected regions, it is necessary to calculate the segmentation endpoints. Thus, an operation is performed on the binary image and the MCH image using equation (14). A binary image of the background image for connected rice grains within the MCH region is thus obtained, as shown in Figure 5(d).(14)Cx,y=Ax,y−Bx,y,where Ax,y is the binary image of the MCH in the connected region, Bx,y is the binary image of connected BR grains, and Cx,y is the binary image of the background of the image of the connected BR grains.In the binary image, the connected features between BR grains images have been completely extracted, so the background image is represented by a curve. The topographic and connection features in the background image are preserved in the skeleton image, as shown in Figure5(e). The following shows the calculation of the skeleton:(15)SA=∪n=0NSnA,(16)SnA=A⊖nB−A⊖nB∘B,(17)A⊖nB=⋯A⊖B⊖B⊖⋯⊖B,where ⊖ is defined as the morphological erosion operator, the background image is represented by A, the skeleton of the background image is represented by SA, the structural element is represented by B, N is the value in A⊖nB≠∅ and A⊖n+1B=∅, SnA is the subset of nth skeleton, and A⊖nB signifies the use of B to erode A for an n number of consecutive times.The skeleton image endpoint look-up table function is used to generate a neighborhood pixel value look-up table Lut. Subsequently, the segmentation endpoints are searched for in the look-up table for the background skeleton feature image in Figure5(e). The segmentation points in the image’s coordinates are calculated. The segmentation endpoints in Figure 5(f) are the white and shaded endpoints. The skeleton image endpoint look-up table function is described by the following: in terms of a binary image, if there is one and only one pixel with the value of 1 within the 3×3 neighborhood of a pixel with the value of 1, the function returns the value of 1; otherwise, the function returns the value of 0. Detailed calculation method is shown as follows:(18)Mx,y=Ax−1,y−1Ax,y−1Ax+1,y−1Ax−1,yAx,yAx+1,yAx−1,y+1Ax,y+1Ax+1,y+1,(19)SumMx,y=∑i=x−1x+1∑j=y−1y+1Ai,j,(20)fMx,y=1,SMx,y=2andAx,y=1,0,other,where Mx,y is a 3×3 neighborhood with Ax,y as the center, Ax,y is a pixel with the value of 1 in the background skeleton feature image, SMx,y is totaled elements in Mx,y, and the skeleton endpoint extraction function is represented by fMx,y.The background skeleton endpoints are fixed in binary image. Because some skeleton endpoints are formed by the MCH boundary, not all of these skeleton endpoints can be defined as segmentation endpoints. Thus, the edges of the minimum convex image are detected based on the Sobel operator equations (21)–(23) to extract its boundary curve. Then, the minimum Euclidian distance between the boundary curve of MCH and each skeleton endpoint is calculated. The results form a minimum Euclidian distance set. The nonsegmentation endpoints are then removed based on(21)Gx=−101−202−101A,Gy=−1−2−1000121A,(22)G=Gx2+Gy2,(23)ϕ=arctanGxGy,(24)J=d1,d2,…dN,(25)Tdi=1,di>th,0,other,where A is the binary image of the MCH, G is the gradient of the image, Gx and Gy are, respectively, represented by the gradient of the image in x-axis and y-axis direction. ϕ is defined as the direction angle of gradient, J is defined as the set of the minimum Euclidian distances between endpoints and MCH, N is defined as total of endpoints, th is the segmentation endpoint distance judgment threshold (th is set to 4), Tdi=1 means that the point is a segmentation endpoint, and Tdi=0 means that the point is not a segmentation endpoint. In Figure 5(f), the minimum distances between shaded endpoints and boundary contours are no more than th. Thus, these shaded endpoints need to be removed. In comparison, the white endpoints are determined to be segmentation endpoints.Figure 6
Matching angle diagram.


### 2.3. Feature Point Matching Steps

(1)
Select a segmented end point as the reference end point in the calculated segmented end point library, and confirm the segmented matching area in accordance with the position characters of the reference end point in image. First, perform morphological expansion operation in the skeleton area where the segmentation endpoints are located. Then, the boundary features of the image are extracted (dotted lineT in Figure 6). In order to more distinctly show the spatial positional relationship in the image, an original contour curve of the background region of binary image is plotted (solid line T′ in Figure 6). In Figure 6, X is the selected segmentation endpoint as reference. Through traversal of the pixels of the boundary curve T, the boundary point (X2) closest to the reference segmentation endpoint is determined between the vector XX2. Then, the boundary curve where the boundary point X2 is located is smoothed by filtering boundary spike noise, as shown in equation (27). Based on smooth handling of contour curve, the points (X1 and X3) near X2 are selected. Thus, matching direction angle of the reference endpoint is determined on the basis of X1, X2, and X3. The specific algorithm of matching angle is calculated by the following equations:(26)Ax=A⊕B,(27)Xi=Ai−2+Ai−1+Ai+Ai+1+Ai+25,(28)Xmid=xmid,ymid=x1+x32,y1+y32,(29)θ=arctany2−ymidx2−xmid+mπ,(30)m=1,y2−ymid≥0&x2−xmid<0,0,x2−xmid≥0,−1,y2−ymid<0&x2−xmid<0,whereAX is defined as dilated skeleton image, A is original skeleton of binary image, B is defined as a 2-radius circular structural element, Ai is defined as coordinates of the contour boundary, Xi is defined as coordinates of boundary points after smooth handling of boundary curve, Xmid is defined as the midpoint between X1 and X3, and θ is defined as the angle of the vector XmidX2⟶ and the horizontal x-axis (i.e., the direction angle of the reference point; θ =  −π,π).(2)
The selected reference point is matched with the corresponding segment end point, referring to the angle of the end point, and±45° is selected as the matching radiating region (the shaded region in Figure 7). In Figure 7, the shaded region shows the search area for the points to be matched.During the matching process, a seed image is first generated using the region generation algorithm. The seed image can be obtained by eroding the binary image multiple times with structural elements differing in shape and size. The seed image, unlike the binary image obtained during the background threshold segmentation process, does not place a premium on BR contour integrity. In the generation of the binary image needed for the seed image, there is no need to reduce the contrast, because this also reduces the clarity of the boundaries and, thereby, causes the boundaries to connect with one another. To avoid insufficiently clear gaps between the BR grains during binarization due to uneven lighting, the image can be subjected to a top-hat transformation with a relatively large structural element first. The following equation shows the top-hat transformation equation (31), where f is the input image, B is the structural element, and g is the output image. Figure 8(a) shows the binary image of connected BR grains. Figure 8(b) shows the obtained seed image.(31)g=f−f∘B.Figure 7
Baseline endpoint matching diagram.Figure 8
Baseline endpoint matching diagram. (a) The binary image. (b) The obtained seed image.
(a)(b)Afterwards, a reference point randomly selected is considered as the center of the circle and the length of rice grain is regarded as the radius with the purpose of searching all reference points in the circle. Meanwhile, the image is considered as the grid map and the pixels of reference points are treated as coordinates; then the routine is planned by the ant colony algorithm. The shortest path of all reference points in the ergodic circle is calculated (the starting point and the ending point are not connected).The ant colony algorithm is constructed by a kind of biological heuristic optimization algorithm with regard to cooperative behaviors of ant colony in foraging, because its significant globally researching capability is widely applied in combinational optimization problems. Based on distance matrixD, the ant colony algorithm is used to calculate a group of patrol order X for minimization of patrol distance function VX.The number of patrol pointsδ and the number of ants in its colony m and distance matrix D are known; then all points passed by ant K are recorded in tabu list tk. After the first loop, the tabu list is reset to zero; then ants are free to select again. Based on pheromone and the distribution of heuristic information, the state transition probabilities of ants are identified in its movement process, as expressed in(32)pijk=τijαtηijβt∑s∈αkτisαtηisβt,j∈αk,0,other,(33)ηijt=1dij.Here,pijk is state transition probability, the larger number indicates the greater probability of selecting the next point for ants. i and j indicate, respectively, the current node and the next node. τijt is the amount of pheromone on the line of i and j at t moment. ηijt is heuristic function. ak is searchable nodes collection not in tabu list. a is pheromone index of relative importance. β is visibility index of relative importance. dij is the data of line i and line j in distance matrix D. According to equations (1) and (2) to execute the choice of next node until all patrol points are traversed, the length of motion path for each ant is recorded as Lkk=1,2,…m; meanwhile, the current shortest path Lmin and patrol order X are recorded. Then the pheromone is updated pursuant to following equation and Δτij is reset to 0, as expressed in(34)τijt+1=1−ρτijt+Δτijt,(35)Δτijt=∑k=1mΔτijkt,(36)Δτijkt=ϑLk,Thekthant passes through nodei,j,0,Thekthant doed not passes through nodei,j.Here, pheromone concentrationτijt is pheromone concentration in previous t time. Δτij is the increasing number of pheromone in the node within unit interval. Δτijkt is the increasing number of pheromone ant K. ρ is volatility coefficient of pheromone. Lk is the distance from the starting point to the end walked by ant K. θ is enhancement coefficient of pheromone. When iterations are up to its maximum Ncmax, jump out of the loop and then output the optimal patrol path X at this time as the final result. The shortest path planned by ant colony algorithm is as shown in image.After shortest-distance path planned, the first step is to identify which two consecutive points are matching endpoints in the path and then select two points according to the order planned by the ant colony algorithm. Through the shortest distance between the reference endpoint and seed image, the nearest seed from two endpoints is judged whether it is the same one or not. If the seeds are the same one, then they are matched endpoints. Otherwise, they are unmatched endpoints. The specific judgment method is as expressed in Figure9.Figure 9
Match verification diagram.The result as shown in Figure10 is concluded from the path planning and distance judgment of seeds. In the image, the endpoints of black line are matched endpoints. Then the rest of endpoints are reused to execute endpoint matching through ant colony algorithm until the endpoints are finally matched.Figure 10
Ant colony algorithm path planning image.


### 2.4. Image Identification

This study presents a CNN model for identifying BR germ positions as well as a CNN model for classifying BR germ integrity. In addition, these two models are used in combination to calculate the BR germ integrity.


#### 2.4.1. Identification of the Germ Direction

After grain image segmentation, the images of the single BR grains show that the germ is generally located in the upper left, upper right, lower left, or lower right corner of the grain. The direction of the germ is relatively notable. Furthermore, the images of single BR grains have already been corrected horizontally along the long axis. As a result, the germ can only go in four different directions. Thus, there are only four types of VGG network outputs. Therefore, no complex feature parameters are needed, as shown in Figure11.Figure 11
Image of the germ position.


#### 2.4.2. Germ Integrity Identification

By position identification, the images of the BR germs are rotated to ensure that they are in the same direction. Then, the germ integrity is identified using an optimized Inception-v3 model. The simplest method for improving a network is to increase its depth and breadth and increase the numbers of hidden layers and neurons in each layer. However, there are some problems with this method. This method leads to a larger parameter space, renders fitting easier, and requires more computational resources. The deeper the network is, the easier it is for the gradient to vanish, and the more difficult it is to optimize the network. The core of the Inception-v3 model involves the use of 1 × 1, 3 × 3, and 5 × 5 small convolutional layers to replace the large convolutional layers of the network. This can significantly reduce the number of weighting parameters. Figure12 shows the results obtained using the Inception-v3 model.Figure 12
The Inception-v3 model (Block1, Block2, and Block3 from left to right).During the training process, the activation input value of the deep NN deepens prior to nonlinear transformation, resulting in the movement of the NN distribution. The convergence is slow during training because the overall distribution gradually becomes a nonlinear function. This results in the vanishing of the gradient of the low-level NN during backpropagation.The batch normalization method normalizes each data output (mean: 0; variance: 1) as well as the mean and standard deviation of the data after the convolution pool, as shown in the following equation:(37)x^k=xk−ExkVarxk.The Inception-v3 model is used in combination with a proposed loss function, which is referred to as the mutual-channel loss (MC-loss) function, to distinguish the features of different channels. In addition, the MC-loss function is used to limit the distribution of the channels. The MC-loss function contains two channel-related components, namely, a discriminality component and a diversity component [28]. Figure 13 shows the overall model framework.Figure 13
Model frame diagram.The input imageX∈RW×H is used as the input of the basic network. The depth feature F∈RN×W×H corresponding to the image is extracted, where N is the number of channels of the depth feature and W and H are the width and height of each feature map, respectively. In addition, it is required that N equal c × ξ, where c is the total number of classes in the training dataset and ξ is the number of feature maps allocated to each class. Therefore, the nth-dimensional feature map of the depth feature F can be represented by Fn∈RWH, n = 1, 2, ... , N. Moreover, the depth feature of the class i can be represented by Fi∈Rξ×WH, i = 0, 1, 2, ... , c-1, as expressed in the following equation:(38)Fi=Fi×ξ+1,Fi×ξ+2,…,Fi×ξ+ξ,.Inputting two optimization objectives with different purposes into two different branches facilitates their calculation and optimization. In Figure13, the cross-entropy loss branch treats the feature as input. Through a fully connected layer, the conventional classification loss LCE is calculated. The cross-entropy loss can drive the network to extract the meaningful discriminality information and focus on the global discriminality region. In addition, the MC-loss branch supervises the network, allowing it to learn features and focus on different local discriminality regions. Subsequently, an ultimate loss function is obtained by adding the MC-loss weighted by μ and LCE. Thus, the loss function of the whole network is as follows:(39)LossF=LCEF+μLMCF.In addition, the MC-loss is obtained by a weighted addition of a discriminality component lossLdis and a diversity component loss Ldiv. Thus, the MC-loss is defined as follows:(40)LMCF=LdisF+λLdivF.The left boxes in Figure14show the procedure for the distinguishability component. The diversity component is used to measure the similarities between the feature channels. As shown by the right side of Figure 14, the diversity component can drive the feature channels of the same group to become different, as shown in Figure 14.Figure 14
Overview of the MC-loss.The orthogonal classification layer adds an improved dropout, which reduces unnecessary link items of various categories, as shown in the following equation:(41)r=soft maxM•Wv.M is a mask matrix of W, constant diagonal matrix, as expressed in the following equation:(42)M=M1,101,2⋯01,k02,1M2,2⋯02,k⋮⋮⋱⋮0k,10k,2⋯Mk,k.Based on Inception-v3 model, two branches are added. The first branch is the original germ rice image branch. The second branch is adding mlpconv to the Inception-v3 model which can help the model to better approximate the abstract representation of the potential space. The simple convolutional neural network cannot extract the abstract features of the germ well, and the inception structure is too complicated, due to the small size of the brown rice germ. Therefore, mlpconv is used to effectively extract the germ characteristics through the subsequent1∗1 convolution. Figure 15 shows the structure.Figure 15
mlpconv layer.At the same time, ResNet is added into the second branch [45–47]. These residual connections allow us to have a deeper network structure without increasing parameters. The second branch is multiple MBR connection including mlpconv, batch normalization, and ReLU. The integral structure is as shown in Figures 16 and 17.Figure 16
The scheme of a possible residual connection in a CNN.Figure 17
The overall structure of MBR.The input image is divided into three parts. Through the improved Inception-v3 loss function, the first part becomes the original part (the input image is299∗299). In the second part (the input image is 35∗35; after upsampling it becomes 225∗225), the BR germs are extracted based on the position direction. As the germ portion is too small in size, the number of pixels is increased by upsampling. Subsequently, the germ image is transferred to 7-layer MBR network. Through the MBR network, the deeper abstract features of the germ can be extracted. It is beneficial to improve the detection accuracy of embryo integrity. The upper part of the network focuses on the extraction of the overall features of brown rice, and the lower part of the network focuses on the feature extraction of the germ part of brown rice. Combining the two parts not only preserves the overall information of the rice, but also accurately detects the integrity of the germ. Finally, the first two parts are combined with the original germ image by global average pooling and input into a complete connection layer. Figure 18 shows the network structure.Figure 18
Improved Inception-v3 network structure diagram.During the production and processing of BR, all the processing procedures are completed by experienced workers through visual assessment, which ensures processing quality and facilitates the adjustment of the equipment. Human visual assessment is based on a general feeling and cannot accurately determine the germ integrity. To more accurately describe the germ integrity of BR and improve its quality, BR grains are classified by observation into eight types, ranging from germ-free BR to whole-germ BR. The classification of each type of BR is based on the germ integrity. Figure19 shows the results.Figure 19
Rice germ integrity classification standard.


## 2.1. Preprocessing

There are two steps to preprocessing a BR image. (1) After binarization, small, scattered white areas are often generated in an image due to the miscellaneous items on the platforms where the husks, bran, and rice are placed during production and processing as well as noise from some image processing procedures. These small areas may subsequently affect the identification of the effective connected areas and thus need to be eliminated. (2) The training set is image preprocessed to enhance the features of the germ portion in the images in order to improve subsequent training performance for the germ integrity of single rice grains.


### 2.1.1. Image Preprocessing

In the first step of image preprocessing, conventional spatial and frequency domain filters cannot simply and effectively eliminate the small areas, whereas a morphological filter can easily eliminate them. To eliminate the small areas, a structural element with a suitable dimension can be used to erode the original image. After erosion, these small areas can be eliminated. An erosion operation, on the other hand, enlarges the voids and separates the previously connected regions. A rice area can also be reduced by an erosion operation. To eliminate the negative effects of an erosion operation, a dilation operation can be subsequently performed. The combination consisting of first an erosion operation and a subsequent dilation operation is referred to as an opening operation. The “。” in the following equation is the operator of an opening operation and signifies that a structural element is first used to erode the image and then dilate the results of the erosion operation. We have(1)X∘B=X⊖B⊕B.A morphological filter is used to remove the small white area in the following left image. A 5 × 5 square is used as a structural element. The comparison image shows that the small white area is removed. The results are shown in Figure1.Figure 1
Comparison of images before and after morphological filtering. (a) The original image. (b) The image processed by morphological filter.
(a)(b)


### 2.1.2. Image Segmentation Preprocessing

In the second step of image segmentation preprocessing, the training set is subjected to top-hat transformation to enhance the germ features in the image. This step involves reducing the grayscale levels of the dark regions and increasing the grayscale levels of the bright regions using the following equation to increase the contrast:(2)h=f−f∘b.Figure2(a) shows the original image. Figure 2(b) shows the result of politeness transformation. As a result, after the contrast stretching process, there is a sharper contrast between the germs and other parts of rice. This is beneficial to the subsequent germ integrity identification training.Figure 2
Image comparison after segmented transformation. (a) The original image. (b) The image processed by top-hat transformation.
(a)(b)


## 2.1.1. Image Preprocessing

In the first step of image preprocessing, conventional spatial and frequency domain filters cannot simply and effectively eliminate the small areas, whereas a morphological filter can easily eliminate them. To eliminate the small areas, a structural element with a suitable dimension can be used to erode the original image. After erosion, these small areas can be eliminated. An erosion operation, on the other hand, enlarges the voids and separates the previously connected regions. A rice area can also be reduced by an erosion operation. To eliminate the negative effects of an erosion operation, a dilation operation can be subsequently performed. The combination consisting of first an erosion operation and a subsequent dilation operation is referred to as an opening operation. The “。” in the following equation is the operator of an opening operation and signifies that a structural element is first used to erode the image and then dilate the results of the erosion operation. We have(1)X∘B=X⊖B⊕B.A morphological filter is used to remove the small white area in the following left image. A 5 × 5 square is used as a structural element. The comparison image shows that the small white area is removed. The results are shown in Figure1.Figure 1
Comparison of images before and after morphological filtering. (a) The original image. (b) The image processed by morphological filter.
(a)(b)


## 2.1.2. Image Segmentation Preprocessing

In the second step of image segmentation preprocessing, the training set is subjected to top-hat transformation to enhance the germ features in the image. This step involves reducing the grayscale levels of the dark regions and increasing the grayscale levels of the bright regions using the following equation to increase the contrast:(2)h=f−f∘b.Figure2(a) shows the original image. Figure 2(b) shows the result of politeness transformation. As a result, after the contrast stretching process, there is a sharper contrast between the germs and other parts of rice. This is beneficial to the subsequent germ integrity identification training.Figure 2
Image comparison after segmented transformation. (a) The original image. (b) The image processed by top-hat transformation.
(a)(b)


## 2.2. Image Segmentation

To segment small targets, this paper uses an improved image segmentation algorithm based on background skeleton features. In addition, the endpoint matching process in this algorithm is improved. To facilitate the analysis of BR grain features, it is necessary to separate the BR and the background based on image preprocessing. The background in the image is black, and its grayscale level differs relatively significantly from that of the BR in the image. To segment the BR grains, this study first uses a threshold segmentation algorithm. The image is segmented using the threshold segmentation algorithm based on the grayscale level. This method is applied to distinguish the BR foreground image from background image and then converts the grayscale image into a binary image. The segmentation equation is shown as follows:(3)gx,y=1,mx,y>N,0,mx,y≤N,where mx,y is the grayscale image, gx,y is the binary image, and N is the segmentation threshold (N is calculated using a classic and highly robust algorithm—the maximum between-class variance (Otsu) algorithm). A threshold is calculated using the Otsu algorithm primarily on the basis of maximum between-class difference between grayscale levels on the BR and the background. Based on the grayscale probability distribution, the Otsu algorithm for digital images is represented by the following equations:(4)w0=∑i=0zPi,w1=∑i=z+1L−1Pi,(5)μ0=∑i=0ziPiw0,μ1=∑i=z+1L−1iPiw1,(6)μ=w0μ0+w1μ1,(7)σ2=w0μ0−μ2+w1μ1−μ2,where z is the set grayscale segmentation threshold (grayscale levels are classified into classes A and B), Pi is the probability of each gray level, w0 and w1 are the probability sums of grayscale levels for classes A and B, respectively, the probability measures of classes A and B grayscale levels are μ0 and μ1, respectively, and σ2 is the between-class variance. The optimum segmentation threshold T that leads to a maximum σ2 is calculated by traversing z=0,255.The segmentation threshold for the grayscale BR image is calculated using the Otsu algorithm. Grayscale levels greater and less than the segmentation threshold in the image are set to 1 and 0, respectively. Thus, a binary BR image is obtained. Figure3 is a binary BR image obtained by processing a grayscale BR image.Figure 3
Binary image of brown rice.Background segmentation is followed by region segmentation. Based on the similarities between the spatial regions, in the region segmentation process, the regions are merged according to the similarity criteria. Below, the region growing algorithm is primarily used to segment the regions in the binary BR image to complete the marking and characterization of the regions.In region marking, the same connected components in an image are assigned the same mark, and different connected regions are differentiated with different marks. A binary image is used to mark the connected regions. The background is labeled with the same label (generally 0). Finally, the image is transformed into a mark matrix.The main algorithm used in region marking is region growing, which was originally used in grayscale image region segmentation. The idea of this algorithm is as follows. A seed pixel is set or found in the region that needs to be segmented. The region is expanded with this seed pixel as its initial point. Subsequently, the pixels within the neighborhood of the seed pixel are examined to determine their similarities to the seed pixel. The pixels that meet the similarity criteria are merged into the seed region. After that, until no more pixels meet the similarity criteria, the newly formed region is used as a seed for the similarity judgment and region merging steps. Finally, the merged region is one connected region that is independent.After subjecting the binary BR image to the background segmentation and region marking procedures, it is necessary to further segment different regions by marking their geometric properties. A method for describing the region features, which is involved in the extraction of the shape features from the binary BR image, is described by the following. The BR features include the perimeter, area, direction of the long axis, and center of gravity. The perimeter of a BR grain is the length of the boundary of the region. On a two-dimensional plane, the perimeter of a region can be approximately calculated by counting the number of pixels of its boundary. Sometimes, to obtain a more accurate representation, pixels are treated as points, and the bend and linear boundaries of a right angle are distinguished by chain-code representations. When the chain-code value is an odd number (i.e., bend boundary), the length is denoted by2. When the chain-code value is an even number (i.e., linear boundary), the length is denoted by 1. Thus, the perimeter p is represented by the following:(8)p=Ne+2No,where Ne is the number of chain codes with even-number values and No is the number of chain codes with odd-number values.The area of a region is its basic property. For the regionR in the BR image, its area is calculated based on the number of pixels contained within it. The area of the region R is represented by the following:(9)SR=∑x,y∈R1.The above equation calculates the cumulative number of pixels in all the regions plus 1.The position of a target region in the image is characterized by its center of gravity. In the binary BR image, all the pixels within the region have a value of 1 and thus follow a uniform distribution. Letxi,yi be the coordinates of the position of a pixel in the image. The equation for calculating the center of gravity is as follows:(10)x¯=1SR∑xi,yi∈Rxi,y¯=1SR∑xi,yi∈Ryi,where x¯ and y¯ are the coordinates of the position of the center of gravity and SR is the area of the region.During BR identification correction, the direction of each BR grain plays an important role and is the main feature parameter for extraction correction. Under normal circumstances, for a long and narrow region, the direction of the long side of its minimum bounding rectangle is selected as the direction of its long axis.(11)μpq=∑∑Rx−x¯py−y¯qfx,y,θ=12arctan2μ11μ20−μ02,where μpq is the central moment of the region, fx,y is the pixel value of the point x,y in the region, and θ is the angle of the direction of the long axis.The connected regions in the binary BR image are marked using the region growing algorithm, as shown in Figure4. In this paper, the connected regions of the image after binarization are numbered, and each number is at the center of each connected domain.Figure 4
Rice area marker image.The connected rice grains are segmented after the BR image has been subjected to the background threshold segmentation and region segmentation procedures. To segment the connected rice grains in the image, it is necessary to first determine the positions of the connected regions. The connected regions in the BR image are marked and described during the region segmentation procedure. Two of them were selected as examples for analysis. The original image is shown in Figure5(a). First, the image is binarized, as shown in Figure 5(b). Then, according to the calculation, the minimum convex hull (MCH) (i.e., minimum convex polygon) of each connected domain image is obtained. The MCH image is shown in Figure 5(c). Furthermore, the MCH region’s area is calculated using the region marking method. Equations (12) and (13) can be used to determine the connected regions.(12)fi=1,Si<0.85andAi>H,0,else,(13)solidity=areaconvex,where i∈1,m, fi is applied to judge the region connection condition (fi=1 means that the current region is a connected region; fi=0 means the current region is a nonconnected region), Ai is the area of the region, convex is the area of the MCH in the region, S is the ratio of the area of the region to the area of the MCH, and H is the area threshold (H is set based on the area of a single rice grain in the binary image; in this study, H is set to 2,400, which is a little bit smaller than the area of a single rice grain in the binary image).Figure 5
Schematic diagram of feature segmentation endpoints. (a) Original image. (b) Binary image. (c) MCH. (d) Background image. (e) Skeleton image. (f) Skeleton endpoints.
(a)(b)(c)(d)(e)(f)After determining the connected regions, it is necessary to calculate the segmentation endpoints. Thus, an operation is performed on the binary image and the MCH image using equation (14). A binary image of the background image for connected rice grains within the MCH region is thus obtained, as shown in Figure 5(d).(14)Cx,y=Ax,y−Bx,y,where Ax,y is the binary image of the MCH in the connected region, Bx,y is the binary image of connected BR grains, and Cx,y is the binary image of the background of the image of the connected BR grains.In the binary image, the connected features between BR grains images have been completely extracted, so the background image is represented by a curve. The topographic and connection features in the background image are preserved in the skeleton image, as shown in Figure5(e). The following shows the calculation of the skeleton:(15)SA=∪n=0NSnA,(16)SnA=A⊖nB−A⊖nB∘B,(17)A⊖nB=⋯A⊖B⊖B⊖⋯⊖B,where ⊖ is defined as the morphological erosion operator, the background image is represented by A, the skeleton of the background image is represented by SA, the structural element is represented by B, N is the value in A⊖nB≠∅ and A⊖n+1B=∅, SnA is the subset of nth skeleton, and A⊖nB signifies the use of B to erode A for an n number of consecutive times.The skeleton image endpoint look-up table function is used to generate a neighborhood pixel value look-up table Lut. Subsequently, the segmentation endpoints are searched for in the look-up table for the background skeleton feature image in Figure5(e). The segmentation points in the image’s coordinates are calculated. The segmentation endpoints in Figure 5(f) are the white and shaded endpoints. The skeleton image endpoint look-up table function is described by the following: in terms of a binary image, if there is one and only one pixel with the value of 1 within the 3×3 neighborhood of a pixel with the value of 1, the function returns the value of 1; otherwise, the function returns the value of 0. Detailed calculation method is shown as follows:(18)Mx,y=Ax−1,y−1Ax,y−1Ax+1,y−1Ax−1,yAx,yAx+1,yAx−1,y+1Ax,y+1Ax+1,y+1,(19)SumMx,y=∑i=x−1x+1∑j=y−1y+1Ai,j,(20)fMx,y=1,SMx,y=2andAx,y=1,0,other,where Mx,y is a 3×3 neighborhood with Ax,y as the center, Ax,y is a pixel with the value of 1 in the background skeleton feature image, SMx,y is totaled elements in Mx,y, and the skeleton endpoint extraction function is represented by fMx,y.The background skeleton endpoints are fixed in binary image. Because some skeleton endpoints are formed by the MCH boundary, not all of these skeleton endpoints can be defined as segmentation endpoints. Thus, the edges of the minimum convex image are detected based on the Sobel operator equations (21)–(23) to extract its boundary curve. Then, the minimum Euclidian distance between the boundary curve of MCH and each skeleton endpoint is calculated. The results form a minimum Euclidian distance set. The nonsegmentation endpoints are then removed based on(21)Gx=−101−202−101A,Gy=−1−2−1000121A,(22)G=Gx2+Gy2,(23)ϕ=arctanGxGy,(24)J=d1,d2,…dN,(25)Tdi=1,di>th,0,other,where A is the binary image of the MCH, G is the gradient of the image, Gx and Gy are, respectively, represented by the gradient of the image in x-axis and y-axis direction. ϕ is defined as the direction angle of gradient, J is defined as the set of the minimum Euclidian distances between endpoints and MCH, N is defined as total of endpoints, th is the segmentation endpoint distance judgment threshold (th is set to 4), Tdi=1 means that the point is a segmentation endpoint, and Tdi=0 means that the point is not a segmentation endpoint. In Figure 5(f), the minimum distances between shaded endpoints and boundary contours are no more than th. Thus, these shaded endpoints need to be removed. In comparison, the white endpoints are determined to be segmentation endpoints.Figure 6
Matching angle diagram.


## 2.3. Feature Point Matching Steps

(1)
Select a segmented end point as the reference end point in the calculated segmented end point library, and confirm the segmented matching area in accordance with the position characters of the reference end point in image. First, perform morphological expansion operation in the skeleton area where the segmentation endpoints are located. Then, the boundary features of the image are extracted (dotted lineT in Figure 6). In order to more distinctly show the spatial positional relationship in the image, an original contour curve of the background region of binary image is plotted (solid line T′ in Figure 6). In Figure 6, X is the selected segmentation endpoint as reference. Through traversal of the pixels of the boundary curve T, the boundary point (X2) closest to the reference segmentation endpoint is determined between the vector XX2. Then, the boundary curve where the boundary point X2 is located is smoothed by filtering boundary spike noise, as shown in equation (27). Based on smooth handling of contour curve, the points (X1 and X3) near X2 are selected. Thus, matching direction angle of the reference endpoint is determined on the basis of X1, X2, and X3. The specific algorithm of matching angle is calculated by the following equations:(26)Ax=A⊕B,(27)Xi=Ai−2+Ai−1+Ai+Ai+1+Ai+25,(28)Xmid=xmid,ymid=x1+x32,y1+y32,(29)θ=arctany2−ymidx2−xmid+mπ,(30)m=1,y2−ymid≥0&x2−xmid<0,0,x2−xmid≥0,−1,y2−ymid<0&x2−xmid<0,whereAX is defined as dilated skeleton image, A is original skeleton of binary image, B is defined as a 2-radius circular structural element, Ai is defined as coordinates of the contour boundary, Xi is defined as coordinates of boundary points after smooth handling of boundary curve, Xmid is defined as the midpoint between X1 and X3, and θ is defined as the angle of the vector XmidX2⟶ and the horizontal x-axis (i.e., the direction angle of the reference point; θ =  −π,π).(2)
The selected reference point is matched with the corresponding segment end point, referring to the angle of the end point, and±45° is selected as the matching radiating region (the shaded region in Figure 7). In Figure 7, the shaded region shows the search area for the points to be matched.During the matching process, a seed image is first generated using the region generation algorithm. The seed image can be obtained by eroding the binary image multiple times with structural elements differing in shape and size. The seed image, unlike the binary image obtained during the background threshold segmentation process, does not place a premium on BR contour integrity. In the generation of the binary image needed for the seed image, there is no need to reduce the contrast, because this also reduces the clarity of the boundaries and, thereby, causes the boundaries to connect with one another. To avoid insufficiently clear gaps between the BR grains during binarization due to uneven lighting, the image can be subjected to a top-hat transformation with a relatively large structural element first. The following equation shows the top-hat transformation equation (31), where f is the input image, B is the structural element, and g is the output image. Figure 8(a) shows the binary image of connected BR grains. Figure 8(b) shows the obtained seed image.(31)g=f−f∘B.Figure 7
Baseline endpoint matching diagram.Figure 8
Baseline endpoint matching diagram. (a) The binary image. (b) The obtained seed image.
(a)(b)Afterwards, a reference point randomly selected is considered as the center of the circle and the length of rice grain is regarded as the radius with the purpose of searching all reference points in the circle. Meanwhile, the image is considered as the grid map and the pixels of reference points are treated as coordinates; then the routine is planned by the ant colony algorithm. The shortest path of all reference points in the ergodic circle is calculated (the starting point and the ending point are not connected).The ant colony algorithm is constructed by a kind of biological heuristic optimization algorithm with regard to cooperative behaviors of ant colony in foraging, because its significant globally researching capability is widely applied in combinational optimization problems. Based on distance matrixD, the ant colony algorithm is used to calculate a group of patrol order X for minimization of patrol distance function VX.The number of patrol pointsδ and the number of ants in its colony m and distance matrix D are known; then all points passed by ant K are recorded in tabu list tk. After the first loop, the tabu list is reset to zero; then ants are free to select again. Based on pheromone and the distribution of heuristic information, the state transition probabilities of ants are identified in its movement process, as expressed in(32)pijk=τijαtηijβt∑s∈αkτisαtηisβt,j∈αk,0,other,(33)ηijt=1dij.Here,pijk is state transition probability, the larger number indicates the greater probability of selecting the next point for ants. i and j indicate, respectively, the current node and the next node. τijt is the amount of pheromone on the line of i and j at t moment. ηijt is heuristic function. ak is searchable nodes collection not in tabu list. a is pheromone index of relative importance. β is visibility index of relative importance. dij is the data of line i and line j in distance matrix D. According to equations (1) and (2) to execute the choice of next node until all patrol points are traversed, the length of motion path for each ant is recorded as Lkk=1,2,…m; meanwhile, the current shortest path Lmin and patrol order X are recorded. Then the pheromone is updated pursuant to following equation and Δτij is reset to 0, as expressed in(34)τijt+1=1−ρτijt+Δτijt,(35)Δτijt=∑k=1mΔτijkt,(36)Δτijkt=ϑLk,Thekthant passes through nodei,j,0,Thekthant doed not passes through nodei,j.Here, pheromone concentrationτijt is pheromone concentration in previous t time. Δτij is the increasing number of pheromone in the node within unit interval. Δτijkt is the increasing number of pheromone ant K. ρ is volatility coefficient of pheromone. Lk is the distance from the starting point to the end walked by ant K. θ is enhancement coefficient of pheromone. When iterations are up to its maximum Ncmax, jump out of the loop and then output the optimal patrol path X at this time as the final result. The shortest path planned by ant colony algorithm is as shown in image.After shortest-distance path planned, the first step is to identify which two consecutive points are matching endpoints in the path and then select two points according to the order planned by the ant colony algorithm. Through the shortest distance between the reference endpoint and seed image, the nearest seed from two endpoints is judged whether it is the same one or not. If the seeds are the same one, then they are matched endpoints. Otherwise, they are unmatched endpoints. The specific judgment method is as expressed in Figure9.Figure 9
Match verification diagram.The result as shown in Figure10 is concluded from the path planning and distance judgment of seeds. In the image, the endpoints of black line are matched endpoints. Then the rest of endpoints are reused to execute endpoint matching through ant colony algorithm until the endpoints are finally matched.Figure 10
Ant colony algorithm path planning image.


## 2.4. Image Identification

This study presents a CNN model for identifying BR germ positions as well as a CNN model for classifying BR germ integrity. In addition, these two models are used in combination to calculate the BR germ integrity.


### 2.4.1. Identification of the Germ Direction

After grain image segmentation, the images of the single BR grains show that the germ is generally located in the upper left, upper right, lower left, or lower right corner of the grain. The direction of the germ is relatively notable. Furthermore, the images of single BR grains have already been corrected horizontally along the long axis. As a result, the germ can only go in four different directions. Thus, there are only four types of VGG network outputs. Therefore, no complex feature parameters are needed, as shown in Figure11.Figure 11
Image of the germ position.


### 2.4.2. Germ Integrity Identification

By position identification, the images of the BR germs are rotated to ensure that they are in the same direction. Then, the germ integrity is identified using an optimized Inception-v3 model. The simplest method for improving a network is to increase its depth and breadth and increase the numbers of hidden layers and neurons in each layer. However, there are some problems with this method. This method leads to a larger parameter space, renders fitting easier, and requires more computational resources. The deeper the network is, the easier it is for the gradient to vanish, and the more difficult it is to optimize the network. The core of the Inception-v3 model involves the use of 1 × 1, 3 × 3, and 5 × 5 small convolutional layers to replace the large convolutional layers of the network. This can significantly reduce the number of weighting parameters. Figure12 shows the results obtained using the Inception-v3 model.Figure 12
The Inception-v3 model (Block1, Block2, and Block3 from left to right).During the training process, the activation input value of the deep NN deepens prior to nonlinear transformation, resulting in the movement of the NN distribution. The convergence is slow during training because the overall distribution gradually becomes a nonlinear function. This results in the vanishing of the gradient of the low-level NN during backpropagation.The batch normalization method normalizes each data output (mean: 0; variance: 1) as well as the mean and standard deviation of the data after the convolution pool, as shown in the following equation:(37)x^k=xk−ExkVarxk.The Inception-v3 model is used in combination with a proposed loss function, which is referred to as the mutual-channel loss (MC-loss) function, to distinguish the features of different channels. In addition, the MC-loss function is used to limit the distribution of the channels. The MC-loss function contains two channel-related components, namely, a discriminality component and a diversity component [28]. Figure 13 shows the overall model framework.Figure 13
Model frame diagram.The input imageX∈RW×H is used as the input of the basic network. The depth feature F∈RN×W×H corresponding to the image is extracted, where N is the number of channels of the depth feature and W and H are the width and height of each feature map, respectively. In addition, it is required that N equal c × ξ, where c is the total number of classes in the training dataset and ξ is the number of feature maps allocated to each class. Therefore, the nth-dimensional feature map of the depth feature F can be represented by Fn∈RWH, n = 1, 2, ... , N. Moreover, the depth feature of the class i can be represented by Fi∈Rξ×WH, i = 0, 1, 2, ... , c-1, as expressed in the following equation:(38)Fi=Fi×ξ+1,Fi×ξ+2,…,Fi×ξ+ξ,.Inputting two optimization objectives with different purposes into two different branches facilitates their calculation and optimization. In Figure13, the cross-entropy loss branch treats the feature as input. Through a fully connected layer, the conventional classification loss LCE is calculated. The cross-entropy loss can drive the network to extract the meaningful discriminality information and focus on the global discriminality region. In addition, the MC-loss branch supervises the network, allowing it to learn features and focus on different local discriminality regions. Subsequently, an ultimate loss function is obtained by adding the MC-loss weighted by μ and LCE. Thus, the loss function of the whole network is as follows:(39)LossF=LCEF+μLMCF.In addition, the MC-loss is obtained by a weighted addition of a discriminality component lossLdis and a diversity component loss Ldiv. Thus, the MC-loss is defined as follows:(40)LMCF=LdisF+λLdivF.The left boxes in Figure14show the procedure for the distinguishability component. The diversity component is used to measure the similarities between the feature channels. As shown by the right side of Figure 14, the diversity component can drive the feature channels of the same group to become different, as shown in Figure 14.Figure 14
Overview of the MC-loss.The orthogonal classification layer adds an improved dropout, which reduces unnecessary link items of various categories, as shown in the following equation:(41)r=soft maxM•Wv.M is a mask matrix of W, constant diagonal matrix, as expressed in the following equation:(42)M=M1,101,2⋯01,k02,1M2,2⋯02,k⋮⋮⋱⋮0k,10k,2⋯Mk,k.Based on Inception-v3 model, two branches are added. The first branch is the original germ rice image branch. The second branch is adding mlpconv to the Inception-v3 model which can help the model to better approximate the abstract representation of the potential space. The simple convolutional neural network cannot extract the abstract features of the germ well, and the inception structure is too complicated, due to the small size of the brown rice germ. Therefore, mlpconv is used to effectively extract the germ characteristics through the subsequent1∗1 convolution. Figure 15 shows the structure.Figure 15
mlpconv layer.At the same time, ResNet is added into the second branch [45–47]. These residual connections allow us to have a deeper network structure without increasing parameters. The second branch is multiple MBR connection including mlpconv, batch normalization, and ReLU. The integral structure is as shown in Figures 16 and 17.Figure 16
The scheme of a possible residual connection in a CNN.Figure 17
The overall structure of MBR.The input image is divided into three parts. Through the improved Inception-v3 loss function, the first part becomes the original part (the input image is299∗299). In the second part (the input image is 35∗35; after upsampling it becomes 225∗225), the BR germs are extracted based on the position direction. As the germ portion is too small in size, the number of pixels is increased by upsampling. Subsequently, the germ image is transferred to 7-layer MBR network. Through the MBR network, the deeper abstract features of the germ can be extracted. It is beneficial to improve the detection accuracy of embryo integrity. The upper part of the network focuses on the extraction of the overall features of brown rice, and the lower part of the network focuses on the feature extraction of the germ part of brown rice. Combining the two parts not only preserves the overall information of the rice, but also accurately detects the integrity of the germ. Finally, the first two parts are combined with the original germ image by global average pooling and input into a complete connection layer. Figure 18 shows the network structure.Figure 18
Improved Inception-v3 network structure diagram.During the production and processing of BR, all the processing procedures are completed by experienced workers through visual assessment, which ensures processing quality and facilitates the adjustment of the equipment. Human visual assessment is based on a general feeling and cannot accurately determine the germ integrity. To more accurately describe the germ integrity of BR and improve its quality, BR grains are classified by observation into eight types, ranging from germ-free BR to whole-germ BR. The classification of each type of BR is based on the germ integrity. Figure19 shows the results.Figure 19
Rice germ integrity classification standard.


## 2.4.1. Identification of the Germ Direction

After grain image segmentation, the images of the single BR grains show that the germ is generally located in the upper left, upper right, lower left, or lower right corner of the grain. The direction of the germ is relatively notable. Furthermore, the images of single BR grains have already been corrected horizontally along the long axis. As a result, the germ can only go in four different directions. Thus, there are only four types of VGG network outputs. Therefore, no complex feature parameters are needed, as shown in Figure11.Figure 11
Image of the germ position.


## 2.4.2. Germ Integrity Identification

By position identification, the images of the BR germs are rotated to ensure that they are in the same direction. Then, the germ integrity is identified using an optimized Inception-v3 model. The simplest method for improving a network is to increase its depth and breadth and increase the numbers of hidden layers and neurons in each layer. However, there are some problems with this method. This method leads to a larger parameter space, renders fitting easier, and requires more computational resources. The deeper the network is, the easier it is for the gradient to vanish, and the more difficult it is to optimize the network. The core of the Inception-v3 model involves the use of 1 × 1, 3 × 3, and 5 × 5 small convolutional layers to replace the large convolutional layers of the network. This can significantly reduce the number of weighting parameters. Figure12 shows the results obtained using the Inception-v3 model.Figure 12
The Inception-v3 model (Block1, Block2, and Block3 from left to right).During the training process, the activation input value of the deep NN deepens prior to nonlinear transformation, resulting in the movement of the NN distribution. The convergence is slow during training because the overall distribution gradually becomes a nonlinear function. This results in the vanishing of the gradient of the low-level NN during backpropagation.The batch normalization method normalizes each data output (mean: 0; variance: 1) as well as the mean and standard deviation of the data after the convolution pool, as shown in the following equation:(37)x^k=xk−ExkVarxk.The Inception-v3 model is used in combination with a proposed loss function, which is referred to as the mutual-channel loss (MC-loss) function, to distinguish the features of different channels. In addition, the MC-loss function is used to limit the distribution of the channels. The MC-loss function contains two channel-related components, namely, a discriminality component and a diversity component [28]. Figure 13 shows the overall model framework.Figure 13
Model frame diagram.The input imageX∈RW×H is used as the input of the basic network. The depth feature F∈RN×W×H corresponding to the image is extracted, where N is the number of channels of the depth feature and W and H are the width and height of each feature map, respectively. In addition, it is required that N equal c × ξ, where c is the total number of classes in the training dataset and ξ is the number of feature maps allocated to each class. Therefore, the nth-dimensional feature map of the depth feature F can be represented by Fn∈RWH, n = 1, 2, ... , N. Moreover, the depth feature of the class i can be represented by Fi∈Rξ×WH, i = 0, 1, 2, ... , c-1, as expressed in the following equation:(38)Fi=Fi×ξ+1,Fi×ξ+2,…,Fi×ξ+ξ,.Inputting two optimization objectives with different purposes into two different branches facilitates their calculation and optimization. In Figure13, the cross-entropy loss branch treats the feature as input. Through a fully connected layer, the conventional classification loss LCE is calculated. The cross-entropy loss can drive the network to extract the meaningful discriminality information and focus on the global discriminality region. In addition, the MC-loss branch supervises the network, allowing it to learn features and focus on different local discriminality regions. Subsequently, an ultimate loss function is obtained by adding the MC-loss weighted by μ and LCE. Thus, the loss function of the whole network is as follows:(39)LossF=LCEF+μLMCF.In addition, the MC-loss is obtained by a weighted addition of a discriminality component lossLdis and a diversity component loss Ldiv. Thus, the MC-loss is defined as follows:(40)LMCF=LdisF+λLdivF.The left boxes in Figure14show the procedure for the distinguishability component. The diversity component is used to measure the similarities between the feature channels. As shown by the right side of Figure 14, the diversity component can drive the feature channels of the same group to become different, as shown in Figure 14.Figure 14
Overview of the MC-loss.The orthogonal classification layer adds an improved dropout, which reduces unnecessary link items of various categories, as shown in the following equation:(41)r=soft maxM•Wv.M is a mask matrix of W, constant diagonal matrix, as expressed in the following equation:(42)M=M1,101,2⋯01,k02,1M2,2⋯02,k⋮⋮⋱⋮0k,10k,2⋯Mk,k.Based on Inception-v3 model, two branches are added. The first branch is the original germ rice image branch. The second branch is adding mlpconv to the Inception-v3 model which can help the model to better approximate the abstract representation of the potential space. The simple convolutional neural network cannot extract the abstract features of the germ well, and the inception structure is too complicated, due to the small size of the brown rice germ. Therefore, mlpconv is used to effectively extract the germ characteristics through the subsequent1∗1 convolution. Figure 15 shows the structure.Figure 15
mlpconv layer.At the same time, ResNet is added into the second branch [45–47]. These residual connections allow us to have a deeper network structure without increasing parameters. The second branch is multiple MBR connection including mlpconv, batch normalization, and ReLU. The integral structure is as shown in Figures 16 and 17.Figure 16
The scheme of a possible residual connection in a CNN.Figure 17
The overall structure of MBR.The input image is divided into three parts. Through the improved Inception-v3 loss function, the first part becomes the original part (the input image is299∗299). In the second part (the input image is 35∗35; after upsampling it becomes 225∗225), the BR germs are extracted based on the position direction. As the germ portion is too small in size, the number of pixels is increased by upsampling. Subsequently, the germ image is transferred to 7-layer MBR network. Through the MBR network, the deeper abstract features of the germ can be extracted. It is beneficial to improve the detection accuracy of embryo integrity. The upper part of the network focuses on the extraction of the overall features of brown rice, and the lower part of the network focuses on the feature extraction of the germ part of brown rice. Combining the two parts not only preserves the overall information of the rice, but also accurately detects the integrity of the germ. Finally, the first two parts are combined with the original germ image by global average pooling and input into a complete connection layer. Figure 18 shows the network structure.Figure 18
Improved Inception-v3 network structure diagram.During the production and processing of BR, all the processing procedures are completed by experienced workers through visual assessment, which ensures processing quality and facilitates the adjustment of the equipment. Human visual assessment is based on a general feeling and cannot accurately determine the germ integrity. To more accurately describe the germ integrity of BR and improve its quality, BR grains are classified by observation into eight types, ranging from germ-free BR to whole-germ BR. The classification of each type of BR is based on the germ integrity. Figure19 shows the results.Figure 19
Rice germ integrity classification standard.


## 3. Experimental Method

### 3.1. Segmentation Experiment

Connected BR grains were segmented using the improved background skeleton-based segmentation algorithm proposed in this study. Figure20 shows the segmentation results.Figure 20
(a) Binary image after segmentation. (b) Germ rice image after segmentation.In view of the limitations of the background skeleton-based segmentation algorithm for single rice grains, the original endpoint matching method was improved in this study. A segmentation experiment on multiple connected BR grains was performed by integrating the region growing algorithm. In addition, the image of the most complex BR grain connection condition was used in a comparison experiment. Figures21 and 22 show the experimental results for the segmentation of randomly distributed BR grains.Figure 21
(a) The unimproved endpoint matching effect picture. (b) The improved endpoint matching effect picture.Figure 22
(a) The unimproved segmentation effect picture. (b) The improved segmentation effect picture.Images are of various types of cereals in which rice grains are connected on a small scale. These images were processed using the classic range-transformation watershed algorithm, the edge detection algorithm with different operators, the region growing algorithm, a fully convolutional NN (FCNN), and the proposed algorithm. The ultimate segmentation results obtained using these four algorithms are shown in Figures23–27.Figure 23
The classic range-transformation watershed algorithm.Figure 24
The region growing algorithm.Figure 25
The edge detection algorithm with different operators.Figure 26
Fully convolutional neural network segmentation.Figure 27
The proposed algorithm.During the extraction of single BR grains, operations, such as image rotation, were performed. As a result, the BR grains were horizontal throughout the process. Figure28 shows the results for single rice grains.Figure 28
Single grain rice extraction image after segmentation.To examine the segmentation performance for various BR grains, 2000 grain samples were collected for three types of BR grains differing in LWR. The connected BR grains were segmented (25 images were collected for each 2000 grains). For the range-transformation watershed algorithm, the edge detection algorithm with different operators, the region growing algorithm, the FCNN, and the proposed algorithm, the number of correctly segmented BR grains and the segmentation time were determined. In addition, the accuracy of each algorithm was calculated. Correct segmentation should ensure that single BR grains are completely segmented and that there are no oversegmentation lines in the BR grains. Tables1–6 summarize the results.Table 1
The classic range-transformation watershed algorithm segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000139669.844.28Short2000153276.642.21Intermediate2000142871.445.22Table 2
The region growing algorithm segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000157878.968.10Short2000170685.372.52Intermediate2000173686.876.63Table 3
The edge detection algorithm segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000105652.890.76Short2000124462.299.21Intermediate2000127663.892.61Table 4
Fully convolutional neural network segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000175287.642.36Short2000182891.445.69Intermediate2000187493.747.54Table 5
The proposed algorithm segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000184292.176.31Short2000190695.372.95Intermediate2000195097.571.44Table 6
Test results.
Germ integrity classResult/grainsThe results of the algorithm/grainsGerm integrity accuracy/%Average accuracy (%)0%∼12.5%70066795.295.5312.5%∼25%70067195.825%∼37.5%70066194.337.5%∼50%70066695.150%∼62.5%70067596.362.5%∼75%70066895.375%∼87.5%70067796.687.5%∼100%70067095.6Tables1–6 summarize the results. The average segmentation accuracies of the classic range-transformation watershed algorithm, the edge detection algorithm, the region growing algorithm, the FCNN, and the proposed algorithm for the three types of BR grains differing in LWR (long, short, and intermediate) were 72.6%, 83.67%, 90.9%, 91.3%, and 94.96%, respectively. The segmentation accuracy of the proposed algorithm was higher than those of the other segmentation algorithms. With regard to the computational time for the segmentation operation, of the three algorithms, the proposed algorithm required the longest computational time. This is mainly because the proposed algorithm involves a relatively large number of steps in the segmentation operation and is more complex, thus requiring more computational time in processing. Because of this, the proposed algorithm effectively inhibited erroneous segmentation, had a higher segmentation accuracy, and provided more valid images of BR grains for the subsequent extraction of single BR grains.


### 3.2. Identification Experiment

#### 3.2.1. Germ Direction Identification Experiment

The 6,000 BR grains obtained after image segmentation were used as a training set. A total of 500 training iterations using the VGG model were performed. In addition, the accuracy of the VGG model was examined by k-fold cross-validation. The accuracy of the VGG model was found to be 98.6%, meeting the accuracy requirements for germ position and direction identification.


#### 3.2.2. Germ Integrity Identification Experiment

First, the position of the germ of each single BR grain was determined using the germ position identification model. Then, the BR grains were rotated so that their germs were in the same direction.A total of 500 training iterations were performed using the improved Inception-v3 model. Afterwards, training was performed in intervals of 10. The accuracy of the improved Inception-v3 model was examined based on the test set. Ultimately, the error rate on the test set converged to 5.9%. After 80 training iterations, there was a significant decrease in the rate of convergence. After 500 training iterations, k-fold cross-validation showed that the identification accuracy of the optimized Inception-v3 model was 95.3%.First, an experiment was conducted to examine the germ integrity of single BR grains. In the experiment, Daohuaxiang-2 BR grains with various levels of germ integrity were used as samples. The whole batch of BR grains was first classified by visual observation according to the eight-type BR classification standard. For each type, 700 grains were selected for testing. A total of 5600 BR grain samples were used. The number of BR grains that could be collected from each image was limited. Therefore, after manual identification, the samples were divided into eight groups. Each group consisted of 25 BR grains. These samples were then subjected to image collection analysis. Image collection was performed a total of five times. Finally, the manual classification and identification results were compared with the BR test results. Table6 summarizes the experimental comparison results.In this section, the optimized Inception-v3 model was used to identify the germ integrity. To demonstrate its advantages, the optimized Inception-v3 model was compared with several currently popular deep network identification models (VGGnet16, VGGnet19, Inception-v3, and ResNet50). The performance of these models was evaluated. In addition, a comparison with the original dataset was performed. Table7 summarizes the experimental comparison results.Table 7
Contrast test results.
Types of riceExperiment methodRecallPrecisionAccuracyTraining samplesLongVGGnet1683.6982.3482.912000VGGnet1990.5490.3289.982000Inception-v392.2192.5591.882000ResNet5092.1192.3491.862000Optimized Inception-v393.7394.1694.832000ShortVGGnet1683.5683.2483.182000VGGnet1991.2191.4891.332000Inception-v392.7993.1291.932000ResNet5089.1590.4489.872000Optimized Inception-v395.5694.9795.532000IntermediateVGGnet1683.8683.1683.542000VGGnet1986.8587.6686.822000Inception-v392.7791.6891.722000ResNet5091.3692.6392.492000Optimized Inception-v394.8793.6994.562000A comparison of the models in Table7 shows that the identification accuracy (94.97%) of the optimized Inception-v3 model was higher than those of the other models. As the initialization parameters for training the BR identification model, this model learned domain knowledge from large amounts of easily accessible same-domain data. This addressed the problem of insufficient training data for the BR identification task and further improved the identification accuracy.Finally, an overall and comprehensive image segmentation and identification experiment was performed to examine the germ integrity. The segmentation algorithm was used to determine the position of each BR grain (first segment according to the background skeleton, and then recognize through the improved Inception-v3 network (BSF)). The BSF network was compared with faster R-CNN, SSD, and YOLO-v3 in terms of accuracy. Table8 summarizes the experimental comparison results.Table 8
Contrast test results.
Types of riceNetworkRecallPrecisionAccuracyTraining samplesLongFaster R-CNN77.5991.2392.162000YOLO v388.1891.7991.562000SSD (VGG-16)90.1692.3492.092000BSF network92.1094.1694.692000ShortFaster R-CNN79.3891.5291.362000YOLO v389.4991.9291.552000SSD (VGG-16)90.2692.7192.692000BSF network95.3094.1694.252000IntermediateFaster R-CNN79.8192.2292.392000YOLO v389.5792.6692.912000SSD (VGG-16)90.8692.8993.212000BSF network97.5094.8795.532000A comparison of the methods in the above table shows that the identification accuracy (94.83%) of the BSF network was higher than those of the other algorithms. As a result, the BSF network performed relatively well in image segmentation. The recall rate of the BSF network was far higher than those of the other detection algorithms.


## 3.1. Segmentation Experiment

Connected BR grains were segmented using the improved background skeleton-based segmentation algorithm proposed in this study. Figure20 shows the segmentation results.Figure 20
(a) Binary image after segmentation. (b) Germ rice image after segmentation.In view of the limitations of the background skeleton-based segmentation algorithm for single rice grains, the original endpoint matching method was improved in this study. A segmentation experiment on multiple connected BR grains was performed by integrating the region growing algorithm. In addition, the image of the most complex BR grain connection condition was used in a comparison experiment. Figures21 and 22 show the experimental results for the segmentation of randomly distributed BR grains.Figure 21
(a) The unimproved endpoint matching effect picture. (b) The improved endpoint matching effect picture.Figure 22
(a) The unimproved segmentation effect picture. (b) The improved segmentation effect picture.Images are of various types of cereals in which rice grains are connected on a small scale. These images were processed using the classic range-transformation watershed algorithm, the edge detection algorithm with different operators, the region growing algorithm, a fully convolutional NN (FCNN), and the proposed algorithm. The ultimate segmentation results obtained using these four algorithms are shown in Figures23–27.Figure 23
The classic range-transformation watershed algorithm.Figure 24
The region growing algorithm.Figure 25
The edge detection algorithm with different operators.Figure 26
Fully convolutional neural network segmentation.Figure 27
The proposed algorithm.During the extraction of single BR grains, operations, such as image rotation, were performed. As a result, the BR grains were horizontal throughout the process. Figure28 shows the results for single rice grains.Figure 28
Single grain rice extraction image after segmentation.To examine the segmentation performance for various BR grains, 2000 grain samples were collected for three types of BR grains differing in LWR. The connected BR grains were segmented (25 images were collected for each 2000 grains). For the range-transformation watershed algorithm, the edge detection algorithm with different operators, the region growing algorithm, the FCNN, and the proposed algorithm, the number of correctly segmented BR grains and the segmentation time were determined. In addition, the accuracy of each algorithm was calculated. Correct segmentation should ensure that single BR grains are completely segmented and that there are no oversegmentation lines in the BR grains. Tables1–6 summarize the results.Table 1
The classic range-transformation watershed algorithm segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000139669.844.28Short2000153276.642.21Intermediate2000142871.445.22Table 2
The region growing algorithm segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000157878.968.10Short2000170685.372.52Intermediate2000173686.876.63Table 3
The edge detection algorithm segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000105652.890.76Short2000124462.299.21Intermediate2000127663.892.61Table 4
Fully convolutional neural network segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000175287.642.36Short2000182891.445.69Intermediate2000187493.747.54Table 5
The proposed algorithm segmentation result.
Types of riceNumber of rice grains/grainNumber of correctly segmented grains/grainAccuracy (%)Segmentation total time/sLong2000184292.176.31Short2000190695.372.95Intermediate2000195097.571.44Table 6
Test results.
Germ integrity classResult/grainsThe results of the algorithm/grainsGerm integrity accuracy/%Average accuracy (%)0%∼12.5%70066795.295.5312.5%∼25%70067195.825%∼37.5%70066194.337.5%∼50%70066695.150%∼62.5%70067596.362.5%∼75%70066895.375%∼87.5%70067796.687.5%∼100%70067095.6Tables1–6 summarize the results. The average segmentation accuracies of the classic range-transformation watershed algorithm, the edge detection algorithm, the region growing algorithm, the FCNN, and the proposed algorithm for the three types of BR grains differing in LWR (long, short, and intermediate) were 72.6%, 83.67%, 90.9%, 91.3%, and 94.96%, respectively. The segmentation accuracy of the proposed algorithm was higher than those of the other segmentation algorithms. With regard to the computational time for the segmentation operation, of the three algorithms, the proposed algorithm required the longest computational time. This is mainly because the proposed algorithm involves a relatively large number of steps in the segmentation operation and is more complex, thus requiring more computational time in processing. Because of this, the proposed algorithm effectively inhibited erroneous segmentation, had a higher segmentation accuracy, and provided more valid images of BR grains for the subsequent extraction of single BR grains.


## 3.2. Identification Experiment

### 3.2.1. Germ Direction Identification Experiment

The 6,000 BR grains obtained after image segmentation were used as a training set. A total of 500 training iterations using the VGG model were performed. In addition, the accuracy of the VGG model was examined by k-fold cross-validation. The accuracy of the VGG model was found to be 98.6%, meeting the accuracy requirements for germ position and direction identification.


### 3.2.2. Germ Integrity Identification Experiment

First, the position of the germ of each single BR grain was determined using the germ position identification model. Then, the BR grains were rotated so that their germs were in the same direction.A total of 500 training iterations were performed using the improved Inception-v3 model. Afterwards, training was performed in intervals of 10. The accuracy of the improved Inception-v3 model was examined based on the test set. Ultimately, the error rate on the test set converged to 5.9%. After 80 training iterations, there was a significant decrease in the rate of convergence. After 500 training iterations, k-fold cross-validation showed that the identification accuracy of the optimized Inception-v3 model was 95.3%.First, an experiment was conducted to examine the germ integrity of single BR grains. In the experiment, Daohuaxiang-2 BR grains with various levels of germ integrity were used as samples. The whole batch of BR grains was first classified by visual observation according to the eight-type BR classification standard. For each type, 700 grains were selected for testing. A total of 5600 BR grain samples were used. The number of BR grains that could be collected from each image was limited. Therefore, after manual identification, the samples were divided into eight groups. Each group consisted of 25 BR grains. These samples were then subjected to image collection analysis. Image collection was performed a total of five times. Finally, the manual classification and identification results were compared with the BR test results. Table6 summarizes the experimental comparison results.In this section, the optimized Inception-v3 model was used to identify the germ integrity. To demonstrate its advantages, the optimized Inception-v3 model was compared with several currently popular deep network identification models (VGGnet16, VGGnet19, Inception-v3, and ResNet50). The performance of these models was evaluated. In addition, a comparison with the original dataset was performed. Table7 summarizes the experimental comparison results.Table 7
Contrast test results.
Types of riceExperiment methodRecallPrecisionAccuracyTraining samplesLongVGGnet1683.6982.3482.912000VGGnet1990.5490.3289.982000Inception-v392.2192.5591.882000ResNet5092.1192.3491.862000Optimized Inception-v393.7394.1694.832000ShortVGGnet1683.5683.2483.182000VGGnet1991.2191.4891.332000Inception-v392.7993.1291.932000ResNet5089.1590.4489.872000Optimized Inception-v395.5694.9795.532000IntermediateVGGnet1683.8683.1683.542000VGGnet1986.8587.6686.822000Inception-v392.7791.6891.722000ResNet5091.3692.6392.492000Optimized Inception-v394.8793.6994.562000A comparison of the models in Table7 shows that the identification accuracy (94.97%) of the optimized Inception-v3 model was higher than those of the other models. As the initialization parameters for training the BR identification model, this model learned domain knowledge from large amounts of easily accessible same-domain data. This addressed the problem of insufficient training data for the BR identification task and further improved the identification accuracy.Finally, an overall and comprehensive image segmentation and identification experiment was performed to examine the germ integrity. The segmentation algorithm was used to determine the position of each BR grain (first segment according to the background skeleton, and then recognize through the improved Inception-v3 network (BSF)). The BSF network was compared with faster R-CNN, SSD, and YOLO-v3 in terms of accuracy. Table8 summarizes the experimental comparison results.Table 8
Contrast test results.
Types of riceNetworkRecallPrecisionAccuracyTraining samplesLongFaster R-CNN77.5991.2392.162000YOLO v388.1891.7991.562000SSD (VGG-16)90.1692.3492.092000BSF network92.1094.1694.692000ShortFaster R-CNN79.3891.5291.362000YOLO v389.4991.9291.552000SSD (VGG-16)90.2692.7192.692000BSF network95.3094.1694.252000IntermediateFaster R-CNN79.8192.2292.392000YOLO v389.5792.6692.912000SSD (VGG-16)90.8692.8993.212000BSF network97.5094.8795.532000A comparison of the methods in the above table shows that the identification accuracy (94.83%) of the BSF network was higher than those of the other algorithms. As a result, the BSF network performed relatively well in image segmentation. The recall rate of the BSF network was far higher than those of the other detection algorithms.


## 3.2.1. Germ Direction Identification Experiment

The 6,000 BR grains obtained after image segmentation were used as a training set. A total of 500 training iterations using the VGG model were performed. In addition, the accuracy of the VGG model was examined by k-fold cross-validation. The accuracy of the VGG model was found to be 98.6%, meeting the accuracy requirements for germ position and direction identification.


## 3.2.2. Germ Integrity Identification Experiment

First, the position of the germ of each single BR grain was determined using the germ position identification model. Then, the BR grains were rotated so that their germs were in the same direction.A total of 500 training iterations were performed using the improved Inception-v3 model. Afterwards, training was performed in intervals of 10. The accuracy of the improved Inception-v3 model was examined based on the test set. Ultimately, the error rate on the test set converged to 5.9%. After 80 training iterations, there was a significant decrease in the rate of convergence. After 500 training iterations, k-fold cross-validation showed that the identification accuracy of the optimized Inception-v3 model was 95.3%.First, an experiment was conducted to examine the germ integrity of single BR grains. In the experiment, Daohuaxiang-2 BR grains with various levels of germ integrity were used as samples. The whole batch of BR grains was first classified by visual observation according to the eight-type BR classification standard. For each type, 700 grains were selected for testing. A total of 5600 BR grain samples were used. The number of BR grains that could be collected from each image was limited. Therefore, after manual identification, the samples were divided into eight groups. Each group consisted of 25 BR grains. These samples were then subjected to image collection analysis. Image collection was performed a total of five times. Finally, the manual classification and identification results were compared with the BR test results. Table6 summarizes the experimental comparison results.In this section, the optimized Inception-v3 model was used to identify the germ integrity. To demonstrate its advantages, the optimized Inception-v3 model was compared with several currently popular deep network identification models (VGGnet16, VGGnet19, Inception-v3, and ResNet50). The performance of these models was evaluated. In addition, a comparison with the original dataset was performed. Table7 summarizes the experimental comparison results.Table 7
Contrast test results.
Types of riceExperiment methodRecallPrecisionAccuracyTraining samplesLongVGGnet1683.6982.3482.912000VGGnet1990.5490.3289.982000Inception-v392.2192.5591.882000ResNet5092.1192.3491.862000Optimized Inception-v393.7394.1694.832000ShortVGGnet1683.5683.2483.182000VGGnet1991.2191.4891.332000Inception-v392.7993.1291.932000ResNet5089.1590.4489.872000Optimized Inception-v395.5694.9795.532000IntermediateVGGnet1683.8683.1683.542000VGGnet1986.8587.6686.822000Inception-v392.7791.6891.722000ResNet5091.3692.6392.492000Optimized Inception-v394.8793.6994.562000A comparison of the models in Table7 shows that the identification accuracy (94.97%) of the optimized Inception-v3 model was higher than those of the other models. As the initialization parameters for training the BR identification model, this model learned domain knowledge from large amounts of easily accessible same-domain data. This addressed the problem of insufficient training data for the BR identification task and further improved the identification accuracy.Finally, an overall and comprehensive image segmentation and identification experiment was performed to examine the germ integrity. The segmentation algorithm was used to determine the position of each BR grain (first segment according to the background skeleton, and then recognize through the improved Inception-v3 network (BSF)). The BSF network was compared with faster R-CNN, SSD, and YOLO-v3 in terms of accuracy. Table8 summarizes the experimental comparison results.Table 8
Contrast test results.
Types of riceNetworkRecallPrecisionAccuracyTraining samplesLongFaster R-CNN77.5991.2392.162000YOLO v388.1891.7991.562000SSD (VGG-16)90.1692.3492.092000BSF network92.1094.1694.692000ShortFaster R-CNN79.3891.5291.362000YOLO v389.4991.9291.552000SSD (VGG-16)90.2692.7192.692000BSF network95.3094.1694.252000IntermediateFaster R-CNN79.8192.2292.392000YOLO v389.5792.6692.912000SSD (VGG-16)90.8692.8993.212000BSF network97.5094.8795.532000A comparison of the methods in the above table shows that the identification accuracy (94.83%) of the BSF network was higher than those of the other algorithms. As a result, the BSF network performed relatively well in image segmentation. The recall rate of the BSF network was far higher than those of the other detection algorithms.


## 4. Conclusions

In this study, the background skeleton-based algorithm for image segmentation is improved. Besides, segmented endpoints are obtained by detecting skeleton features of the background images of connected BR grains. Meanwhile, adopting segmentation endpoint matching algorithm is beneficial for achieving segmentation successfully. This paper addresses image problems of connected BR grains. And the algorithm accuracy applied to image segmentation of connected BR grains in various shapes is above 96% as presented in this paper. In addition, this algorithm is suitable for segmenting images with large regions of connected BR grains and is highly robust. Moreover, compared to the classic segmentation algorithm, this algorithm is suitable for relatively wide BR grains.In addition, two networks are proposed to identify germ integrity. A position network model is used to unify the directions of BR germs. Furthermore, an algorithm based on an improved Inception-v3 model is proposed for determining BR germ integrity. The network outputs are divided into eight categories based on their germ integrity. The accuracy of the proposed algorithm is higher than those of other NN algorithms. The comprehensive accuracy of the proposed algorithm is 94.83%. This algorithm can effectively suppress interference from the endosperm surface of BR.Finally, an overall detection function is achieved based on background skeleton features and the deep learning network algorithm. First, identifying a large number of connected BR grains can ensure high accuracy performance. Second, this can also help rapidly and accurately determine the positions of BR grains. In the experiment, this method was found effective in identifying the germ integrity of a large number of BR grains.



---

*Source: 6709787-2022-03-27.xml*