# Two-Stage Intelligent DarkNet-SqueezeNet Architecture-Based Framework for Multiclass Rice Grain Variety Identification

Authors: Maryam Fatima; Muhammad Attique Khan; Muhammad Sharif; Majed Alhaisoni; Abdullah Alqahtani; Usman Tariqe; Ye Jin Kim; Byoungchol Chang
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Image processing is an important domain for identifying various crop varieties. Due to the large amount of rice and its varieties, manually detecting its qualities is a very tedious and time-consuming task. In this work, we propose a two-stage deep learning framework for detecting and classifying multiclass rice grain varieties. A series of steps is included in the proposed framework. The first step is to perform preprocessing on the selected dataset. The second step involves selecting and fine-tuning pretrained deep models from Darknet19 and SqueezeNet. Transfer learning is used to train the fine-tuned models on the selected dataset. The 50% sample images are employed for the training and rest 50% are used for the testing. Features are extracted and fused using a maximum correlation-based approach. This approach improved the classification performance; however, redundant information has also been included. An improved butterfly optimization algorithm (BOA) is proposed, in the next step, for the selection of the best features that are finally classified using several machine learning classifiers. The experimental process was conducted on selected rice datasets that include five types of rice varieties and achieves a maximum accuracy of 100% that was improved than the recent method. The average accuracy of the proposed method is obtained at 99.2%, through confidence interval-based analysis that shows the significance of this work.


---

## Body

## 1. Introduction

In the field of agriculture, rice is an unavoidable staple diet and the most used around the world [1]. In addition, it is the third most cultivated field in different areas, including China, East Asia, and the south [2, 3]. Approximately 90% of Asians prefer rice food, and their demand for rice increases day by day [4]. Myanmar is the largest rice exporter in the world and the sixth-largest rice-producing country [5]. Pakistan exports 8% of its rice to the world and the export rate is dependent on the quality of the rice [6]. To estimate the characteristic of a rice grain, various physical variables are utilized including shape, size, and texture, which are the most helpful feature to determine the quality of grain through image processing [7]. In other ways, it consumes much time in most procedures and despite that results were not better or more accurate, but image processing is a better option for identification of rice varieties [8]. The actual goal is to detect the rice grain quality within less time and cost.Both physical and chemical methods are used for seed varieties of rice. In other words, destructive and nondestructive methods are also used for solving the rice problems [9]. These nondestructive methods are better as compared to destructive methods because nondestructive methods use the digital image processing (DIP) process, and now, people want to detect rice grain with less time and at cheaper rates. Stone, weed seeds, chaff, etc. are the main causes of damaging the seed quality [10]. Flatbed scanners are used for rice grading and inspection in which rice kernels, glass plate, and scanner head are used. The scanner detects the size, shape, density, and percentage of broken kernels [11]. Rice has different types. C4Raja rice [12], and kernel rice features depend on length, width, and perimeters [13]. Several rice types, such as Chenab Basmati, Kissan Basmati, Basmati 2000, KSK 133, KSK 434, Punjab Basmati, and PK 1121 aromatic, are mostly grown in Pakistan, are utilized for rice seed categorization, and have been gathered [14]. Some varieties are shown in Figure 1.Figure 1
Rice varieties: (a) Khazar, (b) Gharib, (c) Ghasrdashti, (d) Gerdeh, and (e) Mohammadi [15].
(a)(b)(c)(d)(e)Rice grain quality and their varieties is an important factor in cultivation and import/export [16]. Manual labor and some machines work to solve these rice grain problems. Current machines are costly, so, to solve the quality problem, humans propose a machine vision algorithm [17] to identify the rice grain problem and solve them through classification method and based on images and videos, which is easy and fast in the detection of color, shape, and texture feature [18]. Different classification methods are used to classify rice into different groups [1]. On the contrary, automatic detection of the rice grain is a challenging task in the farming industry. Different applications of the technology are used for the automatic identification of rice grains [19]. To remove the impurities from the rice, different methods of DIP and probabilistic neural network (PNN) algorithms are utilized. To determine the grain’s moisture content, machine learning (ML) techniques are applied [20]. Moreover, morphological methods are much important and play a significant part in determining the rice’s quality.In the field of agriculture, identifying and classifying a rice grain are one of the most interesting and demanding tasks. In the ongoing years, different machine learning-based approaches have been introduced. However, still, there are some gaps that need to be resolved in this area [21]. Several techniques have been introduced in the literature for rice variety detection and classification using computer vision and machine learning techniques [22]. Normally, they perform preprocessing techniques at the initial phase for better detection [23]. Robert Singh and Chaudhury [24] presented texture, wavelet, morphological, and color features which were used to classify the four different types of rice grain images that are taken using a 12-megapixel camera. The HSI channel has been used for image conversion and then these images (grayscale) converted into a binary image. Sethi et al. [6] used adaptive thresholding to convert the image into grayscale. Low pass and Gaussian filters are also used to smooth and denoise the image. Masking is used for highlighting the rice object. Histogram equalization (HE) is reviewing the gray level of the image, and image enhancement gives the idea that which techniques are applied to the grayscale and color image [25]. In the preprocessing process, the auto-align method is available to automatically align the arrangement and positioning of grain. The exact position around the grain used the contour method [26]. Wongruen [27] utilized four techniques based on segmentation such as fast K-means of Dixit, fuzzy C-means (FC), multilevel thresholding, and Otsu’s multilevel thresholding (OMT). The OMT’s and FC accuracy rates are 89.83% and 95.53%. Itharatet al. [28] used Khao Dawk MAli 105 dataset and applied global thresholding to segment the chalk area on images of rice. RiceNet-based segmentation has improved the quality of adhesive rice grain. We compare the result with region-based convolutional neural network (R-CNN) and give an 89.5% accuracy [29]. Bhupendra et al. [30] presented a deep learning architecture-based framework for rice grains’ classification. They used several CNN models, and EfficientNetB0 successfully attained the accuracy of 98.32%. Moreover, classification is performed for several other fruit varieties by computer vision researchers such as date fruit [31], Pistachio Species [32], corn [33], and coffee beans [34].In the abovementioned studies, the major problem is the appearance variation of several rice varieties. The similar shape, texture, irregularity, and rice width affect the rice segmentation accuracy. Furthermore, the incorrect segmentation step produced irrelevant features that later degrade the classification accuracy. Furthermore, in the classification phase, the presence of a few redundant and irrelevant features impacts the proposed framework’s computational time (during the prediction process) and can decrease the classification accuracy. Another challenge is the selection of a classifier for final accuracy because several classifiers exist and many of them consumed a lot of time for the prediction process. Therefore, we proposed a two-stage framework for rice variety detection and classification. Our major contributions are as follows:(i)
We proposed a rice segmentation technique based on a color-based saliency map and thresholding(ii)
Two pretrained deep models have been fine-tuned and have fused their information based on maximum correlation(iii)
Best features are selected using an improved butterfly optimization algorithm


## 2. Proposed Methodology

In the proposed work, the main steps are the acquisition of a dataset, deep saliency-based rice segmentation, deep features extraction using pretrained models trained on the selected dataset, feature fusion, feature selection using an optimization technique, and finally classification through machine learning classifiers. The main flow of the proposed framework is illustrated in Figure2. In Figure 2, it is noted that the saliency map is constructed that further processed in the form of a color map. Mathematically, each substep presented in this framework has been discussed below.Figure 2
Proposed deep learning and optimization-based framework of rice variety classification.


### 2.1. Saliency-Based Rice Segmentation

Segmentation is an important step in the domain of computer vision using plant application [35]. In this study, we introduced a saliency-based approach using Wang and Peng [36] technique. In this technique, we initially picked the superpixel image for the saliency method and select top, left, and right pixel for background seeds, the vector Bv = B1,B2,…,BnZ, and their values Bp=1 and Bp=0, this represents the labeled and unlabeled nodes, whereas p represents superpixel. The results of this method are shown in Figure 3. The background saliency is estimated as follows:(1)Iv=K−∝EJ1−1MJ1−1EJ1−1/2Bv,v=q,u,x,Ibp=1−normIqp.Iup.Ixp.Figure 3
(a) Original data. (b) Proposed saliency-based segmented images.
(a)(b)Diffusion-based compactness is defined as follows:(2)Ifp=1−normIvp+Idp,where Ivp is the spatial variance of superpixel and Idp is the spatial distance of superpixel. Later on, we combined the background saliency and compactness as follows:(3)I1p=Ibp+Ifp.We jointly take into consideration the initial saliency values into the superior affinity network with the goal of creating a perspective of coarse-to-fine optimization to diffuse the first saliency map. Second, the robust affinity graphMJ2 is computed as follows:(4)mijJ2=mijJ1+mijy∙mijIf∙mijyZ,whereas mijIf is the affinity graph. We insert this graph into the manifold ranking objective function in order to achieve our goal of working the coarse-to-fine saliency result I2:(5)Iv=K−∝EJ2−1MJ2−1EJ2−1/2I2.The lab color transformation is applied onI2, and the color mapped images are obtained which are later utilized for the training of pretrained deep learning models.


### 2.2. Pretrained Deep Models

In this work, two pretrained deep models have been employed for the training on rice dataset. The selected datasets are SqueezeNet and DarkNet19.


#### 2.2.1. SqueezeNet Model

A convolutional neural network of 18 layers deep is called SqueezeNet. The ImageNet database contains a pretrained version of the network that has been trained on almost a million photos. Figure4 shows SqueezeNet architecture and 3 CNNs highlighting the subset of layers. The SqueezeNet model is used for feature extraction. Final layer activation is dependent on the classification module, and these layers are complicated networks. There are two layers of the classification module: the global pooling layer and the convolutional layer. The remaining 2 layers are arbitrarily and pretrained on ImageNet. The detailed representation of the SqueezeNet model is shown in Table 1. Classification is performed on images through pretrained networks.Figure 4
SqueezeNet architecture.Table 1
Description of SqueezeNet architecture.
Layer name/typeOutput sizeFilter size/stride(if not a fire layer)DepthS1 × 1 (#1 × 1 squeeze)e1 × 1 (#1 × 1 expand)e3 × 3 (#3 × 3expand)Input image224 × 224Conv1111 × 111 × 967 × 7/ (x96)1Maxpool155 × 55 × 963 × 3/20Fire 255 × 55 × 1282166464Fire 355 × 55 × 1282166464Fire 455 × 55 × 256232128128Maxpool427 × 27 × 2563 × 3/20Fire 527 × 27 × 256232128128Fire 627 × 27 × 384248192192Fire 727 × 27 × 384248192192Fire 827 × 27 × 512264256256Maxpool813 × 12 × 5123 × 3/20Fire 913 × 13 × 512264256256Conv1013 × 13 × 51 × 1/1 (×5)1Avgpool101 × 1 × 513 × 13/10


#### 2.2.2. DarkNet19

DarkNet19 is the convolutional neural network. We used the DarkNet19 model, in which 19 convolutional layers are known as DarkNet19. There are five max-pooling layers and 19 convolutional layers (CL) in DarkNet19, with several 1 × 1 CL to the minimum triangle parameters involving 3 × 3. We train the pretrained network by using a transfer learning technique that is applied to 75,000 images which provide the features of the network. In this stage, DarkNet19 was used to classify the images into 1000 classes, but we used only 5 classes for the results. Moreover, the image size of the input layer of DarkNet19 is 256 × 256. The view of the architecture neural network of DarkNet19 is shown in Figure5. This network accepts the input of dimensional 244 × 244. The value of parameters and more details of the architecture neural network of DarkNet19 are shown in Table 2.Figure 5
Architecture of DarkNet19.Table 2
Description of DarkNet19 architecture.
TypeFiltersSize/strideOutputConvolutional343 × 3224 × 2 24Maxpool2 × 2/2112 × 112Convolutional643 × 3112 × 112Maxpool2 × 2/256 × 56Convolutional1283 × 356 × 56Convolutional641 × 156 × 56Convolutional1283 × 356 × 56Maxpool2 × 2/228 × 28Convolutional2563 × 328 × 28Convolutional1281 × 128 × 28Convolutional2563 × 328 × 28Maxpool2 × 2/214 × 14Convolutional5123 × 314 × 14Convolutional2561 × 114 × 14Convolutional5123 × 314 × 14Convolutional2561 × 114 × 14Convolutional5123 × 314 × 14Maxpool2 × 2/27 × 7Convolutional10243 × 37 × 7Convolutional5121 × 17 × 7Convolutional10243 × 37 × 7Convolutional5121 × 17 × 7Convolutional10243 × 37 × 7Convolutional51 × 17 × 7New_softmaxGlobal5


### 2.3. Transfer Learning

Transfer learning is used to improve the efficiency of the process and reduce the number of resources required. When elements of a pretrained machine learning model are reused in a new machine learning model, this is known as transfer learning. In transfer learning, we define feature vector and probability distribution asA=fv,Pfv and fv=v1,v2,…,vn, in which, the ground truth G=g1,g2,…,gn and objective function O=G,lx, whereas lx is unknown label class. Pg|x is a probabilistic representation of the function. Transfer learning and learning rate are denoted as To and Lo. Tf will be used to show the targeted function and the targeted output. The main goal of transfer learning is to improve the learning rate for predicting the targeted item using the recognition function (lx) depending upon the training learned from To and Tf, where To≠Tf and Lo≠Tf. Pattern recognition is improved via inductive transfer learning. When using inductive transfer learning, you will need an annotated database for fast training and testing. Visually, the process of transfer learning for the training of model on rice images is illustrated in Figure 6.Figure 6
Deep transfer learning.


### 2.4. Features’ Extraction and Fusion

Two feature vectors have been extracted from newly trained fine-tuned deep models. The first feature vector is extracted from SqueezeNet fine-tuned model and performed activation on second last feature layer. On this layer, 1024 features are extracted for each image. The second feature vector is extracted from DarkNet19 fine-tuned model and performed activation on second last convolutional layer. On this layer, 1000 features are extracted for each image. We fused both feature vectors using a proposed maximum correlation-based approach that combine the important information from both vectors.We consider two feature vectorsFv1Dar, and Fv2Squ, and fusion vector represented as Fusv. The dimension of these vectors is R×N, where N is represented as the number of images. Each one has a vector length of R×1024 and R×1000, respectively. The following formulation is performed to compute the correlation coefficient between two characteristics:(6)fDar,Squ=COVDar,SquVarDarVarSqu.The range of these values,Dar and Squ, lies between (−1 and 1). -1 stands for weak correlation, and 1 stands for strong correlation. The equation of maximum correlation vector is defined as follows:(7)CVDar,Squ=φfm1Dar,m2Squwhere φ denotes the supremum of the overall Borel functions [37], Squ: ω⟶ω is located between (0 and 1), and CVDar,Squ is the maximum correlation. If the correlation is close to 1, then we drop it; if it is 0, then we discard both features through fused vector. Based on this formulation, a fusion vector is obtained having dimension N×931 that later processes in the optimization algorithm for the best feature selection.


### 2.5. Improved Butterfly Optimization Algorithm-Based Feature Selection

A technique for reducing the input variable to your model is feature selection, which involves removing noise and using just useful data. It involves selecting the best characteristics for your machine learning model in accordance with the kind of issue you are seeking to tackle automatically. The butterfly optimization algorithm (BOA) is the metaheuristic algorithm [38]. Metaheuristic is a relatively new type of optimization algorithm that is used to find the best solution and is influenced by the food investigating behavior of butterflies. Initially, the physical stimulus intensity work on the perceived intensity of the fragrance (f) is defined as follows:(8)f=eLd,where e, d, and L are sensory mandatory, stimulus intensity, and power exponent that depends on modality. The best global butterfly position can be represented as(9)Pjm+1=Pjm+rn2×bp−Pjm×fj,where Pjm is jth butterfly position at time m and rn is the random number [0,1]. fj is the fragrance of jth butterfly and bp is the best current position. The local walk is defined as(10)Pjm+1=Pjm+rn2×Pkm−Plm×fj,where rn is the random number of between [0,1] and Pkm and Plm are used for kth and lth butterfly’s position. In the following step, the selected features based on the above equation are passed to the cross-entropy (CE) function. CE is used for the global search for butterfly’s movement. The following is an optimization problem:(11)z∈ZMinRz,where Z defined the final states and R is the real value performance function Z. An auxiliary problem and probability distribution estimation problem are described as follows:(12)Lγ=PmSR≤γ=EoFSR≤γ,where Pm is the probability measure, SR is the random state, Eo is the expected operator, γ show the thresholding parameter, and F shows the indicator function (the value of F is 1 if SR≤γ,0). In the CE method, the significance sampling methodology is employed to lower the sample size. As a result, the equation can be rewritten like this(13)Lγ=1n∑jnFSR≤γeyi;ahyi,where eyi;a represents the random sample and hyi represents optimal density that can be discovered by reducing the Kullback–Leibler divergence:(14)mina1n∑jnFSR≤γIneyi;a.Hence, the final optimization can be done as follows:(15)ak+1∗=∝a++1−∝ak∗,whereas the smoothing parameter is 0≤∝≤1. This improved algorithm continues until the number of iterations is not completed. In this work, the numbers of iterations are 200. At the end, we obtained a feature vector of dimensional N×522 which was finally classified using machine learning classifiers.


## 2.1. Saliency-Based Rice Segmentation

Segmentation is an important step in the domain of computer vision using plant application [35]. In this study, we introduced a saliency-based approach using Wang and Peng [36] technique. In this technique, we initially picked the superpixel image for the saliency method and select top, left, and right pixel for background seeds, the vector Bv = B1,B2,…,BnZ, and their values Bp=1 and Bp=0, this represents the labeled and unlabeled nodes, whereas p represents superpixel. The results of this method are shown in Figure 3. The background saliency is estimated as follows:(1)Iv=K−∝EJ1−1MJ1−1EJ1−1/2Bv,v=q,u,x,Ibp=1−normIqp.Iup.Ixp.Figure 3
(a) Original data. (b) Proposed saliency-based segmented images.
(a)(b)Diffusion-based compactness is defined as follows:(2)Ifp=1−normIvp+Idp,where Ivp is the spatial variance of superpixel and Idp is the spatial distance of superpixel. Later on, we combined the background saliency and compactness as follows:(3)I1p=Ibp+Ifp.We jointly take into consideration the initial saliency values into the superior affinity network with the goal of creating a perspective of coarse-to-fine optimization to diffuse the first saliency map. Second, the robust affinity graphMJ2 is computed as follows:(4)mijJ2=mijJ1+mijy∙mijIf∙mijyZ,whereas mijIf is the affinity graph. We insert this graph into the manifold ranking objective function in order to achieve our goal of working the coarse-to-fine saliency result I2:(5)Iv=K−∝EJ2−1MJ2−1EJ2−1/2I2.The lab color transformation is applied onI2, and the color mapped images are obtained which are later utilized for the training of pretrained deep learning models.


## 2.2. Pretrained Deep Models

In this work, two pretrained deep models have been employed for the training on rice dataset. The selected datasets are SqueezeNet and DarkNet19.


### 2.2.1. SqueezeNet Model

A convolutional neural network of 18 layers deep is called SqueezeNet. The ImageNet database contains a pretrained version of the network that has been trained on almost a million photos. Figure4 shows SqueezeNet architecture and 3 CNNs highlighting the subset of layers. The SqueezeNet model is used for feature extraction. Final layer activation is dependent on the classification module, and these layers are complicated networks. There are two layers of the classification module: the global pooling layer and the convolutional layer. The remaining 2 layers are arbitrarily and pretrained on ImageNet. The detailed representation of the SqueezeNet model is shown in Table 1. Classification is performed on images through pretrained networks.Figure 4
SqueezeNet architecture.Table 1
Description of SqueezeNet architecture.
Layer name/typeOutput sizeFilter size/stride(if not a fire layer)DepthS1 × 1 (#1 × 1 squeeze)e1 × 1 (#1 × 1 expand)e3 × 3 (#3 × 3expand)Input image224 × 224Conv1111 × 111 × 967 × 7/ (x96)1Maxpool155 × 55 × 963 × 3/20Fire 255 × 55 × 1282166464Fire 355 × 55 × 1282166464Fire 455 × 55 × 256232128128Maxpool427 × 27 × 2563 × 3/20Fire 527 × 27 × 256232128128Fire 627 × 27 × 384248192192Fire 727 × 27 × 384248192192Fire 827 × 27 × 512264256256Maxpool813 × 12 × 5123 × 3/20Fire 913 × 13 × 512264256256Conv1013 × 13 × 51 × 1/1 (×5)1Avgpool101 × 1 × 513 × 13/10


### 2.2.2. DarkNet19

DarkNet19 is the convolutional neural network. We used the DarkNet19 model, in which 19 convolutional layers are known as DarkNet19. There are five max-pooling layers and 19 convolutional layers (CL) in DarkNet19, with several 1 × 1 CL to the minimum triangle parameters involving 3 × 3. We train the pretrained network by using a transfer learning technique that is applied to 75,000 images which provide the features of the network. In this stage, DarkNet19 was used to classify the images into 1000 classes, but we used only 5 classes for the results. Moreover, the image size of the input layer of DarkNet19 is 256 × 256. The view of the architecture neural network of DarkNet19 is shown in Figure5. This network accepts the input of dimensional 244 × 244. The value of parameters and more details of the architecture neural network of DarkNet19 are shown in Table 2.Figure 5
Architecture of DarkNet19.Table 2
Description of DarkNet19 architecture.
TypeFiltersSize/strideOutputConvolutional343 × 3224 × 2 24Maxpool2 × 2/2112 × 112Convolutional643 × 3112 × 112Maxpool2 × 2/256 × 56Convolutional1283 × 356 × 56Convolutional641 × 156 × 56Convolutional1283 × 356 × 56Maxpool2 × 2/228 × 28Convolutional2563 × 328 × 28Convolutional1281 × 128 × 28Convolutional2563 × 328 × 28Maxpool2 × 2/214 × 14Convolutional5123 × 314 × 14Convolutional2561 × 114 × 14Convolutional5123 × 314 × 14Convolutional2561 × 114 × 14Convolutional5123 × 314 × 14Maxpool2 × 2/27 × 7Convolutional10243 × 37 × 7Convolutional5121 × 17 × 7Convolutional10243 × 37 × 7Convolutional5121 × 17 × 7Convolutional10243 × 37 × 7Convolutional51 × 17 × 7New_softmaxGlobal5


## 2.2.1. SqueezeNet Model

A convolutional neural network of 18 layers deep is called SqueezeNet. The ImageNet database contains a pretrained version of the network that has been trained on almost a million photos. Figure4 shows SqueezeNet architecture and 3 CNNs highlighting the subset of layers. The SqueezeNet model is used for feature extraction. Final layer activation is dependent on the classification module, and these layers are complicated networks. There are two layers of the classification module: the global pooling layer and the convolutional layer. The remaining 2 layers are arbitrarily and pretrained on ImageNet. The detailed representation of the SqueezeNet model is shown in Table 1. Classification is performed on images through pretrained networks.Figure 4
SqueezeNet architecture.Table 1
Description of SqueezeNet architecture.
Layer name/typeOutput sizeFilter size/stride(if not a fire layer)DepthS1 × 1 (#1 × 1 squeeze)e1 × 1 (#1 × 1 expand)e3 × 3 (#3 × 3expand)Input image224 × 224Conv1111 × 111 × 967 × 7/ (x96)1Maxpool155 × 55 × 963 × 3/20Fire 255 × 55 × 1282166464Fire 355 × 55 × 1282166464Fire 455 × 55 × 256232128128Maxpool427 × 27 × 2563 × 3/20Fire 527 × 27 × 256232128128Fire 627 × 27 × 384248192192Fire 727 × 27 × 384248192192Fire 827 × 27 × 512264256256Maxpool813 × 12 × 5123 × 3/20Fire 913 × 13 × 512264256256Conv1013 × 13 × 51 × 1/1 (×5)1Avgpool101 × 1 × 513 × 13/10


## 2.2.2. DarkNet19

DarkNet19 is the convolutional neural network. We used the DarkNet19 model, in which 19 convolutional layers are known as DarkNet19. There are five max-pooling layers and 19 convolutional layers (CL) in DarkNet19, with several 1 × 1 CL to the minimum triangle parameters involving 3 × 3. We train the pretrained network by using a transfer learning technique that is applied to 75,000 images which provide the features of the network. In this stage, DarkNet19 was used to classify the images into 1000 classes, but we used only 5 classes for the results. Moreover, the image size of the input layer of DarkNet19 is 256 × 256. The view of the architecture neural network of DarkNet19 is shown in Figure5. This network accepts the input of dimensional 244 × 244. The value of parameters and more details of the architecture neural network of DarkNet19 are shown in Table 2.Figure 5
Architecture of DarkNet19.Table 2
Description of DarkNet19 architecture.
TypeFiltersSize/strideOutputConvolutional343 × 3224 × 2 24Maxpool2 × 2/2112 × 112Convolutional643 × 3112 × 112Maxpool2 × 2/256 × 56Convolutional1283 × 356 × 56Convolutional641 × 156 × 56Convolutional1283 × 356 × 56Maxpool2 × 2/228 × 28Convolutional2563 × 328 × 28Convolutional1281 × 128 × 28Convolutional2563 × 328 × 28Maxpool2 × 2/214 × 14Convolutional5123 × 314 × 14Convolutional2561 × 114 × 14Convolutional5123 × 314 × 14Convolutional2561 × 114 × 14Convolutional5123 × 314 × 14Maxpool2 × 2/27 × 7Convolutional10243 × 37 × 7Convolutional5121 × 17 × 7Convolutional10243 × 37 × 7Convolutional5121 × 17 × 7Convolutional10243 × 37 × 7Convolutional51 × 17 × 7New_softmaxGlobal5


## 2.3. Transfer Learning

Transfer learning is used to improve the efficiency of the process and reduce the number of resources required. When elements of a pretrained machine learning model are reused in a new machine learning model, this is known as transfer learning. In transfer learning, we define feature vector and probability distribution asA=fv,Pfv and fv=v1,v2,…,vn, in which, the ground truth G=g1,g2,…,gn and objective function O=G,lx, whereas lx is unknown label class. Pg|x is a probabilistic representation of the function. Transfer learning and learning rate are denoted as To and Lo. Tf will be used to show the targeted function and the targeted output. The main goal of transfer learning is to improve the learning rate for predicting the targeted item using the recognition function (lx) depending upon the training learned from To and Tf, where To≠Tf and Lo≠Tf. Pattern recognition is improved via inductive transfer learning. When using inductive transfer learning, you will need an annotated database for fast training and testing. Visually, the process of transfer learning for the training of model on rice images is illustrated in Figure 6.Figure 6
Deep transfer learning.


## 2.4. Features’ Extraction and Fusion

Two feature vectors have been extracted from newly trained fine-tuned deep models. The first feature vector is extracted from SqueezeNet fine-tuned model and performed activation on second last feature layer. On this layer, 1024 features are extracted for each image. The second feature vector is extracted from DarkNet19 fine-tuned model and performed activation on second last convolutional layer. On this layer, 1000 features are extracted for each image. We fused both feature vectors using a proposed maximum correlation-based approach that combine the important information from both vectors.We consider two feature vectorsFv1Dar, and Fv2Squ, and fusion vector represented as Fusv. The dimension of these vectors is R×N, where N is represented as the number of images. Each one has a vector length of R×1024 and R×1000, respectively. The following formulation is performed to compute the correlation coefficient between two characteristics:(6)fDar,Squ=COVDar,SquVarDarVarSqu.The range of these values,Dar and Squ, lies between (−1 and 1). -1 stands for weak correlation, and 1 stands for strong correlation. The equation of maximum correlation vector is defined as follows:(7)CVDar,Squ=φfm1Dar,m2Squwhere φ denotes the supremum of the overall Borel functions [37], Squ: ω⟶ω is located between (0 and 1), and CVDar,Squ is the maximum correlation. If the correlation is close to 1, then we drop it; if it is 0, then we discard both features through fused vector. Based on this formulation, a fusion vector is obtained having dimension N×931 that later processes in the optimization algorithm for the best feature selection.


## 2.5. Improved Butterfly Optimization Algorithm-Based Feature Selection

A technique for reducing the input variable to your model is feature selection, which involves removing noise and using just useful data. It involves selecting the best characteristics for your machine learning model in accordance with the kind of issue you are seeking to tackle automatically. The butterfly optimization algorithm (BOA) is the metaheuristic algorithm [38]. Metaheuristic is a relatively new type of optimization algorithm that is used to find the best solution and is influenced by the food investigating behavior of butterflies. Initially, the physical stimulus intensity work on the perceived intensity of the fragrance (f) is defined as follows:(8)f=eLd,where e, d, and L are sensory mandatory, stimulus intensity, and power exponent that depends on modality. The best global butterfly position can be represented as(9)Pjm+1=Pjm+rn2×bp−Pjm×fj,where Pjm is jth butterfly position at time m and rn is the random number [0,1]. fj is the fragrance of jth butterfly and bp is the best current position. The local walk is defined as(10)Pjm+1=Pjm+rn2×Pkm−Plm×fj,where rn is the random number of between [0,1] and Pkm and Plm are used for kth and lth butterfly’s position. In the following step, the selected features based on the above equation are passed to the cross-entropy (CE) function. CE is used for the global search for butterfly’s movement. The following is an optimization problem:(11)z∈ZMinRz,where Z defined the final states and R is the real value performance function Z. An auxiliary problem and probability distribution estimation problem are described as follows:(12)Lγ=PmSR≤γ=EoFSR≤γ,where Pm is the probability measure, SR is the random state, Eo is the expected operator, γ show the thresholding parameter, and F shows the indicator function (the value of F is 1 if SR≤γ,0). In the CE method, the significance sampling methodology is employed to lower the sample size. As a result, the equation can be rewritten like this(13)Lγ=1n∑jnFSR≤γeyi;ahyi,where eyi;a represents the random sample and hyi represents optimal density that can be discovered by reducing the Kullback–Leibler divergence:(14)mina1n∑jnFSR≤γIneyi;a.Hence, the final optimization can be done as follows:(15)ak+1∗=∝a++1−∝ak∗,whereas the smoothing parameter is 0≤∝≤1. This improved algorithm continues until the number of iterations is not completed. In this work, the numbers of iterations are 200. At the end, we obtained a feature vector of dimensional N×522 which was finally classified using machine learning classifiers.


## 3. Results and Discussion

### 3.1. Dataset

In this work, we utilized a publically available rice dataset that include five types of rice varieties such as (a) Khazar, (b) Gharib, (c) Ghasrdashti, (d) Gerdeh, and (e) Mohammadi [39]. A sample image is shown in Figure 1. This dataset consists of total 75,000 images and each class consists of 15,000 images (https://www.kaggle.com/datasets/muratkokludataset/rice-image-dataset). We employed 50% of the images from each class for training and rest for testing.


### 3.2. Experimental Setup

The experimental process of the proposed method is conducted on publically available rice imaging dataset [39], as discussed under Section 3.1. We performed the 50 : 50 approach for both training and testing, whereas the cross-validation value is 10. Four different experiments have been performed for the detailed analysis of the proposed method. In the first experiment, the DarkNet19 model is used after applying deep saliency-based segmentation on the datasets and fed to the classifier. In the second experiment, we train the dataset through the SqueezeNet model and fed it to the classifier. In the third experiment, we applied feature fusion both the first and second models fused the features and fed them to the classifier. At the end, the best value of features is selected through improved butterfly optimization. Several classifiers have been opted for the classification, whereas the performance of each classifier is computed based on recall rate, precision rate, F1 score, and accuracy. Moreover, the computational time is noted during the testing process. The entire proposed method is implemented on MATLAB2022a using Desktop Computer with 16 GB of RAM and 6 GB NVIDIA RTX graphics card.


### 3.3. Experiment 1: DarkNet19 Features

In experiment 1, segmentation is performed on the selected dataset through the proposed approach. After segmentation, we train the fine-tuned DarkNet19 model and extract deep features. On this model, 1024 features have been extracted. Ten different classification methods have been opted and performed classification. The testing accuracy on multiple classifiers is depicted in Table3. In this table, the accuracy of quadratic SVM is 98.4%, which is better compared to other well-known classification methods such as linear SVM, weight KNN, cosine KNN, linear discriminant, and five types of neural networks. The second highest accuracy of 98.3% was obtained on linear SVM and linear discriminant classifiers. The linear discriminants have taken less time as compared to other classifiers. The performance measure of quadratic SVM is also illustrated by a confusion matrix (CM) shown in Figure 7.Table 3
Classification results of the DarkNet19 model (experiment 1).
Classifier (%)Recall (%)Precision (%)F1 score (%)FNR (%)Accuracy (%)Time (sec)Linear SVM98.2898.398.291.7298.3277.71Quadratic SVM98.3698.3898.371.6498.4291.46Weight KNN98.2298.298.211.7898.21444.8Cosine KNN98.298.2298.211.898.21781.5Linear discriminant98.2698.2898.271.7498.373.55Medium neural network97.8697.8897.872.1497.91203.3Narrow neural network97.7697.7497.752.2497.81943.7Wide neural network97.997.8897.892.197.91699.8Bilayered neural network97.8297.8497.832.1897.82330.2Trilayered neural network97.8297.8897.852.1897.81922.5Figure 7
Confusion matrix of quadratic SVM (experiment 1).


### 3.4. Experiment 2: SqueezeNet Features

In experiment 2, SequeezeNet testing features have been extracted and classification has been performed. From this model, 1000 features are extracted. Ten classifiers have been utilized for the classification results and obtained the maximum accuracy of 99.5% by quadratic SVM, as presented in Table4. The other classifiers also give better accuracy, but the maximum accuracy is obtained on quadratic SVM. The computational time is also noted for all classifiers, and it is observed that the linear discriminant consumes less time than the rest of the classifiers. Moreover, the confusion matrix of quadratic SVM is also provided as illustrated in Figure 8.Table 4
Classification results of the SqueezeNet model (experiment 2).
Classifier (%)Recall (%)Sensitive (%)F1 score (%)FNR (%)Accuracy (%)Time (sec)Linear SVM99.2299.2499.230.7899.2301.74Quadratic SVM99.4699.4699.40.5499.5304.67Weight KNN98.5898.4498.511.4298.6362.9Cosine KNN97.997.9497.922.197.9387.4Linear discriminant99.3299.3299.320.6899.372.783Medium neural network99.4299.4299.420.5899.4178.13Narrow neural network99.3899.3899.380.6299.4244.41Wide neural network99.4199.499.410.5899.4249.99Bilayered neural network99.3699.3699.360.6499.4336.25Trilayered neural network99.3799.3899.370.6499.4458.32Figure 8
Confusion matrix of quadratic SVM (experiment 2).


### 3.5. Experiment 3: Feature Fusion

In the third experiment, extracted deep learning features are fused using the proposed maximum correlation-based approach. Similar to experiment 1 and experiment 2, ten different classification methods have been utilized, and the results are presented in Table5. In this table, it is noted that the accuracy is improved after the fusion process, whereas the time is slightly decreased. The wide neural network better performed than the rest of the classifiers as the obtained accuracy is 100% and computational time is 139.86 (sec). Compared to experiment 1 and experiment 2, all classifiers performed better after the fusion process. Moreover, the confusion matrix is also provided as illustrated in Figure 9. In this figure, it is clearly observed that each class correct prediction rate is above 99%.Table 5
Classification results of fusion data (experiment 3).
Classifier (%)Recall (%)Sensitive (%)F1 score (%)FNR (%)Accuracy (%)Time (sec)Linear SVM99.9699.9499.950.0499.9275.3Quadratic SVM99.9699.9699.960.04100221.12Weight KNN99.8899.799.790.1299.9133.5Cosine KNN99.8699.8899.870.1499.9146.3Linear discriminant99.7699.7699.760.2499.7143.63Medium neural network99.9499.9499.940.0699.9223.75Narrow neural network99.9299.9699.940.0899.9209.29Wide neural network99.9699.9699.960.04100139.86Bilayered neural network99.9499.9699.950.0699.9204.31Trilayered neural network99.999.9299.910.199.9229.86Figure 9
Confusion matrix of wide neural network (experiment 3).


### 3.6. Experiment 4: Best Selected Features

In experiment 4, the best features are selected using the proposed improved butterfly optimization algorithm. The total 522 features are selected after employing the proposed optimization algorithm. Results of this experiment are given in Table6. The best noted accuracy of 100% was achieved by Cubic SVM, whereas the recall rate is 99.98, precision rate is 99.96, and F1 score is 99.96%, respectively. The other classifiers also obtained better accuracy and consumed less time than the previous experiments. The confusion matrix of quadratic SVM is also given as illustrated in Figure 10.Table 6
Classification result after proposed optimization algorithm (experiment 4).
ClassifierRecall (%)Precision rate (%)F1 score (%)FNR (%)Accuracy (%)Time (sec)Linear SVM99.499.9499.940.699.9108.64Quadratic SVM99.9899.9699.960.02100142.31Weight KNN99.8899.999.890.1299.9179.7Cosine KNN99.8699.8699.860.1499.951.49Linear discriminant99.6899.799.690.3299.753.094Medium neural network99.7499.7499.740.2699.963.839Narrow neural network99.999.799.810.199.971.045Wide neural network99.9299.9699.940.0899.943.207Bilayered neural network99.9499.9299.930.0699.986.704Trilayered neural network99.9499.9299.930.0699.997.371Figure 10
Confusion matrix of quadratic SVM (experiment 4).


### 3.7. Analysis and Comparison

In this section, a detailed analysis of the proposed system is conducted based on numerical values and some visual plots. The proposed method was evaluated on a publicly available dataset including five classes as illustrated in Figure1. The proposed method includes some important steps such as rice segmentation, deep learning feature extraction, fusion of deep features using the proposed approach, and selection of best features using improved butterfly optimization; the main flow is given in Figure 2. The proposed segmentation technique clearly highlights the rice region shown in Figure 3 which is later utilized for the training of deep models. The results of each step are computed as given in Tables 3–6. Moreover, the confusion matrix of step is illustrated in Figures 7–10. Based on the results, it is clearly observed that the fusion process reduces the computational time that further reduces after the best feature selection. Moreover, we conducted a comparison of the proposed method with several other neural networks as illustrated in Figure 11. Based on this figure, it is clearly shown that the proposed fusion and selection methods give better accuracy.Figure 11
Comparison of the proposed method with other neural networks.Finally, a comparison is conducted with recent techniques on rice_image_dataset, as presented in Table7. Koklu et al. [39] used a rice image dataset and applied three classification methods. The classifier ANN, DNN, and CNN and their results are 99.87%, 99.95%, and 100%, respectively. Cinar and Koklu [40] used the same dataset and achieved maximum accuracy of 99.9%. In [41], authors presented a deep learning-based framework for rice variety classification and obtained accuracy of 98.2%. In [42], authors presented a computerized method and obtained 93.04% accuracy. The proposed method achieved the maximum 100% accuracy with minimum computational time.Table 7
Comparison results of others’ work.
ReferencesYearDatasetAccuracy (%)[39]2021Rice_image_datasetANN99.87DNN99.95CNN100[40]2022Rice_image_datasetLogistic regression99.25Multiperceptron99.9Classification97.4[41]2022Rice_Image_datasetClassification98.2[42]2022Rice datasetClassification93.04Proposed2022Rice_image_datasetLinear SVM99.9Quadratic SVM100Weight CNN99.9Cosine CNN99.9Linear discriminant99.7Medium NN99.9Narrow NN99.9Wide NN99.9Bilayered NN99.9Trilayered NN99.9


## 3.1. Dataset

In this work, we utilized a publically available rice dataset that include five types of rice varieties such as (a) Khazar, (b) Gharib, (c) Ghasrdashti, (d) Gerdeh, and (e) Mohammadi [39]. A sample image is shown in Figure 1. This dataset consists of total 75,000 images and each class consists of 15,000 images (https://www.kaggle.com/datasets/muratkokludataset/rice-image-dataset). We employed 50% of the images from each class for training and rest for testing.


## 3.2. Experimental Setup

The experimental process of the proposed method is conducted on publically available rice imaging dataset [39], as discussed under Section 3.1. We performed the 50 : 50 approach for both training and testing, whereas the cross-validation value is 10. Four different experiments have been performed for the detailed analysis of the proposed method. In the first experiment, the DarkNet19 model is used after applying deep saliency-based segmentation on the datasets and fed to the classifier. In the second experiment, we train the dataset through the SqueezeNet model and fed it to the classifier. In the third experiment, we applied feature fusion both the first and second models fused the features and fed them to the classifier. At the end, the best value of features is selected through improved butterfly optimization. Several classifiers have been opted for the classification, whereas the performance of each classifier is computed based on recall rate, precision rate, F1 score, and accuracy. Moreover, the computational time is noted during the testing process. The entire proposed method is implemented on MATLAB2022a using Desktop Computer with 16 GB of RAM and 6 GB NVIDIA RTX graphics card.


## 3.3. Experiment 1: DarkNet19 Features

In experiment 1, segmentation is performed on the selected dataset through the proposed approach. After segmentation, we train the fine-tuned DarkNet19 model and extract deep features. On this model, 1024 features have been extracted. Ten different classification methods have been opted and performed classification. The testing accuracy on multiple classifiers is depicted in Table3. In this table, the accuracy of quadratic SVM is 98.4%, which is better compared to other well-known classification methods such as linear SVM, weight KNN, cosine KNN, linear discriminant, and five types of neural networks. The second highest accuracy of 98.3% was obtained on linear SVM and linear discriminant classifiers. The linear discriminants have taken less time as compared to other classifiers. The performance measure of quadratic SVM is also illustrated by a confusion matrix (CM) shown in Figure 7.Table 3
Classification results of the DarkNet19 model (experiment 1).
Classifier (%)Recall (%)Precision (%)F1 score (%)FNR (%)Accuracy (%)Time (sec)Linear SVM98.2898.398.291.7298.3277.71Quadratic SVM98.3698.3898.371.6498.4291.46Weight KNN98.2298.298.211.7898.21444.8Cosine KNN98.298.2298.211.898.21781.5Linear discriminant98.2698.2898.271.7498.373.55Medium neural network97.8697.8897.872.1497.91203.3Narrow neural network97.7697.7497.752.2497.81943.7Wide neural network97.997.8897.892.197.91699.8Bilayered neural network97.8297.8497.832.1897.82330.2Trilayered neural network97.8297.8897.852.1897.81922.5Figure 7
Confusion matrix of quadratic SVM (experiment 1).


## 3.4. Experiment 2: SqueezeNet Features

In experiment 2, SequeezeNet testing features have been extracted and classification has been performed. From this model, 1000 features are extracted. Ten classifiers have been utilized for the classification results and obtained the maximum accuracy of 99.5% by quadratic SVM, as presented in Table4. The other classifiers also give better accuracy, but the maximum accuracy is obtained on quadratic SVM. The computational time is also noted for all classifiers, and it is observed that the linear discriminant consumes less time than the rest of the classifiers. Moreover, the confusion matrix of quadratic SVM is also provided as illustrated in Figure 8.Table 4
Classification results of the SqueezeNet model (experiment 2).
Classifier (%)Recall (%)Sensitive (%)F1 score (%)FNR (%)Accuracy (%)Time (sec)Linear SVM99.2299.2499.230.7899.2301.74Quadratic SVM99.4699.4699.40.5499.5304.67Weight KNN98.5898.4498.511.4298.6362.9Cosine KNN97.997.9497.922.197.9387.4Linear discriminant99.3299.3299.320.6899.372.783Medium neural network99.4299.4299.420.5899.4178.13Narrow neural network99.3899.3899.380.6299.4244.41Wide neural network99.4199.499.410.5899.4249.99Bilayered neural network99.3699.3699.360.6499.4336.25Trilayered neural network99.3799.3899.370.6499.4458.32Figure 8
Confusion matrix of quadratic SVM (experiment 2).


## 3.5. Experiment 3: Feature Fusion

In the third experiment, extracted deep learning features are fused using the proposed maximum correlation-based approach. Similar to experiment 1 and experiment 2, ten different classification methods have been utilized, and the results are presented in Table5. In this table, it is noted that the accuracy is improved after the fusion process, whereas the time is slightly decreased. The wide neural network better performed than the rest of the classifiers as the obtained accuracy is 100% and computational time is 139.86 (sec). Compared to experiment 1 and experiment 2, all classifiers performed better after the fusion process. Moreover, the confusion matrix is also provided as illustrated in Figure 9. In this figure, it is clearly observed that each class correct prediction rate is above 99%.Table 5
Classification results of fusion data (experiment 3).
Classifier (%)Recall (%)Sensitive (%)F1 score (%)FNR (%)Accuracy (%)Time (sec)Linear SVM99.9699.9499.950.0499.9275.3Quadratic SVM99.9699.9699.960.04100221.12Weight KNN99.8899.799.790.1299.9133.5Cosine KNN99.8699.8899.870.1499.9146.3Linear discriminant99.7699.7699.760.2499.7143.63Medium neural network99.9499.9499.940.0699.9223.75Narrow neural network99.9299.9699.940.0899.9209.29Wide neural network99.9699.9699.960.04100139.86Bilayered neural network99.9499.9699.950.0699.9204.31Trilayered neural network99.999.9299.910.199.9229.86Figure 9
Confusion matrix of wide neural network (experiment 3).


## 3.6. Experiment 4: Best Selected Features

In experiment 4, the best features are selected using the proposed improved butterfly optimization algorithm. The total 522 features are selected after employing the proposed optimization algorithm. Results of this experiment are given in Table6. The best noted accuracy of 100% was achieved by Cubic SVM, whereas the recall rate is 99.98, precision rate is 99.96, and F1 score is 99.96%, respectively. The other classifiers also obtained better accuracy and consumed less time than the previous experiments. The confusion matrix of quadratic SVM is also given as illustrated in Figure 10.Table 6
Classification result after proposed optimization algorithm (experiment 4).
ClassifierRecall (%)Precision rate (%)F1 score (%)FNR (%)Accuracy (%)Time (sec)Linear SVM99.499.9499.940.699.9108.64Quadratic SVM99.9899.9699.960.02100142.31Weight KNN99.8899.999.890.1299.9179.7Cosine KNN99.8699.8699.860.1499.951.49Linear discriminant99.6899.799.690.3299.753.094Medium neural network99.7499.7499.740.2699.963.839Narrow neural network99.999.799.810.199.971.045Wide neural network99.9299.9699.940.0899.943.207Bilayered neural network99.9499.9299.930.0699.986.704Trilayered neural network99.9499.9299.930.0699.997.371Figure 10
Confusion matrix of quadratic SVM (experiment 4).


## 3.7. Analysis and Comparison

In this section, a detailed analysis of the proposed system is conducted based on numerical values and some visual plots. The proposed method was evaluated on a publicly available dataset including five classes as illustrated in Figure1. The proposed method includes some important steps such as rice segmentation, deep learning feature extraction, fusion of deep features using the proposed approach, and selection of best features using improved butterfly optimization; the main flow is given in Figure 2. The proposed segmentation technique clearly highlights the rice region shown in Figure 3 which is later utilized for the training of deep models. The results of each step are computed as given in Tables 3–6. Moreover, the confusion matrix of step is illustrated in Figures 7–10. Based on the results, it is clearly observed that the fusion process reduces the computational time that further reduces after the best feature selection. Moreover, we conducted a comparison of the proposed method with several other neural networks as illustrated in Figure 11. Based on this figure, it is clearly shown that the proposed fusion and selection methods give better accuracy.Figure 11
Comparison of the proposed method with other neural networks.Finally, a comparison is conducted with recent techniques on rice_image_dataset, as presented in Table7. Koklu et al. [39] used a rice image dataset and applied three classification methods. The classifier ANN, DNN, and CNN and their results are 99.87%, 99.95%, and 100%, respectively. Cinar and Koklu [40] used the same dataset and achieved maximum accuracy of 99.9%. In [41], authors presented a deep learning-based framework for rice variety classification and obtained accuracy of 98.2%. In [42], authors presented a computerized method and obtained 93.04% accuracy. The proposed method achieved the maximum 100% accuracy with minimum computational time.Table 7
Comparison results of others’ work.
ReferencesYearDatasetAccuracy (%)[39]2021Rice_image_datasetANN99.87DNN99.95CNN100[40]2022Rice_image_datasetLogistic regression99.25Multiperceptron99.9Classification97.4[41]2022Rice_Image_datasetClassification98.2[42]2022Rice datasetClassification93.04Proposed2022Rice_image_datasetLinear SVM99.9Quadratic SVM100Weight CNN99.9Cosine CNN99.9Linear discriminant99.7Medium NN99.9Narrow NN99.9Wide NN99.9Bilayered NN99.9Trilayered NN99.9


## 4. Conclusion

In this study, we proposed an automated deep learning-based framework for rice segmentation and variety classification. The proposed framework includes several critical steps. For better visualization, the segmentation step localized the rice region and mapped color. The main goal of this step is to provide better images for deep learning model training. Two pretrained deep models extract features from segmented mapped images. The extracted features are fused using the proposed maximum correlation-based technique. This step reduced the number of features and selected only the most important ones for the fused matrix. However, it should be noted that, during the experimental process, some redundant features are also added to the fused matrix, which does not affect accuracy but increases computational time. As a result, we proposed an improved butterfly optimization algorithm that not only maintains accuracy but also reduces testing time. The selection of important features during the fusion process is the main limitation of this work using the static threshold value. In the future, the fusion process will be optimized for less computational time by employing new techniques. Moreover, in the future, plant and rice disease detection and classification problems will be considered [43–47].



---

*Source: 1339469-2022-11-25.xml*