# Identification of Apple Disease Grades Based on the Attention Mechanism of Lesion Location and Improved Data Enhancement Method

Authors: Tianzhen Yin; Dian Wang
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

The output and quality of apples were greatly threatened by plant diseases. Identifying the types and grades of diseases in time was helpful to the management of diseases. When the disease occurs on a leaf, there was little change in the leaf except for the affected area. The traditional attention mechanism changed the weight of the network to all pixels in the image, which affected the ability of the network to extract the features of the lesion area. And the traditional method of data enhancement was easy to cause local similarity and local discontinuity of all sample features in the same disease grade. In this paper, the improved metric matrix for kernel regression (IMMKR) was used to reduce the influence of local similarity and local discontinuity of all sample features in the same class. Then, a new attention mechanism by fusing the lesion location based on visual features was proposed, and the attention of the model to the lesion area was strengthened. The experiments were carried out on three different diseases of apple named black rot, scab, and rust. The accuracy rate and recall rate of the new method on the combined dataset of PlantVillage and PlantDoc were 91.55% and 92.06%, respectively, which was superior to existing methods. This algorithm has important reference significance for the identification and promotion of crop diseases.


---

## Body

## 1. Introduction

Plant disease is one of the most common factors restricting the yield and quality of crops and has brought a great threat to agriculture [1]. The traditional disease diagnosis method is that experts observe the characteristics of diseases with naked eyes. For experienced pathologists and agronomists, it is difficult to identify diseases and their severity in time, because there is a large number of arable land crops [2]. Moreover, the diagnostic results of experts with different botanical attainments are also very different. In addition, pesticides remain the main means to protect crops from diseases. However, in most cases, regardless of the severity of the disease, crops suffering from the same disease are given the same amount of pesticides [3]. Therefore, identifying the types and severity of diseases in time is helpful to the management of diseases [4].With the development of computer vision technology, pattern recognition has been widely used in plant disease diagnosis [5]. The traditional plant disease diagnosis methods include features extraction and analysis [6]. First, a single threshold or multi-threshold was used to obtain the lesion. Then, different methods were used to analyze the features of the lesion area. Principal component analysis, fuzzy logic, and other tools have been used to construct plant disease classification models [7]. And the fungal colonies were classified by genetic algorithm [8]. In 2004, Support Vector Machine, K-nearest neighbor algorithm, and decision tree were also widely used to solve the problem of disease classification [9]. In 2008, the chromaticity moment analysis was applied to plant disease identification [10]. In recent years, with the development of computer technology, deep learning methods have become popular because they can automatically extract features instead of manually extracting them [11]. The convolutional neural network was widely considered a reliable method for image classification, object detection, semantic segmentation, and other computer vision tasks. For the classification of plant disease images, some research has been put forward, and good results have been achieved [12]. And deep learning is widely used in disease classification, such as in cassava [13], tomato [14], and apple [15]. In addition, deep learning has gradually been applied to the detection of disease severity [16]. In 2018, Namita Sengar et al. took RLT as the disease grading standard of cherry powdery mildew [17]. Fang Tao et al. calculated the ratio of the number of pixels in the diseased area to the number of pixels in the diseased leaf, put it into Resnet-50 as the classification threshold of disease grade, and classified 10 diseases of 8 plants [18].Deep learning gives equal weight to each pixel in the image, and the key pixels cannot be highlighted. To improve the performance of the network, attention mechanisms were introduced into deep learning. Gamaleldin F et al. proposed the saccader attention model, which narrows the performance gap with ImageNet and reduces the demanding standard of training data [19]. Xie et al. had comprehensive depth characteristics and local characteristics to build a composite insulator hydrophobicity grade discrimination model [20]. Ying et al. used the fusion of image and prior knowledge of clinical diagnosis to improve the classification performance of chronic obstructive pulmonary disease severity [21]. Wang et al. proposed a crop disease recognition model based on image and text dual-modal joint representation learning.The traditional attention mechanism changes the weight of the network to all pixels in the image, which affects the ability of the network to extract the features of the lesion area. Zhang et al. said that 80% to 90% of crop diseases would appear on the leaves [22]. To apple scab, black rot, and rust [23–28], the image of apple leaves is simple. At the time of onset, except for the diseased area, the leaf area hardly changes. Moreover, traditional data enhancement methods are easy to cause local similarity and local discontinuity of all sample features in the same disease grade. Then, by fusing the lesion location based on visual features, the attention of the model to the lesion area was strengthened. Therefore, how to make use of the characteristics of diseases to improve the network’s attention to the lesion area is the focus of this paper. Based on a convolutional neural network, this paper proposes a new model of disease severity recognition which integrates visual features and neural networks.


## 2. Methodology

The purpose of this study is to put forward a new disease grade identification model. The proposed method includes two main phases. In the first stage, an improved data enhancement method was used. In the second stage, a new attention mechanism was used. The details of each process are given below.


### 2.1. Improved Data Enhancement Method

When CNN is applied to the classification of plant diseases, to reduce the influence of too few images on classification results, several data enhancement methods, such as flipping images and rotating images, are introduced. Traditional data enhancement makes the number of pictures in different categories equal. When a leaf is diseased, the change of the spots will change with time. Under each category, if only the number of samples in each grade was considered, it was easy to cause local similarity and local discontinuity of all samples in each disease grade. Because the disease severity of samples may be too concentrated in the same growth stage, there may be data imbalance in the same category.Metric learning for kernel regression is a supervised dimension reduction algorithm. In this paper, the metric learning of kernel regression is to fit the measurement matrix through the characteristics of the central samples and the adjoint labels. And the measurement matrix is used to make the feature vectors transformed from all sample features more distinguishable. In this paper, the center samples were selected manually. 100 samples are selected from each category, totaling 1,100. Based on the metric matrix, the distance between the center sample and other samples is calculated, and five intervals are divided according to the average distance. Then, traditional data enhancement methods are used to make the number of samples in each interval equal, to reduce the local similarity and local discontinuity of all sample features in the same disease grade.However, metric learning for kernel regression failed to consider the correlation between multidimensional parameters and the weight of each parameter. In this paper, IMMKR is used to solve the problems that the correlation and weight coefficients of the parameters under different diseases grade are different. The kernel function used in this article is Equation (1)
(1)kij=12πσexp−dxi,xjσ2i∈1,100,j∈101,1500.In the previous study,dxi,xj is mahala Nobis distance,σ2=1. The loss function is shown in Equation (2).
(2)L=∑iyi−y∧i2.The improved distance function is shown in Equation (3).
(3)dxijc=∑k=132∑l=132rklcxjk−xikxjl−xilqlcqkc1/2i∈1,100,j∈101,1500,c∈1,11.I and j represent the center samples and the rested samples, respectively. C represents different disease grades. L and k represent different parameters, respectively.
(4)rklc=∑n=1100xkn−xkn¯xln−xin¯∑n=1100xkn−xkn¯2∑xln−xin¯2l∈1,32,k∈1,32,c∈1,11,ynk=xnk∑n=1100xnk,ynl=xnl∑n=1100xnl,qkc=1+∑i=1100ynklnynk/ln100∑i=1321+∑i=1100ynklnynk/ln100,qlc=1+∑i=1100ynllnynl/ln100∑i=1321+∑i=1100ynllnynl/ln100.C represents different disease grades. N is the number of center samples. L and k represent different parameters, respectively.In this study, 32 parameters were selected to measure disease grades label more accurately, which are from color, texture, and shape of lesions. The color features include mean value, variance, and kurtosis of RGB value of the lesion. Texture characteristic parameters include contrast, entropy, angular second moment, and inverse differential moment of the gray level co-occurrence matrix in four directions of the lesion image. They reflect the depth of the texture of the image, including the randomness of the amount of information, the uniformity of the image gray distribution, the measurement of the texture thickness, and the clarity and regularity of the texture, respectively. Shape features include the seven invariant moments of Hu moment obtained by using the second-order and third-order central moments of the image, which reflects the invariance of the translation, expansion, and rotation of the shape parameters.When the measurement matrix fitting is completed, the distance interval [dxijcmin, dxijcmax] between the center samples and other samples can be obtained. Then, the number of samples in each interval is obtained. The interval is shown in formula:
(5)dxijcmin+dxijcmax−dxijcmin5×m,dxijcmin+dxijcmax−dxijcmin5×m+1.In Equation (5), m∈1,2,3,4,5. If the number of samples exceeds 300, the excess samples are randomly rounded off. If the number is less than 300, data enhancement is used to increase the number of samples to 300.


### 2.2. Improved Attention Mechanism

In the previous studies, many attention mechanisms were put forward. In the field of images classification, attention mechanism in the convolutional neural network includes Convolution Block Attention Module (CBAM) [29], SE block [30], and their mixture. There are also many attention mechanisms in target detection, such as Single Shot MultiBox Detector (SSD) [31] and Feature Pyramid Networks (FPN) [32]. The traditional attention mechanism changes the weight of the network to all pixels in the image, which affects the ability of the network to extract the features of the lesion area. In this paper, a new attention mechanism is proposed by integrating the acquired information of the lesion area with global information. This method can increase the attention of the network to the lesion area, while the influence of other areas on the attention of the network to the lesion area can be ignored. The details were given below. The proposed attention mechanism includes two main phases: (a) acquisition of lesion location; (b) fusion of lesion location features. The details of each process are given below.


#### 2.2.1. Acquisition of Lesion Area

First, the RGB values of the image are converted to HSV values. Then, Equation (6) is used to obtain the difference distribution between HSV values of each pixel and HSV values of the lesion.
(6)fIx=expkx−xLxU−xxU−xL2.Among them,x represents any pixel of the image, and xU and xL are the upper and lower limits of hue value in the lesion area, respectively. K is used to adjust the contrast between the lesion and the background. In this paper, K is 1.5.
(7)px=cexp−x−a2b2.A and c represent the center and peak of the interval [fIxmin,1], respectively, and b is the variance of all values in the interval.
(8)ω0=∑i=fIxmintpi,ω1=∑i=tfIxmaxpi,μ0=∑i=fIxmintipiω0,μ1=∑i=tfIxmaxipiω1,μ=∑i=0fIxmaxipi,δ2t=ω0μ0−μ2+ω1μ0−μ12.pi is the proportion of each Px. When δ2t is the smallest, t is denoted as t0. The location of the lesion is the location of every pixel of the image belonging to [fIxmin,t0].


#### 2.2.2. Fusion of Lesion Area Features

After obtaining the location of the lesion, to compare the influence of fusion depth on the classification effect, different fusion structures were explored. The purpose of this study is to put forward a new disease severity identification model by fusing the features of lesion area based on visual features and CNN. In recent years, some popular deep learning models, such as AlexNet, Inception, VGGNet, and ResNet, have been used to identify plant diseases. In these modes, VGG is a relatively simple network. Therefore, VGG 16 is selected to highlight the advantages of improvement.The original VGG16 is shown in Figure1(a). Figures 1(b)–1(e) show four fusion structures that increase the information input of the sample lesion location on the original network, which are named C0, C1, C2, and C4. These fusion structures only replace the convolution layer corresponding to the original network and do not change the subsequent structure of the original network.Figure 1
Original CNN and feature fusion structures. (a) Original CNN. (b) Fusion structure C0. (c) Fusion structure C1. (d) Fusion structure C2. (e) Fusion structure C4.
(a)(b)(c)(d)(e)The main parameters of the network structure are also shown in this diagram. Generally, CNN maps input featuresX=x1,x2,⋯,xC×H×W to predicted labels Y=y1,y2,⋯,yM in classification tasks by convolution operation. C is the number of channels of the convolution kernel. H and W are the length and width of the input image, respectively. M is the number of categories. Each component yi of Y is calculated by Equation (9).
(9)yi=ki∗X=∑j=1Ckij∗xj.In the equation, “∗” denote convolution calculation, and ki is the convolution kernel. And loss function is following:
(10)L=1N∑iLi=−1N∑i∑c=1Myiclogpic.M is the number of categories. N is the number of samples. If the sample I belong to category c, yic=1; otherwise, yic=0, pic is the prediction probability that sample I belongs to category c.In this article, the input sample feature of the original network is assumed to beXi=x1,x2,⋯,xC×H×W and its lesion location is assumed to be Xi′=x1,x2,⋯,xN. Based on the expansion mechanism, the new input feature vector Xi′ is from [fIxmin,t0]. The new convolution formula is Equations 17 and 18
(11)Xi′′′=Xi+Xi∗Xi′,yi=ki∗Xi′′′=∑j=1Ckij∗xi′′′.


## 2.1. Improved Data Enhancement Method

When CNN is applied to the classification of plant diseases, to reduce the influence of too few images on classification results, several data enhancement methods, such as flipping images and rotating images, are introduced. Traditional data enhancement makes the number of pictures in different categories equal. When a leaf is diseased, the change of the spots will change with time. Under each category, if only the number of samples in each grade was considered, it was easy to cause local similarity and local discontinuity of all samples in each disease grade. Because the disease severity of samples may be too concentrated in the same growth stage, there may be data imbalance in the same category.Metric learning for kernel regression is a supervised dimension reduction algorithm. In this paper, the metric learning of kernel regression is to fit the measurement matrix through the characteristics of the central samples and the adjoint labels. And the measurement matrix is used to make the feature vectors transformed from all sample features more distinguishable. In this paper, the center samples were selected manually. 100 samples are selected from each category, totaling 1,100. Based on the metric matrix, the distance between the center sample and other samples is calculated, and five intervals are divided according to the average distance. Then, traditional data enhancement methods are used to make the number of samples in each interval equal, to reduce the local similarity and local discontinuity of all sample features in the same disease grade.However, metric learning for kernel regression failed to consider the correlation between multidimensional parameters and the weight of each parameter. In this paper, IMMKR is used to solve the problems that the correlation and weight coefficients of the parameters under different diseases grade are different. The kernel function used in this article is Equation (1)
(1)kij=12πσexp−dxi,xjσ2i∈1,100,j∈101,1500.In the previous study,dxi,xj is mahala Nobis distance,σ2=1. The loss function is shown in Equation (2).
(2)L=∑iyi−y∧i2.The improved distance function is shown in Equation (3).
(3)dxijc=∑k=132∑l=132rklcxjk−xikxjl−xilqlcqkc1/2i∈1,100,j∈101,1500,c∈1,11.I and j represent the center samples and the rested samples, respectively. C represents different disease grades. L and k represent different parameters, respectively.
(4)rklc=∑n=1100xkn−xkn¯xln−xin¯∑n=1100xkn−xkn¯2∑xln−xin¯2l∈1,32,k∈1,32,c∈1,11,ynk=xnk∑n=1100xnk,ynl=xnl∑n=1100xnl,qkc=1+∑i=1100ynklnynk/ln100∑i=1321+∑i=1100ynklnynk/ln100,qlc=1+∑i=1100ynllnynl/ln100∑i=1321+∑i=1100ynllnynl/ln100.C represents different disease grades. N is the number of center samples. L and k represent different parameters, respectively.In this study, 32 parameters were selected to measure disease grades label more accurately, which are from color, texture, and shape of lesions. The color features include mean value, variance, and kurtosis of RGB value of the lesion. Texture characteristic parameters include contrast, entropy, angular second moment, and inverse differential moment of the gray level co-occurrence matrix in four directions of the lesion image. They reflect the depth of the texture of the image, including the randomness of the amount of information, the uniformity of the image gray distribution, the measurement of the texture thickness, and the clarity and regularity of the texture, respectively. Shape features include the seven invariant moments of Hu moment obtained by using the second-order and third-order central moments of the image, which reflects the invariance of the translation, expansion, and rotation of the shape parameters.When the measurement matrix fitting is completed, the distance interval [dxijcmin, dxijcmax] between the center samples and other samples can be obtained. Then, the number of samples in each interval is obtained. The interval is shown in formula:
(5)dxijcmin+dxijcmax−dxijcmin5×m,dxijcmin+dxijcmax−dxijcmin5×m+1.In Equation (5), m∈1,2,3,4,5. If the number of samples exceeds 300, the excess samples are randomly rounded off. If the number is less than 300, data enhancement is used to increase the number of samples to 300.


## 2.2. Improved Attention Mechanism

In the previous studies, many attention mechanisms were put forward. In the field of images classification, attention mechanism in the convolutional neural network includes Convolution Block Attention Module (CBAM) [29], SE block [30], and their mixture. There are also many attention mechanisms in target detection, such as Single Shot MultiBox Detector (SSD) [31] and Feature Pyramid Networks (FPN) [32]. The traditional attention mechanism changes the weight of the network to all pixels in the image, which affects the ability of the network to extract the features of the lesion area. In this paper, a new attention mechanism is proposed by integrating the acquired information of the lesion area with global information. This method can increase the attention of the network to the lesion area, while the influence of other areas on the attention of the network to the lesion area can be ignored. The details were given below. The proposed attention mechanism includes two main phases: (a) acquisition of lesion location; (b) fusion of lesion location features. The details of each process are given below.


### 2.2.1. Acquisition of Lesion Area

First, the RGB values of the image are converted to HSV values. Then, Equation (6) is used to obtain the difference distribution between HSV values of each pixel and HSV values of the lesion.
(6)fIx=expkx−xLxU−xxU−xL2.Among them,x represents any pixel of the image, and xU and xL are the upper and lower limits of hue value in the lesion area, respectively. K is used to adjust the contrast between the lesion and the background. In this paper, K is 1.5.
(7)px=cexp−x−a2b2.A and c represent the center and peak of the interval [fIxmin,1], respectively, and b is the variance of all values in the interval.
(8)ω0=∑i=fIxmintpi,ω1=∑i=tfIxmaxpi,μ0=∑i=fIxmintipiω0,μ1=∑i=tfIxmaxipiω1,μ=∑i=0fIxmaxipi,δ2t=ω0μ0−μ2+ω1μ0−μ12.pi is the proportion of each Px. When δ2t is the smallest, t is denoted as t0. The location of the lesion is the location of every pixel of the image belonging to [fIxmin,t0].


### 2.2.2. Fusion of Lesion Area Features

After obtaining the location of the lesion, to compare the influence of fusion depth on the classification effect, different fusion structures were explored. The purpose of this study is to put forward a new disease severity identification model by fusing the features of lesion area based on visual features and CNN. In recent years, some popular deep learning models, such as AlexNet, Inception, VGGNet, and ResNet, have been used to identify plant diseases. In these modes, VGG is a relatively simple network. Therefore, VGG 16 is selected to highlight the advantages of improvement.The original VGG16 is shown in Figure1(a). Figures 1(b)–1(e) show four fusion structures that increase the information input of the sample lesion location on the original network, which are named C0, C1, C2, and C4. These fusion structures only replace the convolution layer corresponding to the original network and do not change the subsequent structure of the original network.Figure 1
Original CNN and feature fusion structures. (a) Original CNN. (b) Fusion structure C0. (c) Fusion structure C1. (d) Fusion structure C2. (e) Fusion structure C4.
(a)(b)(c)(d)(e)The main parameters of the network structure are also shown in this diagram. Generally, CNN maps input featuresX=x1,x2,⋯,xC×H×W to predicted labels Y=y1,y2,⋯,yM in classification tasks by convolution operation. C is the number of channels of the convolution kernel. H and W are the length and width of the input image, respectively. M is the number of categories. Each component yi of Y is calculated by Equation (9).
(9)yi=ki∗X=∑j=1Ckij∗xj.In the equation, “∗” denote convolution calculation, and ki is the convolution kernel. And loss function is following:
(10)L=1N∑iLi=−1N∑i∑c=1Myiclogpic.M is the number of categories. N is the number of samples. If the sample I belong to category c, yic=1; otherwise, yic=0, pic is the prediction probability that sample I belongs to category c.In this article, the input sample feature of the original network is assumed to beXi=x1,x2,⋯,xC×H×W and its lesion location is assumed to be Xi′=x1,x2,⋯,xN. Based on the expansion mechanism, the new input feature vector Xi′ is from [fIxmin,t0]. The new convolution formula is Equations 17 and 18
(11)Xi′′′=Xi+Xi∗Xi′,yi=ki∗Xi′′′=∑j=1Ckij∗xi′′′.


## 2.2.1. Acquisition of Lesion Area

First, the RGB values of the image are converted to HSV values. Then, Equation (6) is used to obtain the difference distribution between HSV values of each pixel and HSV values of the lesion.
(6)fIx=expkx−xLxU−xxU−xL2.Among them,x represents any pixel of the image, and xU and xL are the upper and lower limits of hue value in the lesion area, respectively. K is used to adjust the contrast between the lesion and the background. In this paper, K is 1.5.
(7)px=cexp−x−a2b2.A and c represent the center and peak of the interval [fIxmin,1], respectively, and b is the variance of all values in the interval.
(8)ω0=∑i=fIxmintpi,ω1=∑i=tfIxmaxpi,μ0=∑i=fIxmintipiω0,μ1=∑i=tfIxmaxipiω1,μ=∑i=0fIxmaxipi,δ2t=ω0μ0−μ2+ω1μ0−μ12.pi is the proportion of each Px. When δ2t is the smallest, t is denoted as t0. The location of the lesion is the location of every pixel of the image belonging to [fIxmin,t0].


## 2.2.2. Fusion of Lesion Area Features

After obtaining the location of the lesion, to compare the influence of fusion depth on the classification effect, different fusion structures were explored. The purpose of this study is to put forward a new disease severity identification model by fusing the features of lesion area based on visual features and CNN. In recent years, some popular deep learning models, such as AlexNet, Inception, VGGNet, and ResNet, have been used to identify plant diseases. In these modes, VGG is a relatively simple network. Therefore, VGG 16 is selected to highlight the advantages of improvement.The original VGG16 is shown in Figure1(a). Figures 1(b)–1(e) show four fusion structures that increase the information input of the sample lesion location on the original network, which are named C0, C1, C2, and C4. These fusion structures only replace the convolution layer corresponding to the original network and do not change the subsequent structure of the original network.Figure 1
Original CNN and feature fusion structures. (a) Original CNN. (b) Fusion structure C0. (c) Fusion structure C1. (d) Fusion structure C2. (e) Fusion structure C4.
(a)(b)(c)(d)(e)The main parameters of the network structure are also shown in this diagram. Generally, CNN maps input featuresX=x1,x2,⋯,xC×H×W to predicted labels Y=y1,y2,⋯,yM in classification tasks by convolution operation. C is the number of channels of the convolution kernel. H and W are the length and width of the input image, respectively. M is the number of categories. Each component yi of Y is calculated by Equation (9).
(9)yi=ki∗X=∑j=1Ckij∗xj.In the equation, “∗” denote convolution calculation, and ki is the convolution kernel. And loss function is following:
(10)L=1N∑iLi=−1N∑i∑c=1Myiclogpic.M is the number of categories. N is the number of samples. If the sample I belong to category c, yic=1; otherwise, yic=0, pic is the prediction probability that sample I belongs to category c.In this article, the input sample feature of the original network is assumed to beXi=x1,x2,⋯,xC×H×W and its lesion location is assumed to be Xi′=x1,x2,⋯,xN. Based on the expansion mechanism, the new input feature vector Xi′ is from [fIxmin,t0]. The new convolution formula is Equations 17 and 18
(11)Xi′′′=Xi+Xi∗Xi′,yi=ki∗Xi′′′=∑j=1Ckij∗xi′′′.


## 3. Results

### 3.1. Experiment Setup

The experiment was carried out on a computer equipped with an NVIDIA GeForce RTX 2060Ti GPU. The CPU is Intel (R) Core (TM) i7-9700 CPU and runs under a 64-bit Windows 10 operating system. The programming language version is Python 3.6, the deep learning framework is TensorFlow 2.4.0, and the number of iterations is 100. The learning rate is 0. 001.


### 3.2. Data Acquisition

The dataset used in the experiment cones from PlantVillage and PlantDoc. The dataset contained three diseases of apple leaves named black rot, scab, and rust. The number of samples for each disease grade is 1500. Among them, the number of leaves belonging to black rot 4 in the original dataset is less. Moreover, data enhancement will cause the network to over-fit the samples under this grade, so the leaves belonging to black rot 4 will be discarded in the dataset. 90% of the dataset is the training set and 10% of the dataset is the test set. The performance of the proposed algorithm is evaluated on PlantVillage and the combination of PlantVillage and PlantDoc. Some sample lesions of this dataset are shown in Figure2.Figure 2
Leaf samples of three apple diseases in the dataset.


### 3.3. Performance Metrics

The research in this paper belongs to the multi-classification task. To better measure the performance of the model, the following indicators are selected in this paper including accuracy, recall, the average accuracy of every disease, and average recall of every disease. To evaluate these indicators, we need paraments such as true positive (TP), false positive (FP), true negative (TN), and false negative (FN). TP is the correct classification of images in a particular category. FP is the number of incorrectly classified images, TN is the sum of correctly classified images in all other categories, and FN is the number of incorrectly classified images in related categories.(12)Accuracy=TP+TNTP+FP+TN+FN,Recall=TPTP+FN.Average accuracy is the average value of the accuracy of four grades for each disease. The average recall is the average recall of four grades for each disease. As mentioned above, the leaves belonging to black rot 4 are removed.


### 3.4. Identification Results

In this paper, to reduce the influence of the random division of dataset on the experimental results, each group of experiments was repeated five times. And the average values of five groups of experimental results are shown in this paper.


#### 3.4.1. Identification Results of the Attention Mechanism using Different Fusion Structures

In theMethodology section, a new attention mechanism was put forward and different fusion structures are put forward of the new attention mechanism. Table 1 shows the recognition results of attention mechanisms using different fusion structures. These experiments were compared with the experiment without fusion structure. Table 2 shows the accuracy of each category for each experiment. The dataset used in these experiments is the combined dataset of PlantVillage and PlantDoc, and the data enhancement method used in these experiments is traditional data enhancement method.Table 1
Identification results of the attention mechanism using different fusion structures.
Fusion structure nameAccuracy/(%)Recall/(%)Average accuracy/(%)Average recall/(%)Black rotScabRustBlack rotScabRustNo fusion84.8185.8683.6773.7596.2584.2077.1596.23C086.8687.7387.3379.0094.2585.5782.6195.01C187.2288.1384.8783.4196.1185.6780.2595.75C287.5688.1384.6784.2593.7583.9084.9995.49C489.7590.7790.0080.7598.5089.7384.0098.58Notes: All values are in percentage.Table 2
Accuracy of the attention mechanism using different fusion structures.
No fusionC0C1C2C4Black rot 187.3384.0076.0079.3383.33Black rot 266.0080.0084.0078.0090.00Black rot 398.0098.0097.3397.3397.33Scab 168.0074.6775.3374.0074.00Scab 264.0068.0074.0079.3378.00Scab 369.3371.3378.0090.0081.33Scab 494.0094.0094.0094.0094.00Rust 193.3393.3398.0097.3398.67Rust 292.0086.0083.3392.0094.00Rust 398.6798.0099.3397.3398.00Rust 498.0096.6798.0098.0098.00Notes: All values are in percentage.


#### 3.4.2. Identification Results Using Different Data Enhancement Methods

Table3 shows the accuracy rate and recall rate of three experiments using different data enhancement methods with the new attention mechanism proposed in this paper. The first experiment did not use the method of data enhancement. The second experiment used the traditional method of data enhancement, and the third experiment used the method of data enhancement proposed in this paper. Figure 3 shows the confusion matrix of three groups of experiments. The dataset used in these experiments is the combined dataset of PlantVillage and PlantDoc.Table 3
Identification results using different data enhancement methods.
Accuracy/(%)Recall/(%)Average accuracy/(%)Average recall/(%)Black rotScabRustBlack rotBlack rotScabNo data enhancement72.1473.0578.6763.0074.7570.4868.9679.71Original data enhancement89.7590.7790.0080.7598.5089.7384.0098.58New methods proposed in this paper91.5592.0690.3385.7598.6789.6787.3298.58Notes: All values are in percentage.Figure 3
Confusion matrix of using different data enhancement methods on the combined dataset of PlantVillage and PlantDoc. (a) No data enhancement. (b) Original data enhancement. (c) New methods proposed in this paper.
(a)(b)(c)


#### 3.4.3. Identification Results Using Different Attention Mechanisms

Figure4 shows the convergence curves of the attention mechanism proposed in this paper and recent classic attention mechanisms. Among them, the attention mechanism proposed in this paper uses the fusion structure C4. Tables 4 and 5 show how identification results that different attention mechanisms were tested on the dataset of PlantVillage and the combination of PlantVillage and PlantDoc respectively.Figure 4
Convergence comparison using different attention mechanisms on the combined dataset of PlantVillage and PlantDoc.Table 4
Identification results using different attention mechanisms on the combined dataset of PlantVillage and PlantDoc.
NO attention mechanismCBAMSE blockSSDFPNNew approachIdentification accuracy84.8185.3685.6285.7284.7291.55Notes: All values are in percentage.Table 5
Identification results using different attention mechanism on PlantVillage.
NO attention mechanismCBAMSE blockSSDFPNNew approachIdentification accuracy85.2686.3786.3387.0185.2392.38Notes: All values are in percentage.


## 3.1. Experiment Setup

The experiment was carried out on a computer equipped with an NVIDIA GeForce RTX 2060Ti GPU. The CPU is Intel (R) Core (TM) i7-9700 CPU and runs under a 64-bit Windows 10 operating system. The programming language version is Python 3.6, the deep learning framework is TensorFlow 2.4.0, and the number of iterations is 100. The learning rate is 0. 001.


## 3.2. Data Acquisition

The dataset used in the experiment cones from PlantVillage and PlantDoc. The dataset contained three diseases of apple leaves named black rot, scab, and rust. The number of samples for each disease grade is 1500. Among them, the number of leaves belonging to black rot 4 in the original dataset is less. Moreover, data enhancement will cause the network to over-fit the samples under this grade, so the leaves belonging to black rot 4 will be discarded in the dataset. 90% of the dataset is the training set and 10% of the dataset is the test set. The performance of the proposed algorithm is evaluated on PlantVillage and the combination of PlantVillage and PlantDoc. Some sample lesions of this dataset are shown in Figure2.Figure 2
Leaf samples of three apple diseases in the dataset.


## 3.3. Performance Metrics

The research in this paper belongs to the multi-classification task. To better measure the performance of the model, the following indicators are selected in this paper including accuracy, recall, the average accuracy of every disease, and average recall of every disease. To evaluate these indicators, we need paraments such as true positive (TP), false positive (FP), true negative (TN), and false negative (FN). TP is the correct classification of images in a particular category. FP is the number of incorrectly classified images, TN is the sum of correctly classified images in all other categories, and FN is the number of incorrectly classified images in related categories.(12)Accuracy=TP+TNTP+FP+TN+FN,Recall=TPTP+FN.Average accuracy is the average value of the accuracy of four grades for each disease. The average recall is the average recall of four grades for each disease. As mentioned above, the leaves belonging to black rot 4 are removed.


## 3.4. Identification Results

In this paper, to reduce the influence of the random division of dataset on the experimental results, each group of experiments was repeated five times. And the average values of five groups of experimental results are shown in this paper.


### 3.4.1. Identification Results of the Attention Mechanism using Different Fusion Structures

In theMethodology section, a new attention mechanism was put forward and different fusion structures are put forward of the new attention mechanism. Table 1 shows the recognition results of attention mechanisms using different fusion structures. These experiments were compared with the experiment without fusion structure. Table 2 shows the accuracy of each category for each experiment. The dataset used in these experiments is the combined dataset of PlantVillage and PlantDoc, and the data enhancement method used in these experiments is traditional data enhancement method.Table 1
Identification results of the attention mechanism using different fusion structures.
Fusion structure nameAccuracy/(%)Recall/(%)Average accuracy/(%)Average recall/(%)Black rotScabRustBlack rotScabRustNo fusion84.8185.8683.6773.7596.2584.2077.1596.23C086.8687.7387.3379.0094.2585.5782.6195.01C187.2288.1384.8783.4196.1185.6780.2595.75C287.5688.1384.6784.2593.7583.9084.9995.49C489.7590.7790.0080.7598.5089.7384.0098.58Notes: All values are in percentage.Table 2
Accuracy of the attention mechanism using different fusion structures.
No fusionC0C1C2C4Black rot 187.3384.0076.0079.3383.33Black rot 266.0080.0084.0078.0090.00Black rot 398.0098.0097.3397.3397.33Scab 168.0074.6775.3374.0074.00Scab 264.0068.0074.0079.3378.00Scab 369.3371.3378.0090.0081.33Scab 494.0094.0094.0094.0094.00Rust 193.3393.3398.0097.3398.67Rust 292.0086.0083.3392.0094.00Rust 398.6798.0099.3397.3398.00Rust 498.0096.6798.0098.0098.00Notes: All values are in percentage.


### 3.4.2. Identification Results Using Different Data Enhancement Methods

Table3 shows the accuracy rate and recall rate of three experiments using different data enhancement methods with the new attention mechanism proposed in this paper. The first experiment did not use the method of data enhancement. The second experiment used the traditional method of data enhancement, and the third experiment used the method of data enhancement proposed in this paper. Figure 3 shows the confusion matrix of three groups of experiments. The dataset used in these experiments is the combined dataset of PlantVillage and PlantDoc.Table 3
Identification results using different data enhancement methods.
Accuracy/(%)Recall/(%)Average accuracy/(%)Average recall/(%)Black rotScabRustBlack rotBlack rotScabNo data enhancement72.1473.0578.6763.0074.7570.4868.9679.71Original data enhancement89.7590.7790.0080.7598.5089.7384.0098.58New methods proposed in this paper91.5592.0690.3385.7598.6789.6787.3298.58Notes: All values are in percentage.Figure 3
Confusion matrix of using different data enhancement methods on the combined dataset of PlantVillage and PlantDoc. (a) No data enhancement. (b) Original data enhancement. (c) New methods proposed in this paper.
(a)(b)(c)


### 3.4.3. Identification Results Using Different Attention Mechanisms

Figure4 shows the convergence curves of the attention mechanism proposed in this paper and recent classic attention mechanisms. Among them, the attention mechanism proposed in this paper uses the fusion structure C4. Tables 4 and 5 show how identification results that different attention mechanisms were tested on the dataset of PlantVillage and the combination of PlantVillage and PlantDoc respectively.Figure 4
Convergence comparison using different attention mechanisms on the combined dataset of PlantVillage and PlantDoc.Table 4
Identification results using different attention mechanisms on the combined dataset of PlantVillage and PlantDoc.
NO attention mechanismCBAMSE blockSSDFPNNew approachIdentification accuracy84.8185.3685.6285.7284.7291.55Notes: All values are in percentage.Table 5
Identification results using different attention mechanism on PlantVillage.
NO attention mechanismCBAMSE blockSSDFPNNew approachIdentification accuracy85.2686.3786.3387.0185.2392.38Notes: All values are in percentage.


## 3.4.1. Identification Results of the Attention Mechanism using Different Fusion Structures

In theMethodology section, a new attention mechanism was put forward and different fusion structures are put forward of the new attention mechanism. Table 1 shows the recognition results of attention mechanisms using different fusion structures. These experiments were compared with the experiment without fusion structure. Table 2 shows the accuracy of each category for each experiment. The dataset used in these experiments is the combined dataset of PlantVillage and PlantDoc, and the data enhancement method used in these experiments is traditional data enhancement method.Table 1
Identification results of the attention mechanism using different fusion structures.
Fusion structure nameAccuracy/(%)Recall/(%)Average accuracy/(%)Average recall/(%)Black rotScabRustBlack rotScabRustNo fusion84.8185.8683.6773.7596.2584.2077.1596.23C086.8687.7387.3379.0094.2585.5782.6195.01C187.2288.1384.8783.4196.1185.6780.2595.75C287.5688.1384.6784.2593.7583.9084.9995.49C489.7590.7790.0080.7598.5089.7384.0098.58Notes: All values are in percentage.Table 2
Accuracy of the attention mechanism using different fusion structures.
No fusionC0C1C2C4Black rot 187.3384.0076.0079.3383.33Black rot 266.0080.0084.0078.0090.00Black rot 398.0098.0097.3397.3397.33Scab 168.0074.6775.3374.0074.00Scab 264.0068.0074.0079.3378.00Scab 369.3371.3378.0090.0081.33Scab 494.0094.0094.0094.0094.00Rust 193.3393.3398.0097.3398.67Rust 292.0086.0083.3392.0094.00Rust 398.6798.0099.3397.3398.00Rust 498.0096.6798.0098.0098.00Notes: All values are in percentage.


## 3.4.2. Identification Results Using Different Data Enhancement Methods

Table3 shows the accuracy rate and recall rate of three experiments using different data enhancement methods with the new attention mechanism proposed in this paper. The first experiment did not use the method of data enhancement. The second experiment used the traditional method of data enhancement, and the third experiment used the method of data enhancement proposed in this paper. Figure 3 shows the confusion matrix of three groups of experiments. The dataset used in these experiments is the combined dataset of PlantVillage and PlantDoc.Table 3
Identification results using different data enhancement methods.
Accuracy/(%)Recall/(%)Average accuracy/(%)Average recall/(%)Black rotScabRustBlack rotBlack rotScabNo data enhancement72.1473.0578.6763.0074.7570.4868.9679.71Original data enhancement89.7590.7790.0080.7598.5089.7384.0098.58New methods proposed in this paper91.5592.0690.3385.7598.6789.6787.3298.58Notes: All values are in percentage.Figure 3
Confusion matrix of using different data enhancement methods on the combined dataset of PlantVillage and PlantDoc. (a) No data enhancement. (b) Original data enhancement. (c) New methods proposed in this paper.
(a)(b)(c)


## 3.4.3. Identification Results Using Different Attention Mechanisms

Figure4 shows the convergence curves of the attention mechanism proposed in this paper and recent classic attention mechanisms. Among them, the attention mechanism proposed in this paper uses the fusion structure C4. Tables 4 and 5 show how identification results that different attention mechanisms were tested on the dataset of PlantVillage and the combination of PlantVillage and PlantDoc respectively.Figure 4
Convergence comparison using different attention mechanisms on the combined dataset of PlantVillage and PlantDoc.Table 4
Identification results using different attention mechanisms on the combined dataset of PlantVillage and PlantDoc.
NO attention mechanismCBAMSE blockSSDFPNNew approachIdentification accuracy84.8185.3685.6285.7284.7291.55Notes: All values are in percentage.Table 5
Identification results using different attention mechanism on PlantVillage.
NO attention mechanismCBAMSE blockSSDFPNNew approachIdentification accuracy85.2686.3786.3387.0185.2392.38Notes: All values are in percentage.


## 4. Discussion

From Table1, it can be seen that the experiment with the fusion structure C4 has the best performance. The accuracy of the experiment with the fusion structure C4 is 89.75% and the recall rate is 90.77%. Compared with the experiment without attention mechanism, the accuracy rate and recall rate were improved by 4.94% and 4.91%, respectively. The lowest average accuracy was 80.75% and the highest average accuracy was 98.50%. The lowest average recall rate is 84.00%, and the highest average recall rate is 98.58%. And all average accuracy and average recall of the three diseases are higher than those of the experiment without attention mechanism. Therefore, the location of the fused lesion strengthens the difference between lesion area and other area, which makes the network pay more attention to lesion area.Compared with fusion structure C0, the accuracy and recall rate of the fusion structure C1 increased by 3.12% and 2.46%, respectively. Compared with fusion structure C1, the accuracy and recall rate of the fusion structure C2 increased by 0.52% and 0.09%, respectively. Compared with fusion structure C2, the accuracy and recall rate of fusion structure C4 are improved by 0.48% and 0.97%, respectively. It can be seen from Figures1(b)–1(e) that the fusion structures CO, C1, C2, and C4 are different in the location of regional information fusion. With the deepening of the fusion localization, the classification effect of this model is getting better and better. The reason is that the deeper feature vector is derived from the intermediate feature vector by convolution operation and the deeper feature vector is a better expression than the intermediate feature vector.As can be seen from Table2, the classification accuracy of the experiment without attention mechanism on the samples of black rot 1, black rot 2, scab 1, scab 2, and scab 3 is low, which are no higher than 70%. Most of the mistaken identification comes from slight disease grade to mild disease grade. In the early stage of disease growth, the change of disease characteristics is vaguer than that in the late stage of disease, and it is more likely to lead to the wrong judgment of disease grade. The accuracy of these experiments with attention mechanism using different fusion structures was higher than the experiment without attention mechanism. After using the attention mechanism proposed in this paper, CNN pays more attention to the lesion area, which improves the feature extraction ability of the network.It can be seen from Table3 that the accuracy and recall are 72.14% and 73.05%, respectively, before using data enhancements to the dataset. In the experiment that the traditional data enhancement method was used, the accuracy rate and recall rate are 84.81% and 85.86%, respectively. And the accuracy rate and recall rate were improved by 12.67% and 12.81%, respectively, compared to the experiments without data enhancement method. In the experiment that the new data enhancement method was used, the accuracy and recall are 91.55% and 92.06%, respectively, which are increased by 1.68% and 0.72% compared with the experiment with the traditional data enhancement method.Figures3(a)–3(c) show the confusion matrix of three groups of experiments, which introduces in detail the accuracy predicted image scores of three apple leaf diseases in each grade. Compared with the experiments without data enhancement, the average accuracy of black rot, scab, and rust of the experiment with traditional data enhancement methods increased by 11.33%, 16.25%, and 23.75%, respectively. Compared with the experiments without data enhancement, the average accuracy of black rot, scab, and rust of the experiment with traditional data enhancement methods increased by 11.67%, 21.25%, and 24.08%, respectively. The analysis of the above three diseases shows that, compared with the traditional data enhancement method, the data enhancement method proposed in this paper can better improve the classification effect of scab disease grades. The reasons can be divided into two parts: (1) Traditional data enhancement can reduce the influence of too few images on classification results, by making the number of pictures in different categories equal. (2) The local similarity and local discontinuity of all sample features in the same disease grade were reduced.Figures3(a)–3(c) show the identification accuracy of different categories in the method proposed in this paper. Most of the mistaken identification comes from disease grade 1 to disease grade 2. In the early stage of disease growth, the change of disease characteristics is vaguer than that in the late stage of disease, and it is more likely to lead to the wrong judgment of disease grade.The attention mechanism proposed in this paper was compared with the classical attention mechanism on the PlantVillage dataset and the combined dataset of PlantVillage and PlantDoc. These classical attention mechanisms include CBAM, SE block, SSD, and FPN. Figure4 shows the convergence curve of the accuracy of these experiments. As shown in Figure 4, the network without attention mechanism, four classic attention mechanism networks, and the proposed attention mechanism began to converge after a certain period of time, and finally achieved the best recognition performance. Generally speaking, the new attention mechanism is basically stable after 30 stages, while other models have satisfactory convergence after 40 stages.Tables4 and 5 show identification results that different attention mechanisms were tested on the dataset of PlantVillage and the combination of PlantVillage and PlantDoc, respectively. The recognition accuracy of 11 categories without attention mechanism is 84.81% on the combined dataset of PlantVillage and PlantDoc. The recognition accuracy with CBAM, SE block, SSD, and FPN is 85.36%, 85.62%, 85.72%, and 84.72%, respectively, on the combined dataset of PlantVillage and PlantDoc. And the recognition accuracy of 11 categories with attention mechanisms is 91.55%, which is superior to existing methods.The traditional attention mechanism changes the weight of the network to all pixels in the image, which affects the ability of the network to extract the features of the lesion area. The new attention mechanism is proposed by integrating the acquired information of the lesion area with global information. This method can increase the attention of the network to the lesion area, while the influence of other areas on the attention of the network to the lesion area can be ignored. Compared with the classic attention mechanism, the new attention mechanism improves the convergence speed of the network model, and provides higher recognition accuracy for apple leaf disease grades. Those methods were also verified on the dataset of PlantVillage.


## 5. Conclusions

Images of apple leaf diseases collected under natural conditions have similar characteristics, but with different degrees. It is difficult to identify apple leaf diseases and their severity quickly and accurately by the existing methods. The traditional attention mechanism changes the weight of the network to all pixels in the image, which affects the ability of the network to extract the features of the lesion area. And the traditional method of data enhancement is easy to cause local similarity and local discontinuity of all sample features in the same disease grade. In this paper, the improved metric matrix for kernel regression (IMMKR) proposed was firstly used to reduce the influence of local similarity and local discontinuity of all sample features in the same class. Then, a new attention mechanism by fusing the lesion location based on visual features was proposed, and the attention of the model to the lesion area was strengthened. The experimental results show that the algorithm can improve the recognition accuracy of apple leaf disease grades, and meet the intelligent recognition requirements of different apple leaf diseases. This algorithm has good reference significance for the identification and promotion of crop diseases. In the future, the algorithm proposed in this study will be introduced into software or hardware products, such as apple leaf disease identification software, fine-grained visual classification, and temporal prediction [33–41].



---

*Source: 3781859-2022-04-23.xml*