Ensemble cross‐stage partial attention network for image classification

IET Image ProcessingVolume 16, Issue 1 p. 102-112 ORIGINAL RESEARCH PAPEROpen Access Ensemble cross-stage partial attention network for image classification Hai Lin, Corresponding Author Hai Lin linhai@zhjpec.edu.cn orcid.org/0000-0002-5598-9149 Department of Information Science, ZhanJiang Preschool Education College, Guangdong, China Correspondence Hai Lin, Department of Information Science, ZhanJiang Preschool Education College, Guangdong, China. Email: linhai@zhjpec.edu.cnSearch for more papers by this authorJunJie Yang, JunJie Yang College of Information Science and Technology, Lingnan Normal University, Guangdong, ChinaSearch for more papers by this author Hai Lin, Corresponding Author Hai Lin linhai@zhjpec.edu.cn orcid.org/0000-0002-5598-9149 Department of Information Science, ZhanJiang Preschool Education College, Guangdong, China Correspondence Hai Lin, Department of Information Science, ZhanJiang Preschool Education College, Guangdong, China. Email: linhai@zhjpec.edu.cnSearch for more papers by this authorJunJie Yang, JunJie Yang College of Information Science and Technology, Lingnan Normal University, Guangdong, ChinaSearch for more papers by this author First published: 24 September 2021 https://doi.org/10.1049/ipr2.12335AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract This paper proposes a novel image classification architecture named ensemble cross-stage partial attention network based on the backbone network DarkNet53 of Yolov3 to improve the feature extraction capability and the interpretability of image classification. This network has multiple advantages, including light model parameters, fast classification speed, and high classification accuracy for small objects and complex images. Local network architectures of different cross-phases are added in the proposed network structure to reduce the calculation. Furthermore, channel and hybrid domain attention modules, which, respectively, fuse the branch feature with the extracted channel and spatial attention features, are designed for feature extraction of images. Experimental results confirm the improved performance of the proposed approach on the CIFAR-100, ImageNet, and UCMerced datasets. In addition, experiments on the MSCOCO dataset suggest the application of the proposed method to object detection with satisfactory accuracy. 1 INTRODUCTION Convolutional neural network (CNN) in deep learning has been widely applied in the field of image classification. Different from the traditional image classification methods, neural network does not need to set and extract the features of the object image artificially and automatically learns the closely related image features with classification results from training datasets. Thus, neural networks effectively address the problems of artificial feature extraction and low classification accuracy of traditional methods. At present, deep learning has been successfully applied in computer vision, super-resolution, image classification reconstruction, and object detection. In many methods of deep learning [1] [2], CNN has gained extensive attention from researchers. As the image classification network from the early LeNet-5 [3], Alex [4], ZF-net [5], and VGG [6] to GoogLeNet [7], ResNet [8], SENet [9], and lightweight mobilenet [10] with high classification accuracy, new network architectures based on CNN, such as [11] and [12], are emerging. Effective feature representation can be obtained by increasing the depth and width of the network [13]. However, such representation is subsequently followed by the increase in computation, easy overfitting, gradient disappearance, and gradient explosion. In addition, the convolution kernel of fixed size, which can only obtain the features of the local fixed receptive field [14] and cannot consider the global image features, is commonly used in the image classification process based on CNN. Therefore, this paper proposes an image classification network named ensemble cross-stage partial attention (ECSPA) network to meet the real-time requirements in the actual environment. ECSPA, which is based on the network structure of DarkNet53, adds two new cross-stage partial (CSP) network gradient combinations to FirstStage and four stages to reduce the repeated gradient information of the network. Many experiments show that the two kinds of CSP combinations reduce the computation and the number of model parameters by 33% and 30%, respectively. The attention mechanism is introduced to the aspect of image feature extraction, the CBAM and SENet attention modules are, respectively, added to the blocks of the FirstStage and the next four stage units, and the branch feature is fused with the extracted channel attention and spatial attention feature. Several experiments reveal the proposed method's high precision on CIFAR-100, ImageNet, and UCMerced_LandUse. This network also has strong robustness, in which ECSPA is the backbone of object detection, and the AP value of the network is considerably better than the state-of-the-art method on the MSCOCO dataset. The main contributions of this paper are as follows. 1) The computation of the classification network and the model parameters are reduced, and the classification accuracy is improved by using two kinds of new CSP gradient combinations. 2) Channel and spatial attention features are fused by, respectively, using SENet and CBAM attention modules to improve the accuracy of image feature extraction. The rest of this paper is arranged as follows. Section 2 reviews the related research work. Section 3 comprehensively describes ADBNet. Section 4 analyzes the experimental data and results. Finally, Section 5 draws the conclusions. 2 RELATED WORK This section introduces and discusses the cross-phase local network, attention mechanism, and DarkNet53. 2.1 Cross-phase local network The current network model, which can achieve good results in computer vision tasks, often substantially depends on expensive computing resources and increases the cost of computing [15, 16]. Therefore, Wang et al. proposed a cross-stage local network CSP [17] to solve the problem, which required many computations in previous studies, from the perspective of network architecture. The research demonstrates the presence of many repeated gradient information in the network. The importance of gradient variability in the image feature mapping at the beginning and end of the network must be emphasized to address this problem. First, the feature graph is divided into two parts and then combined on the basis of the cross-stage hierarchical structure. The idea is to divide the gradient flow to facilitate distribution through different network paths. The gradient information distribution can vary remarkably through switching cascading and conversion steps. This difference offsets a considerable amount of repeated gradient information and reduces the computation. Implementing CSPNet is easy and is generally sufficient to handle architectures based on ResNet, VGG, and DenseNet [18]. At present, the CSP architecture has four gradient combinations as shown in Figure 1. Section 3 further designs the two new combinations suitable for ECSPA and CSP. FIGURE 1Open in figure viewerPowerPoint Four gradient combinations of CSP architecture 2.2 Visual attention mechanism The basic idea of the visual attention mechanism is to facilitate attention learning of the system, ignore irrelevant information, and focus on the important information. In recent years, research on the combination of deep learning and visual attention mechanism focused on the use of masks to form the attention mechanism. The principle is to establish a new weight to mark the features related to the category in the image feature data. The network can then learn the areas requiring focus in each new image to form attention via machine learning training. The attention mechanism models in computer vision [19] can be divided into three categories: spatial, channel, and hybrid domains. (1) Spatial domain: The key information can be extracted by the corresponding spatial transformation of the spatial domain information in the picture, and the mask of the space can be generated and scored. (2) Channel domain: This domain is similar to establishing a new weight for the signal of each channel to represent its correlation level and the key information; a large weight value indicates a high correlation. The mask is then generated for the channel for scoring . Hu et al. proposed the channel domain attention module of SENet by utilizing and innovating the dependence of feature channels in the network [20]. The SENet module is characterized by its simple structure, easy implementation, and easy loading into the existing network module framework. SENet adaptively recalibrates the feature response between channels through the establishment of interdependence between channels and learns the correlation between channels by studying the train to screen the attention that affects the channel. Such attention slightly increases the computation but improves the classification accuracy. (3) Hybrid domain: The attention in the spatial domain ignores the information in the channel domain and equally processes the image features in each channel. Thus, the attention in the spatial domain is only generally applied to the shallow feature extraction, in which interpretability will be weak if applied to the other layers of the neural network. The attention of the channel domain aims to avgpool the information directly and globally and ignore the local image feature information in each channel, which is a violent behavior. Therefore, Woo et al. [21] proposed the attention module CBAM of the hybrid domain by combining the two ideas. The attention of hybrid domain not only indicates what should be considered but also improves the expression of concerns, focuses on important features, and suppresses unnecessary features. The author successively applies channel and spatial attention modules to learn what and where to focus in the channel and spatial dimensions, respectively, to emphasize the features of space and channel. In addition, understanding the information to be stressed or suppressed is conducive to the information flow in the network. Determining the applicable attention module for each stage through the comparison of theoretical analysis and experimental data is necessary for a complete network [22]. Section 3 further discusses the design of the ECSPA attention module. 2.3 DarkNet-53 feature extraction network DarkNet53, which is the backbone network of image feature extraction of the object detection network YOLOv3, was proposed by Joseph et al. [23]. This network comprises a FirstStage unit, which contains one block and four stage units , each containing repeated blocks. The network model combines the characteristics of deep residual network ResNet and DarkNet10 [24], which is comparable with the most advanced classifier and has minimal floating computation and fast speed. For example, the performance of DarkNet 53 is better than ResNet101, and the classification speed is 1.5 times faster. This finding is due to the excessive number of layers of ResNet101, which increases the computing cost. The DarkNet can also achieve the highest floating operations per second; thus, the network structure can effectively use GPU to realize efficient and fast classification. With the update of YOLOv4 [25], the author proposed an updated version of the backbone network CSPDarkNet53, which adds CSP on the basis of the DarkNet53. However, the comparison of accuracy experimental data on the ImageNet classification dataset presented in Table 1 shows that the speed is reduced by four times despite the accuracy improvement by 0.24. The experimental research revealed that the speed reduction is attributed to the unreasonable structure design of CSP. Therefore, designing a reasonable CSP structure and introducing the attention mechanism are necessary to further improve the speed and accuracy of classification. TABLE 1. Accuracy and time consumption of classification on the ImageNet dataset model input_size Top1 acc i5 speed(10times) DarkNet53 256 × 256 76.52% 0.2 s cspDarkNet53 256 × 256 76.76% 0.8 s 3 ECSPA NETWORK This section describes the structure of the ECSPA network. Then, the hybrid CSP architecture, SENet module, CBAM module, and adopted activation function of ECSPA are introduced and comprehensively discussed. 3.1 Construction of ECSPA network ECSPA is an image classification network based on DarkNet53. This network uses the filter size of the network, adds two new CSP gradient combinations in the downsampling layer of the FirstStage and four stage units, reduces the parameter for easy training, and introduces two attention mechanisms (i.e., CBAM and SENet), thus improving the classification accuracy. Figure 2 shows the network structure. The input image outputs 32 shallow image feature channels by 3 × 3 convolutions and enters the FirstStage unit of the network followed by four groups of stage units with the same structure. Each structure comprises the following: a downsampling layer; a CSP architecture; a group of repeated executing blocks that repeat 1, 2, 8, 8, and 4 times; an attention module; a cat fusion; and a 1 × 1 convolution. The image features of outputs go through the Avgpool, Connected, and Softmax layers after finishing the four stages. Finally, the table of classification results is obtained. FIGURE 2Open in figure viewerPowerPoint ECSPA network structure 3.2 Cross-Stage Partial Network (CSP) architecture in ECSPA Two new CSP gradient combination designs are adopted in the FirstStage and Stage of ECSPA. (1) CSP gradient combination of the FirstStage unit A total of 64 channels are outputted through downsampling in the FirstStage unit. The image features can be divided into two branches according to the principle of CSP. The first branch input requires blocks with 64 channels, and image features in the second branch are used for fusion. Without changing the number of input channels, Figure 3 shows the CSP architecture adopted by CSPDarkNet53. Instead of splitting the image feature x0 , image feature x0 is inputted to obtain two features x 0 ′ and x 0 ′ ′ through the same 1 × 1 convolutional layer. The first part x 0 ′ is inputted to the blocks, and the second part x 0 ′ ′ is directly connected to the cat operation for fusion with the output of the blocks, which adopts two 1 × 1 convolutional layers and consequently increases the calculation time. The same CSP design is implemented in the FirstStage unit and the following four stage units, which increases the computational burden of 10 convolutional layers compared with the original DarkNet53. Therefore, the actual test speed of CSPDarkNet53 is slower than that of DarkNet53. FIGURE 3Open in figure viewerPowerPoint CSP gradient combination This paper innovatively designs a CSP gradient combination as shown in Figure 4 to reduce the computational cost. The image feature x 0 ′ is first outputted through a 1 × 1 convolutional layer with 64 channels, and then x 0 ′ is assigned to x 0 ′ ′ . Therefore, two same image features are obtained. The first part x 0 ′ is inputted to the blocks, and the second part x 0 ′ ′ is directly connected to the cat operation for fusion with the output of the blocks. This design reduces a 1 × 1 convolutional layer and shortens the computational time. FIGURE 4Open in figure viewerPowerPoint New CSP gradient combination design of the FirstStage unit (2) CSP gradient combination of the stage unit Following the FirstStage unit, four groups of stage units have the same structure. Each stage contains numerous repeated blocks. Therefore, each block should comprise k layers before providing the CSP design. The output of the k − 1 layer will be stacked with the input of the k layer, and the stacked output will become the input of the k + 1 layer. The forward propagation formula is as follows: x 1 = d 1 × x 0 , (1) x 2 = d 2 × [ x 0 , x 1 ] , (2) … … (3) x k = d k × [ x 0 , x 1 , … , x k − 1 ] , (4)where the notation × represents the convolution operation, and [ x 0 , x 1 , … , x k − 1 ] represents stacking. x 0 , x 1 , … , and d k and x k are the weights of k and the output layers, respectively. The backpropagation algorithm is adopted to update the formula as follows: d 1 ′ = f ( d 1 , g 0 ) , (5) d 2 ′ = f ( d 1 , g 0 , g 1 ) , (6) d 3 ′ = f ( d 1 , g 0 , g 1 , g 2 ) , (7) … … (8) d k ′ = f ( d 1 , g 0 , g 1 , g 2 , … , g k − 1 ) , (9)where f is the weight update function that represents the gradient propagated to the k layer. Most gradient information is repeatedly used to update the weights of different layers, thus allowing different layers to learn the same gradient information repeatedly. Therefore, a CSP structure is necessary to offset the same gradient information. Figure 5 shows the new CSP gradient combination design of the four stage units. The image features of each stage unit first pass through the downsampling layer and are equally split into two parts x 0 = [ x 0 ′ , x 0 ′ ′ ] . The number of channels after splitting is similar to that of input channels of the blocks. For example, the output of the downsampling layer is 128 channels in the first stage unit, and the number of channels after splitting is 64. Therefore, adding a 1 × 1 convolutional layer similar to the FirstStage unit is unnecessary for conversion, which reduces the computational cost. In the first part, x 0 ′ is directly connected to the cat operation to be fused with the output of SENet. In the second part, x 0 ′ ′ is directly inputted into the blocks. The calculation steps in the blocks are as follows. First, [ x 0 ′ ′ , x 1 , … , x k ] will pass through one block. Then, the output of the blocks will be stacked with the previous blocks and pass through another block. Finally, x u is outputted. The formula of forward propagation weight is as follows: x k = d k × x 0 ′ ′ , x 1 , … , x k − 1 , (10) x h = d h × x 0 ′ ′ , x 1 , … , x k , (11) x u = d u × x 0 ′ , x h . (12) FIGURE 5Open in figure viewerPowerPoint New CSP gradient combination design of the stage unit The weights are updated using the backpropagation algorithm, and the updated formula is as follows: d k ′ = f d k , g 0 ′ ′ , g 1 , g 2 , … , g k − 1 , (13) d h ′ = f d h , g 0 ′ ′ , g 1 , g 2 , … , g k , (14) d u ′ = f d u , g 0 ′ , g h . (15) Notably, the gradients from blocks are integrated accordingly. In addition, the feature map x 0 ′ , which does not pass through the blocks, is also integrated independently, and the gradient information used for weight updating contains no redundant gradient information of each other. Therefore, the same gradient information can be effectively offset. When the number of input channels of stage units is twice as large as that of the blocks, the CSP structure can be utilized to segment the image features equally, and the number of channels is halved similar to that of input channels of the blocks. This method can solve nearly half of the computational bottleneck and reduce the memory flow. Suppose the size of the basic feature map of the blocks in the stage is w × h × k , the growth rate is e, and a total of m blocks are available. Then, the Convolutional Input/Output (CIO) of the blocks is ( k × m ) + ( ( m 2 + m ) × e ) / 2 , whereas that of some blocks is ( ( k × m ) + ( m 2 + m ) × e ) / 2 . m and e are substantially smaller than k; thus, the latter can save up to half of the memory flow and further reduce the model parameters. 3.3 Soft attention mechanism in ECSPA (1) Structure of convolutional block attention module (CBAM) The application of the soft attention mechanism can improve the accuracy of the classification network. Figure 2 shows that the CBAM hybrid domain attention module is connected to the block output of the FirstStage unit of ECSPA, and the channel domain attention module SENet is connected to the block output of other stage units, which is determined by the location of different stage units. The FirstStage units only have two convolutional layers in the blocks, and the channel domain is relatively small. Therefore, the computing cost is low. Moreover, taking the shallow features of the image as the input allows the suitable utilization of the CBAM mixed domain attention module, which has a relatively high computing cost. First, with a channel attention module, the input feature maps are processed through global max and average pooling on the bases of width and height, respectively, and then pass through MLP. The MLP output features are fused and then fed to a Sigmoid activation function to generate the final channel attention O C . The formula of the channel attention mechanism is as follows: O C = s M L P ( A v g P o o l f + M L P M a x P o o l f , (16)where s represents the Sigmoid activation function. Channel attention for every single image focuses on the important content information on the image. The average pooling provides feedback for every pixel in the feature map, whereas the max pooling only has feedback for the maximum response in the feature map. O C is the input feature map in the design of the spatial attention module CBAM. First, the feature map is transformed by a channel-based global max and average pooling, and then the two results are concatenated on the basis of the channel. The dimension is reduced to one channel after a convolution operation. A sigmoid activation function is then used to generate the spatial attention feature, which is later multiplied by the input feature of this module to obtain the final feature. The formula of spatial attention mechanism can be expressed as follows: O S O C = s C 7 × 7 ( A v g P o o l O C ; M a x P o o l O C , (17)where s represents the Sigmoid activation function, and 7 × 7 is the size of the convolution kernel. The experiments show that a 7 × 7 convolution kernel performs better than a 5 × 5 convolution kernels. (2) Structure of the attention module of SENet In other stages of ECSPA, the SENet channel attention module is appropriate due to its low computational cost and a large number of repeated blocks. The structure of this module is also the same as that of CBAM. A one-dimensional vector with the same number of channels is obtained as the evaluation score of each channel by processing the convolutional feature map, and then the score is applied to the corresponding channel to obtain the result. The ablation experiment comparison of attention module in Section 4.1 reveals that the actual effects of using CBAM or SENet attention module alone or using SENet before CBAM in ECSPA are not as good as the attention mode of CBAM before SENet. 3.4 Activation function of ECSPA The activation function aims to improve the learning capability of the network and the transfer efficiency of the gradient. The commonly used activation functions [26] of CNN are continuously developing. In recent years, the common activation functions include ReLU, LeakyReLU, and softplus, as well as Swish and Mish. A smooth activation function allows effective information to penetrate the neural network to obtain improved accuracy and generalization. The calculation formula of the Mish activation function is as follows: f ( x ) = x ∗ t a n h s o f t p l u s x . (18) The calculation formula of the LeakyReLU activation function is L r e L U ( x ) = x , i f x > 0 n e g a t i v e _ s l o p e × x , otherwise . (19) The comparison of the two aforementioned formulas shows that the computational complexity of the Mish activation function is considerably higher than that of LeakyReLU, and Mish is also an indispensable activation function for CSP. Therefore, the Mish activation function is utilized in all structures, except for the blocks in ECSPA, and the LeakyReLU activation function with relatively low computing cost is adopted for a large number of blocks. The reasonable use of the two activation functions not only improves the accuracy of classification but also reduces the calculation. 4 EXPERIMENTS We have conducted an image classification error rate experiment and an attention module ablation experiment on the CIFAR-100 dataset [27] to verify the advancement of the combination of the two attention modules and demonstrated the superiority in the ability to extract and aggregate image features comprehensively through the heat map and convolutional layer feature map visualization experiments. Experiments are carried out on the UCMerced_LandUse dataset to verify the ability of ECSPA to classify images with small objects and complex backgrounds. Experiments are also carried out on ImageNet [28] and MSCOCO datasets to verify the comprehensive performance of ECSPA in Top1, Top5, BFLOPs, AP, and training loss rate. Subsequently, the existing methods are compared on the bases of the CSPNet architecture in the two datasets to verify the outstanding performance of the two new CSP gradient combinations proposed in this paper in reducing the amount of calculation. 4.1 Ablation experiment on CIFAR-100 dataset ECSPA, RESNet50, RESNet101, VGG16 [6], DarkNet 53, and mobilenetv2 [10] are used to conduct image classification error rate experiment on the CIFAR-100 dataset to verify the classification capability of the network. Figure 6 shows that the Top1 err and Top5 err of ECSPA on the CIFAR-100 dataset are the lowest in the six classification networks, which are 2.83 and 2.15 lower than the original model DarkNet53 in Top1 err and in Top5 err, respectively. This finding proves that the application of CSP architecture and attention module in ECSPA improves the classification accuracy of small images. FIGURE 6Open in figure viewerPowerPoint Comparison of Top1 err and Top5 err of six classification networks on the CIFAR-100 dataset We also conduct the ablation experiment of attention module on the dataset to verify the superiority of the attention module combination of our network. As is shown in Table 2, DarkNet53 represents the original network without CSP and attention module; moreover, the Top1 err value is 22.33 on CIFAR-100. ECSPA# represents the two new CSP gradient combinations without attention module. Then, we conduct the ablation experiment by adding different attention module combinations in the FirstStage and four stage units. Apparently, the Top1 err of DarkNet53 is the highest among all the other methods, and that of ESCPA# without attention module is reduced by 0.23. The Top1 err of ECSPA#+CBAM+CBAM is 20.7, which results in substantial progress and is reduced by 1.63%, whereas that of the network with only one attention module in the FirstStage and four Stage units reduces slightly. Then, when it changes to ECSPA#+SENet+SENet by fine-tuning, the Top1 err value is 19.8 and is reduced by 0.9 again, which proves that the double SENet models are better than the double CBAM models. While changing to ECSPA#+SENet+CBAM, the Top1 err increases instead. Finally, when it changes to ECSPA#+CBAM+SENet as adopted in this paper, the Top1 err reduces to the lowest 19.5, which verifies that this attention module is the best combination as adopted in Section 3.3. TABLE 2. Ablation experiments with different attention modules embedded model FirstStage unit 4 Stage units Top1 err DarkNet53 22.33 ECSPA# 22.1 ECSPA# CBAM 21.5 ECSPA# SENet 21.1 ECSPA# CBAM 21.3 ECSPA# SENet 22.1 ECSPA# CBAM CBAM 20.7 ECSPA# SENet SENet 19.8 ECSPA# SENet CBAM 20.3 ECSPA# CBAM SENet 19.5 4.2 Heat map experiment Visualization of neural network heatmap [29, 30] can help understand the area of the image neural network that needs attention. The highlighted area (dark red) in the heat map represents the area related to the predicted category. The more discriminative areas in the heat map restricting the predicted category, the more accurate the model's recognition of the image. Sensing points are set up in the previous layer of AvgPool of ECSPA, RESNet50, and DarkNet53, and the position viewed by the neural network is restored by compiling. As shown in Figure 7, we circled the dark red central area of the heat map with a blue coil. The dark red area of the car heat map of the RESNet50 network is relatively large, with only one center in the lower position between the two wheels and half of the ineffective dark red area on the road than the discrimination area (body) of the predicted category. Compared with the discriminant area of the predicted category, the dark red area of the heat map of the passenger car has only one center above the window, and part of the dark red area falls outs