# Lightweight Smoke Recognition Based on Deep Convolution and Self-Attention

Authors: Yang Zhao; Yigang Wang; Hoi-Kyung Jung; Yongqiang Jin; Dan Hua; Sen Xu
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Deep convolutional networks have better smoke recognition performance. However, a lightweight network model and high recognition accuracy cannot be balanced when deployed on hardware with limited computing resources such as edge computing. Based on this background, we propose a novel smoke recognition network that combines convolutional networks (CNN) and self-attention. The core ideas of this framework are as follows: (1) Combine the depthwise convolution and asymmetric convolution of large convolution kernels to construct a lightweight CNN model, and realize multiscale extraction of feature information with slight model complexity. (2) Combined with the self-attention in transformer, a skip-connection branch is designed, which improves the feature extraction capability of the backbone network through parallel processing and fusion of feature map information. (3) Fusion multicomponent discrete cosine transform (DCT) is used to compress channel information and expand the ability of global average pooling (GAP) to aggregate feature maps. The proposed DCT-GAP improves the accuracy of the network without adding additional computational costs. Experimental results show that the proposed CSANet achieves an average accuracy of over 98.3% with 238 M FLOPs and 5.8 M parameters on the homemade smoke dataset, outperforming state-of-the-art competitors.


---

## Body

## 1. Introduction

Fire has always been a social problem related to people's lives and property safety [1]. In 2021, 748,000 fires occurred in China, killing 1,987 people. Therefore, early warning of a fire is highly critical. As a clue to determine fire occurrence, smoke is produced early and widely distributed and has stable characteristics, which make it a more reliable indicator than flame [2, 3]. Traditional smoke detectors are primarily based on particle collection. This working principle will inevitably lead to a relatively slow detection speed and unstable accuracy, especially in tall spaces with good gas mobility [4]. With the popularization of video surveillance systems and the development of computer vision technology, vision-based smoke recognition has become an important research topic for the early detection of fires [5].Traditional identification methods first need to design features, which are further divided into physical features such as pixel values [6] and signal features such as slope entropy [7]. The designed features are extracted by means of image processing or variational modal decomposition (VMD) [8], and then these features are input into classifiers such as k-nearest neighbor [9] for identification. Yuan et al. [10] proposed a method to concatenate all suboriented histograms of the local binary pattern (LBP) for smoke classification, better capturing the spatial distribution of texture features. Alamgir et al. [11] propose a new feature extraction method that improves the utilization of local and global texture attributes by local binary co-occurrence patterns for the RGB colour space (RGB_LBCoP). Dubey et al. [12] proposed two multichannel local binary patterns based on adders and decoders, which improved the extraction of multichannel local information. The above three methods effectively improved the accuracy of smoke recognition. However, manually extracted hand-crafted features are insufficient to handle the complex changes in smoke images due to lighting conditions and the surrounding environment, so these methods are not very accurate.Recently, deep learning has succeeded in general image classification [13–19], and many scholars have also achieved good recognition results in smoke recognition [20–23]. These models are trained end-to-end and automatically learn robust features from input images. The low-level network of a deep learning model extracts simple features such as edges and textures, and the high-level network can learn abstract features such as local spatial information correlation. Multiscale networks can extract different levels of information, which is very important for image recognition. Therefore, the use of a deep learning model to solve the problem of smoke recognition is a good choice. Yin et al. [24] first proposed a new deep normalized convolutional neural network (DNCNN) to classify smoke images. Khan et al. [25] performed feature extraction in smoke images by fine-tuning the VGG [15] architecture. Muhammad et al. [26] also improved the accuracy of the smoke recognition model by fine-tuning the MobileNetV2 model. With the development of the transformer architecture, more and more studies have shown the importance of self-attention for image recognition [27]. To this end, some scholars try to combine CNN and self-attention. Jiang et al. [28] used self-attention as a separate module to learn global information on high-dimensional features of the network, thus improving the ability to extract features from smoke images.The above recognition algorithms improved the accuracy of smoke identification to some extent. However, the actual application process still has difficulties and they are as follows: (1) The smoke features are complex and difficult to extract. (2) Objects such as clouds and fog will significantly interfere with the smoke recognition effect, resulting in a high false detection rate. (3) Self-attention makes the algorithm too complicated and takes up too many computing resources.We propose CSANet, a smoke recognition algorithm that combines CNN and self-attention to adders the above problems. The core ideas of the algorithm are as follows: (1) The proposed CSA module uses 7× 7 depthwise convolutions to replace part of 1 × 1 convolutions to reduce model complexity and uses asymmetric convolutions at specific network layers to reduce computational cost. (2) The CSA Block comprises two CSA modules superimposed to extract features in channel expansion and channel compression, respectively. The multiscale CSA Blocks are connected using convolutions with the same stride and convolution kernel. (3) The transformer architecture first captures the dependencies between spatial information through self-attention and then extracts the correlation between channel information, so it has certain advantages. The proposed Skip Block further extends this advantage. Subsequently, the Skip Block is connected to the backbone network in parallel. (4) Verify that global average pooling (GAP) is a specific frequency domain component after discrete cosine transformation (DCT) of the feature maps [29]. The DCT-GAP obtained by redesigning the DCT components fusion method improves the ability of the GAP to aggregate feature information without increasing the computational cost. In short, our contributions are mainly the following three points:(1)
By stacking the proposed CSA Block, the network can achieve better smoke recognition accuracy with minimal model complexity.(2)
Combined with the proposed Skip Block and DCT-GAP, CSANet performs well on smoke recognition. According to experiments, CSANet achieves an average accuracy of more than 98.3% with 258M FLOPs and 5.8 M parameters, better than the current state-of-the-art competitors.(3)
We build a dataset of 14,475 smoke images and 15,463 non-smoke images through manual screening and labelling. This dataset contains images of smoke in various weather and lighting conditions and non-smoke images of clouds and fogs resembling smoke. This dataset is of great significance for studying smoke recognition in real-world scenarios.


## 2. Materials and Methods

### 2.1. CSA Module

Many convolutions are usually included in deep convolutional networks, resulting in a high computational cost. Although recent work like GhostNet [30] has introduced depthwise convolutions to reduce the vast FLOPs caused by 1 × 1 convolutions, small-size depthwise convolutions still bring considerable computational overhead. At the same time, the receptive field of small-scale depthwise convolution is small, and the ability to extract information correlation in an extensive spatial range is poor.To this end, we introduce 7× 7 depthwise convolution and asymmetric convolution to reduce the required computational resources and design a three-pass fusion method to make up for the lack of depthwise convolution to extract channel information. As shown in Figure 1, the CSA module is divided into the expansion channel’s CSA-expansion module and the compression channel’s CSA-compression module.Figure 1
Two categories of CSA modules. (a) CSA-expansion module. (b) CSA-compression module.
(a)(b)Figure1(a) shows that in the CSA-expansion module, 1 × 1 convolution and 7 × 7 depthwise convolution are used to operate on the input data, respectively. Then 7 × 7 depthwise convolution is performed on the data after 1 × 1 convolution adjustment channels. Finally, the three-part result is spliced to produce an output with the number of channels expanded by exp times. In practice, given input data can be represented as X∈RH×W×C, where H and W are height and width, and C is the number of channels. The feature map generated after the CSA-expansion module is(1)Y′=catconvX,depthwiseX,depthwiseconvX,where Y′∈RH×W×C×exp is the output feature map, convX∈RC×exp−1/2×H×W is the result of a 1 × 1 convolution operation on the input, depthwiseX∈RC×H×W is the result of a 7 × 7 depthwise convolution operation on the input, and depthwiseconvX∈RC×exp−1/2×H×W is the result of a 7 × 7 depthwise convolution operation on the result of a 1 × 1 convolution operation.As shown in Figure1(b), in the CSA-compression module, a 1 × 1 convolution is first used to reduce the number of channels of the intermediate layer Y′∈RC×exp×H×W to reduce model parameters. Then two parallel 7 × 7 depthwise convolutions are concatenated after the 1 × 1 convolution to extract features. The feature map generated after the CSA-compression module is(2)Y=catconvY′,depthwiseY′,depthwiseconvY′,where Y∈RC×H×W is the output feature map with the number of channels C, convY′∈RC/3×H×W is the result of a 1 × 1 convolution operation on the Y′, and depthwiseconvY′∈RC/3×H×W is the result of a 7 × 7 depthwise convolution operation on the result of a 1 × 1 convolution operation.Although 7× 7 depthwise convolutions reduce some model parameters, follow-up will be combined with self-attention, requiring a high computational cost. The overall model complexity needs to be further controlled. Therefore, in the CSA module with a feature map scale of 14 × 14, we replace the 7 × 7 depthwise convolution by concatenating a 7 × 1 depthwise convolution with a kernel of 1 × 7 depthwise convolution [31], which significantly reduces the computational complexity of the model.After designing the CSA module, BatchNorm and ReLU must be inserted to reduce the occurrence of overfitting. At the same time, to solve the problem of the poor ability of deep convolution to learn the correlation between channels, the SENet module [32] is connected in series between the CSA-expansion module and the CSA-compression module. The CSA Block of the inverted residual structure composed of CSA-expansion and CSA-compression module is shown in Figure 2.Figure 2
The schematic of the CSA Block.


### 2.2. Skip Block

In order to use self-attention to expand the network's attention region for spatial information without introducing too much extra computational cost, we design Skip Block. Skip Block is calculated in parallel with the backbone network, and multiscale fusion learning combining high-level and low-level are performed at the same time. The Skip Block is mainly composed of two skip modules in series, and the complete frame is shown in Figure3.Figure 3
The schematic of the skip block.In common transformer frameworks, self-attention is used to model the dependencies of spatial information [18]. To mine inter-channel correlations, multilayer perceptron (MLP) components are usually concatenated after self-attention. Some papers have shown that this network construction order of modelling spatial dependencies first and then mining the correlations between channels is an essential reason for improving the algorithm's performance [33]. Therefore, we further layer the network based on this network construction order. By concatenating ECANet [34] behind self-attention, the relationship between channels is learned with a small amount of computation. By adding a depthwise convolution in front of the MLP, the ability of Skip Block to capture local spatial information is further improved. Finally, we combine the output dimensions into the backbone network using a convolution.Self-attention directly captures long-range dependencies by computing the relationship between any two locations rather than being limited to adjacent points. It is equivalent to constructing a vast convolution kernel to extract more helpful information while avoiding using more network layers' superpositions. However, the shortcomings of self-attention are also prominent. Large-scale spatial information or even attention to global information has a very high computational cost. On the other hand, local self-attention often limits the interaction area of each window. Therefore, we used a cross-window self-attention [35] for parallel computation of self-attention for horizontal and vertical stripes forming a cross-shaped window. Each stripe is obtained by dividing the input features into equal-width stripes. Furthermore, locally enhanced location encoding is added to handle local location information better.Figure4 is a schematic diagram of local self-attention for horizontal or vertical stripes in cross-window self-attention. The essence of cross-window self-attention is multihead attention, and we add a lightweight position information encoding mechanism using depthwise convolution. The definition of cross-window self-attention is as follows:(3)Attention=SoftmaxQKTdkV+depthwiseV,where QQuery, KKey, and VValue in the formula are three matrices obtained by linear mapping. In addition, dk is the scaling of the inner product for Softmax operation, and depthwise is the depthwise convolution operation.Figure 4
The computer flow diagram of cross-window self-attention.The computation of attention is divided into four steps. The first step is to calculate the similarity of the query with each key through the dot product to obtain the weight. The second step is usually normalizing these weights using a softmax function. The third step is to add the weight and the corresponding key value to get attention. Finally, the location information directly applied to the critical value is added to the attention to obtain the final attention.In the actual operation of self-attention, the inputX∈RH×W×C is linearly mapped into K heads according to the multihead mechanism. Then, in order to realize the local self-attention of horizontal stripes and vertical stripes, respectively, the input is divided into two parts according to the channel, which is defined as follows:(4)X=X1,X2,…,XK,Xk=XkH,XkV,s.t.k∈1,2,…,K,where Xk∈RH×W×C/K is the input of the kth head, HXk∈RH×W×C/2K is the input of local attention to the horizontal stripes of the kth head, and VXk∈RH×W×C/2K is the input of local attention to the vertical stripes of the kth head.Take the local self-attention of the horizontal stripes of thekth head as an example. In order to process the self-attention area of interest into stripes, the input HXk is divided into non-overlapping stripes of width sw uniformly. Then the local self-attention output of the kth head horizontal stripe is(5)YkiH=attentionXkiHQki,XkiHKki,XkiHVki,H−attentionkXkH=Yk1H,Yk2H,…,YkMH,s.t.i∈1,2,…,M,where Xki∈Rsw×W×C/2K and M=H/sw. Qki∈Rsw×W×C/2K, Kki∈Rsw×W×C/2K, and Vki∈Rsw×W×C/2K represent the projection matrices of queries, keys, and values for the kth head, respectively. Then the vertical stripes of self-attention can be similarly derived, and its output for kth head is denoted as V−attentionkXkV.Finally, the outputs of the horizontal and vertical stripes are stitched together. Then the outputs of the multis are wired together. The output of the final cross-window self-attention can be expressed as(6)cross−windowself−attentionX=cathead1,head2,…,headKWO,headk=H−attentionk,V−attentionk,where WO∈RC×C is the commonly used projection matrix that projects self-attention results into the target output dimension (set as C by default).


### 2.3. DCT-GAP

GAP is a commonly used means to integrate global information, prevent overfitting in convolutional neural networks, and reduce the learning parameters of fully connected layers in the network. Recently, attention mechanisms, especially channel attention [32], have achieved great success in computer vision. Many studies have focused on how to design efficient channel attention mechanisms. However, the channel attention mechanism uses scalars to represent channels, which is challenging to design due to the loss of information. Therefore, some scholars regard the channel representation problem as a compression process using frequency analysis from a different perspective [36]. Based on frequency domain analysis, Qin et al. [29] proved that the traditional GAP is a particular case of frequency domain feature decomposition. At the same time, they extended GAP using multicomponent frequency domain feature fusion. However, in multicomponent frequency domain feature selection, the influence of different scale frequency-domain feature settings on network performance in different scale feature maps is not considered. We design DCT-GAP to fuse discrete cosine transform (DCT) frequency domain feature components of different scales according to the characteristics of using GAP only for 14 × 14 and 7 × 7 feature maps in CSANet.The essential function of the two-dimensional (2D) DCT is(7)Bh,wi,j=cosπhHi+12cosπωWj+12.After simplifying some constant normalization factors that do not affect the result, the 2D DCT can be written as follows:(8)fh,w2d=∑i=0H−1∑j=0W−1xi,j2dBh,wi,j,s.t.h∈0,1,…,H−1,w∈0,1,…,W−1,in which f2d∈RH×W is the 2D DCT frequency spectrum, x2d∈RH×W is the input, H is the height of the input, and W is the width of the input.Typically, GAP is the de-facto compression method for simplicity and effectiveness. GAP uses a scalar for the inputx2d∈RC×H×W to represent the global information within one of the channels. That is, RC×H×W⟶RC.Therefore, in equation (8), when h and w take 0, the result of GAP is proportional to the lowest frequency result of 2D DCT as shown in the following equation:(9)f0,02d=∑i=0H−1∑j=0W−1xi,j2dcos0Hi+12cos0Wj+12=∑i=0H−1∑j=0W−1xi,j2d=GAPx2dHW.To better compress the channel and introduce more information, we generalize GAP to multiple frequency components of 2D DCT. First, the inputX∈RC×H×W is divided into many parts along the channel dimension. Denote X0,X1,…,XK−1 as the parts, in which Xi∈RCi×H×W, i∈0,1,…,K−1, and ∑0K−1Ci=C. For each section, a corresponding 2D DCT frequency component is assigned. Thus, we have the following:(10)Freqi=DCTui,viXi=∑h=0H−1∑w=0W−1X:,h,wiBh,wui,vi,s.t.i∈1,…,K−1,in which ui,vi are the frequency component 2D indices corresponding to Xi,andFreqi∈RCi is the Ci dimensional vector. The whole vector can be obtained by concatenation.(11)Freq=DCT−GAPX=catFreq0,Freq1,…,FreqK−1.Although we introduce other frequency components, this does not mean that these components are used to replace the lowest frequency component completely. Introducing a small number of other frequency components will fuse more information. Introducing many other frequency components will cause the loss of the unique information of the lowest frequency components. Therefore, we limit the proportion of other frequency components used when designing multifrequency component fusion. Considering that the number of channels will increase with reducing the size of the feature map in the CSANet network, we gradually increase the proportion of the lowest frequency components. In order to facilitate the division of channels, we have made the following designs. When operating on a feature map with a size of 14× 14, set the proportion of the lowest frequency component to 1/4. When operating on a feature map with a size of 7 × 7, set the proportion of the lowest frequency component to 1/2. So for Ci, the value of C0, which represents the lowest frequency component, is R, and its value is C/2 or C/4. For the remaining components Ci=C−R/K−1, i∈1,2,…,K−1. The general description of DCT-GAP is shown in Figure 5.Figure 5
The general description of DCT-GAP.When selecting those frequency domain components that can complement the lowest frequency components, it is a reasonable approach to use the accuracy of a single component as the importance score of that component. However, we consider whether the frequency also affects the choice. We will therefore investigate which decision is more favorable in Section3.4 with the help of more detailed experimental results.


### 2.4. CSANet

In order to obtain multiscale feature map information, CSANet concatenates independent downsampling layers between CSA Block groups, and the overall architecture is shown in Table1. Convolutions with equal stride and kernel are used as stem and downsampling layers to better learn correlations between distant spaces. Feature extraction is performed on the output feature map of the stem layer through the Skip Block, and the results are concatenated into the backbone network in stage 5. The parameter #exp represents the factor by which to expand the channel in the CSA-expansion module. The SENet module works better in high-dimensional, so the SE module is used in the CSA Block in stages 5 and 7. Furthermore, in the SENet module, ECANet, and stage 9, GAP is replaced by DCT-GAP.Table 1
The overall architecture of CSANet. #exp represents the multiple of channel expansion. SE indicates whether to use the SENet module. The sixth column of the table represents the Skip block input as the output feature map of stage stem. Meanwhile, the output of size142×56 is connected to the network in stage 5.
StageOperatorInput#expSEStem4× 4 conv s = 4, BN2242×3——Skip Block1CSA Block× 2562×242—2BN, 2× 2 conv s = 2562×24——3CSA Block× 2282×483—4BN, 2× 2 conv s = 2282×48——5CSA Block×6142×14441Concat (142×56)6BN, 2× 2 conv s = 2142×144——7CSA Block× 272×288618CSA-expansion module72×2886—9DCT-GAP72×1728——10BN, FC72×1728——The cross-entropy loss function is often used for binary classification problems. Therefore, we choose it to measure the similarity between two probability distributions of actual and predicted labels. The cross-entropy loss can be described as follows:(12)Hpx,qx=−∑ipxilogqxi,where pxiandqxi are input and output labels, respectively.


## 2.1. CSA Module

Many convolutions are usually included in deep convolutional networks, resulting in a high computational cost. Although recent work like GhostNet [30] has introduced depthwise convolutions to reduce the vast FLOPs caused by 1 × 1 convolutions, small-size depthwise convolutions still bring considerable computational overhead. At the same time, the receptive field of small-scale depthwise convolution is small, and the ability to extract information correlation in an extensive spatial range is poor.To this end, we introduce 7× 7 depthwise convolution and asymmetric convolution to reduce the required computational resources and design a three-pass fusion method to make up for the lack of depthwise convolution to extract channel information. As shown in Figure 1, the CSA module is divided into the expansion channel’s CSA-expansion module and the compression channel’s CSA-compression module.Figure 1
Two categories of CSA modules. (a) CSA-expansion module. (b) CSA-compression module.
(a)(b)Figure1(a) shows that in the CSA-expansion module, 1 × 1 convolution and 7 × 7 depthwise convolution are used to operate on the input data, respectively. Then 7 × 7 depthwise convolution is performed on the data after 1 × 1 convolution adjustment channels. Finally, the three-part result is spliced to produce an output with the number of channels expanded by exp times. In practice, given input data can be represented as X∈RH×W×C, where H and W are height and width, and C is the number of channels. The feature map generated after the CSA-expansion module is(1)Y′=catconvX,depthwiseX,depthwiseconvX,where Y′∈RH×W×C×exp is the output feature map, convX∈RC×exp−1/2×H×W is the result of a 1 × 1 convolution operation on the input, depthwiseX∈RC×H×W is the result of a 7 × 7 depthwise convolution operation on the input, and depthwiseconvX∈RC×exp−1/2×H×W is the result of a 7 × 7 depthwise convolution operation on the result of a 1 × 1 convolution operation.As shown in Figure1(b), in the CSA-compression module, a 1 × 1 convolution is first used to reduce the number of channels of the intermediate layer Y′∈RC×exp×H×W to reduce model parameters. Then two parallel 7 × 7 depthwise convolutions are concatenated after the 1 × 1 convolution to extract features. The feature map generated after the CSA-compression module is(2)Y=catconvY′,depthwiseY′,depthwiseconvY′,where Y∈RC×H×W is the output feature map with the number of channels C, convY′∈RC/3×H×W is the result of a 1 × 1 convolution operation on the Y′, and depthwiseconvY′∈RC/3×H×W is the result of a 7 × 7 depthwise convolution operation on the result of a 1 × 1 convolution operation.Although 7× 7 depthwise convolutions reduce some model parameters, follow-up will be combined with self-attention, requiring a high computational cost. The overall model complexity needs to be further controlled. Therefore, in the CSA module with a feature map scale of 14 × 14, we replace the 7 × 7 depthwise convolution by concatenating a 7 × 1 depthwise convolution with a kernel of 1 × 7 depthwise convolution [31], which significantly reduces the computational complexity of the model.After designing the CSA module, BatchNorm and ReLU must be inserted to reduce the occurrence of overfitting. At the same time, to solve the problem of the poor ability of deep convolution to learn the correlation between channels, the SENet module [32] is connected in series between the CSA-expansion module and the CSA-compression module. The CSA Block of the inverted residual structure composed of CSA-expansion and CSA-compression module is shown in Figure 2.Figure 2
The schematic of the CSA Block.


## 2.2. Skip Block

In order to use self-attention to expand the network's attention region for spatial information without introducing too much extra computational cost, we design Skip Block. Skip Block is calculated in parallel with the backbone network, and multiscale fusion learning combining high-level and low-level are performed at the same time. The Skip Block is mainly composed of two skip modules in series, and the complete frame is shown in Figure3.Figure 3
The schematic of the skip block.In common transformer frameworks, self-attention is used to model the dependencies of spatial information [18]. To mine inter-channel correlations, multilayer perceptron (MLP) components are usually concatenated after self-attention. Some papers have shown that this network construction order of modelling spatial dependencies first and then mining the correlations between channels is an essential reason for improving the algorithm's performance [33]. Therefore, we further layer the network based on this network construction order. By concatenating ECANet [34] behind self-attention, the relationship between channels is learned with a small amount of computation. By adding a depthwise convolution in front of the MLP, the ability of Skip Block to capture local spatial information is further improved. Finally, we combine the output dimensions into the backbone network using a convolution.Self-attention directly captures long-range dependencies by computing the relationship between any two locations rather than being limited to adjacent points. It is equivalent to constructing a vast convolution kernel to extract more helpful information while avoiding using more network layers' superpositions. However, the shortcomings of self-attention are also prominent. Large-scale spatial information or even attention to global information has a very high computational cost. On the other hand, local self-attention often limits the interaction area of each window. Therefore, we used a cross-window self-attention [35] for parallel computation of self-attention for horizontal and vertical stripes forming a cross-shaped window. Each stripe is obtained by dividing the input features into equal-width stripes. Furthermore, locally enhanced location encoding is added to handle local location information better.Figure4 is a schematic diagram of local self-attention for horizontal or vertical stripes in cross-window self-attention. The essence of cross-window self-attention is multihead attention, and we add a lightweight position information encoding mechanism using depthwise convolution. The definition of cross-window self-attention is as follows:(3)Attention=SoftmaxQKTdkV+depthwiseV,where QQuery, KKey, and VValue in the formula are three matrices obtained by linear mapping. In addition, dk is the scaling of the inner product for Softmax operation, and depthwise is the depthwise convolution operation.Figure 4
The computer flow diagram of cross-window self-attention.The computation of attention is divided into four steps. The first step is to calculate the similarity of the query with each key through the dot product to obtain the weight. The second step is usually normalizing these weights using a softmax function. The third step is to add the weight and the corresponding key value to get attention. Finally, the location information directly applied to the critical value is added to the attention to obtain the final attention.In the actual operation of self-attention, the inputX∈RH×W×C is linearly mapped into K heads according to the multihead mechanism. Then, in order to realize the local self-attention of horizontal stripes and vertical stripes, respectively, the input is divided into two parts according to the channel, which is defined as follows:(4)X=X1,X2,…,XK,Xk=XkH,XkV,s.t.k∈1,2,…,K,where Xk∈RH×W×C/K is the input of the kth head, HXk∈RH×W×C/2K is the input of local attention to the horizontal stripes of the kth head, and VXk∈RH×W×C/2K is the input of local attention to the vertical stripes of the kth head.Take the local self-attention of the horizontal stripes of thekth head as an example. In order to process the self-attention area of interest into stripes, the input HXk is divided into non-overlapping stripes of width sw uniformly. Then the local self-attention output of the kth head horizontal stripe is(5)YkiH=attentionXkiHQki,XkiHKki,XkiHVki,H−attentionkXkH=Yk1H,Yk2H,…,YkMH,s.t.i∈1,2,…,M,where Xki∈Rsw×W×C/2K and M=H/sw. Qki∈Rsw×W×C/2K, Kki∈Rsw×W×C/2K, and Vki∈Rsw×W×C/2K represent the projection matrices of queries, keys, and values for the kth head, respectively. Then the vertical stripes of self-attention can be similarly derived, and its output for kth head is denoted as V−attentionkXkV.Finally, the outputs of the horizontal and vertical stripes are stitched together. Then the outputs of the multis are wired together. The output of the final cross-window self-attention can be expressed as(6)cross−windowself−attentionX=cathead1,head2,…,headKWO,headk=H−attentionk,V−attentionk,where WO∈RC×C is the commonly used projection matrix that projects self-attention results into the target output dimension (set as C by default).


## 2.3. DCT-GAP

GAP is a commonly used means to integrate global information, prevent overfitting in convolutional neural networks, and reduce the learning parameters of fully connected layers in the network. Recently, attention mechanisms, especially channel attention [32], have achieved great success in computer vision. Many studies have focused on how to design efficient channel attention mechanisms. However, the channel attention mechanism uses scalars to represent channels, which is challenging to design due to the loss of information. Therefore, some scholars regard the channel representation problem as a compression process using frequency analysis from a different perspective [36]. Based on frequency domain analysis, Qin et al. [29] proved that the traditional GAP is a particular case of frequency domain feature decomposition. At the same time, they extended GAP using multicomponent frequency domain feature fusion. However, in multicomponent frequency domain feature selection, the influence of different scale frequency-domain feature settings on network performance in different scale feature maps is not considered. We design DCT-GAP to fuse discrete cosine transform (DCT) frequency domain feature components of different scales according to the characteristics of using GAP only for 14 × 14 and 7 × 7 feature maps in CSANet.The essential function of the two-dimensional (2D) DCT is(7)Bh,wi,j=cosπhHi+12cosπωWj+12.After simplifying some constant normalization factors that do not affect the result, the 2D DCT can be written as follows:(8)fh,w2d=∑i=0H−1∑j=0W−1xi,j2dBh,wi,j,s.t.h∈0,1,…,H−1,w∈0,1,…,W−1,in which f2d∈RH×W is the 2D DCT frequency spectrum, x2d∈RH×W is the input, H is the height of the input, and W is the width of the input.Typically, GAP is the de-facto compression method for simplicity and effectiveness. GAP uses a scalar for the inputx2d∈RC×H×W to represent the global information within one of the channels. That is, RC×H×W⟶RC.Therefore, in equation (8), when h and w take 0, the result of GAP is proportional to the lowest frequency result of 2D DCT as shown in the following equation:(9)f0,02d=∑i=0H−1∑j=0W−1xi,j2dcos0Hi+12cos0Wj+12=∑i=0H−1∑j=0W−1xi,j2d=GAPx2dHW.To better compress the channel and introduce more information, we generalize GAP to multiple frequency components of 2D DCT. First, the inputX∈RC×H×W is divided into many parts along the channel dimension. Denote X0,X1,…,XK−1 as the parts, in which Xi∈RCi×H×W, i∈0,1,…,K−1, and ∑0K−1Ci=C. For each section, a corresponding 2D DCT frequency component is assigned. Thus, we have the following:(10)Freqi=DCTui,viXi=∑h=0H−1∑w=0W−1X:,h,wiBh,wui,vi,s.t.i∈1,…,K−1,in which ui,vi are the frequency component 2D indices corresponding to Xi,andFreqi∈RCi is the Ci dimensional vector. The whole vector can be obtained by concatenation.(11)Freq=DCT−GAPX=catFreq0,Freq1,…,FreqK−1.Although we introduce other frequency components, this does not mean that these components are used to replace the lowest frequency component completely. Introducing a small number of other frequency components will fuse more information. Introducing many other frequency components will cause the loss of the unique information of the lowest frequency components. Therefore, we limit the proportion of other frequency components used when designing multifrequency component fusion. Considering that the number of channels will increase with reducing the size of the feature map in the CSANet network, we gradually increase the proportion of the lowest frequency components. In order to facilitate the division of channels, we have made the following designs. When operating on a feature map with a size of 14× 14, set the proportion of the lowest frequency component to 1/4. When operating on a feature map with a size of 7 × 7, set the proportion of the lowest frequency component to 1/2. So for Ci, the value of C0, which represents the lowest frequency component, is R, and its value is C/2 or C/4. For the remaining components Ci=C−R/K−1, i∈1,2,…,K−1. The general description of DCT-GAP is shown in Figure 5.Figure 5
The general description of DCT-GAP.When selecting those frequency domain components that can complement the lowest frequency components, it is a reasonable approach to use the accuracy of a single component as the importance score of that component. However, we consider whether the frequency also affects the choice. We will therefore investigate which decision is more favorable in Section3.4 with the help of more detailed experimental results.


## 2.4. CSANet

In order to obtain multiscale feature map information, CSANet concatenates independent downsampling layers between CSA Block groups, and the overall architecture is shown in Table1. Convolutions with equal stride and kernel are used as stem and downsampling layers to better learn correlations between distant spaces. Feature extraction is performed on the output feature map of the stem layer through the Skip Block, and the results are concatenated into the backbone network in stage 5. The parameter #exp represents the factor by which to expand the channel in the CSA-expansion module. The SENet module works better in high-dimensional, so the SE module is used in the CSA Block in stages 5 and 7. Furthermore, in the SENet module, ECANet, and stage 9, GAP is replaced by DCT-GAP.Table 1
The overall architecture of CSANet. #exp represents the multiple of channel expansion. SE indicates whether to use the SENet module. The sixth column of the table represents the Skip block input as the output feature map of stage stem. Meanwhile, the output of size142×56 is connected to the network in stage 5.
StageOperatorInput#expSEStem4× 4 conv s = 4, BN2242×3——Skip Block1CSA Block× 2562×242—2BN, 2× 2 conv s = 2562×24——3CSA Block× 2282×483—4BN, 2× 2 conv s = 2282×48——5CSA Block×6142×14441Concat (142×56)6BN, 2× 2 conv s = 2142×144——7CSA Block× 272×288618CSA-expansion module72×2886—9DCT-GAP72×1728——10BN, FC72×1728——The cross-entropy loss function is often used for binary classification problems. Therefore, we choose it to measure the similarity between two probability distributions of actual and predicted labels. The cross-entropy loss can be described as follows:(12)Hpx,qx=−∑ipxilogqxi,where pxiandqxi are input and output labels, respectively.


## 3. Results and Discussion

### 3.1. Experimental Platform

All experiments are carried out in the Ubuntu 16.04 operating system. The graphics processing unit (GPU) is Tesla P100, which relies on Python 3.7, NumPy 1.20.2, Torchvision 0.4.0, PyTorch 1.2.0, and OpenCV 4.5.1. The hyperparameter settings are summarized in Table2 below.Table 2
Hyperparameter settings during network training.
Hyper-parametersValueEpochs300OptimizerAdamBatch size16Learning rate0.001Attenuation coefficients0.9 and 0.999Input size224× 224


### 3.2. Smoke Dataset

There are 14,475 smoke images and 15,463 non-smoke images obtained through government departments and other research groups. The smoke images in the dataset are divided into three parts: white smoke samples, black smoke samples, and hard samples. To enhance the usability of the dataset, the non-smoke images contain rich backgrounds such as clouds and fog. Some samples of the images in this dataset are shown in Figure6.Figure 6
The sample dataset. (a) White smoke samples. (b) Black smoke samples. (c) Hard samples. (d) Non-smoke samples.
(a)(b)(c)(d)In order to use the dataset reasonably, all images were randomly scrambled and divided into the training set, the validation set, and the test set according to 7 : 2 : 1. The complex background and rich scenes make this dataset more challenging, so this dataset has specific research significance and theoretical value.


### 3.3. Evaluation Criterion

The experiment uses the number of learnable parameters and the total floating-point operations (FLOPs) to assess the algorithm’s complexity. Positive sample accuracy (PA) and negative sample accuracy (NA) were used to quantify the algorithm’s recognition performance for smoke. PA and NA are defined as follows:(13)PA=PQsmoke×100%,NA=NQnon−smoke×100%.In the formula,P represents the number of correctly identified smoke samples, Qsmoke represents the number of smoke samples in the dataset, N represents the number of correctly identified non-smoke samples, and Qnon−smoke represents the number of non-smoke samples in the dataset.


### 3.4. DCT-GAP Design

For DCT-GAP, the choice of frequency domain components is dominant. One way is to select low-frequency components. Some studies have shown that CNN is more suitable for low-frequency components [37], and we name it DCT-Low. Another method is to select it according to the importance of frequency domain components, which we call DCT-Top. For DCT-Top, it is reasonable that the accuracy of each frequency domain component in the network is equal to its importance. Figure 7(a) shows the recognition accuracy for each frequency domain component applied to the smoke dataset in CSANet. Then we choose the K highest precision components to construct DCT-Top. To facilitate channel division and performance comparison, K can be set to 5, 7, 10, 13, or 19.Figure 7
DCT-GAP design. (a) PA (%) on smoke dataset using different frequency components in CSANet. (b) PA (%) with different numbers of components.
(a)(b)Figure7(b) is the recognition accuracy of CSANet using different numbers of components in the smoke dataset. We can see three phenomena. (1) All models that fuse frequency domain components have significant recognition accuracy advantages over models that only use GAP. This comparison demonstrates the correctness of our continued exploration of multifrequency component fusion to improve GAP. (2) The recognition accuracy of some models exceeds that of models using the FcaNet fusion method. This result verifies the effectiveness of our method of setting different fusion scales in feature maps of different scales. (3) The DCT-Top performance of frequency components at 13 is the best of all experiments, so we use this setting in subsequent experiments.


### 3.5. Comparative Experiments

To verify the superiority of the proposed CSANet in smoke recognition, we selected ten algorithms with advanced performance for comparison. First, five classes of algorithms using the CNN architecture are selected, including MobileNetV2 [38], ShuffleNetV2 [39], MobileNetV3 [40], GhostNet [30], and SkipNet [41]. For reasonableness of the comparison, we choose the version with the FLOPs index of 200 M to 300 M from the above algorithms. Then, five lightweight algorithms using the transformer architecture are selected, which are T2T-ViT [42], DeiT [43], ConViT [44], ConT [45], and PVT [46], for comparative experiments. In order to ensure the fairness of the experiments, our experiments adopt a unified image preprocessing method and input size. Then we use a unified experimental platform to train each algorithm.Table3 shows the experimental results of the proposed CSANet algorithm and its competitors. The network structures of MobileNetV2 and ShuffleNetV2 are relatively monotonous and straightforward, and the algorithm parameters are small. However, the recognition accuracy is poor. MobileNetV3 improves the recognition performance while reducing the FLOPs index by adding SENet modules and modifying the network structure. GhostNet reduces the substantial computational cost of 1 × 1 convolution through the Ghost module. Therefore, GhostNet has better recognition performance than MobileNetV3 among models with similar FLOPs. SkipNet enhances MobileNetV3 by adding skip-connection branches and outperforms GhostNet in recognition accuracy. Due to the high computational cost of self-attention, the recognition algorithm using the transformer architecture is computationally expensive in floating-point. T2T-ViT improves the algorithm's ability to mine local information by gradually structuring images into tokens. DeiT proposes a token-based distillation strategy to improve the algorithm's ability to learn global information. ConViT improves DeiT by introducing self-attention modulated between local and global features. ConT further improves recognition performance by combining self-attention and convolution algorithms. PVT combines progressively shrinking pyramids and spatially shrinking attention layers to obtain high-resolution and multiscale feature maps. Although the recognition accuracy of the PVT algorithm is relatively high, the model complexity is also relatively high. The advantage of the CSANet proposed in this paper is that the network of the CNN framework is used as the backbone, combined with the powerful self-attention to extract different location information, improving accuracy. At the same time, self-attention is added to the head of the network as part of the skip-connection branch, which significantly controls the complexity of the entire model. Comprehensively analyzing the PA and NA indicators of CSANet and other competing methods, the average accuracy of CSANet has reached more than 98.3%, with good smoke recognition performance and a low false detection rate.Table 3
Comparison of CSANet and its competitors.
NetworksInput sizeParameters (M)FLOPs (M)PA (%)NA (%)MobileNetV2 1.0×22423.530091.8293.66ShuffleNetV2 1.5×22423.529991.9193.84MobileNetV3 large 1.0×22425.421996.7897.53GhostNet 1.3×22427.322697.3197.82SkipNet-M22425.524697.7998.35T2T-ViT-722424.31,30291.5492.92DeiT-Ti22425.71,07592.1795.31ConViT-Tiny22426.01,15592.6494.77ConT-Ti22425.81,56895.5796.46PVT-Tiny224213.21,95596.2897.19CSANet22425.825898.1398.58The best value in each evaluation metrics is marked in bold.


### 3.6. Ablation Experiments

To comprehensively evaluate the proposed CSANet and better understand the impact of its various modules on model performance, we also set up ablation experiments. Ablation experiments selectively strip some model modules to judge the impact of missing parts on the model. Modules that affect model complexity and accuracy can be fully evaluated through ablation experiments. As shown in Table4, we divide two modules, Skip Block and DCT-GAP, from the network.Table 4
The result of ablation experiment.
PartsCompositionSkip Block✓✓DCT-GAP✓✓Parameters (M)4.55.84.55.8FLOPs (M)138258138258PA (%)95.4997.5296.3798.13NA (%)96.4897.9997.3398.58As shown in Table4, the experimental results show that without using Skip Block and DCT-GAP, the model achieves 95.49% and 96.48% for PA and NA with a complexity of 4.5 M parameters and 132 M FLOPs. After adding Skip Block, the parameters increased by 1.3 M, the FLOPs increased by 120 M, and the PA and NA indicators increased by 2.03% and 1.51%, respectively. By comparing the data in the second and fourth columns, it can be found that using DCT-GAP improves model accuracy without increasing the number of parameters and FLOPs. The improvement method designed in this paper is effective.From this experimental result, it can be concluded that the use of Skip Block can significantly improve the ability of the network to identify smoke. In contrast, DCT-GAP can further improve the model's accuracy without adding additional computational costs. In summary, the improvement method designed in this paper is effective.


## 3.1. Experimental Platform

All experiments are carried out in the Ubuntu 16.04 operating system. The graphics processing unit (GPU) is Tesla P100, which relies on Python 3.7, NumPy 1.20.2, Torchvision 0.4.0, PyTorch 1.2.0, and OpenCV 4.5.1. The hyperparameter settings are summarized in Table2 below.Table 2
Hyperparameter settings during network training.
Hyper-parametersValueEpochs300OptimizerAdamBatch size16Learning rate0.001Attenuation coefficients0.9 and 0.999Input size224× 224


## 3.2. Smoke Dataset

There are 14,475 smoke images and 15,463 non-smoke images obtained through government departments and other research groups. The smoke images in the dataset are divided into three parts: white smoke samples, black smoke samples, and hard samples. To enhance the usability of the dataset, the non-smoke images contain rich backgrounds such as clouds and fog. Some samples of the images in this dataset are shown in Figure6.Figure 6
The sample dataset. (a) White smoke samples. (b) Black smoke samples. (c) Hard samples. (d) Non-smoke samples.
(a)(b)(c)(d)In order to use the dataset reasonably, all images were randomly scrambled and divided into the training set, the validation set, and the test set according to 7 : 2 : 1. The complex background and rich scenes make this dataset more challenging, so this dataset has specific research significance and theoretical value.


## 3.3. Evaluation Criterion

The experiment uses the number of learnable parameters and the total floating-point operations (FLOPs) to assess the algorithm’s complexity. Positive sample accuracy (PA) and negative sample accuracy (NA) were used to quantify the algorithm’s recognition performance for smoke. PA and NA are defined as follows:(13)PA=PQsmoke×100%,NA=NQnon−smoke×100%.In the formula,P represents the number of correctly identified smoke samples, Qsmoke represents the number of smoke samples in the dataset, N represents the number of correctly identified non-smoke samples, and Qnon−smoke represents the number of non-smoke samples in the dataset.


## 3.4. DCT-GAP Design

For DCT-GAP, the choice of frequency domain components is dominant. One way is to select low-frequency components. Some studies have shown that CNN is more suitable for low-frequency components [37], and we name it DCT-Low. Another method is to select it according to the importance of frequency domain components, which we call DCT-Top. For DCT-Top, it is reasonable that the accuracy of each frequency domain component in the network is equal to its importance. Figure 7(a) shows the recognition accuracy for each frequency domain component applied to the smoke dataset in CSANet. Then we choose the K highest precision components to construct DCT-Top. To facilitate channel division and performance comparison, K can be set to 5, 7, 10, 13, or 19.Figure 7
DCT-GAP design. (a) PA (%) on smoke dataset using different frequency components in CSANet. (b) PA (%) with different numbers of components.
(a)(b)Figure7(b) is the recognition accuracy of CSANet using different numbers of components in the smoke dataset. We can see three phenomena. (1) All models that fuse frequency domain components have significant recognition accuracy advantages over models that only use GAP. This comparison demonstrates the correctness of our continued exploration of multifrequency component fusion to improve GAP. (2) The recognition accuracy of some models exceeds that of models using the FcaNet fusion method. This result verifies the effectiveness of our method of setting different fusion scales in feature maps of different scales. (3) The DCT-Top performance of frequency components at 13 is the best of all experiments, so we use this setting in subsequent experiments.


## 3.5. Comparative Experiments

To verify the superiority of the proposed CSANet in smoke recognition, we selected ten algorithms with advanced performance for comparison. First, five classes of algorithms using the CNN architecture are selected, including MobileNetV2 [38], ShuffleNetV2 [39], MobileNetV3 [40], GhostNet [30], and SkipNet [41]. For reasonableness of the comparison, we choose the version with the FLOPs index of 200 M to 300 M from the above algorithms. Then, five lightweight algorithms using the transformer architecture are selected, which are T2T-ViT [42], DeiT [43], ConViT [44], ConT [45], and PVT [46], for comparative experiments. In order to ensure the fairness of the experiments, our experiments adopt a unified image preprocessing method and input size. Then we use a unified experimental platform to train each algorithm.Table3 shows the experimental results of the proposed CSANet algorithm and its competitors. The network structures of MobileNetV2 and ShuffleNetV2 are relatively monotonous and straightforward, and the algorithm parameters are small. However, the recognition accuracy is poor. MobileNetV3 improves the recognition performance while reducing the FLOPs index by adding SENet modules and modifying the network structure. GhostNet reduces the substantial computational cost of 1 × 1 convolution through the Ghost module. Therefore, GhostNet has better recognition performance than MobileNetV3 among models with similar FLOPs. SkipNet enhances MobileNetV3 by adding skip-connection branches and outperforms GhostNet in recognition accuracy. Due to the high computational cost of self-attention, the recognition algorithm using the transformer architecture is computationally expensive in floating-point. T2T-ViT improves the algorithm's ability to mine local information by gradually structuring images into tokens. DeiT proposes a token-based distillation strategy to improve the algorithm's ability to learn global information. ConViT improves DeiT by introducing self-attention modulated between local and global features. ConT further improves recognition performance by combining self-attention and convolution algorithms. PVT combines progressively shrinking pyramids and spatially shrinking attention layers to obtain high-resolution and multiscale feature maps. Although the recognition accuracy of the PVT algorithm is relatively high, the model complexity is also relatively high. The advantage of the CSANet proposed in this paper is that the network of the CNN framework is used as the backbone, combined with the powerful self-attention to extract different location information, improving accuracy. At the same time, self-attention is added to the head of the network as part of the skip-connection branch, which significantly controls the complexity of the entire model. Comprehensively analyzing the PA and NA indicators of CSANet and other competing methods, the average accuracy of CSANet has reached more than 98.3%, with good smoke recognition performance and a low false detection rate.Table 3
Comparison of CSANet and its competitors.
NetworksInput sizeParameters (M)FLOPs (M)PA (%)NA (%)MobileNetV2 1.0×22423.530091.8293.66ShuffleNetV2 1.5×22423.529991.9193.84MobileNetV3 large 1.0×22425.421996.7897.53GhostNet 1.3×22427.322697.3197.82SkipNet-M22425.524697.7998.35T2T-ViT-722424.31,30291.5492.92DeiT-Ti22425.71,07592.1795.31ConViT-Tiny22426.01,15592.6494.77ConT-Ti22425.81,56895.5796.46PVT-Tiny224213.21,95596.2897.19CSANet22425.825898.1398.58The best value in each evaluation metrics is marked in bold.


## 3.6. Ablation Experiments

To comprehensively evaluate the proposed CSANet and better understand the impact of its various modules on model performance, we also set up ablation experiments. Ablation experiments selectively strip some model modules to judge the impact of missing parts on the model. Modules that affect model complexity and accuracy can be fully evaluated through ablation experiments. As shown in Table4, we divide two modules, Skip Block and DCT-GAP, from the network.Table 4
The result of ablation experiment.
PartsCompositionSkip Block✓✓DCT-GAP✓✓Parameters (M)4.55.84.55.8FLOPs (M)138258138258PA (%)95.4997.5296.3798.13NA (%)96.4897.9997.3398.58As shown in Table4, the experimental results show that without using Skip Block and DCT-GAP, the model achieves 95.49% and 96.48% for PA and NA with a complexity of 4.5 M parameters and 132 M FLOPs. After adding Skip Block, the parameters increased by 1.3 M, the FLOPs increased by 120 M, and the PA and NA indicators increased by 2.03% and 1.51%, respectively. By comparing the data in the second and fourth columns, it can be found that using DCT-GAP improves model accuracy without increasing the number of parameters and FLOPs. The improvement method designed in this paper is effective.From this experimental result, it can be concluded that the use of Skip Block can significantly improve the ability of the network to identify smoke. In contrast, DCT-GAP can further improve the model's accuracy without adding additional computational costs. In summary, the improvement method designed in this paper is effective.


## 4. Conclusions

This paper proposed a new design idea for a lightweight network. We design an accurate and lightweight neural network that combines CNN and self-attention to improve the performance of smoke recognition. CSA Block aimed to reduce model complexity by applying depthwise and asymmetric convolutions with large convolution kernels. Furthermore, Skip Block was proposed to enhance the learning of nonlocal information. Skip Block uses self-attention with less computation and improves the rationality of network construction by adding spatial information and channel information extraction components. DCT-GAP was proposed to further generalize global average pooling (GAP) by reasonably fusing frequency domain components. This paper constructed a smoke dataset for training and experimentation. The constructed dataset currently contains smoke, clouds, and other interference images and rich background images in various natural environments, which can be used to study smoke recognition algorithms under actual conditions. Experiments show that, compared with ten state-of-the-art methods, the proposed CSANet method achieved the best performance under reasonable model complexity, with an average recognition accuracy of over 98.3%. In future work, we will generalize this network to fire recognition work.



---

*Source: 1218713-2022-08-21.xml*