# A Lightweight Attention-Based Network towards Distracted Driving Behavior Recognition
## 3 Methods
### Proposed Inverted Residual Attention Module (IRAM)  
Currently, attention mechanism is a hot research topic. The presence of some effective and lightweight attention modules can improve the network performance with negligible computation cost. The channel domain in the soft attention mechanism is about \"what to look\" for. The spatial domain is about \"where to look\". As for the real-time image classification of distracted driving, the camera is always fixed. This means that for most images, the spatial information is almost unchanged. An effective attention module in this scenario should focus on the channel domain features. The schematic of the proposed IRAM is shown in Figure 3.  
Figure 3: Schematic of Inverted Residual Attention Module (IRAM). Figure 2: Schematic of depthwise separable convolution. Consider a feature map \(F_{s}\in R^{H\times W\times C}\) is input into the IRAM. The channel attention module will infer a 1D channel-wise attention map \(M_{c}\in R^{1\times 1\times C}\). The spatial attention module will infer a 2D spatial-wise attention map \(M_{s}\in R^{H\times W\times 1}\). The whole attention mechanism process can be summarized as follows:  
\[F_{s}^{\prime} =M_{c}(F_{s})\otimes F_{s}, \tag{5}\] \[F_{s}^{\prime\prime} =M_{s}\big{(}F_{s}^{\prime}\big{)}\otimes F_{s}^{\prime}, \tag{6}\]  
where \(\otimes\) denotes the element-wise dot product. The structure of channel-wise attention is inspired by the inverted residual module. The process of a normal residual module can be concluded as \"compression-convolution-expansion\". Comparatively, the process of an inverted residual module [42] can be concluded as \"expansion-convolution-compression\". Pointwise convolution is first used to expand feature channels and is finally used to compress them to the initial amount. The purpose is to extract more higher-level features. Hence, the output weighted channel-wise feature map can be more specific and precise. For the channel attention module, the input feature map will first experience a global average pooling. The pooling value of each channel can be regarded with a global region of interest. Its calculation process is shown as:  
\[P_{c}=\mathrm{GAP}(U_{c})=\frac{1}{H\times W}\sum_{i=1}^{H}\sum_{j=1}^{W}U_{c} (i,j), \tag{7}\]  
where \(P_{c}\) is the pooling value of the \(c\)th channel and \(U_{c}(i,j)\) is the value at position \((i,j)\) of the \(c\)th channel in the input feature map. Pointwise convolution is performed after GAP to increase the channels. It can be represented as:  
\[P_{c}^{\prime}=\sum_{i=1}^{C}W_{c}\times P_{i}, \tag{8}\]  
where \(P_{c}^{\prime}\) is the pooling value of the \(c\)th channel, \(c\in[0,\ C\cdot r]\), \(r\) is the expand ratio, \(W_{c}\) is the weight of the \(c\)th filter, and \(P_{i}\) is the pooling value of the \(i\)th channel. Depthwise convolution is performed after the first pointwise convolution to extract the features of each channel. Since depthwise convolution cannot change the feature channels, another pointwise convolution is performed to reduce the channels. The spatial attention module is connected after the channel attention module. It utilizes the channel-wise weighted feature map as its input. It conducts the convolution with a standard convolution layer. We select a 3 \(\times\) 3 convolution kernel here to reduce the computation cost. It is tested that a 7 \(\times\) 7 convolution kernel produces a slightly higher accuracy. However, considering the increased model complexity, we still use a 3 \(\times\) 3 kernel. The process can be represented as:  
\[M_{s}\big{(}F_{s}^{\prime}\big{)}=\delta\Big{(}f^{3\times 3}\big{(}[avgpool \big{(}F_{s}^{\prime}\big{)};maxpool\big{(}F_{s}^{\prime}\big{)}]\big{)} \Big{)}, \tag{9}\]  
where \(\delta\) denotes a sigmoid function. The attention module can be configured in three different configurations regarding the series or parallel connection. We will compare their performances in the experiment part.