# Efficient Attention Mechanism for Dynamic Convolution in Lightweight Neural Network
## 3 Proposed Efficient Attention Mechanism for Dynamic Convolution Network (SAM-DyNet)
### Efficient Attention Mechanism for Dynamic Convolution Network (EAM-DyNet)  
While calculating the attention weights, the networks need to compress the information by the pooling layer and pass through two fully connected layers. A ReLU activation function is required in the middle of the fully connected layer. The size of the output from the fully connected layer should be consistent with the number of attention weights, that  
Figure 4: (a) Model diagram of the dynamic neural networks (DyNet) in which the size of the output from the pooling layer is \(C\times 1\times 1\), (b) model diagram of EAM-DyNet. Our method first selects representative spatial information, then uses adaptive pooling to compress the spatial information, \(k_{i}\) is kernel\({}_{i}\) in Equation (5). is, k, and finally, obtain the attention weight after softmax. In this process, we can select a part of the size of the information as the input of the attention weight to better reduce its redundancy because the input image will have a lot of redundant information. Therefore, we conducted the following processing for the spatial information of channel size \(\widehat{C}\):  
\[\widehat{C}=\beta C, \tag{8}\]  
Among them, \(\widehat{C}\) is the channel size of the selected channel information, where \(\beta\) is a value from 0 to 1, and the size of the picture information should be \(\beta\times C\times H\times W\). As it also verified different sizes of \(\beta\) and selected different spatial information through different experiments, the accuracy rate will also change. For the channel selection method, the main methods are channel adjacent reduction, channel equal interval reduction, random channel reduction (RCR) and channel group reduction (CGR). The compression method is shown in Figure 5. If we assume the input channel \(D=\{D_{0},D_{1},\cdots,D_{i},\cdots,D_{C-1}\}\). Channel Adjacent Reduction: This method selects continuous spatial information from the spatial information of channel size \(C\). When selecting the channel of size \(\widehat{C}\), we can select the first, middle, or last \(\beta C\) channels, and select that these channels are adjacent and continuous. The formula can be expressed as:  
\[\widehat{D}=\bigg{\{}D_{j},D_{j+1},D_{j+2},\cdots,D_{j+\widehat{C}}\bigg{\}}, \tag{9}\]  
\[0\leq j\leq C-\widehat{C}-1, \tag{10}\]  
Channel Equal Interval Reduction: To avoid the possibility of redundant information generated between adjacent channels, it selects spatial information at equal intervals. When selecting \(\widehat{C}\) channels from the spatial information of channel size \(C\), we will take one time  
Figure 5: In the figure, \(C_{i}\) channel is the input channel, \(C_{i}^{\prime}\) is the selected channel, and the main selection methods are as follows: (**a**) Channel adjacent reduction: \(\widehat{C}\) channels are selected from adjacent channels; (**b**) Channel equal interval reduction: \(1/\beta\) channels are selected at every interval, and \(\beta C\) channels are selected, namely \(\widehat{C}\) channels; (**c**) Random channel reduction (RCR): take out k channels randomly from C input channels; (**d**) Channel group reduction (CGR): divide C input channels into k blocks and take out one channel from each block, when k channels are taken out at random, it is recorded as CGR-Random; when taken out, the k largest channels from each block are denoted as CGR-MAX. every \(1/\beta\) channels, then \(\beta\times C\) channels are taken out, that is, \(\widehat{C}\) channels. Its expression is shown in Equation (11). \[\widehat{D}=\left\{D_{0},D_{\frac{1}{p}},D_{\frac{2}{p}},\cdots,D_{\frac{\beta \times C-1}{\beta}}\right\}, \tag{11}\]  
Random Channel Reduction (RCR): To reduce redundancy to a greater extent, it randomly takes out \(\widehat{C}\) from the spatial information of channel size \(C\) in this method. When the value of \(\widehat{C}\) is equal to the number of attention cores \(k\), the effect is best by a series of experiments. Its expression is shown in Equation (12).