# Efficient Attention Mechanism for Dynamic Convolution in Lightweight Neural Network
## 3 Proposed Efficient Attention Mechanism for Dynamic Convolution Network (SAM-DyNet)
### Efficient Attention Mechanism for Dynamic Convolution Network (EAM-DyNet)
\[\widehat{D}=\left\{\widehat{D}_{0},\widehat{D}_{1},\cdots,\widehat{D}_{ \widehat{C}}\right\}, \tag{12}\]  
\[\widehat{D}\subseteq D, \tag{13}\]  
Channel Group Reduction (CGR): In the RCR-based method, we can optimize it by grouping these channels to avoid the uncertainty of using the RCR method. To obtain the best spatial information of the \(k\) channels, it equally divides the \(C\) channels into \(k\) blocks, selects one channel from each block, and selects \(k\) channels in total. When randomly selecting from each block, it is called group random compression (CGR-Random). When the maximum spatial information is selected from each block, it is called CGR-MAX. Its expression is shown in Equation (14). \[\widehat{D}=\left\{\widehat{D}_{0},\widehat{D}_{1},\cdots,\widehat{D}_{j}, \cdots,\widehat{D}_{k}\right\}, \tag{14}\]  
\[\widehat{C}=\beta C, \tag{8}\]0  
In the process of addressing the attention weight, DyNet use pooling to compress the images, but the ordinary pooling layer compresses an \(H\times W\) size image into a \(1\times 1\) size image, which leads to the loss of spatial information due to the reduction in channels. Therefore, to address this problem, EAM-DyNet uses adaptive pooling to compress the channel information to output the channel information of the specified size. Finally, after pooling, the size of the spatial information will be reduced by \(\beta^{2}\) times, that is, the size of the spatial information should be \(\beta^{2}\times H\times W\) at this time. In the adaptive pooling layer, the output tensor size is determined by the input size, kernel size, and stride. The calculation of the output size is as follows:  
\[\widehat{C}=\beta C, \tag{8}\]1  
Therefore, to obtain the output size we need, the adaptive pooling layer will automatically adjust its kernel\({}_{size}\) and \(stride\) to achieve an adaptive effect. However, the choice of different pooling also has a certain impact on the experimental results. Commonly used adaptive pooling layers include the adaptive average pooling layer and adaptive maximum pooling layer. The average pooling layer divides the input information into several groups, compresses these groups of data by averaging them, and finally outputs the required results, while the maximum pooling layer compresses each group of data by taking the maximum value. We can find which pooling layer is more effective by comparing the accuracy of two pooling layers experimentally. After this, we input the feature map with the size of \(\beta^{2}\times C\times H\times W\) into the fully connected layer. After passing through the first fully connected layer, we reduced the channel size to 10, then the channel size was reduced to k by the second fully connected layer, which is the number of attention cores. In addition, there is the ReLU activation function between two fully connected layers. Finally, it uses softmax to obtain the attention weights. However, when the softmax is set to 1 and multiple kernels cannot be optimized at the same time, this will lead to slow training convergence speed and affect the experimental results. Therefore, to address this low-efficiency problem, we set it to a larger temperature, and the calculation method is as shown in the formula:  
\[\widehat{C}=\beta C, \tag{8}\]2  
Among them, \(c_{i}\) is the output result of the \(i\)th attention weight from the second fully connected layer, and \(\tau\) is the temperature. By setting the size of different temperatures, we finally found that the training result would be more effective when \(\tau\) is 30, and the final accuracy rate will reach the highest.