# Adaptive Aggregated Attention Network for Pulmonary
## 3 Materials and Methods
### Methods
#### 3.2.2 Attention Mechanism  
To capture fine-grained features, we applied a residual connection, and bottom-up and top-down structure to the attentive dual path network. The attention mechanism was named RAN. Based on the up-sample and down-sample operations, feature maps with different resolutions were generated, and the mixed features maps were generated by fusing features of different resolutions. A residual connection was helpful to capture deep feature information and to transmit structural features generated by the shallow layers of the DPN to the deep layers of the DPN. It was efficient to fuse structural features and semantic features. Besides, the RAN module can avoid overfitting to some extent. Based on the advantages of the RAN module, we can add the RAN module in the shallow, middle, and deep layers of a DPN to capture different structural features and semantic features and the multiple RAN modules do not result in a reduction in network performance. To our best knowledge, it is the first time RAN was applied to lung nodule classification. The RAN mechanism is shown in Figure 5, in which the up-sample method is trilinear interpolation, the down-sample is max-pooling, and the RB is a basic residual block. Besides, we apply the SEN module in the last layer of a DPN to capture the relationship among channels. Especially, we proposed a novel attention module named MSAN, which aggregates the multi-scale spatial attention mechanism and contextual attention mechanism. Based on our careful observation and experiments, we utilized two convolution paths to extract different features. One convolution path was designed with kernel size \(1\times 1\times 1\), and the other one was designed with a dilated convolution with kernel size \(3\times 3\times 3\) and dilation  
Figure 4: Illustration of the gradient boosting machine (GBM) algorithm. Figure 5: Illustration of the residual attention block. size \(2\times 2\times 2\). This process can be formulated as Equation (3), in which \(Z\) consists of \(Z[:d]\) and \(Z[d:]\), \(d\) is a hyperparameter, and we set \(d=1/2\). \[\left\{ \begin{array}{ll}Z[d:]&=W_{f}x\\ Z[:d]&=W_{g}x\end{array} \right. \tag{3}\]  
A convolution with kernel size \(1\times 1\) is efficient in extracting dense features with a small receptive field, and it is helpful to find subtle differences among nodules. A dilation convolution with kernel size \(3\times 3\) is efficient in extracting sparse features with a large receptive field, and it is helpful to capture the overall features of nodules. We chose to concatenate the two group feature maps rather than add them together. After the concatenation operation, the SE module was utilized to screen important features and to filter out unimportant features. The global average pooling operation was utilized to transform \(Z\in\mathbb{R}^{H\times W\times D\times C}\) to \(P\in\mathbb{R}^{1\times 1\times 1\times C}\), and \(c\)th element of \(Z\) is calculated by Equation (4), in which \(H\times W\times D\) is the spatial dimensions of input feature maps \(Z\). \[P_{c}=\frac{1}{H\times W\times D}\underset{i=1}{\sum}\underset{j=1}{\sum} \underset{d=1}{\sum}\underset{d=1}{\sum}\underset{d=1}{\sum}\underset{d=1}{ \sum}\underset{d=1}{Z_{c}(i,j,d)} \tag{4}\]  
To capture the channel-wise dependencies, a fully connected layer was employed, and it can be expressed as Equation (5), in which \(\delta\) represents the ReLU function, \(W_{1}\in\mathbb{R}^{C/r\times C}\), and \(W_{2}\in\mathbb{R}^{C\times C/r}\). \(r\) is the reduction ratio, which is used to reduce the parameters of the model. \[M=F_{fc}(P)=\delta(W_{2}\delta(W_{1}P)) \tag{5}\]  
The final results of the MSA block can be calculated by Equation (6), in which \(O=[O_{1},O_{2},\ldots,O_{c}]\) and \(F_{scale}(Z_{c},M_{c})\) refers to a channel-wise multiplication between the scalar \(M_{c}\) and the feature map \(Z_{c}\in\mathbb{R}^{H\times W\times D}\).