# DANet: Temporal Action Localization with Double Attention
## 3 Double Attention Network
### Double Attention Block  
The backbone module consists of a number of \(L_{\text{emd}}\) projection layers denoted as \(E\) and a number of \(L_{\text{att}}\) DABlock layers. Specifically, each projection layer contains a 1D convolutional neural network followed by layer normalization and RELU. Let the output of all projection layers be denoted by P, which is given by:  
\[P=E(X), \tag{1}\]  
where \(P\in\mathbb{R}^{T\times\text{C}_{\text{emd}}}\) and \(\text{C}_{\text{emd}}\) denotes the dimension of all projection layers. \(T\) varies depending on the video length. Subsequently, the output of all projection layers, \(P\), is used as the input of the 0 th DABlock \(D^{0}\). In addition, let \(D^{i}\) denote the input of the \(i\)th DABlock, where \(i\in\{0,1,\ldots,L_{att}-1\}\). As shown in Figure 1, the DABlock comprises two attention modules that capture the local and global temporal relationships in an untrimmed video, named MLSA and MA. In MLSA, to focus on the local temporal context and enhance local awareness, we employ  
Figure 1: Overview of the proposed DANet, comprising feature extractor, backbone, neck, and head. multi-headed self-attention to model temporal relationships, which is computed as follows:  
\[S=\textit{softmax}(QK^{T}/\sqrt{D_{q}})V, \tag{2}\]  
where \(Q=D^{i}W_{Q},K=D^{i}W_{K},V=D^{i}W_{V}\), and are referred to as query, key, and value, respectively. \(W_{Q}\in\mathbb{R}^{C_{\text{end}}\times C_{q}},W_{K}\in\mathbb{R}^{C_{\text{ end}}\times C_{q}},W_{V}\in\mathbb{R}^{C_{\text{end}}\times C_{q}}\) denote the weight of query, key, and value, respectively. Multi-headed self-attention performs self-attention operations on \(N_{\text{head}}\) heads in parallel, and thus has \(C_{\text{end}}/N_{\text{head}}\) channels at each head. After calculation, let \(h_{m}^{i}\) denote the output of the \(m\)th head. Therefore, the output of the \(i\)th MLSA is as follows:  
\[M^{i}=\textit{Concat}(h_{1}^{i},h_{2}^{i},\dots,h_{N_{\text{head}}}^{i}). \tag{3}\]  
To preserve the most critical local features and aggregate global feature information, we propose MA, whose structure is illustrated in Figure 2. We apply MA to maintain the discriminative features from local embeddings through the use of max-pooling, while aggregating global feature information via average-pooling. In addition, we apply polling operations on both the channel and temporal dimensions and generate corresponding attention weights. Specifically, for the channel-wise attention weights, we utilize max-pooling to preserve the most salient traits of local clip embeddings. We employ average-pooling to aggregate information for each channel. Subsequently, we utilize two fully connected layers, along with RELU and sigmoid functions, to calculate the channel-wise attention weights \(W_{c}\)  
Figure 2: Illustration of the structure of the proposed MA. We use MA to preserve critical local features by employing max-pooling, and aggregate global feature information via average-pooling, denoted as Avg Pooling. Attention weights spanning both channel and temporal dimensions are generated to aid in this process. These weight variables are computed based on the importance of different feature elements and are crucial for improving the model’s representation capability. Finally, we upon multiplication of channel-wise weights with the original inputs, whose formula is as follows:  
\[C^{i}=W_{c}\otimes M^{i}. \tag{4}\]  
In addition, to focus on which part of each snippet is the most salient feature at time step \(t\), we generate temporal-wise attention weights and use \(C^{i}\) as input. We want to focus on which part of each snippet is the most salient feature at time step \(t\). We also use average-pooling and max-pooling, but the pooling operation is performed on the temporal axis and concatenate them to generate an efficient feature descriptor without fully connected layers. Subsequently, \(D^{i+1}\) is computed as follows:  
\[D^{i+1}=W_{t}\otimes C^{i}. \tag{5}\]  
Therefore, we successfully obtain the multi-scale representation input.