# Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms
## 3 Proposed Method
### Transcription Models
For each head \(i\) (\(1\leq i\leq I\)), let \(\mathbf{Q}_{i}\triangleq[\mathbf{q}_{i,1},\ldots,\mathbf{q}_{i,N}]\in\mathbb{ R}^{D_{K}\times N}\), \(\mathbf{K}_{i}\triangleq[\mathbf{k}_{i,1},\ldots,\mathbf{k}_{i,N}]\in\mathbb{ R}^{D_{K}\times N}\), and \(\mathbf{V}_{i}\triangleq[\mathbf{v}_{i,1},\ldots,\mathbf{v}_{i,N}]\in\mathbb{ R}^{D_{K}\times N}\) be query, key, and value matrices given by  
\[\mathbf{Q}_{i} =\mathbf{W}_{i}^{(Q)}\text{LayerNorm}(\mathbf{Z})+\mathbf{b}_{i }^{(Q)}, \tag{4}\] \[\mathbf{K}_{i} =\mathbf{W}_{i}^{(K)}\text{LayerNorm}(\mathbf{Z})+\mathbf{b}_{i }^{(K)},\] (5) \[\mathbf{V}_{i} =\mathbf{W}_{i}^{(V)}\text{LayerNorm}(\mathbf{Z})+\mathbf{b}_{i }^{(V)}, \tag{6}\]  
where \(D_{k}\) is the feature dimension of each head (\(D_{k}{=}\frac{D_{F}}{I}\) in this paper as in [26]): \(\mathbf{q}_{i}\in\mathbb{R}^{D_{K}}\), \(\mathbf{k}_{i}\in\mathbb{R}^{D_{K}}\), and \(\mathbf{v}_{i}\in\mathbb{R}^{D_{K}}\) are query, key, and value vectors, respectively; \(\mathbf{W}_{i}^{(Q)}\in\mathbb{R}^{D_{K}\times D_{F}}\), \(\mathbf{W}_{i}^{(K)}\in\mathbb{R}^{D_{K}\times D_{F}}\), and \(\mathbf{W}_{i}^{(V)}\in\mathbb{R}^{D_{K}\times D_{F}}\) are weight matrices; and \(\mathbf{b}_{i}^{(Q)}\in\mathbb{R}^{D_{K}\times N}\), \(\mathbf{b}_{i}^{(K)}\in\mathbb{R}^{D_{K}\times N}\), and \(\mathbf{b}_{i}^{(V)}\in\mathbb{R}^{D_{K}\times N}\) are bias vectors. Let \(\mathbf{\alpha}\in\mathbb{R}^{N\times N}\) be a self-attention matrix consisting of the degrees of self-relevance of the extended latent features \(\mathbf{Z}\), which is given by  
Figure 2: The original positional encoding described in [26] (**left**) and the proposed one (**right**) with \(D_{F}=512\) and \(N=256\). \[e_{i,n,n^{\prime}} =\frac{\mathbf{q}_{i,n}^{\top}\mathbf{k}_{i,n^{\prime}}}{\sqrt{D_{K}}}, \tag{7}\] \[\alpha_{i,n,n^{\prime}} =\frac{\exp\left(e_{i,n,n^{\prime}}\right)}{\sum_{n^{\prime}=1}^{N }\exp\left(e_{i,n,n^{\prime}}\right)}, \tag{8}\]  
where \({}^{\top}\) represents the matrix or vector transpose, and \(n\) and \(n^{\prime}\) represent the feature indices of \(\mathbf{Q}\) and \(\mathbf{K}\), respectively. Let \(\mathbf{H}\triangleq[\mathbf{H}_{1},\dots,\mathbf{H}_{l}]\in\mathbb{R}^{D_{F} \times N}\) be a feature matrix obtained by concatenating all the heads, where \(\mathbf{H}_{i}\triangleq\mathbf{V}_{i}\mathbf{A}_{i}^{\top}\in\mathbb{R}^{D_ {K}\times N}\). The extended latent features and the extracted features \(\mathbf{H}\) with Dropout (\(p=0.1\)) [60] are then fed into a feed forward network (FFN) with a rectified linear unit (ReLU) as follows:  
\[\mathbf{Z}\leftarrow\mathbf{W}_{2}^{(H)}\text{max}\Big{\{}0,\mathbf{W}_{1}^{( H)}(\text{Dropout}(\mathbf{H})+\mathbf{Z})+\mathbf{b}_{1}^{(H)}\Big{\}}+\mathbf{b}_{2}^{ (H)}, \tag{9}\]  
where \(\mathbf{W}_{1}^{(H)}\in\mathbb{R}^{D_{\text{FFN}}\times D_{F}}\) and \(\mathbf{W}_{2}^{(H)}\in\mathbb{R}^{D_{F}\times D_{\text{FFN}}}\) are weight matrices, \(\mathbf{b}_{1}^{(H)}\in\mathbb{R}^{D_{\text{FFN}}\times N}\) and \(\mathbf{b}_{2}^{(H)}\in\mathbb{R}^{D_{F}\times N}\) are bias vectors, and \(D_{\text{FFN}}\) is the dimension of the output. Equation (4) to Equation (9) are repeated \(L\) times with different parameters. The onset probabilities \(\boldsymbol{\phi}\) are finally calculated as follows:  
\[\boldsymbol{\phi}=\sigma\Big{(}\mathbf{W}_{3}^{(H)}\mathbf{Z}+\mathbf{b}_{3} ^{(H)}\Big{)}, \tag{10}\]  
where \(\sigma(\cdot)\) is a sigmoid function, \(\mathbf{W}_{3}^{(H)}\in\mathbb{R}^{M\times D_{F}}\) is a weight matrix, and \(\mathbf{b}_{3}^{(H)}\in\mathbb{R}^{M\times N}\) is a bias vector.