# Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms
## 4 Evaluation
### Investigation of Self-Attention Mechanism  
We further investigated the behaviors of the self-attention mechanisms used in the transcription and language models. To validate the effectiveness of the proposed tatum-synchronous positional encodings (SyncPE), we compared two versions of the proposed transcription model, denoted by CNN-SelfAtt and CNN-SelfAtt-SyncPE, in terms of the F-measure and TER. As shown in Table 7, CNN-SelfAtt-SyncPE always outperformed CNN-SelfAtt by a large margin. To investigate the impact of the data size used for training the transcription models (CNN-BiGRU and CNN-SelfAtt-SyncPE), we compared the performances obtained by using 1/32, 1/16, 1/4, 1/2, and all of the training data in the Slakh or RWC datasets. As shown in Figure 8, CNN-SelfAtt-SyncPE was severely affected by the data size and CNN-BiGRU worked better than CNN-SelfAtt when only a small amount of paired data were available. To investigate the impact of the data size used for training the language models (bi-gram, GRU, and MLM-SyncPE), we compared the performances obtained by CNN-SelfAtt-SyncPE that was regularized with a language model pretrained with 512 or 51 external drum scores. As shown in Table 8, the pretrained language models with a larger number of drum scores achieved higher performances. The effect of the MLM-SyncPE-based regularization severely depends on the data size, whereas the bi-gram model was scarcely affected by the data size. \begin{table}
\begin{tabular}{c l c c c c c c c c c c} \hline \hline  & **Decoder** & **Time** & \multicolumn{6}{c}{**Madmom**} & \multicolumn{6}{c}{**Ground-Truth**} \\ \hline  & **+ Language Model** & & **BD** & **SD** & **HH** & **Total** & **TER** & **BD** & **SD** & **HH** & **Total** & **TER** \\ \hline \multirow{8}{*}{Slakh} & BiGRU [18] & & 55.8 & 93.6 & 92.7 & 71.5 & 85.9 & 20.6 & 93.0 & 92.5 & 71.4 & 85.6 & 8.4 \\  & BiGRU & & 15.3 & 95.6 & 90.2 & 75.5 & 87.1 & 19.6 & 95.2 & 90.2 & 75.8 & 87.1 & 7.2 \\  & + MLM-SyncPE & (512) & 137.0 & 95.3 & 90.7 & 78.4 & 88.1 & 19.0 & 94.7 & 90.9 & 77.6 & 87.8 & 7.1 \\  & SelfAtt-SyncPE & & 26.7 & 95.8 & 93.1 & 79.9 & 89.6 & 18.7 & 95.6 & 92.9 & 79.5 & 89.3 & 6.8 \\  & + Bi-gram & (512) & 15.6 & 95.8 & 92.7 & 80.5 & 89.7 & **18.5** & 95.4 & 93.1 & 80.5 & 89.7 & **6.3** \\  & + GRU & (512) & 15.7 & 96.4 & 93.0 & 80.5 & **90.0** & **18.5** & 96.1 & 92.9 & 80.4 & 89.8 & 6.4 \\  & + MLM-SyncPE & (512) & 42.0 & 96.1 & 93.2 & 80.7 & **90.0** & **18.5** & 95.8 & 93.3 & 80.9 & **90.0** & **6.3** \\ \hline \multirow{8}{*}{RWC} & BiGRU [18] & & 58.2 & 86.0 & 74.0 & 70.5 & 76.8 & 16.7 & 86.6 & 74.1 & 70.9 & 77.2 & 12.3 \\  & BiGRU & & 24.5 & 86.9 & 76.2 & 77.9 & 80.3 & 15.2 & 86.7 & 76.5 & 76.9 & 80.0 & 11.4 \\  & + MLM-SyncPE & (512) & 130.1 & 88.0 & 76.5 & 79.7 & **81.4** & **14.0** & 88.1 & 76.5 & 79.3 & **81.3** & **10.3** \\  & SelfAtt-SyncPE & & 30.5 & 87.5 & 76.4 & 72.6 & 78.8 & 17.0 & 88.0 & 75.6 & 72.9 & 78.8 & 13.2 \\  & + Bi-gram & (512) & 31.8 & 86.0 & 76.5 & 69.6 & 77.4 & 16.8 & 87.0 & 76.0 & 69.6 & 77.5 & 12.7 \\  & + GRU & (512) & 24.1 & 87.6 & 76.2 & 73.2 & 79.0 & 16.6 & 87.8 & 76.3 & 73.9 & 79.4 & 12.5 \\  & + MLM-SyncPE & (512) & 51.6 & 88.1 & 74.9 & 75.4 & 79.5 & 16.2 & 87.9 & 74.3 & 71.7 & 78.0 & 12.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Training times per song (s/song), F-measures (%), and TERs obtained by the conventional and proposed methods.