# Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms
## 4 Evaluation
### Model Configurations  
The configurations of the two transcription models (CNN-BiGRU and CNN-SelfAtt(G-SyncPE) described in Section 3.1) are shown in Figure 5 and Table 1. The encoders were the same CNN consisting of four convolutional layers with a kernel size of \(3\times 3\), and the decoders were based on the BiGRU and the multi-head self-attention mechanism. The influential hyperparameters were automatically determined with a Bayesian optimizercalled Optuna [65] for the validation data in the Slakh dataset or with 10-fold cross validation in the RWC dataset under a condition that \(D_{\text{FFN}}=4D_{\text{F}}\). As a result, CNN-SelfAtt had about twice as many parameters as CNN-BiGRU. In the training phase, the batch size was set to 10, and the max length was set to 256 for CNN-SelfAtt. We used the AdamW optimizer [66] with an initial learning rate of \(10^{-3}\). The learning rate of CNN-SelfAtt was changed according to [26], and \(warmup\_steps\) was set to 4000. To prevent over-fitting, we used weight regularization (\(\lambda=10^{-4}\)), drop-out just before all of the fully connected layers of CNN-BiGRU (\(p=0.2\)) and each layer of CNN-SelfAtt (\(p=0.1\)), and datum-level SpecAugment [67] for the RWC dataset, where 15% of all tatums were masked in the training phase. The weights of the convolutional and BiGRU layers were initialized based on [68], the fully connected layer was initialized by the sampling from \(\text{Uniform}(0,1)\), and the biases were initialized to 0. In the testing phase, the average of the ten parameters before and after the epoch that achieved the smallest loss for the validation data was used in CNN-SelfAtt. The threshold for \(\mathbf{\phi}\) was set to \(\delta=0.2\).  
The configurations of the three language models (bi-gram, GRU, and MLM(-SyncPE) described in Section 3.2) are shown in Table 2. Each model was trained with 512 external drum scores of Japanese popular songs and Beatles songs. To investigate the impact of the data size for predictive performance, each model was also trained by using only randomly selected 51 scores. The influential hyperparameters of the neural language models, i.e., the number of layers and the dimension of hidden states in the GRU model and \(h\), \(l\), \(D_{\text{F}}\), and \(D_{\text{FFN}}\) in the MLM, were automatically determined with Optuna [65] via three-fold cross validation with the 512 scores under a condition that \(D_{\text{FFN}}=4D_{\text{F}}\). As a result, the MLM had about twice as many parameters as the GRU model. The bi-gram model was defined by only two parameters \(\pi_{0,1}\) and \(\pi_{1,1}\).  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**BiGRU**} & \multicolumn{5}{c}{**SelfAtt**} \\ \hline  & **Layer** & **Dim** & **Size** & \(h\) & \(l\) & \(D_{\text{F}}\) & \(D_{\text{FFN}}\) & **Size** \\ \hline Slakh & 1 & 131 & 573 k & 2 & 8 & 96 & 384 & 1.03 M \\ RWC & 3 & 98 & 774 k & 8 & 7 & 120 & 480 & 1.38 M \\ \hline \hline \end{tabular}
\end{table}
Table 1: Hyperparameters of the decoders of the two transcription models optimized by Optuna.  
Figure 5: Configurations of the transcription models (CNN-BiGRU and CNN-SelfAtt).  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**BiGRU**} & \multicolumn{5}{c}{**SelfAtt**} \\ \hline  & **Layer** & **Dim** & **Size** & \(h\) & \(l\) & \(D_{\text{F}}\) & \(D_{\text{FFN}}\) & **Size** \\ \hline Slakh & 1 & 131 & 573 k & 2 & 8 & 96 & 384 & 1.03 M \\ RWC & 3 & 98 & 774 k & 8 & 7 & 120 & 480 & 1.38 M \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyperparameters of the three language models optimized by Optuna.