# High-Quality and Reproducible Automatic Drum Transcription From Crowdsourced Data
## 4 Results
### Evaluating Training Procedures  
In the following, we evaluate the reference (frame-synchronous RNN) architecture when trained with nine different combinations of datasets. Specifically, we trained a model with and without specific datasets to measure (1) the contribution of the different datasets toward the model's quality and (2) the generalization capabilities achieved on the datasets that were not used for the training. The results are presented in Table 5, where the top five rows refer to models trained without pre-training, while the bottom four rows refer to models that were pre-trained on the TMIDT dataset. A comparison of the first two rows of Table 5 helps to understand the contribution (to the training) of the small non-crowdsourced datasets (RBMA, ENST, and MDB) when added to the crowdsourced data (ADTOF-RGW and YT). It can be observed that adding RBMA, ENST, and MDB to the training data did not improve the results of the model, except when testing on ADTOF-RGW, but only slightly (from 0.82 to 0.83). On the one hand, this reveals that the limited diversity of these datasets (RBMA, ENST, and MDB) prevented an improvement in the generalization of the model, even with the boosted sampling probability (Section 3.2). On the other hand, this also shows that the model trained exclusively on ADTOF-RGW and ADTOF-YT already performed well on other datasets (zero-shot performance of ADTOF-RGW plus ADTOF-YT on RBMA, ENST, and MDB). \begin{table}
\begin{tabular}{l l c c c c c} \hline \hline \(\#\) & **Training Dataset (s)** & **ADTOF-RGW** & **ADTOF-YT** & **RBMA** & **ENST** & **MDB** \\ \hline
1 & All five & **0.83** & **0.85** & **0.65** & **0.78** & **0.81** \\
2 & ADTOF-RGW, ADTOF-YT & 0.82 & **0.85** & **0.65** & **0.78** & **0.81** \\
3 & ADTOF-RGW ([PERSON] et al. [3]) & 0.79 & 0.73 & 0.63 & 0.72 & 0.76 \\
4 & ADTOF-YT & 0.76 & **0.85** & 0.57 & 0.73 & 0.78 \\
5 & RBMA, ENST, MDB & 0.63 & 0.48 & 0.57 & 0.73 & 0.74 \\ \hline
6 & All five & 0.79 & 0.81 & 0.62 & 0.77 & 0.77 \\
7 & ADTOF-RGW, ADTOF-YT & 0.79 & 0.81 & 0.62 & 0.76 & 0.77 \\
8 & RBMA, ENST, MDB ([PERSON] et al. [2]) & 0.70 & 0.56 & 0.63 & 0.76 & 0.75 \\
9 & - & 0.70 & 0.62 & 0.60 & 0.75 & 0.68 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: F-measure (between 0 and 1) for different training procedures achieved by the “frame RNN” architecture. By comparing rows 2 and 3, and rows 2 and 4, one can see the contribution of the datasets ADTOF-YT and ADTOF-RGW with respect to the training carried out using both. As expected, since these datasets are biased towards different music genres, their joint usage achieved noticeably better results for zero-shot performance. For instance, the addition of ADTOF-RGW to ADTOF-YT brought the F-score on RBMA from 0.57 to 0.65; likewise, the addition of ADTOF-YT to ADTOF-RGW brought the F-score on ENST and MDB from 0.72 and 0.76 to 0.78 and 0.81, respectively. In other words, since mixing these datasets increased the diversity of the training data, the zero-shot performance of the model also increased. However, it is rather surprising to notice that while the addition of ADTOF-YT improved the performance on ADTOF-RGW (from 0.79 to 0.82), no improvement was observed on ADTOF-YT when ADTOF-RGW was added (the F-score stayed at 0.85). We attribute this behavior to the fact that ADTOF-RGW contains mistakes in the annotations and is smaller than ADTOF-YT. This effectively prevented the model trained exclusively on ADTOF-RGW from achieving the best performance on this dataset. Nonetheless, as already observed in [3], thanks to its large size and diversity, ADTOF-RGW was still better for training than any of the non-crowdsourced datasets (row 5). Finally, we quantified the contribution of pre-training on the large synthetic dataset TMIDT, following a method suggested by [PERSON] et al. [2] and known to counter the data paucity of real-word recordings.