# A Deeper Look at Sheet Music Composer Classification Using Self-Supervised Pretraining
## 3 Results
### Fragment Classification Results  
We compare the performance of four different models: AWD-LSTM, GPT-2, RoBERTa, and a baseline CNN model. The CNN model follows the approach described by [24] for a composer classification task using ***kern scores. Their approach is to feed a piano roll-like representation into two convolutional layers, perform average pooling of the activations along the sequence dimension, and then apply a final linear classification layer. This CNN model can be interpreted as the state-of-the-art approach as of 2019. For each of the four models, we compare the performance under three different pretraining conditions. The first condition is no pretraining, in which the model is trained from scratch on the labeled fragment data. The second condition is target pretraining, in which a language model is first pretrained on the labeled (full-page) dataset, and then the pretrained classifier is finetuned on the labeled fragment data. The third condition is \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Composer** & **Pieces** & **Pages** & **Features** \\ \hline [PERSON] & 176 & 969 & 245k \\ [PERSON] & 70 & 666 & 153k \\ [PERSON] & 82 & 665 & 118k \\ [PERSON] & 50 & 347 & 88k \\ [PERSON] & 169 & 2272 & 394k \\ [PERSON] & 54 & 468 & 116k \\ [PERSON] & 75 & 578 & 139k \\ [PERSON] & 37 & 500 & 105k \\ [PERSON] & 74 & 686 & 108k \\ \hline Total & 787 & 7151 & 1.47M \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Overview of the labeled (full-page) dataset used for classifier finetuning. IMSLP pretraining, which consists of three steps: (1) training a language model on the full IMSLP dataset, (2) finetuning the language model on the labeled dataset, and (3) finetuning the pretrained classifier on the labeled fragment data. The third condition corresponds to the ULMFit [33] method. Comparing the performance under these three conditions will allow us to measure how much pretraining improves the performance of the classifier. Figure 4 shows the performance of all models on the fragment classification task. While our original goal is to classify full-page images, the performance on the test fragment data is useful because the dataset is balanced and more reliable due to its much larger size (i.e., number of samples). The three groups in the figure correspond to fragment sizes of 64 (left), 128 (middle), and 256 (right). For a given fragment size, the bars indicate the classification accuracy of all models, where different colors correspond to different pretraining conditions. Note that the CNN model only has results for the no pretraining condition, since it is not a language model. The bars indicate the performance of each model with training data augmentation \(K=3\) and test time augmentation \(L=2\). These settings were found to be best for the best-performing model (see Section 4.1) and were applied to all models in Figure 4 for fair comparison. The performance with no data augmentation (\(K=0\), \(L=0\)) is also overlaid as black rectangles for comparison. There are four things to notice about Figure 4. First, pretraining makes a big difference. For all language model architectures, we see a large and consistent improvement from pretraining. For example, for the RoBERTa model with fragment size 64, the accuracy improves from 50.9% to 72.0% to 81.2% across the three pretraining conditions. Second, the augmentation strategies make a big difference. Regardless of model architecture or pretraining condition, we see a very large improvement in classification performance. For example, the AWD-LSTM model with IMSLP pretraining and fragment size 64 improves from 53.0% to 75.9% when incorporating data augmentation. The effect of the training and test-time augmentation will be studied in more depth in Section 4.1. Third, the Transformer-based models outperform the LSTM and CNN models. The best model (GPT-2 with IMSLP pretraining) achieves a classification accuracy of 90.2% for fragment size 256. Fourth,  
Figure 4: Results on the fragment classification task.