# A Deeper Look at Sheet Music Composer Classification Using Self-Supervised Pretraining
## 2 Materials and Methods
### Pretraining  
The first stage is to pretrain the model using self-supervised language modeling. Given a set of unlabeled sheet music images, the output of the pretraining stage is a trained language model. The pretraining consists of four steps, as shown at the top of Figure 1.  
The first step of pretraining is to convert each sheet music image into a bootleg score representation. The bootleg score [41] is a mid-level feature representation that encodes the sequence and positions of filled noteheads in sheet music. It was originally proposed for a sheet image-MIDI retrieval task, but has been successfully applied to other tasks such as sheet music identification [2] and audio-sheet image synchronization [43]. The bootleg score itself is a \(62\times N\) binary matrix, where 62 indicates the total number of possible staff line positions in both the left- and right-hand staves and \(N\) indicates the total number of grouped note events after collapsing simultaneous note onsets (e.g., a chord containing four notes would constitute a single grouped note event). Figure 2 shows a short excerpt of sheet music and its corresponding bootleg score which contains \(N=23\) grouped note events. The bootleg score is computed by detecting three types of musical objects in the sheet music: staff lines, bar lines, and filled noteheads. Because these objects all have simple geometric shapes (i.e., straight lines and circles), they can be detected reliably and efficiently using classical computer vision techniques such as erosion, dilation, and blob detection. The feature extraction has no trainable weights and only about 30 hyperparameters, so it is very robust to overfitting. Indeed, the feature representation has been successfully used with very different types of data, including synthetic sheet music, scanned sheet music, and cell phone pictures of physical sheet music. For more details on the bootleg score feature representation, the reader is referred to [41]. Figure 1: Overview of approach to composer style classification of sheet music images. The input \(x\) is a raw sheet music image, and the label \(y\) indicates the composer. The second step in pretraining is to augment the data by pitch shifting. One interesting characteristic about the bootleg score representation is that key transpositions correspond to shifts in the feature representation. Therefore, one easy way to augment the data is to consider shifted versions of each bootleg score up to a maximum of \(\pm K\) shifts. For example, when \(K=2\), this augmentation strategy will result in five times more data: the original data plus versions with shifts of \(+2\), \(+1\), \(-1\), and \(-2\). When pitch shifting causes a notehead to \"fall off\" the bootleg score canvas (e.g., a notehead near the top of the left-hand staff gets shifted up beyond the staff's range), the notehead is simply eliminated. In our experiments, we considered values of \(K\) up to 3. The third step in pretraining is to tokenize each bootleg score into a sequence of words or subwords. This is done in one of two ways, depending on the underlying language model. For word-based language models (e.g., AWD-LSTM [44]), each column of the bootleg score is treated as a single word. In practice, this can be done easily by simply representing each column of 62 bits as a single 64-bit integer. Words that occur fewer than three times in the data are mapped to a special unknown token <UNK>. For subword-based language models (e.g., GPT-2 [36], RoBERTa [45]), each column of the bootleg score is represented as a sequence of 8 bytes, where each byte constitutes a single character. The list of unique characters forms the initial vocabulary set. We then apply a byte pair encoding (BPE) algorithm [46] to iteratively merge the most frequently occurring pairs of adjacent vocabulary items until we reach a desired subword vocabulary size. In our experiments, we use a vocabulary size of 30,000 subwords. The (trained) byte pair encoder can be used to transform a sequence of 8-bit characters into a sequence of subwords.