# Generating Music with Data: Application of Deep Learning Models for Symbolic Music Composition
## 2 Materials and Methods
### Models
#### Custom GPT2  
The Custom GPT2 model was developed with the aim to be an alternative to the Music Transformers and MuseNet models. This model started from the implementation of a GPT2 causal language model [28] offered by the transformers library [22] through the HuggingFace platform.  
1. Tokenizer  
A \"tokenizer\" was trained to generate the vocabulary of the problem, using the pre-processed dataset of textual sequences as a base. The original tokenization training strategy of the GPT2 model [28] was based on the byte pair encoding (BPE) technique due to the extensive size of the worked vocabulary (50,257 tokens). Taking into account the fact that the data pre-processing strategy adopted in this work resulted in a relatively small vocabulary (390 tokens), the word level training technique was chosen for the tokenizer, which directly relates an identification number to each vocabulary token, thus making an event-to-token mapping. The HuggingFace tokenizers library was chosen to carry out this task due to its ease of manipulation and high performance.  
1. Inputs and Model  
The GPT2 model must receive a pre-defined tensor dimensionality for consistency of the matrix calculation in the input layer. The original implementation by [PERSON] et al. (2019) uses sequences of 1024 tokens in \"batches\" of 512 sequences (input of [1024 \(\times\) 512]). Unfortunately, these values could not be replicated due to a hardware limitation. Therefore, to adapt the  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Learning rate & 0.001 \\ Batch size & 64 \\ Window size & 200 \\ Stride & 10 \\ Transposition use & False \\ Control relationship & 1 \\ Teacher forcing proportion & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: List of hyperparameters for training the Custom GRU model.  
model to the available hardware resources, sequences of 64 tokens in batches of 64 sequences were chosen (input of [64 \(\times\) 64]). Sequences with more than 64 tokens were therefore truncated into multiple sequences less than or equal to 64 tokens, e.g., a sequence of 200 tokens [200 \(\times\) 1] was transformed into four sequences of 64 [64 \(\times\) 4]. It is worth noting in this example that the last sequence would initially have 8 tokens (200 = 64 + 64 + 64 + 8).  
However, as mentioned earlier, the input tensor needs to have consistent dimensionality. Because of this, strings of tokens smaller than 64 had their difference in \"PAD\" tokens. \"PAD\" tokens are ignored by the model. Therefore, in the previous example, the last sequence of 8 tokens was extended with 56 \"PAD\" tokens, thus adapting the example tensor of [200 \(\times\) 1] to a tensor of [64 \(\times\) 4].  
Training  
The Custom GPT2 model managed to achieve a loss function value of 2.50 on the validation data after being trained for 116 h on two 24 GB NVIDIA GeForce RTX 3090 GPUs with the following hyperparameters listed in Table 4.  
Detokonizer and Decoder  
After training, the model was used to generate a musical language. Given one or more tokens in sequence and a maximum of M tokens to be generated, the GPT2 model output a sequence of dimensionality tokens [1 \(\times\) M] in an autoregressive manner. The model outputs were stochastically sampled using the beam search technique, thus replicating the same strategy used in the Performance RNN, Music Transformers and Custom GRU models. From this point on, the data reverse engineering process could begin. The tokenizer was used in this step, but this time acting in the mapping of tokens to events, which were consequently processed via reverse engineering the techniques used in the pre-processing, obtaining a MIDI file as a result. However, the model was not able to converge in results interesting enough for the research, probably due to the complexity of the model faced by the limitation of available hardware.