# Generating Music with Data: Application of Deep Learning Models for Symbolic Music Composition
## 2 Materials and Methods
### Models
#### 2.2.4 Training Framework and Music Generation for the Custom GRU and Custom GPT2 Model  
The Custom GRU and Custom GPT2 models were completely developed and trained in this work. These models used the result of data pre-processing as the dataset, as detailed in Section 2.1.2. It is worth noting that the Custom GPT2 model failed to achieve interesting results to be included in the blind test due to hardware limitations in view of the complexity of the model. This section details the entire \"pipeline\" used for training music generation models. The Custom GRU model was developed with the aim to be an alternative to the Performance RNN model, comprising an architecture that was also based on recurrent neural networks. Unlike the Performance RNN model, which is based on layers of long short-term memory (LSTM [14]), the GRU model is based on layers of gated recurrent units [15]. The development of the model started from the work of [PERSON] [25], which was built using the Python programming language combined with the PyTorch framework [21]. Table 2 below details the layers used in the model alongside their respective parameter quantities. The model had 5,232,880 parameters in its entirety. All linear and embedding layers were initialized according to the method described by [PERSON] and [PERSON] [26] using a normal distribution. Therefore, the initial weights of the layers mentioned above were the result of the sampled values of N(0, std2), as defined in Equation (1):  
\[std=gain\ \times\sqrt{\frac{2}{fan_{in}+fan_{out}}} \tag{1}\]  
where gain: optional scaling factor, \(fan_{in}:\) number of layer inputs and \(fan_{out}\): number of layer outputs. On all initializations, gain assumed its default value of 1 (resulting in tensors not scaling). The model architecture started with a linear layer of 32 inputs and 1536 outputs, defined by Equation (2):  
\[y=x\ W^{T}\ +b \tag{2}\]  
where \(x\): layer input tensor, \(W^{T}\): layer weight transposed matrix and \(b\): bias matrix. The first linear layer was followed by an activation function of the hyperbolic tangent type, defined by Equation (3):  
\[tanh=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} \tag{3}\]  
where \(x\): output tensor of the previous layer. The result of this activation function was then passed to an embedding layer of dimensionality [240 \(\times\) 240]. The output of this layer was passed to another linear layer, followed by a leaky rectified linear unit (Leaky ReLU) activation function, defined by Equation (4):  
\[LeakyRelu(x)=(0,\ x)+slope\times(0,x) \tag{4}\]  
where \(x\): output tensor of the previous layer and \(slope\): negative slope. The negative slope value used in this architecture was 0.1. Then, the model followed with three layers of gated recurrent units [15], where each layer computed the system of equations, defined in Equation (5):  
\[ \begin{split} r_{t}\ =\sigma\Big{(}W_{ir}\ x_{t}\ +b_{ir}+W_{hr}\ h_{(t-1)}+b_{hr}\Big{)}\\ z_{t}=\sigma\Big{(}W_{iz}x_{t}+b_{iz}+W_{hz}h_{(t-1)}+b_{hz} \Big{)}\\ n_{t}=tanh\ \Big{(}W_{in}x_{t}+b_{in}+r_{t}\ *\ \Big{(}W_{hn}\ h_{(t-1)}+b_{ hm}\Big{)}\Big{)}\\ h_{t}\ =(1-\ z_{t})\ *\ n_{t}+z_{t}\ *\ h_{(t-1)}\end{split}  \tag{5}\]  
where \(r_{t}:\) reset port; \(\sigma\): sigmoid function; \(W_{ir}\,\ b_{ir},\ W_{hr}\) and \(b_{hr}\): learnable weight matrices and the bias vector for the reset port; \(x_{t}\): entry at time \(t\); \(h_{(t-1)}:\) hidden state of the layer at time \(t-1\) or the initial hidden state at time \(0\); \(z_{t}\): update port; \(W_{iz}\), \(b_{iz}\), \(W_{hz}\) and \(b_{hz}\): learnable weight matrices and the bias vector for the update port. \(n_{t}:\) new state port; \(W_{in}\), \(b_{in}\), \(W_{hn}\) and \(b_{hn}\): learnable weight matrices and the bias vector for the new state port. *: Hadamard product; and \(h_{t}:\) hidden state at time \(t\).