# A Review of Neural Network-Based Emulation of Guitar Amplifiers
## 3 Neural Network-Based Methods
### Architectures
The results of listening tests indicated that a small difference was perceived among the models in comparison to the reference. The first article presenting the use of recurrent neural networks for vacuum tube amplifier modelling dates back to 2013 where a Nonlinear AutoRegressive eXogenous (NARX) \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **WaveNet 1** & **WaveNet 2** & **WaveNet 3** \\ \hline
**Layers** & 10 & 18 & 18 \\
**Channels** & 16 & 8 & 16 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Feedforward WaveNet configuration summary. The nomenclature for these configurations is not generalized, and the names used here are taken from [26]. Figure 1: The general WaveNet architecture comprising various layers of dilated convolutions, illustrated here with a receptive field of size 8. Figure adapted from [26]. network was applied to the task in [30]. A NARX network is similar to a traditional RNN (i.e., a fully-connected network incorporating recurrence) but with limited connectivity to remedy the training problems usually present in RNN which are linked to vanishing and exploding gradients due to their recursive nature. The architecture used was a two-layer, feedforward network with sigmoid activation functions in the hidden layer and a single linear neuron in the output layer. The audio quality of this method was reported to be low when modelling a 4W Vox AC4 TV tube amplifier either due to insufficient training or limited model capacity. [PERSON] et al. [31] follow up the work on the NARX network with a study using Long Short Term Memory (LSTM), first proposed in [32], for the task of amplifier modelling. Again, a 4W Vox AC4 TV was chosen for this work. LSTM is a RNN variant that incorporates the use of various gates to control information flow through each recurrent layer in order to avoid the vanishing and exploding gradients of regular RNN. These gates are the following:  
\[\text{Input gate:}\ \mathbf{i}^{(t)}=\sigma(W_{ix}\mathbf{x}^{(t)}+W_{ih} \mathbf{h}^{(t-1)}+\mathbf{b}_{i}) \tag{3}\] \[\text{Forget gate:}\ \mathbf{f}^{(t)}=\sigma(W_{fx}\mathbf{x}^{(t)}+W_{fh} \mathbf{h}^{(t-1)}+\mathbf{b}_{f})\] (4) \[\text{Input node:}\ \mathbf{g}^{(t)}=\tanh(W_{cx}\mathbf{x}^{(t)}+W_{ ch}\mathbf{h}^{(t-1)}+\mathbf{b}_{c})\] (5) \[\text{Output gate:}\ \mathbf{o}^{(t)}=\sigma(W_{ox}\mathbf{x}^{(t)}+W_{ oh}\mathbf{h}^{(t-1)}+\mathbf{b}_{o})\] (6) \[\text{Cell state:}\ \mathbf{c}^{(t)}=\mathbf{g}^{(t)}\odot \mathbf{i}^{(t)}+\mathbf{c}^{(t-1)}\odot\mathbf{f}^{(t)}\] (7) \[\text{Hidden state:}\ \mathbf{h}^{(t)}=\tanh(\mathbf{c}^{(t)}) \odot\mathbf{o}^{(t)} \tag{8}\]  
where \(\sigma\) is the sigmoid activation function, the \(W\)s represent the weights of the layer, the \(b\)s represent the biases, and \(\odot\) is the Hadamard product. The \(\mathbf{x}\) and the \(\mathbf{h}\) refer to the input of the cell and the hidden state, respectively. The models used were configured as three or four-layer networks with a structure comprising a variable length input, followed by LSTM with a variable number of units (including either linear or tanh activation) and a linear output unit. The number of units used in the LSTM layers varied from 5 to 20, with a number of hidden units varying in the same range and a sequence length in the range of [1; 5]. During subjective listening tests, the audio quality of the network output, when compared to the target device, was not deemed satisfactory by semiprofessional guitarists. [PERSON] et al. tested a new LSTM architecture along with another variant on the traditional RNN, the Gated Recurrent Unit (GRU) [33] and both were compared to the WaveNet architecture from [26]. The GRU, like LSTM, aims to remedy exploding and vanishing gradient problems by controlling information flow through using an update gate instead of the four gates of LSTM, leading to improved computational efficiency.