# Real-Time Guitar Amplifier Emulation with Deep Learning +
## 4 Validation
### Listening Tests  
To investigate the impact of the ESR on the perceptual quality of the models, listening tests were conducted. The listening tests were conducted using the multiple stimuli with hidden reference and anchor (MUSHRA) methodology [38]. To avoid presenting test subjects with an excessive number of test conditions, some models were excluded from the tests. The tests included the hidden reference, the WaveNet1 and WaveNet3 models, the RNN models with LSTM hidden sizes of 32 and 96, a multilayer perceptron (MLP) model with an input size of 1024 and 8 hidden layers of 16 neurons each, and an anchor that was created using a static nonlinear function.  
The listening test items consisted of six clips, each approximately 2-4 s long, taken from the validation and test sets of the dataset. The MUSHRA test was carried out using the WebMUSHRA interface [39], shown in Figure 12. For each test the user was presented with the reference and the seven test items. For each test the user was asked to rate each of the test items based on \"how accurately they emulate the timbre of the reference\". The ratings were given on a scale of 0-100. Each of the six test items were processed by the two amplifiers being modelled, resulting in a total of 12 MUSHRA  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **WaveNet1** & **WaveNet2** & **WaveNet3** & **RNN32** & **RNN64** & **RNN96** \\ \hline HT5M & 2.8\% & 0.60\% & **0.32\%** & 4.2\% & 3.2\% & 1.8\% \\ Mesa 5:50 & 0.64\% & 0.46\% & 0.29\% & 0.68\% & 0.29\% & **0.20\%** \\ \hline Speed (x RT) & 1.9 & 1.6 & 1.1 & **8.3** & 4.2 & 2.5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Error-to-signal ratio for the test set and model processing speeds.  
Figure 11: The frequency response of the low-passed A-weighting pre-emphasis filter.  
trials. Audio examples from the MUSHRA test are available online at [[http://research.spa.aalto.fi/publications/papers/applsci-deep/](http://research.spa.aalto.fi/publications/papers/applsci-deep/)]([http://research.spa.aalto.fi/publications/papers/applsci-deep/](http://research.spa.aalto.fi/publications/papers/applsci-deep/)).  
A total of 14 people completed the MUSHRA test. The subjects all played at least one musical instrument, and none reported any hearing impairments. The listening tests were conducted at the Aalto Acoustics Lab in a silent room using Sennheiser HD-650 headphones. The test subjects were asked to choose a loud, but not uncomfortable, volume during the two training examples at the beginning of the test, and then instructed not to change the volume throughout the rest of the trials. The data of five of the test participants was excluded from the test results as they rated the hidden reference below 90 in more than 15% of the trials.  
The results of the MUSHRA trials with 95% confidence interval are shown in Figures 13 and 14 for the Mesa 5:50 and HT5M, respectively. The Mean MUSHRA scores are shown in Table 3. For the Mesa 5:50 the mean MUSHRA scores are all above 90, indicating that the participants generally rated the models as Excellent or better. It is also worth noting that both the WaveNet3 and the RNN-96 received a score of 100 in approximately 80% of the trials conducted, meaning that the subjects were often unable to hear any difference between the models and the reference.  
For the HT5M the scores are generally lower, however for the larger WaveNet-style and RNN models the mean score is at or above 90, indicating that they were rated as Excellent on average. The best performing model is the WaveNet3, although as the confidence intervals of the RNN-96 and WaveNet3 perceptual scores are overlapping, it is not clear that the difference between the two is statistically significant or not.