# A Review of Neural Network-Based Emulation of Guitar Amplifiers
## 3 Neural Network-Based Methods
### Evaluation Methods  
The metrics presented in the previous section must be differentiable in order to be used within the optimization problem of the training phase. These metrics can also be used for the evaluation of the network after training and validation, as well as non-differentiable functions and subjective listening tests.  
While objective metrics struggle to properly reflect the perceptual aspects of the output audio, listening tests take time to implement and hinder continuous integration of ML systems. Therefore, there is a real need for objective evaluation metrics that do not rely on human participation.  
Most of the evaluation methods used for this modelling task rely on reuse of the loss functions used during training or a variation thereof. For example, [PERSON] et al. [26] used pre-emphasized ESR for training and plain ESR for evaluation. While simple to implement, a single objective metric cannot replace subjective listening tests.  
The listening tests that are mainly used for this task rely on MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) testing. In a MUSHRA test, the participants are presented with a labeled reference, various test samples, an unlabeled reference, and an anchor. This method is detailled in the Recommendation ITU-R BS.1534-3 [55].  
A similar framework used to carried out listening tests with human participants is the Web Audio Evaluation Tool [56] which is based on the HTML5 Web Audio API for perceptual audio evaluation.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Objective Function** & **Details** \\ \hline MSE & Non-perceptual, used in [31; 41; 43] \\ \hline RMS & MSE based loss, non-perceptual, used in [30] \\ \hline NRMSE & MSE based loss, non-perceptual \\ \hline ESR + \(H(z)=1-0.95z^{-1}\) & Pre-emphasis for better high-frequency content \\  & in [25; 26]. \\ \hline ESR + \(H_{\text{FD}}(z)\) & Studied for perceptual relevance in [53]. \\  & Includes DC offset. \\ \hline ESR + \(H_{\text{HP}}(z)\) & Studied for perceptual relevance in [53]. \\  & Includes DC offset. \\ \hline ESR + \(H_{\text{LP}}(z)\) & Studied for perceptual relevance in [53]. \\  & Includes DC offset and A-weighting. \\ \hline MAE & Non-perceptual, used in [37; 38; 39] \\ \hline STFT + MAE & Time-Frequency content, used in [28] \\ \hline \hline \end{tabular}
\end{table}
Table 5: Summary of the loss functions and their perceptual relevance.  
For the objective evaluation of the models compared in [28], three metrics were used: the MAE and the STFT loss described above and a perceptually informed loudness metric that uses the loudness algorithm from the ITU-R BS. 1770 Recommendation [57]. Both the causal and noncausal versions of the network achieved comparable performance but the noncausal variants performed slightly better in the time-domain. A listening test similar to MUSHRA was also carried out using the WebMUSHRA interface [58] to further validate the accuracy of the models. The results of this test indicated that a small difference was perceived among the models in comparison to the reference.  
Subjective evaluation was provided in [23] in the form of a MUSHRA test, carried out using the WebMUSHRA interface [58]. These tests showed the WaveNet3 to be the most perceptually accurate model of the HT5M amplifier, although this prediction could still be distinguished from the original amplifier, whereas the LSTM96 proved to be closest to the Mesa 5:50 in terms of subjective quality and most people could not tell the difference between the model and the target.  
For the work established in [PERSON]'s PhD thesis [35], a number of objective metrics as well as listening tests were presented. One listening test studies the number of parameters that can be reduced without loss of accuracy and the other is used to determine the threshold of a given metric above which the accuracy is no longer improved. An overview of the evaluation methods is presented in Table 6.