## 9 Evaluation Methods for RRE
### Perceptual Evaluation  
To assess the audio quality, listening tests have to be performed following an appropriate procedure. Many proposals for the perceptual evaluation of an audio system can be found in the literature [240; 241]. However, focusing on RRE and referring to the state-of-the-art, the perceptual assessment of RRE systems should adhere to the following standards:  
* ITU-R BS.1116-1 [242]: \"Methods for subjective assessment of small impairments in audio systems including multichannel sound systems\",
* ITU-R BS.1534-1 [243]: \"Method for the subjective assessment of intermediate quality level of coding systems\",
* ITU-R BS.1284-1 [244]: \"General methods for the subjective assessment of sound quality\".  
All these recommendations provide a description of the test methodology, test procedure, and statistical methods to elaborate the acquired data. However due to the broadness of this topic, the discussion will be focused only on the most relevant procedures that have been applied to RRE.  
The ITU-R BS.1284-1 recommendation provides a guide to the general assessment of perceived audio quality, and has been applied in [75; 145] for the assessment of RRE. It is worth noting that ITU-R BS.1284-1 is based on ITU-R BS.1116-1. According to the guidelines of [244], expert listeners should be preferred to _\"give a better and a quicker indication of the likely results in the long term.\"_ The subjective listening test is conceived as a comparison test, and the listeners should be instructed to provide a score using a seven-grade scale with a recommended resolution of 1 decimal place, as reported in [244]. The test is based on paired comparisons with references, and the score is given after listening to a repetition (four times consecutively) of the predetermined programme sequence. In the case of the assessment of an equalization procedure, the following sequence is considered in [75; 145]:  
1. reference sequence without equalization;
2. same sequence, equalized with one of the selected equalization techniques;
3. reference sequence without equalization (repeated);
4. same sequence, equalized with one of the selected equalization techniques (repeated).  
As recommended in [244], the stimuli should never exceeded 20 s in length, thus lengths were limited between 15 to 20 s. Moreover, care was taken in order to guarantee that the tested musical items did not appear to be interrupted. For each reference signal, the presentation order of the different equalization methods was randomized and the listener did not know which equalization methodology was under test. Following the recommendation, before the listening test, a training set was subministrated to the listeners. As reported in [145], in order to familiarize with the test procedure, the test materials and the test environment, the subjects had the possibility to listen to each audio item in all conditions under evaluation. While the ITU-R BS.1284-1 recommendation suggests several attributes for characterizing the perceived sound quality, in [75; 145] three attributes have been considered; i.e., \"transparency\" (all details of the performance can be clearly perceived), \"timbre\" (accurate portrayal of the different sound), \"main impression\" (the integrity of the total sound image and the interaction between other parameters). In order to test the room response equalizer using different spectral content, different music genres were considered as reference signals.  
Finally, the obtained results were processed to derive the mean values and the confidence intervals. A significance level of 0.05 was considered for computing the confidence intervals.