# Low-Order Spherical Harmonic HRTF Restoration Using a Neural Network Approach
## 3 Machine Learning HRTF Restoration
### Baseline Model  
As mentioned in Section 3, there are numerous ML (machine learning) models throughout the literature that have the potential for SH-interpolated HRTF restoration as they have shown some promising results in similar tasks in the visual domain. We aimed to find a model that has a simple architecture whilst being able to produce viable results. The reason for using a simple model is based on the consideration of the limited number of HRTF datasets, i.e., a simpler model is less likely to over-fit the training data. For comparison, all the models in this paper were trained with 500 epochs. The majority were trained with an NVIDIA Quadro P4000M GPU with 32 GB of computer RAM. For the models with extensive amount of data in Section 3.3.6, a NVIDIA GeForce RTX2080 Ti with 40 GB of computer RAM (random-access memory) was used. The proposed model can be seen as a simplified version of an inception network [52]. Separate models for left and right channels are trained individually, whilst the input of the model takes both channels to provide additional information which improves the results as shown in Table 4. Here, the overall mean is the average MSE loss across all training, validation and test results and the test mean is the average MSE loss of SADIE subject 20 and Bernschutz KU100 test data. The phase difference between two channels was handled with dual-band time alignment (TA), as discussed in Section 2. The model input size was \(2\times 256\) (left and right channels of the interpolated HRTFs with the length of 256 samples per channel) and the output was 256 (either left or right channel). \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline \multicolumn{6}{c}{**Comparison of Results Between Mono and Stereo Inputs (Lower Is Better)**} \\ \hline \hline
**Model** & **Overall Mean** & **Training** & **Validation** & **SADIE 20** & **Bernschutz** & **Test Mean** \\ \hline Baseline (mono) & 50.66 & 31.40 & 33.94 & 62.27 & 75.04 & 68.65 \\ \hline Baseline (stereo) & 44.46 & 28.17 & 30.17 & 52.34 & 67.17 & 59.75 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison between stereo input and mono input with the baseline model demonstrating that stereo input performs better than mono input. The model proposed here uses a combination of a convolutional auto-encoder (CAE) and a denoising auto-encoder (DAE) [53]. Preliminary test results showed that the DAE is better with the main contour of the frequency response (Figure 4) and CAE is better with the finer details (Figure 5). The combination of the two yields positive results. Similar results were also observed in image research [54]. The results from the convolutional CAE and DAE are concatenated and passed through a fully connected layer for the voting process. Figure 4: Restoration result example for ipsilateral HRTF response with DAE only (orange: model restored output, green: actual HRTF measurement, blue: SH-interpolated HRTF.) Figure 5: Restoration result example for ipsilateral HRTF response with CAE only (orange: model restored output, green: actual HRTF measurement, blue: SH-interpolated HRTF). The complete model is shown in Figure 6. Note that batch normalisation is performed after each convolution layer and transposed convolution layer, with the exception of the very last transposed convolution layer so the output of the CAE should have similar magnitude with the DAE. The models are built and trained with PyTorch [55; 56] using smooth L1 loss with Adam optimiser (learning rate: 0.000001, beta 1:0.9, beta 2:0.999). Different loss functions were compared and the results are shown in Table 5. Note that all the losses were calculated with the HRTF values in dB. The mean square error (MSE) loss, also known as the L2 loss, is given by the following equation:  
\[\text{L2 Loss}=\sum_{i=1}^{n}\left(y_{\text{label}}-y_{\text{predicted}} \right)^{2} \tag{9}\]  
where \(y_{\text{label}}\) is the target value of the output and \(y_{\text{predicted}}\) is the prediction from the model.