# A Neural Parametric Singing Synthesizer Modeling Timbre and Expression from Natural Songs +
## 5 Results
### Qualitative Results  
Results of the listening tests comparing different systems trained on natural singing are listed in Table 5. For systems trained on pseudo singing, results of the preference listening test are shown in Figure 7.  
\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline \multirow{2}{*}{**Voice (Language)**} & \multirow{2}{*}{**System**} & \multicolumn{2}{c}{**Harmonic**} & \multicolumn{2}{c}{**Aperiodic**} & \multicolumn{2}{c}{**V/UV**} \\ \cline{3-6}  & & **MCD (dB)** & **MS-LSD (\textless{}25 Hz/Full, dB)** & **BAPD (dB)** & **FPR (\%)** & **FNR (\%)** \\ \hline \multirow{2}{*}{M1 (Eng.)} & **HTS** & **4.95** & 11.09/22.44 & 2.72 & 16.10 & **2.46** \\  & **NPSS** & 5.14 & **7.79/8.18** & **2.44** & **11.22** & 2.65 \\ \hline \multirow{2}{*}{F1 (Eng.)} & **HTS** & **4.75** & 10.25/22.09 & 4.07 & **15.60** & 1.01 \\  & **NPSS** & 4.95 & **5.68/9.04** & **3.83** & 15.79 & **0.56** \\ \hline \multirow{2}{*}{F2 (Spa.)} & **HTS** & **4.88** & 11.07/22.28 & 3.62 & 1.85 & **2.21** \\  & **NPSS** & 5.27 & **8.02/6.59** & **3.38** & **1.40** & 3.20 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Quantitative results for the timbre models trained on pseudo singing, separated by voice/language. The IS16 system is excluded from the quantitative metrics because removing utterances from the dataset to use for testing would mean missing diphones would have to be replaced. The Sinsky-HMM and Sinsky-DNN systems were excluded from this comparison, as the only available models are trained on natural singing. The listed metrics are Mel-Cepstral Distortion (MCD) and Modulation Spectrum Log Spectral Distortion (MS-LSD) for harmonic features, Band Aperiodicity Distortion (BAPD) for aperiodic features, and False Positive Rate (FPR) and False Negative Rate (FNR) for voiced/unvoiced (V/UV) features.  
In the listening tests, NPSS is clearly ahead of competing systems. In the MOS test for systems trained on natural singing, NPSS is around a third between the second best rated system (Sinsy-HMM) and the reference. Here, it should be noted that the concatenative system, IS16, performs worst, showing that this kind of system is poorly suited for this kind of data. In contrast, the preference test for systems trained on pseudo singing, shows a strong preference for NPSS over the HTS system, and a moderate preference over the IS16 system, which was designed for this kind of data. The correlation between the qualitative results and the quantitative metrics based on the modulation spectrum indicate that this may be a metric with higher perceptual relevance than the framewise metrics such as MCD.  
In our experience NPSS, HMM and DNN systems all produce quite coherent timbres. The concatenative system in contrast tends to produce more discontinuous timbres, especially when using a dataset of natural singing, or other artifacts at concatenation boundaries, e.g., in fast singing or when phonetic segmentation is not perfect. We found NPSS to generally produce less static features over time, and less coloring of timbre. Compared to HMM and DNN systems, the autoregressive generation of NPSS seems to help in reproducing rapidly varying consonants, although these can occasionally sound better still in the concatenative system. In terms of expression, the HMM system produces very coherent behavior, which while perhaps a little less human, tends to generally sound quite pleasant. NPSS on the other hand, seems to be more varied, but this also means that results are sometimes better than other times. One notable quality of NPSS is that the framewise conditioning of timbre on pitch means that vibratos produce natural, synchronized modulations in both pitch and timbre (see, e.g., Figure 5), unlike in the other systems which condition on note pitch.