<figure><img src="image_1.png" /></figure>

HAL
open science

# A neural voice transformation framework for modification of pitch and intensity

Frederik Bous

<figure><img src="image_2.png" /></figure>

## To cite this version:

Frederik Bous. A neural voice transformation framework for modification of pitch and intensity. Signal and Image Processing. Sorbonne Université, 2023. English. NNT: 2023SORUS382. tel-04353490

HAL Id: tel-04353490

https://theses.hal.science/tel-04353490

Submitted on 19 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
ircam
Centre
Pompidou

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

SORBONNE
UNIVERSITÉ

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

MINISTÈRE
DE LA CULTURE

Liberté
Égalité
Fraternité

THÈSE DE DOCTORAT DE SORBONNE UNIVERSITÉ
Spécialité Traitement du signal

ED130 - École doctorale Informatique, Télécommunications et Électronique (Paris)

Sciences et Technologie de la Musique et du Son (UMR 9912 STMS)

Institut de Recherche et de Coordination Acoustique Musique (IRCAM)

Équipe Analyse/Synthèse des Sons.

A neural voice transformation framework
for modification of pitch and intensity

Présenté par

Frederik Bous

Dirigée par Axel Roebel

Soutenue le 21 septembre 2023 devant le jury composé de :

<table><tr><td>Thierry Dutoit</td><td>Rapporteur</td></tr><tr><td>Yannis Stylianou</td><td>Rapporteur</td></tr><tr><td>Christophe d'Alessandro*</td><td>Examinateur</td></tr><tr><td>Jordi Bonada</td><td>Examinateur</td></tr><tr><td>Natalie Henrich</td><td>Examinateur</td></tr></table>

*Président du jury
UMR 9912 STMS

IRCAM, Sorbonne Université, CNRS, Ministère de la Culture

Équipe Analyse/Synthèse des Sons

1, place Igor Stravinsky
75004 Paris, FRANCE

PhD thesis presented by Frederik Bous

Thesis supervision by Axel Roebel

Version 1.1

Screen edition

Compiled on 2023-12-18

This thesis is accompanied by additional media, which can be downloaded from http://thesis.fnab.xyz.

<figure><img src="image_7.png" /></figure>

Copyright © 2023 Frederik Bous

<figure><img src="image_8.png" /></figure>

orcid.org/0000-0002-7477-7600

<figure><img src="image_9.png" /></figure>

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. The full text of the licence can be found under http://creativecommons.org/licenses/by-nc-nd/4.0/.
Abstract

Human voice has been a great source of fascination and an object of research for over
100 years. During that time numerous technologies have sprouted around the voice,
such as the vocoder, which provides a parametric representation of the voice, com-
monly used for voice transformation. From this tradition, the limitations of purely
signal processing based approaches are evident: To create meaningful transforma-
tions the codependencies between different voice properties have to be understood
well and modelled precisely. Modelling these correlations with heuristics obtained
by empiric studies is not sufficient to create natural results. It is necessary to ex-
tract information about the voice systematically and use this information during
the transformation process automatically. Recent advances in computer hardware
permit this systematic analysis of data by means of machine learning. This thesis
thus uses machine learning to create a neural voice transformation framework.

The proposed neural voice transformation framework works in two stages: First a neural vocoder allows mapping between a raw audio and a mel-spectrogram representation of voice signals. Secondly, an auto-encoder with information bottleneck allows disentangling various voice properties from the remaining information. The auto-encoder allows changing one voice property while automatically adjusting the remaining voice properties. In the first part of this thesis, we discuss different approaches to neural vocoding and reason why the mel-spectrogram is better suited for neural voice transformations than conventional parametric vocoder spaces.

In the second part we discuss the information bottleneck auto-encoder. The auto-
encoder creates a latent code that is independent of its conditional input. Using
the latent code the synthesizer can perform the transformation by combining the
original latent code with a modified parameter curve. We transform the voice using
two control parameters: the fundamental frequency and the voice level. Transform-
ation of the fundamental frequency is an objective with a long history. Using the
fundamental frequency allows us to compare our approach to existing techniques
and study how the auto-encoder models the dependency on other properties in a
well known environment. For the voice level, we face the problem that annotations
hardly exist. Therefore, first we provide a new estimation technique for voice level
in large voice databases, and subsequently use the voice level annotations to train a
bottleneck auto-encoder that allows changing the voice level.
Abstract

La voix humaine est une grande source de fascination et un objet de recherche depuis plus de 100 ans. Pendant ce temps, de nombreuses technologies ont germées autour de la voix, comme le vocodeur, qui fournit une représentation paramétrique de la voix, couramment utilisée pour la transformation de la voix. Dans cette tradition, les limites des approches basées uniquement sur le traitement du signal sont évidentes : Pour créer des transformations cohérentes, les dépendances entre les différentes propriétés vocales doivent être bien comprises et modélisées avec précision. Modéliser ces corrélations avec des heuristiques obtenues par des études empiriques ne suffit pas à créer des résultats naturels. Il est nécessaire d'extraire systématiquement des informations sur la voix et d'utiliser automatiquement ces informations lors du processus de transformation. Les progrès récents de la puissance de calcul permettent cette analyse systématique des données au moyen de l'apprentissage automatique. Cette thèse utilise donc l'apprentissage automatique pour créer un système neuronal de transformation de la voix.

Le système neuronal de transformation de la voix, présenté ici, fonctionne en deux étapes : Tout d'abord, un vocodeur neuronal permet d'établir une correspondance entre la forme d'onde et une représentation mel-spectrogramme des signaux vocaux. Ensuite, un auto-encodeur avec un goulot d'étranglement permet de démêler différentes propriétés de la voix du reste de l'information. L'auto-encodeur permet de modifier une propriété de la voix tout en ajustant automatiquement d'autres caractéristiques de façon à en conserver le réalisme. Dans la première partie de cette thèse, nous comparons différentes approches du vocodage neuronal et nous expliquons pourquoi la représentation mel-spectrogramme est plus adapté pour la transformation neuronale de la voix plutôt que les espaces paramétriques du vocodeur conventionnels.

Dans la deuxième partie, nous présentons l'auto-encodeur avec goulot d'étranglement de l'information. L'auto-encodeur crée un code latent indépendant du conditionnement en entrée. En utilisant ce code latent, le synthétiseur peut effectuer la transformation en combinant le code latent original avec une courbe de paramètres modifiée. Nous transformons la voix en utilisant deux paramètres de contrôle : la fréquence fondamentale et le niveau sonore vocal. La transformation de la fréquence fondamentale est un problème qui a longtemps été abordé : Notre approche est comparable aux techniques existantes puisqu'elles utilisent la fréquence fondamentale comme paramètre. Cela nous permet également d'étudier comment l'auto-encodeur modélise les dépendances entre la fréquence fondamentale et d'autres propriétés de la voix dans un environnement connu. Quant au niveau sonore vocal, nous sommes confrontés au problème de la rareté des annotations. Par conséquent, nous proposons d'abord une nouvelle technique d'estimation du niveau sonore vocal dans
de grandes bases de données de voix ; puis nous utilisons ces annotations pour entraîner un auto-encodeur avec goulot d'étranglement permettant de modifier le niveau sonore vocal.
# Acknowledgement

Though our own vanity sometimes lets us forget this, but we are nothing without the help of others. This thesis would not have been possible if it was not due to the help of my advisors, colleagues, friends, and family. Thus, commanded by deepest gratitude, it is necessary to mention a few names to acknowledge their contribution and attest my indebtedness.

My sincerest thank goes to my thesis advisor **Axel Roebel**. The amount of advice and support I received from Axel, professional and personal, is innumerable and invaluable. The amount of attention I got from Axel during my thesis was extraordinary, which is not a matter of course for thesis supervisors. I am honoured to have been a student of Axel Roebel.

During most of my thesis I had the pleasure of working together with the artist **Judith Deschamps**. After Axel, Judith was the second most influential person on my thesis. Our collaboration was a great motivator of my work and allowed me to explore the abilities and applications of my research in an interesting project. I am happy to have had Judith at my side as a collaborator and as a friend.

It was a great pleasure to do my PhD at IRCAM thanks to all the people that make this place magical. In particular, I am grateful to **Nicolas Obin** for being persistent, where I had already given up and to **Hugues Vinet** for his support allowing me to finish my PhD without worries. I'm grateful to **Luc Ardaillon** for his preliminary work I could build on, and I'm glad I could work together with Luc to extend some of his work. Thank you to **Darick Lean**, who helped me organize all the databases, thank you to everyone, who helped me by proofreading my manuscripts, **Emily Graber**, **Théodor Lemerle**, **Lenny Renault**, **Léane Salais**, and **Daniel Wolf**, and thank you to **Yann Teytaut** and **Clément Le Moine Veillon** for sharing their LaTeX source. A big thank you to **Pierre Guillot** for helping me get my software CIRCE ready for production which was a hell of a hassle and would not have been possible without Pierre.

My work has been greatly influenced by the artistic work of the composers **Aïda Shirazi**, **Sachie Kobayashi**, and **Omer Barash**, whom I am thankful for using my research in their compositions, helping me develop the tools for a wider audience. Likewise, collaborating with **António Breitenfeld Sá-Dantas** on the Farinelli project was a great pleasure and a source of many ideas. Furthermore, I am grateful to all the singers who gave their voice to the Farinelli project, allowing us to create a new, weird, and interesting voice.

Finally, I would like to acknowledge my friends and family who have been a great support to me during these years. Thank you to **Sascha Knüttel**, **Kay Wohlfarth**, and **Christoph von Erffa** for standing by me when things got rough, and to my family that I can rely on and that is always there for me when I need them.
# Contents

<table><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td>1. Introduction</td><td></td><td>1</td></tr><tr><td>1.1. Objectives</td><td></td><td>1</td></tr><tr><td>1.2. Context</td><td></td><td>2</td></tr><tr><td>1.2.1. Artistic residency Farinelli</td><td></td><td>2</td></tr><tr><td>1.2.2. ANR Project ARS</td><td></td><td>3</td></tr><tr><td>1.2.3. IRCAM Singing Synthesizer</td><td></td><td>3</td></tr><tr><td>1.3. Main contributions</td><td></td><td>3</td></tr><tr><td>1.4. Outline</td><td></td><td>5</td></tr><tr><td>2. History and fundamentals</td><td></td><td>6</td></tr><tr><td>2.1. A short history of voice synthesis and transformation</td><td></td><td>6</td></tr><tr><td>2.2. Human perception of sound</td><td></td><td>9</td></tr><tr><td>2.2.1. Perception of pitch</td><td></td><td>11</td></tr><tr><td>2.2.2. Pitch scales</td><td></td><td>12</td></tr><tr><td>2.2.3. Loudness</td><td></td><td>14</td></tr><tr><td>2.3. Fundamentals of the voice</td><td></td><td>15</td></tr><tr><td>2.3.1. The theory of speech production</td><td></td><td>16</td></tr><tr><td>2.3.2. Glottal flow models</td><td></td><td>18</td></tr><tr><td>2.4. Neural networks</td><td></td><td>21</td></tr><tr><td>2.4.1. Neural arithmetic on voice signals</td><td></td><td>24</td></tr><tr><td>2.4.2. Common network architecture types for voice</td><td></td><td>27</td></tr><tr><td>2.5. Datasets</td><td></td><td>30</td></tr><tr><td>2.5.1. Speech datasets</td><td></td><td>31</td></tr><tr><td>2.5.2. Singing datasets</td><td></td><td>32</td></tr><tr><td>2.5.3. Combining datasets</td><td></td><td>34</td></tr><tr><td>2.5.4. Dataset analysis</td><td></td><td>36</td></tr><tr><td>2.6. Perceptive tests</td><td></td><td>39</td></tr><tr><td>2.6.1. Conception</td><td></td><td>39</td></tr><tr><td>2.6.2. Procedure</td><td></td><td>41</td></tr><tr><td>2.6.3. Evaluation</td><td></td><td>42</td></tr><tr><td>2.7. Conclusion</td><td></td><td>46</td></tr><tr><td>3. State of the art</td><td></td><td>47</td></tr><tr><td>3.1. Analysis of voice</td><td></td><td>47</td></tr><tr><td>3.1.1. Fundamental frequency</td><td></td><td>48</td></tr><tr><td>3.1.2. Glottal pulse signal</td><td></td><td>48</td></tr><tr><td>3.1.3. Voice timbre</td><td></td><td>50</td></tr><tr><td>3.1.4. Noise component</td><td></td><td>50</td></tr></tbody></table>
<table><tbody><tr><td>3.2.</td><td>Vocoders</td><td>51</td></tr><tr><td>3.2.1.</td><td>Pitch synchronous overlap and add</td><td>52</td></tr><tr><td>3.2.2.</td><td>Non-parametric vocoders</td><td>52</td></tr><tr><td>3.2.3.</td><td>Source-filter parametric vocoder</td><td>53</td></tr><tr><td>3.2.4.</td><td>Neural vocoder</td><td>53</td></tr><tr><td>3.2.5.</td><td>Neural source-filter vocoders</td><td>53</td></tr><tr><td>3.2.6.</td><td>Neural vocoder using the mel-spectrogram</td><td>54</td></tr><tr><td>3.3.</td><td>Synthesis</td><td>55</td></tr><tr><td>3.3.1.</td><td>Concatenative synthesis</td><td>55</td></tr><tr><td>3.3.2.</td><td>Statistical parametric speech (singing) synthesis</td><td>56</td></tr><tr><td>3.3.3.</td><td>Neural parametric synthesis</td><td>56</td></tr><tr><td>3.3.4.</td><td>Neural end-to-end synthesis</td><td>56</td></tr><tr><td>3.4.</td><td>Transformation</td><td>57</td></tr><tr><td>3.4.1.</td><td>Modelling dependencies</td><td>57</td></tr><tr><td>3.4.2.</td><td>Disentanglement of parameters</td><td>58</td></tr><tr><td>3.4.3.</td><td>Higher level voice conversion</td><td>60</td></tr><tr><td>3.5.</td><td>Conclusion</td><td>61</td></tr><tr><td>4.</td><td>Glottal closure instance extraction</td><td>63</td></tr><tr><td>4.1.</td><td>Introduction</td><td>63</td></tr><tr><td>4.1.1.</td><td>Glottal closure instant</td><td>64</td></tr><tr><td>4.1.2.</td><td>Overview</td><td>64</td></tr><tr><td>4.2.</td><td>Approach</td><td>65</td></tr><tr><td>4.2.1.</td><td>The synthetic database</td><td>65</td></tr><tr><td>4.2.2.</td><td>The analysis-synthesis framework</td><td>65</td></tr><tr><td>4.2.3.</td><td>Loss functions</td><td>66</td></tr><tr><td>4.2.4.</td><td>Training procedure</td><td>69</td></tr><tr><td>4.3.</td><td>Evaluation</td><td>69</td></tr><tr><td>4.3.1.</td><td>Test setup</td><td>70</td></tr><tr><td>4.3.2.</td><td>Evaluation metrics</td><td>70</td></tr><tr><td>4.4.</td><td>Results</td><td>71</td></tr><tr><td>4.5.</td><td>Conclusion</td><td>72</td></tr><tr><td>5.</td><td>Neural vocoder</td><td>73</td></tr><tr><td>5.1.</td><td>Desired properties of a vocoder</td><td>73</td></tr><tr><td>5.1.1.</td><td>Mathematical model</td><td>74</td></tr><tr><td>5.1.2.</td><td>Desired properties of the parametric space</td><td>75</td></tr><tr><td>5.1.3.</td><td>Practical requirements on the vocoder implementation</td><td>77</td></tr><tr><td>5.2.</td><td>Parametric spaces for voice</td><td>77</td></tr><tr><td>5.2.1.</td><td>Time-domain signal and end-to-end processing</td><td>78</td></tr><tr><td>5.2.2.</td><td>Classical vocoder parameters</td><td>78</td></tr><tr><td>5.2.3.</td><td>Mel-spectrogram</td><td>80</td></tr><tr><td>5.2.4.</td><td>Conclusion</td><td>83</td></tr><tr><td>5.3.</td><td>Neural vocoding using the mel-spectrogram</td><td>83</td></tr></tbody></table>
<table>
  <tbody>
    <tr>
      <td>5.4.</td>
      <td>Multi-Band Excited WaveNet</td>
      <td>85</td>
    </tr>
    <tr>
      <td>5.4.1.</td>
      <td>Network architecture</td>
      <td>85</td>
    </tr>
    <tr>
      <td>5.4.2.</td>
      <td>Training</td>
      <td>87</td>
    </tr>
    <tr>
      <td>5.4.3.</td>
      <td>Discussion</td>
      <td>88</td>
    </tr>
    <tr>
      <td>5.5.</td>
      <td>Conclusion</td>
      <td>90</td>
    </tr>
    <tr>
      <td>6.</td>
      <td>Bottleneck auto-encoder</td>
      <td>91</td>
    </tr>
    <tr>
      <td>6.1.</td>
      <td>Introduction</td>
      <td>92</td>
    </tr>
    <tr>
      <td>6.1.1.</td>
      <td>Applications</td>
      <td>92</td>
    </tr>
    <tr>
      <td>6.1.2.</td>
      <td>Challenges of pitch transformation</td>
      <td>93</td>
    </tr>
    <tr>
      <td>6.1.3.</td>
      <td>Mathematical description of disentanglement</td>
      <td>93</td>
    </tr>
    <tr>
      <td>6.1.4.</td>
      <td>Goals of this chapter</td>
      <td>95</td>
    </tr>
    <tr>
      <td>6.1.5.</td>
      <td>Outline</td>
      <td>96</td>
    </tr>
    <tr>
      <td>6.2.</td>
      <td>Methodology</td>
      <td>96</td>
    </tr>
    <tr>
      <td>6.2.1.</td>
      <td>Proposed model</td>
      <td>96</td>
    </tr>
    <tr>
      <td>6.2.2.</td>
      <td>Datasets</td>
      <td>98</td>
    </tr>
    <tr>
      <td>6.2.3.</td>
      <td>Nomenclature</td>
      <td>99</td>
    </tr>
    <tr>
      <td>6.2.4.</td>
      <td>Experimental setup</td>
      <td>99</td>
    </tr>
    <tr>
      <td>6.3.</td>
      <td>Experimental results</td>
      <td>101</td>
    </tr>
    <tr>
      <td>6.3.1.</td>
      <td>f<sub>0</sub> accuracy</td>
      <td>101</td>
    </tr>
    <tr>
      <td>6.3.2.</td>
      <td>Synthesis quality</td>
      <td>104</td>
    </tr>
    <tr>
      <td>6.3.3.</td>
      <td>Investigation of the latent code</td>
      <td>107</td>
    </tr>
    <tr>
      <td>6.4.</td>
      <td>Conclusion</td>
      <td>111</td>
    </tr>
    <tr>
      <td>6.5.</td>
      <td>Future work</td>
      <td>111</td>
    </tr>
    <tr>
      <td>7.</td>
      <td>Transformation of perceived voice level</td>
      <td>113</td>
    </tr>
    <tr>
      <td>7.1.</td>
      <td>Introduction</td>
      <td>113</td>
    </tr>
    <tr>
      <td>7.1.1.</td>
      <td>Terminology</td>
      <td>113</td>
    </tr>
    <tr>
      <td>7.1.2.</td>
      <td>Problem formulation</td>
      <td>115</td>
    </tr>
    <tr>
      <td>7.1.3.</td>
      <td>Outline</td>
      <td>115</td>
    </tr>
    <tr>
      <td>7.2.</td>
      <td>Extracting voice level from audio recordings</td>
      <td>116</td>
    </tr>
    <tr>
      <td>7.2.1.</td>
      <td>Objective</td>
      <td>116</td>
    </tr>
    <tr>
      <td>7.2.2.</td>
      <td>Learnt recording factor</td>
      <td>117</td>
    </tr>
    <tr>
      <td>7.2.3.</td>
      <td>Adaptive recording factor</td>
      <td>118</td>
    </tr>
    <tr>
      <td>7.3.</td>
      <td>Proposed voice level transformations</td>
      <td>120</td>
    </tr>
    <tr>
      <td>7.4.</td>
      <td>Experiments</td>
      <td>121</td>
    </tr>
    <tr>
      <td>7.4.1.</td>
      <td>Data</td>
      <td>121</td>
    </tr>
    <tr>
      <td>7.4.2.</td>
      <td>Target curves</td>
      <td>121</td>
    </tr>
    <tr>
      <td>7.4.3.</td>
      <td>Architecture</td>
      <td>122</td>
    </tr>
    <tr>
      <td>7.4.4.</td>
      <td>Auto-encoders</td>
      <td>122</td>
    </tr>
    <tr>
      <td>7.4.5.</td>
      <td>Evaluation methods</td>
      <td>122</td>
    </tr>
    <tr>
      <td>7.5.</td>
      <td>Results</td>
      <td>123</td>
    </tr>
    <tr>
      <td>7.5.1.</td>
      <td>Classification of dynamic</td>
      <td>123</td>
    </tr>
    <tr>
      <td>7.5.2.</td>
      <td>Accuracies</td>
      <td>125</td>
    </tr>
    <tr>
      <td>7.5.3.</td>
      <td>Perceptive test</td>
      <td>125</td>
    </tr>
  </tbody>
</table>
<table><tbody><tr><td>7.6. Conclusions</td><td>127</td></tr><tr><td><b>8. Applications</b></td><td><b>129</b></td></tr><tr><td>8.1. Farinelli singing voice</td><td>129</td></tr><tr><td>8.1.1. The castrato voice of Farinelli</td><td>130</td></tr><tr><td>8.1.2. Related work</td><td>130</td></tr><tr><td>8.1.3. Musical material</td><td>131</td></tr><tr><td>8.1.4. Creating a hybrid voice</td><td>132</td></tr><tr><td>8.1.5. Improvisations over Quell'usignolo</td><td>134</td></tr><tr><td>8.1.6. Observations and difficulties</td><td>136</td></tr><tr><td>8.1.7. Conclusion</td><td>137</td></tr><tr><td>8.2. Musical compositions with CIRCE</td><td>138</td></tr><tr><td>8.2.1. Aïda Shirazi: Né entre corps (2022)</td><td>138</td></tr><tr><td>8.2.2. Sachie Kobayashi: Day 0 - trans-instrumentalism (2022)</td><td>139</td></tr><tr><td>8.2.3. Omer Barash: G.N.Z. (2023)</td><td>141</td></tr><tr><td><b>9. Conclusion</b></td><td><b>143</b></td></tr><tr><td>9.1. Recapitulation</td><td>143</td></tr><tr><td>9.1.1. Neural vocoders</td><td>143</td></tr><tr><td>9.1.2. Bottleneck auto-encoder</td><td>144</td></tr><tr><td>9.2. The neural voice transformation framework</td><td>144</td></tr><tr><td>9.3. Applications</td><td>145</td></tr><tr><td>9.3.1. Audio post-production</td><td>146</td></tr><tr><td>9.3.2. Creation of new, hybrid-voices</td><td>146</td></tr><tr><td>9.3.3. Obfuscation and anonymization</td><td>146</td></tr><tr><td>9.3.4. Singing voice synthesis</td><td>146</td></tr><tr><td>9.3.5. Speech enhancement and humanization</td><td>147</td></tr><tr><td>9.4. Limitations and Future work</td><td>147</td></tr><tr><td>9.4.1. Issues with synthesis quality</td><td>148</td></tr><tr><td>9.4.2. Controllable voice properties</td><td>152</td></tr><tr><td>9.4.3. Understanding the latent space of the auto-encoder</td><td>155</td></tr><tr><td><b>List of figures</b></td><td><b>157</b></td></tr><tr><td><b>List of tables</b></td><td><b>158</b></td></tr><tr><td><b>A. List of model configurations and their layers</b></td><td><b>159</b></td></tr><tr><td>A.1. Models from Chapter 4: Pulse detection</td><td>160</td></tr><tr><td>A.2. Models from Chapter 6: Bottleneck auto encoder</td><td>161</td></tr><tr><td>A.3. Models from Chapter 7: Voice level transformation</td><td>162</td></tr><tr><td><b>Glossary</b></td><td><b>163</b></td></tr><tr><td><b>Bibliography</b></td><td><b>170</b></td></tr></tbody></table>
# 1. Introduction

This thesis is the result of the last three years of my research at IRCAM. It extends the research from my master thesis (Bous, 2019) where we were investigating means to synthesize spectral envelopes from the fundamental frequency and phonetic transcription. The limitations of previous methods of modelling the voice were apparent already then: The classical parameters used to describe voice, spectral envelope and fundamental frequency, are highly dependent on each other. Changing the fundamental frequency without changing the spectral envelope creates unnatural results and previous attempts to adjust the spectral envelope to changes of the fundamental frequency were unsatisfactory. The voice has been an object of research for a long time and relationships between different aspects of voice, like the vocal tract filter and the fundamental frequency are well known. Nevertheless, this knowledge is rather approximative than precise: While we know from the research on voice production mechanism that different aspects of the voice interact, the theories remain approximative and do not provide mathematical models that are precise enough for natural voice synthesis. To obtain mathematical descriptions other methods are required.

## 1.1. Objectives

Machine learning provides a tool to systematically and automatically obtain math-
ematical models from data. Technological advance in computational power has
caught up with the requirements for large-scale deep learning, which makes it a
suitable tool to formalize the relationships of different aspects in the voice. This
leads to the central research objective of this thesis:

> We want to use deep learning to transform interesting voice
> properties in voice recordings with high audio quality and
> natural sounding voice.

The objective of this work is to develop a neural voice transformation framework that allows modelling the relationships between different voice properties. The neural voice transformation framework should be able to learn these relationships from available voice recordings. When changing one voice property, the neural voice transformation framework should automatically adapt the remaining voice properties such that the speech remains natural. Notably, we want to address the following problems:

**Transformable properties** In the past, Voice transformation has largely been ori-
ented around transformations of pitch. Nowadays, we observe that abstract
properties like voice identity are popular in the voice conversion community. In this thesis, the goal is to extend the canon of signal related transformable properties. While in this research, we are mainly occupied with pitch, we will do this by developing a transformation framework that can be applied to other properties as well, and we will use the voice level as another such property.

**Transformation range** In traditional pitch transformation methods the range of possible transformations is limited. If the transposition is too large the result sounds unnatural. We want to extend the amount of transposition that keeps producing natural voice. This requires automatic adjustments in either voice identity, vocal technique or both. Similar reasoning applies to other voice properties.

While pioneer work has already established solid ground for voice synthesis us-
ing deep learning, the area of transformation has still large unexplored territories.
Only a handful of topics of voice transformation are covered by neural signal pro-
cessing, notably speaker identity transformation and some speaking style tokens.
The present work addresses transformation of fundamental signal parameters, i.e.
the pitch and intensity. We will be concerned with human voice in general but
give special attention to singing voice. The treatment of speech is addressed to use
synergies between speech and singing voice. Since training data for singing voice is
scarce, we hope to use additional information about human voice from speech to
improve the quality of our models and would obtain models that work on speech
and singing as well.

**1.2. Context**

During the last three years the research around this thesis was part of three larger projects: the artistic residency by Judith Deschamps, the ANR project ARS, and the IRCAM Singing Synthesizer. While this thesis was originally motivated by the shortcomings of existing methods in voice processing, the surrounding projects have given this thesis a purpose and thus have influenced the direction of this research.

**1.2.1. Artistic residency Farinelli**

During the last two years I had the pleasure of working together closely with Judith Deschamps in the context of her artistic research residency at IRCAM. The subject of her artistic residency was to recreate the voice of 18th century castrato singer Carlo Maria Michelangelo Nicola Broschi, known as Farinelli. The research behind this thesis is closely tied to the Farinelli project as a lot of the research questions were focused around developing the technologies that would allow us to recreate the voice of Farinelli. The recreated voice was a central device in the documentary film *La Mue, un conte vidéographique* (Deschamps, 2022b) and was part of an installation at Casino Luxembourg in early 2023. The project will be discussed in more detail in Chapter 8.
## 1.2.2. ANR Project ARS

The *Agence nationale de la recherche* (French national research agency, ANR) provides research grants for various research topics. Just before the beginning of the thesis such a grant was acquired in the context of the project *Analyse et transformation du style de chant* (Analysis and transformation of singing style, ARS). As the name suggests, the objective of the project is to investigate singing style and find new means to analyse and transform it. The ARS project is an opportunity to work together with musicologists and use voice transformation to investigate the effect of the changed parameter on the style. Furthermore, the collaboration with the private sector has motivated the creation of a robust product with high expectations to synthesis quality. The ARS project is a collaboration between the *STMS lab*, the research institutes *Institut Jean Le Rond d'Alembert* and *Passages Arts & Littératures* (XX-XXI), and the company *Flux Software Engineering*.

## 1.2.3. IRCAM Singing Synthesizer

The IRCAM Singing Synthesizer (ISiS) is a parametric concatenative synthesizer. To synthesize singing voice from the text and score, first the phonetic sequence is synthesized by concatenating parameter contours of the corresponding diphones using the parameters of a parametric vocoder. Rhythm and melody are imposed by time-stretch and pitch-shift using the same parametric vocoder. This thesis was in part motivated by the idea to improve IRCAM Singing Synthesizer by using the parametric space of a parametric vocoder and use the neural transformation algorithms for the time-stretch and pitch-shift operations. Furthermore, the singing voice could be customized by other voice transformations, in particular the voice level changes that were developed in this thesis. While this thesis is not directly involved in the development of the IRCAM Singing Synthesizer, it provides one very interesting application that we want to address in the future.

# 1.3. Main contributions

As scientific research is a collective effort it is difficult to disentangle individual
contributions from the scientific results. Therefore, not all ideas presented here
are my own ideas. Attribution is given where necessary either as citations or a men-
tioning of the original author of an idea. My main contributions that are presented
here in the thesis are the application of the bottleneck auto-encoder to the funda-
mental frequency (Chapter 6) and the development of the voice level estimation
model (Chapter 7). Furthermore, I contributed in the development of the neural
vocoder presented in Chapter 5 through fundamental groundwork, in particular the
analysis-synthesis framework for glottal closure instant detection from Chapter 4.
I used the results of this research to write the software application CIRCE<fnref n="1" /> which
provides a graphical interface for the neural pitch and voice level transformations
developed in this thesis.

<footnote n="1">1. https://forum.ircam.fr/projects/detail/circe/</footnote>
## Contributions of Chapter 4: Glottal closure instant detection

The results of Chapter 4 were published in Bous, Ardaillon, et al. (2020). Chapter 4 builds on existing work of Luc Ardaillon and extends the glottal closure instant detection of Ardaillon and Roebel (2020). A novel analysis-synthesis framework is introduced allowing to train the glottal pulse estimator in parallel with a neural vocoder.

## Contributions of Chapter 5: Neural vocoder

A significant fraction of my research was involved in developing a neural vocoder using the mel-spectrogram as its parametric space. While I contributed to the theoretical considerations and performed my own experiments, the Multi-Band Excited WaveNet presented at the end of Chapter 5 cannot be seen as an independent contribution of this thesis.

## Contributions of Chapter 6: Bottleneck auto-encoder

The results of Chapter 6 were published in Bous and Roebel (2022). In that chapter we propose the first deep-learning based algorithm explicitly focusing on transformation of fundamental frequency $f_0$ of singing and spoken voice using the explicit $f_0$ values (previous work like (Qian, Jin, et al., 2020) use $f_0$-contours normalized by speaker and only focus on speech). The proposed system is able to accurately follow the desired $f_0$ within typical ranges of human voice and even outside the range of the source voice, while surpassing the audio quality of classical vocoders. We study the proposed bottleneck auto-encoder with regard to several aspects:

1. The quality of the pitch transformation is evaluated by analysing the $f_0$ error on the test set in Section 6.3.1 and by a perceptive test which we present in Section 6.3.2.

2. Since the proposed system uses an auto-encoder with information bottleneck like (Qian, Yang Zhang, Chang, X. Yang, et al., 2019), we perform a thorough study on how disentanglement depends on different hyperparameters. We will investigate the bottleneck size, the size of the neural network, dataset size and the voice type (speech vs. singing voice). The results of this analysis are given in Section 6.3.1.

3. Finally, we carry out a visual analysis on the latent code of the auto-encoder where we find phonetic content to be represented as periodic discontinuous patterns by the auto-encoder.

## Contributions of Chapter 7: Voice level

The results of Chapter 7 were published in Bous and Roebel (2023). In Chapter 7 we use the bottleneck auto-encoder to transform the perceived voice level of singing voice recordings. None of the available singing voice recordings are annotated with
voice level. Thus, we provide a method to train a neural network to extract voice level without the need for explicitly annotated voice level. Voice level is given with regard to a fixed reference point where the signal power (or loudness) would be theoretically identical to the (perceived) voice level. The voice level estimation presented here, can be calibrated using the reference point of any recording condition present in the singing database. The contributions are the following:

1. We present an unsupervised algorithm that allows training a deep neural network to predict the voice level without the need to create annotated data.

2. We use this voice level measure to train the bottleneck auto-encoder from Chapter 6 to transform the voice level in recordings of singing voice.

## 1.4. Outline

The remainder of this thesis is structured as follows: We begin this thesis with an introduction to the fundamentals related to the topics of voice transformation in Chapter 2. There, we will discuss topics that are not necessarily state of the art but nevertheless important in the context of this thesis. This includes a historic overview (Section 2.1), an introduction to hearing (Section 2.2), speech production (Section 2.3), and neural networks (Section 2.4), followed by a discussion about the datasets (Section 2.5), and the evaluation methods (Section 2.6). We continue with a review of the state of the art in Chapter 3 where we will discuss work closely related to pitch transformation. In particular, we will discuss voice analysis methods (Section 3.1), vocoders (Section 3.2), speech- and singing synthesis (Section 3.3), and voice transformation (Section 3.4).

The main part of this thesis consist of Chapters 4 to 7 and includes the principal contributions of this thesis. Chapter 4 describes the initial work on neural vocoding resulting in an improvement of glottal closure instant detection using a neural vocoder. In Chapter 5 we will be discussing the neural vocoder for the purpose of voice transformation. In that chapter we will motivate the choices that led to the development of the neural vocoder that provides the parametric space on which we will implement the voice transformations in the subsequent chapters. Chapter 6 describes the bottleneck auto-encoder, which is used for pitch- and vocal effort transformation. In this chapter we discuss the general working principle of the auto-encoder and evaluate it on pitch transformation. In Chapter 7 we develop a method to estimate the voice level in singing voice from the timbre using an unsupervised estimator. The voice level measure is used to condition the bottleneck auto-encoder to allow transformation of voice level.

The last part consists of Chapters 8 and 9 and puts the presented material into perspective. In Chapter 8 we present a few applications that testify about the usefulness of the technology developed in this thesis. We will discuss the residency of Judith Deschamps and a few musical compositions. Chapter 9 is a recapitulation of the thesis, puts the presented material into a larger context and discusses possible future work.
## 2. History and fundamentals

Human voice has been subject of study for a long time. This thesis builds on research that has been going on for over 100 years. Throughout this long period, different technologies were utilized and had their popularity at one time or another. We see voice technology being realized through mechanics, electrical circuitry, digital signal processing and recently through neural networks. While this work heavily uses deep learning to achieve the goals we set out for us, the other technologies have not vanished from the domain of voice processing. Rather, they have become such common visitors that we grew used to their presence and do not notice them every time any more. In this chapter we introduce the main characters of this thesis. We pay tribute to the extensive work that has allowed my own work to be produced and provide a context within the research of this thesis resides.

### 2.1. A short history of voice synthesis and transformation

The first documented systematic investigation on speech production is the work by Kempelen (1791) in the late 18th century. Kempelen was an infamous inventor and engineer, at his time most known for his chess-playing machine, an apparatus that was able to play chess but which was later exposed as a hoax; the central logic unit was an actual human chess player concealed from the spectators (Linggard, 1985, pp. 4-8). While the conceptualization and construction of the chess-playing hoax took six months to finish, Kempelen's next goal was much more ambitious. For the next 20 years he would work fiercely on his speaking machine, this time for real. This invention-driven research yielded some groundbreaking insights into the speech production mechanism. Finally, Kempelen built his speaking machine using a bagpipe reed for the glottis to produce the excitation signal, a bellow for the lungs to control the strength of the sound and different resonant pipes to form different phonemes.

Kempelen's apparatus was reproduced several times during the 19th century by Farber, Willis, Wheatstone, and Scripture (Dudley, 1939). Others succeeded to imitate voice sounds by other means, like Helmholtz, using tuning forks, or Miller, using organ pipes (Stewart, 1922). One noteworthy invention was Joseph Farber's *Euphonia*, maybe the first singing synthesizer ever to be built (Blaauw, 2022, Sec. 2.1).

In the 20th century electronic devices started to replace mechanical ones, and we see voice synthesis being implemented with electronic circuits. The first such device was the speech synthesizer of Stewart (1922). This synthesizer implemented a source-filter model by filtering different source signals by analogue filters. A circuit diagram of this synthesizer is shown in Fig. 2.1. The filters were obtained by
<figure><img src="image_10.png" /><figcaption>**Figure 2.1:** Electrical circuit of the first electrical speech synthesizer. A motor-driven circuit interrupter produces a periodic excitation that is filtered by the analogue filter to produce a speech signal fed to the telephone receiver. Different vowels can be produced by adjusting the variable components to generate different resonant frequencies. [From the original publication (Stewart, 1922). Public Domain.]</figcaption></figure>

experimentation and Stewart provides frequency and damping constant of the filters impulse response for six different vowels in the English language. The excitation signals are generated from a hiss for noise excitation and a motor-driven circuit interrupter for the voiced excitation. The synthesizer was able to produce singing, speech, and whispering of vowels, semi-vowels, diphthongs and fricative consonants but failed to produce plosives.

The speech synthesizer of Stewart can be seen as an electrical implementation of the mechanical speech apparatûs in the sense that the mechanical components have been replaced by electrical ones. Apart from electrical filtering and pulse generation being more flexible than their mechanical counterpart, the real benefits of using electrical components became clear through the invention of the vocoder (Dudley, 1939). Stewart relied on experimental fine-tuning of the circuits in his speech synthesizer to match the filter characteristics by adjusting the variable components by hand and fixing the values when the result sounded correct. Dudley, on the other hand, automated the analysis process and obtained an automatic transcription of the synthesis parameters. This analysis-synthesis cycle was named Vocoder as a portmanteau of the words voice and encoder. An immediate consequence is the increased number of parameters: As the speech analysis happens automatically Dudley was able to parametrize the voice using ten frequency bands and the fundamental frequency thus yielding an 11-dimensional parametric space. Further noteworthy analogue voice synthesizers were Voder, a voice synthesizer based on Dudley's vocoder, Orator Vox Electrica (OVE), a speech synthesizer created at the Royal Institute of Technology (KTH) in Stockholm (Fant and Martony, 1962; Liljencrants, 1968), and the Music and Singing Synthesis Equipment (MUSSE) singing synthesizer (also from KTH) (Larsson, 1977; Sundberg, 2006).

At this point the need arose to understand the acoustics of the voice to utilize the possibilities of the increasingly parameter rich synthesizers. Thus, the analysis of voice and its acoustical properties emerged as its own research topic. Among the enormous amount of publications we find a great variety of different subtopics. Vocal intensity has been studied with its relationship to various other voice properties. Ladefoged and McKinney (1963) and Isshiki (1964) provide first studies about how vocal intensity is controlled physically by the voice. Further studies have deepened
our understandings about how different voice properties depend on each other and interact (Holmberg et al., 1988; Titze and Sundberg, 1992; Sundberg et al., 1993; Henrich et al., 2005). Studies on the effect of external factors on the voice, such as distance between speakers can be found in Traunmüller and Eriksson (2000). The study of the voice production mechanism has yielded several glottal pulse models, like the Rosenberg C model (Rosenberg, 1971), the LF model (Fant, Liljencrants, et al., 1985), the KLGLOTT88 model (D. H. Klatt and L. C. Klatt, 1990) and the R++ model (Veldhuis, 1998).

With the advent of computers the vocoder could be reimplemented digitally and synthesis could be performed even more flexibly. Again, new digital synthesizers were first just an implementation of analogue synthesizers in software which came with some benefit. The big power of digital synthesis, however, became apparent through inventions like the phase vocoder (Flanagan and Golden, 1966), linear predictive coding (LPC) (Atal and Hanauer, 1971) and FM synthesis (Chowning, 1989). Computer based calculations of the parameters required by these new technologies made their application possible and had opened new possibilities for voice research. Early digital synthesizers were the work of Kelly and Lochbaum (1962), MUSSE DIG, a digital reimplementation of MUSSE in 1978 (Sundberg, 2006), and Chant (Rodet et al., 1984) developed at IRCAM. Voice transformations could be achieved thanks to pitch synchronous overlap and add (Moulines and Charpentier, 1990), harmonic plus noise model (Laroche et al., 1993; Stylianou et al., 1995) and the shape-invariant sinusoidal models including the phase vocoder (Quatieri and McAulay, 1992; Roebel, 2010).

Around the turn of the century concatenative synthesis (Moulines and Charpentier, 1990; Sagisaka et al., 1992; Hunt and Black, 1996) was a popular choice for speech synthesis. The idea of concatenative synthesis is to have a large database of linguistic units (phonemes, diphones, etc.) and concatenate the units according to the target text. The concatenation could be done on the audio signal or on the parameters of a parametric vocoder. The latter is called parametric concatenative synthesis and was particularly popular in singing synthesis (Umbert, Bonada, and Blaauw, 2013; Bonada et al., 2016; Ardaillon, 2017). An alternative was statistical parametric speech synthesis (Zen, Tokuda, et al., 2009), where a statistical model, like hidden Markov models (HMMs) (Yoshimura, 2002), is used to generate vocoder parameters from the target text. These methods could produce voice with high flexibility and reasonable quality thanks to the development of high quality parametric vocoders (Kawahara, 2006; Morise et al., 2016; Huber and Roebel, 2015).

Since the last decade, we can observe a new technology emerging within the voice research community, the technology of deep learning. Like in other domains, deep learning is in the process of revolutionizing the field. Early work using deep learning applied to voice can be found in statistical parametric speech synthesis, where neural networks were used as the statistical models (Zen, Senior, et al., 2013; Zen, Agiomyrgiannakis, et al., 2016) while the actual audio synthesis was still performed by a parametric synthesizer. In recent years, however, thanks to deep learning we have seen many breakthroughs that would have not been possible using previous approaches. Nowadays, the best speech synthesis (Donahue et al., 2021; Ren et al.,
<figure><img src="image_11.png" /><figcaption>**Figure 2.2:** Cross-section of the auditory system. Sound enters through the ear, travels through the auditory canal and excites the tympanic membrane. The vibrations are transmitted via the malleus, incus, and stapes to the cochlea where the sound is split into its frequency components. [Adapted from Wikimedia Commons Anatomy_of_the_Human_Ear.svg. Creative Commons Attribution 2.5 Generic Licence.]</figcaption></figure>

2021; J. Kim et al., 2021; C. Wang et al., 2023; K. Shen et al., 2023), singing synthesis (J. Wu and Luan, 2020; P. Lu et al., 2020) and speaker identity transformation (J.-X. Zhang et al., 2019) and other applications are achieved by neural networks. We see the same pattern unfolding as we have before during the introduction of electrical circuitry and later with digital signal processing: a new technology emerges which causes us to rethink the way we approach the problem. Great inventions are being discovered continuously with more foreseeable in the future. While we have arrived at the present in our narrative of historic overview, this is far from the end of the story of voice research.

## 2.2. Human perception of sound

In this thesis we aim to synthesize audio signals that *sound* like human voice. The verb 'to sound like' is a subjective action. We need to emphasize at this point that we are not trying to recreate human voice exactly but rather to create signals that are indistinguishable from real voice to human listeners. This slight difference is often neglected but gives us leeway to reduce complexity and focus our resources on relevant acoustic features. In order to produce the necessary features in the synthetic voice signals we first need to study how humans perceive voice.

Sound is perceived in the auditory system (Moore, 2012, Ch. 1.6), which is depicted in Fig. 2.2. Incoming sound is caught by the *auricle*, or *pinna*, and channelled into the *auditory canal* which due to its decreasing diameter amplifies the sound pressure. At the end of the auditory canal is the *tympanic membrane*. The variations in sound pressure are picked up by the tympanic membrane and transmitted via the hearing bones, *malleus, incus and stapes*, to the *inner ear* (depicted in purple in Fig. 2.2).

In the inner ear, the sound is perceived in the cochlea (the other part, the semi-circular canals, are responsible for our sense of balance). The cochlea is a lengthy cavity surrounded by bony structure. It is about 30 mm long and winded like a snail turning 2.75 times. The interior of the cochlea is filled by almost incompressible
<figure><img src="image_12.png" /><figcaption>**Figure 2.3: Cross-section of the cochlea.** Vibration of the basilar membrane is what we perceive as sound. [Adaption from Wikimedia Commons Cochlea-crosssection.svg. Creative Commons Attribution-Share Alike 3.0 Unported Licence.]</figcaption></figure>

fluids. At its base are two membrane-covered openings, the *oval window* and the *round window*. The oval window connects the interior of the cochlea to the stapes and allows sound to enter the cochlea. Since the cochlea is filled with incompressible fluid, the round window exists to allow the displaced volume to be absorbed by connecting the interior of the cochlea to the tympanic cavity. The interior of the cochlea is split lengthwise by two membranes, *Reissner's membrane* and the *basilar membrane*. A cross-section of the cochlea can be seen in Fig. 2.3.

The basilar membrane is where the vibrations of the incoming sound are detected. The mechanical properties of the basilar membrane change along the length of the cochlea, making it react differently to each frequency: Incoming sinusoids cause the basilar membrane to vibrate. The amplitude of the vibration is different along the basilar membrane with its maximum somewhere along the length of the cochlea. The exact position of the maximum depends on the frequency of the sinusoid; the part closer to the oval window (the base) reacts more sensitive to high frequencies, while the tip (the apex) reacts more sensitive to low frequencies (Ruggero, 1992). Each point along the length of the basilar membrane can be thought of as a band pass filter. Measurements have shown, that the bandwidth of these band pass filters are about 1/8 to 1/2 of an octave (Moore, 2012, Ch. 1.6). Thus, the basilar membrane decomposes the frequency spectrum over space. Finally, the vibrations on the basilar membrane are detected along its length, translated into nervous impulses and sent to the brain.

There is much more to the hearing mechanism which is beyond the scope of this thesis. A very interesting and detailed description can be found in Moore (2012). The important take-away for us is, that sound is decomposed into its frequency components before it enters the brain. Hearing is the perception of frequencies, not air pressure itself.
2.2.1. Perception of pitch

Pitch and frequency are related but different concepts. Frequency is a physical quantity and describes how often a periodic process repeats. Pitch is a psychoacous- tical property related to the frequency of sinus tones and can be extended to more complex sounds that evoke the same perception of pitch (Moore, 2012, Ch. 6.1). That is, in the world of pure sine tones, pitch and frequency are the same scales. Further stimuli can evoke the perception of the same pitch as a sine tone, though their harmonic structure may be very different.

While the phenomenon of pitch is well known to everyone with functioning ears, how pitch is perceived is still a matter of debate (Cheveigné, 2010). While several theories for the perception of pitch exist, ‘physiological and psychophysical invest- igations have failed to rule decisively in favor of either hypothesis [...]’ (Cheveigné, 2010, Sec. 4.12). Nevertheless, most naïve models of pitch perception can be ruled out through simple counter examples. Camacho (2007, Sec. 1.2) provides illustrative counter examples, which I will summarize here:

**Pitch is not the lowest partial** Generally, the pitch remains the same when adding or removing harmonics. This is even true for the fundamental frequency. Removing all the energy from the fundamental frequency does not necessarily change the pitch.

**Pitch is not the distance between partials** Camacho (2007) shows the example of a square wave, which has only odd harmonics. Thus, the distance between the partials is $2f_0$, but the pitch is perceived as $f_0$.

**Pitch is not the fundamental frequency** Adding subharmonics does not change the perceived pitch if the energy from the subharmonics is low compared to the remaining signal. This effect can be observed in rough voice where subhar- monics might be introduced through the glottal source.

**Pitch can be evoked by inharmonic signals** As shown by Patel and Balaban (2001), even inharmonic signals can evoke the perception of pitch. Stimuli consisting of very low periodicity might evoke a perceived pitch of much higher fre- quency. For example, Patel and Balaban (2001) show that the combination of sinusoids of 650 Hz, 950 Hz and 1250 Hz evoke a pitch of 315 Hz or 650 Hz.

**Pitch can be evoked by binaural effects** A particular curiosity is the phenomenon of *Huggins pitch* (Cramer and Huggins, 1958; Culling et al., 1998). This stimulus is created by an all pass applied to white nosie. Then the original and the filtered signal are played simulatenously in the left ear and the right ear respectively. Though each of the channels, the filtered and the unfiltered component individually are perceived only as white noise, listening to them at the same time creates the perception of a pitch at the frequency, where the phase changes (Cheveigné, 2010, Sec. 4.8).

**Pitch is not a linear scale** Though generally we distinguish between low and high pitches, there is no transitivity on the perceived pitches, as can be seen
through Shepard scales (Shepard, 1964). In a rising Shepard scale, the next stimulus appears to have a higher pitch than the previous, even though the whole scale is periodic resulting in a circular scale of rising steps, similar to the Penrose steps.

The observations above reveal that pitch cannot be easily obtained from frequency. Though, in general, counter examples exist, in voice, the pitch usually corresponds to the fundamental frequency. Thus, in voice, we can measure pitch through the fundamental frequency $f_0$, by taking a proper pitch scale into account. In the following we will use the terms ‘pitch’ and ‘fundamental frequency’ interchangeably.

### 2.2.2. Pitch scales

Different approaches to quantify pitch exist. Pitch can be quantified as the frequency of the corresponding sinusoidal stimulus. However, since pitch is a perceptive quantity, a proper pitch scale should reflect the perceptive component quantitatively as well. Two approaches exist: either through perceived sizes of intervals, leading to the mel-scale, or through the frequency resolution of the basilar membrane, leading to the ERB-scale.

#### The mel-scale

Observations of non-uniform perception of pitch intervals versus frequency ratios have lead Stevens et al. (1937) to the development of the mel-scale, which maps frequency to a perceptive pitch value in mel. This scale is quantized by assigning 1000 Hz = 1000 mel and fitting the observations from the perceptive measurements. Stevens et al. (1937) asked participants to set specific pitch ratios, e.g. a pitch of 2000 mel would be twice as high as a pitch of 1000 mel. Later formulas have been introduced to fit the measured data and typically the formula

$$m = m_0 \log \left( 1 + \frac{f}{f_m} \right) \qquad (2.1)$$

is used to map between values *m* on the mel-scale and the frequencies *f* in Hz. The cut-off frequency $f_m$ can be seen as a soft transition where the scale switches from linear to logarithmic. Today normally we use $f_m = 700$ Hz, while $f_m = 1000$ Hz is also common in particular in older literature.

The mel-scale has found its application through the *mel-filtered cepstral coefficients (MFCCs)* in speech recognition (Rabiner and Juang, 1993) and is currently very prominent through the mel-spectrogram in deep-learning based speech processing (J. Shen et al., 2018).

#### The mel-spectrogram

The mel-spectrogram is a spectrogram with the frequency axis compressed to the mel-scale. To calculate mel-spectrograms we need to remap the frequency axis to the mel-scale. If we were dealing with continuous-time signals we could simply
<figure><img src="image_13.png" /><figcaption>**Figure 2.4.:** Mel filter bank for a configuration with 20 mel-bands. In this thesis we use 80 mel-bands, but this would make the graph unreadable.</figcaption></figure>

apply Eq. (2.1) to the frequency axis. We are, however, dealing with discrete-time signals. That means the spectrograms have discrete frequency bands. Similarly, the mel-spectrogram has discrete mel-bands. Here is really where the strength of the mel-spectrogram comes into play as using the mel-spectrogram allows us to reduce the number of bands dramatically. In a typical application the mel-spectrogram is used to compress 1025 frequency bands to 80 mel-bands. Thus, one mel-band represents multiple frequency bands, especially for higher frequencies. To reflect the information from all the frequency bands it does not suffice to interpolate the frequency axis at the centre frequencies of the mel-bands. Since each mel-band represents multiple short time Fourier transform (STFT) bands, we average over the bandwidth of each mel-band with triangular weights. This creates the mel-filter bank; an example is shown in Fig. 2.4. Since each filter is only used around one centre-frequency we can write all the filters into a matrix and shift them to the position of the mel-band. Thus mapping from a spectrogram S to its mel-spectrogram M can be achieved by the linear operator B as

$$M_{\text{lin}} = B |S|. \qquad (2.2)$$

In practice often we use the log-amplitude mel-spectrograms in our processing pipeline rather than the linear-amplitude mel-spectrogram which changes Eq. (2.2) to

$$M_{\text{log}} = \log(B|S|). \qquad (2.3)$$

Since log / exp are bijective these representations are equivalent, and therefore we will use the term *mel-spectrogram* to refer to either of these representations.
## The equivalent rectangular bandwidth filter bank

An alternative to the mel-scale are equivalent rectangular bandwidth (ERB) filter banks. The ERB scale is based on the study of the filter characteristics of the basilar membrane. Every point along the basilar membrane acts as a band pass filter with a different centre frequency and a different bandwidth. The bandwidth can be given as a function of centre frequency which is approximated by Glasberg and Moore (1990) as

$$w_{\text{ERB}} = \alpha (1 + \beta f_c), \qquad (2.4)$$

where $f_c$ is the centre frequency, $w_{\text{ERB}}$ is the equivalent rectangular bandwidth, and $\alpha = 24.7$ Hz and $\beta = 4.37$ ms are heuristic constants. Equation (2.4) can be used to create the ERB-scale $s_{\text{ERB}}$ through

$$s_{\text{ERB}} = \int_{0}^{f} \frac{1}{w_{\text{ERB}}(f_c)} df_c \qquad (2.5)$$

$$= \frac{1}{\alpha\beta} \log(1 + \beta f) \qquad (2.6)$$

$$\approx 9.265 \log \left(1 + \frac{f}{229 \text{ Hz}}\right). \qquad (2.7)$$

Thus, the ERB scale resembles the mel-scale in structure. Note that on the ERB scale the transition from linear to exponential is much lower (230 Hz instead of 700 Hz). The other significant difference is that ERB bands have exactly width 1 on the ERB scale and are constant height rather than triangles. The width of the mel-bands depends on the number of mel-bands and at any frequency exactly two mel-bands overlap. The width of the ERB-bands is always the same and how many bands overlap depends on the overlap value $r$. Given an overlap value $r$, an ERB filter bank can be constructed by taking intervals of length 1 with step size $1 - r$ on the ERB scale and mapping the boundaries back to the frequency scale in Hz.

### 2.2.3. Loudness

The amplification through the middle ear does not occur with flat transfer function (Glasberg and Moore, 2002) and the basilar membrane reacts different to each frequency (Moore, 2012, Ch. 1.6). As a consequence our hearing has different sensitivity for different frequencies. First measurements were done by Fletcher and Munson (1933) where the authors measured how loud people perceive sounds of different frequencies. For this, they introduced the phon-scale. For 1 kHz sinusoids the phon-scale corresponds to the sound pressure level in dB. A sinusoid at a different frequency gets assigned the loudness level of a 1 kHz sinusoid that is perceived as equally loud. Using the phon-scale we can obtain equal-loudness contours. An example can be seen in Fig. 2.5. From these curves we can visualize how our hearing is dependent on the frequency, especially for lower levels. Later the equal-loudness contours were revised by D. W. Robinson and Dadson (1956) to create the initial ISO 226 standard. The standard was revised in 2003.
<figure><img src="image_14.png" /><figcaption>**Figure 2.5.:** Equal loudness contours from ISO 226: 2003 standard for phon values 0, 10, ..., 90. The 0 phon value is the minimal audible threshold. Hearing is most sensitive between 1 kHz to 5 kHz.</figcaption></figure>

These equal-loundess contours are obtained using sinusoids at different frequencies. However, a more complex frequency spectrum and temporal changes have an effect on the perceived loudness. These issues are addressed by Glasberg and Moore (2002) who provide a loudness model for non-stationary sounds.

The phon-scale is measured in units of dB. However, observations of Stevens et al. (1937) have shown that a sound is perceived twice as loud with each increase by 10 dB. Thus, Stevens et al. (1937) introduced the sone scale to capture the perceived loudness. We assign the value of 1 sone to the phon value of 40 dB and doubles with each increase by 10 dB. Thus, the loudness in sone can be obtained through the formula

$$L_s = 2^{\frac{L_p - 40 \text{ dB}}{10 \text{ dB}}} \text{sone} \qquad (2.8)$$

$$= (10^{-4} L_e)^{0.30103} \text{sone,} \qquad (2.9)$$

where $L_s$ is the loudness in sone and $L_p$ is the loudness in phon and $L_e$ is the signal power. Thus, the perceived loudness increases with the power of 0.3 of the signal's power.

## 2.3. Fundamentals of the voice

To transform human voice we need to understand how human voice is produced. In this section we will have a look at speech production. We will introduce the source filter model and the glottal pulse model of Fant, Liljencrans, et al. (1985)
<figure><img src="image_15.png" /></figure>

**Figure 2.6:** Cross-section of the larynx, the main speech production organ. Located in the neck, the bottom part connects through the trachea to the lungs. The other end is connected to the nasal and oral cavities. [Adaption from Wikimedia Commons *Illu_larynx*.jpg. Public Domain]

which provides us with various parameters of the voice that we can use to describe different aspects of it.

### 2.3.1. The theory of speech production

During the production of human voice, three principal components influence its acoustic appearance: the glottal source, the noise component and the vocal tract filter. Voiced speech is produced in the glottis, which generates periodic or sometimes irregular pulses. Additionally, noise originates in various places of the vocal tract. Finally, the glottal pulse and the noise is shaped acoustically through the vocal tract.

#### Phonation and the glottal source

The act of producing voiced speech is called phonation. Voiced speech originates in the glottis and is formed in the nasal and oral cavities. An illustration of the Larynx can be found in Fig. 2.6. During phonation the vocal folds are tensed such that they are pushed together and close the larynx. At the same time the lungs build up air pressure that pushes the vocal folds aside. Once the air pressure is high enough, the vocal folds open and air bursts out. As the air leaks through the now opened vocal folds, the air pressure pushing the vocal folds apart drops, allowing the vocal folds to close again, and the process repeats anew. This happens periodically and creates a pulse signal, called *glottal pulse signal*.

#### Noise component

In phonated voice the glottal pulse is the dominant sound source, but it is not the only source. Turbulences in the air flow through the vocal tract create noise which can be most prominently heard in unvoiced fricatives like [f], [θ], [s] or [ʃ], but is in fact an audible component in all other modes of speech, including vowels.

We can divide the noise component in speech into two categories (Richard and Alessandro, 1996):
<figure><img src="image_16.png" /><figcaption>**Figure 2.7:** Cross-section of the vocal tract. Glottal pulses originate in the larynx and are filtered by travelling through the nasal and oral cavities. The filter is highly dependent on the position of the tongue. [Adaption from Wikime-dia Commons *Illu01_head_neck*.jpg. Public Domain]</figcaption></figure>

**Additive noise** which is generated independently of the glottal source and can be assumed to be superimposed additively.

**Structural noise** which originates from variations in the glottal pulse.

The additive noise further be subdivided into *transient noise*, *quasi-stationary noise* and *modulated noise*. Transient noise occurs through plosive consonants like [p], [t] or [k] or through mouth noise, lip noise and clicks. Quasi-stationary noise is generated through turbulent airflow in the vocal tract or by the glottis if the vocal folds remain open. It is prominently dominant in unvoiced fricatives. Quasi-stationary noise is used as the excitation signal in whispering. Modulated noise is generated from turbulent airflow which is modulated by the vocal fold vibration and occurs most notably in breathy vowels and voiced fricatives. The structural noise can be subdivided into *jitter* (random variation in the pulse positions), and *shimmer* (random variation in the pulse amplitudes) (Richard and Alessandro, 1996).

## Vocal tract filter and spectral envelopes

After the glottal pulse signal originates in the glottis, it has to leave the body to be heard by the listener. This happens mainly through the oral and nasal cavities (see Fig. 2.7) though a small fraction of the sound may also be transported through the body itself. Acoustically, the oral and nasal cavities can be thought of as long tubes that reflect and absorb the sound waves. As a result they don't transmit all frequencies equally well and thus act as filters on the glottal pulse signal. The superposition of the different paths that the signal can take creates the *vocal tract filter* (VTF). This filter depends very much on the geometry of the oral cavity which in turn depends on the position of the tongue and the opening of the mouth. This allows the speaker to create different sounds from the same glottal source signal such as vowels and voiced consonants.

Similarly, the noise component of the voice is filtered through the vocal tract. Since parts of the noise can be created in the vocal tract after the glottis, the mech-
anism is slightly different. Depending on where the noise component originates only a fraction of the vocal tract filter acts on the noise component.

The combination of these different effects creates a filter that is characterized by resonances and zeros. The resonances are called *formants*, and it was shown (e.g. by Stewart (1922)) that the formants are the key acoustic feature that allows us to distinguish different vowels.

### 2.3.2. Glottal flow models

The glottal pulse signal has been studied extensively in the last 50 years and several glottal flow models have emerged. Common glottal pulse models are the Rosenberg C model (Rosenberg, 1971), the Liljencrants-Fant (LF) model (Fant, Liljencrants, et al., 1985), the KLGLOTT88 model (D. H. Klatt and L. C. Klatt, 1990) and the R++ model (Veldhuis, 1998). A discussion on them can be found in Doval et al. (2006). Of particular interest for this work is the Liljencrants-Fant (LF) model (Fant, Liljencrants, et al., 1985) as we will be using it in Chapters 4 and 5. Here it shall serve as an example how the glottal pulse can be parametrized, allowing us to introduce various pulse qualities like the *open quotient (OQ)* or the *glottal closure instant (GCI)*.

The classical Liljencrants-Fant (LF) model of Fant, Liljencrants, et al. (1985) describes the glottal pulse by three parameters (four if you count the amplitude $E_e$, or five if you also count the period $t_0$). A simplified model has later been given by Fant, 1995 using only one parameter. The glottal pulse of the LF model alternates between two phases, the open- and the closing phase. A visualization can be found in Fig. 2.8. The three LF pulse parameters are:

$t_p$ the time when the maximal flow occurs,

$t_e$ the time when the vocal folds close,

$t_a$ a measure of how long it takes for the vocal folds to close, the *return phase*. As the glottal flow in the LF model is never truly zero, we define the closing duration as the time between the closure instant $t_e$ and the time $t'_a$ where the tangential at $t_e$ (red dashed line in Fig. 2.8) crosses the time-axis.

The pulse parameters follow the restriction

$$0 < \frac{t_e}{2} \le t_p < t_e \le t_e + t_a \le t_0. \qquad (2.10)$$

Furthermore, we identify the parameters

$t_0$ the period of the pulse,

$U_0$ the amplitude of the glottal flow, and

$E_e$ the amplitude of the glottal flow derivative.
<figure><img src="image_17.png" /><figcaption>**Figure 2.8.:** Glottal flow and glottal flow derivative of the Liljencrants-Fant glottal pulse model, annotated with the defining parameters $t_p$, $t_e$, and $t'_a = t_e + t_a$. The curves are normalized in time by the period and in value their maxima.</figcaption></figure>

The LF model defines the glottal flow derivative $E: [0, t_0) \to \mathbb{R}$ using two separate cases for open- and closing phase:

$$E(t) = \begin{cases} E_o(t) & t \le t_e \\ E_c(t) & t > t_e \end{cases} \qquad (2.11)$$

with

$$E_o(t) = E_o e^{\alpha t} \sin \frac{\pi t}{t_p} \qquad (2.12)$$

during the open phase ($t \le t_e$) and

$$E_c(t) = -E_1 (e^{-\epsilon(t-t_e)} - e^{-\epsilon(1-t_e)}) \qquad (2.13)$$

during the closing phase ($t > t_e$). Equations (2.12) and (2.13) define the pulse only implicitly. Furthermore, we require the following two conditions:

$$E_o(t_e) = E_c(t_e) \qquad (2.14)$$
to ensure *E* is continuous (except for the special case where *t*<sub>a</sub> = 0, in which case *E*<sub>c</sub>(*t*) = 0 for all *t*), and

$$
\int_{0}^{t_0} E(t) \, dt = 0 \qquad (2.15)
$$

to ensure *E* is a derivative of a periodic function. There is no closed form solution for *E*₀, *E*₁, *α* and *ε* but an algorithm to obtain the necessary parameters numerically can be found in Fant, Liljencrants, et al. (1985).

Equivalently the glottal pulse model can be described by

$O_q$ (open quotient) The ratio of duration of the open phase and the whole pulse duration.

$$
O_q = \frac{t_e}{t_0} \tag{2.16}
$$

$\alpha_m$ (open phase skewness) The ratio between $t_p$ and $t_e$.

$$
\alpha_m = \frac{t_p}{t_e} \tag{2.17}
$$

$f_a$ (glottal formant) The frequency where the spectral tilt changes from 20 dB per decade to 40 dB per decade.

$$
f_a = \frac{1}{2\pi t_a} \tag{2.18}
$$

or by the ratios

$$
R_a = \frac{t_a}{t_0} \tag{2.19}
$$

$$
R_g = \frac{t_0}{2t_p} \qquad (2.20)
$$

$$
R_k = \frac{t_e - t_p}{t_p}. \qquad (2.21)
$$

The simplified LF model

The full LF model allows a flexible parametrization of the glottal pulse but in fact
these parameters are not independent. Thus, not all combinations of pulse para-
meters make sense even if they stay within their mathematically permitted bounds.
Furthermore, estimation of the pulse parameters from voice recordings is unreli-
able as the glottal pulse cannot be observed directly. Thus, Fant (1995) provides a
heuristic that defines a simplified LF model in which the relevant parameters are
derived from one meta-parameter *R*<sub>d</sub>. The *R*<sub>d</sub>-parameter is defined as

$$
R_d := \frac{f_0 U_0}{110 E_e}, \qquad (2.22)
$$
where $U_0$ is the amplitude of the glottal flow and $E_e$ is the amplitude of the glottal flow derivative.

Using the statistical relations from Fant and Kruckenberg (1994) the other ratios can be predicted as

$$R_a = \frac{-1 + 4.8R_d}{100} \qquad (2.23)$$

$$R_k = \frac{22.4 + 11.8R_d}{100} \qquad (2.24)$$

$$R_g = \frac{0.125R_k + 0.3R_k^2}{0.11R_d - 0.5R_a - 1.2R_aR_k} \qquad (2.25)$$

The numerical range for $R_d$, is provided empirically by Fant (1995) as $R_d \in [0.3, 2.7]$.

Different pulses from the simplified LF pulse model are visualized in Fig. 2.9. For high $R_d$ values the pulse becomes more and more smooth, and for the highest value, 2.7, the pulse closely resembles a sine curve except for a small dent towards the end. In the spectrum we can see that with higher $R_d$, the energy moves more and more towards the lower Fourier coefficients. For the highest $R_d$ value most energy is located in the fundamental frequency and already the second partial (coefficient index 2) is attenuated by 20 dB. Conversely, with lower $R_d$ value, the 'dent' becomes sharper and turns into a sharp peak for the lower $R_d$ values. This peak creates a rich overtone spectrum and moves the energy more and more to the higher Fourier coefficients so that the higher partials actually carry more energy than the fundamental frequency.

## 2.4. Neural networks

The main tool for voice treatment in this thesis are neural networks. Although we're not studying neural networks in this work explicitly, it is an important tool for modelling the voice, so a deep understanding of the underlying principles is essential. Neural networks are universal approximators (Hornik et al., 1989), which means that any<fnref n="1" /> mathematical function $f: \mathbb{R}^n \to \mathbb{R}^m$ can be approximated arbitrarily well using a sufficiently complex neural network $\hat{f}_{\theta}$. The parameters of a neural network are called weights and are typically denoted by $\theta$.

Neural networks can have very complex structures (called architecture) but generally a neural network is characterized trough alternating applications of affine operations $\Phi_{\theta_k}^k: \mathbb{R}^{n_k} \to \mathbb{R}^{n_{k+1}}$ and non-linear scalar functions $a: \mathbb{R} \to \mathbb{R}$ applied to each dimension individually ($a(\mathbf{x}) := (a(x_1), ..., a(x_n))$). In its most basic form the neural network function can be written as

$$\hat{f}_{\theta_1, ..., \theta_n} = \Phi_{\theta_n}^n \circ a \circ \Phi_{\theta_{n-1}}^{n-1} \circ ... \circ a \circ \Phi_{\theta_1}^1, \qquad (2.26)$$

<footnote n="1">1. The original paper of Hornik et al. (1989) proves the approximation theorem for any Borel-measurable function between finite dimensional spaces. The claim has since been extended to more functional spaces. In practice this means we can approximate any function that is not specifically designed as a counter example.</footnote>
<figure><img src="image_18.png" /></figure>

**Figure 2.9.:** Liljencrats-Fant glottal flow for different $R_d$ values. Top: flow derivative. Bottom: Fourier coefficients and glottal formants. The pulses are normalized to constant energy. For low $R_d$ values, the glottal closure instant increasingly becomes a sharp peak. The sharp peaks create a rich overtone spectrum which we can see in lower graph. For high $R_d$ values the energy moves more towards the lower harmonics. In the extreme case of $R_d$ = 2.7, already the second partial (coefficient index 2) is attenuated by 20 dB and apart from a small dent at the end, the pulse shape looks much like a sinusoid. Since for all $R_d$ values the glottal flow is continuous but not differentiable, eventually all Fourier coefficients decay with 40 dB per decade. The transition from 20 dB decay to 40 dB decay is called glottal formant (marked by the vertical lines) and occurs at higher frequency for lower $R_d$ values.
where ◦ denotes functional composition (|[f ◦ g](x) = f(g(x))). An iteration of a ◦ Φk is called a layer. The linear components of the affine operations Φ do not necessarily need to be dense matrices. For example, convolutional neural networks (LeCun et al., 1989) use convolutions instead of matrix multiplications which yields much fewer parameters. Likewise, other operations may be interwoven between layers such as combinations of previous layers (called skip connections (Long et al., 2015; Ronneberger et al., 2015)) or DSP operations (called differentiable digital signal processing (DDSP) (Engel et al., 2019))

Normally the target function f is not available explicitly, which is why we want to approximate it by a neural network. Instead, the function f is described implicitly through samples

$$D = \{(x, y) : y = f(x), x \in X\} \qquad (2.27)$$

at specific points X. The set of known points D is called dataset. Typically, we divide the dataset into two subsets:

$$D_{\text{train}} = \{(x, y) : y = f(x), x \in X_{\text{train}} \subseteq X\}, \qquad (2.28)$$

the training set, and

$$D_{\text{test}} = \{(x, y) : y = f(x), x \in X \setminus X_{\text{train}}\}, \qquad (2.29)$$

the test set.

We optimize the weights θ such that the approximation error L is minimal on the training set:

$$\mathcal{L}(\theta) = \sum_{(x,y) \in X_{\text{train}}} d(y, \hat{f}_{\theta}(x)), \qquad (2.30)$$

$$\hat{\theta} = \underset{\theta}{\operatorname{argmin}} \mathcal{L}(\theta), \qquad (2.31)$$

for some appropriate function d that quantifies how well x matches<fnref n="2" /> $\hat{f}_{\theta}(x)$. This optimization is called training. To estimate how well the trained network $\hat{f}_{\theta}$ approximates the true function f we can evaluate its ability to generalize on the test set.

To train the neural network, a convenient algorithm exists, the backpropagation algorithm (Rumelhart et al., 1986), which is based on gradient descent. Backpropagation allows efficient calculation of the gradient for all the weights by propagating the error through the neural network in reverse order.

While neural networks are a simple and powerful tool, their success depends on the proper application. In many cases the dataset D can be easily compiled by collecting the sufficiently many data pairs (x, y). From the machine learning standpoint, these cases are considered solved and are hardly interesting research questions. Many

<footnote n="2">2. We can most intuitively think of d as a 'distance' function though in strict mathematical terms not all requirements for a distance in the mathematical sense are necessary. The Kullback-Leibler divergence, for instance, is not a distance due to its asymmetry but is a very common choice for d.</footnote>
applications of neural networks, however, are not as straightforward. Often the relationship we want to model cannot be easily expressed with input-output pairs required to create the training data *D*.

Consider, for example, the task of transposing voice. To train a neural network transpose directly as discussed above, we would need to compile a dataset *D* containing pairs of original and transposed voice. Firstly, the amount of data required to sufficiently densely sample this mapping is unpractical. Creating a dataset that represent voice well, already requires large amounts of data. For the task of transposition, however, we would need a representative amount of pairs for every transposition value. Even assuming that we would have enough time to compile such a dataset, the attempt would still be futile, since proper pairs of original and transposed voice do not exist: To transpose voice, different equally likely solutions exist. Nevertheless, as we will see in Chapter 6, we are able to use neural networks to do transpositions. The trick is, not to model the transposition directly, but to train multiple neural networks on more efficient sub-tasks. Finding the right relationships to model with neural networks is what makes the field of machine learning challenging and interesting.

### 2.4.1. Neural arithmetic on voice signals

Voice signals, as a sub-category of audio signals, are functions that map $Z \to R$, though we usually only look at a finite length section of the signal.<fnref n="3" /> We assume shift invariance for the underlying distributions in the sense that it does not matter at which time we look at the signal. Audio recordings are almost never of fixed length and the network architecture should be able to process audio recordings of arbitrary length.<fnref n="4" /> The prior knowledge about shift invariance can be hard-wired into the neural network architecture by convolutional layers (LeCun et al., 1989), recurrent layers (Werbos, 1990; R. J. Williams and Zipser, 1995; Sutskever et al., 2014) or transformers (Vaswani et al., 2017), together with pooling and resampling. We will very rarely see fully connected layers that stretch out across the whole time axis of the signal.

Inside the neural network the voice signal can be represented in different ways. The most obvious signal representation is the time-domain signal as it is provided through audio recording and required for audio playback.<fnref n="5" /> The time-domain representation, however, has some undesired properties that make processing in a neural network difficult. Time-domain audio signals have an unwieldy shape: As they are

<footnote n="3">3. While ultimately sound is a continuous-time signal in this thesis we address only the signal processing part that happens after digitalization and do not concern ourselves with analogue-digital conversion.</footnote>

<footnote n="4">4. Computational limitations apply for large durations, where memory constraints might prevent us from processing a whole recording at once. In this case the recording must be split, and each subsection must be processed individually. Thanks to the shift invariance assumption this is possible, but a sufficient overlap must be provided to avoid boundary effects.</footnote>

<footnote n="5">5. 100 years of electronic signal processing may make this point seem obvious but as we have seen in Section 2.2, the human brain works completely different. In the brain the signal never appears in the time domain as the frequency decomposition occurs in the ear before the signal is passed to the brain.</footnote>
one-dimensional, unlike for two-dimensional images, convolutional layers are not as efficient. Audio signals encode their information through oscillation which creates dependencies spread far across the signal. In audio, signals can have significant correlation across multiple seconds. In high fidelity audio, this would require the neural networks to have receptive fields of several 10 000 audio samples which is unfeasible with stacks of one-dimensional convolutional layers. As discussed in Section 2.2, human hearing does not perceive the temporal evolution of sound waves explicitly but rather the frequency components within the sound waves. Thus, the time-domain signal does not only represent the information that arrives in the brain but also perceptually unimportant additional information. Furthermore, the information from different frequency bands is superimposed in the time-domain signal. In particular low amplitude frequency components can be easily perceived but might be almost invisible in the time domain. Finally, comparing audio signals in the time domain is difficult. Two sinusoids with the same frequency might arguably be equivalent but might produce a large difference if the phases are different. While phase information is not completely irrelevant, it emphasizes problems that usually have little significance if comparisons are made in the time domain.

All these issues can be addressed through spectral representations by using the magnitude spectrogram and similar representations. Nowadays, the mel-spectrogram is a very popular choice to representing voice recordings (J. Shen et al., 2018). As a positive side effect, spectral representations are two-dimensional data, like images. We can often borrow ideas from image processing by applying the same (or slightly modified) systems to spectral representations of audio. Ultimately, however, we need to generate a time-domain signal if we want to listen to the result. Therefore, several strategies exist to efficiently model raw-audio.

### Strided, transposed and dilated convolutions

Strided convolutions can be thought of as a convolution with a sub-sampling applied afterwards: A strided convolution with stride *n* would shift the filter in steps of *n*. Thus, the number of required computations is reduced by *n* in the first layer and cumulatively in the deeper layers. A *strided transposed convolution* can be thought of as first upsampling by filling *n* − 1 zeros between each input value and then performing the convolution. Consequently, a stride in a transposed convolution increases the sample rate again which allows us to obtain the required sample rate. *Dilated convolutions* can be thought of as a convolution where the filter is upsampled by filling *n* − 1 zeros between each filter coefficient. Dilated convolutions do not change the sample rate. Using stacks of varying dilation rates, allows creating large receptive fields using only few filter parameters. Typically, the dilation rate is increased by a factor of two after each layer (Oord, Dieleman, et al., 2016). This way, Prenger et al. (2019), for example, can use only filters of size 3 but have effective receptive fields of several seconds at 24 kHz.

<footnote n="6">6. We describe the operations in this paragraph through illustrative but inefficient implementations. Practical implementations avoid unnecessary operations like multiplications by zero which makes these operations much faster than their non-sparse counterparts.</footnote>
Sample rate reduction

To decrease the sample rate of the synthesized signal and to utilize the benefit of
parallel computation, we can use a reshape operation to group short time frames into
a single time step. This way we are creating an *n*-dimensional signal of length *t*/*n*
containing *n* sub-sampled versions of the 1-dimensional original signal of length *t*
(Prenger et al., 2019). Since no filtering occurs before the re-sampling the sub-
sampled signals contain aliasing from the higher frequency bands. This aliasing is
necessary to represent the higher frequency bands and can properly be reproduced
by neural networks in the synthetic signals. If combined with a convolution, this
representation is equivalent to using a strided convolution with filter size *s* · *n* and
stride *n* instead of reshape and filter with filter size *s*. Similarly to map back from
the *n* channels to the original sample rate a strided transposed convolution can be
used instead of a regular convolution followed by a reshape.

Alternatively, we can use quadrature mirror filters to represent the different
frequency bands in separate signals (C. Yu et al., 2020). In this approach the signal
is split into *n* signals, each representing one frequency band. The frequency bands
are obtained by filtering with a specific filter for each frequency band that removes
all frequency components outside the specific band. The band-limited signals are
then sub-sampled by the factor *n*. The frequency bands are of equal width such that
the whole frequency band fits into the bandwidth of the sub-sampled signal. To
recreate the original signal, the sub-bands need to be up-sampled and modulated on
the centre frequency of their respective band. The lowest band is filtered by a low
pass and does only require resampling. The band-pass filters can be approximated
by pseudo quadrature mirror filter bank (PQMF). These PQMF will be used in the
neural vocoder of Chapter 5.

Finally, the complex short time Fourier transform (STFT) can be used to create
a signal with multiple frequency bands and reduced sample rate. Kaneko, Tanaka,
et al. (2022) showed that an STFT with 8 or 16 frequency bands can provide a good
reconstruction with significantly reduced computational cost. While an STFT with 8
or 16 frequency bands is not meaningful in the way we would usually expect from a
spectral representation, this representation can work well for sample rate reduction.

Spectral losses

The short time Fourier transform (STFT) is a linear operation and thus differentiable.
The differentiability remains when removing the phase by taking the magnitudes
of each frequency bin.<fnref n="7" /> Therefore, the magnitude spectrogram can be used to
define a loss between two time-domain signals allowing us to circumvent issues with
unknown phase information. This is useful for example if we want to synthesize
a periodic signal whose pulse positions are unknown. Furthermore, we can use

<footnote n="7">7. Note that in terms of complex analysis, many common operations like the absolute value |·| and real- and imaginary part (Re and Im resp.) are not holomorphic (complex differentiable) and thus holomorphy is lost if these operations are involved. Usually, however, we only require differentiability with respect to real variables, in which case differentiability is preserved through the Wirtinger derivatives (Caracalla and Roebel, 2017).</footnote>
<figure><img src="image_19.png" /><figcaption>**Figure 2.10:** *A deep dream transformed image of the Mona Lisa using the VGG16 classifier trained on the ImageNet dataset [From Wikimedia Commons, File: "Mona_Lisa"_with_DeepDream_effect_using_VGG16_network_trained_on_ImageNet.jpg. Creative Commons CCO 1.0 Universal Public Domain Dedication.]*</figcaption></figure>

multiple STFTs with different window lengths to obtain a multi-resolution spectral loss (X. Wang et al., 2019) to match the spectral composition on different time scales.

### 2.4.2. Common network architecture types for voice

There are different network architectures which are being used in signal synthesis and transformation, notably for images and sound, all with their own advantages and drawbacks. Since voice can be parametrized by spectral representations, in many applications we can treat it as an image. Thus, architectures for image transformation and synthesis can often be easily adapted to voice signals. We will discuss the most common ones here as the choice of the right tool depends on the application and should be carefully made depending on the requirements.

#### Optimization based transformations

Signals can be transformed by synthesizing new signals that are constraint to contain specific information from a source signal. Synthesis can be performed by the same technique that we use to train neural networks, with gradient descent. In this strategy we initialize a prototype signal with zeros or noise and minimize the distance between some feature metric of the target signal and the synthetic signal.

Optimization based transformation has been used for style transfer (Gatys et al., 2016), where the content of a photograph is matched with the 'artistic style' of famous paintings. This allows creating a version of the photograph that looks like a painting of famous painters like van Gogh, Picasso, or Kandinsky. Another application of this technique is in deep dream (Olah et al., 2017, see Fig. 2.10). Here a classifier is used to increase the response to specific classes, making these objects appear everywhere in the image, creating a dream-like hallucinogenic image. This principle has been applied by Caracalla and Roebel (2020) to sound texture synthesis.
The drawback of these methods is that an optimization is required during the synthesis. This results in long synthesis durations as thousands of optimization steps are required. (Gatys et al. (2016) report about one hour for a 512 × 512 image on a dedicated GPU.)

## Generative adversarial network

Generative adversarial network are systems of two neural networks that are trained adversarially (Goodfellow et al., 2020). Generative adversarial network are typically applied in synthesis tasks. A generator creates samples that resemble a set of target samples and a discriminator that is trained to distinguish between real samples from the example set, and fake samples generated by the generator. The training procedure is adversarial in the sense that a low loss value for the generator creates a high loss value for the discriminator and vice versa. Since in purely adversarial setups the loss is only provided by the discriminator it is impossible to deduce the quality of the generator from the loss value. Convergence may occur at any loss value and there is no relationship between final loss value and quality of the generated samples. Thus, when training generative adversarial network, convergence issues are likely to occur. These issues can be addressed through the use of auxiliary classifiers (Odena et al., 2017).

GANs have been applied to voice in different ways. MelGAN (K. Kumar et al., 2019) is used to generate audio from mel-spectrograms. An extension MelGAN, Universal MelGAN (Jang et al., 2020), uses multi-resolution time-domain and spectrogram discriminators, multiple discriminators operating on the time-domain signal at different sample rates and on different spectrograms with different window lengths. Similarly, multiple discriminators may be used to operate on different sections of an image or different frequency bands of a spectrogram (C. Li and Wand, 2016). Multiple variants exist for signal transformation, in particular stemming from identity transformation where we wish to map a signal between a set of discrete variants (different speaker). Here we wish to ensure consistency when mapping back and forth between different classes. CycleGAN (J.-Y. Zhu et al., 2017) maps between two classes using two generative adversarial network that work as inverses of each other. A cycle consistency loss ensures that the generators act as inverses. CycleGAN has been applied to voice identity conversion by Kaneko and Kameoka (2018). StarGAN (Y. Choi et al., 2018) allows mapping between multiple classes using a single generator conditioned on the input image and the target domain. An auxiliary classifier guides the convergence. StarGAN has first been applied to voice by Kaneko, Kameoka, et al. (2019). AttGAN (He et al., 2019) combines the ideas of StarGAN and auto-encoders by using an auto-encoder as the generator. Nowadays, we often see discriminators applied to other network architectures combining adversarial losses and losses with known target values (Roebel and Bous, 2022; Bous, Benaroya, et al., 2022).
## Auto-encoders

An auto-encoder is a system of two neural networks that are trained to be inverses of each other (Lange and Riedmiller, 2010). An encoder produces an abstract representation (typically called hidden representation or latent representation) whereas a decoder maps back from the abstract representation to the original input. The point of this encoder-decoder loop could be to create a compressed representation of the input if the latent space has fewer dimensions than the input space. Of interest here, however, is signal transformation. Through the use of additional inputs the auto-encoder may be able to create an abstract representation of its input that is disentangled from a specific feature. For example Lample et al. (2017) allows interpolation in pictures of faces between binary classes like young / old, smile / no smile or glasses / no glasses. The disentanglement can be achieved through a bottleneck auto-encoder, like AutoVC (Qian, Yang Zhang, Chang, X. Yang, et al., 2019), or an adversarial classifier operating on the latent space (Lample et al., 2017). The auto-encoder with information bottleneck is the central tool of Chapter 6 and will be discussed in more detail there.

Generally, auto-encoders have no constraint for their latent space, so the organization of if tends to be highly irregular and elements close to each other in the latent space may correspond to completely different inputs. To regularize the latent space, variational auto-encoders (Kingma and Welling, 2013) encode their input by a probability distribution on the latent space rather than a single point. In practice this means that the encoder returns the parameters of a parametric probability distribution, usually mean vector and covariance matrix of a multi variate Gaussian. During training, we sample from this distribution before mapping back to the original space for reconstruction. This forces the auto-encoder to create smooth transitions in its latent space and allows interpolating between samples by following a path in the latent space, for example between the timbre of different instruments (Esling, Bitton, et al., 2018).

## Generative flow

The idea of a generative flow is to create a complex probability distribution from a simple elementary distribution, like the multivariate Gaussian distribution. This can then be used to model the probability distribution of images of faces (Kingma and Dhariwal, 2018) or time domain representation of speech (Prenger et al., 2019). To obtain samples from this distribution we sample from the multivariate Gaussian distribution and use the generative flow to obtain a sample of the desired distribution.

The flow is trained by applying the inverse of the flow to the training samples and using a loss that constraints the output to be Gaussian distributed. For this the flow needs to be invertible, which is not the case for arbitrary neural networks. A flow consists of (invertible) affine operations and (bijective) activation functions, but the weights of the affine operations are not trainable weights — the weights of the affine operations are calculated through a convolutional neural network from a subset of the input. The subset that is used to calculate the affine transformation is
passed unchanged to the next layer. Thus, the affine operation can be reconstructed
on the other side, making each layer invertible.

The drawback of this architecture is that each layer in the flow requires a whole
neural network to calculate its weights. Therefore, neural networks based on gen-
erative flows tend to be extremely large and training and inference are relatively
slow.

Differentiable digital signal processing

Many elements of digital signal processing are differentiable and can thus be used
as elements in a neural network. In particular the parameters of filters, windows,
wavetable synthesis and additive- and subtractive synthesis can be learnt as weights
of a neural network or be computed by a neural network. During training, the
gradient can be transported through these operations allowing us to apply losses
on synthesized signals. This hybrid approach between digital signal processing and
deep learning is called *differentiable digital signal processing* (DDSP) (Engel et al., 2019).
DDSP allows integrating prior knowledge about the signal into the architecture. For
example, melody instruments like the flute can only play one note at a time, so it
makes sense to restrict the synthesis to use only one fundamental frequency. At the
same time we get interpretable parameters. In the example with the flute, we would
obtain the fundamental frequency, which corresponds to the note that is being
played. The imposed restriction can guide convergence of the models while also
being very expressive. We can learn hundreds of filter coefficients or wavetables
with hundreds of points as functions of the input parameters.

A drawback of this approach is that the assumptions may be too restrictive for
special cases. In speech, we generally assume that the glottal pulses happen peri-
odically following a fundamental frequency. This is true during stable phonation
but is violated in rough speech, creaky voice and even at the boundary between
voiced and unvoiced speech. In these modes of speech production glottal pulses
may happen irregularly either through varying amplitude, varying pulse positions,
or missing of some pulses. By means of over-parametrisation,<fnref n="8" /> a DDSP model could
in theory model many of these effects as well, but convergence is unlikely as these
re-parametrisations tend to lie in different local minima.

2.5. Datasets

Deep learning relies on the availability of large datasets. Since all models are learnt
from data we cannot expect to produce much more than what is contained in the
data. Any deep learning model reflects the data it has been trained with. Thus, it is
worth to pay closer attention to the datasets that were available during the time
this research carried out.

<footnote n="8">8. We could for example construct an equivalent representation for wavetable synthesis with wavetable *w* and frequency *f* by concatenating *w* to itself (and resampling) and dividing *f* by 2.</footnote>
Most of the time we treat singing voice and speech separately due to their differences. Voice conversion systems for singing voice have rather different requirements as voice conversion systems for speech. As both forms of voice rarely occur together it is justified to develop different models for either voice type if the quality of both models can be increased by separating the voice types. Thus, we will have different datasets for singing voice applications and for speech applications. A lot of datasets with voice are publicly available for both singing voice and speech.

### 2.5.1. Speech datasets

Speech datasets are available in large quantities. Since consumer microphones are widely available these days, anyone can record themselves reading a book or a newspaper. Projects like LibriVox<fnref n="9" /> create large corpus of audiobooks by crowdsourcing the recording process, which can be used to generate large speech datasets like LibriSpeech (Panayotov et al., 2015). Furthermore, vast amounts of voice recordings can be easily found on video hosting platforms and as podcasts. Projects like Vox-Celeb (Nagrani et al., 2019) collect voice from celebrities from the internet resulting in thousands of hours of speech.

The result, however, of these large scale efforts are huge amounts of data with poor quality. In the case of amateurs reading audiobooks the recording conditions may be suboptimal resulting in high background noise or distortions from cheap recording hardware. Additionally, through audio compression further artefacts are introduced. In the case of videos from the internet we face the difficulty that voice is often not isolated. Additional signals such as sound effects, background music, background noise or other speakers interrupting each other result in unclean data. Furthermore, the same difficulties as before arise if the videos are created by amateurs. Due to the huge size of these datasets it is impossible to ensure consistent quality.

Depending on the application, similar arguments may apply to the input data processed by the user. If a user wishes to transform a recording that has been made under poor recording conditions, the transformation might fail if the transformation system has never seen unclean data. Thus, to increase robustness it might actually be desirable to include noisy or poor quality data.

On the other hand, deep learning approaches reproduce the data they are trained with. Unclean training data degrades the quality of the derived models as it may teach the model to generate noisy sounds as a result of a transformation. Thus, if the goal is to process and produce high-quality audio, these crowdsourced large-scale datasets may not be suitable.

There are a few high-quality datasets available for speech. The datasets that satisfy our quality demands and were thus used in this thesis are summarized in Table 2.1 and described below:

**Att-HACK** Database for French expressive speech recorded by Moine and Obin (2020). The database includes 20 speakers speaking a set of phrases in four

<footnote n="9">9. https://librivox.org</footnote>
<table><caption>**Table 2.1.:** Speech datasets used in this thesis.</caption><thead><tr><th>Name</th><th>Language</th><th>#Spk.</th><th>Dur.</th><th>Credit</th></tr></thead><tbody><tr><td>Att-HACK</td><td>French</td><td>20</td><td>21:51:55</td><td>(Moine and Obin, 2020)</td></tr><tr><td>BREF120</td><td>French</td><td>120</td><td>~ 100 h</td><td>(Gauvain et al., 1990)</td></tr><tr><td>BREF80</td><td>French</td><td>80</td><td>10:28:45</td><td>(Gauvain et al., 1990)</td></tr><tr><td>CMU Arctic</td><td>English</td><td>3</td><td>2:42:56</td><td>(Komelek and Black, 2004)</td></tr><tr><td>TIMIT</td><td>English</td><td>630</td><td>5:11:03</td><td>(Zue et al., 1990)</td></tr><tr><td>VCTK</td><td>English</td><td>110</td><td>33:47:56</td><td>(Yamagishi et al., 2019)</td></tr><tr><td>LJ Speech</td><td>English</td><td>1</td><td>23:55:17</td><td>(Ito and Johnson, 2017)</td></tr></tbody></table>

different attitudes, friendly, seductive, dominant and distant.

**BREF120** Large commercial database containing 100 hours of French speech from 120 speakers. The speakers read from a newspaper.

**BREF80** Smaller subset of the BREF120 database, containing only 80 speakers and less material per speaker.

**CMU Arctic** A dataset created for speech synthesis at Carnegie Mellon University by Komelek and Black (2004). The dataset is accompanied with electroglottograph (EGG) measurements which allow detection of pulse positions.

**TIMIT** A dataset created for speech recognition in 1990 by Zue et al. (1990). The dataset contains 630 native English speakers covering a large variety of American dialects.

**VCTK** Multispeaker dataset recorded by Yamagishi et al. (2019). The dataset includes speech data from 110 English speakers. Note that in this dataset each utterance is recorded with two microphones. Thus the total available audio is twice as much as the total duration.

**LJ Speech** Single speaker dataset by Ito and Johnson (2017). The speaker is reading 7 non-fiction books which amounts to approximately 24 hours of speech.

### 2.5.2. Singing datasets

Singing datasets are much rarer than speech datasets. There are multiple reasons behind this. First, the research community around singing voice is much smaller than for speech. Thus, much fewer datasets are produced with singing voice than there are for speech. Recording high quality databases requires a lot of money to pay for the artists, technical equipment and staff to operate the equipment. Although more and more commercial applications of singing voice technology (and music technology in general) have emerged in recent years, the market for speech technology is still much bigger (and probably always will be). Thus also funding from the private sector is much larger for speech databases.
Another reason is how we encounter singing voice in the wild. While we generally appreciate isolated speech, as it maximizes intelligibility, the opposite is true for singing voice. Solo a cappella singing is very rare in music. Singing voice is in most cases accompanied by other instruments, other voices or sound effects. Thus finding isolated singing on the internet yields much fewer data than for speech and thus makes projects like VoxCeleb unfeasible.

Finally, singing requires much more physical effort than speaking. While it can be exhausting to speak non-stop for hours, it is almost impossible for singing. It takes years of practice for professional singers to be able to sing through an opera of several hours. Even then, scores are deliberately written in a way that allows enough breaks for each role so that a singer does not have to sing for too long. Thus recording the same duration of singing requires much more effort than speech which means that singing databases generally tend to be shorter.

Additional difficulty arises from the variability in singing techniques. Different singing styles have emerged in different cultures and for different purposes. Each of these styles typically consists of a set of different possible singing techniques. Some of these techniques may overlap between different styles while some may be unique to a specific style and sometimes the same technique is used in a different context in different singing styles. To represent singing voice in general a comprehensive dataset needs to cover all different singing styles in addition to the variety of voices. Thus, a representative singing voice dataset should be even larger than a speech dataset with a similar ambition. Alternatively a singing voice dataset may focus on only one singing style. Most singing voice datasets opt for the second choice while other datasets try to cover a large variety of singing styles and consequently under-represent all of them.

Many of the existing singing voice datasets have been recorded for different purposes than deep learning. Concatenative synthesizers like Vocaloid or the IRCAM Singing Synthesizer (Ardaillon, 2017) require all diphones of a language to be recorded by one singer. The desired phonetic sequence is generated by concatenating the required diphones and the desired melody is obtained by transposition. The required dataset is recorded on a constant pitch and with minimal expression. These datasets are relatively large and common but are not optimal for deep learning as they do not represent real singing; they are samples of how singing voice sounds like but do not reflect how singers would interpret a piece and lack expressive features like $f_0$ contours and articulations.

The datasets that were available for this thesis are summarized in Table 2.2 and described below.

**CREL** A Research Database recorded by Tsirulnik and Dubnov (2019). It contains singing voice from three professional singers, singing scales and the song *Twinkle Twinkle Little Star* in different singing styles.

**NUS-SMC** The NUS-48E Sung and SpokenLyrics Corpus, a dataset from Duan et al. (2013) at the Sound and Music Computing Lab at National University of Singapore. It consists of singing and speech from 9 amateur singers.
<table><caption>**Table 2.2.:** Singing datasets used in this thesis.</caption><thead><tr><th>Name</th><th>Language</th><th>#Spk.</th><th>Dur.</th><th>Credit</th></tr></thead><tbody><tr><td>CREL</td><td>English</td><td>3</td><td>1:48:45</td><td>(Tsirulnik and Dubnov, 2019)</td></tr><tr><td>NUS-SMC</td><td>English</td><td>9</td><td>1:18:34</td><td>(Duan et al., 2013)</td></tr><tr><td>ICH</td><td>Greek</td><td>1</td><td>2:49:27</td><td>(Grammalidis et al., 2016)</td></tr><tr><td>PJS</td><td>Japanese</td><td>1</td><td>30:44</td><td>(Koguchi et al., 2020)</td></tr><tr><td>JVS-MuSiC</td><td>Japanese</td><td>100</td><td>1:36:24</td><td>(Tamaru et al., 2020)</td></tr><tr><td>Tohoku</td><td>Japanese</td><td>2</td><td>3:44:46</td><td>(Ogawa and Morise, 2021)</td></tr><tr><td>VocalSet</td><td>Multiple</td><td>20</td><td>8:35:17</td><td>(Wilkins et al., 2018)</td></tr><tr><td>ISiS</td><td>French</td><td>3</td><td>4:53:08</td><td>(Ardaillon, 2017)</td></tr><tr><td>Farinelli</td><td>Italian</td><td>7</td><td>6:30:37</td><td>(Deschamps, 2022a)</td></tr></tbody></table>

**ICH** Byzantine singing from the i-Treasures Intangible Cultural Heritage dataset created by Grammalidis et al. (2016). It consists of 4 hours of Byzantine singing sung by a single singer in baritone range.

**PJS** Phoneme-balanced Japanese singing-voice corpus by Koguchi et al. (2020). The dataset consists of 100 phrases both sung and spoken by the same singer. The singing is pop tenor voice.

**JVS-MuSiC** Japanese multispeaker singing-voice corpus by Tamaru et al. (2020). The dataset contains singing voice from 100 amateur singers singing two songs. One song is shared by all singers and the other may be different for each singer.

**Tohoku** Two voices, Tohoku Kiritan and Tohoku Itako, from the singing synthesisers Voiceroid and Utau. Recordings of the Tohoku Kiritan voice include singing of real pop songs which have been made available by Ogawa and Morise (2021).

**VocalSet** A singing voice dataset with 20 professional singers made available by Wilkins et al. (2018). The dataset includes vocalises as well as songs from folklore, opera and clerical repertoire.

**ISiS** Datasets recorded by Ardaillon (2017) for the IRCAM Singing Synthesizer. It consist of singing from three French professional singers from opera, jazz and pop. Most of the singing is performed on constant pitch but several real songs are also included.

**Farinelli** Dataset recorded by Judith Deschamps for her artistic residency at IRCAM. Seven singers were recorded singing the song *Quell’usignolo* by Giacomelli as well as vocalises and songs from their own repertoire.

### 2.5.3. Combining datasets

Some datasets presented here are rather small and in particular for singing, none of those datasets is large enough to train models that are general enough for consumer
purposes. To increase the effective dataset size we combine datasets to obtain a
wider coverage of different singing styles, languages, voice identities etc. We can
even combine singing and speech to train universal voice models that work on any
type of voice.

There is no uniform standard in dataset organization thus each dataset comes in
its own format. The simplest form is a folder that contains all the recordings of the
dataset split into short audio files. Often, however, datasets come with additional
metadata. Multi-speaker and multi-singer datasets typically annotate their files
with the speaker / singer information. Especially in speech, phonetic or word
transcriptions are common and can be provided in a separate file for each audio file
or in a single file containing all the text. Some datasets are even unprocessed or
contain long files that are unhandy and thus need to be cut into smaller files.

To standardize all the metadata we developed an ecosystem for databases. Main-
taining this framework is a significant part of necessary housework for the re-
searcher in machine learning and an ongoing chore as every new database needs to
be integrated in the unified framework. The standardized framework can be seen as
a contribution of this thesis to other research within our research group as many of
the preprocessing scripts have been reused in other research projects. We extended
the existing database structure with import scripts, notably with the possibility
to automatically cut long files during silences and remove excessive leading and
trailing silences around the imported audio files.

Sampling

When working with combined datasets we have to address the question of sampling.
If we combine for instance a very large dataset in one language with another not
so large dataset in another language we might over-represent the language of the
larger dataset if we sample uniformly from all training samples. Similarly, some
datasets have shorter phrases and thus consist of many short audio files while others
might consist of longer audio files. If we uniformly sample from all audio files we
will over-represent the dataset which contains short audio files. The effect of the
latter can be mitigated by cutting all audio files into the same length, but this might
not be desired as it would require cutting within a speech- or musical phrase.

In other cases we want to over-represent some samples, for instance, to ensure
that even rare cases are synthesized reasonably well. For example when we were
developing the mel-inverter we had the problem that singing voice above 700 Hz
(F5) would not work well (or at all) although typical soprano ranges go up to 1200 Hz
(D6) or even higher and samples of this were included in the training data. However,
inspecting the training data for these models revealed that far less than 1% of all the
audio frames in the training set contained singing with a fundamental frequency
above 700 Hz. Thus, the error on these frames was insignificant for the training loss.
The issue was resolved by drawing audio samples that contained *high-pitched* singing
far more often.

Whether we need to equalize or emphasize some properties in the training data,
a good sampling strategy is essential. Which sampling strategy is appropriate will
<figure><img src="image_20.png" /><figcaption>**Figure 2.11.:** Pie charts showing the distribution of different attributes in the combined singing dataset. Clockwise from top left: Voice type, gender, database, and language.</figcaption></figure>

depend on the application and needs to be found through experimentation.

### 2.5.4. Dataset analysis

The neural models that we train are reflections of the data we train them with. They distil information from the training set. It is therefore essential to be very familiar with the datasets that we use and be aware of their properties.

We start our analysis with the singing databases. As we will almost always train our effects on the combined database of the databases from Table 2.2, we will analyse how the databases mix together. The distribution of different attributes over the duration can be seen in Fig. 2.11. We see that the dataset is astonishingly well-balanced between male and female. This is surprising as most of the databases themselves are not well-balanced, not even the ones containing many singers. We can see that the dataset is dominated by soprano, baritone and female pop. Within pop singing, the dataset is strongly biased against female singers. This is because we have 3 female but only one male pop singer and the database with the male pop singer (PJS) is not very large. The soprano range is dominating against other female classical voices as it is the most common female classical voice and is represented by
<figure><img src="image_21.png" /><figcaption>**Figure 2.12.:** Histogram of fundamental frequency in different datasets. The singing dataset consists of all the databases from Table 2.2. The speech dataset is a combination of VCTK and Att-HACK as used in Chapter 6. The histogram has a resolution of 6.25 cent and is normalized by width.</figcaption></figure>

11 distinct singers while Alto and Mezzo only have 2 each. Similarly, in our dataset the baritone range is most common among male classical voices. In this case, this is due to two singers with a large number of recordings, (from ICH and ISiS) who together already amount for 17% of the total voices. Finally, we can see that the dataset is well-balanced between 5 languages, Japanese, English, Italian, Greek and French, while a large portion is unknown. However, this distribution should not be taken too seriously as much of the singing is in coloraturas or vocalises which is rather different from the actual language but was not treated separately in this analysis.

A lot of time will be spent discussing the fundamental frequency of the voiced segments in particular as we will be changing the fundamental frequency in Chapter 6. Thus, it is worth to have a look at the distribution of the fundamental frequency inside the dataset. We obtain the fundamental frequency annotation by using the analysis method proposed by Ardaillon and Roebel (2019). Histograms of the fundamental frequency for different datasets can be seen in Figs. 2.12 to 2.14. One big difference between speech and singing is the distribution of the *f*₀, as can be clearly seen in Fig. 2.12: In singing we generally observe higher *f*₀ than in speech. For both speech datasets (see Fig. 2.13) we can identify two distinguishable modes corresponding to male and female speakers. The modes for male speakers are very similar between VCTK and Att-HACK, while the distribution of *f*₀ for the female speaker has a lower average for the Att-HACK database and a larger variance.

In the histogram of the full singing voice dataset, Fig. 2.12, a striking feature are the peaks at different locations, most prominently at E4, G4, C4 and C5. Further peaks can be found at D4, A4, C3 and closer inspection also reveals peaks that can
<figure><img src="image_22.png" /><figcaption>**Figure 2.13.:** Histogram of fundamental frequency in speech different datasets. Both histograms show two modes for male and female voices respectively.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>**Figure 2.14.:** Histogram of fundamental frequency in different singing datasets. The ISiS dataset was recorded for concatenative singing synthesis and shows the sparse distribution of pitches for those datasets. The Farinelli dataset consists of singing of mostly the same song, Quell'usignolo, the song is in C-major, and we see peaks for many notes from the C-major scale.</figcaption></figure>
be associated with other notes from the 12-tone scale. All large peaks are part of the C major scale. Furthermore, the most prominent peaks are from the C major triad (C, E, G). This can be explained by the fact that most of the material in the dataset is in fact recorded in C major or some related key that shares most of the scale with C major. A significant amount of data comes from datasets which were recorded specifically for concatenative synthesis (most of ISiS, in part Tohoku) where coverage of phonetic material is far more important than coverage of pitches. Thus, in those datasets only one or a few pitches are represented, which explains the most prominent peaks (C3, D4, E4, G4, A4, C4). The difference between a dataset for concatenative singing synthesis and a dataset of real singing is demonstrated in Fig. 2.14 where we compare ISiS, a dataset recorded for concatenative singing synthesis, versus Farinelli, a dataset containing real singing (mostly in C major).

Another interesting observation from Fig. 2.12 is that although the soprano range typically exceeds C6 (1047 Hz) and frequently reaches up to F6 (1400 Hz), fundamental frequencies above G5 (784 Hz) rarely occur in the dataset, even though 1/6th of the recordings come from actual sopranos. Similarly, for the lower notes we observe that below B2 (124 Hz) there are only very few recordings. As discussed in the previous section, the lack of representation of extreme pitches within the dataset is an issue that we will have to overcome for both mel-inversion (Chapter 5) and pitch transformation (Chapter 6).

## 2.6. Perceptive tests

When studying synthesis methods, we investigate how to create something new. This can lead to fascinating results and gives us new possibilities in creativity. The drawback of this kind of discipline is that the problem is often not defined exactly which leads to multiple equally valid but very different solutions. As a consequence evaluating the results and by extension evaluating the solution methods is subjective. To allow for scientific rigour we rely on reliable metrics that allow comparing one method to another. This is typically done in perceptive tests. A subjective quality can be measured by asking human participants for their subjective preference. Most of the time we want to evaluate the audio quality, but we might also ask about other qualities like how loud someone is singing, as we did in Chapter 7, or the similarity to a certain speaker, as is common for voice identity conversion.

### 2.6.1. Conception

The International Telecommunication Union (ITU) provides a recommendation for subjective testing in the ITU-T P.800 and ITU-R BS.1534-3 recommendations. ITU-T P.800 gives guidelines for evaluating telecommunication systems and measuring the subjective degradation. ITU-R BS.1534-3 give recommendations how to subjectively evaluate the intermediate quality of audio systems. We can borrow many of the recommendations from ITU-T P.800 and ITU-R BS.1534-3 for our own subjective test design.
## Common test designs

In a perceptive test we let participants (test subjects) evaluate randomly drawn audio samples (test samples) generated by the synthesis methods we want to evaluate. The provided scores are averaged over all test subjects and test samples to generate a mean opinion score (MOS). Recommendations ITU-T P.800 and ITU-R BS.1534-3 provide a few common test designs:

**Absolute Category Rating (ACR)** According to ITU-T P.800 the quality of a sound recording can be quantified by translating the following subjective values into the corresponding numbers:

<table><thead><tr><td>Quality of the speech</td><td>Score</td></tr></thead><tbody><tr><td>Excellent</td><td>5</td></tr><tr><td>Good</td><td>4</td></tr><tr><td>Fair</td><td>3</td></tr><tr><td>Poor</td><td>2</td></tr><tr><td>Bad</td><td>1</td></tr></tbody></table>

**Degradation Category Rating (DCR)** The vocoder can be seen as a communication channel which introduces degradation with respect to the original recording. From this perspective on can employ a degradation scale instead of the quality scale using a Degradation Category Rating (DCR) (ITU-T P.800, Annex D)

<table><thead><tr><td>Degradation</td><td>Score</td></tr></thead><tbody><tr><td>Degradation is inaudible</td><td>5</td></tr><tr><td>Degradation is audible but not annoying</td><td>4</td></tr><tr><td>Degradation is slightly annoying</td><td>3</td></tr><tr><td>Degradation is annoying</td><td>2</td></tr><tr><td>Degradation is very annoying</td><td>1</td></tr></tbody></table>

**Comparison Category Rating (CCR)** When comparing two synthesis methods against each other it can be advisable to compare samples from both methods directly through a Comparison Category Rating (CCR) (ITU-T P.800, Annex E):
<table><thead><tr><td>Comparison</td><td>Score</td></tr></thead><tbody><tr><td>Much better</td><td>3</td></tr><tr><td>Better</td><td>2</td></tr><tr><td>Slightly better</td><td>1</td></tr><tr><td>About the same</td><td>0</td></tr><tr><td>Slightly worse</td><td>-1</td></tr><tr><td>Worse</td><td>-2</td></tr><tr><td>Much worse</td><td>-3</td></tr></tbody></table>

A CCR is particularly useful if we want to evaluate the impact of a specific hyper-parameter to our model by comparing two versions of the model with only the hyperparameter changed.

**MUSHRA** The Multi Stimulus test with Hidden Reference and Anchor (MUSHRA) test is recommended by the ITU-R BS.1534-3. Grading is done on a continuous scale (typically discretized by steps of one) from 0 to 100 with the following nominal correspondences (ITU-R BS.1534-3)

<table><thead><tr><td>Quality of the speech</td><td>Score</td></tr></thead><tbody><tr><td>Excellent</td><td>80-100</td></tr><tr><td>Good</td><td>60-80</td></tr><tr><td>Fair</td><td>40-60</td></tr><tr><td>Poor</td><td>20-40</td></tr><tr><td>Bad</td><td>0-20</td></tr></tbody></table>

Test subjects are asked to rate the samples from the models that we want to evaluate together with original, unprocessed recordings (hidden reference) as well as two hidden anchors that undergo standardized degradation. The degradation is achieved by low-pass filtering at 3.5 kHz and 7 kHz respectively (ITU-R BS.1534-3, Section 5.1). The test subjects are not told which sample comes from which source.

### 2.6.2. Procedure

Standards ITU-T P.800 and ITU-R BS.1534-3 both recommend a controlled test environment with the same equipment for each participant. While a controlled test environment yields the most consistent results, this requirement makes the evaluation expensive and thus impractical for our purposes. The progress in our domain is fast-paced with new models being created every week. We need to perform perceptive tests regularly to obtain indications which methods really provide improvements. The ITU recommendations are designed for testing consumer products with anticipated longer life-spans. In that context it makes sense to invest more money to obtain a reliable rating of the finished product. In our domain, new improvements can appear every few weeks making results of perceptive test viable for much shorter time.
For us the trade-off between cost of testing vs. frequency of testing is leaning more
towards frequency rather than quality. Therefore, we permit ourselves to deviate
from the recommendations with regard to testing environment. This requires more
strict post-screening which will be discussed in Section 2.6.3. To be able to reach as
many participants as possible our test are carried out online, thus allowing many
people to participate in a short time for relatively low cost.

Our online test consist of a web-page that presents audio samples and requires
the participant to give a rating with respect to the quality to be evaluated. Fig. 2.15
shows a screenshot of the perceptive test from Chapter 7. The test starts with an
explanation about what is to be tested together with some instructions how to carry
out the test. To evaluate the quality of the synthesized samples from Chapters 6
and 7, we perform ACR tests and let the test subjects evaluate audio samples on the
perceptive scale from 1 to 5. The order of the samples is randomized according to
ITU-R BS.1534-3, Section 3. Furthermore, MUSHRA is performed to evaluate the
quality of the audio samples generated by the model in Chapter 5, and we use a
modified CCR to compare evaluate the effect of the voice transformation in Chapter 7
(depicted in Fig. 2.15).

The ITU-R BS.1534-3 standard recommends expert listeners as test subjects. Using the same argument as before we permit ourselves to deviate from this recommendation to allow faster and more frequent testing. The ITU-T P.800 standard recommends ‘Subjects [...] are chosen at random from the normal telephone using population’ (ITU-T P.800, Section A.4.1). Thus, according to ITU-T P.800, any person that is able to hear would be a suitable test subject. To recruit many participants in a short time we use an online platform to hire paid test subjects. The platform allows targeting specific demographics and hiring people with specific skills. This is especially useful when evaluating samples containing speech as a high proficiency is required in the language that is being synthesized. According to ITU-T P.800 in order to get people to do the tests carefully the test should not be too long so that the participants are finished before they get tired. It is recommended that ‘Ideally no session should last for more than 20 minutes [...]’ (ITU-T P.800, Section B.3). However, we found that for internet based tests even short durations are advisable and believe durations between 5 and 10 minutes are best to get valuable answers.

**2.6.3. Evaluation**

Post-screening

The studies are carried out online by anonymous participants. Carrying out the
study online allows us to reach a lot of people from a wide range of demographics
for a relatively cheap price. On the other hand we have no control over the test
environment. Participants are advised to take the test in a quiet environment using
good quality headphones, but we have no means to make sure these instructions
are properly honoured. Since test participants earn more money the more surveys
they participate in, there is an incentive for them to finish the test quickly. As a
consequence some submissions may be of bad quality. This could be either because
participants did not read the instructions properly, because they didn't take the
<figure><img src="image_24.png" /><figcaption>**Figure 2.15.:** Screenshot of the perceptive test website from Chapter 7. The test follows the typical design pattern: First a few instructions are given on how to carry out the test (yellow box). After that the actual test follows. This test does pairwise comparisons and the participant is asked to rate which one has been sung in a "stronger" voice.</figcaption></figure>
time to properly listen, or because they could not hear the audio samples well due to inadequate environment or equipment. Some participants might even give random answers just to finish the test as quick as possible. Thus, to improve the quality of the results, it might be necessary to remove submissions with remarkable abnormality.

The ITU-R BS.1534-3 standard recommends a post-screening of the test subjects by evaluating their response of the hidden reference. Subjects that rated the hidden reference too often too low should be excluded from the aggregated scores if they rated more than 15% of the hidden references with a score lower than 90 (on a scale from 0 to 100) (ITU-R BS.1534-3, Section 4.1.2). With this approach we found that we can significantly improve the test accuracy in terms of confidence intervals while losing only few (less than 5%) of the submissions.

## Confidence intervals

A perceptive test is a very unreliable measurement method. Obviously, since responses are subjective and every participant has a different background, different expectations, tastes, and habits responses will vary enormously, and we can expect frequently contradicting responses even from the same person. Thus, it is important to quantify the quality of the estimate itself. The ITU-T P.800 standard recommends the mean opinion score to be accompanied by 95%-confidence intervals. Rather than providing a single number as our estimate, we provide an interval within we believe the true value to be in with 95% certainty.

Derivations for 95%-confidence intervals can be found in standard literature on statistics, e.g. in EMS (2020). A typical derivation goes as follows: Assume we get a set of $n$ ratings $X = (X_1, ..., X_n)$ and assume that $X_1, ..., X_n$ are independent and identically distributed Gaussian random variables with distribution $X_1 \sim N(\mu, \sigma^2)$ with unknown mean $\mu$ and unknown standard deviation $\sigma$. Thus, the mean $\mu$ represents the mean opinion score, the quantity we want to estimate. The standard deviation $\sigma$ represents how controversial the opinion is, that is, if $\sigma = 0$ everyone agrees whereas the higher $\sigma$ is, the more different the opinion of each individual is. A high standard deviation $\sigma$ does not necessarily mean that the confidence interval will be large. The confidence interval measures how sure we are that the mean lies within a certain region but makes no statement about how controversial this mean is. As we will see in the derivation, however, the standard deviation $\sigma$ has an influence on the confidence interval insofar as a higher $\sigma$ requires more samples to achieve a small confidence interval.

We are thus interested in an estimator $M_n : \mathbb{R}^n \to \mathbb{R}$ such that the expectation $E[M_n(X)] = \mu$ and the variance $V[M_n(X)]$ is minimal. To estimate $\mu$ we use the arithmetic mean, thus our estimate $\bar{X}$ of $\mu$ is

$$ \bar{X} = M_n(\mathbf{X}) = \frac{1}{n} \sum_{k=1}^{n} X_k. \qquad (2.32) $$

The estimate $\bar{X}$ is itself a random variable and since the Gaussian distribution is
stable, $\bar{X}$, as a sum of Gaussian random variables, is Gaussian again. In particular

$$ \bar{X} \sim \mathcal{N}\left(\mu, \frac{\sigma^2}{n}\right). \qquad (2.33) $$

Since $\bar{X}$ is a continuous random variable it is unlikely that the estimate $\bar{X}$ is exactly correct ($\mu \neq \bar{X}$ almost certainly). In fact the error can be given as

$$ \bar{X} - \mu \sim \mathcal{N}\left(0, \frac{\sigma^2}{n}\right) \qquad (2.34) $$

and thus using the 2.5% quantile $q \approx 2$ of the standard normal distribution we get

$$ 95\% = \mathbf{P}(\bar{X} - \mu \in [-c, c]) \qquad (2.35) $$

$$ = \mathbf{P}(\mu \in [\bar{X} - c, \bar{X} + c]), \qquad (2.36) $$

where $c = q\sigma/\sqrt{n}$. Thus, with 95% probability the real mean $\mu$ lies within the interval

$$ \left[ \bar{X} - \frac{q\sigma}{\sqrt{n}}, \bar{X} + \frac{q\sigma}{\sqrt{n}} \right]. \qquad (2.37) $$

In most cases, however, the standard deviation $\sigma$ is unknown as well and thus the confidence interval from Eq. (2.37) is useless for practical purposes. We can estimate the variance as

$$ S^2 = \frac{1}{n-1} \sum_{k=1}^{n} (X_k - \bar{X})^2, \qquad (2.38) $$

but now the estimated variance is itself a random variable and thus Eq. (2.34) does not make sense if we replace $\sigma^2$ with $S^2$. We normalize $\bar{X} - \mu$ by its estimated standard deviation and observe that

$$ T := \frac{\bar{X} - \mu}{S/\sqrt{n}} \qquad (2.39) $$

follows Student's t-distribution with $n-1$ degrees of freedom (Schmetterer, 2012, Section I.29). Thus, we can obtain the confidence interval from

$$ 95\% = \mathbf{P}(T \in [-q_{n-1}, q_{n-1}]) \qquad (2.40) $$

$$ = \mathbf{P}\left(\frac{\bar{X} - \mu}{S/\sqrt{n}} \in [-q_{n-1}, q_{n-1}]\right) \qquad (2.41) $$

$$ = \mathbf{P}\left(\mu \in \left[\bar{X} - \frac{q_{n-1}S}{\sqrt{n}}, \bar{X} + \frac{q_{n-1}S}{\sqrt{n}}\right]\right), \qquad (2.42) $$

where $q_{n-1}$ is the 2.5% quantile of Student's t-distribution with $n-1$ degrees of freedom. Thus, we can say that with 95% probability the real mean $\mu$ lies within the interval

$$ \left[ \bar{X} - \frac{q_{n-1}S}{\sqrt{n}}, \bar{X} + \frac{q_{n-1}S}{\sqrt{n}} \right]. \qquad (2.43) $$
**2.7. Conclusion**

Voice synthesis and transformation as a scientific research topic has a history of over 100 years. In this period we came to gain substantial insight about the voice such that today we have a broad and deep understanding of human hearing, the speech production mechanism and acoustical properties of voice. Nevertheless, voice transformation still finds its limitations for extreme invasive changes resulting in poor audio quality or in inaccurate changes.

In the history of voice research we have seen on several occasions how emerging
technology can revolutionize the field and allow approaching the issues from a com-
pletely different angle, resulting in new possibilities that were unthinkable before:
Electronic circuitry allowed synthesis in small compact devices (Stewart, 1922) and
allowed the development of the vocoder (Dudley, 1939). Digital signal processing
allowed storing the voice in computers and performing complex transformations on
the voice. Deep learning allows modelling stochastic relationships and extracting
those from collected data. We are still in the process of rethinking our perspective
on voice due to the possibilities of deep learning.

In this chapter we had a look at the underlying theories about how we perceive
sound, how human voice is produced and what are its physical properties. We have
seen a short introduction to neural networks, the main tool used in this thesis, and
how they are used in audio signal processing. As deep learning is relying on its
training data, we had a closer look at the available data. Finally, we discussed how
we can evaluate synthetic voice by means of perceptive tests.
## 3. State of the art

In this chapter we will discuss the state of the art of voice processing. The central tool for processing the voice is the vocoder which is or particular interest for voice transformation. The vocoder identifies voice recordings with parameters in a parametric space: It consists of two conceptually inverse operations. The *analysis* operation extracts voice parameters from the voice recording. The *synthesis* operation uses the extracted voice parameters to reconstruct the original signal per-ceptually. We can approach the vocoder from different angles. First, in Section 3.1, we will discuss how different voice properties can be extracted from a voice signal and which analysis methods can be used for that task. Then, in Section 3.2, we will discuss how extracted voice properties can be used to resynthesize voice through different vocoders, and we will discuss the strengths and drawbacks of state-of-the-art vocoders. As the parametric space of a vocoder is used to parametrize the resynthesis operation, the parameters tend to describe the acoustic properties of the signal rather than abstract concepts, like text and melody. To synthesize voice from text or a musical score, more elaborate techniques are necessary, which we will discuss in Section 3.3. Similarly, vocoders can be used to transform some properties of the voice directly but find their limitations quickly as they do not automatically model the relationships between different parameters. Thus, in Section 3.4 we will discuss how a vocoder can be used to transform the voice and when it can be avoided.

### 3.1. Analysis of voice

Signal analysis is the extraction of specific information from a signal. The extracted information can be global, as is the case when detecting the speaker of an utterance. Other information changes over time with the signal and is itself a signal (typically with a lower sample rate). An example of a time-varying voice property is fundamental frequency of voiced speech which is estimated on short signal frames. The variety of signal analysis applications is large and ranges from low level parameter extraction for vocoders and parametric voice synthesis to high level voice processing like singing voice transcription or automated speech recognition.

The extracted signal information parametrizes the underlying signal model. Generally, we assume the source-filter model (Fant, 1981; Markel and Gray, 2013) which we discussed in Section 2.3. Differences in signal models originate from how we model the glottal source and the noise component of the voice. The glottal source can be represented either as a sequence of pulses (Huber and Roebel, 2015) or by a combination of harmonically related sinusoids (Stylianou et al., 1995). The noise component can be modelled non-parametrically (leaving it as is), or through a filter
ter applied to white noise. In the following subsections, we will present the voice
parameters that are used in or related to this thesis.

**3.1.1. Fundamental frequency**

The fundamental frequency, often denoted by $f_0$ or F0, comes from the assumption that an oscillator generates a quasi-periodic excitation signal with frequency $f_0$. In the source-filter model this quasi-periodic excitation is then further transformed to produce the observed signal. This model is not constrained to only voice. Many signals follow this pattern so classical signal processing offers a wide range of $f_0$ analysis tools. These can be categorized into three different approaches, time-based analysis, that typically work on the auto-correlation function (Sondhi, 1968; Rabiner, 1977; Cheveigné and Kawahara, 2002; Mauch and Dixon, 2014), analysis on the (log-)spectrum (Schroeder, 1968; Duifhuis et al., 1982; Hermes, 1988; Sun, 2000; Camacho, 2007), or analysis on the cepstrum (Noll, 1967). An overview over the different classical $f_0$ analysis techniques can be found in Camacho (2007, Chapter 2) and a qualitative study was done by Babacan et al. (2013a).

Recently, neural networks have been used to create domain specific fundamental frequency estimators for musical instruments (J. W. Kim et al., 2018; Singh et al., 2021) and speech (Ardaillon and Roebel, 2019; Roebel and Bous, 2022). As these methods are trained on and applied to specific types of signals (specific musical instruments or voice) they can use additional information, e.g. from the timbre, to improve the accuracy or resolve ambiguities, which could not be done in the general case. The pitch estimation works on raw audio (J. W. Kim et al., 2018; Ardaillon and Roebel, 2019) as well as on the mel-spectrogram (Roebel and Bous, 2022) and might be trained on any other representation of voice.

**3.1.2. Glottal pulse signal**

An alternative characterization of the glottal excitation signal is through the glottal pulses. Here, we don't necessarily assume periodic excitation, but rather simply a set of pulses (with position and possibly shape). The characteristic pointy tip of the glottal closure instant $t_e$ (see Fig. 2.8 in Section 2.3.2, Page 19) allows us to define the pulse position in terms of the glottal closure instant (GCI). Using a specific glottal pulse model allows parametrizing the glottal pulse shape.

The behaviour of the vocal folds can be analysed by electroglottographic (EGG) measurements, where the electric resistance through the glottis is measured by electrodes placed on the laryngeal prominence. The electric resistance through the glottis depends on the contact area of the vocal folds. Thus, during phonation an oscillation can be measured and the glottal closure instant can be obtained through peak-picking in the electroglottographic derivative. Using parallel recordings of the sound- and electroglottographic signal, we can annotate the voice signal with the glottal closure instants.
Pulse position

While electroglottographic measurements allow non-intrusive analysis of glottal activity, the requirement for a specific measurement device makes the procedure uncommon. Thus signal processing based analysis methods have been developed to extract the glottal closure instants from the audio recordings. Glottal inverse filtering (Alku, 2011) can be used to estimate the glottal pulse signal from which the glottal closure instants can be extracted. Typically, this is done using the linear prediction residual (Naylor et al., 2007; Prathosh, Ananthapadmanabha, et al., 2013), the zero-frequency filtered signal (Murty and Yegnanarayana, 2008) or through smoothing (Drugman and Dutoit, 2009).

Unlike the fundamental frequency, the glottal closure instants cannot be extracted from audio so easily by means of classical signal processing (Babacan et al., 2013b). Thus, we still observe lots of research activity about GCI extraction in recent years which use more and more data-driven approaches. S. Yang et al. (2018) and Reddy et al. (2018) use a convolutional neural network to classify the peaks in a low-pass filtered speech signal and similar representation derived from low-pass filtering. The signal pre-processing is addressed by Goyal et al. (2019) by training a neural network to create a representation best suited for the classification task upstream. Deepak et al. (2019) and Prathosh, Srivastava, et al. (2019) use generative adversarial network and adversarial approximate inference respectively to generate synthetic electroglottographic measurements, which measures the vibrations of the vocal folds using additional hardware. Ardaillon and Roebel (2020) use the PaN vocoder (Ardaillon, 2017) to create a synthetic database with known pulse signal and a convolutional neural network (based on the $f_0$ estimator of Ardaillon and Roebel (2019)) to obtain the glottal pulse signal from raw audio. A refinement of Ardaillon and Roebel (2020) will be presented in Chapter 4.

Glottal closure instant detection can be used for speech analysis and processing (Drugman, Alku, et al., 2014) such as analysis of vocal disorders (Reddy et al., 2018; Deshpande and Manikandan, 2018), formant estimation (Anand et al., 2006) or speech synthesis (Drugman and Dutoit, 2012).

Pulse shape

Glottal closure instant detection is concerned with finding the time instants of the glottal pulse but generally make no statement about the pulse shape. Some glottal closure instant extraction methods, like Deepak et al. (2019), Prathosh, Srivastava, et al. (2019), and Ardaillon and Roebel (2020), generate the non-parametric pulse signal (the latter) or an electroglottographic signal (the former two) which could be used to infer some information about the pulse shape. In fact many approaches to extract the glottal pulse signal from the voice signal have been proposed, known as *glottal inverse filtering* (Alku, 2011). However, obtaining the glottal pulse signal is only half the work. The estimated glottal pulse signal has to be paired with an underlying pulse model to provide a meaningful parametrization. A method to obtain the $R_d$ parameter from the simplified LF-model (see Section 2.3.2 and Fig. 2.9, Pages 18 and 22 resp.) is provided by Huber, Roebel, and Degottex (2012) which uses phase
minimization (Degottex, Roebel, et al., 2010).

### 3.1.3. Voice timbre

In the classical vocoder model, the glottal source signal and the noise component are filtered by the vocal tract filter. We can characterize the result by its spectral envelope. The spectral envelope can be represented by linear predictive coding (LPC) (Atal and Hanauer, 1971) which yields filter coefficients that in effect describe the spectral envelope of the signal. More recently the spectral envelope has been parametrized in the frequency domain using the TrueEnv algorithm (Roebel and Rodet, 2005) or the *Cheap Trick* algorithm (Morise, 2015). The spectral envelope can be used to whiten the signal, by inverse-filtering the original signal by the spectral envelope. Parametric vocoders use the LPC coefficients or spectral envelope for resynthesis, by filtering a white excitation. In the past the spectral envelope was also used for speech recognition (Morgan et al., 2005).

Another characterization of the vocal tract and the voice timbre is through the formant frequencies. Formants are the resonance frequencies of the vocal tract and appear as peaks in the spectral envelope. The formant frequencies can be found using group delay (Murthy and Yegnanarayana, 1991; Anand et al., 2006), using peaks of the smoothed spectrum or spectral envelope (Schafer and Rabiner, 1975) or using the linear prediction spectrum by picking the appropriate poles or peaks (McCandless, 1974). Performance can be improved by incorporating the bandwidth of the pole (Evanini et al., 2009).

### 3.1.4. Noise component

Due to its random nature, the noise component can be difficult to describe precisely. To human ears, two different noise signals are almost indistinguishable if the spectral density is the same. Thus, the spectral density could be used to parametrize the noise source. In practice, however, obtaining the spectral density of the noise source, is not easy as it is often masked by sinusoidal components and may change rapidly, violating the necessary stationarity assumptions. The spectral envelope contains an implicit description of the noise component during unvoiced speech. During voiced speech some information about the noise component of the signal may be inferred from the spectral envelope in the higher frequency bands, where the noise component is stronger than the partials of the voiced component. In both cases, however, it is difficult to disentangle the timbre of the voiced component and the noisy component in the spectral envelope alone.

Assuming no other background noise in the signal, the noise component can be obtained by subtracting the deterministic component (Laroche et al., 1993). This approach relies heavily on the quality of the deterministic signal component as a faulty estimation will leave deterministic components in the estimated noise source. This is less of a problem in pitch-synchronous methods (Moulines and Charpentier, 1990; Stylianou et al., 1995) where analysis windows are very small. Alternatively, subtraction can be improved by resampling the time axis depending on the $f_0$, to
obtain a deterministic excitation with constant pitch (Kawahara, 2006; Huber, 2015, Sec. 6.3.1.3). Alessandro, Yegnanarayana, et al. (1995) use an iterative procedure in the frequency domain of the LPC residual to estimate the noise source directly. Here the noise component is estimated first, and the deterministic component can be obtained by subtracting the noise component.

The extracted noise component can be used to estimate a filter or a spectral envelope to obtain a parametrized description of the noise source. The noise component is used in vocoders and synthesizers. During resynthesis, the original noise source might be replaced by white noise. In breathy voice, however, we observe that a large part of the noise is modulated noise (as discussed in Section 2.3.1). Many models account for this effect by modulating the noise source with the fundamental frequency (Stylianou et al., 1995; Mehta and Quatieri, 2005; Degottex, Lanchantin, Roebel, et al., 2013).

Other methods to extract the noise component have been discussed by and Degottex, Lanchantin, and Gales (2016).

## 3.2. Vocoders

A vocoder is an auto-encoder on the voice signal. Any form of analysis-synthesis cycle can be seen as a vocoder and in the history of vocoders we have seen the vocoder principle taking many forms. The first vocoder was invented by Dudley (1939) who also coined the term ‘vocoder’. Dudley’s vocoder was able to convert an analogue speech signal into an $f_0$ signal and a multidimensional signal describing the timbre. These parameters could then be used to resynthesize the original phrase.

Generally the vocoder maps between raw-audio and the parametric space of the vocoder. In most cases a vocoder performs lossy encoding and perfect reconstruction is not guaranteed even if the vocoder parameters are not changed. Thus, any application using a vocoder is necessarily bounded by the capability of the vocoder. Vocoder can be categorized by their parametric space. We distinguish between parametric vocoders, like Dudley (1939), that define the signal by some acoustic parameters, and non-parametric vocoders, like the phase vocoder or pitch synchronous overlap and add (PSOLA) (Moulines and Charpentier, 1990), where the signal is cut into frames but not compressed through a parametric description. Furthermore, we can distinguish between fixed rate parametrizations where the parametric space is a (multidimensional) signal of fixed sample rate (typically lower than audio rate), and pitch-synchronous methods (Moulines and Charpentier, 1990) where the step size of analysis frames depends on the pitch. Finally, we need to distinguish between neural vocoders and classical, DSP-based vocoders. An implementation of a vocoder with neural networks allows for parametric spaces fundamentally different from classical parametric space, which allows higher level signal descriptions.

The main three applications of vocoders are transformation, synthesis, and compression. Vocoder designed for voice compression are used in telecommunication to reduce bandwidth and can achieve much higher compression rates than general
purpose audio compression.<fnref n="1" /> The main design objective is to achieve a good quality to bitrate ratio. The requirements for vocoders designed for transformation and synthesis are very different. The main design objective is to achieve good resynthesis quality. Furthermore, when generating vocoder parameters through synthesis or transformation it is likely that the vocoder parameters are not generated perfectly, and different parameter groups may contain inconsistent values. Thus, A secondary objective is to achieve robustness towards inconsistent parameter values. We will discuss properties and applications of the vocoder in more detail in Chapter 5. In this section we will discuss the different types of existing vocoders used in voice synthesis and transformation.

### 3.2.1. Pitch synchronous overlap and add

Idea of pitch synchronous overlap and add (PSOLA) is to place an analysis window around each period of the deterministic excitation. If the window placement is correct, consecutive analysis windows are phase-synchronous which allows comparisons in the time domain. Multiple periods are used in one window, typically around 2 to 4 in (Moulines and Charpentier, 1990). Signals can be stretched in time by inserting additional, repeated frames.

Pitch-synchronous vocoders can be non-parametric (Moulines and Charpentier, 1990). In this case the frames are treated in the time domain. The non-parametric time-domain PSOLA can be used to change the pitch of a signal, by resampling the audio frames and then adding or removing additional frames to compensate for the change in duration.

Furthermore, the short audio-frames can be parametrized as is done in with harmonic plus noise models (HNMs) (Stylianou et al., 1995). The HNM assumes that the audio-frames consist of harmonically coupled sinusoids and noise. The amplitudes and phases of sinusoids are estimated through an underdetermined linear set of equations and the noise is obtained through subtraction of the harmonic part. The parametrization of the harmonics allows more flexible transformations than the non-parametric approach.

### 3.2.2. Non-parametric vocoders

Non-parametric vocoders are based around the idea of splitting the voice signal in small analysis windows. The analysis windows can be left in the time-domain, as is the case in PSOLA (Moulines and Charpentier, 1990) or transformed into the frequency domain as is done by the phase vocoder. To change the durations, additional frames have to be added or removed. In the case of the phase vocoder intermediate frames can be generated by interpolating the amplitudes and adjusting the phase. In general the phase vocoder does not preserve the shape of the sound waves. This creates audible degradation in voice signals (Quatieri and McAulay, 1992). Thus, until

<footnote n="1">1. Typical compression performed by vocoders used in telecommunication achieve bit-rates between 16 kbit/s to 40 kbit/s (International Telecommunication Union standard, ITU-T G.726).</footnote>
the shape-invariant phase vocoder (Roebel, 2010), the phase vocoder was generally
used only for transformation of sounds other than voice.

In the time domain, we face the problem that we cannot easily create additional
frames from the neighbouring frames. This is addressed by the aforementioned
pitch-synchronous step-size. Since the audio frames in PSOLA have a step size of
exactly one period, the phases align in the harmonic component. This way individual
frames can be dropped or repeated without creating discontinuities in the phase.
Since the step size of PSOLA is relatively low, durations can be controlled with
sufficient flexibility only by inserting and removing frames.

3.2.3. Source-filter parametric vocoder

The source-filter model can be used to parametrize a voice signal by the frequency of the excitation source $f_0$, the timbre, the noise source, and possibly additional parameters. A periodic excitation signal is filtered and mixed with noise. The first such approach was through the use of linear predictive coding (LPC) (Atal and Hanauer, 1971). The LPC approach parametrizes the timbre by the filter linear prediction filter. The linear prediction residual can be further parametrized through fundamental frequency and additive noise.

While linear predictive coding has become out of fashion for voice transformation
vocoders, the source-filter principal has been used in many recent vocoders. Popular
such vocoders are STRAIGHT (Kawahara, 2006; Kawahara et al., 2008), WORLD (Morise
et al., 2016), SVLN (Degottex, Lanchantin, Roebel, et al., 2013; Huber and Roebel,
2015) or Pulse and Noise (PaN) (Ardaillon, 2017).

3.2.4. Neural vocoder

Neural vocoder allow exploring novel possibilities to parametrize the voice. For
example, H.-S. Choi, J. Yang, et al. (2022) use Wav2Vec (Baevski et al., 2020) features
to describe the phonetic content and unsupervised speaker embeddings to define
the timbre of voice alongside the conventional vocoder parameters $f_0$ and signal
energy. Other neural vocoders completely omit conventional parameters and use the
mel-spectrogram as their parametric space instead (see Section 3.2.6). Furthermore,
neural networks can be used in vocoding to model the acoustic dependencies within
the source filter model.

3.2.5. Neural source-filter vocoders

DSP-based source-filter vocoders largely treat the glottal source and the noise
excitation independently due to the limited possibilities of DSP. In reality, however,
all these properties depend on each other. Therefore, it is desirable to generate the
glottal source signal *e* and the noise signal *n* together to take their co-dependencies
into account. Deep learning allows modelling these dependencies, and we find a
lot of recent work, where glottal source and noise source are generated by neural
networks (Song et al., 2019; Hwang et al., 2020; Oh et al., 2020). Song et al. (2019)
use a WaveNet (Oord, Dieleman, et al., 2016) to generate the excitation signal by
conditioning the WaveNet on classical vocoder parameters. Oh et al. (2020) use a WaveGlow (Prenger et al., 2019) to generate the excitation signal. H.-S. Choi, J. Lee, et al. (2021) generate the spectrograms of source and filter separately in their synthesis module. X. Wang et al. (2019) on the other hand generate an excitation from a classical sinusiods-plus-noise model and replace the filter operation by a neural network.

### 3.2.6. Neural vocoder using the mel-spectrogram

Mel-inversion aims at reconstructing a signal from its mel-spectrogram. The first occurrence of a neural mel-inverter was used by J. Shen et al. (2018) in their second version of the text-to-speech engine Tacotron (Y. Wang et al., 2017). That publication synthesized mel-spectrograms from character sequences transcribing the target utterance and then calculated the raw audio using a WaveNet (Oord, Dieleman, et al., 2016) conditioned on the synthesized mel-spectrograms. The audio quality was excellent, but the procedure had some significant drawbacks:

1. Due to its recurrent nature the WaveNet can be trained efficiently, but synthesis can occur only one audio sample after the other. Thus, inference speed is extremely slow (Oord, Y. Li, et al. (2017) report 172 audio samples per second for the classical WaveNet on a Titan V100 GPU, thus 140 times slower than real-time for 24 kHz audio).

2. The mel-inverter is speaker-dependent thus can only be used for a specific voice. Furthermore, the required amount of data to train this mel-inverter is very large, J. Shen et al. (2018) used 24 h of a single speaker. This amount of data is normally not available for a single speaker (see Table 2.2).

The first problem can be addressed by the parallel WaveNet approach (Oord, Y. Li, et al., 2017) where a parallel version of the WaveNet is inferred from a pretrained classical (non-parallel) WaveNet. This procedure allows fast inference speed in production<fnref n="2" /> but is impractical for research purposes due to its very long and complex training procedure.

The above work introduced many possible applications for voice processing on the mel-spectrogram but due to the difficulties above, the mel-inversion was a restrictive element that limited the results. Therefore, a lot of work happened that tried to address these concerns at many places at the same time using different approaches. Lorenzo-Trueba et al. (2018) used recurrent neural networks to create a universal (speaker independent) mel-inverter, Prenger et al. (2019) used generative flows to create a faster mel-inverter. MelGAN (K. Kumar et al., 2019) used generative adversarial network (GAN) based on convolutional neural network (CNN) but was not a universal vocoder, only a single speaker could be properly synthesized with one model. MelGAN has been extended by Jang et al. (2020) using multi-resolution spectrogram discriminators to allow universal vocoding and by G. Yang et al. (2021)

<footnote n="2">2. Parallel WaveNet was developed by a large tech-company that deployed this neural text-to-speech engine on its consumer devices.</footnote>
using synthesizing the audio signal in multiple bands using pseudo quadrature mirror filter decomposition to reduce model complexity and further increase inference speed. HiFi-GAN (Kong et al., 2020) use Multi-Receptive Field Fusion blocks in the generator of their generative adversarial network. GELP (Juvela et al., 2019) uses linear prediction and adversarial training. iSTFT-Net (Kaneko, Tanaka, et al., 2022) produces a complex valued STFT from the mel-spectrogram that can be inverted using overlap-add. Big-VGAN (S.-g. Lee et al., 2023) uses a large generative adversarial network trained on LibriTTS, and the authors show examples where it can even synthesize music, even though it has never been trained on music (though as discussed in Section 2.5, some music might have been present as background noise). Roebel and Bous (2022) use a source filter model and adversarial training and will be discussed in Section 5.3.

## 3.3. Synthesis

Voice synthesis, in the most general sense, is the generation of audio that contains human voice. More specifically when we talk about synthesis as a research question itself we mean *text-to-speech* in the domain of speech or *score-to-singing* in the domain of singing. The former transforms a character sequence representing written text into a recording of someone reading the text. The latter transforms a musical score including notes and lyrics into a recording of someone singing the piece written in the score. Often we would like to have some control over how a text is read, or a piece is performed. Thus, in *expressive synthesis* we use additional control inputs to define the expression and style of the performance. The typical synthesis methods are described below:

### 3.3.1. Concatenative synthesis

To synthesize voice from a phoneme sequence concatenative synthesis can be used. For a concatenative synthesizer we need a database containing all elementary building blocks, called *units*, of a language. To synthesize a phrase, the text is converted into a sequence of units and the respective samples are concatenated to create a speech signal. Depending on the application different types of units can be chosen, e.g. phonemes, diphones, syllables of whole words. Larger units yield more natural speech but require much larger databases to cover all possible units. In singing, a popular choice is to use diphones (Moulines and Charpentier, 1990) that is, the transitions between two phonemes, as they provide natural transitions between phonemes. Examples of speech synthesizers using concatenative synthesis are Moulines and Charpentier (1990), Sagisaka et al. (1992) or Hunt and Black (1996). Timing and speech melody can be imposed by time-stretch and pitch-shifting respectively.

The concatenation can happen using the audio samples or using the parameters of a parametric vocoder. In the latter case only the parameters that are predominantly responsible for the phonetic content need to be concatenated, while other parameters, in particular the *f*₀, can be freely chosen. This is particularly useful in singing synthesis where the *f*₀ is dictated independently of the phonetic content. Since to
impose the $f_0$ a vocoder is needed regardless whether the concatenation occurs on
the audio samples or on the vocoder parameters, the parametric concatenative ap-
proach is more efficient. Examples of parametric concatenative singing synthesizers
are Umbert, Bonada, and Blaauw (2013), Bonada et al. (2016) and Ardaillon (2017).

3.3.2. Statistical parametric speech (singing) synthesis

Similar to parametric concatenative synthesis, statistical parametric speech syn-
thesis (SPSS) (Zen, Tokuda, et al., 2009) aims to synthesize vocoder parameters for a
given utterance. In the case of statistical parametric speech synthesis, however, the
vocoder parameters are not found by concatenating pre-recorded units of speech,
but rather statistical models are used to generate the vocoder parameters. The stat-
istical model is typically obtained from hidden Markov models (HMM) (Yoshimura,
2002) or more recently by neural networks (Zen, Senior, et al., 2013).

The statistical model obtained for statistical parametric speech synthesizers is much smaller than the whole dataset required for concatenative synthesizers. Thus, disk space requirements are much smaller for statistical parametric speech synthesizers. Likewise, the statistical models are more flexible as the unit-selection approach of concatenative synthesizers and allow changing ‘voice characteristics, speaking styles and emotions’ (Zen, Tokuda, et al., 2009). The increased flexibility, however, is bought at the cost of synthesis quality. The main weak-points of statistical parametric speech synthesizers are the statistical model and the underlying vocoder.

3.3.3. Neural parametric synthesis

Speech- and singing synthesis came to rely more and more on deep learning. Early
use of neural networks can be found in statistical parametric speech synthesis,
where the statistical model is implemented through a neural network (Zen, Senior,
et al., 2013; Zen, Agiomyrgiannakis, et al., 2016; Blaauw and Bonada, 2017). These
approaches use neural networks to generate parameters of a classical vocoder. Since
then, neural networks were used to replace other components of the synthesis
system such that today almost all operations rely on neural networks. Neural
voice synthesis typically occurs in two steps. First the high-level representation is
transformed into an intermediate acoustic representation and then the audio signal
is generated from the acoustic representation. The intermediate representation
can be vocoder parameters of a parametric vocoder (Zen, Senior, et al., 2013), the
spectrogram (Y. Wang et al., 2017) or the mel-spectrogram (J. Shen et al., 2018; Ping
et al., 2018). With mel-inversion techniques becoming more and more efficient, the
latter has become a popular choice in text-to-speech and related domains.

3.3.4. Neural end-to-end synthesis

Neural end-to-end synthesizers generate audio waveforms from the text or score
without intermediate representation. Mel-spectrogram based speech synthesizers
can be turned into end-to-end systems by jointly training mel-synthesis and mel-
inversion or fine-tuning the mel-inverter to the synthetic mel-spectrograms, as
suggested by J. Shen et al. (2018). In this case we obtain an end-to-end system
that is guided through the use of the mel-spectrogram as an initial intermediate
representation. The advantage of this end-to-end setup is that the mel-inverter can
be trained to account for errors produced by the mel-generator.

Donahue et al. (2021) present an end-to-end system that requires no pre-defined
intermediate representation at all, using a generative adversarial network. Fast-
Speech 2 (Ren et al., 2021) use a mel-spectrogram decoder on an intermediate layer
that allows creating a loss based on the mel-spectrogram during training but is
disregarded during inference, to avoid generating the mel-spectrogram directly.
J. Kim et al. (2021) use normalizing flows and variational auto-encoder (VAE) to
guide the end-to-end speech synthesis with the linear spectrogram.

**3.4. Transformation**

Transformation can be seen as a special kind of synthesis that reuses some inform-
ation extracted from an already existing recording together with some external
information. For example in pitch transformation we want to synthesize voice
with the phonetic content and the timing of the original phrase together with a
freely chosen fundamental frequency. Which properties should be used from the
original recording, which properties are imposed explicitly, and which properties
are inferred, depends on the specific application.

**3.4.1. Modelling dependencies**

We have already discussed briefly in Section 3.2 how pitch and duration can be
changed using different vocoders. Using a vocoder it is not difficult to resynthesize
a signal with a different pitch. The difficulty lies in designing transformations where
the result still sounds natural.

Most vocoders model the acoustic dependencies of pitch changes, that is how to
change the signal without introducing artefacts, but not how the voice of a speaker
or singer changes depending on the change. This can work reasonably well for
small changes but can quickly create unrealistic results if the new f₀ values are
too different from the original. The problem is that the other voice properties also
implicitly depend on the f₀. Thus, any serious attempt of voice transformation
needs to address these dependencies as well, either by reducing the dependencies
through re-parametrization or by also regenerating the other parameters from the
new f₀ values. These points have been addressed by Farner et al. (2009), Degottex,
Lanchantin, Roebel, et al. (2013), and Ardaillon (2017). The PaN vocoder (Ardail-
lon, 2017) reduces the dependency of the timbre parameter on the fundamental
frequency by using the vocal tract filter instead of the spectral envelope of the full
signal. The vocal tract filter depends much less on the fundamental frequency than
the spectral envelope since the spectral colour of the glottal pulse signal which
highly depends on the f₀ is removed. Thus, this parametrization allows a higher
range of $f_0$ changes where the result is still acceptable but still finds its limitation, since the vocal tract itself also depends on the pitch.

Other voice properties that are not directly exposed by vocoders can be manipulated similarly: the relationship between the vocoder parameters and the property that is to be changed needs to be modelled. Degottex, Roebel, et al. (2011); Degottex, Lanchantin, Roebel, et al. (2013) modify the breathiness in voice by changing the pulse shape parameter from the underlying pulse model using a parametric vocoder. Huber (2015) changes the voice identity by adjusting the pitch and timbre to convert between source and target speaker. Gentilucci et al. (2018); Gentilucci et al. (2019) and Liuni et al. (2020) created a rough voice effect for singing voice by introducing sub-harmonics through amplitude modulation.

Voice level manipulations can be achieved by modification of timbre and glottal source (Perrotin and Alessandro, 2016). To increase the perceived voice level, Perrotin and Alessandro (2016) suggest increasing number of harmonics in the glottal source by 'increasing the source signals impulsiveness using time distortion' (Perrotin and Alessandro, 2016) and adjust the spectral envelope. The spectral envelope can be adjusted through spectral modelling (Alessandro and Doval, 1998; Alessandro and Doval, 2003) which increases the spectral tilt with increasing voice level. In concatenative synthesis spectral morphing (Turk et al., 2005; Defez et al., 2013) can be used to create intermediate values of voice level between two recordings of different voice levels. Furthermore, voice level affects formant positions, which has been addressed by Molina et al. (2014) by shifting formants according to observed statistics. Ardaillon and Roebel (2017) model the effect of increased mouth opening, due to increased vocal effort, on the vocal tract through the first formant $f_1$. Raitio et al. (2020) use the spectral tilt as a control parameter to affect the voice level in text-to-speech.

### 3.4.2. Disentanglement of parameters

The re-parametrization of the spectral envelope $E$ in the PaN vocoder through the vocal tract filter $V$ and the glottal pulse parameter $R_d$, can be seen as adjusting the spectral envelope to the new pitch. The dependency of the spectral envelope on the pitch is modelled through the pulse shape, which is changed accordingly, while the vocal tract filter is left unchanged. Another perspective is to view this as an attempt to remove the dependency of the fundamental frequency from the other vocoder parameters. The removal of dependency of one parameter from another is called *disentanglement*. In general, disentanglement is a re-parametrization of a parameter set such that its parameter subsets become independent of each other.

Disentanglement is a fundamental discipline in representation learning (Y. Bengio et al., 2013). The goal of disentanglement is to find a representation of the data using independent factors. Due to its many applications it is a research field of high interest (E. Bengio et al., 2017; Higgins, Amos, et al., 2018; X. Liu et al., 2022). Disentanglement using neural networks has been used in many domains; for example, in image processing for face attributes (Lample et al., 2017), face pose (Peng et al., 2017), face identity (Nitzan et al., 2020); in anomaly detection for domain changes
(Dohi et al., 2022) and in voice conversion for voice identity (J.-X. Zhang et al., 2019; Qian, Yang Zhang, Chang, X. Yang, et al., 2019; Qian, Jin, et al., 2020; Qian, Yang Zhang, Chang, Hasegawa-Johnson, et al., 2020), accent (Z. Wang et al., 2021), fundamental frequency (Qian, Jin, et al., 2020; Qian, Yang Zhang, Chang, Hasegawa-Johnson, et al., 2020), speech duration (Qian, Yang Zhang, Chang, Hasegawa-Johnson, et al., 2020) and gender (Benaroya et al., 2023).

## Measuring disentanglement

Disentanglement is difficult to quantify as there is no canonic way to define it. To evaluate different disentanglement methods, however, numerous metrics have been proposed measuring different properties of the disentangled space but sometimes with contradicting indications (Sepliarskaia et al., 2019; Carbonneau et al., 2022). The disentanglement metrics are divided by Carbonneau et al. (2022) into three categories:

**Intervention-based** metrics measure disentanglement by evaluating how points in the code space generated from the same factor relate to each other. The $\beta$-VAE (Higgins, Matthey, et al., 2017), Factor VAE (H. Kim and Mnih, 2018) and R-Factor VAE (M. Kim et al., 2019) belong to this category.

**Predictor-based** metrics rely on a predictor that is trained to predict the factors from the code. This predictor is analysed to quantify the disentanglement. Disentanglement, Completeness, and Informativeness (DCI) (Eastwood and C. K. Williams, 2018), the Explicitness Score (Ridgeway and Mozer, 2018), and the Attribute Predictability Score (A. Kumar et al., 2018) belong into this category.

**Information-based** metrics measure the disentanglement by the mutual information between the factors and the code. Most metrics in this category rely on the Mutual Information Gap (R. T. Chen et al., 2018; Do and Tran, 2020) and derived quantities.

More details and in-depth comparisons between these metrics can be found in Carbonneau et al. (2022).

## Bottleneck auto-encoder

Of particular interest for this work is AutoVC Qian, Yang Zhang, Chang, X. Yang, et al. (2019); Qian, Jin, et al. (2020). AutoVC is a bottleneck auto-encoder which serves as a basis of Chapter 6. The bottleneck auto-encoder of Qian, Yang Zhang, Chang, X. Yang, et al. (2019) disentangles the speaker dependent information from the voice through an auto-encoder structure. The encoder produces a speaker-independent representation through the use of an information bottleneck that forces the encoder to prioritize important information and to remove redundant information from its output (the latent code). By supplying the decoder with the speaker identity, the speaker identity becomes redundant in the latent code (the output of the encoder) and is thus removed. Qian, Jin, et al. (2020) follows a hybrid approach, adjusting the
fundamental frequency through remapping first and second order moments but using the auto-encoder for the remaining information. A further development of this approach is given by (Qian, Yang Zhang, Chang, Hasegawa-Johnson, et al., 2020) where speech is split into rhythm, pitch, timbre, and content simultaneously.

### 3.4.3. Higher level voice conversion

In this thesis we introduce a framework that allows transforming low-level, volatile properties of voice. This framework is used to transform pitch and intensity.

Voice conversion has also been applied to more global parameters like voice identity and social attitude. While the timescale of the property of interest is different, we find many similarities, that allow ideas from these applications to be borrowed in our applications and vice versa. It is thus interesting to discuss these related applications here briefly.

#### Identity transformation

In recent years voice identity conversion has become a popular research activity. With the use of neural networks, first introduced by Desai et al. (2009), modelling the influence of the voice identity on the parametric space of a vocoder became feasible. Today voice conversion is typically done using a neural vocoder based on the mel-spectrogram (Kameoka et al., 2018; J.-X. Zhang et al., 2019; Qian, Yang Zhang, Chang, X. Yang, et al., 2019; Proszewska et al., 2022; Y. A. Li et al., 2023) while end-to-end approaches are becoming popular as well (Nguyen and Cardinaux, 2022; Xie et al., 2022; H.-S. Choi, J. Yang, et al., 2022).

There are two different types of properties that constitute voice identity, physical constraints and habitual features (Bous, Benaroya, et al., 2022). Physical constraints, like the size and shape of the vocal tract, influence the voice's acoustic features. Habitual features are effects that happen not because of the physical setup of the vocal tract but rather from habits. Examples of this are accent, speech rate, speech melody and the way how the speaker accentuates. Voice that defies these constraints is not impossible but improbable for the given speaker.

Which parameters need to be transformed depends on the application. Thus, we see a wide range of sub-disciplines of voice conversion: From complete conversion (Kameoka et al., 2018; J.-X. Zhang et al., 2019) where only the phonetic content is preserved, and the speech is resynthesized from that, to voice reenactment (Bous, Benaroya, et al., 2022) which preserves all habitual features to allow actors to impersonate someone in film and theatre. In between we find voice conversion with preserved timing (Qian, Yang Zhang, Chang, Hasegawa-Johnson, et al., 2020; Ferro et al., 2020) that allows temporal alignment or conversions that preserve the speaker but change their habitual features like accent conversion (Zhao et al., 2018). In Chapters 6 and 8 we will observe that the pitch transformation introduced in this thesis is also implicitly capable of changing the voice identity.
## Transformation of emotion and attitude

Another type of voice conversion is the transformation of emotion and social attitude. In this flavour we want to preserve the voice identity and phonetic content but change the emotion or attitude behind the utterance. While social attitudes influence many parameters of speech production (Salais et al., 2022) its most prominent influence is on the fundamental frequency (F. Chen et al., 2004; Puts et al., 2007; Goupil et al., 2021). Emotional voice conversion can be achieved by converting the speech melody encoded through the fundamental frequency $f_0$, and several studies have been proposed to model the dependency between $f_0$ and emotion (Veaux and Rodet, 2011; Luo, Takiguchi, et al., 2016; Luo, Jinhui Chen, et al., 2017; C. Robinson et al., 2019). More holistic approaches can be found in K. Zhou et al. (2021); K. Zhou et al. (2022) using generative adversarial network (GAN) and Moine, Obin, and Roebel (2021) using transformer networks.

## 3.5. Conclusion

This chapter has provided an overview of the state of the art in voice signal processing technology related to the transformation of pitch and voice level. The different analysis methods were introduced, that allow us to obtain the voice properties necessary to model the voice sufficiently (Section 3.1). Then we have discussed how these extracted properties are used in vocoders that provide a parametric space of the voice (Section 3.2). The vocoder was a central tool in voice transformation in the past, and it remains at the heart of many voice transformation systems through the latest generation of neural vocoders. We have discussed applications of the vocoder such as voice synthesis (Section 3.3) and voice transformation (Section 3.4). Regarding voice transformation we have discussed different approaches, using classical vocoders, and more recent approaches that rely more and more on deep learning. In this context briefly discussed disentanglement, which is the central tool of voice transformation used in this work. We have discussed several applications of voice transformation, including pitch transformation and other voice properties, voice identity conversion and social attitude conversion.

### Motivation of research work

The past three years have been fruitful not only for my own research but for the whole voice-research community. Many of the problems we observed at the onset of this thesis have been adequately addressed and at least partially solved. When I started this work in 2019 there were no universal vocoders available. In particular, singing voice was not supported adequately with neural vocoders. In 2019, it was not at all clear whether the mel-spectrogram contained enough information about the voice that it could serve as a sufficient parametrization of the voice in general. Mel-inversion was still very costly, and it was not certain that cheap and high quality mel-inversion was possible. Thus, in this thesis we begin our investigation of the neural vocoder with two complementary approaches.
1. We investigate the possibilities of neural vocoding using classical vocoder parameters with an analysis-synthesis framework in Chapter 4. While this framework has turned out to be too impractical for voice transformation, we have gained valuable insight with this approach about neural vocoding and the modelling capacities of neural networks with respect to voice. The resulting vocoder exposes glottal pulse positions which is useful for voice analysis and might be used in modelling rough voice.

2. In parallel, in the context of the ARS project, we investigate into possibilities of overcoming the difficulties, that were present at the time, for neural vocoders based on the mel-spectrogram. The findings of that investigation are found in Chapter 5. There we introduce the vocoder than will provide the voice representation used in the subsequent chapters Chapters 6 and 7.
# 4. Glottal closure instance extraction

We begin this thesis with a study on glottal closure instant extraction. This was the first investigation in the context of this thesis and resulted in its first publication (Bous, Ardaillon, et al., 2020). The approach extends recent work of Ardaillon and Roebel (2020) who worked in the same research lab and co-authored with me on the publication that this chapter is based on. Since at the heart of the approach is a neural vocoder, this approach serves well as a starting point to the research around neural vocoding. We will use the insights gained from this study again in Chapter 5 where we will develop a neural vocoder designed for voice transformation.

This chapter been extracted from Bous, Ardaillon, et al. (2020) which has been published in the conference proceedings of the 2020 European Signal Processing Conference (EUSIPCO) and the article and the inference code together with a pre-trained model can be found under the links below.

<figure><img src="image_25.png" /><figcaption>Article</figcaption></figure>

Article: https://arxiv.org/abs/2003.01220

Inference code: http://thesis.fnab.xyz/ch4/code

<figure><img src="image_26.png" /><figcaption>Inference code</figcaption></figure>

## 4.1. Introduction

Classical parametric neural vocoders typically use the fundamental frequency as one of their vocoder parameters (Kawahara, 2006; Degottex, Lanchantin, Roebel, et al., 2013; Morise et al., 2016; Ardaillon, 2017). This is motivated by the source filter model (Fant, 1981; Markel and Gray, 2013) and allow easy modelling the periodic excitation signal as an interpretable parameter. Using the fundamental frequency, however, finds its limitations in rough speech and vocal fry, which is generated by irregular pulses. In this situation, the fundamental frequency does not exist and vocoders that rely on this parametrization do not reproduce well the voiced excitation with irregular pulses.

An alternative representation of the voiced excitation is through the pulse positions rather than through the fundamental frequency. Some parametric vocoders, like PaN (Ardaillon, 2017), use the pulse positions, derived from the fundamental frequency, internally, but rely on the presence of regular pulses to work properly. To better model speech with irregular glottal pulses, the glottal pulse positions need
to be estimated individually. Unlike fundamental frequency estimation, however,
the pulse detection is still unreliable, such that voice transformation of speech with
irregular pulses is still uncommon in parametric vocoders.

**4.1.1. Glottal closure instant**

The position of a glottal pulse can best be characterized by the glottal closure instant (glottal closure instant (GCI)) due to its prominent appearance in the glottal flow derivative. The glottal closure instant is the time instant where the glottal pulse transitions from the open phase to the closing phase (see Fig. 2.8, Page 19). This transition is marked by a sharp peak in the glottal flow derivative and can often be recognized visually in the time domain signal. A more detailed introduction to the state of the art on glottal closure instant detection can be found in Section 3.1.2.

4.1.2. Overview

Recently, Ardaillon and Roebel (2020) used a neural network to generate a synthetic glottal source signal (based on Fant (1995)) by using the PaN vocoder (Ardaillon, 2017) to create a synthetic database with perfect annotation (J. W. Kim et al., 2018; Ardaillon and Roebel, 2019). Using the annotated synthetic database, Ardaillon and Roebel (2020) were able to train the glottal source estimator by matching the synthetic audio to the annotated glottal pulse signal. The resulting estimator could then be applied to real speech.

Here, we expand the ideas of Ardaillon and Roebel (2020). While using a synthetic database provides us with perfect annotation, the synthetic data is limited by the generating model and does not cover all the types of phonation that we encounter in speech and singing. In particular at voice/unvoiced boundaries glottal pulses may occur irregularly. Since the PaN vocoder is driven by the fundamental frequency it cannot properly generate data for these cases. To cover these cases we need to train on real data.

In this chapter we will develop a semi-supervised training procedure to train on both synthetic data, using the existing annotations and on real data using a neural vocoder that uses the glottal pulse signal as its source signal. Using a neural vocoder allows us to train both signal analysis and synthesis simultaneously, which allows the analysis module, which extracts the glottal excitation signal from the voice signal, to be trained semi-supervised on data without explicit pulse annotations. (Semi-supervised, because it still needs to be simultaneously trained on synthetic voice as well, where annotation exists.)

One common difficulty in neural vocoding is that time domain signals usually
cannot be compared by their difference. While perceptually insignificant, a differ-
ence in phase can create an arbitrary difference between two sinusoids. In deep
learning, however, we rely on distance metrics that allow us to compare two signals
to evaluate how close a result is from the target value. Using the pulse positions of
the input signal, allows us to evade this problem, as the vocoder is in fact capable to
reproduce the phase of the original signal as well. Thus, we can compare the input
and output signal in the time domain, to create a training loss directly, which is not
possible if only the fundamental frequency is used.

**4.2. Approach**

**4.2.1. The synthetic database**

The power of deep learning depends on the available training data. While there exist numerous speech databases with speech, only few of these databases also contain electroglottographic (EGG) signals. Therefore, GCI extraction methods that rely on the presence of EGG signals cannot fully exploit the benefits of deep learning based methods.

Ardaillon and Roebel (2020) proposed to use a synthetic database in order to have glottal flow signals available as ground truth. The model, even though trained on synthetic data, still generalized well to real speech and achieved comparable results with state-of-the-art methods. The dataset consists of the publicly-available BREF80 (Gauvain et al., 1990) and TIMIT (Zue et al., 1990) datasets, that were analysed and re-synthesised using the PaN vocoder (Huber and Roebel, 2015; Ardaillon, 2017). The PaN vocoder uses the LF glottal source model (Fant, Liljencrants, et al., 1985; Fant, 1995) as a source signal. For the resynthesis, the *f*₀ estimation of Ardaillon and Roebel (2019) was used to generate the source signal by creating pulses which are separated by the fundamental period. Thus, an almost perfect ground truth glottal pulse can be generated from the vocoder model.

By means of gathering the speech recordings available from previous projects we increased the size of the dataset from a few hours to about 165 hours of human voice including normal speech, singing, whispering and shouting. The simplified LF-model as described by Fant (1995), which is used by the PaN vocoder, uses a single parameter to describe the pulse, the *R*<sub>d</sub> parameter. Ardaillon and Roebel (2020) resynthesized the database multiple times using different *R*<sub>d</sub>-values in order to create a large variety of different pulse shapes. Assuming that the increased database size would provide a sufficiently large variety of *R*<sub>d</sub> values, only one resynthesis of the database using the *R*<sub>d</sub>-values provided by the *R*<sub>d</sub> extraction algorithm (Degottex, Roebel, et al., 2010; Huber, Roebel, and Degottex, 2012) was used in this work.

The training database thus contains a resynthesis of all speech signals and all real speech signals. The PaN resynthesis is annotated with all PaN parameters: GCI, R<sub>d</sub>, f<sub>0</sub>, voiced/unvoiced segments, spectral envelops and unvoiced components (see below). For the real speech data a restricted set of annotations will be used: f<sub>0</sub>, voiced/unvoiced segments, R<sub>d</sub> analysis, spectral envelops.

**4.2.2. The analysis-synthesis framework**

To allow the glottal flow extraction module to model the various phenomena from
real speech that are not captured by the PaN resynthesis, we need to train the
glottal flow extraction module on real data. However, for the real data we do
not have corresponding glottal flow annotation. To allow using real voice data
nevertheless, we train the glottal flow extraction module in an analysis-synthesis
framework. In this framework, an analyser (the glottal flow extraction module) and
a synthesizer (see below) are cascaded and trained simultaneously. We use special
loss functions, described in more detail in Section 4.2.3, to ensure the result of the
analyser represents the glottal flow.

Following the approach from Ardaillon and Roebel (2020) the modules of this
paper also work on audio given at a sample rate of 16 kHz. Due to pooling layers in
the analyser, its output sample rate is 2 kHz. To match the required input sample
rate of the synthesizer, upsampling using linear interpolation is performed during
training. When inferring the GCI, upsampling is performed using cubic splines in
order to increase accuracy. The architecture is identical from Ardaillon and Roebel
(2020) and can be seen in Table A.1 (Page 160).

Synthesizer

We use a convolutional neural network (CNN) with dilated convolution based on
WaveNet (Oord, Dieleman, et al., 2016) to transform the glottal pulses generated by
the analysis module, and an additional white-noise source, to raw audio.

Unlike the original WaveNet, and similar to X. Wang et al. (2019) and parallel
WaveNet (Oord, Y. Li, et al., 2017) we create whole frames in one forward pass.
Avoiding the recursive structure of the original WaveNet is possible due to the
different objective: WaveNet requires recursion to create periodic signals. Our
synthesizer has the periodic pulses as input and thus performs not much more
than filtering the input according to the spectral envelopes. Furthermore, since the
pulses are coherent with the speech signal, the ambiguity in the phase is minimized
for the voiced speech component. Consequently, rather than learning a general
distribution of raw audio speech signals under their control parameters, here we
can use the speech signals as *ground truth*. This allows us to require that the actual
audio at hand is properly reproduced according to appropriate loss metrics (see
below), which sufficiently models the speech properties when the objective is to
infer the pulse positions.

The synthesis task is much simpler than for the original WaveNet, even simpler
than for J. Shen et al. (2018), since an oscillation with the correct frequency is already
provided at the input. Thus, we follow the observation of J. Shen et al. (2018) and
use a reduced size with only two stacks of six layers each (rather than three stacks
of ten layers each).

4.2.3. Loss functions

The following section describes the different loss functions that are used to train the
networks. Fig. 4.1 gives a schematic overview of the various losses used to update
the networks and the flow of the signals used to calculate them.

The principal idea is to train the analysis module jointly with the synthesizer. We
use two losses to train this configuration, AS-spectral and AS-time. These compare
the real speech *s* and the resynthesized speech *ŝ* in the frequency and the time