<figure><img src="image_27.png" /><figcaption>**Figure 4.1.:** Schematic of the analysis-synthesis framework and a visualization of Table 4.1. The Analysis modules are in green boxes, the Synthesis module is in a blue box, losses are in orange boxes. The synthesizer is additionally conditioned on a noise source signal (white noise) and the spectral envelopes of the audio and of the noise signal component. All analysis modules share the same weights.</figcaption></figure>

<table><caption>**Table 4.1.:** The loss function of the analysis-synthesis framework is split into five separate losses that are all calculated on different signals ŷ within the framework, each with its own target value y. The details for each component are given below. Spectral loss functions are calculated using a spectrogram $S_w$ with the given window sizes w.</caption><thead><tr><th>Name</th><th>ŷ</th><th>y</th><th>Weight</th><th>Function</th><th>w [ms]</th></tr></thead><tbody><tr><td>AS-spectral</td><td>ŝ</td><td>s</td><td>0.1</td><td>Σ<sub>w</sub> ||S<sub>w</sub>(y) - S<sub>w</sub>(ŷ)||</td><td>160, 53.3, 32.0, 22.9 and 17.8</td></tr><tr><td>AS-time</td><td>s|<sub>noise=0</sub></td><td>s</td><td>10</td><td>||y - ŷ||</td><td>—</td></tr><tr><td>ASA-time</td><td>p̂</td><td>p̂</td><td>1</td><td>||y - ŷ||<sup>2</sup></td><td>—</td></tr><tr><td>A-spectral</td><td>p̂</td><td>p<sub>PaN</sub></td><td>0.02</td><td>Σ<sub>w</sub> ||S<sub>w</sub>(y) - S<sub>w</sub>(ŷ)||</td><td>80 and 40</td></tr><tr><td>A-time</td><td>p̂<sub>PaN</sub></td><td>p<sub>PaN</sub></td><td>10</td><td>||y - ŷ||<sup>2</sup></td><td>—</td></tr></tbody></table>

domain respectively. However, to ensure that the intermediate signal resembles the glottal pulse signal, we need to employ further regularization. Therefore, we additionally use the losses AS-time, AS-spectral, and ASA-time.

**AS-spectral** To train the analysis-synthesis loop, we follow X. Wang et al. (2019), using multi-resolution spectral loss for the resynthesis. The spectral loss is calculated as the mean absolute error (MAE) between multiple STFT with different window lengths. The STFT are obtained by multiplying the output segment with Hann windows of given sizes, (see Table 4.1), applying an FFT to each of the windowed segments and taking the logarithm of the absolute values. The overlap is 1/2 for each resolution. We call this loss AS-spectral.

**AS-time** The STFT considers only absolute values of each frequency bin. Phase differences, therefore, do not affect the spectral loss. This property is required to properly model the noise component of the signal where the phases are random. On
the other hand, the pulse positions are encoded in the phases as well. The spectral
loss is thus not suitable to enforce proper pulse positions.

Since the analysis module aims to provide a coherent representation for the
excitation signal of both the real and the resynthesized speech signal, there is no
ambiguity in phase. We can thus also apply a loss in the time domain between the
real audio and the resynthesis to enforce proper reconstruction of the phases, which
shall be referred to as AS-time. The time domain loss however will not correctly
deal with noise. Therefore, when calculating the loss in the time domain, the noise
input of the synthesizer is set to zero, enforcing the synthesizer to only produce the
deterministic waveform.

Further regularization losses

When training the GCI analyser in the analysis-synthesis setup, the analyser and synthesizer networks need to be constrained such that they focus on their respective tasks. Notably, the analyser should not start to represent more than the glottal pulse form in its output even if such a strategy clearly would help to improve the synthesis. For imposing these constraints we rely on two strategies: In each training step, the analyser is also trained on the synthetic data, as done by Ardaillon and Roebel (2020), creating the A-time loss, and additionally, we include further regularization losses.

**ASA-time** In the analysis-synthesis setup it is desired that the original signal and the resynthesized signal have the same glottal pulses. Thus, we reanalyse the resynthesized audio and apply a loss in the time domain between the reanalysis and the original analysis.

Note that a global optimum for this loss is reached if the analyser does not produce
any pulses. In that case all pulse signals are zero and as a result the loss is trivially
zero. This behaviour was observed when the synthesizer was not pre-trained and
no other regularization was performed on the analysis of the real audio.

**A-spectral** While we do not know the exact pulse positions *p* in the real audio s,
we know where pulses are present (the voiced segments), the approximate pulse
shape (from the *R*<sub>d</sub>-analysis), and the pulse frequency (from the *f*<sub>0</sub>-analysis). All
these parameters are also in the synthetic glottal pulse signal *p*<sub>PaN</sub>. Therefore, the
two spectrograms of the unknown and unobservable pulse signal of the real speech
*â* and the pulse signal used in the PaN synthesis *p*<sub>PaN</sub> should be similar. To enforce
this invariant, we apply a spectral loss between the synthetic glottal pulse signal
*p*<sub>PaN</sub> and the analysed glottal pulse signal *â*. This loss is of major importance of the
analyser as it prevents the network to create two distinct behaviours for real and
synthetic speech, which will be demonstrated in an ablation study in Section 4.3.

Total loss

The total loss is obtained as a weighted sum of all the losses. The weights were chosen
manually such that the individual losses were in the same order of magnitude. The
weight values can be seen in Table 4.1.

### 4.2.4. Training procedure

#### Step 1 – Initialization

Both the analysis and the synthesis modules are pre-trained separately on the synthetic database. The analysis module is trained only using the loss *A-time*. The synthesizer is trained to imitate the PaN vocoder by using synthetic pulses as input and applying an L2 loss between the synthesizer output and the output generated by the PaN vocoder using the same synthetic pulses.

For the analysis module we use the same hyperparameters as Ardaillon and Roebel (2020): a batch size of 128, an epoch length of 500 updates, an initial learning rate of $2 \cdot 10^{-4}$ and decrease the learning rate by a factor of 0.75 if no improvement on the validation loss was observed for 10 epochs. The synthesizer is trained with a batch size of 8 training-samples from 8 different files and a length of 2560 audio-samples, an epoch length of 512 updates, an initial learning rate of $5 \cdot 10^{-5}$ and the same learning rate schedule as for the analyser with a minimum of $1 \cdot 10^{-6}$. After pre-training, the batch-normalisation layers in the analyser are frozen.

#### Step 2 – Fine-tuning with analysis-synthesis

After pre-training, the analysis and synthesis modules are combined in the analysis-synthesis setup. Both real and synthetic data are used to generate the different losses. For details see Table 4.1 and Fig. 4.1. Each training step consists of two updates. First the losses involving real data (*AS-spectral*, *AS-time*, *ASA-time*, *A-spectral*) are calculated according to Table 4.1 and the gradients are propagated to both analyser and synthesizer. Then a *conventional* training step (as in Step I) is performed for the analyser on synthetic data to ensure that the output of the analyser does not diverge too far from the LF-model.

For the analysis-synthesis we use a batch size of 8 with a length of 3553 audio samples (2560 remain as input to the synthesizer due to valid-padding in the analyser), an initial learning rate of $1 \cdot 10^{-5}$ with the usual schedule (cf. Step 1) with no minimal learning rate.

### 4.3. Evaluation

We evaluate the performance of the analyser on the CMU arctic (Komek and Black, 2004) databases using the metrics from Ardaillon and Roebel (2020), which are based on the metrics defined by Thomas and Naylor (2009). The results are summarized in Table 4.2. The GCI are obtained using a simple peak-picking procedure on the negative peaks of the derivative of the predicted glottal flow, as was done by Ardaillon and Roebel (2020).
4.3.1. Test setup

To evaluate the proposed method we compare the following five models:

**SEDREAMS** For comparison, we include the results from SEDREAMS (Drugman and Dutoit, 2009) as a baseline.

**FCN-BT** As another baseline we use the configuration 'FCN-synth-GF' from Ardaillon and Roebel (2020), which is trained on the BREF80 and TIMIT dataset. This was assumed to be the state-of-the-art at the time of the study.

**FCN-UNI** The same network and training procedure as FCN-BT but trained on the synthetic (not augmented) part of the *new*, universal voice database, described in Section 4.2.1. This is the same as Step 1 from Section 4.2.4 and without the joint training of Step 2.

**AS-UNI** The analysis module from the analysis-synthesis as introduced in the previous section. We do not pre-train a new analysis module for this configuration but use FCN-UNI as initialization to Step 2.

**AS-UNI*** The analysis module from the analysis-synthesis, however trained without *A-spectral*. We do not pre-train a new analysis module for this configuration but use FCN-UNI as initialization to Step 2.

As baselines, we use SEDREAMS from Drugman and Dutoit (2009) and FCN-BT from Ardaillon and Roebel (2020). To compare the effect of the new database we include FCN-UNI which has the same architecture and is trained the same way as FCN-BT but is trained on the universal voice dataset proposed in Section 4.2.1. Furthermore, FCN-UNI allows us to isolate the effect of the analysis synthesis training framework by comparing it to AS-UNI, which uses FCN-UNI as the pre-trained initialization to Step 2. Finally, we include the model AS-UNI* which is trained without using the *A-spectral* loss, to investigate the effect of the *A-spectral* loss.

4.3.2. Evaluation metrics

We use the same evaluation metrics that were used by Ardaillon and Roebel (2020) for Table 2, which are as follows:

**IDR** (*identification rate*) The number of unique identifications between generated pulses and pulses in the ground truth divided by the total number of pulses in the ground truth.

**MR** (*miss rate*) The number of pulses in the ground truth that were not detected by the system divided by the total number of pulses in the ground truth.

**FAR** (*false alarm rate*) The number of generated pulses that could not be associated with any pulse from the ground truth divided by the total number of pulses in the ground truth.
<table><caption>**Table 4.2.:** Results on CMU arctic with metrics considering all GCI.</caption><thead><tr><th>Model</th><th>IDR [%]</th><th>MR [%]</th><th>FAR [%]</th><th>IDA [ms]</th></tr></thead><tbody><tr><td>AS-UNI</td><td>94.21</td><td>1.66</td><td>9.71</td><td>0.51</td></tr><tr><td>AS-UNI*</td><td>92.55</td><td>0.87</td><td>15.21</td><td>0.64</td></tr><tr><td>FCN-UNI</td><td>91.90</td><td>0.88</td><td>22.33</td><td>0.75</td></tr><tr><td>FCN-BT†</td><td>92.73</td><td>2.86</td><td>10.87</td><td>0.47</td></tr><tr><td>SEDREAMS‡</td><td>90.74</td><td>0.19</td><td>61.38</td><td>0.30</td></tr></tbody></table>

† Ardaillon and Roebel (2020)

‡ Drugman and Dutoit (2009)

**IDA** (*identification accuracy*) Standard deviation of the misalignment between the associated detected and ground truth GCI.

For each model we calculate the metrics for each file separately and create an average for each of the speakers, *bdl*, *jmk* and *slt*. We then create total averages, as the average over each speaker, i.e. equally representing each *speaker*, and display the result in Table 4.2.

## 4.4. Results

Having a large variety of pulse shapes in the synthetic data is important

Although the database for FCN-UNI was much larger than the database for FCN-BT, the identification rate of FCN-UNI is slightly worse than the one of FCN-BT. This can be explained by the larger variety of $R_d$ values in the synthetic database for FCN-BT obtained by the data augmentation (multiple resynthesis with different $R_d$-values).

It is worth noting that although here we focus on estimating the glottal closure instant (GCI), the analyser learns much more than just the GCI: It learns the whole pulse shape, that is, in the context of the underlying pulse model (LF-model (Fant, 1995)) it learns the proper $R_d$-value. Thus augmenting the variety in $R_d$-values / pulse-shapes should improve generalization. Furthermore, the pulses generated by the analysis module could potentially be used to infer the $R_d$-value, which might improve the $R_d$-analysis and consequently the resynthesis of the synthetic database.

Fine-tuning the analysis as part of the analysis-synthesis framework does improve the GCI detection

We see that the identification rate of the fine-tuned model AS-UNI the model it was initialized with, FCN-UNI. We can conclude that fine-tuning the analysis in
the analysis-synthesis framework does positively impact the performance of the
analysis module.

This can be explained by the fact that the fine-tuning happens on real data and
the real data is closer to the evaluation dataset, CMU arctic, than the synthetic data.
This suggests that the analysis-synthesis setup is able to force the analysis module
to generate a more expressive pulse model. Consequently, the model generalizes
better to real data,

Observe that none of the losses used for training explicitly represents the GCI.
Nevertheless, our model is able to detect the GCI better than any other method. This
suggests that the model does not only learn the pulse positions, it really learns the
pulse shapes from which the GCI can be derived.

Regularization of the pulse signal is crucial

The improvement of AS-UNI* is smaller than for AS-UNI compared to FCN-UNI
and still does not allow surpassing the results of FCN-BT. This shows that the
regularization in terms of the *A-spectral* is a crucial element of this set-up. One
problem with the analysis-synthesis method is the fact that the synthesizer learns
only to use the pulse input in voiced speech. Consequently, the predicted pulse
signal has no effect at the synthesizer output during unvoiced speech and silence and
thus does not affect the AS losses. Since most of the error happens in the unvoiced
segments, the improvement in configuration AS-UNI* is minimal.

In an additional experiment it was shown that training the analyser on real data
with *A-spectral* and on synthetic data with *A-time*, hence avoiding the use of a
synthesizer, does not allow improving over the FCN-UNI baseline.

4.5. Conclusion

In order to further improve the results from Ardaillon and Roebel (2020), we pro-
posed a new semi-supervised approach for training a neural GCI detector that allows
training with controlled synthetic speech and real speech data. The main idea of
this approach is to incorporate the neural network for predicting the glottal flow,
introduced by Ardaillon and Roebel (2020), into a constrained analysis-synthesis
loop. The end-to-end system trained using a reconstruction loss and a collection
of additional loss functions that ensure proper segregation of the analysis and syn-
thesis tasks. To better represent all the variability of voice signals, we also proposed
to use here a much larger database with many types of signals.

For the task of GCI detection, evaluation on the CMU arctic database has shown
that the fine-tuning using real data significantly improves the identification rate of
the estimator with respect to training it solely on the synthetic database. The iden-
tification rate also surpasses all previous systems establishing the proposed method
as the new state-of-the-art. While we only assessed in this paper the performance of
our analyser for GCI detection, the predicted glottal flow may also be used for other
tasks like R<sub>d</sub> analysis.
# 5. Neural vocoder

Vocoders are used in many applications. Each application has its own requirements such that different approaches to vocoding exist, designed to accommodate the specific applications. This thesis is concerned with transformation of voice. Thus, in this chapter we will develop a neural vocoder that has a parametric space suitable to transform voice. We will discuss the different requirements that we pose to a vocoder used for transformation and the difficulties to implement such a vocoder. The neural vocoder developed in this chapter, and which is used in the following chapters as well, uses the mel-spectrogram as its parametric space. Due to its many possible applications, today many neural vocoders based on the mel-spectrogram exist and have been developed independently in different places.

The topics described in this chapter have been researched collaboratively with Axel Roebel who worked on the same issues for a different project. My own contribution is experimentation with the parametric space in particular in connection with the previous chapter and improvement on practical issues with MelGAN (K. Kumar et al., 2019). The final vocoder, as described in Section 5.4, has been developed in parallel and cannot be seen as a contribution of this thesis.

## 5.1. Desired properties of a vocoder

We shall first discuss the requirements posed to a vocoder used for transformation. There are two types of requirements: Foremost, we have demands on the properties of the parametric space. The desired properties allow us to design transformations and ensure the limitations of the vocoder do not interfere with our objectives. Furthermore, we have practical requirements on the implementation of the vocoder, that arise, when the voice transformation together with the vocoder is implemented on physical hardware, trained with real data and used in a real application. This allows us to motivate, why we chose to use the mel-spectrogram as the parametric space of our vocoder and why our vocoder is implemented the way it is.

Generally a vocoder can be used to perform two tasks:

1. A vocoder removes unnecessary information.

2. A vocoder restructures and presents the relevant information in a more accessible way.

Which information to remove depends on the application. In voice transformation, we want to remove the perceptually irrelevant information, like certain phase
properties or the exact characteristics of the noise. Removing unnecessary information simplifies the transformation task since this unnecessary information does not have to be treated by the transformation.

Since the time-domain signal is one-dimensional, the contained information is spread across time through oscillations. The fundamental frequency for example, while a time varying property, cannot be localized to a fixed point in time but rather is a property of a window in time in which we can observe an oscillation. (We need to observe at least two periods for a frequency to exist.) A vocoder bundles the information that is spread across time to a degree, by decoding the information that is encoded through the oscillations. This simplifies the transformation task as the information is more explicitly available.

When discussing different vocoders we can evaluate them on the basis of how well they perform these two tasks. Not all vocoders necessarily need to perform both tasks, the phase vocoder, for example, does not remove any information and is still a useful tool for specific kinds of transformation. In the following we will first introduce some mathematical modelling of voice signals and then use the mathematical model to discuss different desirable properties of vocoders.

### 5.1.1. Mathematical model

In this chapter we model signals as finite length sequences. Let $X$ be the space of all finite length signals:

$$X = \bigcup_{n=1}^{\infty} \mathbb{R}^n \qquad (5.1)$$

A vocoder is a pair of mappings $(a, s)$ between the space of time-domain signals $X$ and the vocoder parameters $P$.

$$P \subseteq X^N \qquad (5.2)$$

$$a : X \to P \qquad (5.3)$$

$$s : P \to X \qquad (5.4)$$

We call $a$ the analysis operation and $s$ the synthesis operation. Analysis and synthesis change the lengths uniformly, that is, for all signals $x \in \mathbb{R}^n$ of fixed length $n$ the output of $a$ has always the same lengths $m_1, ..., m_N$.

$$a(x) \in \mathbb{R}^{m_1} \times \dots \times \mathbb{R}^{m_N} \qquad (5.5)$$

$$s(a(x)) \in \mathbb{R}^n \qquad (5.6)$$

Usually $m_1, ..., m_N < n$, that is, the time-domain signal $x$ is represented by multiple vocoder parameters with lower sample rate.
We define a perception function $\pi : X \times X \to [0, \infty)$ with the properties:

$$ (\text{symmetry}) \qquad \pi(x, y) = \pi(y, x) \qquad (5.7) $$

$$ (\text{triangle}) \qquad \pi(x, z) \le \pi(x, y) + \pi(y, z) \qquad (5.8) $$

for any $x, y, z \in X$. The function $\pi$ is similar to a metric, except that different elements might have perceptive 'distance' zero. The perceptive function measures how similar two signals $x$ and $y$ sound like: if they sound very different, $\pi(x, y)$ will be large, if they sound alike, $\pi(x, y)$ will be zero.

Using the perception function $\pi$ we can define the relationship between analysis function $a$ and synthesis function $s$ as inverses of each other under the perception function: Let $V \subseteq X$ be the set of voice signals we want to represent with the vocoder $(a, s)$. For all voice signals $x \in V$ we require

$$ \pi(s(a(x)), x) = 0. \qquad (5.9) $$

In practice a vocoder can be designed by fixing either $a$ or $s$ and then optimizing the other such that

$$ s = \underset{\bar{s}}{\operatorname{argmin}} \mathbf{E} [\pi(\bar{s}(a(V)), V)] \qquad (5.10) $$

or

$$ a = \underset{\bar{a}}{\operatorname{argmin}} \mathbf{E} [\pi(s(\bar{a}(V)), V)], \qquad (5.11) $$

for some random variable $V$ that maps to $V$ reflecting the distribution of the voice signals that we want to represent with the vocoder $(a, s)$.

### 5.1.2. Desired properties of the parametric space

The vocoder exposes relevant information about the voice through its parametric space and presents it in a more accessible way. Below we will discuss what is meant by 'relevant information' and 'accessible' through more concrete properties.

**Expressiveness** The parametric space of the vocoder should be expressive enough to be able to encode even subtle changes to the voice. In the days of classical parametric vocoders, an insufficiently expressive parametric space would result in degradation for the cases that cannot be expressed in the vocoder's parametric space. With neural networks the problem can be much more hidden. Neural networks can generate realistically sounding voice without any conditioning (Oord, Dieleman, et al., 2016). Any additional conditioning can guide the result into a desired direction. An under-parametrised neural vocoder might be able to produce realistically sounding voice in any case but the result after an analysis-synthesis cycle might be very different from its input. Thus, it is much more difficult to say whether a parametrization is sufficiently expressive in the context of neural vocoding and can in practice only be shown with certainty by providing examples of synthesizers that can differentiate relevant features.
In mathematical terms, expressiveness can describe how well perceptually relevant details can be distinguished under the analysis function $a$. For two voice signals $x_1, x_2 \in V$ this property can be expressed as

$$ \pi(x_1, x_2) = 0 \Leftrightarrow a(x_1) = a(x_2). \qquad (5.12) $$

The left-to-right implication (⇒) requires that the vocoder removes redundant information: if $x_1$ and $x_2$ sound alike, there is no need to distinguish between them in the parametric space of the vocoder. The right-to-left implication (⇔) is the actual expressiveness requirement as discussed above. If multiple elements $x_1$ and $x_2$ are mapped to the same point in the parametric space, the synthesis function $s$ is not perceptually unique, and the vocoder cannot distinguish between the acoustic feature that makes $x_1$ and $x_2$ perceptually different.

**Comparability** Unless the parametric space of the vocoder consists only of perfectly disentangled parameters, the parametric space must be transformed as a whole to stay coherent. Even if the parametric space would consist of perfectly disentangled parameters, changing a property that is not exposed as a parameter would require to change all parameter simultaneously. We believe this can best be achieved with deep learning. When dealing with neural networks we need to be able to compare different parameter values to define training losses.

Formally, the parametric space of a vocoder needs to form a metric space $(\mathbf{P}, d)$ such that we can compare two parametrizations $p, q \in \mathbf{P}$ through the distance function $d$. This distance function should reflect the perceptive function $\pi$ in the underlying signal space. That is, for two voice signals $x, y \in V$

$$ \pi(x, y) = d(a(x), a(y)). \qquad (5.13) $$

If the parametric space consists of many parameter sets that are too different from each other structurally, finding a suitable distance function $d$, might require additional hyperparameters, which may add additional difficulties. Thus, for a neural vocoder a single set of closely related parameters is more desirable.

**Interpretability** The parametric space should be meaningful. When designing transformations on the parametric space of a vocoder, we need to have an idea, what the individual parameters mean to make informed decisions about what to change and how.

The above properties are desired features of the parametric space. The mathematical formulations express ideals but are in practice not enforceable. Thus, we have to interpret the equations as approximations rather than strict requirements. Mathematically Eq. (5.13) implies Eq. (5.12), but they express different concepts such that in an approximative context they can be seen as two separate requirements.
**5.1.3. Practical requirements on the vocoder implementation**

In addition to the aforementioned features, we face additional difficulties when a neural vocoder is implemented. Since computational hardware is not infinitely fast, the amount of training data is limited and neural networks might not always generalize well. We have to consider the following requirements when creating a neural vocoder after we have decided on a parametric space. The choice of a parametric space, might also be influenced by the possibility of satisfying the requirements below for a specific parametric space.

**Inference speed** For all practical purposes the speed of the neural vocoder was a concern in 2020 (and still is today). WaveNet (Oord, Dieleman, et al., 2016), which was popular in speech synthesis at the time (J. Shen et al., 2018), took minutes to produce a second of audio. Thus, the speed of the vocoder is a serious consideration in any practical application.

**Universality** To perform transformation on the voice a universal vocoder is necessary that allows working on any voice since we cannot know in advance which voice the vocoder will be applied to and what kind of transformations will be used. Historically this was not such a big issue as the vocoders of the past only relied on acoustic models that were speaker independent. With neural networks we are able to learn from speaker specific data and early vocoders worked only on a single speaker (J. Shen et al., 2018) or needed to be conditioned on the specific speaker (Arik, Diamos, et al., 2017). In speech synthesis (where these vocoders were used) this does not pose such a big problem as the synthesized voice identity is known in advance. With voice cloning (Arik, Jitong Chen, et al., 2018), however, this becomes an issue, even for speech synthesis.

**Training data** The availability of training data has grown rapidly in recent years. Still, training data for singing voice is rare with only few publicly available singing voice databases and with many singing styles not covered. Thus, for voice processing that includes singing voice, it is necessary that a neural vocoder can be trained on the limited examples that exist and is able to generalize to singing styles that are only sparsely covered. Similarly, for practical reasons, the same condition applies to speech. Smaller datasets allow results to be reproduced more easily and allow shorter training procedures as iterating over the dataset is faster.

**5.2. Parametric spaces for voice**

In this section we will discuss different representations of the voice in regard to their suitability for neural vocoding and voice transformation. We will argue, why in our opinion the mel-spectrogram is best suited to satisfy the requirements from the previous section.
### 5.2.1. Time-domain signal and end-to-end processing

Though this thesis is concerned with voice transformation using a neural vocoder, it is worth to consider the possibility not to use a vocoder at all and to perform the transformation on the signal directly. End-to-end approaches exist in very recent literature of related fields for text-to-speech synthesis (Donahue et al., 2021; Ren et al., 2021; J. Kim et al., 2021) and for voice identity conversion (Nguyen and Cardinaux, 2022; Xie et al., 2022). When we designed the neural vocoder in 2020, however, the state-of-the-art was very different and end-to-end approaches had serious drawbacks such that an approach with a neural vocoder was the safer choice.

As discussed at the beginning of Section 5.1 the information in the time-domain signal is heavily distributed over time. An end-to-end approach needs to disentangle the information that is spread across time and find an internal representation where these properties are independent. Finding this representation automatically makes sense in an application, where we only want to change one property since computing this representation from the time-domain signal without intermediate steps is the most efficient.

While the parametric space of a vocoder might not be optimal for a specific property, it can be reused in a chain of transformations. Thus, in applications, where we may want to change multiple properties it is more desirable to use a vocoder to avoid the need to reconstruct the raw-audio signal between each transformation step.

### 5.2.2. Classical vocoder parameters

The parametric space of classical vocoders parametrizes signal processing operations used for the signal synthesis. A typical parametrization would be through the fundamental frequency $f_0$, which defines the frequency of a source oscillator, the spectral envelope, which defines a filter and the aperiodicity, also defining a filter. Other vocoder parameters have similar origins in signal processing. While not strictly necessary in the context of neural vocoding, we can use the existing parametric spaces as a basis for the parametric space of a neural vocoder. We will discuss possible drawbacks and benefits, according to the properties from Section 5.1, below.

#### Interpretability

Many classical neural vocoders operate on a parametric space that is interpretable. The underlying physical or signal models give meaning to the parameters that can be understood by humans. For our neural vocoder we could borrow some existing vocoder parameters for their interpretability or make some adjustments to make them fit better to processing with neural networks.

Changing properties that are exposed as a parameter can be achieved easily but requires adjusting the remaining parameters to ensure consistency. For example, the fundamental frequency can easily be changed using a classical parametric vocoder if that vocoder uses the fundamental frequency as one of its parameters. To generate
natural voice, however, the remaining vocoder parameters need to be adjusted to remain coherent with the new fundamental frequency contour. Some neural vocoders, like PaN (Ardaillon, 2017), are designed to reduce the influence of the fundamental frequency on the other parameters. In PaN, this is achieved by using the vocal tract filter, rather than the spectral envelope. Nevertheless, we find limitations for large transformations with the PaN approach when listening to the results.

## Comparability

While the parametric space of a classical vocoder exposes some properties that might be of interest, we cannot reasonably expect all interesting properties to be exposed in one parametric space. Thus, we are faced with two possibilities: Either use a different vocoder for different sets of properties, or fix one parametric space and resynthesize the vocoder parameters depending on the changed property. The former approach has the same drawbacks as the end-to-end approach with the additional problem that now the parametric space might not even be optimal for the specific transformation. If using the parametric space of a classical vocoder we would have to implement the transformation on each of the parameter sets and ensure consistency between the different parameter sets.

This brings us to the requirement of comparability. To change a voice property in the parametric space, each of the sub-sets, e.g. $f_0$, spectral envelope and aperiodicity have to be transformed to reflect the change of the modification in the modified parameter. When treating the parameters individually, inconsistencies might occur between the vocoder parameters if the property change can be achieved by different effects. Thus, consistency between the parameters needs to be enforced. Consistency could be achieved by transforming all parameter groups simultaneously. In this case we face the difficulty of defining proper training metrics that give each parameter the correct weight:

The full parametric space $\mathbf{P}$ can be described as the Cartesian product between the different parameter groups $\mathbf{P}_1 \times \cdots \times \mathbf{P}_n$. Each parameter group ($\mathbf{P}_k$) forms a metric subspace ($\mathbf{P}_k, d^k$). We can obtain a metric $d$ on the product of the sub-spaces as a weighted sum over the distance of the metric sub-spaces:

$$d((p_1, \dots, p_n), (q_1, \dots, q_n)) := \sum_{k=1}^{n} \lambda_k d^k(p_k, q_k) \qquad (5.14)$$

with the individual weights $\lambda_1, \dots, \lambda_n$. The problem of this approach is that it introduces additional hyperparameters ($\lambda_k$) which can have a large influence on the result. There may not be a straightforward optimal solution, especially if the metric sub-spaces ($\mathbf{P}_k, d^k$) are structured very differently, which is the case between fundamental frequency and spectral envelope.

Alternatively, consistency could be achieved, by successively changing one parameter after the other, taking all the previously changed parameters into account. For example changing the $f_0$ based on the property $x$ first, then changing the spectral envelope $E$ based on $x$ and the new value of $f_0$ and finally changing the aperiodicity
based *x*, *f*₀ and *E*. While the discussed issues could be overcome somehow, it is worth
to consider whether other parametrizations could be easier to use. The parametric
space of classical vocoders is motivated by the synthesis operation. With neural
networks we are not restricted in the same way and are free to explore different
parametrizations, possibly inspired by classical vocoder parameters.

Expressiveness

Many classical vocoders suffer from limited expressiveness. The use of the fun-
damental frequency f₀, in particular, poses serious restrictions on the possibility
to represent voice with irregular pulses, like glottal fry. In the context of neural
vocoding it might be possible to infer the existence of irregular pulses, but there is
no explicit control. In Chapter 4 we experimented with a neural vocoder that uses
a non-parametric glottal pulse signal instead of the fundamental frequency. This
approach allows irregular pulses, which is why we investigated into its possibilities.
Originally the idea was that this vocoder could also be used for transformation.
Using a non-parametric glottal pulse signal, however, poses difficulties with in-
terpretability: To change pulse positions, additional tools are required to infer
the pulse positions and to resynthesize the pulse signal from the new positions.
Furthermore, coherency issues arise, as the glottal pulse signal is more strongly
correlated with the spectral envelope and the aperiodicity. Finally, the experiments
with the neural vocoder of Chapter 4 suggested that the spectral envelopes of full
signal and noise signal might miss certain characteristics of the voice thus making
its parametric space not expressive enough. For these reasons the vocoder from
Chapter 4 was found to be unsuitable as a general vocoder on which to base the
voice transformations on.

5.2.3. Mel-spectrogram

When we started our work on the neural vocoder in 2020, the mel-spectrogram was
an emerging representation for voice used in text-to-speech (J. Shen et al., 2018;
Arik, Diamos, et al., 2017; Ping et al., 2018) and voice identity conversion (J.-X. Zhang
et al., 2019).

In a signal processing context, the spectral envelope is a handy description of a
signal as it describes a filter that can be easily applied to any input signal. We can see
it as a means to remove the direct<fnref n="1" /> dependency of the fundamental frequency on the
spectrogram by smoothly connecting the partials. Implicit dependency, however, is
still contained in the spectral envelope. Another drawback is that the noise source
is not described by the spectral envelope in the frequency bands where the sound is
dominated by harmonic sounds. The noise source needs to be described with another
parameter, like the spectral envelope of the noise source. In classical synthesis this
is a desired feature since noise and harmonic sounds are treated independently. In

<footnote n="1">1. From a signal processing point of view, the frequency of an oscillation and the spectral envelope *are* independent: one describes the position of the partials, the other the amplitude of the partials. In the context of voice, however, dependencies exist between the pitch and the timbre, which manifests itself as correlations between *f*₀ and spectral envelope.</footnote>
the context of neural synthesis this is undesired as we want to generate harmonic
and noise input at the same time, to properly model their interactions. Furthermore,
the parametrization of the noise source depends on analysis methods that may
themselves be imprecise.

For the reasons above it might be worth to consider using a property derived from
the spectrogram that does not hide its dependency on the fundamental frequency
and contains information about the noise source explicitly. The mel-spectrogram
fulfils these criteria and has successfully been used in speech synthesis (J. Shen et al.,
2018). The reduced dimensionality from remapping the frequency axis allows it to be
efficiently used in neural networks, and it was a promising candidate for a trade-off
between removing redundant information while still describing the signal explicitly.
Below we will discuss the strengths and drawbacks of the mel-spectrogram as a
parametric space based on the requirements defined in Section 5.1.

Interpretability

The mel-spectrogram can be seen as a visual representation of audio. We can
inspect the mel-spectrogram visually and easily detect regions of voice activity
and distinguish between voiced and unvoiced sections. With more experience
further details can be inferred from a visual inspection, such as distinction between
speech and singing, the rough amount of breathiness and approximate pitch and
the change of some voice property can be seen in the mel-spectrogram. This allows
us to confirm that a transformation is changing the voice in the desired direction —
though, quality assessment is not possible through visual inspection alone, it can
serve as a first sanity check.

Since the mel-spectrogram does not make any attempt in disentangling different
voice properties, it can be difficult to inspect the mel-spectrogram more systemat-
ically. While derived properties, like fundamental frequency and spectral colour,
can be inferred from the mel-spectrogram, this requires additional tools which may
make additional assumptions or introduce estimation errors or biases. Changing
the mel-spectrogram by hand is impossible due to the high correlation between
adjacent frequency bins. Thus, the mel-spectrogram is interpretable for informal
visual inspection and preliminary checks, but unsuitable for systematic analysis or
adjustments by hand.

Comparability

Having one coherent set of parameters allows comparing results easily which is
essential for deep learning to create meaningful gradients. Since all the parameters
are on the same scale, we can compute the loss without relying on heuristics how
to weight the different loss components. Furthermore, the mel-scale reflects some
aspects of human perception of sound which naturally assigns higher weight to
those frequency components that are perceptually more relevant.

Typically, the values of mel-spectrogram are given on a logarithmic scale. The
logarithmic scale models well the perception of loudness and allows distinguish-
ing between low amplitude frequency components even in the presence of high
amplitude sinusoids in other frequency bands. Calculating differences on the log-
arithmetic scale is robust to changes in total signal gain. An average error over the
frequency bins generates a loss with units (typically in dB) that are common in
signal processing and thus easily understood and allow meaningful interpretation of
the results. Thus, absolute or quadratic differences between log-mel-spectrograms,
generate a distance metric, that provides a relevant approximation of Eq. (5.13).
The ease of comparing two log-mel-spectrograms through differences yielding a
meaningful value is one of the main strengths of the mel-spectrogram.

Expressiveness

Evaluating the expressiveness of a parametric space is difficult for any parametric
space and can in practice only be observed through examples. This is especially
the case if deep learning models are involved as these can use information from
the parametric space that is not obviously visible. When we evaluated the mel-
spectrogram for its suitability as a parametric space, sufficient expressiveness was
still an open question, and it still remains uncertain for a few subdomains. The
application in text-to-speech (J. Shen et al., 2018; Arik, Diamos, et al., 2017; Ping
et al., 2018) showed that the mel-spectrogram was sufficiently expressive to model
the speech of a single speaker. Multiple speaker could only be modelled, at the time,
through additional speaker conditioning. Further work on universal vocoding by
Lorenzo-Trueba et al. (2018) indicated that the mel-spectrogram could be used to
differentiate between speech of multiple speaker. Whether the mel-spectrogram
capture all perceptually relevant details was still an open question but an essential
limitation for this work.

Throughout the work around this chapter we were finally able to show that the
mel-spectrogram could also be used for singing voice by presenting a universal
vocoder that works on both speech and singing simultaneously. Recent work of S.-g.
Lee et al. (2023) has shown examples where the mel-spectrogram could even be used
to resynthesize musical accompaniment and other kinds of sounds. Whether the
mel-spectrogram can represent perceptually relevant details in rough voice, glottal
fry and other modes of irregular phonation may be likely but remains to be seen.
More and more examples where the mel-spectrogram is a suitable representation
have been appearing in the literature lately. Still, it might be possible that eventually
we will find limitations of the modelling capacity of the mel-spectrogram which
may necessitate using a different representation of the voice. Nevertheless, for the
tasks set out in this thesis, all the indications are there that the mel-spectrogram is
sufficiently expressive.

Practical concerns

In 2020, it was not clear at all from the literature, whether universal vocoding using
the mel-spectrogram was practical. The mel-spectrogram was already commonly
used in text-to-speech applications like J. Shen et al. (2018), Arik, Diamos, et al. (2017)
and Ping et al. (2018). However, mapping from the mel-spectrogram to raw-audio was either achieved by single-speaker WaveNet (J. Shen et al., 2018), or a WaveNet with speaker conditioning (Arik, Diamos, et al., 2017). No universal vocoder existed that could be used for both speech and singing voice. Since transparent resynthesis could only be achieved using WaveNet, inference speed was very slow or took a very elaborate refinement process (Oord, Y. Li, et al., 2017). Furthermore, training single-speaker neural vocoders required large amounts of training data for a single speaker. Thus, practical concerns existed with regard to inference speed, universality and required amount of training data.

With progress towards speaker independent universal vocoders (Lorenzo-Trueba et al., 2018) and faster network architectures through WaveGlow (Prenger et al., 2019) and MelGAN (K. Kumar et al., 2019) it became probable, that computationally efficient universal vocoding on the mel-spectrogram would be feasible. Today, all these practical concerns are overcome by various neural vocoders (Jang et al., 2020; Jiao et al., 2021; Roebel and Bous, 2022; S.-g. Lee et al., 2023), one of which will be presented in Section 5.4.

### 5.2.4. Conclusion

Neural synthesizers are able to generate audio by extracting implicit information from its parametric space. Thus, new possibilities arise for the way how we represent voice and when designing a neural vocoder we have the chance to rethink the parametric space. The choice of parametric space is important as it influences the limits of what kind of transformation is possible. In this section we have discussed different representations of voice, from classical vocoder parameters to the mel-spectrogram and have contemplated the possibilities of end-to-end approaches.

From the discussion above, we can conclude that the mel-spectrogram is a promising representation of the voice and could provide interesting possibilities for voice transformation. Most of the apparent drawbacks that were unclear when we first began working on the neural vocoder in 2020, have been resolved in recent years. Today we find that the mel-spectrogram has established itself as the de facto standard for neural vocoding.

## 5.3. Neural vocoding using the mel-spectrogram

Even though there were a few open questions about the expressiveness of the mel-spectrogram and practical concerns with regard to the implementation of such a neural vocoder, the mel-spectrogram was a promising parametrization of the voice. Thus, much of the early work of my thesis, was dedicated to addressing the existing concerns. Since other projects depended on an efficient neural vocoder as well (Benaroya et al., 2023), means to optimize the existing mel-inversion approaches and to allow unconditioned multi speaker synthesis were investigated in parallel work by me and others. While I concentrated on singing voice and universal vocoding using MelGAN, others also investigated into WaveGlow and other architectures. Finally, the MelGAN approach was dropped due to insufficient quality, and we started using
a multi speaker version of WaveGlow which was developed for a project on gender conversion (Benaroya et al., 2023). This approach was later replaced by a custom architecture that more closely implemented the source-filter model. Since the question of suitability of the mel-spectrogram was solved at that point, I started to work on transformations on the mel-spectrogram. The final mel-inverter is summarized in Section 5.4 for completeness. My own contribution, however, was only minor, thus insufficient to constitute an independent contribution for this thesis. This section summarizes the research on the mel-spectrogram inversion I was involved in.

## Initial experiments with state-of-the-art vocoders

Our first experiments on creating a mel-inverter were based on WaveGlow (Prenger et al., 2019) and MelGAN (K. Kumar et al., 2019). The WaveGlow approach consists of creating a bijective mapping between the data that we want to synthesize and white noise. During training, the mapping is trained to generate white noise from audio recordings, during inference, the inverse mapping is used to generate audio recordings from white noise. These mappings can be conditioned on the mel-spectrogram to enforce that the generated audio follows the provided mel-spectrogram. One problem with this approach is that the mapping requires huge receptive fields<fnref n="2" /> (several seconds) to create oscillations with consistent phase and even then often struggles with incoherent phases at different times. This phase inconsistency would sometimes cause the phase to jump suddenly, spreading the peaks of the partials and creating audible artefacts. Thus, any attempt to reduce the size of the models to make them more efficient was futile.

Similarly, with MelGAN we observed that oscillations would not come naturally as well. During early training stages one could clearly see periodic patterns along the frequency axis that emerged not from partials of the sound but rather from the upsampling operations. The resulting sound resembled the classic robot-like vocoder effect typically found in pop music of the '80. With increasing iterations the patterns would become more fine-grained and follow the desired spectral shape until the static periodicity was (barely) audible.

## Difficulties with oscillations

What both these approaches have in common is that they try to generate oscillating signals from non-oscillating data. This can easily be achieved by recursive structures like WaveNet, as an autoregressive system can have an oscillating impulse response with only two recursive coefficients. Non-recursive, finite impulse response systems on the other hand, cannot naturally produce oscillations and therefore have to compensate with large receptive fields to fake an oscillating impulse response. As a result inconsistencies in the phase may occur which introduce artefacts as the phase differences cause cancellation of harmonics.

<footnote n="2">2. Receptive field here means that audio samples seconds apart share a large common fraction of the same noise input.</footnote>
In Chapter 4 we discussed a vocoder that used the estimated glottal pulse signal as its input. This vocoder did not face the difficulty of creating an oscillation as it was already provided to it. We adopted a similar approach for the mel-spectrogram based vocoder of this chapter. However, as the pulse positions cannot be extracted from the mel-spectrogram, we will be using the fundamental frequency to generate the oscillation. This fundamental frequency is extracted from the mel-spectrogram and then used to operate a wavetable synthesizer. With this approach we can resynthesize the signal with a relatively small neural network.

## 5.4. Multi-Band Excited WaveNet

The remainder of this section has only minor contribution from myself. It is a summary of Roebel and Bous (2022) and is provided for completeness since the remaining chapters build on the existence of a universal vocoder and audio sample are generated, using this mel-inverter. It cannot be seen as an independent contribution for this thesis.

The Multi-Band Excited WaveNet (MBExWN) reflects the insights we gained in this chapter. The core features are:

**Externally generated oscillation** The Multi-Band Excited WaveNet generates its oscillation by estimating the fundamental frequency from the mel-spectrogram and synthesizing an excitation using a wavetable.

**Sample-rate reduction via PQMF** Computations with neural networks in the time domain are costly but are often redundant. Using the pseudo quadrature mirror filter bank (PQMF) the Multi-Band Excited WaveNet reduces the sample rate of the main component from 24 kHz to 1.6 kHz.

**Spectral domain filter** The final output is obtained through a filter implemented on the STFT. The filter is obtained through a neural network and allows creating narrow resonances without requiring long time consistency of the model.

A summary of the full architecture can be found in Fig. 5.1.

### 5.4.1. Network architecture

First a neural network (the F0-Net in Fig. 5.1) estimates the fundamental frequency from the mel-spectrogram. The network performs an upsampling by a factor of 100 to match the sampling rate of the later elements. The estimated $f_0$ is used in a wavetable generator to produce the necessary oscillation. To prevent aliasing but provide enough harmonics at the same time the wavetable generator can choose from 13 band-limited wavetables with increasing number of harmonics, according to the input $f_0$, and averages over two adjacent wavetables to create smooth differentiable transitions. The wavetable generator generates a signal at a sampling rate of 8 kHz. To improve efficiency, the signal is reshaped such that each time step of the output signal contains five time steps of the input signal. Finally, random
<figure><img src="image_28.png" /><figcaption>**Figure 5.1.:** Schematic of the Multi-Band Excited WaveNet (MBExWN). Green boxes are neural networks, yellow boxes are differentiable DSP operations, red ovals are losses. The dimensions inside the boxes are denoting temporal × spectral dimensions, where the temporal dimension is given as upsampling with regard to the input sample rate (here 80 Hz). [Illustration by Axel Roebel, from Roebel and Bous (2022)]</figcaption></figure>

noise is concatenated with the pulse signal to allow mixing and switching between harmonic and noise components of the source.

The periodic excitation, together with the noise source are shaped through a WaveNet (Oord, Dieleman, et al., 2016) to create the glottal source signal. Like in Chapter 4, the WaveNet does not have to create an oscillation and thus neither needs to be recursive nor requires a large receptive field. Here the WaveNet consists of two stacks with five layers each. A final convolution ('PostNet' in Fig. 5.1) maps the output of the WaveNet to the necessary number of channels required for the next step. The WaveNet is conditioned on the mel-spectrogram. To match the number of channels inside the WaveNet, the mel-spectrogram is first mapped through a convolution that maps the mel-frequency axis to the required number of features. The WaveNet does not change the sample rate of its input. To map back from the 1.8 kHz to the required 24 kHz at the output, a pseudo quadrature mirror filter bank with 15 bands is used.

The synthesized source is filtered by a vocal tract filter. The vocal tract filter is estimated from the mel-spectrogram. The vocal tract filter estimator is a neural
network with five layers. It produces the cepstral coefficients of the vocal tract
filter. Using the cepstral representation allows us to enforce minimum phase for
the resulting filter by interpreting the coefficients as causal cepstral coefficients
and to control the formant bandwidth by the number of cepstral coefficients. The
filter allows crating narrow resonances without requiring long time consistency of
the model.

5.4.2. Training

Training the Multi-Band Excited WaveNet relies on three losses; a multi-resolution
spectral loss (X. Wang et al., 2019) matches the signal to the target perceptually,
the f₀-loss guides the f₀ estimation and a discriminator improves the quality of the
noise component.

When calculating the mel-spectrogram the phase is disregarded removing all
time information inside a frame and in particular the pulse positions of voiced
speech. Though some information about pulse positions might be inferred through
inter-frame relations, in particular due to frame overlap, the information does not
suffice to properly restore the pulse positions. Perceptually, this is not necessary as
individual pulses of regular phonation are not perceived in voiced speech. The prob-
lem arises when we want to compare the synthesized signal with the original: As the
phase difference between the two could be arbitrary, comparing the synthesized sig-
nal to the target signal in the time domain will not yield any useful information. Thus,
we rely on a spectral loss to properly compare the two signals. When calculating
the spectral loss, again, the phase is disregarded before comparing the magnitudes
of the spectra. To best capture the differences, the spectrogram is calculated with
different window lengths thus providing us with multiple resolutions.

The estimated fundamental frequency can and must be trained with existing an-
notations. This approach is a double-edged sword. Ideally we would like the network
to figure out the best matching f₀ by itself: First of all the f₀ annotations are not
perfect but above all the existing f₀ models do not work well for speech boundaries
where pulses happen irregularly. One of the reasons why we do not simply use
a parametric vocoder is because the underlying assumptions are inaccurate. We
utilize deep learning not to rely on parameters like the f₀ and its analysis tools that
rely on these assumptions and let the dependencies be modelled by the data instead.
On the other hand it is necessary to provide the fundamental frequency as it will
not be able to find the correct f₀ from random initialization. The problem is that
using a spectral loss does not produce a useful gradient for the f₀ if the f₀ is too far
off: The spectral loss defines how spectrum needs to be adjusted along the energy
axis, the f₀, however, is in a sense orthogonal to this, as changes in the f₀ move
the spectrogram along the frequency axis. If a partial of the synthesized signal is
close to a partial in the target signal, the spectral loss may move the f₀ such that the
partials move closer together, regardless whether these are the same partials or not.
Thus, the spectral loss matches harmonics from the synthetic signal to harmonics of
the target signal that do not necessarily belong together. In particular, the direction
to the closest partial in the target signal could be different for each partial, such
that the net gradient is cancelled out. As a compromise we train the F0-Net using
the auxiliary f₀-loss, but apply the f₀-loss only on segments of stable phonation. In
this context, anything more than 50 ms away from voice/unvoiced boundaries, is
assumed to have a stable fundamental frequency. For any other regions only the
spectral loss has an influence on the generated f₀.

The multi-resolution spectral loss can perceptually capture the deterministic part of the signal. The noise component, on the other hand, cannot be evaluated so well with this method. Perceptually we cannot distinguish two noise signals well if the spectral density is the same. The spectrogram of a noise signal, however, does not estimate the spectral density well. Instead, we obtain a noisy version of the spectral density which is different for each realization of the random signal. Due to the unpredictability of the noise input, the synthesizer is inclined to disregard the noise input and to simulate the noise part with complex but deterministic patterns instead. To increase the quality of the noise, an adversarial strategy is used to teach the synthesizer to generate realistic noise signals. A discriminator, operating on two different time-resolutions (Jang et al., 2020), is trained to distinguish synthesized and real voice on their waveforms. The two discriminators work on different sample-rates to provide different viewpoints to the audio waveform. The discriminators help to make a difference between somewhat acceptable quality and almost transparent synthesis but during the initial stages of training the information provided by the discriminators is rather misleading and slows down or even prohibits convergence. We thus perform training in three steps: First, the F0-Net is pre-trained on the mel-spectrogram with the f₀ annotations alone. Then the whole synthesizer (without the discriminator) is pre-trained, using only the spectral- and the f₀-loss initializes the weights to already produce proper voice. Afterwards, during the fine-tuning, the synthesizer is trained together with the discriminators using all losses.

5.4.3. Discussion

Perceptive evaluation

The Multi-Band Excited WaveNet was evaluated against Multi-Band MelGAN (G. Yang et al., 2021) and Universal MelGAN (Jang et al., 2020) in a perceptive test for Roebel and Bous (2022). An excerpt of the results is shown in Tables 5.1 and 5.2. From these tables we can observe the following: On speech, the voice model MW-VO, trained on both speech and singing, performs worse than the domain specific speech model MW-SP, trained only on speech. On singing however, training on both speech and singing, improves the quality compared with the model MW-SI, trained only on singing. For singing voice, the voice model MW-VO performs best on both seen and unseen singing, while on speech the speech model MW-SP performs comparable with existing state-of-the-art neural vocoders. The perceptive tests have shown that the quality loss through analysis-synthesis is very small for regular speech and singing, even for unseen voices. The mel-inverter struggles with rough and creaky voice and in particular with growling. These are cases where the assumption of a fundamental frequency is violated. For these special cases further treatment is necessary to generate irregular pulses or allow the use of sub-harmonics.
<table><caption>**Table 5.1.:** Mean opinion score for MBExWN and other baselines on speech: Evaluation on LJ Speech (see Table 2.1, Page 32), and the evaluation data from the demo website† of Jang et al. (2020) using the samples from LJ Speech (UMG(LJ)) and other samples (UMG(other)). Values on a scale from 0 to 100. [From Roebel and Bous (2022), Table 4]</caption><thead><tr><th></th><th>HREF</th><th>MW-SP</th><th>MW-VO</th><th>UMG-LJ+LT</th><th>MMG-LJ</th></tr></thead><tbody><tr><td>LJ</td><td>94 ± 1.9</td><td>85 ± 4.3</td><td>82 ± 5.2</td><td>—</td><td>86 ± 4.1</td></tr><tr><td>UMG(LJ)</td><td>93 ± 3.7</td><td>86 ± 6.8</td><td>77 ± 11.4</td><td>80 ± 10.6</td><td>—</td></tr><tr><td>UMG(other)</td><td>94 ± 2.2</td><td>78 ± 6.4</td><td>74 ± 7.5</td><td>81 ± 6.3</td><td>—</td></tr></tbody></table>

<table><caption>**Table 5.2.:** Mean opinion score for MBExWN and other baselines on singing: Evaluation on ICH(see Table 2.2, Page 34), samples of pop singing from MusDB (Rafii et al., 2017) (Pop(MusDB)) and samples of rough singing voice (Rough). Values on a scale from 0 to 100. [From Roebel and Bous (2022), Table 5]</caption><thead><tr><th></th><th>HREF</th><th>MW-SI</th><th>MW-VO</th><th>MMG-ICH</th></tr></thead><tbody><tr><td>ICH</td><td>94 ± 2.7</td><td>83 ± 8.4</td><td>86 ± 6.2</td><td>85 ± 5.5</td></tr><tr><td>Pop(MusDB)</td><td>93 ± 2.7</td><td>73 ± 8.3</td><td>79 ± 8.1</td><td>—</td></tr><tr><td>Rough</td><td>96 ± 2.7</td><td>60 ± 12.7</td><td>68 ± 11.3</td><td>—</td></tr></tbody></table>

**HREF** Hidden reference

**MW-SP** Multi-Band Excited WaveNet trained only on speech

**MW-SI** Multi-Band Excited WaveNet trained only on singing

**MW-VO** Multi-Band Excited WaveNet trained on speech and singing

**UMG-LJ+LT** Universal MelGAN (Jang et al., 2020) trained on LJ speech and Libri-TTS

**MMG-LJ** Multiband MelGAN (G. Yang et al., 2021) trained on LJ speech.

**MMG-ICH** Multiband MelGAN (G. Yang et al., 2021) trained on ICH dataset

<footnote>† https://kallavinka8045.github.io/icassp2021/#unseen-domains-speaker-emotion-language-1</footnote>
## Computational speed

The Multi-Band Excited WaveNet allows us to efficiently map between time-domain audio and mel-spectrogram. It is able to generate 50 000 audio samples per second on a single CPU core thus allowing real-time processing (with latency).

## 5.5. Conclusion

In this chapter we have developed the vocoder framework that we will use to perform voice transformations in. The development of a new vocoder model was motivated by the new possibilities in synthesis due to neural networks. These new possibilities allowed us to reinvent the vocoder using a parametric space, the mel-spectrogram that has long been thought to be insufficient to fully describe the voice. However, more and more work appeared presenting methods to resynthesize voice from mel-spectrogram with almost no perceptual difference. Together with the results of Section 5.3, it has become clear that the mel-spectrogram can be used in a universal neural vocoder for a sufficiently large variety of voice signals.

Today a large quantity of mel-inversion algorithms exist, all with different strengths and drawbacks. We will keep using the Multi-Band Excited WaveNet throughout the remainder of this thesis because it is available to us, relatively easy to retrain on new data and extremely fast during synthesis. Due to its universality, however, the voice transformations proposed in the next chapters, are independent of the mel-inverter which would permit using any other mel-inversion technique that allows inversion of arbitrary mel-spectrogram containing voice.

For the remainder of this thesis we may assume time-domain audio and mel-spectrogram to be equivalent.
# 6. Bottleneck auto-encoder

After extensive considerations we finally have settled for the mel-spectrogram as the parametric space of our neural vocoder. Having a means to freely map between mel-spectrogram and raw audio, it remains to develop transformations on the mel-spectrogram. We will see in the following chapters that the mel-spectrogram is indeed a very suitable representation of the voice for the task of transformation.

The tool of choice for the voice transformations will be the bottleneck auto-encoder. The auto-encoder is a type of neural network architecture that is trained to reproduce its own input. Two neural networks, the Encoder, and the Decoder, are cascaded and trained to act as inverses of each-other. The auto-encoder produces a latent space between encoder and decoder where the input signal is re-parametrised. The study of auto-encoders is concerned with influencing the auto-encoder in a way that the latent space inhibits certain desirable properties. In our particular case, we will train the auto-encoder to produce a latent space that is independent (disentangled) from the pitch (this chapter) and also from the voice level (Chapter 7).

We will develop the bottleneck auto-encoder with the transformation of the pitch. Pitch, as encoded through the fundamental frequency is easily available for any voice recording thanks to extensive previous work on the subject (see Section 3.1.1, Page 48). These methods work reasonably well so that we have reliable annotations which will allow us to concentrate on the auto-encoder architecture itself without having to worry too much about the underlying data. Furthermore, the result can be evaluated both systematically by the existing $f_0$ estimation tools and subjectively by looking at the mel-spectrogram or listening to the resynthesis: The fundamental frequency is visible in the mel-spectrogram so that we can see whether and where the method is working or not. Similarly, our hearing is well-trained on recognizing pitches to that we can evaluate the result by hearing. Applications for pitch transformations are plentiful and the two motivating applications for us were the artistic residency with Judith Deschamps and the IRCAM Singing Synthesizer.

<figure><img src="image_29.png" /><figcaption>Article</figcaption></figure>

<figure><img src="image_30.png" /><figcaption>Audio samples</figcaption></figure>

Article: https://www.mdpi.com/2078-2489/13/3/102

Audio Samples: http://thesis.fnab.xyz/ch6/demo

This chapter was originally published under the title *A Bottleneck Auto-Encoder for F0*
*Transformations on Speech and Singing Voice* (Bous and Roebel, 2022) in the open-access journal *Information* by the Multidisciplinary Digital Publishing Institute (MDPI).

**6.1. Introduction**

Since the invention of the vocoder over 80 years ago by Dudley (1939), people have been studying means to transform the acoustical properties of speech recordings. Since then, enabling transparent pitch transformations has been a prominent topic inside the speech processing research community and attempts to achieve this were plentiful. Prominent approaches include the phase vocoder (Flanagan and Golden, 1966), pitch synchronous overlap and add (Moulines and Charpentier, 1990), shape-invariant additive models (Quatieri and McAulay, 1992), shape-invariant phase vocoder (Roebel, 2010), harmonic plus noise model (Laroche et al., 1993; Stylianou et al., 1995), parametric vocoders (Kawahara, 2006; Morise et al., 2016), or extended source-filter models (Degottex, Lanchantin, Roebel, et al., 2013; Huber and Roebel, 2015). Recently, as speech signal processing has become dominated by deep-learning based techniques, the community has focused on more abstract, high-level voice properties, like speaker identity (Desai et al., 2009; Kameoka et al., 2018; J.-X. Zhang et al., 2019; Qian, Yang Zhang, Chang, X. Yang, et al., 2019; Qian, Jin, et al., 2020; Ferro et al., 2020) but also emotion (C. Robinson et al., 2019; Moine, Obin, and Roebel, 2021) and accent (Zhao et al., 2018). These methods may change the fundamental frequency implicitly as part of their transformation. Nevertheless, explicit transformation of pitch remains an interesting and challenging research topic, in particular in artistic applications.

Musical pitch and speech melody are determined by the fundamental frequency $f_0$ of the quasi-periodic oscillation during phonation. In singing voice, the $f_0$ does not only carry the melody but also the singer's individual expression used to interpret the piece which is highly dependent on the genre of the piece (Umbert, Bonada, Goto, et al., 2015). In speech the $f_0$ plays a different role than in singing. It carries important information such as mood, intent, and identity (Salais et al., 2022). The $f_0$ contour is strongly related with expressivity (Veaux and Rodet, 2011), and editing the $f_0$ contour may therefore be used to manipulate the expressivity of a given voice signal.

**6.1.1. Applications**

For singing synthesis, it does not suffice to simply map the notes to their corresponding frequencies and durations. Singing synthesizers need to translate the required melody to an $f_0$ curve to convey the expression and style required by the score (Umbert, Bonada, and Blaauw, 2013; Ardaillon, Chabot-Canet, et al., 2016). A common approach is unit-selection with $f_0$ modification as is done by Ardaillon, Degottex, et al. (2015) or Bonada et al. (2016). Thus, transforming the $f_0$ of singing voice is crucial for parametric concatenative singing synthesis and could also help refine existing recordings. The quality of these synthesizers is thus limited by the quality of the transposition algorithm. Combining singing units in mel-spectral
representation with the neural $f_0$ transformation introduced in this chapter, along with our mel-inverter MBExWN described in Chapter 5 could allow creating a neural singing synthesizer.

As for voice modification, $f_0$ transformation can also be used on real singing recordings to alter the intonation, expression, and even melody in post-production. The $f_0$ transformation algorithm presented here could also expand the range of a singer. If pushed far enough, thanks to the implicit adaptation of voice features, it will create a coherent new personality.

One of the most prominent differences between the sexes is the mean $f_0$, due to different rates of growth of the vocal folds in puberty. Thus changing the $f_0$ in speech can result in changing or obfuscating the speaker's gender (Farner et al., 2009), an effect which could be used in scientific studies to research the effect of perceived gender in the voice in social interactions (Arias et al., 2021).

### 6.1.2. Challenges of pitch transformation

The fundamental frequency has an effect on most other voice properties. A mismatch during synthesis is often perceived as unnatural (Degottex, Roebel, et al., 2011). Due to the inherent complexity of the relationships between $f_0$, glottal pulse and vocal tract filter, we think that these effects can best be modelled with data-driven approaches. While deep-learning has successfully been applied to numerous similar domains, the transformation of pitch has not been addressed adequately since widespread accessibility of deep-learning. In this chapter we will address the issues that cannot be addressed adequately by classical parametric vocoders. As discussed in Chapter 5, using the mel-spectrogram as parametric space allows us to train a neural network to model the dependency between $f_0$ and other voice properties more efficiently than using a classical parametric vocoder space.

### 6.1.3. Mathematical description of disentanglement

In the literature, disentanglement is commonly approached by a full disentanglement of explaining factors (Higgins, Amos, et al., 2018). In this approach independent factors are learnt unsupervised. The resulting parameters can be manipulated independently but might not correspond to meaningful parameters or to parameters of interest. In our applications we have a specific property that we want to change. We are not interested in a complete description using disentangled factors but rather disentangling individual factors from the remaining parameter set. Thus, here we approach disentanglement from a slightly different angle.

We define disentanglement through stochastic independence on a parametric feature space $\mathbf{F} = \mathbf{F}_1 \times \cdots \times \mathbf{F}_n$. Let $\mathbf{X}$ be the space of all finite length signals

$$ \mathbf{X} = \bigcup_{n=1}^{\infty} \mathbb{R}^n. \qquad (6.1) $$

Furthermore, let $(\Omega, \mathcal{F}, \mathbf{P})$ be a probability space, and let $(F_k : \Omega \to F_k, k = 1, \dots, N)$
be a finite set of voice features.<fnref n="1" /> Assume there exists a synthesis function s such that

$$s : \mathbf{F}_1 \times \cdots \times \mathbf{F}_N \to \mathbf{X} \qquad (6.2)$$

$$V(\omega) = s(F_1(\omega), \dots, F_N(\omega)). \qquad (6.3)$$

Assuming there exist probability densities for ($F_1, \dots, F_N$), we can evaluate a signal by the likelihood of its features. This defines the signal's likelihood

$$L : \mathbf{X} \to \mathbb{R}_+^N \qquad (6.4)$$

$$L(s(f_1, \dots, f_N)) := p_{F_1, \dots, F_N}(f_1, \dots, f_N). \qquad (6.5)$$

If all features ($F_k$) are mutually independent, we say that $\mathbf{F}$ is *disentangled*. Then the likelihood $L$ can be written as the product of the likelihoods of the individual features:

$$L(s(f_1, \dots, f_N)) = \prod_{k=1}^{N} p_{F_k}(f_k) \qquad (6.6)$$

In this case changing one feature changes the overall likelihood of the voice signal only with regard to changes in the likelihood of that feature. Assume without loss of generality we change the value of the first feature from $f_1$ to $f'_1$. Then the voice signal changes from $v = s(f_1, \dots)$ to $v' := s(f'_1, \dots)$. Accordingly, it's likelihood changes as

$$L(v') = L(v) \cdot \frac{p_{F_1}(f'_1)}{p_{F_1}(f_1)}. \qquad (6.7)$$

This property ensures that changing a feature to a reasonable value will not result in improbable, unnatural voice.

Generally, however, different voice features are not independent, and thus the feature space is not disentangled. This means that (6.7) does not hold, and often we find

$$L(v') \ll L(v) \cdot \frac{p_{F_1}(f'_1)}{p_{F_1}(f_1)}. \qquad (6.8)$$

In this case, for a reasonable change in the voice property, the changed signal can become very unlikely ($L(v')$ may be even 0) and may no longer represent a real world voice signal. This is a problem as changing one voice feature may have unpredictable and undesirable results, which is an issue with the parametric spaces of classical vocoders. Therefore, disentanglement aims at a re-parametrization of $\mathbf{F}$ through the invertible mapping $d : \mathbf{F} \to \mathbf{D}$. The goal of disentanglement is to find such a mapping $d$, such that Eq. (6.7) holds for its subcomponents $\mathbf{D}_1, \dots, \mathbf{D}_M$. Note that the number $M$ of subcomponents of $\mathbf{D}$ does not necessarily need to be the same as the number $N$ of subcomponents of $\mathbf{F}$.

<footnote n="1">1. Here we use the term 'feature' instead of 'property' to match the variable name $f$. We use $f$ instead of $p$ (for property) to avoid a naming clash with variables related to probabilities. The terms 'voice property' and 'voice feature' can be thought of synonyms in this context.</footnote>
If we want to change a feature, it has to be a subspace of the disentangled space **D**.
Due to the re-parametrization *d*, however, features may be lost in **D**. Thus, for
disentanglement to be useful, we pose the additional constraint that the features
we want to change must be still present in **D**. If two features *F*₁ and *F*₂ are not
independent, both cannot be in **D**. Thus, for practical reasons, unless two features
are already independent, we disentangle into the space **D** = **F**₁ × **D**' with the inde-
pendent subcomponents *F*₁ and *d*'(*F*₁, ..., *F*ₙ). In this case we say, *feature 1 has been
disentangled (from the other features)*.

If we still want to be able to change two features, two options arise.

1. Find two disentanglements $d_1$ and $d_2$ that disentangle feature 1 and feature 2 respectively, and transform the features one after the other. In this case the feature that was transformed first will not be preserved, but we may hope that it will still remain close to what we wanted.

2. Model $F_1$ and $F_2$ as a joint feature $F_{1,2} = (F_1, F_2)$ and disentangle $F_{1,2}$ from all the remaining features. In this case Eq. (6.7) holds for the joint distribution $p_{F_1,F_2}$, and it is up to the user to provide combinations of $f_1$ and $f_2$ that make sense in the context of voice.

**6.1.4. Goals of this chapter**

This chapter introduces the neural voice transformation framework, which is the
central tool of this thesis. The neural voice transformation framework is a framework
that allows disentanglement and consequently transformation of aligned signal
properties. We introduce the neural voice transformation framework using the
fundamental frequency $f_0$ as an example property as transformation of the $f_0$
is a well-established and well-studied problem. Thus, we will address the free
modification of the fundamental frequency $f_0$ in recordings of speech and singing
voice. The algorithm requires only minimal input from the user and allows one to
change the $f_0$ by means of providing a fixed transposition value or redrawing the
$f_0$ curve.

The transformation system is based on the bottleneck auto-encoder of AutoVC
(Qian, Yang Zhang, Chang, Hasegawa-Johnson, et al., 2020). Qian, Yang Zhang, Chang,
Hasegawa-Johnson, et al. (2020) use the part of speech melody that is independent of
rhythm, timbre, and content. In particular, the $f_0$ is normalized by mean and vari-
ance for each speaker. Contrary to that, here we use the non-normalized, absolute
$f_0$ as control parameter. Therefore, the control parameter for our auto-encoder
contains some information about the speaker identity and timbre. Changing the raw
$f_0$ signal may change the component of speaker identity and timbre that depends
on the $f_0$. The system should apply the given $f_0$ contour onto the output signal such
that the output signal contains exactly the same $f_0$ contour. This implies that parts
of the identity will change if the change is incompatible with the speaker identity.

Two main goals arise for a potential $f_0$ transformation system:

1. The desired $f_0$ contour should be followed exactly.
2. The result should sound like a human.

The first goal, the $f_0$ accuracy, can be evaluated objectively by resynthesizing the transformed mel-spectrogram and checking that the $f_0$ in the resynthesized audio matches the target $f_0$. Our findings are stated in Section 6.3.1. The second objective, the naturalness of the syntheses, can only be evaluated by listening to the generated samples. Therefore, we have performed a perceptive study by asking a selected group of participants to listen to synthesized and non-synthesized audio and give a perceptive rating. The results of the perceptive test are presented in Section 6.3.2.

One crucial hyperparameter of the bottleneck auto-encoder is the size of the bottleneck. We therefore perform a systematic analysis of the dependence between disentanglement and bottleneck size and present the results in Section 6.3.1.

### 6.1.5. Outline

The remainder of the chapter is structured as follows. In Section 6.2 we introduce the auto-encoder setup, its architecture and how to train it, the datasets used in the experiments and the experimental setup itself. In Section 6.3 we present the results from a perceptive test and a transformation accuracy analysis. We compare the proposed system to other $f_0$ transformation algorithms, as well as a visual analysis of the latent code for the case where the bottleneck size is 2. A conclusion and an outlook for future work will be found in Section 6.4.

## 6.2. Methodology

### 6.2.1. Proposed model

The auto-encoder operates on the mel-spectrogram. To change the $f_0$ we first extract the mel-spectrogram and the $f_0$ from the original audio recording. Then we use the encoder to produce a latent code from the mel-spectrogram and the original $f_0$. We resynthesize the mel-spectrogram by applying the decoder to the latent code and the target $f_0$. Finally, the transformed audio is obtained by inverting the transformed mel-spectrogram using the neural vocoder MBExWN, which we have presented in Chapter 5. Figure 6.1 gives an overview of the idea of the bottleneck auto-encoder setup.

#### Input data

The mel-spectrograms are computed on 24 kHz audio. We use 80 mel-bands to represent the frequencies between 0 and 8 kHz. Frequencies above 8 kHz are discarded and expected to be completed coherently by the final mel-inverter. The mel-spectrogram has a hop size of 12.5 ms resulting in a 80-dimensional signal with 80 Hz sample rate. The mel-spectrogram is given with logarithmic amplitudes. The
<figure><img src="image_31.png" /><figcaption>**Figure 6.1.:** Schematic overview of the bottleneck auto-encoder idea. The encoder and the decoder are trained jointly to regenerate the original mel-spectrogram using a reconstruction loss. The decoder receives the $f_0$ as additional information. The information bottleneck ensures that the encoder removes the $f_0$ from the latent code and the decoder uses the $f_0$ from the conditional input. Additionally conditioning the encoder on the $f_0$ helps the encoder to identify the information that needs to be removed.</figcaption></figure>

$f_0$ curve is provided as a number pair (log($f_0$), voiced/unvoiced mask) and is generated from the audio signal by the $f_0$ estimation algorithm, F0-FCN from Ardaillon and Roebel (2019).

To obtain normalized input data we rescale the input data such that the interval [-1, 1] represents [-120 dB, -20 dB] for the mel-spectrogram and [45 Hz, 1400 Hz] for the fundamental frequency $f_0$.

## Network architecture

The auto-encoder is implemented as a fully CNN where the encoder and decoder are resembling mirrored structures. To benefit from the natural topology of the frequency axis we treat the mel-spectrogram as an image and perform 2D-convolutions on them, which facilitates training according to Bous and Roebel (2019). Through strides in frequency direction, the encoder reduces the frequency-axis to length 1 but keeps the time-axis unchanged. The decoder uses transposed convolutions to incrementally rebuild the frequency-axis. The exact architecture can be taken from Tables A.3 and A.4 (Page 161).

In the encoder, we start with an *image* with the structure time × frequency. This is stacked with the $f_0$ signal, which is resampled and extended on the frequency axis to match the size of the mel-frequency axis with a trainable linear projection (cf. Table A.3, Page 161). We then reduce the size of the frequency axis step-by-step through down-sampling until it has length 1. Down-sampling is performed with stride convolutions across the frequency dimension, and the down-sampling operations are intercalated with classic 3 × 3 convolutions. The last layer of the encoder, a 3 × 1 convolution, maps the features to the desired bottleneck size.

The decoder has the inverse structure with respect to the encoder. We start
with the concatenation of the latent code and the target $f_0$ curve and broaden the frequency axis step-by-step by up-sampling. Up-sampling is performed through transposed convolutions. The final convolution maps the number of features back to 1 recreating the shape of the input mel-spectrogram.

All convolutions use zero-padding to keep the shapes consistent. Activation functions are ReLU (Nair and Hinton, 2010). Encoder and decoder have 10 and 11 convolutional layers respectively, where each layer has 512 filters. The models have roughly 12.5 M parameters each.

Contrary to AutoVC (Qian, Yang Zhang, Chang, X. Yang, et al., 2019), there is no temporal down-sampling in our approach. As AutoVC does speaker identity conversion, the conditional input is constant in time. Thus, there is a precise temporal resolution is not important. Furthermore, the temporal down-sampling allows the bottleneck auto-encoder of Qian, Yang Zhang, Chang, X. Yang, et al. (2019) to adjust timings, which may render the voice identity conversion more natural. Here, however, we condition on the $f_0$ signal, which varies over time, and aim to preserve the temporal structure implied by the $f_0$ signal as precise as possible. Similarly, we use convolutional layers instead of recurrent, to focus the neural network to use more local features.

We also experimented with 1D-convolutions but found the results' quality to be inferior with respect to the 2D-versions. We tried growing the number of features after each down-sampling of the frequency-axis in the encoder, while equally decreasing the number of features in the decoder. However, we found no significant difference in the results when keeping the number of parameters and inference speed similar. Thus, we opted to keep the same number of features for each layer, regardless of the size of the frequency-axis.

## Training procedure

The auto-encoder is trained only on self-reconstruction. We apply a point-wise mean-absolute-error loss between the input mel-spectrogram and the output of the decoder. Optimization is performed with Adam (Kingma and Ba, 2014), with a learning rate of $1 \cdot 10^{-4}$ and the parameters $\beta_1 = 0.9$ and $\beta_2 = 0.999$. We use a batch size of 16 training-samples per batch with 80 time steps (equivalent to 1 s) per sample. All models were trained for 250 k updates.

### 6.2.2. Datasets

In our experiments we use four different datasets:

**si** A pure singing-voice dataset, obtained by combining several publicly available datasets and our own proprietary datasets (CREL, NUS-SMC, ICH, PJS, JVS-MuSiC, Tohoku, VocalSet, ISiS, and Farinelli (see Table 2.2, Page 34).)

**sp** A pure speech dataset consisting of the English dataset VCTK (Yamagishi et al., 2019) and the French dataset Att-HACK (Moine and Obin, 2020)

**vo** A hybrid dataset of speech and singing, consisting of the two datasets above
**sp-xs** A smaller speech dataset, consisting of a reduced number of VCTK speakers, to match the duration of the singing-voice dataset

The singing-voice dataset is a combined dataset of CREL, NUS-SMC, ICH, PJS, JVS-MuSiC, Tohoku, VocalSet, ISiS, and Farinelli (see Table 2.2, Page 34). In total, the singing-dataset contains 26.5 h of singing voice from 136 different singers, however 100 of them are from JVS-MuSiC and 94% of the total duration is sung by the remaining 36 singers. Most of the singers in the dataset are professionals, but a few of them are also amateurs. For datasets which also contained speech, only the singing recordings were used. The dataset contains singing in English, French, Greek, Italian, Japanese, and Latin.

For speech, we train on both VCTK (Yamagishi et al., 2019) and Att-HACK (Moine and Obin, 2020) which amounts to a combined dataset of English and French speech. The speech dataset contains a total of 134 speakers (109 from VCTK, 25 from Att-HACK) and 54 h of speech (32 h from VCTK, 22 h from Att-HACK).

To better compare the results between speech and singing voice, we also train models on a smaller speech dataset, consisting of only a subset of the VCTK speakers, resulting in 26.5 h of speech.

## Dataset splits

For evaluation, we withhold four speakers from the VCTK dataset in the following section, two female and two male speakers. These are p361, p362, p374 and p376. Thus, all evaluation of the speech models is performed on unseen speakers.

Test set for singing voice, consists of five samples each from a set of different singers, aimed to include a representative of each voice class. In particular the test set includes baritone, tenor, mezzo-soprano, soprano, child voice and countertenor, and singing style, including classical, pop, j-pop and Byzantine singing.

### 6.2.3. Nomenclature

To distinguish the models we use the following nomenclature: A model is named after the dataset it is trained on, e.g. si (singing), sp (speech) or vo (voice, speech + singing). The bottleneck size is given in the subscript. Additionally, the size of the model may be given by the number of features in the hidden layers indicated in the superscript. If no superscript is given, the default model size of 512 is implied. Thus, the model si₃²⁵₆ is a singing model with bottleneck size nᵢ = 3 and 256 features in the hidden layers. The model vo₆ has been trained on both speech and singing, uses a bottleneck size of nᵢ = 6 and has 512 features in the hidden layers. A table with all models trained for this chapter is given in Table 6.1, and explained in more details in the section below.

### 6.2.4. Experimental setup

The variety of sounds that can be produced with the human voice is rich and human voice is used in different contexts. Often a distinction is made between speaking
<table><caption>**Table 6.1.:** Final loss values (training (validation)) for the models used in Section 6.3. All values in dB. Nomenclature according to Section 6.2.3.</caption><thead><tr><th>n<sub>b</sub></th><th>si<sub>n<sub>b</sub></sub></th><th>sp<sub>n<sub>b</sub></sub></th><th>sp-xs<sub>n<sub>b</sub></sub></th><th>vo<sub>n<sub>b</sub></sub></th><th>sp<sub>n<sub>b</sub></sub><sup>256</sup></th><th>sp<sub>n<sub>b</sub></sub><sup>128</sup></th></tr></thead><tbody><tr><td>1</td><td>3.13 (3.46)</td><td>3.52 (3.55)</td><td></td><td></td><td></td><td>3.68 (3.79)</td></tr><tr><td>2</td><td>2.47 (2.91)</td><td>2.84 (2.89)</td><td>2.90 (3.36)</td><td>3.06 (3.17)</td><td>2.99 (3.03)</td><td>3.09 (3.12)</td></tr><tr><td>3</td><td>2.41 (2.70)</td><td>2.51 (2.55)</td><td></td><td>2.61 (2.72)</td><td></td><td>2.85 (2.84)</td></tr><tr><td>4</td><td>2.06 (2.44)</td><td>2.36 (2.39)</td><td>2.42 (2.74)</td><td>2.54 (2.64)</td><td>2.48 (2.51)</td><td>2.68 (2.68)</td></tr><tr><td>5</td><td>2.13 (2.45)</td><td>2.19 (2.21)</td><td></td><td></td><td></td><td>2.47 (2.46)</td></tr><tr><td>6</td><td>2.03 (2.29)</td><td>2.08 (2.11)</td><td>2.14 (2.38)</td><td>2.30 (2.39)</td><td>2.16 (2.19)</td><td>2.32 (2.33)</td></tr><tr><td>7</td><td></td><td>1.96 (1.99)</td><td></td><td></td><td></td><td>2.24 (2.24)</td></tr><tr><td>8</td><td></td><td>1.87 (1.90)</td><td>1.94 (2.17)</td><td></td><td>1.97 (2.00)</td><td>2.18 (2.21)</td></tr><tr><td>9</td><td></td><td>1.83 (1.85)</td><td>1.88 (2.14)</td><td></td><td>1.91 (1.93)</td><td>2.10 (2.10)</td></tr><tr><td>10</td><td></td><td>1.75 (1.76)</td><td>1.81 (1.98)</td><td></td><td>1.83 (1.85)</td><td>1.97 (1.96)</td></tr><tr><td>11</td><td></td><td>1.69 (1.71)</td><td></td><td></td><td>1.76 (1.77)</td><td>1.95 (1.95)</td></tr><tr><td>12</td><td></td><td>1.64 (1.66)</td><td>1.67 (1.87)</td><td></td><td>1.71 (1.73)</td><td>1.92 (1.92)</td></tr><tr><td>13</td><td></td><td>1.56 (1.58)</td><td></td><td></td><td></td><td>1.89 (1.89)</td></tr><tr><td>14</td><td></td><td>1.52 (1.54)</td><td>1.56 (1.75)</td><td></td><td>1.63 (1.65)</td><td>1.85 (1.85)</td></tr><tr><td>16</td><td></td><td></td><td></td><td></td><td>1.53 (1.54)</td><td></td></tr></tbody></table>

$n_b$ Bottleneck size

si Singing model (trained on singing dataset)

sp Speech model (trained on speech dataset)

sp-xs Speech model (trained on the small speech dataset)

vo Voice model (trained on speech and voice)

256, 128 Number of features in the hidden layers (default is 512)

and singing voice due to their different acoustic properties and applications. We compare two different approaches: either to train separate models for speech and singing or to train a combined voice model using training data of speech and singing voice. As speech and singing have their differences, training a model only on one domain may improve its quality as the complexity of the task is reduced. On the other hand, speech and singing share many similarities; training a universal model might allow using overlapping information and improve overall quality as a consequence. We will see, however, that for our application this is not the case neither for $f_0$ accuracy nor for audio quality.

Since this is the first time the $f_0$ is used as the only feature that is manipulated in a neural network, we cannot compare the proposed method to other neural-based baselines. However, the repertoire of classical signal processing based $f_0$ transformation algorithms is rich, and so we compare our auto-encoder with the PaN vocoder (Ardaillon, 2017), a variant of the parametric vocoders built upon the Liljencrats-Fant pulse model (Fant, Liljencrats, et al., 1985; Fant, 1995).

Since the size of the information bottleneck is the crucial part of the disentangle-
ment, we train models with different values for the bottleneck size. In our study
about the effect of bottleneck size we also vary the count of filters in the convolution
operations from the set {128, 256, 512} Unless explicitly stated, the count of filters is
512.

The loss values for the models we trained for this chapter can be found in Table 6.1.
We can see that for every configuration the final loss decreases with the increase
of bottleneck size. For singing and speech, small dataset we observe substantial over-
fitting due to the smaller dataset size; however, while the validation error is sig-
nificantly larger than the training error, it did not increase over time. For the
configurations trained on the full speech dataset almost no over-fitting can be
observed.

To allow the reader to get their own perceptive impression on the proposed
method, we created a website<fnref n="2" /> with audio examples from the models used in the
studies discussed below.

6.3. Experimental results

6.3.1. $f_0$ accuracy

To check if the proposed method actually changes the $f_0$ we apply the transposition
algorithm to our test sets and measure the difference between requested the $f_0$ and
the $f_0$ extracted from the synthesized audio.

Since our algorithm learns the correspondence between $f_0$ and mel-spectrogram
from the training data, it is not able to produce mel-spectrograms with $f_0$ values
that are not contained in the training dataset. We therefore limit the transpositions
to files from the test set such that the target $f_0$ lies between 53.8 Hz and 377 Hz for
speech and between 105 Hz and 746 Hz for singing. These ranges were chosen based
on the distribution of $f_0$ in the dataset, such that only 1% of the $f_0$ values from the
training set lie below the lower limit and only 1% above the upper limit.

Figure 6.2 plots the mean absolute $f_0$ error against the transposition in cent for
speech and singing separately for selected models. As one might expect, the larger
the transposition (in magnitude), the higher the mean $f_0$ error in most of the cases,
although the error does not increase much for many of the presented models. In
particular for singing, for models for which the bottleneck is sufficiently small the
error seems almost constant across a range of about -2300 to +1300 cent. It is
worth noting that for singing down-transpositions seems to work better, whereas
for speech up-transpositions seem to be slightly more stable. The transposition
seems to be more accurate for singing in general.

The models which were trained on both speech and singing provide a special
insight into how speech and singing are treated inside the model, as they appear in
both plots of Fig. 6.2. One might expect that they work similarly for both speech
and singing, but this is not at all the case for the model with bottleneck size 6, which
seems to work relatively well for speech but effectively fails for singing. In fact for

<footnote n="2">2. http://thesis.fnab.xyz/ch6/demo</footnote>
<figure><img src="image_32.png" /><figcaption>**Figure 6.2.:** Accuracy of the *f*₀ transformation given as mean absolute error between requested *f*₀ and synthesized *f*₀. Nomenclature according to Section 6.2.3. (Top) speech, (Bottom) singing.</figcaption></figure>
singing, bottleneck size 6 has roughly the same $f_0$ error, regardless whether it was trained only on singing or both singing and speech. One might expect that due to the higher complexity of the problem, the channel capacity needs to increase to properly accommodate all possibilities from the presented data. Instead, the model operates differently on speech and singing, successfully disentangling the former from the $f_0$ while keeping the $f_0$ in the latent code for the latter.

## Bottleneck size analysis

To choose the right dimensionality for the latent code we evaluate the disentanglement for various sizes of the bottleneck, by measuring the accuracy of the $f_0$ transformation. Let $f_t$ be the target $f_0$, let $f_s$ be the $f_0$ measured in the synthesized signal and $\tau$ the transposition, then we define the normalized mean $f_0$ error (NMFE) as

$$ \text{NMFE} = E \left[ \frac{|\log f_t - \log f_s|}{|\log \tau|} \right]. \qquad (6.9) $$

If the $f_0$ is perfectly disentangled from the latent code, the auto-encoder will produce a signal with the $f_0$ provided to the decoder and the NMFE will be zero.<fnref n="3" /> If, on the other hand, the decoder only uses the $f_0$ information contained in the latent code, then the $f_0$ error will be equal to the transposition and thus the NMFE will be one. Any value in between should give us a measure for the degree of entanglement.

We plot the NMFE against the bottleneck size for different configurations in Fig. 6.3.

For all configurations on speech we see a drastic increase in NMFE at around 8 or 9. For the speech configuration with 512 filters per convolutional layer (the default configuration) the NMFE increases between 9 and 10 but drops again for 12 and 13 to almost the same value as for the lower bottleneck sizes. At 14 the NMFE is finally multiple times worse. There is no reason to believe that models with bottleneck size 12 or 13 should generally perform better than bottleneck sizes 10 or 11 in terms of NMFE. This may be an effect of network initialization, since each of the data points is generated by training only one neural network and measuring its NMFE. It might well be that here for bottleneck size 13 the model by chance did not find the way to entangle the $f_0$ into the 14 dimensions preserving the small reconstruction error. Regardless of the explanation, for bottleneck sizes above 9 the disentanglement becomes unlikely.

For speech the critical point seems to be always the same, regardless of the modelling capacity of the neural network of the size of the training dataset. This is counter to the intuition that the computational capacity of the model is responsible to create the information bottleneck. However, as we do not actually measure the modelling capacities of the neural networks, no clear conclusions can be drawn from this observation. It seems that the critical point is rather stable for a wide range

<footnote n="3">3. Of course due to the approximations with neural networks, the NMFE will never be zero and some fraction of the NMFE will be due to inaccuracies of the model. The NMFE should be seen more as an indication rather than a proper measure of disentanglement.</footnote>
<figure><img src="image_33.png" /><figcaption>**Figure 6.3.:** Normalized mean $f_0$ error (NMFE) averaged over different transpositions ±660, ±1320 and ±1980 as a function of the bottleneck size $n_b$ for various configurations (see Table 6.1). The NMFE is calculated according to Eq. (6.9). Nomenclature according to Section 6.2.3.</figcaption></figure>

of neural networks with practical size, the networks in the study here have 50 M, 12.5 M and 3.13 M parameters respectively.

For singing voice the NMFE behaves quite differently. The NMFE is much lower for low bottleneck sizes (1-4) but already for a bottleneck size of 4 the NMFE is more than twice as high as at the beginning.

### 6.3.2. Synthesis quality

A low mean $f_0$ error for a wide range of transpositions implies disentanglement between $f_0$ and latent code. This does not necessarily mean that the model is in

<table><caption>**Table 6.2.:** Quality rating of the algorithms for speech averaged over all transposition values. Value is given with 95% confidence interval. Nomenclature according to Section 6.2.3.</caption><thead><tr><th>transp.</th><th>GT</th><th>PaN</th><th>sp<sub>2</sub></th><th>sp<sub>3</sub></th><th>vo<sub>3</sub></th><th>sp<sub>8</sub></th></tr></thead><tbody><tr><td>1760</td><td></td><td>1.41 ± 0.31</td><td>2.28 ± 0.54</td><td>2.50 ± 0.65</td><td>2.61 ± 0.62</td><td>2.11 ± 0.40</td></tr><tr><td>1320</td><td></td><td>1.74 ± 0.32</td><td>3.17 ± 0.52</td><td>3.65 ± 0.49</td><td>3.26 ± 0.46</td><td>3.00 ± 0.46</td></tr><tr><td>880</td><td></td><td>2.50 ± 0.53</td><td>3.72 ± 0.52</td><td>3.61 ± 0.50</td><td>3.33 ± 0.62</td><td>3.47 ± 0.42</td></tr><tr><td>440</td><td></td><td>2.48 ± 0.55</td><td>3.73 ± 0.50</td><td>3.41 ± 0.53</td><td>3.55 ± 0.48</td><td>4.09 ± 0.44</td></tr><tr><td>0 4.79 ± 0.09</td><td></td><td>2.90 ± 0.37</td><td>3.67 ± 0.37</td><td>4.10 ± 0.22</td><td>3.83 ± 0.31</td><td>4.10 ± 0.38</td></tr><tr><td>-440</td><td></td><td>3.06 ± 0.56</td><td>3.28 ± 0.49</td><td>3.83 ± 0.56</td><td>3.50 ± 0.60</td><td>4.18 ± 0.51</td></tr><tr><td>-880</td><td></td><td>2.22 ± 0.42</td><td>3.17 ± 0.57</td><td>3.41 ± 0.48</td><td>3.35 ± 0.58</td><td>3.61 ± 0.47</td></tr><tr><td>-1320</td><td></td><td>1.65 ± 0.43</td><td>2.83 ± 0.63</td><td>3.39 ± 0.58</td><td>3.72 ± 0.54</td><td>3.61 ± 0.53</td></tr><tr><td>-1760</td><td></td><td>1.61 ± 0.35</td><td>2.50 ± 0.53</td><td>2.68 ± 0.51</td><td>2.82 ± 0.53</td><td>2.50 ± 0.51</td></tr><tr><td>average</td><td>4.79 ± 0.09</td><td>2.25 ± 0.16</td><td>3.21 ± 0.18</td><td>3.47 ± 0.16</td><td>3.38 ± 0.17</td><td>3.48 ± 0.17</td></tr></tbody></table>
<table><caption>**Table 6.3.:** Quality rating of the algorithms for singing averaged over all transposition values. Value is given with 95% confidence interval. Nomenclature according to Section 6.2.3.</caption><thead><tr><th>transp.</th><th>GT</th><th>PaN</th><th>si<sub>2</sub></th><th>vo<sub>3</sub></th><th>si<sub>3</sub></th></tr></thead><tbody><tr><td>2200</td><td></td><td>2.00 ± 0.40</td><td>2.88 ± 0.68</td><td>2.41 ± 0.68</td><td>2.36 ± 0.74</td></tr><tr><td>1760</td><td></td><td>2.00 ± 0.61</td><td>2.40 ± 0.60</td><td>2.73 ± 0.59</td><td>2.80 ± 0.58</td></tr><tr><td>1320</td><td></td><td>2.65 ± 0.58</td><td>2.65 ± 0.68</td><td>2.88 ± 0.61</td><td>3.31 ± 0.56</td></tr><tr><td>880</td><td></td><td>3.93 ± 0.68</td><td>3.47 ± 0.57</td><td>3.73 ± 0.51</td><td>3.53 ± 0.73</td></tr><tr><td>440</td><td></td><td>4.35 ± 0.43</td><td>4.00 ± 0.47</td><td>3.35 ± 0.70</td><td>4.06 ± 0.57</td></tr><tr><td>0</td><td>4.75 ± 0.11</td><td>3.91 ± 0.48</td><td>4.25 ± 0.39</td><td>3.94 ± 0.46</td><td>4.09 ± 0.35</td></tr><tr><td>-440</td><td></td><td>3.67 ± 0.72</td><td>4.13 ± 0.45</td><td>3.87 ± 0.57</td><td>3.93 ± 0.47</td></tr><tr><td>-880</td><td></td><td>3.18 ± 0.69</td><td>3.35 ± 0.66</td><td>3.62 ± 0.65</td><td>3.47 ± 0.77</td></tr><tr><td>-1320</td><td></td><td>2.20 ± 0.61</td><td>2.80 ± 0.76</td><td>2.53 ± 0.53</td><td>3.36 ± 0.68</td></tr><tr><td>-1760</td><td></td><td>2.18 ± 0.62</td><td>2.56 ± 0.73</td><td>2.62 ± 0.77</td><td>3.12 ± 0.70</td></tr><tr><td>-2200</td><td></td><td>2.60 ± 0.66</td><td>2.87 ± 0.78</td><td>2.13 ± 0.60</td><td>2.57 ± 0.68</td></tr><tr><td>average</td><td>4.75 ± 0.11</td><td>3.06 ± 0.21</td><td>3.31 ± 0.19</td><td>3.14 ± 0.19</td><td>3.41 ± 0.19</td></tr></tbody></table>

fact useful for performing actual transpositions. The accuracy of the $f_0$ could be achieved by generating an arbitrary signal with the target $f_0$ unrelated to speech. In the case where the bottleneck size is 1, phonetic content is greatly reduced and the audio quality of the resynthesized mel-spectrogram actually degrades quite noticeably. Therefore, to analyse the synthesis quality, we conduct a perceptive test.

In the perceptive test we asked 96 participants to rate audio samples on a scale from 1 to 5, 1 being the worst and 5 being the best possible quality. The samples were randomly drawn from the test set, the models, various transpositions, as well as from the ground truth. The results are given in Table 6.2 for speech, Table 6.3 for singing and in Fig. 6.4 as a plot over the transposition amount.

Tables 6.2 and 6.3 contain 46 and 45 data points respectively. Thus, the coverage for each of the data points is rather thin, despite the large number of participants. Consequently, the confidence intervals are rather large.

For speech, the auto-encoder outperforms the classical vocoder with a strong margin and for all transpositions. For singing the classical PaN vocoder is not as bad. For transpositions up of +440 and +880 cent the classical vocoder is even slightly outperforming the auto-encoder, while for transpositions down of -440 and -880 cent the auto-encoder is slightly better. The auto-encoder shows an approximately symmetric performance for transpositions up and down. For the classical vocoder the performance is asymmetric, which is due to the fact that internally the vocoder preserves the spectral envelope (the formant structure) of the original signal. For higher $f_0$ the harmonics are spread further apart, which decreases the details in the spectral envelope (the formant structure). Therefore, preserving the spectral envelope for upwards transpositions does not pose a problem. On the other hand, for lower $f_0$ the harmonics are spread more densely, and more information about the spectral envelope (the formant structure) is perceptually expected. As the classical vocoder cannot create more details its perceptual result is worse. Note that the difference in this intermediate range of transpositions remains
<figure><img src="image_34.png" /><figcaption>**Figure 6.4.:** Mean opinion score plotted against transposition amount for speech (Tap) and singing (Bottom). The curves are slightly offset horizontally to prevent overlapping of the error bars. The error bars denote the 95% confidence interval. All data points are integer multiples of 440. The exact values can be read from Tables 6.2 and 6.3. Nomenclature according to Section 6.2.3.</figcaption></figure>
below 1 point in the MOS scale, and it is smaller than the 95% confidence interval observed in the perceptual tests.

For transposition above +/− 880 cent the auto-encoder considerably outperforms the classical vocoder by about 1 point on the MOS scale. Here the signal needs to change more dramatically to remain coherent, a change that the auto-encoder can create, while the classical vocoder fails.

Averaging over all transpositions, as shown in the bottom row of Table 6.3, reveals a significant dominance of the auto-encoder models over the classical vocoder (with a confidence of over 95%). One weak-point of the PaN vocoder is synthesizing consonants like plosives and fricatives. Those are much more frequent in speech than in singing and could explain the larger perceptual difference we found for speech.

As expected, the models tend to receive the highest rating for transposition 0, with a few exceptions that lie within the 95% confidence interval. For speech, most of the auto-encoder models manage to keep a perceptive rating above 3 by a large margin over the interval [−1320, 1320]. No auto-encoder model stands out among the others. In the average ratings bottleneck size 3 and 8 seem almost equally good. Only the model with bottleneck size 2 has a significantly lower overall rating. In the same way for singing there is no clear dominance of one model over all transpositions. The model with bottleneck size 2 seems to perform better for smaller transpositions while the model with bottleneck size 3 seems to perform better for larger transpositions. The model with bottleneck size 3 manages to keep a perceptive rating above 3 for over an interval of [−1760, 1320].

For both singing and speech we can observe that the universal model (trained on both speech and singing) generally performs worse than its domain specific counterpart with the same bottleneck size.

### 6.3.3. Investigation of the latent code

Since the model with bottleneck size 2 has achieved a rather good rating for singing voice in both mean $f_0$ error and mean opinion score, we can reasonably assume that two dimensions suffice to represent singing voice signals. Since the latent code in that model is only two-dimensional we can visualize it with a 2d-plot. This allows an investigation on how the model uses its two dimensions to represent the features of human voice internally.

In one of our singing datasets we have recordings from one singer with isolated notes for each vowel of the French language and for five different dynamics (pp, mp, mf, f, ff). Plotting the latent code of these notes allows us to see how the dynamics and the phonemes are encoded in the latent code.

The latent code vectors typically have a norm of less than 4 and outliers with a Euclidean norm above 10 are extremely rare. However, for better visualization we train a special model with a tanh activation at the end of the encoder to force all the samples into the unit square. Loss-wise, and in terms of objective ($f_0$) and perceptual accuracy, there is no difference with the equivalent model that uses no activation at the output of the encoder.
<figure><img src="image_35.png" /><figcaption>**Figure 6.5.:** Distribution of vowels in the latent code for various intensities. Top row: all intensities, pp, mp. Bottom row: mf, f, ff. Each vowel is represented in a different colour. The distributions are estimated as a Gaussian mixture model with one mode for each sample fixed standard deviation σ = 0.015 placed at the location of the sample.</figcaption></figure>

Figure 6.5 shows the latent codes for each frame in a scatter plot with each vowel in a different colour. The subplots show only the codes of frames of a fixed intensity. Fig. 6.6 shows the latent codes for each frame with each intensity in a different colour and with a subplot for each individual vowel. From both figures we can clearly see that intensity is ordered along a diagonal axis.

From Fig. 6.6 we can see that the phonemes are arranged in clusters. However, each phoneme forms more than one cluster, typically three to four. Looking at each code as a temporal sequence as plotted in Fig. 6.7, we notice that the code values hop periodically from one cluster to the other. More specifically, each phoneme is represented by a specific gesture usually repeating with a periodicity of 3 to 4 time-steps. Two effects could be used to explain this phenomenon. Firstly, due to the overlapping analysis windows, two consecutive frames are highly correlated. Here, we use an overlap of 75%, which coincides perfectly with the periodicity of 3 to 4. Furthermore, as human voice (and especially singing) normally does not change that quickly over time, there is further redundancy across consecutive frames. In our implementation, the encoder and decoder have rather large receptive fields — 11 and 13 time-steps respectively, which translates to 138 ms and 163 ms. Using their large receptive fields the networks in the auto-encoder seem to be able to discard this redundancy under the large pressure of the information bottleneck: in addition to the two feature dimensions, they also take advantage of the temporal
<figure><img src="image_36.png" /><figcaption>**Figure 6.6.:** Distribution of intensities in the latent code space for all vowels in the French language. Colour brightness translates to musical loudness: Dark blue is pp, yellow is ff. The distributions are estimated as a Gaussian mixture model with one mode for each sample fixed standard deviation $\sigma = 0.015$ placed at the location of the sample.</figcaption></figure>
<figure><img src="image_37.png" /><figcaption>**Figure 6.7.:** A few consecutive steps of latent code for each vowel in the French language showing how the latent code progresses through the latent code space. The arrows point from one point to the next, progression goes from dark to bright. Each vowel forms a specific gesture.</figcaption></figure>
dimension to represent phonemes.

## 6.4. Conclusion

In this chapter we have applied an auto-encoder with information bottleneck (Qian, Jin, et al., 2020) to speech and singing voice to disentangle the $f_0$ from the latent code. With this auto-encoder we were able to perform high quality transpositions of speech and singing voice.

Since it is possible to measure the $f_0$ in the transposed samples we were able to define a disentanglement measure and perform a study over the required size of the information bottleneck. We have seen that for singing voice disentanglement requires a bottleneck size of less than 4 ($n_b \le 3$). For speech, however, disentanglement can be achieved with a much larger bottleneck ($n_b \le 9$). With the appropriate bottleneck size we were able to perform coherent transpositions over a range of more than an octave for both speech and singing voice.

A perceptive test was carried out to study the quality of the transposed audio. While there is still a substantial difference between ground truth and transposed audio, the proposed method surpassed the state-of-the-art classical vocoder PaN Ardaillon, 2017 in perceptual ratings for both singing and speech. For speech the difference between auto-encoder and classical vocoder was severe.

Finally, we performed a deeper analysis of the latent code for the case of $n_b = 2$. We have seen that, while voice intensity corresponds to a clear direction in the latent space, the phonemes were not only encoded in position but rather through temporal gestures within the latent code.

## 6.5. Future work

While the auto-encoder has disentangled the $f_0$ from the latent code successfully, the resynthesized audio quality is still not en par with the original audio. For singing voice this is partly due to the small dataset size, and we indeed noticed an over fitting phenomenon for the singing models. Audio quality could thus be improved by training on a large singing dataset. For speech, we saw almost no over fitting when training the model on the 54 h large combined dataset. This means that improving audio quality here would not be as straight forward as for singing.

An adversarial approach could help improve audio quality by applying a discriminator either on the latent space as in fader networks Lample et al., 2017 or on the output mel-spectrogram.

The models analysed in Section 6.3 all used non-causal filters. However, informal listening tests showed that the quality does not degrade much if using causal filters instead. A causal model would even allow for real-time processing as the only delay would come from buffering the audio-frames to compute the mel-spectrogram.

As a side effect the proposed method is able to compress the mel-spectrogram to as little as 2 to 8 features per frame. It produces an efficient and visually interpretable representation of speech. A more thorough analysis of the latent code may allow
manipulation of the embedding directly in the latent space, potentially leading
to artistic applications in sound design. Currently, using float32 encodings, the
model with bottleneck size 2 compresses human voice to a data rate of 7.76 kB s⁻¹.
Concentrating on the compression could further reduce the data rate and produce a
highly compressed high-quality speech codecs.
# 7. Transformation of perceived voice level

In the previous chapter we have developed the bottleneck auto-encoder as a tool for voice transformation and demonstrated its capabilities on transformation of the fundamental frequency. Using the fundamental frequency as a first parameter is an obvious choice since it is easily available as an input parameter through existing analysis techniques. Its effect can easily be evaluated both perceptually and objectively and several other pitch transformation techniques exist that can be used as a baseline. The next challenge is to apply the bottleneck auto-encoder to other voice parameters. The bottleneck auto-encoder has already been applied to speaker identity through AutoVC (Qian, Yang Zhang, Chang, X. Yang, et al., 2019; Qian, Jin, et al., 2020), but numerous other voice properties exist that have never been addressed before, neither with the bottleneck auto-encoder nor with classical digital signal processing methods. In this chapter we will discuss the voice level and try to use the bottleneck auto-encoder to change the voice level in singing voice recordings.

<figure><img src="image_38.png" /><figcaption>Article</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>Audio samples</figcaption></figure>

<figure><img src="image_40.png" /><figcaption>Inference code</figcaption></figure>

Article: https://arxiv.org/abs/2204.04006

Audio Samples: http://thesis.fnab.xyz/ch7/demo

Inference Code: http://thesis.fnab.xyz/ch7/code

This chapter is an adaption of Bous and Roebel (2023) which was presented at the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

## 7.1. Introduction

### 7.1.1. Terminology

In this chapter, voice level shall denote the power with which voice is emitted from the speaker or singer, as defined by Traunmüller and Eriksson (2000). This is related to the *vocal effort* which is a property that describes how much effort is exerted onto the vocal folds to produce speech (or singing) (Traunmüller and Eriksson, 2000; Ford Baldner et al., 2015; McKenna and Stepp, 2018). Vocal effort is different from voice level as can be illustrated by comparing different phonemes: Singing on the vowel
[a] allows producing a much higher voice level than humming on the consonant [m] using the same vocal effort, or conversely the same voice level requires much more vocal effort for [m] than for [a]. More generally it has been observed that increase in voice level is a combined effect of ‘increased harmonic energy that is boosted by the vocal tract’ (Sundberg et al., 1993) in addition to increased subglottal pressure. Another related term is the *musical dynamic*, which is a musical term that describes the expressive intention relating to loudness. Musical dynamic is difficult to quantify because its execution is part of the interpretation of a piece and strongly depends on the epoch, genre, composer, and the context inside the piece. A general translation of musical dynamic to voice level or vocal effort is impossible.

Sometimes the term *voice intensity* or *vocal intensity* is used to denote voice level (Titze and Sundberg, 1992; Sundberg et al., 1993). A common confusion occurs when conflating these terms with the recorded level or intensity.<fnref n="1" /> Though a higher voice level will produce a recording with higher signal power if the recording conditions remain the same, these quantities are completely independent. The signal power can easily be changed by amplification but changing the volume in the playback device does not change the amount of power that was put into producing the note that is being played. This difference will be obvious to any listener. Similarly, voice level or vocal effort should not be confused with the ratio between voice and other sounds in a recording, for example between singing voice, and it's accompaniment in a musical piece. In classical singing, in order to be able to hear the voice over an accompaniment, the two must be balanced through singing- and playing technique of singer and accompaniment respectively. This will give characteristic changes in the timbre of the different instruments. In post-production we might change the signal power of the singing voice if voice and accompaniment were recorded separately, but the effect will be very different and trained ears will notice the difference. Thus, it is important to distinguish between voice level as a property of the source and signal intensity as a property of the representation of the voice.

Finally, we need to distinguish between power and loudness. Power is a pure
physical quality that is derived from the square of the signal (either as an averaged
frame-wise quantity or instantaneous using the Hilbert transform). Loudness is
a perceptual quantity that depends on how loud a sound is perceived. Perceived
loudness does not have a canonic definition but depends on the used loudness model.
The purpose of a loudness model is to capture empiric observations about human
perception of loudness. More details are discussed in Section 2.2.3. Voice level and
signal power find their perceptive pendant through *perceived voice level* and *signal
loudness* respectively. Signal loudness describes how loud a sound is when it arrives
in the ear and can be calculated with a given loudness model. Perceived voice level
is a spectral property that describes how loud the source sounds, independent of
how loud the source really was or how much power the sound still has in the ear.

<footnote n="1">1. In the literature we see vocal intensity denoted by the abbreviation SPL (sound pressure level) (Titze and Sundberg, 1992; Sundberg et al., 1993) as it is measured by a sound pressure meter at a fixed distance to the mouth. Sound pressure level is a measure of signal power at a specific point. In a controlled experimental set up the SPL can be in fact used to measure the voice level, however, in general these are two different concepts and the distinction is a crucial point in this chapter.</footnote>
It is the perceived voice level that best reflects what we want to change in voice recordings.<fnref n="2" />

## 7.1.2. Problem formulation

The central challenge addressed in this chapter with voice level and vocal effort is, that we cannot extract these qualities from voice recordings. The voice level could be measured with a calibrated sound pressure level meter close to the singer and several studies investigating the relationship between the voice level and other voice parameters have done so in the past (Titze and Sundberg, 1992; Sundberg et al., 1993; Traunmüller and Eriksson, 2000). Generally however, singing recordings do not come with voice level annotations and none of them are available to us. Vocal effort is even more difficult to measure as it requires knowledge about the effort put into the voice which can only really be evaluated psychologically. Though not exactly the same, vocal effort is highly correlated with the subglottal pressure (Ladefoged and McKinney, 1963; Isshiki, 1964; Sundberg et al., 1993). Measuring subglottal pressure, however, requires special equipment which is not widely available. Thus, to be able to manipulate voice level, first we need to find a method to obtain voice level from voice recordings, which is the main contribution of this chapter.

In speech, we face the additional difficulty that modes of speech production change, when speakers adjust their voice level: When speaking very quiet, people tend to whisper, when speaking very loudly, people tend to scream. Articulation and accentuation, speech rhythm and pitch change when the voice level is changed especially when switching to whispering or screaming. Thus, in speech, voice level, is highly dependent on other factors, like the fundamental frequency and timing. To create a convincing change in voice level these properties need to be adjusted accordingly. The bottleneck auto-encoder, as proposed in Chapter 6, however, is not able to change the timing. Experiments to change the voice level in speech using the method presented below have not yielded naturally sounding changes which suggests that heavier adjustments of the approach from Chapter 6 are required to transform voice level in speech.

For singing voice, pitch, and timing are constrained by the melody and music respectively and are not caused by the intended voice level. The bottleneck auto-encoder from Chapter 6 is well suited in this situation as it preserves the timing and, provided it is conditioned on the fundamental frequency, preserves the pitch as well. Thus, in this chapter we focus the discussion on singing voice.

## 7.1.3. Outline

The remaining chapter is structured as follows: In Section 7.2 we introduce the mathematical modelling and present two ways to estimate the voice level from

<footnote n="2">2. Applying the strict definition of voice level here, changing the voice level in a voice recording is impossible. Changing the voice level would require going back in time and getting the singer to change the way they sing during the recording. What we really want is just to modify the recording in a way such that it sounds like the singer had sung differently, thus the distinction regarding perceived voice level.</footnote>
voice recordings where no voice level annotation exists. In Section 7.3 we present the adaptions of Chapter 6 to include the voice level as a controllable parameter. We will explain our experimental validation in Section 7.4 and present and discuss the results in Section 7.5. Finally, we will see a short summary and an outlook in Section 7.6.

## 7.2. Extracting voice level from audio recordings

Audio recordings are not calibrated measurements. The goal of a recording is to produce a signal that when played on a speaker will create sound waves that sound like the original source. The scaling of these signals is irrelevant as it is expected to be adjusted by the consumer or the sound engineer that further processes the sound. Thus, each recording has a different relationship between the source's power and the recorded signal's power as microphones have different directivities and transfer functions and the signal gain is adjusted to minimize quantization noise when digitalizing the microphone signal.

Therefore, without explicit annotation it is impossible to infer the voice level from the power of an audio recording. Still, if we look at the speech production mechanism, we notice that humans cannot increase the voice level without changing other properties of their voice (Traunmüller and Eriksson, 2000; Holmberg et al., 1988; Titze and Sundberg, 1992; Henrich et al., 2005). Thus, we should be able to infer the perceived voice level from the signal's timbre.

The perceived voice level is a function of the timbre. The goal of estimating the (perceived) voice level in this chapter is to change the perceived voice level in a recording. Hence, for this purpose, we would like to assign the same perceived voice level to two recording if the voices in both recordings sound like they have been sung equally loud.<fnref n="3" /> This necessitates using perceptive loudness measures to be able to compare different sounds. Thus, in this derivation we will be using the term 'loudness' rather than 'power' to distinguish between perceptive and objective measures. In the following we will omit the term 'perceived' when talking about perceived voice level, unless it is necessary to distinguish it explicitly from objective voice level.

### 7.2.1. Objective

#### Voice level as a property of the timbre

We want to infer the voice level $v$ as a perceptive quantity of the signal's timbre. The voice level is independent of the scale of the signal. Let $x$ be a signal containing voice. Thus, for all factors $a \in \mathbb{R}$

$$v_x(n) = v_{a \cdot x}(n) \qquad (7.1)$$

<footnote n="3">³. This illustrative description serves to motivate what we expect from the voice level. In fact, we do not assign a voice level value to a whole recording but rather treat the voice level as a time-varying property.</footnote>
(the voice level $v_x$ of $x$ is the same as the voice level $v_{a \cdot x}$ of $a \cdot x$).

Since $v$ is a spectral property we will treat $v$ as a frame-wise property and assume $v$ is a function of the signal frames

$$ \mathbf{x}(n) := (x(n+k))_{k=-N}^{N} \qquad (7.2) $$

$$ = [x(n-N), x(n-N+1), \dots, x(n+N)] \qquad (7.3) $$

with frame length $2N+1$, with $N \in \mathbb{N}$, centred around $n$. To ensure the robustness to arbitrary gain changes from Eq. (7.1), any practical implementation $F$ must remove the scale of the signal during its computation. When using neural networks, the normalization should occur before the frame is fed to the neural network to ensure the arbitrary gain is not used. Thus, we should calculate $v_x(n)$ on the normalized frame $||\mathbf{x}(n)||^{-1} \mathbf{x}(n)$.

$$ v_x(n) = F(\|\mathbf{x}(n)\|^{-1} \mathbf{x}(n)) \qquad (7.4) $$

by some function $F: \mathbb{R}^{2N+1} \to \mathbb{R}$.

In this section we will investigate how $F$ can be approximated using a neural network. The problem that arises from this objective is that ground truth values for $v$ are not available. Thus, we need to find a way to train the neural network with the information that is available from the signal.

## Voice level and signal loudness

In this derivation we define voice level $v$ as the loudness with which the signal is emitted, while the signal loudness $p$ is the loudness of the signal as it is received. We measure the voice level at a fixed point in relationship to the singer, e.g. at their mouth (or a fixed distance from the singer's mouth). When we move around in the room, the signal loudness will change. Similarly, we may change recording settings, like the pre-amplification, which will likewise change the signal loudness. We make the assumption that all these effects can be modelled by a recording factor $a$, that in the general case has to be assumed time varying. This leads to the relationship

$$ p(n) = a(n) \cdot v(n). \qquad (7.5) $$

Different ways to estimate the signal loudness $p$ from the signal exist in the literature and the choice of loudness estimation technique influences the estimation of the perceived voice level. We will discuss practical matters on the loudness estimation in Section 7.4.2. From Eq. (7.5) the voice level could be calculated by solving for $v$. The problem is, however, that the recording factor $a$ is unknown. In the remainder of this section we propose two different techniques to estimate the voice level in the presence of an unknown recording factor $a$.

### 7.2.2. Learnt recording factor

Let $X = \{x_1, x_2, \dots\}$ be a set of voice recordings. Assume that all recordings from $X$ have been recorded under the same conditions and have been processed with the
same post-processing. That is, same microphone, pre-gain, spatial positioning of speaker and microphone, normalization, etc. have been used. In this case we can assume that for each recording $x \in X$ the recording factor $a$ is a constant such that loudness contour of the recorded signal $p$ is proportional to the voice level $v$:

$$p(n) = a \cdot v(n) \qquad (7.6)$$

For multi speaker databases it is highly likely that some of these assumptions are violated; however, it is not unlikely that these assumptions still hold for all recordings of a fixed speaker $s$. Therefore, we get the relationship

$$p(n) = a_s \cdot v(n) \qquad (7.7)$$

for all recordings generated by a speaker $s$ with a different recording factor $a_s$ for each speaker.

The speaker dependent recording factor $a_s$ can be learnt alongside the weights $\theta$ of a neural network $N_{\theta}$. As discussed in Section 7.2.1 we have to normalize the frame $x(n)$ before feeding it to the neural network. We normalize $x(n)$ to ensure the scale invariance of Eq. (7.1) and to make sure the neural network does not use the signal loudness instead of the timbre. This step is crucial for this method to work. Using an $L_2$ error, we get the following error function

$$\sum_n \|p(n) - a_s N_{\theta} (\|x(n)\|^{-1} x(n))\|^2. \qquad (7.8)$$

The resulting neural network $N_{\theta}$ allows estimating the voice level from the normalized signal: After training the neural network, we can calibrate the voice level estimator by using the same recording factor $a$ for all recordings. Once the recording factor has been fixed for all recordings, the relationship between $p$ and $v$ is no longer affected by the recording conditions. In this case we can compare the recordings as if they were made under the same recording conditions. To calibrate the estimations to a specific recording we can use the recording factor of that recording. Note that the speaker specific effects are part of the speaker dependent recording factor $a_s$.

### 7.2.3. Adaptive recording factor

The assumptions from Section 7.2.2 require the recordings from the training dataset to be grouped by same recording conditions. This works well if the number of speakers is small, and we can be sure that the recordings have not been normalized separately. However, for many databases we don't know what kind of post-processing has been performed or whether the samples for a fixed speaker have been created over multiple recording sessions with slightly different conditions. In this case we would have to assign a different recording factor to each recording. With the previous approach this causes problems as a gradient exists for a specific recording factor only if a sample associated with this recording factor is present in the training batch. Thus recording factors with a small percentage of associated recordings in the dataset are learnt very slowly as they are updated rarely. For the
case where we cannot group the recordings into a reasonable amount of classes we propose an adaptive recording factor:

Let $\hat{v} := N_{\theta}(\|\mathbf{x}(n)\|^{-1} \mathbf{x}(n))$ be the output of the neural network and $p$ the loudness curve associated with the input sample. We assume the recording factor is still constant for a fixed recording but now different for each training sample. Prior to calculating the loss we estimate the recording factor $a$ such that the $L_2$ error

$$e_a = \sum_n (p(n) - a \hat{v}(n))^2 \qquad (7.9)$$

is minimal for each training sample:

$$\hat{a} = \underset{a}{\operatorname{argmin}} e_a \qquad (7.10)$$

For a fixed pair of target and estimate $(p, \hat{v})$ there exists a closed form solution for $a$:

$$\hat{a} = \frac{\sum_n p(n) \hat{v}(n)}{\sum_m (\hat{v}(m))^2}. \qquad (7.11)$$

Combining Eqs. (7.9) and (7.11) and normalizing $e_{\hat{a}}$ by $\|p\|^2 = \sum_n (p(n))^2$ yields the scalar product loss:

$$e_{\text{scp}} := \frac{e_{\hat{a}}}{\|p\|^2} = 1 - \frac{\left(\sum_n p(n) \hat{v}(n)\right)^2}{\left(\sum_m (p(m))^2\right) \left(\sum_k (\hat{v}(k))^2\right)} = 1 - \frac{(p \cdot \hat{v})^2}{\|p\|^2 \|\hat{v}\|^2} \qquad (7.12)$$

Using $e_{\text{scp}}$, the neural network $N_{\theta}$ can be trained without knowing the recording factor $a$. To calibrate the estimation during inference to the recording environment of a specific recording, we can calculate the recording factor according to Eq. (7.11) and rescale the network output by that factor.

## Geometric interpretation

Given two vectors $\mathbf{a}, \mathbf{b} \in \mathbb{R}^n$, the scalar product $\cdot$ and their relative angle $\angle$ are related by the equation

$$\mathbf{a} \cdot \mathbf{b} = \|\mathbf{a}\| \|\mathbf{b}\| \cos \angle(\mathbf{a}, \mathbf{b}). \qquad (7.13)$$

Thus, the scalar product loss $e_{\text{scp}}$ can be interpreted geometrically as

$$e_{\text{scp}} = 1 - \frac{(p \cdot \hat{v})^2}{\|p\|^2 \|\hat{v}\|^2} \qquad (7.14)$$

$$= 1 - \cos^2 \angle(p, \hat{v}) \qquad (7.15)$$

$$= \sin^2 \angle(p, \hat{v}) \qquad (7.16)$$

$$\approx (\angle(p, \hat{v}))^2. \qquad (7.17)$$
Minimizing $e_{scp}$ minimizes the angle between the signal loudness $p$ and the estimated vocal effort $\hat{v}$ in the vector space $\mathbb{R}^N$, where $N$ is the length of the signals $p$ and $\hat{v}$. If $e_{scp} = 0$, the signal loudness $p$ and the vocal effort $\hat{v}$ are the same up to a scale factor. This reflects the requirement of Eq. (7.5) and the assumption that $a$ is constant.

The geometric interpretation implies that this method can only work if there is sufficient variation in the training samples. This suggests that the time-dimension of the training samples must be long enough to include enough variation. This method does not learn how to infer the voice level from the timbre itself. The change in voice level that causes a change in timbre is what allows the model to learn the relationship between timbre and voice level.

## 7.3. Proposed voice level transformations

Having a method to infer the voice level from audio recordings could be useful for many applications and in different disciplines. In this chapter we focus our attention on transforming the voice level in singing voice. We use the bottleneck auto-encoder from Chapter 6 to disentangle the voice level from the mel-spectrogram of the voice recordings.

Two possible approaches to include the voice level in the bottleneck auto-encoder arise from the previous chapter: Either replace the $f_0$ with the voice level, or condition the auto-encoder on both $f_0$ and voice level. We experimented with both configurations but found a joint conditioning on both $f_0$ and voice level to work better. Similar to the observations of Qian, Jin, et al. (2020) we noticed that explicitly imposing the $f_0$ helps to enforce consistency. In music, pitch, and intensity are generally treated independently, so we would not want the $f_0$ to change due to a change of voice level, and it makes sense to fix the $f_0$ during the process.

The inverse is not true though: For example, it is very difficult to sing high notes very quietly or low notes with very high voice level. Thus, when changing the pitch it would be acceptable and might even be necessary that the voice level was adjusted accordingly. Thus, to change the voice level, it is recommended to use an auto-encoder conditioned on both voice level and $f_0$. To change the $f_0$, it is recommended to use an auto-encoder only conditioned on the $f_0$ as introduced in Chapter 6.

In speech, the instantaneous $f_0$, is not a very important speech property, but rather the evolution and contour, e.g. the $f_0$ going up at the end of questions. Moreover, as discussed by Traunmüller and Eriksson (2000), the $f_0$ tends to increase for increased voice level and further contour details may change as well. Thus, to transform voice level in speech, it would be necessary to infer a new $f_0$ curve from the imposed voice level. We tried to automatically adjust the $f_0$ by only disentangling the voice level and leaving the to be modelled in the latent code. However, with this approach, we were not able to successfully adjust the $f_0$ in the result. This suggests that the bottleneck auto-encoder trained to disentangle the voice level from the mel-spectrogram in speech was not able to properly disentangle the voice level from the fundamental frequency. A more forced disentanglement between voice level
and fundamental frequency is required for speech. Thus, for the rest of the chapter we focus our attention to the voice level in singing voice and leave the application to speech for future work.

We can use this auto-encoder to validate the proposed voice level estimator that was introduced in Section 7.2 and show that it really represents the (perceived) voice level. If the transformed recordings are perceived to have been sung with a voice level that is coherent with the intended transformation (louder, less loud), we can conclude that the proposed voice level estimators in fact encode the voice level.

## 7.4. Experiments

### 7.4.1. Data

All experiments are trained on the same dataset as in Chapter 6, which is a combined dataset of CREL, NUS-SMC, ICH, PJS, JVS-MuSiC, Tohoku, VocalSet, ISiS and Farinelli (see Table 2.2, Page 34). Most of the singing recordings were not made with deliberate variation in voice level. However, since singing naturally contains variations in dynamic and consequently variations in voice level, we can assume that the dataset contains sufficient variations in voice level.

### 7.4.2. Target curves

During the training process, the voice level estimators are trained to match the signal loudness contour $p$. We calculate the perceived loudness with a simplified version of Glasberg and Moore (2002). To obtain the loudness, first, we normalize the power spectrum by the 70 dB equal loudness contour $L_{70 \text{ dB}}$ of the ISO 226: 2003 standard. Then, we apply an equivalent rectangular bandwidth (ERB) filter bank $B_{\text{ERB}}$ to the spectrum (Glasberg and Moore, 2002). Finally, the values for each ERB bin are converted to sone by using the exponent $\alpha = \log_{10}(2) \approx 0.301$ and added together. Given the spectrum $X$ of a frame of the signal, the corresponding loudness is calculated by the formula

$$p = \sum (B_{\text{ERB}} L_{70 \text{ dB}}^{-1} |X|^2)^{\alpha}, \quad (7.18)$$

where $B_{\text{ERB}} \in \mathbb{R}^{n_{\text{ERB}} \times n_{\text{fft}}}$ is the ERB filter bank and $L_{70 \text{ dB}} \in \mathbb{R}^{n_{\text{fft}} \times n_{\text{fft}}}$ is a diagonal matrix with the ISO 226: 2003 70 dB equal loudness contour sampled at the fft-frequencies on its diagonal. To calculate the spectrum from the signal given at 24 kHz, we use a window size of 32 ms, an fft-size $n_{\text{fft}} = 1024$. To calculate the ERB filter bank $B_{\text{ERB}}$, we consider the frequencies between 20 Hz and 12 kHz with an overlap of 4 which results in $n_{\text{ERB}} = 145$ ERB frequency bands. Significant differences of this approach to Glasberg and Moore (2002) is that we use only a single frame length, use the ISO 226 equal loudness contour and do not take masking and the threshold of hearing into account.

Furthermore, we run a voice activity detection on the voice signal and set the target curve to zero for frames where no voice is present (neither voiced nor unvoiced).
This forces the voice level estimator to produce voice level $v(n) = 0$ if no voice is produced in frame $n$ and thus teaches the model to ignore background noise.

## 7.4.3. Architecture

We train two voice level estimators, one with the learnt recording factor from Section 7.2.2 (Le) and the other with the adaptive recording factor strategy from Section 7.2.3 (Ad). Since we do our voice transformations based on the mel-spectrogram, the voice level estimators receive already pre-computed the mel-spectrogram as input. Furthermore, the mel-spectrogram is normalized frame-wise as required by Eq. (7.4). We use the same analysis parameters as in Chapter 6 to generate the mel-spectrogram.

Both networks Le and Ad have the same architecture which is given in Table A.5 (Page 162). The networks are simple convolutional feed-forward networks with 10 layers. Convolutions are 1d, treating the frequency bins as features. The number of filters is 100 in most of the layers, except the first, which has 80, the second to last, which has 50, and the last which has 1. The filter size is 3 in the first two layers and 1 elsewhere. With a step size of 12.5 ms per frame the voice level estimator has thus a receptive field of 5 frames or 50 ms.

We train the models with a batch size of 256 training samples of 80 frames (or 1 s) each. The models are trained for 500 k updates using the ADAM (Kingma and Ba, 2014) optimizer ($\beta_1 = 0.9, \beta_2 = 0.999$) with an initial learning rate of $1 \cdot 10^{-4}$. The learning rate is reduced by a factor of $\sqrt{0.1}$ if the validation loss does not decrease for a period of 16 k updates, with a minimum learning rate of $1 \cdot 10^{-6}$.

## 7.4.4. Auto-encoders

We train two auto-encoder configurations, one for each voice level estimator. For the auto-encoder we use almost the same architecture as in Chapter 6 (see Table A.3 and Table A.4) only with the conditional input changed, as we add the voice level to the list of conditional inputs. Thus, the auto-encoders are conditioned on the voice level, the $f_0$ and the voiced-unvoiced mask. We use the $f_0$ model of Ardaillon and Roebel (2019).

## 7.4.5. Evaluation methods

Since the dataset that we used to train our models does not include annotations for the voice level, we cannot evaluate the models directly with a ground truth. The hypothesis of this chapter is that we can extract meaningful information from the spectral properties of the mel-spectrogram about the perceived voice level. We will show that the information extracted by our voice level estimators reflects the perceived voice level in two ways. Firstly, by analysing data labelled with musical dynamic and, secondly, by analysing what the auto-encoder that is trained to use the estimated voice level does with the provided information.

We use recordings that are labelled into different dynamics to investigate the relationship between dynamic and the proposed voice level estimator. Labelled
data was obtained by Ardaillon (2017) who asked a singer to sing one note in all combinations of the dynamics pp, mp, mf, f and ff and all French vowels on the same pitch. Thus, we can create histograms over the estimated voice level for different dynamics and vowels. If the proposed method reflects the true voice level, lower dynamics should correspond to lower values in voice level.

Furthermore, we claim that the proposed voice level estimate is useful for changing the perception of how loud a phrase has been sung. Thus, if the proposed auto-encoder indeed succeeds in changing the signals properties in a way that it is perceived as sung with the desired voice level, we can conclude that the proposed voice level estimators indeed encode the voice level. To this end we asked 40 participants in a perceptive online study to rate the perceived voice level change of the transformed audio. Participants were presented pairs of audio where for each pair both files were generated using one of the auto-encoders and where in one file the voice level was changed while for the other the voice level was kept the same. Participants were then asked to rate which recording sounded as if it was sung with a stronger / louder voice and could give an answer of -2, -1, 0, 1 or 2. The order within each pair and the overall order of the pairs were randomized and the volume of each of the files was normalized to the same average loudness according to the loudness model of Glasberg and Moore (2002). In a second test, we asked 26 participants to rate the audio samples for their audio quality and computed a mean-opinion-score for each of the voice level changes of both models and the ground truth reference.

The auto-encoder's precision is evaluated by measuring the average difference between requested voice level and voice level measured in the synthesized mel-spectrograms for various voice level changes. The results are given in Table 7.1. The samples used for the perceptive test are available online<fnref n="4" />.

## 7.5. Results

### 7.5.1. Classification of dynamic

Figure 7.1 visualizes the relationship between estimated voice level estimator of Ad and the symbolic dynamic annotation of the recordings used for this investigation. Similar histograms are produced by estimator Le. First, we observe that louder dynamics produce higher voice level values, so we can conclude that the voice level does indeed include information about how loud a note has been sung. If we look at the individual phonemes, we see that the dynamics create different clusters with increasing mean though for some vowels some dynamics have rather large overlap, e.g. for [e] the pp and mp are largely overlapping: This is because we see the histogram of the voice level for each frame which naturally varies over time for the same dynamic for instance as a side effect of vibrato. Thus while some selected frames from different dynamics may be equally loud, the overall notes still have different loudness and can be measured as averages over larger durations. Furthermore, dynamics are subjective and a matter of interpretation. It could be

<footnote n="4">4. http://thesis.fnab.xyz/ch7/demo</footnote>
<figure><img src="image_41.png" /><figcaption>**Figure 7.1.:** Estimated voice level values for recordings in different dynamics plotted for the voice level model Ad. The graphs show smoothed relative histograms using a Gaussian kernel with σ = 0.31 dB. Only frames inside stable phonation are used, unvoiced frames and frames close (50 ms) to a voiced / unvoiced boundary are ignored. We show histograms for four selected vowels (out of a total of 15 vowels in the French language) and the average over all vowels (bottom). All graphs share the same horizontal axis.</figcaption></figure>
<table><caption>**Table 7.1:** Transformation precision [dB] of the auto-encoders. Errors are calculated between target voice level (as given to the auto-encoder) and voice level of the output (the voice level estimator applied to the transformed mel-spectrogram) for various voice level changes. The error is calculated on a logarithmic scale to provide meaningful values.</caption><thead><tr><th></th><th>-16 dB</th><th>-10 dB</th><th>0 dB</th><th>+10 dB</th><th>+16 dB</th></tr></thead><tbody><tr><th>AdA</th><td>4.02</td><td>2.49</td><td>1.43</td><td>3.22</td><td>4.15</td></tr><tr><th>LeA</th><td>4.50</td><td>2.91</td><td>1.35</td><td>4.97</td><td>7.77</td></tr></tbody></table>

that this particular singer does not make such a big difference between pp and mp for an [e] in terms of voice level.

Comparing the histograms from different vowels we observe that the voice level can be very different for different phonemes under the same dynamic. This is because not all vowels are in fact equally loud. The vowel [a], being an open vowel, can be sung much louder than [u], being a close vowel. Thus, for the same dynamic we can expect the resulting voice level to be different, and it is no surprise that the estimated voice level for [a] is much higher than for [u]. This explains why, if we average over all vowels, adjacent dynamics overlap — because the variations in voice level due to the phoneme are on the same scale as the variations due to the dynamic. Nevertheless, we can still very clearly distinguish between loud dynamics (f, ff) and quiet dynamics (pp, mp).

### 7.5.2. Accuracies

From Table 7.1 we can see that the auto-encoders can use the information provided by the voice level estimator and successfully disentangle the voice level from the mel-spectrogram. The auto-encoder with adapted recording factor (AdA) performs with higher precision than the auto-encoder with learnt recording factor (LeA). This indicates that the adapted recording factor model (Ad) is more robust than the learnt recording factor model (Le). Given that the assumptions for Le are stronger than for Ad, the difference could be explained by some of those assumptions being in fact violated, which caused the voice level estimator Le to provide inconsistent information.

### 7.5.3. Perceptive test

The results of the perceptive test are plotted in Figs. 7.2 and 7.3 and given in Tables 7.2 and 7.3. The perceived voice level increase is given in Table 7.2. We can see, that both models are able to create a noticeable change in voice level for small changes in the target voice level. For the auto-encoder with adaptive recording factor (AdA) we observe that it has trouble creating convincing results for high increases in voice level. Investigating the files that were rated in the opposite direction we noticed that for those files the auto-encoder introduced significant amounts of artefacts which seemed to have created the opposite of the desired effect. The perceptive test suggests that the auto-encoders have a more noticeable impact when decreasing
<figure><img src="image_42.png" /><figcaption>**Figure 7.2.:** Perceived voice level change caused by changing the voice level with the proposed auto-encoders for various amounts. The values in this figure are averaged over all answers. Error bars represent the 95% confidence interval.</figcaption></figure>

<table><caption>**Table 7.2.:** Change in voice level as perceived by participants of a perceptive test on a subjective scale from -2 to 2 with 95% confidence intervals.</caption><thead><tr><th>Voice level</th><th>-16 dB</th><th>-10 dB</th><th>10 dB</th><th>16 dB</th></tr></thead><tbody><tr><td>AdA</td><td>-1.57 ± 0.32</td><td>-0.84 ± 0.20</td><td>0.55 ± 0.20</td><td>0.08 ± 0.32</td></tr><tr><td>LeA</td><td>-1.29 ± 0.33</td><td>-0.74 ± 0.22</td><td>0.69 ± 0.18</td><td>0.83 ± 0.22</td></tr></tbody></table>

the voice level, which can be seen for both models and for all amounts of change. The auto-encoder with adaptive recording factor AdA seems to create a stronger effect when decreasing the voice level than the auto-encoder with learnt recording factor LeA. For increasing the voice level LeA seems to be better suited than AdA.

The quality ratings are given in Table 7.3 For self-reconstruction and small amounts of voice level change both models, AdA and LeA, seem to work equally well. For large changes in voice level the auto-encoder with learnt recording factor (LeA) outperforms the auto-encoder with adaptive recording factor (AdA) by a margin. For increases in voice level LeA is able to hold its level of quality even for an increase of 16 dB. On the other hand LeA does make a larger error in Table 7.1. For decrease in voice level both auto-encoder models suffer strong degradation in quality, although both models were successfully able to convince the participants that the recordings had much less voice level. Listening to these samples reveals that the auto-encoders increase the background noise significantly. Since our mel-inverter MBExWN does not handle synthetic noise well, the overall audio quality is poor in these cases although the conversion itself is realistic.

From the test results we can conclude that the given auto-encoders were able
<figure><img src="image_43.png" /><figcaption>**Figure 7.3.:** Subjective quality as reported by the participants on a scale from 1 to 5. The values in this figure are averaged over all answers. Error bars represent the 95% confidence interval.</figcaption></figure>

<table><caption>**Table 7.3.:** Quality rating given by participants of a perceptive test on a subjective scale from 1 to 5 with 95% confidence intervals.</caption><thead><tr><th>Quality</th><th>-16 dB</th><th>-10 dB</th><th>0 dB</th><th>10 dB</th><th>16 dB</th></tr></thead><tbody><tr><th>GT</th><td></td><td></td><td>4.64 ± 0.12</td><td></td><td></td></tr><tr><th>AdA</th><td>2.12 ± 0.49</td><td>3.32 ± 0.37</td><td>3.82 ± 0.25</td><td>3.55 ± 0.31</td><td>2.78 ± 0.37</td></tr><tr><th>LeA</th><td>2.61 ± 0.44</td><td>3.40 ± 0.39</td><td>3.80 ± 0.26</td><td>3.55 ± 0.30</td><td>3.55 ± 0.39</td></tr></tbody></table>

to change the perceived voice level in singing voice, and therefore the voice level estimators capture the information about the true voice level.

## 7.6. Conclusions

We have introduced a method to estimate the voice level from recordings with unknown amplification factors (recording factor). Two variants to overcome the missing recording factor annotation have been introduced: either by learning the unknown recording factor alongside the weights of the neural network (Le) or by adjusting the loss function to remove scaling and only compare the contours of the network's output and the associated signal loudness (Ad). These voice level estimators have been used to condition a bottleneck auto-encoder to disentangle the voice level from the mel-spectrogram. We have shown that both models produce consistent values and can produce the effect of changed voice level on singing recordings in most cases and with acceptable quality.

While the proposed auto-encoders produce a noticeable change in voice level, the audio quality is still significantly lower than real recordings especially when increasing the voice level. Consequently, this first publication on neural voice level transformation has to be seen as a proof-of-concept rather than a well-polished system. Improvements to the audio quality are required for this method to be used
in actual musical production. Nevertheless, the estimation method for the voice
level opens new ways of voice classification, analysis, and transformation.
# 8. Applications

Seeking knowledge through scientific research can be an exciting goal in itself. Yet often the question arises, what all of this is good for. Thus, we conclude this thesis with a discussion of artistic projects that I was involved in during my thesis and that were using the technology presented in the previous chapters. A discussion of the applications is an important part as it gives justification to what we do and can be used to motivated further directions for development. In the case of this thesis much of the work was motivated by the artistic residency of Judith Deschamps who asked us to reinvent the voice of 18th century castrato singer Farinelli.

In general, research at IRCAM is largely motivated by creating tools for musical composition. Therefore, it is a welcome confirmation of this work that the pitch and voice level transformations introduced in the previous chapters were used in some compositions created at IRCAM, notably by the composers Aïda Shirazi (Shirazi, 2022), Sachie Kobayashi (Kobayashi, 2022) and Omer Barash (Barash, 2023).

## 8.1. Farinelli singing voice

IRCAM, as an institution primarily concerned with artistic creation, offers artistic residencies aimed at composers and artists of other domains. Artists in residence can research their artistic ideas as part of one of IRCAM's research groups. This is meant to encourage collaborations between scientific research and artistic creation. Such an artistic residency happened during my thesis between the artist Judith Deschamps and my research group.

One of the reoccurring themes in Deschamps's work is the transience of human and non-human lives and our (futile) attempts to prevent body changes. Since 2018, she started a series of works about the phonatory apparatus, and how History invested this specific part of the body. This led her to discuss the practice of castration in classical music, an attempt to preserve the youth of a boy's voice. In particular the story of 18th century castrato singer Farinelli was a central interest to her. She started her artistic residency with the goal to reinvent the voice of Farinelli using deep neural networks. The voice was supposed to be the product of hybridization between multiple voices covering different parts of Farinelli's singing range. The artistic residency of Judith Deschamps started in January 2021 and yielded a film (Deschamps, 2022b) and a sound installation.
**8.1.1. The castrato voice of Farinelli**

In the musical tradition of the baroque era, castration of young boys was a common
procedure to produce high male voices. The castration of young boys prevents
puberty due to the lack of testosterone hormones. As a consequence the rapid
growth of the vocal folds, that causes the voice break in uncastrated males, never
occurs. The vocal fold length remains in the range of a child's or soprano's as has
been observed by post-mortem examinations (Tandler and Grosz, 1909). While
the larynx of a castrato does not experience a rapid growth in teenage years, the
remaining vocal tract does. As a result the voice of a castrato, although similar
in pitch, was very different from the voice of a soprano, countertenor, or child
(Jenkins, 1998). In particular extensive training and the greatly increased lung
capacity allowed castrati to sing extremely long phrases in one breath and with
great flexibility and virtuosity. For example, the castrato singer Farinelli was said
to be able to 'hold a note for a whole minute without taking breath' (Jenkins, 1998).

Farinelli (1705 – 1782), born as Carlo Maria Michelangelo Nicola Broschi, was one of the most famous castrati and one of the most famous opera singers of all times (Desler, 2014). In 1737, at the age of 32, and at the height of an extremely successful career throughout Europe, he followed a summoning by the Spanish royal house. At the Spanish court, Farinelli was employed as Philip V's private chamber musician. The Spanish king was experiencing severe depression and Farinelli's task was to sing for the king every night to sooth his melancholy. Contemporary accounts mention Farinelli singing the same few songs every night for over ten years, though this extremely repetitive repertoire has been doubted in later literature (Desler, 2014, pp. 142). The ritual between Farinelli and the Spanish king is subject of Deschamps's work on Farinelli. Farinelli remained at the Spanish court after Philip V's death in 1746 and continued to serve for the new King, Ferdinand VI, until his retirement in 1759.

An account on Farinelli's vocal range can be found in Desler (2014, Ch. 3). Farinelli was famous for his large vocal range, one of the distinguishing abilities that set him apart from other singers. At his prime he was able to sing up to D6¹, but his vocal range was likely diminishing in his later career. By the time he was employed at the Spanish court most of his repertoire was in the range B3–F5 and only rarely exceeded F3–B5. This impressively large vocal range allowed Farinelli in his prime to sing anything written for tenor, alto, or soprano comfortably. This large vocal range is one of the key challenges for anyone trying to recreate the voice of Farinelli in its entirety and was influential to our own design choices.

**8.1.2. Related work**

It is not the first time that IRCAM attempted to recreate the voice of Farinelli. Depalle
et al. (1994) merged the voices of a countertenor and a soprano for the soundtrack
of the 1994 film *Farinelli* (Corbiau, 1994). The soundtrack contains arias sung by the

<footnote n="1">1. Concert pitch at the time was about 100 cent to 200 cent lower than today and varied widely throughout different locations, so an exact frequency cannot be given. D6 *today* corresponds to 1174 Hz.</footnote>
fictional Farinelli voice. These arias were recorded by two singers, a countertenor and a soprano, who together covered the whole range of Farinelli's vocal range. To create the impression of a single voice performing the entire repertoire, the female voice was modified through means of voice identity conversion to the voice of the countertenor. The voice identity conversion was achieved by adjusting the amplitudes of the harmonics in the voiced segments to match the recorded statistics of the countertenor. Separate statistics were recorded for each vowel, pitch and three vocal intensities. The adjustment of the partials was achieved by frequency domain filters with separate rectangles around each partial. The filter amplitudes were selected in such a way that the converted voice had the same relative amplitudes between the harmonics as in the target voice reference recordings.

Resurrecting historic voices has many applications in entertainment. When actors die during the time of a production before it can be finished, additional voice recordings of the deceased actors may be required to finish the production and do justice to the actors last work. In documentaries where acoustic material has not been recorded or has been lost, recreating the material artificially can help to give the narrative more realism. In fictional stories it could be desirable that the deceased should make an appearance. If recordings of the target voice exist, voice reenactment (Bous, Benaroya, et al., 2022) can be used to allow an actor to imitate the voice of the deceased. Another possibility is to train a text-to-speech model on the target voice and synthesizing the voice using the script alone. Voice reenactment has been used by the Analysis/Synthesis team of IRCAM, to recreate the voice of 20th century chanson singer Dalida in the television series *Hôtel de Temps* for the French television network *France 3* and to recover a lost recording of Charles de Gaulle's *appel du 18 juin 1940*.

Several pieces from Farinelli's repertoire are still being sung regularly by other voice types, e.g. sopranos or countertenors. Historically informed performances can be seen as an imitation of Farinelli's voice, not with computers but with other voices. The piece *Quell'usignolo* which we reinterpret in the following sections has been recorded on numerous occasions.

### 8.1.3. Musical material

The goal of this project was to reinterpret the piece *Quell'usignolo*, by Geminiano Giacomelli using a synthetic reinvention of the voice of Farinelli. *Quell'usignolo* is one of the pieces that is mentioned frequently among the pieces that were sang every night and might thus have been Philip V favourite piece. It is an aria from the opera *la Merope* by Geminiano Giacomelli, premiered in 1734 with Farinelli in the cast. This piece was used as the basis for our virtual voice of Farinelli.

This piece is of particular interest since handwritten notes by Farinelli himself exist (Broschi, ). 'This piece [...] has been transcribed by Farinelli for a manuscript sent to Maria Theresa of Spain, to demonstrate the quality of his voice. The score includes his [Farinelli's] own ornamentations [...] ' (Deschamps, 2022a, translation mine), and has been influential to Deschamps's artistic work on Farinelli. The manuscript provides insight on how Farinelli might have interpreted the piece,
<table><caption>**Table 8.1.:** Phrases in the piece Quell’usignolo.</caption><thead><tr><th>Phrase</th><th>Translation</th></tr></thead><tbody><tr><td colspan="2"><strong>Exposition (A) and recapitulation (A')</strong></td></tr><tr><td>(Q) Quell’usignolo che innamorato</td><td>The nightingale who fell in love</td></tr><tr><td>(C) Se canta solo</td><td>When it sings alone among the branches</td></tr><tr><td>(F) Fra fronda e fronda</td><td>From leaf to leaf</td></tr><tr><td>(S) Spiega del fato la Crudeltà</td><td>Laments the cruelty of fate</td></tr><tr><td colspan="2"><strong>Trio (B)</strong></td></tr><tr><td>(W) S'ode pietoso nel bosco Ombroso</td><td>Pitifully heard in the dark forest</td></tr><tr><td>(X) Chi gli risponde con lieto core</td><td>Who will answer with a happy heart</td></tr><tr><td>(Y) Di ramo in ramo cantando va</td><td>Singing it goes from branch to branch</td></tr><tr><td>(Z) Con lieto core nel bosco Ombroso cantando va</td><td>With a happy heart it goes singing through the dark forest</td></tr></tbody></table>

<table><caption>**Table 8.2.:** Structure of the piece Quell’usignolo.</caption><thead><tr><th>Exposition (A)</th><th></th><th>Recapitulation (A')</th></tr></thead><tbody><tr><td>Qa Ca Fa Sa</td><td rowspan="4">Trio (B)</td><td>Qa' Ca' Fa' Sa'</td></tr><tr><td>Qb Cb Sb</td><td>Qb' Cb' Sb'</td></tr><tr><td>Qc Cc Fc Sc</td><td>Qc' Cc' Fc' Sc'</td></tr><tr><td>Qd Fd Cd Sd</td><td>Qd' Fd' Cd' Sd'</td></tr></tbody></table>

about his vocal range and his abilities. This score was used as the basis for an arrangement created by the composer António Breitenfeld Sá-Dantas.

The structure of the original piece is a typical (ABA') structure, with a fast exposition (A), a modified recapitulation (A') and a slower trio (B) in the middle. The exposition (A) consist of 15 phrases, each one beginning with a text part using one line from the lyrics, followed by an optional cadenza. The lyrics of the different lines are given in Table 8.1. The lines (Q), (C) and (S) reoccur four times each. Phrase (F) is omitted once and occurs only three times. The text parts of (A') are slight variations of (A) while the cadenzas are almost completely different between (A) and (A') with much more extended cadenzas in section (A'). The trio takes more liberty in its structure. We can roughly divide the trio into four phrases, (W), (X), (Y) and (Z) with some cadenzas interjected within the phrases. An overview of the structure of the piece is given in Table 8.2.

### 8.1.4. Creating a hybrid voice

To create an artistic impression of Farinelli's voice we combine different voices to generate a hybrid voice that has an unusually large range. This is achieved by the pitch transformer from Chapter 6 utilizing its tendency to convert voice identity depending on the transposition and target pitch.
<figure><img src="image_44.png" /><figcaption>**Figure 8.1.:** Screen shot of CIRCE, where pitch transformations are applied to a section of the piece Quell'usignolo. The left panel shows different elementary transformations, the top panel shows the resulting mel-spectrogram and the bottom panel shows the different $f_0$-curves for original, target and transformed. In the bottom left corner we see a list of applied transformations.</figcaption></figure>

Due to the unusually high pitch range, only very few singers would be able to perform the whole score. Thus, we use two different versions of Quell'usignolo, arranged by Sá-Dantas: The *target score* follows closely the manuscript of Farinelli, and is our interpretation of how Farinelli might have sung the piece; this is what we want as the result. The *record score* is a simplified version of the *target score*. It follows the same rhythm as the *target score* but has a reduced vocal range, obtained by transposing the higher sections down and the lower sections up. Thus, the *record score*, lies in the alto range and can be sung by any trained singer of that range. The Farinelli voice is created by recording an alto singing the *record score*, and transposing the recording of the *record score* to match the notes of the *target score* using the neural voice transformation framework. The *target score* is in C major while the *record score* is in G major.

To create the correct timings for the transformations and apply the transformation to the recording we created a custom software, called CIRCE, for manipulating the mel-spectrogram interactively. Figure 8.1 shows a screenshot of CIRCE applying the pitch transformations on a section of the piece.

The pitch transformer from Chapter 6 has a tendency to change the voice identity
<figure><img src="image_45.png" /><figcaption>**Figure 8.2:** The installation *another voice* by Judith Deschamps, where the Farinelli voice was exhibited. A mysterious object in the wall invites climbing on the ramp. At the top of the ramp a motion sensor is activated, and the Farinelli Voice can be heard through the use of directional speakers. [© Lynn Theisen]</figcaption></figure>

with increasing amount of transposition. This is necessary to create realistic sounding voice even for pitches that are impossible to sing for the source voice. Thus, with increasing amount of transposition we can observe a gradual transition from one voice identity to another. The target voice can be influenced to some degree by carefully choosing the training dataset. If for a specific pitch range only one voice was recorded, the auto-encoder may overfit to this voice on this particular pitch range. Consequently, any sample synthesized in this pitch range will strongly resemble the voice from the training set. Conversely, this effect is mitigated if more voices cover the target pitch range. If there are sufficiently many voices representing the target pitch, the auto-encoder can approximate the source voice more closely in the target range and create a smooth gradual transition.

To reinvent Farinelli’s voice we created a custom dataset with seven voices covering the alleged vocal range of Farinelli: two pre voice-break child voices, one soprano, two altos, one countertenor and one tenor. All singers sang the parts of the *target score* that were within their range, except for the altos who were recorded performing the whole *record score*. Some singers also provided additional material from their own repertoire.

### 8.1.5. Improvisations over Quell’usignolo

In our original approach, the piece was produced using the arrangement by Sá-Dantas as described in the previous section (the ‘*target score*’). Parts of this reinterpretation have been used in the Film *La Mue* (Deschamps, 2022b). For artistic reasons we later decided to create random variations of the score to allow the piece to be different every time it is performed. These random variations were exhibited as part of the show *another voice* at Casino Luxembourg between 17th of February and 14th of April 2023. A photograph of the installation is depicted in Fig. 8.2. In this installation the voice was improvising new versions of *Quell’usignolo* using rule-based rearrangements of sections of the recording. The recording was split into small
fragments, consisting of only a few notes each, that captured elementary melodic
building blocks and individual ornamentations.

The structure of the piece, in particular of the exposition (A) and the recapitulation
(A') allow us to reshuffle the structure on multiple levels.

1. The text-parts of each phrase can be subdivided into smaller fragments. The fragments from (A) and (A') can be used interchangeably to create a new version of the text-part.

2. The cadenzas can be split into smaller fragments. For each fragment we define a set of allowed successor fragments. Possible successor are classified into fragments that cause smooth transitions, rough transitions or end the cadenza. Depending on whether we want a smooth transition, a rough transition or exit the cadenza we pick the next fragment from the appropriate class randomly.

3. The different versions of each phrase can be used interchangeably. At the end of each phrase either a random version of a different phrase or the trio may follow. Between two phrases there is a short rest of a few seconds.

This process constitutes a Markov chain over the set of all fragments. Formally we can define a transition matrix that defines the probabilities of going from one fragment to the next. This transition matrix defines the piece and choosing the non-zero elements is the central part of the recomposition, which was done by Sá-Dantas. To obtain simple controls for the value of the non-zero elements we parametrize the matrix by three elementary probabilities. We define probabilities for smooth transitions $p_s$, rough transitions $p_r$ and cadenza exits $p_e$ and divide the probability by the number of fragments in each transition class.

To further increase the variability of the improvisation we allow key changes to occur:

* Randomly the key can change by going up or down by a semitone. If a maximum or minimum key is exceeded we change the key by a fourth in the opposite direction instead. There is always a larger probability to go back towards the original key rather than to go away from it.

* If a fragment is repeated within the last five fragments the key automatically goes up by a semitone.

Since the key changes depend on the last five fragments, the key changes do not form a Markov chain. We use 11 keys in total, from E to D which further extends the originally-envisioned *target score* by a second up and a minor sixth down.

The improvisation is run on a conventional computer which is part of the installation. The computer can be seen in Fig. 8.3. Since the number of variations is limited, we pre-compute all the required fragments for all the possible keys. Then, during improvisation the program cross-fades between the different fragments. Thus, the neural transformations do not have to be run on the installation hardware which allows for a light-weight setup. During the installation a screen provides information about the state of the improviser through a logging output and a small screen.
<figure><img src="image_46.png" /><figcaption>**Figure 8.3:** *The computer running the improvisation of the Farinelli Voice is part of the installation. A screen shows the state of the improviser and the transition probabilities. Behind the screen the hardware is shown to the visitor in an open case.*

[© Lynn Theisen]</figcaption></figure>

### 8.1.6. Observations and difficulties

For the proposed approach to work, the *record score* could have theoretically used any arbitrary notes as long as the rhythm matched the *target score*. However, to allow the singer who interpreted the *record score* to sing with a similar overall expression as would be required for singing the *target score*, the melody in the *record score* was mostly the same as the *target score* but in a different key. Most of the time, lower parts of the melody were transposed up by a fifth while higher parts were transposed down by a fourth to create the simplified *record score*. This also made the process of creating the transformation of the recording easier, as large sections could be selected at the same time rather than individual notes.

When we applied the transformation to the recording, however, we noticed that the tenor voice would be very prominent in all sections that were transposed down. This effect occurred regardless of whether the result was in a range exclusively covered by the tenor or not. Especially the transitions were noticeable and sometimes created the impression of another voice taking over. Further investigation revealed the following observation: Suppose we use two notes from the same singer on the same vowel, one high note and one low note. We transform both notes to the pitch of the low note. We had expected that both notes (the note that was originally high and the note that was low all along) would sound similar. This is, however, not the case. Singing a high note requires a different singing technique than singing a low note. The auto-encoder preserves the singing technique as part of the latent code. The transposed note cannot be sung by the source singer using the original singing technique. Instead of adjusting the singing technique, the auto-encoder adjusts the voice identity. This is the same principle as the previously-discussed adjustment of the voice identity in the case where the new pitch cannot be sung by the source singer at all and could have been anticipated. In the case of the Farinelli
Voice, the transposition amount usually changes from +500 cent to -700 cent, thus by a whole octave. This big change of transposition creates an audible change in voice quality even if the absolute pitches are closely together. (Usually the change in transposition occurs, when the melody in the *target score* moves in the opposite direction, such that the melody in the *record score* does not have a large jump.) In the context of the installation this is an interesting artefact of the voice and allowed the artist to showcase the weirdness and imperfections of our method. From the scientific point of view, however, this suggests that more precise control of either voice identity or singing technique is necessary to remove the ambiguity created from providing a target pitch alone.

Another difficulty we observed was concerning extremely high pitches. Using the set-up from Chapter 6 we could never transform the pitch to anything higher than F5 (700 Hz) even if samples containing these pitches were drawn with more than 50% probability. Experimentation suggested that the auto-encoder treated these high pitches not as voiced speech but rather as a separate phoneme with the positions of the partials encoded in the latent code (possibly confusing partials with formants): Transforming from fundamental frequencies below 700 Hz to above 700 Hz saturates at 700 Hz and transforming fundamental frequencies above 700 Hz does not change the result in any systematic way. This suggests that the bottleneck size was not narrow enough for the higher pitches and thus different bottleneck sizes are required for different input pitches.

To solve this problem we implemented an adaptive bottleneck size. During training, we apply dropout to the latent code. The dropout probability is calculated for each time step separately depending on the input fundamental frequency. The dropout probability increases linearly from 0 to 1 starting at 0 for the minimal pitch A2 (45 Hz) until complete erasure at the maximal pitch F6 (1400 Hz). No dropout is applied for unvoiced segments. During inference no dropout is applied. With the adaptive bottleneck even the high-pitched sounds could be disentangled from their fundamental frequency. The adaptive bottleneck allowed us to increase the bottleneck size from 3 dimensions to 8 dimensions. Thus, as a welcome side effect, the larger bottleneck transmits more information through the auto-encoder, in particular for the unvoiced segments, which also increased the audio quality significantly.

### 8.1.7. Conclusion

We created an artistic impression of the voice of Farinelli, by merging different voices covering the large vocal range of Farinelli. Previous work by Depalle et al. (1994) created the final recording by patching together the recordings of two singers performing different sections and a voice identity transformation on one of the voices to match the timbre of the other. In this work the final material was obtained by recording only one singer. This recording was then transformed using the neural pitch transformation from this thesis to increase the vocal range of the singer. While the input voice for the transformations was only provided by one singer, the neural transformation itself was trained on different singers covering the whole range
of Farinelli. The voice was used in a sound installation where the exhibited piece was varied by randomly rearranging the sections and cadenzas. The cadenzas were improvised by rearranging fragments from the cadenzas. Key changes were used to reduce repetitions.

## 8.2. Musical compositions with CIRCE

To implement the transformations required for the Farinelli project, the computer program CIRCE was developed. Though originally only envisioned for a single purpose, the program is itself useful for other artistic creation as well. CIRCE has been available for download on the IRCAM Forum website<fnref n="2" /> for free since autumn 2022 and receives updates with new features and new models every few months. Three noteworthy musical compositions have made use of the neural voice transformation framework, developed in this thesis, through CIRCE. The collaborations around these pieces have helped develop the software by uncovering the problems and requirements in a real application.

### 8.2.1. Aïda Shirazi: *Né entre corps* (2022)

Additional media: http://thesis.fnab.xyz/ch8/shirazi

<figure><img src="image_47.png" /></figure>

The composition *Né entre corps* by the composer Aïda Shirazi is a piece for electronics and dance. The dancer is hidden behind a screen and only their shadow is projected onto the screen. Using different light sources, the shadow can be formed in a variety of ways. (See Fig. 8.4 for a visual example.)

The music is a setting of a poem by Haleh Ghassemi. The poem is read in two different versions, the Persian original and a French adaptation by Irène Gayraud. The voices of the poet and the translator were recorded reading their respective versions of the poem with additional voice recordings of the composer herself. The voice recordings were treated using spectral filtering and reverberation as well as the neural voice transformation framework from this thesis. The accompaniment is composed of analogue synthesizers as well as samples of the sounds from silverware, pots, pans, glass, pepper and salt grinders and knives cutting on glass. Droning granular sounds evoke the general atmosphere of the texts. Additionally, to the fixed media, the breathing noise of the dancer is picked up by a headset microphone and mixed with the other electronics.

The visual aspect of the piece shows a female dancer behind a screen. Thus, only the shadow of the dancer is visible. This effect is used to remove the identity of the performer: to create a neutral character of a woman, reduced to only its 'timeless features, free from stereotypes and expectations of the physical appearance of a ballerina.'<fnref n="3" /> The projected image slowly changes from the abstract to the concrete. Initially we see only abstract shapes that are not recognized as a human body, while later a female body becomes recognizable. The projected shadow obscures the female

<footnote n="2">2. https://forum.ircam.fr/projects/detail/circe/</footnote>

<footnote n="3">3. This and the other quotes in this section are from my personal correspondence (PC) with the composer.</footnote>
<figure><img src="image_48.png" /><figcaption>**Figure 8.4:** Shadow play from Né entre corps by Aïda Shirazi. The body of a female dancer is projected on a screen to produce an abstract representation of the body. This is mirrored in the voice using the neural voice transformation framework. [© Aïda Shirazi]</figcaption></figure>

body and plays with ‘the visual representation of the body’ (Shirazi, PC). It questions ‘the preconceived notion of the body of the female dancer’ by ‘deconstructing and expanding and abstracting the image of human and female body’ (Shirazi, PC). A shadow projection from the piece is shown in Fig. 8.4.

The voice parallels the philosophy of the shadow play. Like the visual appearance of the human, the acoustic appearance is masked and distorted through unusual reading and audio effects. Similar to the staged movements of dance, the composer created ‘distortion of the words through different means of utterance’ [during the recording], such as stuttering and other exaggerated moods of expression’ (Shirazi, PC). The voice recordings were further distorted by filtering, reverberation, the neural voice transformation framework and other audio effects. The neural voice transformation framework allowed Shirazi to play in the area between whispering and clear speech. To her, the whisper is the acoustic counterpart of the shadow, creating a mystification of the voice that hides otherwise obvious aspects of the voice. Using the voice-unvoiced mask in the control signal creates a ‘nuanced treatment between [whisper and clear speech]’ (Shirazi, PC).

### 8.2.2. Sachie Kobayashi:

**Day 0 - trans-instrumentalism (2022)**

Additional media: http://thesis.fnab.xyz/ch8/kobayashi

<figure><img src="image_49.png" /></figure>

The composition *Day 0 - trans-instrumentalism* by the composer Sachie Kobayashi is a multimedia piece for fixed media and live electronics. It is a theatrical piece for one performer without dialogue. The piece tells the story of a person waking up one day to find themselves turned into a musical instrument, much like in Kafka’s short story *Die Verwandlung* (Metamorphosis). Throughout the piece the character descends further and further into a dream world that could have been the result of imagination or hallucination.

The fixed media part of the piece is composed with synthesizers, field recordings,
<figure><img src="image_50.png" /><figcaption>**Figure 8.5:** *Scene from Day 0 – trans-instrumentalism by Sachie Kobayashi. The character in the piece discovers that they have turned into a musical instrument and their everyday gestures, like scratching, produce music. [© Hervé Véronèse – Centre Pompidou]*</figcaption></figure>

recordings of previous works of the composer, and audio samples from daily objects and speech. The speech samples are generated by text-to-speech synthesis and from recording the composers and my own voice. The voice plays the role of an announcer on the radio, providing the listener with background information about the topic of the piece by reading excerpts from the Wikipedia article on transhumanism. The voice itself is accompanying the progress into the dream-like state of the meta-morphosis. Initially a neutral-emotionless voice from text-to-speech is presenting the material. Later the strongly-transformed voices of real speech replace the monotonous announcer. The strong voice transformations are implemented with the neural voice transformation framework using CIRCE. The extremeness is achieved by amplifying the pitch variations to range over multiple octaves passing through different voice registers.

The neural transformation introduced an intentional weirdness that was new for the composer. Even within extreme transformation the neural voice transformation framework preserves its acoustic naturalness. This is being opposed by the unnatural control curve, the extreme pitch variations, that are used to drive the bottleneck auto-encoder. This contradiction between natural sound and unnatural behaviour allowed the composer to enter the region of *uncanny valley* and use it as a narrative device in her piece, which made the neural transformation an attractive tool for her composition.

During this process, the voice identity is obfuscated due to the extreme pitch transformations which move the pitch out of the respective voices range. Kobayashi used the pitch transformations to increase the $f_0$ range of the speech melody. As a result, the pitch went through different ranges and created intergender voices. This allowed the composer to use her own voice during the composition process to input the voice's expressivity in the most natural way (simply by speaking the way she wants) without inserting herself into the piece explicitly.
<figure><img src="image_51.png" /></figure>

**Figure 8.6:** One of the anonymous texts used for the piece G.N.Z. by Omer Barash. Half of the page is missing such that we only know the second half of each line of the poem. (Cambridge University Library T-S 13J10.1) [© Syn-dics of Cambridge University Library]

### 8.2.3. Omer Barash: G.N.Z. (2023)

Additional media: http://thesis.fnab.xyz/ch8/barash

The composition G.N.Z. by Omer Barash is a piece for tenor and electronics. It is a musical setting of two anonymous texts from a collection of medieval fragments found in the Cairo Genizah of the Ben Ezra Synagogue. The title G.N.Z. refers to the Hebrew root גנז (gnz) meaning to hide, to archive or to store. The name of the Cairo Genizah itself derives from the root גנז. The main theme of the piece is the exploration of ambiguity and betweenness in particular with respect to the question of identity.

Two texts from the Genizah collection form the two parts of the piece. The first text is a poem of which ‘only half of each line has survived’ (Barash, PC). This poem is depicted in Fig. 8.6. The missing half of the poem gave the composer the liberty to fill the gaps as part of the interpretation and plays on the theme of concealment. The music reflects the fragmented nature of the text and fills the missing parts with a heterophonic choir. The second text is a love song which ‘represents a gay experience, not asimilar to a contemporary one’ (Barash, PC), addressing relatable questions such as unachieved love and makes reference to party and drinking. Similarly, the musical setting makes reference to modern rave music, using elements like periodic kicks and the ‘vocoder effect’.<fnref n="4" /> Alluding to the holes in the paper, the first poem occasionally shines through during the second poem.

The electronic accompaniment is mostly based on samples of the singer, as well as live effects applied to the voice during the performance. The effects used in the piece are spectral freeze using a phase vocoder, chorus, the ‘vocoder effect’ and the voice transformations introduced in this thesis. Voice transformation is used in this piece in two complementary ways, applied at different times in the piece: First, the electronics are used to create an accompaniment which blends in with the singer and creates a response growing out of the original voice. Second, the voice itself is

<figure><img src="image_52.png" /></figure>

<footnote n="4">4. That is, the effect widely used in pop music of the 1970 and '80s, which has become a stereotype of a 'robot voice', thanks to songs like *The Robots* (1978) by *Kraftwerk*.</footnote>
transformed in real time to make it blend in with its surrounding.

The piece plays with the question of ambiguity and betweenness of identity on several levels. First, the themes of the texts themselves evoke questions about identity, which the composer addresses in his work. Second, the anonymity of the authors leaves room for speculation about the identity of the authors based on content, orthography, and calligraphy of the poems. Barash uses the implicit voice identity transformation of *CIRCE* to address the question of identity musically. The composer used the pitch transformation to generate new audio samples of the singers voice by imposing new melodies on recordings of long sustained notes. This process allowed Barash to command over a set of derived voices with slightly different identities to be used in ensemble with the original voice. Here, the voice transformation of the neural voice transformation framework was used 'not so much as an audio effect but rather as a generator' (Barash, PC). The generated samples act as a new source of singing, which can then be used as an input to the more invasive effects.
# 9. Conclusion

My initial research began with the goal of finding deep learning methods that allow us to transform voice properties in interesting ways. The contemplations around this topic have yielded a neural voice transformation framework with a neural vocoder and a bottleneck auto-encoder. This neural voice transformation framework allows changing fundamental frequency and voice level and could easily be adapted to other voice properties as well.

## 9.1. Recapitulation

After an introduction to the material in Chapter 2 we discussed the fundamental approaches of voice transformation in Chapter 3. In particular, we have introduced the vocoder as a tool to represent voice and how it is used in voice synthesis and transformation. This has led us to more in depth discussion of the vocoder in Chapters 4 and 5, which led to the development of an efficient universal neural vocoder as described in Chapter 5. This vocoder was then used in the remaining thesis to provide a practical representation of the voice. In Chapters 6 and 7 we have discussed the bottleneck auto-encoder and applied it to fundamental frequency (Chapter 6) and voice level (Chapter 7). Finally, we have seen some applications of the voice transformation in Chapter 8.

### 9.1.1. Neural vocoders

In Chapter 4 we have seen a neural vocoder that was based on the classical source filter model. The vocoder of Chapter 4 used a non-parametric glottal pulse signal to define the voiced part of the voice. The glottal pulse signal was estimated from the audio by another neural network. This analysis-synthesis framework allowed jointly training signal analysis and synthesis end-to-end. The vocoder improved state of the art GCI estimation. However, the approach came with considerable drawbacks: While using the glottal pulse signal instead of the fundamental frequency allows modelling irregular phonations more precisely, due to its non-parametric nature, the glottal pulse signal is difficult to transform and thus not well suited for voice transformation. Furthermore, the synthesis quality was still rather poor. Therefore, this approach was ultimately dropped for another parametric representation, the mel-spectrogram.

While Chapter 4 served as a warm-up, in Chapter 5 we developed more systematically a vocoder for voice transformations. We have started the chapter by compiling a list of requirements to a vocoder designed for voice transformations. Then existing approaches were discussed with regard to these requirements. This allowed us to