argue, why in the context of neural voice transformations, the mel-spectrogram was a good representation of the voice. In the remainder of Chapter 5 we discussed the Multi-Band Excited WaveNet, a neural vocoder based on the mel-spectrogram that was a result of the previous contemplations, though the actual development of the Multi-Band Excited WaveNet was not my own contribution.

### 9.1.2. Bottleneck auto-encoder

With the neural vocoder providing a practical representation of the voice through the mel-spectrogram, we were able to introduce neural voice transformation framework in Chapter 6. Transformations on the mel-spectrogram were achieved by a bottleneck auto-encoder, which was trained to disentangle the fundamental frequency from the mel-spectrogram. Disentanglement is achieved by an information bottleneck, that forces the auto-encoder to remove information from its latent space. By providing the fundamental frequency to the decoder, the information about the fundamental frequency becomes redundant and is removed. The bottleneck size was studied thoroughly in Chapter 6. We finished the chapter by an analysis of the latent space where we found that information was spread through time through oscillations.

The bottleneck auto-encoder was then applied to the voice level in Chapter 7. With the voice level we have faced the problem that the voice level is not widely available as annotation and no reliable estimation method existed. Thus, in Chapter 7 we have developed an estimation method of perceived voice level that did not rely on explicit voice level annotations. This method assumes that variations in signal loudness are exclusively due to variations in voice level. By matching the loudness contour without regard of scale from the signal timbre alone, the voice level could be successfully estimated. The estimated perceived voice level was used to manipulate the voice level in singing recordings.

## 9.2. The neural voice transformation framework

With all the pieces coming together, we can now summarize the full neural voice transformation framework as depicted in Fig. 9.1:

1. First the mel-spectrogram *M* is calculated on the input audio *a*.

2. Then we estimate the feature *F* that we want to transform from the mel-spectrogram *M*.

3. The mel-spectrogram *M* and the estimated feature *F* are used by the encoder to create the latent code *c*.

4. We define the new value *F'* for the feature we want to transform either by deriving it from the original value *F* or by inventing a completely new curve.

5. The latent code *c* and the transformed feature *F'* are used by the decoder to resynthesize the mel-spectrogram *M*'.
<figure><img src="image_53.png" /><figcaption>Figure 9.1.: Full framework during inference.</figcaption></figure>

6. The transformed mel-spectrogram $M'$ can be further transformed by another auto-encoder or converted to raw audio by the mel-inverter MBExWN.

The neural voice transformation framework consists of two principal components

**Auto-encoder** The auto-encoder disentangles the feature $F$ from the mel-spectrogram $M$. The resulting latent code $c$ is independent of the feature $F$. Resynthesizing the mel-spectrogram using the latent code $c$ and the transformed feature $F'$ yields a transformed mel-spectrogram $M'$. The auto-encoder was discussed in Chapters 6 and 7.

**Mel-inverter** The mel-inverter converts the transformed mel-spectrogram $M'$ back to raw audio. In Chapter 5 an implementation of a mel-inverter was introduced, though any other mel-inverter can be used, provided it was sufficiently trained on the voice type that is being transformed.

Additionally, we may require auxiliary neural networks to extract the feature we want to transform from the mel-spectrogram

**Feature estimator** During training and inference, the auto-encoder needs the correct feature value, to work properly. We use a neural network to estimate the feature $F$ from the same mel-spectrogram $M$ that is being fed into the auto-encoder. This ensures that the feature $F$ does not contain additional information that is not present in the mel-spectrogram $M$. In Chapter 7 we have discussed extensively how to estimate the voice level from the mel-spectrogram.

## 9.3. Applications

In Chapter 8 we have seen several artistic projects where the neural voice transformation framework has been used. They showed different examples of how the neural voice transformation framework could be used in an artistic application. Generally, there are different categories of applications for the neural voice transformation framework that we will discuss in this section.
### 9.3.1. Audio post-production

After a voice recording is made, for different reasons it may be desired that the original recording would be changed. If the singer of the original recording is not available or unable to make a new recording the existing recording may be altered using the tools developed in this thesis. For example some notes might not have been sung correctly, or we may want to generate alterations of a phrase.

### 9.3.2. Creation of new, hybrid-voices

Similar to the above, but with slightly different implications, we may want to extend the range of a singer. If for example a score requires certain notes that are beyond the singer's range, the whole recording might be transposed to map from the original voice to the target range. In this case we benefit from the implicit voice identity conversion that adjusts the voice identity characteristic in a way that allow the higher notes. We will discuss a bit more, the possibilities of controlling the new voice identity in Section 9.4.2 (headline Voice identity).

Furthermore, the voice transformation does not only allow overcoming limitations of a single voice but of what human voice can do in general. In the Farinelli project (Section 8.1) we have created a voice with an unnaturally large ambitus which singers normally do not have. Similarly, Kobayashi (2022) altered speech recordings by using extreme pitch variations, that pass through pitch registers of multiple voices to create a natural sounding voice with very unnatural behaviour. A human might attempt to imitate the pitch contour, but the voice will betray an extreme exertion. With the auto-encoder the most absurd pitch contours can be followed with ease, creating a super-human voice that is not restrained by the same limits as real speech.

### 9.3.3. Obfuscation and anonymization

In the context of speech, the voice transformation can be used to obfuscate or anonymize the original speaker. This could be used if only one speaker is available, but multiple personalities are required. Another application could be to obfuscate specific aspects of speaker identity, in particular gender by moving the average pitch into a region atypical for that gender. While this does not aim to be a 'gender transformation', which is a separate research domain, it allows questioning certain pre-conceptions about voice and identity. This could be used in social or psychological studies to investigate the effect of different voice parameters on the perception of identity.

> It should be noted that the anonymization is not suited to remove all information related to voice identity. It should, therefore, not be used in an application where anonymity is crucial and identification of the original speaker could be consequential.

### 9.3.4. Singing voice synthesis

The pitch transformations can be applied in parametric concatenative singing synthesis (Section 3.3.1) in particular the IRCAM Singing Synthesizer. In concatenative
synthesis the phoneme sequence is generated by concatenating samples of diphones (phoneme pairs) from a recorded database. Rhythm and melody are imposed by time-stretch and pitch-shifting. Currently, the IRCAM Singing Synthesizer uses the PaN vocoder of Ardaillon (2017), a parametric vocoder. Using a neural neural voice transformation framework instead of PaN would increase naturalness especially for the more extreme pitches: the diphone samples are all recorded on only one pitch. For target pitches further away from the original pitch the result can sound unnatural. Currently, the temporal periodicity in the latent code, as observed in Section 6.3.3, prevents time-stretching to be performed on the latent code, because resampling the latent code in time would destroy the characteristic gestures. Two possibilities could be employed: The time-stretching could occur on the mel-spectrogram. Alternatively, the auto-encoder architecture could be modified such that the decoder only sees one frame at a time. In this case the auto-encoder cannot spread information through the time axis and interpolation can be done on the latent code.

Furthermore, the voice level transformation could allow for increased control over the expressiveness of the result. Currently, only the original voice level from the diphone samples can be produced by the IRCAM Singing Synthesizer resulting in everything being sung in *mezzo forte*. While some expressiveness can be controlled using the signal gain alone, the possibilities in the current implementation are limited. Using the neural vocoder with a voice level control would extend the possibilities for expressiveness significantly.

### 9.3.5. Speech enhancement and humanization

Due to the high compression of the auto-encoder, information unrelated to voice is removed. If the auto-encoder has only been trained on clean speech, it is only able to synthesize clean speech. Thus, if during inference, background noise is present, it is suppressed by the auto-encoder. This works to a certain degree but fails completely if the background noise is too strong.

Similarly, if we provide something that isn't voice as the input to the auto-encoder, the auto-encoder has no means to encode it as something else and will conflate it with a voice recording. The result will resemble human voice much more than the input. Though this does not work consistently and reliably the auto-encoder has been observed to transform breakbeats into beatboxing and string sounds into humming.

## 9.4. Limitations and Future work

Possibilities for future work were already discussed in Sections 4.5, 5.5, 6.5 and 7.6 for the specific applications. In the greater scheme, there are a few topics that need to be addressed.

* The disentangled space should ensure that for reasonable input values the transformed result should still sound natural. In many cases, however, we
find that this is not the case. Improvement in audio quality is still necessary.

* While the neural voice transformation framework could be applied to all kinds of voice properties, in this thesis we have only discussed two properties, the fundamental frequency and the voice level. It would be desirable to add more transformable features.

* Although both auto-encoder and mel-inverter are lightweight enough to calculate the audio fast, real-time applications are limited by the non-causal nature of the system. To allow real-time applications with low latency, the architectures have to be adapted not to depend on future values.

* Finally, it would be interesting to understand the latent code better.

### 9.4.1. Issues with synthesis quality

#### Variable information and bottleneck size

Our experiments have indicated that conditional entropy of the mel-spectrogram under the control parameter *F*, is not constant. In Section 6.3.1 we have observed that the required bottleneck size is quite different for singing voice (2-3) and for speech (5-8). Training a model that works well for both speech and singing was thus not feasible. Similarly, in Section 8.1.6 we have observed that the voiced component contains much less information for high pitches than low pitches. This could be explained by the spacing and number of harmonics as in effect the harmonics in the voice sample the vocal tract filter. For high-pitched voice the vocal tract filter is sampled only at few points and might miss significant information like narrow banded formants. This argument only holds for the deterministic component, while the inverse is true for the noise component: The further the harmonics are spaced, the more area is occupied by the noise component between the harmonics. In singing the latter information is far less relevant as, firstly, the most of the information of the noise is perceptually irrelevant and, secondly, the noise component itself is perceptually less relevant as it carries much less energy. In practice this means that for high enough fundamental frequency the information bottleneck is too large to remove the fundamental frequency from the latent code and the decoder will disregard the *f*₀ input. To force disentanglement for all possible *f*₀ values, the idea of an adaptive bottleneck size was touched upon in Section 8.1.6. The idea of a variable bottleneck size showed promising results in the case of disentangling *f*₀ in the presence of a wide range of *f*₀ values.

The bottleneck size can be varied using dropout on the latent code. By applying dropout with rate *r* ∈ [0, 1] to the latent code, the channel capacity of the residual code is multiplied by 1-*r*. The effective bottleneck size can thus be controlled by the dropout rate. By increasing the dropout rate with the *f*₀, we were able to achieve much better disentanglement. In preliminary experiments, disentanglement was achieved for the whole pitch range of human voice, including high pitches.

A rigorous evaluation of this method, however, remains due. In particular the optimal bottleneck size needs to be found depending on the fundamental frequency,
voice type (i.e. singing vs. speech) and other voice properties.

## Fundamental problems with the bottleneck approach

Even if the controllable property is disentangled properly, sometimes we observe substantial audio degradation. The degradation of the audio can be explained by a fundamental flaw of the information bottleneck approach: In theory the encoder should remove all the information related to the voice feature that we want to disentangle first, before it removes other information that is not redundant. This could be the case if the reconstruction loss is zero and the bottleneck size is exactly at the optimal size. In practice, however, since the information amount is variable, the optimal bottleneck size cannot be used, even with the aforementioned variable bottleneck size. Furthermore, we observe that in practice, essential information is being removed, before the latent space is completely disentangled. Thus perfect reconstruction of the original mel-spectrogram is not possible and the audio quality suffers especially after transformation.

The degradation of the audio quality manifests itself through two main symptoms:

* The phonetic content is often degraded. Some vowels are changed towards more common ones and some consonants, in particular plosives are dropped.

* The noise component is over-smoothed rather than having the characteristic texture of noise. As a result the noise component sounds rather poor, especially for voice with high breathiness.

## Phonetic content degradation

In many cases we can observe degradation of the phonetic content. The vowels all become more similar to each other, consonants might be switched, and plosives might be dropped.

During training the network learns to reconstruct the mel-spectrogram, and it can be expected that phonetic content is preserved when the reconstruction error is sufficiently low. On the other hand, the network is not trained to apply transformations explicitly. Since the framework has no mechanism to preserve phonetic content, there is no guarantee that the latent code represents the same phonetic content for different values of the control parameter. The network might choose different coding strategies for different values of the control parameter, as we have observed with pitches above 700 Hz. As a result, changing the control parameter will change the context of the encoded phoneme and thus change its meaning.

Adding additional constraints might be necessary to ensure stability of phonetic content. This could be achieved by forcing consistency through a phoneme recognizer between original and transformed mel-spectrogram as proposed by Moine (2023, Section 7.2.2).
## Noise component

Using the variable bottleneck size, we can improve parts of the noise component by choosing a high dimensionality of the latent space and fully opening the bottleneck during unvoiced segments. Since no disentanglement of the fundamental frequency is necessary during unvoiced segments no bottleneck is required there. This way the unvoiced component can be reconstructed much more easily. The noise component within voiced segments is not affected by this approach. This approach works well for singing voice where the noisy component in the voiced segments is low and even for speech this approach provides an improvement to the overall quality. Particularly breathy voice still finds its limitation with this approach.

Thus, more sophisticated methods are required to force the decoder to produce the right noise texture. Noise is problematic in terms of information theory, as the amount of information is high, but most of the information is perceptually irrelevant. To reconstruct the noisy texture perfectly, many perceptually irrelevant details need to be preserved. When trying to reconstruct the noise component in the mel-spectrogram with insufficient information, the decoder cannot reproduce the noise component. When using any $L^p$ loss, the decoder can only approximate these areas with mean values. Over-smoothing is the result and no noisy texture is generated. The MBExWN, however, which is used to calculate audio signal from the mel-spectrogram, has never seen smooth textures like that which results in unpredictable behaviour. Two possible remedies could be followed. Either we force the decoder not to generate smooth surfaces but rather noisy textures, or we teach the mel-inverter to treat the smooth surfaces the same way as noisy textures. The question is thus, who should handle generating the noise, the decoder of the auto-encoder or the mel-inverter.

## Adversarial training

Over-smoothing of the noisy textures is a problem of the loss function since generating arbitrary textures would create a higher loss than creating mean values. In this sense the $L^p$ loss does not model the perception well, as noisy textures would be preferred over smooth textures. This could be fixed by using adversarial approaches instead. A typical approach of this would be to train a discriminator to distinguish between real- and synthetic mel-spectrograms. Since smooth textures are rare in real mel-spectrograms this could be a distinguishing feature for the discriminator and the decoder would be incentivized to avoid creating smooth surfaces (Bous, Benaroya, et al., 2022).

## Smoothed mel-spectrogram

For noise signals the perceptually relevant detail is the spectral density, that is, the expected value of the energy in a frequency band. The spectral density is mostly a smooth function with little information. Thus, one might argue that the spectral density might be a more suited parametric space for the neural vocoder as it removes
a lot of perceptually irrelevant information. In this context, the over-smoothing
might actually not be such a big problem but rather a welcome side effect.

The big obstacle for this approach is to find the canonic representation. Since
the noise component changes quickly, it is impossible to estimate the true spectral
density. Simply smoothing the mel-spectrogram is not an option either since narrow
resonances in the spectral envelope caused by the formants must be preserved. We
only want to smooth the noise component. Therefore, an alternative method to
approximate the spectral density is required. A possible approach could be to use a
bottleneck auto-encoder without any conditional input. A large advantage of this
approach in the context of the proposed neural voice transformation framework
would be that we have an upper bound to the total information in the voice repres-
entation through the bottleneck size. The auto-encoder that generates the vocoder
parameters could be trained jointly with the mel-inverter, further optimizing the
parametric space.

Fine-tuning one model to the other

Another similar approach could be to train the mel-inverter not on the real mel-
spectrograms but rather on the synthetic, non-transformed mel-spectrograms. This
way we could teach the mel-inversion to invent the missing information and correct
possible errors. The biggest drawback of this approach is the loss of universality of
the vocoder. For each transformation auto-encoder, a new mel-inverter would have
to be trained. Thus, training the mel-inverter on the synthetic mel-spectrograms is
impractical. More feasibly, the mel-inverter could be used to provide a loss to the
auto-encoder. In this case, the auto-encoder and the mel-inverter would be used
simultaneously when training the auto-encoder, but the weights of the mel-inverter
would be frozen. This could force the auto-encoder to produce more realistic mel-
spectrograms.

Hybrid information bottleneck

The above ideas are remedies that address the one or the other symptom of the too
large information reduction through the information bottleneck. None of these
approaches, however, address the underlying issue with the information bottleneck,
which is that the information bottleneck inevitably removes more information than
it should.

This does not mean, however, that the proposed method of this thesis should not
be used at all. The simplicity and the quick disentanglement make the bottleneck
auto encoder a compelling and powerful tool. While the information bottleneck
itself has its limitation, it can be combined with other disentanglement methods
to benefit from multiple approaches. One alternative approach, for example, is
the use of GAN. Using a discriminator, we could evaluate transformed voice during
training. The problem with this approach is, that convergence is tricky. Using the
variable bottleneck size the information bottleneck could be opened and closed
whenever necessary. Thus, the information bottleneck could be used to quickly
achieve disentanglement. Then, the bottleneck could be slowly opened to allow more
information to be exchanged between encoder and decoder, while a discriminator
ensures that the latent space remains disentangled. These approaches could be
further combined with content classifiers that ensure that the relevant content is
well-preserved.

9.4.2. Controllable voice properties

So far only two voice properties can be controlled, the fundamental frequency and
the voice level. We want to be able to add transform more properties, but as with
the voice level, many of these parameters are not easily available.

Breathiness

Breathiness is the ratio between sinusoidal voiced and noisy unvoiced component. Normally, while still an essential part of the voice, the noisy component is very low in voiced segments. However, due to various reasons, the noise component can become much more dominant, for example when speaking very quietly (in the extreme case for whispering). Thus, breathiness is highly correlated with voice level, though, they can be different: For example we can use varying voice level when whispering or speaking very breathy. Therefore, additional control of breathiness would be a desirable feature.

To control the breathiness in voice, the bottleneck auto-encoder should be conditioned on a breathiness parameter. Finding a useful parametrization of breathiness is key to a successful breathiness transformation. We could use some existing breathiness analysis from the literature. Most classical neural vocoders have some kind of parametrization of the noise. To achieve good breathiness transformation, we should investigate, which noise parametrization is best suited for this task.

The configuration of Chapter 6 already allows modifications of breathiness
through the voice-unvoiced mask, which is part of the $f_0$ conditioning. Since a
fundamental frequency is only defined in voiced segments, the conditional input
of Chapter 6 contains an additional binary value, the voiced-unvoiced mask, to
indicate, whether the $f_0$ input can be used or not.

During inference, this input can be used to modify the breathiness, by providing
intermediate values between 0 and 1. Although the auto-encoder has never seen
other values than 0 and 1 during training, it somewhat adds more noise in the
voice segments when changing the voice-unvoiced mask from ‘voiced’ to ‘unvoiced’.
The result, however, is not very realistic and needs substantial improvement. As
previously observed, the auto-encoder generally has a problem with generating the
noise component realistically. With the variable bottleneck size, this problem could
be improved substantially in the unvoiced regions and in regular non-breathy voice,
the poor quality of the noise component did not impact the overall quality much.
When synthesizing breathy voice, however, this approach does not work any more.
Thus, different methods to generate realistic noise are required for this application.

Furthermore, the approach of interpolating the voiced-unvoiced mask does not
provide very accurate control. The noise increases with increased unvoiced para-
meter, but it is impossible to predict, which value really should be used. Finally, the
voiced-unvoiced mask is used to indicate whether the f₀ input should be used or
not. As a consequence, with increased unvoiced parameter the auto-encoder starts
to ignore the f₀ input and generates incorrect f₀ contours.

Thus, while the fractional voiced-unvoiced approach allows some control over the
breathiness of the voice, further research is required, to create a realistic breathiness
modification. For that, a reliable parametrization of the breathiness is required.

Voice identity

We established that in order to achieve extreme pitch transformation the voice identity needs to be adjusted. If the new pitch is out of the range of the original voice, the synthetic recording needs to borrow the timbre of another voice. Two different behaviours with regard to voice identity adjustment can be observed. Either the voice identity is preserved within its vocal range and only if the new pitch falls outside the original voice's range the voice identity changes rather abruptly. The other possibility is that the voice identity is gradually changed with increasing amount of transposition. Currently, both behaviours are possible, and the outcome is rather unpredictable. Thus, it is desirable to have more control about the implicit voice identity transformation.

Depending on the application, the voice identity should change or not:

* A common application of pitch transformation is in audio editing. In this application we have a recording of one singer and may want to change one or a few notes at the same time. Here, the changes in pitch should not cause a noticeable change in voice identity and thus, the voice identity must be preserved.

* Another application is to transform a recording into another domain. For example a piece could be recorded by one voice and then transformed entirely into a different vocal range. Here the voice identity needs to change.

The deep learning approach allows us to influence how the voice identity changes. It, thus, requires us to consider what kind of pitch transformation do we want: Do we want to have the same person sing the passage in a different pitch and accept the limitations of the original voice or do we want the voice identity (voice type) to change with the pitch, as if it was sung by a voice with a different range?

**Explicit voice identity conditioning** The obvious solution to control the voice iden-
tity would be to introduce a voice identity conditioning to the auto-encoder. This
way we would quite explicitly enter the domain of voice identity transformation.
While voice identity transformation is certainly an interesting research domain in
itself, here, this approach has a few undesirable drawbacks: To condition on the
voice identity, some additional information about the voice has to be provided. This
can be done, either by including the voice in the training set or learning some kind of
space of voice identity and mapping the input voice into this space for conditioning.
For our purposes, the former is not feasible, as we do not know in advance on which
voice the transformation will be applied. Likewise, it cannot be expected from the
user to fine tune the model for each voice they use. Thus, zero-shot methods (the
latter), need to be employed if explicit voice identity conditioning is desired.

Another drawback of explicit conditioning is that we lose the implicitness of
voice identity adjustment. If we want to transpose a recording into a new range
the system should find the voice identity in the new range that is closest to the
original voice. Forcing the user to choose a new voice identity themselves might
not be desired in all applications.

**Conditioning on the position in the vocal range**
Instead of conditioning on the
whole voice identity, we could condition only on the characteristics that need
to be changed. Here, what needs to be adjusted is only the vocal range of the
voice. Thus conditioning on the vocal range alone should suffice to obtain sufficient
control over the implicit voice identity change. Two possible ways to convey this
information to the auto-encoder exist. The singer's or speaker's vocal range could
be provided through some kind of parametrization, e.g. minimum and maximum
pitch. Then the auto-encoder can infer where the note lies within the vocal range
and synthesize the timbre accordingly. Similar drawbacks as with the explicit voice
identity conditioning exist: This method requires the user to know the exact vocal
range of the original singer.

Another possibility is to provide information about where the note should lie
within the vocal range of the singer. We call this the *relative pitch*, as it is relative to
the singer's vocal range. This property could be estimated similar to the voice level
estimation of Chapter 7 from the voice timbre. Then both absolute pitch (i.e. the
pitch in Hz) and relative pitch could be provided simultaneously to the auto-encoder.
If we want to preserve the voice identity, the relative pitch needs to change together
with absolute pitch. If we want to adjust the voice identity, the relative pitch needs
to be preserved while the absolute pitch is changed. While the relative pitch only
defines the voice identity implicitly, it describes the voice property that we want to
control most explicitly.

**Relative pitch and voice identity** Absolute pitch, relative pitch and voice identity are pairwise independent but not mutually independent. Two properties can be provided while the third will be inferred. Which two properties are provided depends on the application. Above, we have discussed the pros and cons of accompanying absolute pitch with either voice identity or relative pitch. Furthermore, relative pitch and voice identity can be combined to infer the absolute pitch (Qian, Jin, et al., 2020). This is useful in speech, where the absolute pitch bears little significance and most information is contained in the pitch contour. Thus, we often find this parametrization in speaker identity conversion.
9.4.3. Understanding the latent space of the auto-encoder

We have studied the latent space of the auto-encoder in Chapter 6 (notably Sec-
tion 6.3.3), where we have examined how different vowels are encoded in the latent
space of the auto-encoder. While this helped us to understand how the information
is spread over time, still many open questions remain.

It would be interesting to investigate, how the remaining information is encoded
in the latent space, in particular voice properties related to voice identity. Finding
ways to manipulate the voice directly on the latent space would open numerous
possibilities of artistic applications.

The variable bottleneck size approach, as discussed in Section 9.4.1, forms a kind
of variational auto-encoder (VAE) since the dropout transforms the individual points
into a binary probability distribution. Knowledge about VAEs could be used to reason
about the latent space of our auto-encoder. Further research could be conducted
around using other VAE approaches as well to further stabilize the latent space in
terms of continuity.
List of figures

<table><tbody><tr><td>2.1.</td><td>Electrical circuit of the first electrical speech synthesizer</td><td>7</td></tr><tr><td>2.2.</td><td>Cross-section of the auditory system</td><td>9</td></tr><tr><td>2.3.</td><td>Cross-section of the cochlea</td><td>10</td></tr><tr><td>2.4.</td><td>Mel filter bank for a configuration with 20 mel-bands</td><td>13</td></tr><tr><td>2.5.</td><td>Equal loudness contours from ISO 226: 2003 standard</td><td>15</td></tr><tr><td>2.6.</td><td>Cross-section of the larynx</td><td>16</td></tr><tr><td>2.7.</td><td>Cross-section of the vocal tract</td><td>17</td></tr><tr><td>2.8.</td><td>Liljencrats-Fant glottal pulse and its defining parameters</td><td>19</td></tr><tr><td>2.9.</td><td>Liljencrats-Fant glottal flow derivative for different R<sub>d</sub> values</td><td>22</td></tr><tr><td>2.10.</td><td>Deep dream transformed image of the Mona Lisa</td><td>27</td></tr><tr><td>2.11.</td><td>Pie charts showing the distribution of different attributes</td><td>36</td></tr><tr><td>2.12.</td><td>Histogram of fundamental frequency</td><td>37</td></tr><tr><td>2.13.</td><td>Histogram of fundamental frequency in speech</td><td>38</td></tr><tr><td>2.14.</td><td>Histogram of fundamental frequency in singing</td><td>38</td></tr><tr><td>2.15.</td><td>Screenshot of a perceptive test website</td><td>43</td></tr><tr><td>4.1.</td><td>Schematic of the analysis-synthesis framework</td><td>67</td></tr><tr><td>5.1.</td><td>Schematic of the MBExWN</td><td>86</td></tr><tr><td>6.1.</td><td>Bottleneck auto-encoder schematic overview</td><td>97</td></tr><tr><td>6.2.</td><td>Accuracy of f<sub>0</sub> transformation</td><td>102</td></tr><tr><td>6.3.</td><td>NMFE as a function of the bottleneck size</td><td>104</td></tr><tr><td>6.4.</td><td>Mean opinion score plotted against transposition amount</td><td>106</td></tr><tr><td>6.5.</td><td>Distribution of vowels in the latent code space</td><td>108</td></tr><tr><td>6.6.</td><td>Distribution of intensities in the latent code space</td><td>109</td></tr><tr><td>6.7.</td><td>Progression of the latent code for stable vowels</td><td>110</td></tr><tr><td>7.1.</td><td>Histogram of voice level for different dynamics</td><td>124</td></tr><tr><td>7.2.</td><td>Perceived change in voice level over applied voice level change</td><td>126</td></tr><tr><td>7.3.</td><td>Mean opinion score for transformed recordings</td><td>127</td></tr><tr><td>8.1.</td><td>Screen shot of CIRCE</td><td>133</td></tr><tr><td>8.2.</td><td>The installation another voice by Judith Deschamps</td><td>134</td></tr><tr><td>8.3.</td><td>The computer running the improvisation of the Farinelli Voice</td><td>136</td></tr><tr><td>8.4.</td><td>Shadow play from Né entre corps</td><td>139</td></tr><tr><td>8.5.</td><td>Scene from Day 0 - trans-instrumentalism</td><td>140</td></tr><tr><td>8.6.</td><td>Anonymous text used for G.N.Z.</td><td>141</td></tr></tbody></table>
<table><tr><td>9.1. Full framework during inference</td><td>145</td></tr></table>
List of tables

<table><tr><td>2.1. Speech datasets used in this thesis.</td><td>32</td></tr><tr><td>2.2. Singing datasets used in this thesis.</td><td>34</td></tr><tr><td>4.1. Losses used for the analysis-synthesis framework.</td><td>67</td></tr><tr><td>4.2. Results on CMU arctic with metrics considering all GCI.</td><td>71</td></tr><tr><td>5.1. Mean opinion scores for MBExWN and other baselines on speech</td><td>89</td></tr><tr><td>5.2. Mean opinion scores for MBExWN and other baselines on singing</td><td>89</td></tr><tr><td>6.1. Final loss values for models of Chapter 6</td><td>100</td></tr><tr><td>6.2. Quality ratings of the algorithms for speech</td><td>104</td></tr><tr><td>6.3. Quality ratings of the algorithms for singing</td><td>105</td></tr><tr><td>7.1. Transformation precision of the auto-encoders</td><td>125</td></tr><tr><td>7.2. Perceived change in voice level over applied voice level change</td><td>126</td></tr><tr><td>7.3. Mean opinion score for transformed recordings</td><td>127</td></tr><tr><td>8.1. Phrases in the piece Quell'usignolo.</td><td>132</td></tr><tr><td>8.2. Structure of the piece Quell'usignolo.</td><td>132</td></tr><tr><td>A.1. List of layers in the analysis module.</td><td>160</td></tr><tr><td>A.2. List of layers in the synthesis module.</td><td>160</td></tr><tr><td>A.3. List of layers in the encoder.</td><td>161</td></tr><tr><td>A.4. List of layers in the decoder.</td><td>161</td></tr><tr><td>A.5. List of layers of the voice level estimator.</td><td>162</td></tr></table>
# A. List of model configurations and their layers

This appendix section provides the architectures of the models used in this thesis. The tables follow the same nomenclature in addition to the legend below them:

**Layer types:**

**BNorm** Batch normalization.

**ConCat** Concatenation of inputs in feature dimension.

**Conv₁** 1-D convolutional layer.

**Conv₂** 2-D convolutional layer.

**Conv₂**T Transposed 2-D convolutional layer.

**MaxPool** Local max pooling layer.

**WaveNet Stack** A stack of gated WaveNet layers with skip connections and exponentially increasing dilated convolutions as described by Oord, Dieleman, et al. (2016).

**Activation functions:**

**ReLU** Rectangular Linear Unit $(x + |x|)/2$.

**sigmoid** Sigmoid function $(1 + e^{-x})^{-1}$.

When listing inputs, external inputs (that are not the output of previous layers) are written in *italics*. The dimensions of the output tensors (out. shape) are given as (time, feature) or (time, frequency, feature).
## A.1. Models from Chapter 4: Pulse detection

<table><caption>Table A.1.: List of layers in the analysis module.</caption><thead><tr><th>Layer</th><th>Inputs</th><th>Type</th><th>n<sub>h</sub></th><th>k</th><th>Act.</th><th>Out. shape</th></tr></thead><tbody><tr><td>conv1</td><td>audio</td><td>Conv<sub>1</sub></td><td>512</td><td>32</td><td>ReLU</td><td>(t, 512)</td></tr><tr><td>pool1</td><td>conv1</td><td>MaxPool</td><td>—</td><td>2</td><td>—</td><td>(t/2, 512)</td></tr><tr><td>bn1</td><td>pool1</td><td>BNorm</td><td>—</td><td>—</td><td>—</td><td>(t/2, 512)</td></tr><tr><td>conv2</td><td>bn1</td><td>Conv<sub>1</sub></td><td>64</td><td>32</td><td>ReLU</td><td>(t/2, 64)</td></tr><tr><td>pool2</td><td>conv2</td><td>MaxPool</td><td>—</td><td>2</td><td>—</td><td>(t/4, 64)</td></tr><tr><td>bn2</td><td>pool2</td><td>BNorm</td><td>—</td><td>—</td><td>—</td><td>(t/4, 64)</td></tr><tr><td>conv3</td><td>bn2</td><td>Conv<sub>1</sub></td><td>64</td><td>32</td><td>ReLU</td><td>(t/4, 64)</td></tr><tr><td>pool3</td><td>conv3</td><td>MaxPool</td><td>—</td><td>2</td><td>—</td><td>(t/8, 64)</td></tr><tr><td>bn3</td><td>pool3</td><td>BNorm</td><td>—</td><td>—</td><td>—</td><td>(t/8, 64)</td></tr><tr><td>conv4</td><td>bn3</td><td>Conv<sub>1</sub></td><td>256</td><td>32</td><td>ReLU</td><td>(t/8, 256)</td></tr><tr><td>bn4</td><td>conv4</td><td>BNorm</td><td>—</td><td>—</td><td>—</td><td>(t/8, 256)</td></tr><tr><td>conv5</td><td>bn4</td><td>Conv<sub>1</sub></td><td>512</td><td>32</td><td>ReLU</td><td>(t/8, 512)</td></tr><tr><td>bn5</td><td>conv5</td><td>BNorm</td><td>—</td><td>—</td><td>—</td><td>(t/8, 512)</td></tr><tr><td>conv6</td><td>bn5</td><td>Conv<sub>1</sub></td><td>1024</td><td>32</td><td>ReLU</td><td>(t/8, 1024)</td></tr><tr><td>bn6</td><td>conv6</td><td>BNorm</td><td>—</td><td>—</td><td>—</td><td>(t/8, 1024)</td></tr><tr><td>output</td><td>bn6</td><td>Conv<sub>1</sub></td><td>1</td><td>4</td><td>sigmoid</td><td>(t/8, 1024)</td></tr></tbody></table>

*n*<sub>h</sub> Number of filters in convolution

*k* Kernel size of convolution

*audio* Raw-audio input

<table><caption>Table A.2.: List of layers in the synthesis module.</caption><thead><tr><th>Layer</th><th>Inputs</th><th>Type</th><th>n<sub>h</sub></th><th>k</th><th>n<sub>l</sub></th></tr></thead><tbody><tr><td>stack1</td><td>pulses, noise, env, envn</td><td>WaveNet Stack</td><td>256</td><td>2</td><td>6</td></tr><tr><td>stack2</td><td>stack1, env, envn</td><td>WaveNet Stack</td><td>256</td><td>2</td><td>6</td></tr></tbody></table>

*n*<sub>h</sub> Number of hidden units in stack

*k* Kernel size of dilated convolutions

*n*<sub>l</sub> Number of layer in stack

*pulses* Glottal pulse signal as provided by analysis module

*noise* White noise

*env* Spectral envelope of the audio signal

*envn* Spectral envelope of the noise component
## A.2. Models from Chapter 6: Bottleneck auto encoder

<table><caption>Table A.3.: List of layers in the encoder.</caption><thead><tr><th>Layer</th><th>Inputs</th><th>Type</th><th>n<sub>h</sub></th><th>k</th><th>s</th><th>Act.</th><th>Out. shape</th></tr></thead><tbody><tr><td>f0in</td><td>f<sub>0</sub>, vum</td><td>Conv<sub>2</sub><sup>T</sup></td><td>2</td><td>(1, 80)</td><td>(1, 80)</td><td></td><td>(t, 80, 2)</td></tr><tr><td>concat</td><td>mel, f0in</td><td>ConCat</td><td></td><td></td><td></td><td></td><td>(t, 80, 3)</td></tr><tr><td>hid1</td><td>concat</td><td>Conv<sub>2</sub></td><td>512</td><td>(1, 2)</td><td>(1, 2)</td><td>ReLU</td><td>(t, 40, 512)</td></tr><tr><td>hid2</td><td>hid1</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 40, 512)</td></tr><tr><td>hid3</td><td>hid2</td><td>Conv<sub>2</sub></td><td>512</td><td>(1, 2)</td><td>(1, 2)</td><td>ReLU</td><td>(t, 20, 512)</td></tr><tr><td>hid4</td><td>hid3</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 20, 512)</td></tr><tr><td>hid5</td><td>hid4</td><td>Conv<sub>2</sub></td><td>512</td><td>(1, 2)</td><td>(1, 2)</td><td>ReLU</td><td>(t, 10, 512)</td></tr><tr><td>hid6</td><td>hid5</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 10, 512)</td></tr><tr><td>hid7</td><td>hid6</td><td>Conv<sub>2</sub></td><td>512</td><td>(1, 2)</td><td>(1, 2)</td><td>ReLU</td><td>(t, 5, 512)</td></tr><tr><td>hid8</td><td>hid7</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 5, 512)</td></tr><tr><td>hid9</td><td>hid8</td><td>Conv<sub>2</sub></td><td>512</td><td>(1, 5)</td><td>(1, 5)</td><td>ReLU</td><td>(t, 1, 512)</td></tr><tr><td>out</td><td>hid9</td><td>Conv<sub>2</sub></td><td>n<sub>b</sub></td><td>(3, 1)</td><td></td><td></td><td>(t, 1, 512)</td></tr></tbody></table>

<table><caption>Table A.4.: List of layers in the decoder.</caption><thead><tr><th>Layer</th><th>Inputs</th><th>Type</th><th>n<sub>h</sub></th><th>k</th><th>s</th><th>Act.</th><th>Out. shape</th></tr></thead><tbody><tr><td>concat</td><td>code, f<sub>0</sub></td><td>ConCat</td><td></td><td></td><td></td><td></td><td>(t, 1, n<sub>b</sub> + 2)</td></tr><tr><td>hid0</td><td>concat</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 1, 512)</td></tr><tr><td>hid1</td><td>hid0</td><td>Conv<sub>2</sub><sup>T</sup></td><td>512</td><td>(1, 5)</td><td>(1, 5)</td><td>ReLU</td><td>(t, 5, 512)</td></tr><tr><td>hid2</td><td>hid1</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 5, 512)</td></tr><tr><td>hid3</td><td>hid2</td><td>Conv<sub>2</sub><sup>T</sup></td><td>512</td><td>(1, 2)</td><td>(1, 2)</td><td>ReLU</td><td>(t, 10, 512)</td></tr><tr><td>hid4</td><td>hid3</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 10, 512)</td></tr><tr><td>hid5</td><td>hid4</td><td>Conv<sub>2</sub><sup>T</sup></td><td>512</td><td>(1, 2)</td><td>(1, 2)</td><td>ReLU</td><td>(t, 20, 512)</td></tr><tr><td>hid6</td><td>hid5</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 20, 512)</td></tr><tr><td>hid7</td><td>hid6</td><td>Conv<sub>2</sub><sup>T</sup></td><td>512</td><td>(1, 2)</td><td>(1, 2)</td><td>ReLU</td><td>(t, 40, 512)</td></tr><tr><td>hid8</td><td>hid7</td><td>Conv<sub>2</sub></td><td>512</td><td>(3, 3)</td><td></td><td>ReLU</td><td>(t, 40, 512)</td></tr><tr><td>hid9</td><td>hid8</td><td>Conv<sub>2</sub><sup>T</sup></td><td>512</td><td>(1, 5)</td><td>(1, 5)</td><td>ReLU</td><td>(t, 80, 512)</td></tr><tr><td>out</td><td>hid9</td><td>Conv<sub>2</sub></td><td>1</td><td>(3, 3)</td><td></td><td></td><td>(t, 80, 512)</td></tr></tbody></table>

n<sub>h</sub> Number of filters in convolution

n<sub>b</sub> Bottleneck size

k Kernel size of convolution

s Stride

audio Raw-audio input

f<sub>0</sub> f<sub>0</sub> input

vum Voiced-unvoiced input

mel Mel-spectrogram input

code Latent code input
A.3. Models from **Chapter 7: Voice level transformation**

<table><caption>**Table A.5.:** List of layers of the voice level estimator.</caption><thead><tr><th>Layer</th><th>Inputs</th><th>Type</th><th>n<sub>h</sub></th><th>k</th><th>Act.</th><th>Out. shape</th></tr></thead><tbody><tr><td>hid1</td><td>mel</td><td>Conv<sub>1</sub></td><td>80</td><td>3</td><td>ReLU</td><td>(t, 80)</td></tr><tr><td>hid2</td><td>hid1</td><td>Conv<sub>1</sub></td><td>100</td><td>3</td><td>ReLU</td><td>(t, 100)</td></tr><tr><td>hid3</td><td>hid2</td><td>Conv<sub>1</sub></td><td>100</td><td>1</td><td>ReLU</td><td>(t, 100)</td></tr><tr><td>hid4</td><td>hid3</td><td>Conv<sub>1</sub></td><td>100</td><td>1</td><td>ReLU</td><td>(t, 100)</td></tr><tr><td>hid5</td><td>hid4</td><td>Conv<sub>1</sub></td><td>100</td><td>1</td><td>ReLU</td><td>(t, 100)</td></tr><tr><td>hid6</td><td>hid5</td><td>Conv<sub>1</sub></td><td>100</td><td>1</td><td>ReLU</td><td>(t, 100)</td></tr><tr><td>hid7</td><td>hid6</td><td>Conv<sub>1</sub></td><td>100</td><td>1</td><td>ReLU</td><td>(t, 100)</td></tr><tr><td>hid8</td><td>hid7</td><td>Conv<sub>1</sub></td><td>100</td><td>1</td><td>ReLU</td><td>(t, 100)</td></tr><tr><td>hid9</td><td>hid8</td><td>Conv<sub>1</sub></td><td>50</td><td>1</td><td>ReLU</td><td>(t, 50)</td></tr><tr><td>out</td><td>hid9</td><td>Conv<sub>1</sub></td><td>1</td><td>1</td><td></td><td>(t, 1)</td></tr></tbody></table>

$n_h$ Number of filters in convolution

$k$ Kernel size of convolution

mel Mel-spectrogram of analysed audio
# Glossary

## A

**ACR** Absolute Category Rating
A perceptive test, where ratings are provided for each sample individually on an absolute scale. 40, 42, 167

**ANR** Agence Nationale de la Recherche
The French national research agency. 2, 3, 163

**ARS** Analyse et tRansformation de Style du chant
(Analysis and tRansformation of singing Style)
Research project of the ANR, (ANR-19-CE38-0001-13). This thesis was in part funded by this research project. 2, 3, 62

**AutoVC**
The original bottleneck auto-encoder by Qian, Yang Zhang, Chang, X. Yang, et al. (2019). Used for voice identity conversion. 29, 59, 95, 98, 113

## B

**basilar membrane**
The part of the inner ear, where the incoming sound is decomposed into frequencies and translated into nervous signals. 10, 12, 14, 164

**bottleneck auto-encoder**
Auto-encoder with information bottleneck. The information bottleneck ensures disentanglement of the latent code and an additional control parameter. This is the main technology used for disentanglement in this thesis. 3–5, 29, 59, 91, 95–98, 100, 103, 108, 111, 140, 143, 144, 148, 149, 151, 152, 167

## C

**CCR** Comparison Category Rating
A perceptive test, where pairs of samples are compared against each other. 40–42

**CIRCE** Circe, the IRCAM voice Encoder
A software providing a graphical user interface to the neural voice transformation framework. CIRCE has been developed during this thesis and is available for free on the IRCAM forum at https://forum.ircam.fr/projects/detail/circe/. 3, 133, 138, 140, 142
CNN

Convolutional Neural Network
A neural network architecture, where convolutions are used as linear operations (LeCun et al., 1989). 23-26, 29, 49, 54, 66, 86, 97, 98, 101, 103, 122, 159

CNRS

Centre National de la Recherche Scientifique
(French National Centre for Scientific Research)
French research organization involved in the STMS research lab. 169

concatenative synthesis

Synthesis method where elementary units of voice are concatenated to synthesize a whole phrase. 3, 8, 38, 39, 55, 56, 92, 165

D

DCR

Degradation Category Rating
A perceptive test, where degradation is rated. 40

DDSP

Differentiable Digital Signal Processing
A framework of neural network components based on digital signal processing, through which gradients can be transported. Developed by Engel et al. (2019). 23, 30

DSP

Digital Signal Processing
Signal processing implemented in the computer instead of analogue electrical cirquits. 6, 9, 23, 30, 46, 51, 53, 164

E

EGG

ElectroGlottoGraph
A measurement of conductance through the glottis, used to estimate the contact surface between the vocal folds and thus infer vocal fold activity like GCI. 32, 48, 49, 65

ERB

Equivalent Rectangular Bandwidth
A pitch scale, modelled after the bandwidth of the filter characteristic on the basilar membrane. 12, 14, 121

EUSIPCO

European Signal Processing Conference
A scientific conference specialized in topics around signal processing. The paper on the analysis-synthesis framework for GCI detection from Chapter 4 was published in the conference proceedings of EUSIPCO. 63

F

fft

fast fourier transform
A computationally efficient algorithm to compute the discrete Fourier transform. 121
**FM**

A synthesis technique, where the frequency of a carrier is modulated rapidly to create a complex timbre (Chowning, 1989). 8

G

**GAN**

**Generative Adversarial Networks**

A set of two neural networks, a Generator and a Discriminator. The Generator produces samples that resemble samples from a training set. The Discriminator distinguishes between real (from the training set) and fake (produced by the generator) samples. Generator and Discriminator are trained adversarially in the sense that a low loss for the Generator results in a high loss for the Discriminator and vice versa. 28, 49, 54, 61, 151, 167

**GCI**

**Glottal Closure Instant**

The time instant where the vocal folds start to close. In the glottal flow derivative, this can be seen as a sharp negative peak (see Fig. 2.8 and Section 2.3.2). 3–5, 18, 22, 48, 49, 64–66, 68, 69, 71, 72, 143, 164

**GPU**

**Graphical Processing Unit**

Our main computational device for neural networks. 28, 54

H

**HMM**

**Hidden Markov Model**

Statistical model, where observations are assumed to be generated by a hidden (non-observable) Markov process. 8, 56

**HNM**

**Harmonic plus Noise Model**

A pitch-synchronous parametric vocoder. The each analysis frame is modelled as a sum of harmonically coupled sinusoids (for the voiced component) and noise. 8, 52, 92

I

**ICASSP**

**International Conference on Acoustics, Speech and Signal Processing**

A scientific conference specialized in topics around signal processing. The paper on estimation and transformation of voice level from Chapter 7 was published in the conference proceedings of ICASSP. 113

**IRCAM**

**Institut de Recherche et Coordination Acoustique/Musique**
(Institute of Research and Coordination of Acoustics and Music)

An institution for pairing scientific research with artistic musical creation. Scientific research at IRCAM is carried out to enable its application in musical compositions. 1, 2, 8, 34, 129–131, 163, 169

**ISiS**

**IRCAM Singing Synthesizer**

A parametric concatenative singing synthesizer based on the PaN vocoder. Developed by Ardaillon (2017). 2, 3, 33, 34, 91, 146, 147
**ITU**
International Telecommunication Union
United nations agency responsible for organizing international practices in telecommunication. The ITU provides recommendations and standards regarding practices around topics of telecommunication. Here we use their recommendations on perceptive evaluations. 39, 41, 52

K

**KTH**
Kungliga Tekniska Högskolan
(Royal Institute of Technology)
A research institution of importance for research on the fundamentals of voice. Early speech and singing synthesizers were developed at KTH, like the OVE and MUSSE. 7, 167

L

**LF**
Liljencrants-Fant
Glottal pulse model that describes the glottal pulse signal either with four (Fant, Liljencrants, et al., 1985) or one (Fant, 1995) parameter. Discussed in Section 2.3.2. 8, 18-22, 49, 65, 69, 71, 168

**LibriTTS**
Large speech dataset based on the LibriVox audio book corpus. Used mainly for text-to-speech applications. 55

**LPC**
Linear Predictive Coding
Reparametrization of a signal, where the next sample is predicted from a linear combination of the *n* previous samples. This results in a signal parametrization consisting of an auto-recursive prediction filter and a residual signal. 8, 49-51, 53, 55

M

**MBExWN**
Multi-Band Excited WaveNet
The neural vocoder that is used to transform the mel-spectrogram into raw audio. 4, 85-90, 93, 96, 126, 144, 145, 150

**MDPI**
Multidisciplinary Digital Publishing Institute
A publisher for scientific research articles. 92

**mel-scale**
A pitch scale modelled after the perception of intervals (see Section 2.2.2). 12-14, 81

**mel-spectrogram**
An amplitude spectrogram where the frequency axis is wrapped to the mel-scale. 12, 13, 25, 28, 48, 54-57, 60-62, 73, 77, 80-88, 90, 91, 93, 96-98, 101, 105, 111, 120, 122, 123, 125, 127, 133, 143-145, 147, 149-151, 161, 166, 167, 169
**MelGAN**

An architecture of generative adversarial networks for mel-spectrogram in-
version. First introduced by K. Kumar et al. (2019). 28, 54, 73, 83, 84, 88,
89

**MFCC**

**Mel-Filtered Cepstral Coefficient**

Signal features obtained by Fourier transforming the log-amplitude mel-spectrogram. 12

**MOS**

**Mean Opinion Score**

A quality rating obtained through a perceptive test by asking participants to
rate a recording and averaging their responses. 40, 44, 89, 106, 107

**MUSHRA**

**MUlti Stimulus test with Hidden Reference and Anchor**

A specific kind of Absolute Category Rating perceptive test. 41, 42

**MUSSE**

**Music and Singing Synthesis Equipment**

An early singing synthesizer developed at KTH. 7, 8, 166

**N**

**neural vocoder**

Parametric vocoder implemented with neural networks. The use of neural
networks allows usage of unorthodox parametric spaces. 3-5, 51, 60-64, 73,
75-80, 83, 88, 90, 91, 96, 143, 152, 167

**neural voice transformation framework**

A framework to transform voice with neural networks consisting of a neural
vocoder using the mel-spectrogram as its parametric space and the bottleneck
auto-encoder, which does the actual transformation. The neural voice trans-
formation framework is the central piece of this thesis. 1, 2, 95, 133, 138-140,
142-145, 147, 148, 151, 163

**NMFE**

**Normalized Mean F₀ Error**

A means of analysing disentanglement by measuring how well the external
control parameter is used. 103, 104

**O**

**OQ**

**Open Quotient**

The ratio between open phase $t_e$ and pulse duration $t_0$ (see Section 2.3.2). 18,
20

**OVE**

**Orator Vox Electrica**

An early speech synthesizer developed at KTH. 7, 166
P

PaN

**Pulse and Noise**
A classical vocoder that synthesises the voice using the LF glottal pulse model of and Noise. Developed by Ardaillon (2017). 49, 53, 57, 58, 63–65, 68, 69, 79, 100, 105, 107, 111, 147, 165

**parametric vocoder**

Vocoder that maps between raw-audio and a parametric space. 3, 8, 50, 51, 55, 56, 58, 63, 64, 75, 78, 87, 92, 93, 100, 147

**PC** Quotes stemming from personal correspondence with the author or composer discussing their piece. 138, 139, 141, 142

**phase vocoder**

Non-parametric vocoder that allows signal transformation on the STFT. 8, 51–53, 92

**phon**

Scale of perceived loudness measured in dB above hearing threshold. 14, 15

**PQMF**

**Pseudo Quadrature Mirror Filter bank**
A set of complementary filters that splits the signal into equally large frequency bands. 26, 85, 86

**PSOLA**

**Pitch Synchronous OverLap and Add**
A pitch-synchronous non-parametric vocoder. Short windows are placed around each period of voiced speech. 8, 51–53, 92

R

**ReLU**

**Rectangular Linear Unit**
An activation function that sets the negative components to zero: $ReLU(x) = (x + |x|)/2$. (Nair and Hinton, 2010). 98, 159

**RNN**

**Recurrent Neural Network**
A neural network type, where the output of previous samples is used as additional input. 24, 98

S

**sone**

Scale of perceived loudness that reflects perceived increases. Perceived loudness and thus the sone scale increases with the power of 0.3 of the signal energy. 15, 121

**SPL**

**Sound Pressure Level**
A measure of sound level by measuring the sound pressure at a specific point in space. 114
**SPSS**

**Statistical Parametric Speech Synthesis**

Speech (or singing) synthesis method, where vocoder parameters are generated using a statistical model. 8, 56

**STFT**

**Short Time Fourier Transform**

A spectrogram obtained by splitting the signal in small overlapping frames and applying the discrete Fourier transform to each frame. 13, 26, 27, 67, 168

**STMS**

**Sciences et Technologies de la Musique et du Son**

**(Sciences and Technologies of Music and Sound)**

Joint research unit of IRCAM, Sorbonne Université, CNRS and the French Ministry of Culture. 3, 164

# T

**Tacotron**

One of the first neural text-to-speech engines using the mel-spectrogram. Developed by Y. Wang et al. (2017) (V1) and J. Shen et al. (2018) (V2). 54

**TrueEnv**

A spectral envelope estimation algorithm (Roebel and Rodet, 2005). 50

**TTS**

**Text-To-Speech**

Speech synthesis, where text is used as input and a voice signal reading the text is produced. 54-56, 58, 140, 169

# V

**VAE**

**Variational Auto-Encoder**

An auto-encoder, where the latent space consists of parametric probability distributions, rather than individual points. 57, 59, 155

**VTF**

**Vocal Tract Filter**

A filter that models all the effects that the vocal tract has on the glottal source signal. 16-18, 50, 57, 58

# W

**WaveGlow**

A neural network applying generative flow to audio synthesis. First introduced by Prenger et al. (2019). 54, 83, 84

**WaveNet**

A neural network architecture type designed for audio synthesis. First introduced by Oord, Dieleman, et al. (2016). 53, 54, 66, 77, 83, 84, 86, 159
Bibliography

Alessandro, Christophe d' and Boris Doval (1998). 'Experiments in voice quality modification of natural speech signals: the spectral approach'. In: *The Third ESCA/COCOSDA Workshop (ETRW) on Speech Synthesis* (cit. on p. 58).

- (2003). 'Voice quality modification for emotional speech synthesis'. In: *Eighth European Conference on Speech Communication and Technology* (cit. on p. 58).

Alessandro, Christophe d', Bayya Yegnanarayana, and Vassilios Darsinos (1995). 'Decomposition of speech signals into deterministic and stochastic components'. In: *1995 International Conference on Acoustics, Speech, and Signal Processing. Vol. 1*. IEEE, pp. 760-763 (cit. on p. 51).

Alku, Paavo (2011). 'Glottal inverse filtering analysis of human voice production—A review of estimation and parameterization methods of the glottal excitation and their applications'. In: *Sadhana* 36.5, pp. 623-650 (cit. on p. 49).

Anand, J. M., S. Guruprasad, and Bayya Yegnanarayana (2006). 'Extracting formants from short segments of speech using group delay functions'. In: *7th Annual Conference of the International Speech Communication Association (INTERSPEECH)*, pp. 1009-1012. ISBN: 9781604234497 (cit. on pp. 49, 50).

Ardaillon, Luc (2017). 'Synthesis and expressive transformation of singing voice'. PhD thesis. Université Pierre et Marie Curie. url: https://hal.archives-ouvertes.fr/tel-01710926/document (cit. on pp. 8, 33, 34, 49, 53, 56, 57, 63-65, 79, 100, 111, 123, 147, 165, 168).

Ardaillon, Luc, Celine Chabot-Canet, and Axel Roebel (2016). 'Expressive control of singing voice synthesis using musical contexts and a parametric f0 model'. In: *17th Annual Conference of the International Speech Communication Association (INTER-SPEECH)*. ISCA (cit. on p. 92).

Ardaillon, Luc, Gilles Degottex, and Axel Roebel (2015). 'A multi-layer F0 model for singing voice synthesis using a B-spline representation with intuitive controls'. In: *16th Annual Conference of the International Speech Communication Association (INTERSPEECH)*. ISCA (cit. on p. 92).

Ardaillon, Luc and Axel Roebel (2017). 'A mouth opening effect based on pole modification for expressive singing voice transformation'. In: *18th Annual Conference of the International Speech Communication Association (INTERSPEECH)*, pp. 1288-1292 (cit. on p. 58).

- (2019). 'Fully-convolutional network for pitch estimation of speech signals'. In: *20th Annual Conference of the International Speech Communication Association (INTER-SPEECH)*. ISCA (cit. on pp. 37, 48, 49, 64, 65, 97, 122).

- (2020). 'GCI detection from raw speech using a fully-convolutional network'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 4, 49, 63-66, 68-72).
Arias, Pablo et al. (2021). 'Beyond correlation: acoustic transformation methods for the experimental study of emotional voice and speech'. In: *Emotion Review* 13.1, pp. 12–24 (cit. on p. 93).

Arik, Sercan, Jitong Chen, et al. (2018). 'Neural voice cloning with a few samples'. In: *Advances in neural information processing systems* 31 (cit. on p. 77).

Arik, Sercan, Gregory Diamos, et al. (2017). 'Deep voice 2: Multi-speaker neural text-to-speech'. In: *arXiv preprint arXiv:1705.08947* (cit. on pp. 77, 80, 82, 83).

Atal, Bishnu S and Suzanne L Hanauer (1971). 'Speech analysis and synthesis by linear prediction of the speech wave'. In: *The journal of the acoustical society of America* 50.2B, pp. 637–655 (cit. on pp. 8, 50, 53).

Babacan, Onur et al. (2013a). 'A comparative study of pitch extraction algorithms on a large variety of singing sounds'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on p. 48).

- (2013b). 'A quantitative comparison of glottal closure instant estimation algorithms on a large variety of singing sounds'. In: *14th Annual Conference of the International Speech Communication Association (INTERSPEECH)*, pp. 1702–1706 (cit. on p. 49).

Baevski, Alexei et al. (2020). 'wav2vec 2.0: A framework for self-supervised learning of speech representations'. In: *Advances in neural information processing systems* 33, pp. 12449–12460 (cit. on p. 53).

Barash, Omer (2023). G.N.Z. (Cit. on p. 129).

Benaroya, Laurent, Nicolas Obin, and Axel Roebel (2023). 'Manipulating Voice Attributes by Adversarial Learning of Structured Disentangled Representations'. In: *Entropy* 25.2, p. 375 (cit. on pp. 59, 83, 84).

Bengio, Emmanuel et al. (2017). 'Independently controllable features'. In: *arXiv preprint arXiv:1703.07718* (cit. on p. 58).

Bengio, Yoshua, Aaron Courville, and Pascal Vincent (2013). 'Representation learning: A review and new perspectives'. In: *IEEE transactions on pattern analysis and machine intelligence* 35.8, pp. 1798–1828 (cit. on p. 58).

Blaauw, Merlijn (2022). 'Modeling timbre for neural singing synthesis: methods for data-efficient, reduced effort voice creation, and fast and stable inference'. PhD thesis. Universitat Pompeu Fabra (cit. on p. 6).

Blaauw, Merlijn and Jordi Bonada (2017). 'A Neural Parametric Singing Synthesizer Modeling Timbre and Expression from Natural Songs'. In: *Applied Sciences* 7.12, p. 1313 (cit. on p. 56).

Bonada, Jordi, Marti Umbert, and Merlijn Blaauw (2016). 'Expressive singing synthesis based on unit selection for the singing synthesis challenge 2016'. In: *17th Annual Conference of the International Speech Communication Association (INTERSPEECH)*. ISCA (cit. on pp. 8, 56, 92).

Bous, Frederik (2019). 'Generating Spectral Envelopes for Singing Synthesis Using Neural Networks'. MA thesis. Technische Universität Darmstadt (cit. on p. 1).

Bous, Frederik, Luc Ardaillon, and Axel Roebel (2020). 'Semi-supervised learning of glottal pulse positions in a neural analysis-synthesis framework'. In: *European Signal Processing Conference (EUSIPCO)*. IEEE (cit. on pp. 4, 63).
Bous, Frederik, Laurent Benaroya, et al. (2022). ‘Voice Reenactment with F0 and timing constraints and adversarial learning of conversions’. In: *European Signal Processing Conference (EUSIPCO)*. IEEE (cit. on pp. 28, 60, 131, 150).

Bous, Frederik and Axel Roebel (2019). ‘Analysing deep learning-spectral envelope prediction methods for singing synthesis’. In: *European Signal Processing Conference (EUSIPCO)*. IEEE (cit. on p. 97).

- (2022). ‘A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice’. In: *Information* 13.3, p. 102 (cit. on pp. 4, 92).

- (2023). ‘Analysis and transformations of intensity in singing voice’. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 4, 113).

Broschi, Carlo (.). *Sammlung von Arien für Gesang mit Instrumentalbegleitung*. url: https://digital.onb.ac.at/RepViewer/viewer faces?doc=DTL_4711034 (cit. on p. 131).

Camacho, Arturo (2007). ‘SWIPE: A sawtooth waveform inspired pitch estimator for speech and music’. PhD thesis. University of Florida. url: http://www.kerwa.ucr.ac.cr:8080/handle/10669/536 (cit. on pp. 11, 48).

Caracalla, Hugo and Axel Roebel (2017). ‘Gradient conversion between time and frequency domains using wirtinger calculus’. In: *Digital Audio Effects (DAFx)*, pp. 234–238 (cit. on p. 26).

- (2020). ‘Sound texture synthesis using RI spectrograms’. In: *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 416–420 (cit. on p. 27).

Carbonneau, Marc-André et al. (2022). ‘Measuring disentanglement: A review of metrics’. In: *IEEE Transactions on Neural Networks and Learning Systems* (cit. on p. 59).

Chen, Fangxin et al. (2004). ‘Acoustic analysis of friendly speech’. In: *2004 IEEE International Conference on Acoustics, Speech, and Signal Processing*. Vol. 1. IEEE, pp. I–569 (cit. on p. 61).

Chen, Ricky TQ et al. (2018). ‘Isolating sources of disentanglement in variational autoencoders’. In: *Advances in neural information processing systems* 31 (cit. on p. 59).

Cheveigné, Alain de (Jan. 2010). ‘Pitch perception’. In: *Oxford Handbook of Auditory Science: Hearing*. Oxford University Press. ISBN: 9780199233557. doi: 10.1093/oxfordhb/9780199233557.013.0004 (cit. on p. 11).

Cheveigné, Alain de and Hideki Kawahara (2002). ‘YIN, a fundamental frequency estimator for speech and music’. In: *The Journal of the Acoustical Society of America* 111.4, pp. 1917–1930 (cit. on p. 48).

Choi, Hyeong-Seok, Juheon Lee, et al. (2021). ‘Neural analysis and synthesis: Re-constructing speech from self-supervised representations’. In: *Advances in Neural Information Processing Systems* 34, pp. 16251–16265 (cit. on p. 54).

Choi, Hyeong-Seok, Jinhyeok Yang, et al. (2022). ‘NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis’. In: *The Eleventh International Conference on Learning Representations* (cit. on pp. 53, 60).

Choi, Yunjey et al. (2018). ‘Stargan: Unified generative adversarial networks for multi-domain image-to-image translation’. In: *Conference on Computer Vision and Pattern Recognition (CVPR)*. IEEE (cit. on p. 28).
Chowning, John (1989). 'Frequency Modulation Synthesis of the Singing Voice'. In: *Current directions in computer music research*. Ed. by Max V. Mathews and John Robinson Pierce. Boston: MIT Press. Chap. 6, pp. 57–64 (cit. on pp. 8, 165).

Corbiau, Gérard (1994). *Farinelli* (cit. on p. 130).

Cramer, Elliot M and WH Huggins (1958). 'Creation of pitch through binaural interaction'. In: *The Journal of the Acoustical Society of America* 30.5, pp. 413–417 (cit. on p. 11).

Culling, John F, A Quentin Summerfield, and David H Marshall (1998). 'Dichotic pitches as illusions of binaural unmasking. I. Huggins' pitch and the "binaural edge pitch"'. In: *The Journal of the Acoustical Society of America* 103.6, pp. 3509–3526 (cit. on p. 11).

Deepak, KT et al. (2019). 'Glottal Instants Extraction from Speech Signal Using Generative Adversarial Network'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 5946–5950 (cit. on p. 49).

Defez, Àngel Calzada, Joan Claudi Socoró Carrié, and Robert AJ Clark (2013). 'Parametric model for vocal effort interpolation with Harmonics Plus Noise Models'. In: *8th ISCA Tutorial and Research Workshop on Speech Synthesis, SSW 2013* (cit. on p. 58).

Degottex, Gilles, Pierre Lanchantin, and Mark Gales (2016). 'A Pulse Model in Log-Domain for a Uniform Synthesizer'. In: *9th ISCA Speech Synthesis Workshop*, pp. 230–236 (cit. on p. 51).

Degottex, Gilles, Pierre Lanchantin, Axel Roebel, et al. (2013). 'Mixed source model and its adapted vocal tract filter estimate for voice transformation and synthesis'. In: *Speech Communication* 55.2, pp. 278–294 (cit. on pp. 51, 53, 57, 58, 63, 92).

Degottex, Gilles, Axel Roebel, and Xavier Rodet (2010). 'Phase minimization for glottal model estimation'. In: *IEEE Transactions on Audio, Speech, and Language Processing* 19.5, pp. 1080–1090 (cit. on pp. 50, 65).

- (2011). 'Pitch transposition and breathiness modification using a glottal source model and its adapted vocal-tract filter'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 58, 93).

Depalle, Philippe, Guillermo Garcia, and Xavier Rodet (1994). 'A virtual castrato (!?)'. In: *International Computer Music Conference (ICMC)*, pp. 357–360 (cit. on pp. 130, 137).

Desai, Srinivas et al. (2009). 'Voice conversion using artificial neural networks'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 60, 92).

Deschamps, Judith (2022a). 'La mue, la mort et le chant d'une IA'. In: *Fabula / Les colloques, Souci de l'autre, souci de soi et création, Pour une littérature du care*. url: http://www.fabula.org/colloques/document8285.php (cit. on pp. 34, 131).

- (2022b). *La Mue, un conte vidéographique* (cit. on pp. 2, 129, 134).

Deshpande, P. S. and M. S. Manikandan (2018). 'Effective Glottal Instant Detection and Electroglottographic Parameter Extraction for Automated Voice Pathology Assessment'. In: *IEEE Journal of Biomedical and Health Informatics* 22.2, pp. 398–408. issn: 21682194. doi: 10.1109/JBHI.2017.2654683 (cit. on p. 49).

Desler, Anne (2014). "Il novello Orfeo' Farinelli: vocal profile, aesthetics, rhetoric'. PhD thesis. University of Glasgow (cit. on p. 130).
Do, Kien and Truyen Tran (2020). 'Theory and Evaluation Metrics for Learning Disentangled Representations'. In: *International Conference on Learning Representations*. url: https://openreview.net/forum?id=HJgK0h4Ywr (cit. on p. 59).

Dohi, Kota, Takashi Endo, and Yohei Kawaguchi (2022). 'Disentangling physical parameters for anomalous sound detection under domain shifts'. In: *30th European Signal Processing Conference (EUSIPCO)*. IEEE, pp. 279-283 (cit. on p. 59).

Donahue, Jeff et al. (2021). ‘End-to-end Adversarial Text-to-Speech’. In: *International Conference on Learning Representations*. url: https://openreview.net/forum?id=rsf1z-JSj87 (cit. on pp. 8, 57, 78).

Doval, Boris, Christophe d' Alessandro, and Nathalie Henrich (2006). 'The spectrum of glottal flow models'. In: *Acta acustica united with acustica* 92.6, pp. 1026-1046 (cit. on p. 18).

Drugman, T., Paavo Alku, et al. (2014). ‘Glottal source processing: From analysis to applications’. In: *Computer Speech and Language* 28.5, pp. 1117-1138. issn: 10958363. doi: 10.1016/j.csl.2014.03.003 (cit. on p. 49).

Drugman, T. and T. Dutoit (2009). ‘Glottal Closure and Opening Instant Detection from Speech Signals’. In: *10th Annual Conference of the International Speech Communication Association (INTERSPEECH)* (cit. on pp. 49, 70, 71).

- (2012). 'The Deterministic plus Stochastic model of the residual signal and its applications'. In: *IEEE Transactions on Audio, Speech and Language Processing* 20.3, pp. 968-981. issn: 15587916. doi: 10.1109/TASL.2011.2169787 (cit. on p. 49).

Duan, Zhiyan et al. (2013). ‘The NUS sung and spoken lyrics corpus: A quantitative comparison of singing and speech’. In: *6th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APISPA ASC)*. IEEE (cit. on pp. 33, 34).

Dudley, Homer (1939). 'Remaking speech'. In: *The Journal of the Acoustical Society of America* 11.2, pp. 169-177 (cit. on pp. 6, 7, 46, 51, 92).

Duifhuis, Hendrikus, Lei F Willems, and RJ Sluyter (1982). 'Measurement of pitch in speech: An implementation of Goldstein's theory of pitch perception'. In: *The Journal of the Acoustical Society of America* 71.6, pp. 1568-1580 (cit. on p. 48).

Eastwood, Cian and Christopher KI Williams (2018). 'A framework for the quantitative evaluation of disentangled representations'. In: *International Conference on Learning Representations* (cit. on p. 59).

EMS, European Mathematical Society (2020). ‘Confidence intervals’. In: *Encyclopedia of Mathematics*. Accessed 2023-05-07. EMS Press. url: http://encyclopediaofmath.org/index.php?title=Confidence_estimation&oldid=44400 (cit. on p. 44).

Engel, Jesse, Chenjie Gu, Adam Roberts, et al. (2019). ‘DDSP: Differentiable Digital Signal Processing’. In: *International Conference on Learning Representations (ICLR)* (cit. on pp. 23, 30, 164).

Esling, Philippe, Adrien Bitton, et al. (2018). ‘Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics’. In: *21st International Conference on Digital Audio Effects (DAFx)* (cit. on p. 29).

Evanini, Keelan, Stephen Isard, and Mark Liberman (2009). ‘Automatic formant extraction for sociolinguistic analysis of large corpora’. In: *10th Annual Conference of the International Speech Communication Association (INTERSPEECH)* (cit. on p. 50).
Fant, Gunnar (1981). 'The source filter concept in voice production'. In: *STL-QPSR* 1.1981, pp. 21–37 (cit. on pp. 47, 63).

- (1995). 'The LF-model revisited. Transformations and frequency domain analysis'. In: *Speech Trans. Lab. Q. Rep.*, Royal Inst. of Tech. *Stockholm* 2.3, p. 40 (cit. on pp. 18, 20, 21, 64, 65, 71, 100, 166).

Fant, Gunnar and Anita Kruckenberg (1994). 'Notes on stress and word accent in Swedish'. In: *Proceedings of the international symposium on prosody*. Vol. 18, pp. 2–3 (cit. on p. 21).

Fant, Gunnar, Johan Liljencrans, and Qi-guang Lin (1985). 'A four-parameter model of glottal flow'. In: *STL-QPSR* 4.1985, pp. 1–13 (cit. on pp. 8, 15, 18, 20, 65, 100, 166).

Fant, Gunnar and J Martony (1962). 'Speech synthesis instrumentation for parametric synthesis (OVE II)'. In: *Speech Transmission Laboratory Quarterly Progress and Status Report (KTH)* 2, pp. 18–24 (cit. on p. 7).

Farner, Snorre, Axel Roebel, and Xavier Rodet (2009). 'Natural transformation of type and nature of the voice for extending vocal repertoire in high-fidelity applications'. In: *Audio Engineering Society Conference: 35th International Conference: Audio for Games*. Audio Engineering Society (cit. on pp. 57, 93).

Ferro, Rafael, Nicolas Obin, and Axel Roebel (2020). 'CycleGAN Voice Conversion of Spectral Envelopes using Adversarial Weights'. In: *European Signal Processing Conference (EUSIPCO)*. IEEE (cit. on pp. 60, 92).

Flanagan, James L and Roger M Golden (1966). 'Phase vocoder'. In: *Bell system technical Journal* 45.9, pp. 1493–1509 (cit. on pp. 8, 92).

Fletcher, Harvey and Wilden A Munson (1933). 'Loudness, its definition, measurement and calculation'. In: *Bell System Technical Journal* 12.4, pp. 377–430 (cit. on p. 14).

Ford Baldner, Elizabeth, Emerald Doll, and Miriam Ruth van Mersbergen (2015). 'A Review of Measures of Vocal Effort With a Preliminary Study on the Establishment of a Vocal Effort Measure'. In: *Journal of Voice* 29.5, pp. 530–541. ISSN: 0892-1997. doi: https://doi.org/10.1016/j.jvoice.2014.08.017 (cit. on p. 113).

Gatys, Leon A, Alexander S Ecker, and Matthias Bethge (2016). 'Image style transfer using convolutional neural networks'. In: *Conference on Computer Vision and Pattern Recognition (CVPR)*. IEEE (cit. on pp. 27, 28).

Gauvain, J. L., L. F. Lamel, and M. Eskenazi (1990). 'Design considerations and text selection for BREF, a large French read-speech corpus'. In: *ICSLP*, pp. 1097–1100 (cit. on pp. 32, 65).

Gentilucci, Marta, Luc Ardaillon, and Marco Liuni (2018). 'Vocal distortion and real-time processing of roughness'. In: *International Computer Music Conference (ICMC)* (cit. on p. 58).

- (2019). 'Composing vocal distortion: A tool for real-time generation of roughness'. In: *Computer Music Journal* 42.4, pp. 26–40 (cit. on p. 58).

Glasberg, Brian R and Brian CJ Moore (1990). 'Derivation of auditory filter shapes from notched-noise data'. In: *Hearing research* 47.1-2, pp. 103–138 (cit. on p. 14).

- (2002). 'A model of loudness applicable to time-varying sounds'. In: *Journal of the Audio Engineering Society* 50.5, pp. 331–342 (cit. on pp. 14, 15, 121, 123).
Goodfellow, Ian et al. (2020). 'Generative adversarial networks'. In: *Communications of the ACM* 63.11, pp. 139–144 (cit. on p. 28).

Goupil, Louise et al. (2021). ‘Listeners’ perceptions of the certainty and honesty of a speaker are associated with a common prosodic signature’. In: *Nature communications* 12.1, p. 861 (cit. on p. 61).

Goyal, Mohit, Varun Srivastava, and Prathosh A. P. (2019). ‘Detection of Glottal Closure Instants from Raw Speech Using Convolutional Neural Networks’. In: *20th Annual Conference of the International Speech Communication Association (INTERSPEECH)*. Ed. by Gernot Kubin and Zdravko Kacic. ISCA, pp. 1591–1595. doi: 10.21437/Interspeech.2019-2587 (cit. on p. 49).

Grammalidis, Nikolaos et al. (2016). ‘The i-treasures intangible cultural heritage dataset’. In: *3rd International Symposium on Movement and Computing (MOCO)* (cit. on p. 34).

He, Zhenliang et al. (2019). ‘Attgan: Facial attribute editing by only changing what you want’. In: *Transactions on Image Processing* 28.11, pp. 5464–5478 (cit. on p. 28).

Henrich, Nathalie et al. (2005). ‘Glottal open quotient in singing: Measurements and correlation with laryngeal mechanisms, vocal intensity, and fundamental frequency’. In: *The Journal of the Acoustical Society of America* 117.3, pp. 1417–1430 (cit. on pp. 8, 116).

Hermes, Dik J (1988). ‘Measurement of pitch by subharmonic summation’. In: *The journal of the acoustical society of America* 83.1, pp. 257–264 (cit. on p. 48).

Higgins, Irina, David Amos, et al. (2018). ‘Towards a definition of disentangled representations’. In: *arXiv preprint arXiv:1812.02230* (cit. on pp. 58, 93).

Higgins, Irina, Loic Matthey, et al. (2017). ‘beta-vae: Learning basic visual concepts with a constrained variational framework’. In: *International conference on learning representations* (cit. on p. 59).

Holmberg, Eva B, Robert E Hillman, and Joseph S Perkell (1988). ‘Glottal airflow and transglottal air pressure measurements for male and female speakers in soft, normal, and loud voice’. In: *The Journal of the Acoustical Society of America* 84.2, pp. 511–529 (cit. on pp. 8, 116).

Hornik, Kurt, Maxwell Stinchcombe, and Halbert White (1989). ‘Multilayer feedforward networks are universal approximators’. In: *Neural networks* 2.5, pp. 359–366 (cit. on p. 21).

Huber, Stefan (2015). ‘Voice Conversion by modelling and transformation of extended voice characteristics’. PhD thesis. Université Pierre et Marie Curie-Paris VI (cit. on pp. 51, 58).

Huber, Stefan and Axel Roebel (2015). ‘On glottal source shape parameter transformation using a novel deterministic and stochastic speech analysis and synthesis system’. In: *16th Annual Conference of the International Speech Communication Association (INTERSPEECH)*. ISCA (cit. on pp. 8, 47, 53, 65, 92).

Huber, Stefan, Axel Roebel, and Gilles Degottex (2012). ‘Glottal Source Shape Parameter Estimation Using Phase Minimization Variants’. In: *Thirteenth Annual Conference of the International Speech Communication Association* (cit. on pp. 49, 65).

Hunt, Andrew J and Alan W Black (1996). ‘Unit selection in a concatenative speech synthesis system using a large speech database’. In: *Acoustics, Speech, and Signal*
*Processing*, 1996. ICASSP-96. *Conference Proceedings.*, 1996 IEEE International Conference on. Vol. 1. IEEE, pp. 373–376 (cit. on pp. 8, 55).

Hwang, Min-Jae et al. (2020). ‘LP-WaveNet: Linear prediction-based WaveNet speech synthesis’. In: *Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIGA ASC)*. IEEE, pp. 810–814 (cit. on p. 53).

Isshiki, Nobuhiko (1964). 'Regulatory mechanism of voice intensity variation'. In: *Journal of speech and hearing research* 7.1, pp. 17–29 (cit. on pp. 7, 115).

Ito, Keith and Linda Johnson (2017). *The LJ Speech Dataset*. https://keithito.com/LJ-Speech-Dataset/ (cit. on p. 32).

Jang, Won, Dan Lim, and Jaesam Yoon (2020). ‘Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains’. In: *arXiv preprint arXiv:2011.09631* (cit. on pp. 28, 54, 83, 88, 89).

Jenkins, James S (1998). ‘The voice of the castrato’. In: *The Lancet* 351.9119, pp. 1877–1880 (cit. on p. 130).

Jiao, Yunlong et al. (2021). ‘Universal neural vocoding with parallel wavenet’. In: *ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 6044–6048 (cit. on p. 83).

Juvela, Lauri et al. (2019). ‘GELP: GAN-Excited Liner Prediction for Speech Synthesis from Mel-Spectrogram’. In: *20th Annual Conference of the International Speech Communication Association (INTERSPEECH)*. International Speech Communication Association, pp. 694–698 (cit. on p. 55).

Kameoka, Hirokazu et al. (2018). ‘Stargan-vc: Non-parallel many-to-many voice conversion using star generative adversarial networks’. In: *Spoken Language Technology Workshop (SLT)*. IEEE (cit. on pp. 60, 92).

Kaneko, Takuhiro and Hirokazu Kameoka (2018). ‘Cyclegan-vc: Non-parallel voice conversion using cycle-consistent adversarial networks’. In: *26th European Signal Processing Conference (EUSIPCO)*. IEEE (cit. on p. 28).

Kaneko, Takuhiro, Hirokazu Kameoka, et al. (2019). ‘StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversion’. In: *20th Annual Conference of the International Speech Communication Association (INTERSPEECH)*. ISCA (cit. on p. 28).

Kaneko, Takuhiro, Kou Tanaka, et al. (2022). ‘iSTFTNet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time Fourier transform’. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 6207–6211 (cit. on pp. 26, 55).

Kawahara, Hideki (2006). ‘STRAIGHT, exploitation of the other aspect of VOCODER: Perceptually isomorphic decomposition of speech sounds’. In: *Acoustical science and technology* 27.6, pp. 349–353 (cit. on pp. 8, 51, 53, 63, 92).

Kawahara, Hideki et al. (2008). ‘TANDEM-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum, F0, and aperiodicity estimation’. In: *2008 IEEE International Conference on Acoustics, Speech and Signal Processing*. IEEE, pp. 3933–3936 (cit. on p. 53).

Kelly Jr, John L. and Carol C. Lochbaum (1962). ‘Speech synthesis’. In: *Proceedings of the fourth international congress on acoustics*. Vol. 1. G42, pp. 1–4 (cit. on p. 8).
Kempelen, Wolfgang von (1791). *Mechanismus der menschlichen Sprache*. Degen (cit. on p. 6).

Kim, Hyunjik and Andriy Mnih (2018). 'Disentangling by factorising'. In: *International Conference on Machine Learning*. PMLR, pp. 2649–2658 (cit. on p. 59).

Kim, Jaehyeon, Jungil Kong, and Juhee Son (2021). ‘Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech’. In: *International Conference on Machine Learning (ICML)*. PMLR, pp. 5530–5540 (cit. on pp. 9, 57, 78).

Kim, Jong Wook et al. (2018). ‘Crepe: A convolutional representation for pitch estimation’. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 48, 64).

Kim, Minyoung et al. (2019). ‘Relevance factor vae: Learning and identifying disentangled factors’. In: *arXiv preprint arXiv:1902.01568* (cit. on p. 59).

Kingma, Diederik P and Jimmy Ba (2014). ‘Adam: A method for stochastic optimization’. In: *3rd International Conference on Learning Representations (ICLR)* (cit. on pp. 98, 122).

Kingma, Diederik P and Prafulla Dhariwal (2018). ‘Glow: Generative flow with invertible 1x1 convolutions’. In: *Advances in neural information processing systems* 31 (cit. on p. 29).

Kingma, Diederik P and Max Welling (2013). ‘Auto-encoding variational bayes’. In: *arXiv preprint arXiv:1312.6114* (cit. on p. 29).

Klatt, Dennis H and Laura C Klatt (1990). ‘Analysis, synthesis, and perception of voice quality variations among female and male talkers’. In: *the Journal of the Acoustical Society of America* 87.2, pp. 820–857 (cit. on pp. 8, 18).

Kobayashi, Sachie (2022). *Day 0 – Trans-instrumentalism* (cit. on pp. 129, 146).

Koguchi, Junya, Shinnosuke Takamichi, and Masanori Morise (2020). ‘PJS: phoneme-balanced Japanese singing-voice corpus’. In: *Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APISPA ASC)*. IEEE (cit. on p. 34).

Komelek, John and Alan W Black (2004). ‘The CMU Arctic speech databases’. In: *Fifth ISCA workshop on speech synthesis* (cit. on pp. 32, 69).

Kong, Jungil, Jaehyeon Kim, and Jaekyoung Bae (2020). ‘Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis’. In: *Advances in Neural Information Processing Systems* 33, pp. 17022–17033 (cit. on p. 55).

Kumar, Abhishek, Prasanna Sattigeri, and Avinash Balakrishnan (2018). ‘Variational Inference of Disentangled Latent Concepts from Unlabeled Observations’. In: *International Conference on Learning Representations*. url: https://openreview.net/forum?id=H1kG7GZAW (cit. on p. 59).

Kumar, Kundan et al. (2019). ‘Melgan: Generative adversarial networks for conditional waveform synthesis’. In: *Advances in neural information processing systems* 32 (cit. on pp. 28, 54, 73, 83, 84, 167).

Ladefoged, Peter and Norris P McKinney (1963). ‘Loudness, sound pressure, and subglottal pressure in speech’. In: *The journal of the Acoustical Society of America* 35.4, pp. 454–460 (cit. on pp. 7, 115).

Lample, Guillaume et al. (2017). ‘Fader networks: manipulating images by sliding attributes’. In: *International Conference on Neural Information Processing Systems (NeurIPS)* (cit. on pp. 29, 58, 111).
Lange, Sascha and Martin Riedmiller (2010). 'Deep auto-encoder neural networks in reinforcement learning'. In: *The 2010 International Joint Conference on Neural Networks (IJCNN)*. IEEE, pp. 1-8 (cit. on p. 29).

Laroche, Jean, Yannis Stylianou, and Eric Moulines (1993). 'HNM: A simple, efficient harmonic+ noise model for speech'. In: *Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics*. IEEE, pp. 169-172 (cit. on pp. 8, 50, 92).

Larsson, Bjorn (1977). 'Music and singing synthesis equipment (MUSSE)'. In: *Speech Transmission Laboratory Quarterly Progress and Status Report (STL-QPSR)* 1.1977, pp. 38-40 (cit. on p. 7).

LeCun, Yann et al. (1989). 'Backpropagation applied to handwritten zip code recognition'. In: *Neural computation* 1.4, pp. 541-551 (cit. on pp. 23, 24, 164).

Lee, Sang-gil et al. (2023). 'Bigvgan: A universal neural vocoder with large-scale training'. In: *12th International Conference on Learning Representations (ICLR)* (cit. on pp. 55, 82, 83).

Li, Chuan and Michael Wand (2016). 'Precomputed real-time texture synthesis with markovian generative adversarial networks'. In: *European conference on computer vision*. Springer, pp. 702-716 (cit. on p. 28).

Li, Yinghao Aaron, Cong Han, and Nima Mesgarani (2023). 'Styletts-VC: One-Shot Voice Conversion by Knowledge Transfer From Style-Based TTS Models'. In: *2022 IEEE Spoken Language Technology Workshop (SLT)*. IEEE, pp. 920-927 (cit. on p. 60).

Liljencrants, Johan (1968). 'The OVE III speech synthesizer'. In: *IEEE Transactions on Audio and Electroacoustics* 16.1, pp. 137-140 (cit. on p. 7).

Linggard, Robert (1985). *Electronic synthesis of speech*. CUP Archive (cit. on p. 6).

Liu, Xiao et al. (2022). 'Learning disentangled representations in the imaging domain'. In: *Medical Image Analysis*, p. 102516 (cit. on p. 58).

Liuni, Marco et al. (2020). 'ANGUS: Real-time manipulation of vocal roughness for emotional speech transformations'. In: *arXiv preprint arXiv:2008.11241* (cit. on p. 58).

Long, Jonathan, Evan Shelhamer, and Trevor Darrell (2015). 'Fully convolutional networks for semantic segmentation'. In: *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 3431-3440 (cit. on p. 23).

Lorenzo-Trueba, Jaime et al. (2018). 'Robust universal neural vocoding'. In: *arXiv preprint arXiv:1811.06292* (cit. on pp. 54, 82, 83).

Lu, Peiling et al. (2020). 'Xiaoicesing: A high-quality and integrated singing voice synthesis system'. In: *21st Annual Conference of the International Speech Communication Association (INTERSPEECH)* (cit. on p. 9).

Luo, Zhaojie, Jinhui Chen, et al. (2017). 'Emotional voice conversion using neural networks with arbitrary scales F0 based on wavelet transform'. In: *EURASIP Journal on Audio, Speech, and Music Processing* 2017, pp. 1-13 (cit. on p. 61).

Luo, Zhaojie, Tetsuya Takiguchi, and Yasuo Ariki (2016). 'Emotional voice conversion using deep neural networks with MCC and F0 features'. In: *15th International Conference on Computer and Information Science (ICIS)*. IEEE/ACIS, pp. 1-5 (cit. on p. 61).

Markel, John D and AH Jr Gray (2013). *Linear prediction of speech*. Vol. 12. Springer Science & Business Media (cit. on pp. 47, 63).
Mauch, Matthias and Simon Dixon (2014). 'pYIN: A fundamental frequency estimator using probabilistic threshold distributions'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on p. 48).

McCandless, Stephanie (1974). 'An algorithm for automatic formant extraction using linear prediction spectra'. In: *IEEE Transactions on Acoustics, Speech, and Signal Processing* 22.2, pp. 135-141 (cit. on p. 50).

McKenna, Victoria S and Cara E Stepp (2018). 'The relationship between acoustical and perceptual measures of vocal effort'. In: *The Journal of the Acoustical Society of America* 144.3, pp. 1643-1658 (cit. on p. 113).

Mehta, Daryush and Thomas F Quatieri (2005). 'Synthesis, analysis, and pitch modification of the breathy vowel'. In: *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2005*. IEEE, pp. 199-202 (cit. on p. 51).

Moine, Clément Le (2023). 'Neural conversion of social attitudes in speech signals'. PhD thesis. Sorbonne Université (cit. on p. 149).

Moine, Clément Le and Nicolas Obin (2020). 'Att-HACK: An Expressive Speech Database with Social Attitudes'. In: *Speech Prosody* (cit. on pp. 31, 32, 98, 99).

Moine, Clément Le, Nicolas Obin, and Axel Roebel (2021). 'Towards end-to-end F0 voice conversion based on Dual-GAN with convolutional wavelet kernels'. In: *European Signal Processing Conference (EUSIPCO)*. IEEE (cit. on pp. 61, 92).

Molina, Emilio et al. (2014). 'Parametric model of spectral envelope to synthesize realistic intensity variations in singing voice'. In: *2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 634-638 (cit. on p. 58).

Moore, Brian CJ (2012). *An introduction to the psychology of hearing*. Brill (cit. on pp. 9-11, 14).

Morgan, Nelson et al. (2005). 'Pushing the envelope-aside [speech recognition]'. In: *IEEE Signal Processing Magazine* 22.5, pp. 81-88 (cit. on p. 50).

Morise, Masanori (2015). 'CheapTrick, a spectral envelope estimator for high-quality speech synthesis'. In: *Speech Communication* 67, pp. 1-7 (cit. on p. 50).

Morise, Masanori, Fumiya Yokomori, and Kenji Ozawa (2016). 'WORLD: a vocoder-based high-quality speech synthesis system for real-time applications'. In: *IEICE TRANSACTIONS on Information and Systems* 99.7, pp. 1877-1884 (cit. on pp. 8, 53, 63, 92).

Moulines, Eric and Francis Charpentier (1990). 'Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones'. In: *Speech communication* 9.5-6, pp. 453-467 (cit. on pp. 8, 50-52, 55, 92).

Murthy, Hema A and Bayya Yegnanarayana (1991). 'Formant extraction from group delay function'. In: *speech communication* 10.3, pp. 209-221 (cit. on p. 50).

Murty, K. S. R. and Bayya Yegnanarayana (2008). 'Epoch Extraction From Speech Signals'. In: *IEEE Transactions on Audio, Speech, and Language Processing* 16.8, pp. 1602-1613 (cit. on p. 49).

Nagrani, Arsha et al. (2019). 'Voxceleb: Large-scale speaker verification in the wild'. In: *Computer Science and Language* (cit. on p. 31).

Nair, Vinod and Geoffrey E Hinton (2010). 'Rectified linear units improve restricted boltzmann machines'. In: *International Conference on Machine Learning (ICML)*. PMLR (cit. on pp. 98, 168).
Naylor, Patrick A et al. (2007). 'Estimation of glottal closure instants in voiced speech using the DYPSA algorithm'. In: *IEEE Transactions on Audio, Speech and Language Processing* 15.1, pp. 34–43. issn: 15587916. doi: 10.1109/TASL.2006.876878 (cit. on p. 49).

Nguyen, Bac and Fabien Cardinaux (2022). 'Nvc-net: End-to-end adversarial voice conversion'. In: *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 7012–7016 (cit. on pp. 60, 78).

Nitzan, Yotam et al. (2020). ‘Face identity disentanglement via latent space mapping’. In: *ACM Transactions on Graphics* (TOG) 39.6, pp. 1–14 (cit. on p. 58).

Noll, A Michael (1967). 'Cepstrum pitch determination'. In: *The journal of the acoustical society of America* 41.2, pp. 293–309 (cit. on p. 48).

Odena, Augustus, Christopher Olah, and Jonathon Shlens (2017). ‘Conditional image synthesis with auxiliary classifier gans’. In: *International Conference on Machine Learning (ICML)*. PMLR (cit. on p. 28).

Ogawa, Itsuki and Masanori Morise (2021). ‘Tohoku Kiritan singing database: A singing database for statistical parametric singing synthesis using Japanese pop songs’. In: *Acoustical Science and Technology* 42.3, pp. 140–145 (cit. on p. 34).

Oh, Suhyeon et al. (2020). ‘ExcitGlow: Improving a WaveGlow-based neural vocoder with linear prediction analysis’. In: *Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIDA ASC)*. IEEE, pp. 831–836 (cit. on pp. 53, 54).

Olah, Chris, Alexander Mordvintsev, and Ludwig Schubert (2017). ‘Feature Visualization’. In: *Distill*. https://distill.pub/2017/feature-visualization. doi: 10.23915/distill.00007 (cit. on p. 27).

Oord, Aaron van den, Sander Dieleman, et al. (2016). ‘Wavenet: A generative model for raw audio’. In: *arXiv preprint arXiv:1609.03499* (cit. on pp. 25, 53, 54, 66, 75, 77, 86, 159, 169).

Oord, Aaron van den, Yazhe Li, et al. (2017). ‘Parallel WaveNet: Fast high-fidelity speech synthesis’. In: *arXiv preprint arXiv:1711.10433* (cit. on pp. 54, 66, 83).

Panayotov, Vassil et al. (2015). ‘Librispeech: an asr corpus based on public domain audio books’. In: *2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)*. IEEE, pp. 5206–5210 (cit. on p. 31).

Patel, Aniruddh D and Evan Balaban (2001). ‘Human pitch perception is reflected in the timing of stimulus-related cortical activity’. In: *Nature neuroscience* 4.8, pp. 839–844 (cit. on p. 11).

Peng, Xi et al. (2017). ‘Reconstruction-based disentanglement for pose-invariant face recognition’. In: *Proceedings of the IEEE international conference on computer vision*, pp. 1623–1632 (cit. on p. 58).

Perrotin, Olivier and Christophe d’ Alessandro (2016). ‘Vocal effort modification for singing synthesis’. In: *17th Annual Conference of the International Speech Communica- tion Association (INTERSPEECH 2016)*, pp. 1235–1239 (cit. on p. 58).

Ping, Wei et al. (2018). ‘Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning.’ In: *7th International Conference on Learning Representations (ICLR)* (cit. on pp. 56, 80, 82, 83).
Prathosh, A. P., T. V. Ananthapadmanabha, and A. G. Ramakrishnan (2013). 'Epoch Extraction Based on Integrated Linear Prediction Residual Using Plosion Index'. In: *IEEE Transactions on Audio, Speech, and Language Processing* 21.12, pp. 2471-2480 (cit. on p. 49).

Prathosh, A. P., Varun Srivastava, and Mayank Mishra (2019). 'Adversarial approximate inference for speech to electroglottograph conversion'. In: *IEEE/ACM Transactions on Audio, Speech, and Language Processing* 27.12, pp. 2183-2196 (cit. on p. 49).

Prenger, Ryan, Rafael Valle, and Bryan Catanzaro (2019). 'Waveglow: A flow-based generative network for speech synthesis'. In: *ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 3617-3621 (cit. on pp. 25, 26, 29, 54, 83, 84, 169).

Proszewska, Magdalena et al. (2022). 'GlowVC: Mel-spectrogram space disentangling model for language-independent text-free voice conversion'. In: *23rd Annual Conference of the International Speech Communication Association (INTERSPEECH)*. doi: 10.21437/Interspeech.2022-322 (cit. on p. 60).

Puts, David Andrew et al. (2007). 'Men's voices as dominance signals: vocal fundamental and formant frequencies influence dominance attributions among men'. In: *Evolution and Human Behavior* 28.5, pp. 340-344 (cit. on p. 61).

Qian, Kaizhi, Zeyu Jin, et al. (2020). 'F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 4, 59, 92, 111, 113, 120, 154).

Qian, Kaizhi, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, et al. (2020). 'Unsupervised speech decomposition via triple information bottleneck'. In: *International Conference on Machine Learning (ICML)*. PMLR (cit. on pp. 59, 60, 95).

Qian, Kaizhi, Yang Zhang, Shiyu Chang, Xuesong Yang, et al. (2019). 'Autocvc: Zero-shot voice style transfer with only autoencoder loss'. In: *International Conference on Machine Learning (ICML)*. PMLR (cit. on pp. 4, 29, 59, 60, 92, 98, 113, 163).

Quatieri, Thomas F and Robert J McAulay (1992). 'Shape invariant time-scale and pitch modification of speech'. In: *IEEE Transactions on Signal Processing* 40.3, pp. 497-510 (cit. on pp. 8, 52, 92).

Rabiner, Lawrence (1977). 'On the use of autocorrelation analysis for pitch detection'. In: *IEEE transactions on acoustics, speech, and signal processing* 25.1, pp. 24-33 (cit. on p. 48).

Rabiner, Lawrence and Biing-Hwang Juang (1993). *Fundamentals of speech recognition*. Prentice-Hall, Inc. (cit. on p. 12).

Rafii, Zafar et al. (Dec. 2017). *The MUSDB18 corpus for music separation*. doi: 10.5281/zenodo.1117372 (cit. on p. 89).

Raitio, Tuomo, Ramya Rasipuram, and Dan Castellani (2020). 'Controllable neural text-to-speech synthesis using intuitive prosodic features'. In: *21st Annual Conference of the International Speech Communication Association (INTERSPEECH)* (cit. on p. 58).

Reddy, M. G., T. Mandal, and K. S. Rao (2018). 'Glottal closure instants detection from pathological acoustic speech signal using deep learning'. In: *International Conference on Neural Information Processing Systems (NeurIPS)*. *Machine Learning for Health* (cit. on p. 49).
Ren, Yi et al. (2021). 'FastSpeech 2: Fast and High-Quality End-to-End Text to Speech'. In: *International Conference on Learning Representations*. url: https://openreview.net/forum?id=piLPYqxtWuA (cit. on pp. 8, 57, 78).

Richard, Gaël and Christophe d' Alessandro (1996). 'Analysis/Synthesis and Modification of the Speech Aperiodic Components'. In: *Speech Communication* 19.3, pp. 221–244 (cit. on pp. 16, 17).

Ridgeway, Karl and Michael C Mozer (2018). 'Learning deep disentangled embeddings with the f-statistic loss'. In: *Advances in neural information processing systems* 31 (cit. on p. 59).

Robinson, Carl, Nicolas Obin, and Axel Roebel (2019). 'Sequence-to-sequence modeling of f0 for speech emotion conversion'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 61, 92).

Robinson, Derek W and R So Dadson (1956). 'A re-determination of the equal-loudness relations for pure tones'. In: *British Journal of Applied Physics* 7.5, p. 166 (cit. on p. 14).

Rodet, Xavier, Yves Potard, and Jean-Baptiste Barriere (1984). 'The CHANT project: From the synthesis of the singing voice to synthesis in general'. In: *Computer Music Journal* 8.3, pp. 15–31 (cit. on p. 8).

Roebel, Axel (2010). 'A shape-invariant phase vocoder for speech transformation'. In: *13th International Conference on Digital Audio Effects (DAFx)* (cit. on pp. 8, 53, 92).

Roebel, Axel and Frederik Bous (2022). 'Neural Vocoding for Singing and Speaking Voices with the Multi-band Excited WaveNet'. In: *Information* 13.3, p. 103 (cit. on pp. 28, 48, 55, 83, 85, 86, 88, 89).

Roebel, Axel and Xavier Rodet (2005). 'Efficient spectral envelope estimation and its application to pitch shifting and envelope preservation'. In: *International Conference on Digital Audio Effects*, pp. 30–35 (cit. on pp. 50, 169).

Ronneberger, Olaf, Philipp Fischer, and Thomas Brox (2015). 'U-net: Convolutional networks for biomedical image segmentation'. In: *Medical Image Computing and Computer-Assisted Intervention (MICCAI)*. Springer, pp. 234–241 (cit. on p. 23).

Rosenberg, Aaron E (1971). 'Effect of glottal pulse shape on the quality of natural vowels'. In: *The Journal of the Acoustical Society of America* 49.2B, pp. 583–590 (cit. on pp. 8, 18).

Ruggero, Mario A (1992). 'Responses to sound of the basilar membrane of the mammalian cochlea'. In: *Current opinion in neurobiology* 2.4, pp. 449–456 (cit. on p. 10).

Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams (1986). 'Learning representations by back-propagating errors'. In: *nature* 323.6088, pp. 533–536 (cit. on p. 23).

Sagisaka, Yoshinori et al. (1992). 'ATR μ-talk speech synthesis system.' In: *Second International Conference on Spoken Language Processing*. Vol. 92, pp. 483–486 (cit. on pp. 8, 55).

Salais, Léane et al. (2022). 'Production Strategies of Vocal Attitudes'. In: *23rd Annual Conference of the International Speech Communication Association (INTERSPEECH)*. ISCA, pp. 4985–4989 (cit. on pp. 61, 92).

Schafer, Ronald W and Lawrence Rabiner (1975). 'Digital representations of speech signals'. In: *Proceedings of the IEEE* 63.4, pp. 662–677 (cit. on p. 50).
Schmetterer, Leopold (2012). *Introduction to mathematical statistics*. Vol. 202. Springer Science & Business Media (cit. on p. 45).

Schroeder, Manfred R (1968). 'Period histogram and product spectrum: New methods for fundamental-frequency measurement'. In: *The Journal of the Acoustical Society of America* 43.4, pp. 829-834 (cit. on p. 48).

Sepliarskaia, Anna, Julia Kiseleva, and Maarten de Rijke (2019). 'How to not measure disentanglement'. In: *arXiv preprint arXiv:1910.05587* (cit. on p. 59).

Shen, Jonathan et al. (2018). ‘Natural tts synthesis by conditioning wavenet on mel spectrogram predictions’. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 12, 25, 54, 56, 57, 66, 77, 80–83, 169).

Shen, Kai et al. (2023). ‘NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers’. In: *arXiv preprint arXiv:2304.09116* (cit. on p. 9).

Shepard, Roger N (1964). ‘Circularity in judgments of relative pitch’. In: *The journal of the acoustical society of America* 36.12, pp. 2346–2353 (cit. on p. 12).

Shirazi, Aïda (2022). *Né entre corps* (cit. on p. 129).

Singh, Satwinder, Ruili Wang, and Yuanhang Qiu (2021). ‘DeepF0: End-to-end fundamental frequency estimation for music and speech signals’. In: *ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 61–65 (cit. on p. 48).

Sondhi, Mohan (1968). ‘New methods of pitch extraction’. In: *IEEE Transactions on audio and electroacoustics* 16.2, pp. 262–266 (cit. on p. 48).

Song, Eunwoo, Kyungguen Byun, and Hong-Goo Kang (2019). ‘ExcitNet vocoder: A neural excitation model for parametric speech synthesis systems’. In: *27th European Signal Processing Conference (EUSIPCO)*. IEEE, pp. 1–5 (cit. on p. 53).

Stevens, Stanley Smith, John Volkmann, and Edwin Broomell Newman (1937). ‘A scale for the measurement of the psychological magnitude pitch’. In: *The journal of the acoustical society of america* 8.3, pp. 185–190 (cit. on pp. 12, 15).

Stewart, John Q (1922). ‘An electrical analogue of the vocal organs’. In: *Nature* 110.2757, p. 311 (cit. on pp. 6, 7, 18, 46).

Stylianou, Yannis, Jean Laroche, and Eric Moulines (1995). ‘High-quality speech modification based on a harmonic+ noise model’. In: *Fourth European Conference on Speech Communication and Technology* (cit. on pp. 8, 47, 50–52, 92).

Sun, Xuejing (2000). ‘A pitch determination algorithm based on subharmonic-to-harmonic ratio’. In: *2nd Annual Conference of the International Speech Communication Association (INTERSPEECH)*. ISCA (cit. on p. 48).

Sundberg, Johan (2006). ‘The KTH synthesis of singing’. In: *Advances in cognitive Psychology* 2.2, p. 131 (cit. on pp. 7, 8).

Sundberg, Johan, Ingo Titze, and Ronald Scherer (1993). ‘Phonatory control in male singing: A study of the effects of subglottal pressure, fundamental frequency, and mode of phonation on the voice source’. In: *Journal of Voice* 7.1, pp. 15–29 (cit. on pp. 8, 114, 115).

Sutskever, Ilya, Oriol Vinyals, and Quoc V Le (2014). ‘Sequence to sequence learning with neural networks’. In: *Advances in neural information processing systems* 27 (cit. on p. 24).
Tamaru, Hiroki et al. (2020). ‘JVS-MuSiC: Japanese multispeaker singing-voice corpus’. In: *arXiv preprint arXiv:2001.07044* (cit. on p. 34).

Tandler, Julius and Siegfried Grosz (1909). ‘Über den Einfluß der Kastration auf den Organismus’. In: *Archiv für Entwicklungsmechanik der Organismen* 27.1, pp. 35–61 (cit. on p. 130).

Thomas, Mark RP and Patrick A Naylor (2009). ‘The SIGMA algorithm: A glottal activity detector for electroglottographic signals’. In: *IEEE Transactions on Audio, Speech, and Language Processing* 17.8, pp. 1557–1566 (cit. on p. 69).

Titze, Ingo R and Johan Sundberg (1992). ‘Vocal intensity in speakers and singers’. In: *the Journal of the Acoustical Society of America* 91.5, pp. 2936–2946 (cit. on pp. 8, 114–116).

Traunmüller, Hartmut and Anders Eriksson (2000). ‘Acoustic effects of variation in vocal effort by men, women, and children’. In: *The Journal of the Acoustical Society of America* 107.6, pp. 3438–3451 (cit. on pp. 8, 113, 115, 116, 120).

Tsirulnik, Liliya and Shlomo Dubnov (2019). ‘Singing Voice Database’. In: *International Conference on Speech and Computer (ICSC)*. Springer (cit. on pp. 33, 34).

Turk, Oytun et al. (2005). ‘Voice quality interpolation for emotional text-to-speech synthesis’. In: *6th Annual Conference of the International Speech Communication Association (INTERSPEECH)* (cit. on p. 58).

Umbert, Martı, Jordi Bonada, and Merlijn Blaauw (2013). ‘Generating singing voice expression contours based on unit selection’. In: *Stockholm Music Acoustics Conference (SMAC)* (cit. on pp. 8, 56, 92).

Umbert, Martı, Jordi Bonada, Masataka Goto, et al. (2015). ‘Expression control in singing voice synthesis: Features, approaches, evaluation, and challenges’. In: *IEEE Signal Processing Magazine* 32.6, pp. 55–73 (cit. on p. 92).

Vaswani, Ashish et al. (2017). ‘Attention is all you need’. In: *Advances in neural information processing systems* 30 (cit. on p. 24).

Veaux, Christophe and Xavier Rodet (2011). ‘Intonation conversion from neutral to expressive speech’. In: *Twelfth Annual Conference of the International Speech Communication Association (INTERSPEECH)*. ISCA (cit. on pp. 61, 92).

Veldhuis, Raymond (1998). ‘A computationally efficient alternative for the liljencrants–fant model and its perceptual evaluation’. In: *The Journal of the Acoustical Society of America* 103.1, pp. 566–571 (cit. on pp. 8, 18).

Wang, Chengyi et al. (2023). ‘Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers’. In: *arXiv preprint arXiv:2301.02111* (cit. on p. 9).

Wang, Xin, Shinji Takaki, and Junichi Yamagishi (2019). ‘Neural source-filter wave-form models for statistical parametric speech synthesis’. In: *IEEE/ACM Transactions on Audio, Speech, and Language Processing* 28, pp. 402–415 (cit. on pp. 27, 54, 66, 67, 87).

Wang, Yuxuan et al. (2017). ‘Tacotron: Towards End-to-End Speech Synthesis’. In: *18th Annual Conference of the International Speech Communication Association (INTERSPEECH)*, pp. 4006–4010. doi: 10.21437/Interspeech.2017-1452 (cit. on pp. 54, 56, 169).
Wang, Zhichao et al. (2021). ‘Accent and speaker disentanglement in many-to-many voice conversion’. In: *12th International Symposium on Chinese Spoken Language Processing (ISCSLP)*. IEEE, pp. 1–5 (cit. on p. 59).

Werbos, Paul J (1990). ‘Backpropagation through time: what it does and how to do it’. In: *Proceedings of the IEEE* 78.10, pp. 1550–1560 (cit. on p. 24).

Wilkins, Julia et al. (2018). ‘VocalSet: A Singing Voice Dataset’. In: *International Society for Music Information Retrieval Conference (ISMIR)*. ISMIR (cit. on p. 34).

Williams, Ronald J and David Zipser (1995). ‘Gradient-based learning algorithms for recurrent networks and their computational complexity’. In: *Backpropagation: theory, architectures, and applications*, pp. 433–486 (cit. on p. 24).

Wu, Jie and Jian Luan (2020). ‘Adversarially trained multi-singer sequence-to-sequence singing synthesizer’. In: *21st Annual Conference of the International Speech Communication Association (INTERSPEECH)* (cit. on p. 9).

Xie, Qicong et al. (2022). ‘End-to-End Voice Conversion with Information Perturbation’. In: *arXiv preprint arXiv:2206.07569* (cit. on pp. 60, 78).

Yamagishi, Junichi, Christophe Veaux, and Kirsten MacDonald (2019). *CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92)*. doi: 10.7488/ds/2645 (cit. on pp. 32, 98, 99).

Yang, Geng et al. (2021). ‘Multi-band MelGAN: Faster waveform generation for high-quality text-to-speech’. In: *Spoken Language Technology Workshop (SLT)*. IEEE, pp. 492–498 (cit. on pp. 54, 88, 89).

Yang, S. et al. (2018). ‘Detection of glottal closure instants from speech signals: A convolutional neural network based method’. In: *19th Annual Conference of the International Speech Communication Association (INTERSPEECH)*, pp. 317–321. doi: 10.21437/Interspeech.2018-1281 (cit. on p. 49).

Yoshimura, Takayoshi (2002). ‘Simultaneous modeling of phonetic and prosodic parameters, and characteristic conversion for HMM-based text-to-speech systems’. In: *PhD diss, Nagoya Institute of Technology* (cit. on pp. 8, 56).

Yu, Chengzhu et al. (2020). ‘DurIAN: Duration Informed Attention Network For Speech Synthesis’. In: (cit. on p. 26).

Zen, Heiga, Yannis Agiomyrgiannakis, et al. (2016). ‘Fast, compact, and high quality LSTM-RNN based statistical parametric speech synthesizers for mobile devices’. In: *17th Annual Conference of the International Speech Communication Association (INTERSPEECH)* (cit. on pp. 8, 56).

Zen, Heiga, Andrew Senior, and Mike Schuster (2013). ‘Statistical parametric speech synthesis using deep neural networks’. In: *international conference on acoustics, speech and signal processing (ICASSP)*. IEEE, pp. 7962–7966 (cit. on pp. 8, 56).

Zen, Heiga, Keiichi Tokuda, and Alan W Black (2009). ‘Statistical parametric speech synthesis’. In: *speech communication* 51.11, pp. 1039–1064 (cit. on pp. 8, 56).

Zhang, Jing-Xuan, Zhen-Hua Ling, and Li-Rong Dai (2019). ‘Non-parallel sequence-to-sequence voice conversion with disentangled linguistic and speaker representations’. In: *Transactions on Audio, Speech, and Language Processing* 28, pp. 540–552 (cit. on pp. 9, 59, 60, 80, 92).
Zhao, Guanlong et al. (2018). 'Accent conversion using phonetic posteriors'. In: *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE (cit. on pp. 60, 92).

Zhou, Kun et al. (2021). ‘Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset’. In: *ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, pp. 920–924 (cit. on p. 61).

- (2022). ‘Emotional voice conversion: Theory, databases and ESD’. In: *Speech Communication* 137, pp. 1–18 (cit. on p. 61).

Zhu, Jun-Yan et al. (2017). ‘Unpaired image-to-image translation using cycle-consistent adversarial networks’. In: *International Conference on Computer Vision (ICCV)*. IEEE (cit. on p. 28).

Zue, V., S. Seneff, and J. Glass (1990). ‘Speech database development at MIT: TIMIT and beyond’. In: *Speech communication* 9.4, pp. 351–356 (cit. on pp. 32, 65).