Year: 2021

Abstract:
Understanding and controlling latent representations in deep generative models is a challenging yet important problem for analyzing, transforming and generating various types of data. In speech processing, inspiring from the anatomical mechanisms of phonation, the source-filter model considers that speech signals are produced from a few independent and physically meaningful continuous latent factors, among which the fundamental frequency and the formants are of primary importance. In this work, we show that the source-filter model of speech production naturally arises in the latent space of a variational autoencoder (VAE) trained in an unsupervised fashion on a dataset of natural speech signals. Using only a few seconds of labeled speech signals generated with an artificial speech synthesizer, we experimentally demonstrate that the fundamental frequency and formant frequencies are encoded in orthogonal subspaces of the VAE latent space and we develop a weakly-supervised method to accurately and independently control these speech factors of variation within the learned latent subspaces. Without requiring additional information such as text or human-labeled data, we propose a deep generative model of speech spectrograms that is conditioned on the fundamental frequency and formant frequencies, and which is applied to the transformation of speech signals.

heading: 1 INTRODUCTION
text: High-dimensional data such as natural images or speech signals exhibit some form of regularity which prevents their dimensions from varying independently from each other. This suggests that there exists a latent representation of smaller dimension from which the high-dimensional observed data were generated. Discovering the hidden properties of complex data is the goal of representation learning, and deep latent-variable generative models have emerged as promising unsupervised approaches (Goodfellow et al., 2014; Kingma & Welling, 2014; Rezende et al., 2014; Chen et al., 2016; Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018). The variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014), which is equipped with both a generative and inference model, can be used not only for data generation but also for analysis and transformation. As an explicit model of a probability density function (pdf), the VAE can also be used as a learned prior for solving inverse problems such as compressed sensing (Bora et al., 2017), speech enhancement (Bando et al., 2018; Leglaive et al., 2018), or source separation (Kameoka et al., 2019; Jayaram & Thickstun, 2020). Making sense of the latent representation learned by a VAE and controlling the underlying continuous factors of variation in the data are important challenges to build more expressive and interpretable generative models and probabilistic priors.
Previous works on representation learning with deep generative models, in particular VAEs, have mostly focused on images (Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Locatello et al., 2019; 2020). Yet, it is not always easy to define the ground-truth latent factors of variation involved in the generation of natural images. For speech data, the latent factors of variation can be directly related to the anatomical mechanisms of speech production. This makes speech data interesting for investigating the disentangled representation learning capabilities of VAEs, complementary to studies dealing with images. A key concept for characterizing the structure of speech signals is deduced from the source-filter model proposed by Fant (1970). This model, described in
more detail in Section 2.2, implies that a speech signal is mainly characterized by a few continuous latent factors of variation corresponding to the vibration of the vocal folds (i.e., the source), which defines the fundamental frequency, and the resonances of the vocal tract (i.e., the filter), which define the formants. The source-filter model is at the core of various fundamental speech processing techniques such as cepstral representations and linear predictive coding (LPC) (Rabiner & Schafer, 2010). Valin & Skoglund (2019); Wang et al. (2019) and Juvela et al. (2019) have recently shown that the efficiency of neural speech vocoders can be largely improved by leveraging the sourcefilter model. Other works investigating the interaction between the source-filter model and neural networks include Lee et al. (2019) and Choi et al. (2021). All these studies illustrate the interest of combining deep learning techniques with more traditional signal processing models and algorithms. In this work, we interpret and control the latent space of a VAE from the perspective of the source-filter model of speech production, which can be beneficial for various applications in speech analysis, transformation, and synthesis.
We first train a VAE on a dataset of about 25 hours of unlabeled speech signals. Then, using only a few seconds of labeled speech signals generated with an artificial speech synthesizer, we propose a method to analyze and control the fundamental frequency and the formant frequencies in the latent representation of the previously trained VAE. Our contributions are the following: (i) We experimentally demonstrate that the fundamental frequency and the frequency of the first three formants are encoded in orthogonal subspaces of the VAE latent space. This shows that a vanilla VAE trained in an unsupervised fashion is able to learn a representation that is compliant with the source-filter model of speech production. (ii) We develop a weakly-supervised method to precisely and independently control the source-filter continuous latent factors of speech variation within the learned subspaces. We put in evidence the orthogonality of these subspaces, which allows us to perform speech transformations in a disentangled manner (i.e., modifying one of the factors does not affect the others). (iii) Without requiring additional information such as text or human-labeled data, we propose a deep generative model of speech spectrograms conditioned on the fundamental frequency and formant frequencies. To the best of our knowledge, this is the first study showing the link between the classical source-filter model of speech production and the representation learned in the latent space of a VAE. Thanks to this link, we propose a principled method to generate speech data controlled with interpretable trajectories (of e.g., fundamental frequency and formant frequencies).

heading: 2 BACKGROUND

heading: 2.1 VARIATIONAL AUTOENCODER
text: Generative modeling consists in learning a probabilistic model of an observable random variable x ∈ X ⊂ RD. Let D = {x1, ...,xN ∈ X} be a dataset of N = #D independent and identically distributed (i.i.d.) observations of x. The empirical distribution of x is defined by p̂(x) = 1N ∑ xn∈D δ(x − xn), where δ is the Dirac delta function, which is null everywhere except in 0 where it takes the value 1.
The variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) attempts to approximate p̂(x) with a pdf pθ(x) parametrized by θ. High-dimensional data such as natural images or speech signals exhibit some form of regularity which prevents theD dimensions of x from varying independently from each other. We can thus assume that there exists a latent variable z ∈ RL, with L D, from which the observed data were generated. Accordingly, the model distribution in the VAE is defined by marginalizing the joint distribution of the latent and observed data, pθ(x) =∫ pθ(x|z)p(z)dz.
In this work, the observed data vector x ∈ RD+ denotes the power spectrum of a short frame of speech signal (i.e., a column of the short-time Fourier transform (STFT) power spectrogram). Its entries are non negative and its dimension D equals the number of frequency bins. We use the Itakura-Saito VAE (IS-VAE) (Bando et al., 2018; Leglaive et al., 2018; Girin et al., 2019) defined by
p(z) = N (z;0, I), pθ(x|z) = ∏D
d=1 Exp
( [x]d; [vθ(z)] −1 d ) , (1)
whereN and Exp denote the densities of the multivariate Gaussian and univariate exponential distributions, respectively, and [v]d denotes the d-th entry of v. The inverse scale parameters of pθ(x|z) are provided by a neural network called the decoder, parametrized by θ and taking z as input.
The marginal likelihood pθ(x) and the posterior distribution pθ(z|x) are intractable due to the non linearities of the decoder, so it is necessary to introduce an inference model qφ(z|x) ≈ pθ(z|x), which in the VAE is usually defined by
qφ(z|x) = N (z;µφ(x),diag{vφ(x)}) , (2) where the mean and variance parameters are provided by a neural network called the encoder network, parametrized by φ and taking x as input. Then, the VAE training consists in maximizing a lower-bound of ln pθ(x), called the evidence lower-bound (ELBO) and defined by L(θ, φ) = Ep̂(x) [ Eqφ(z|x) [pθ(x|z)]−DKL (qφ(z|x) ‖ p(z)) ] . During training, the generative and inference model parameters θ and φ are jointly estimated by maximizing the ELBO, using (variants of) stochastic gradient descent with the so-called reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014).

heading: 2.2 SOURCE-FILTER MODEL OF SPEECH PRODUCTION
text: The source-filter model of speech production (Fant, 1970) is at the basis of many speech processing systems. It considers that the production of speech results from the interaction of a source signal with a linear filter. In voiced speech, the source originates from the vibration of the vocal folds, which produces a quasi-periodic glottal sound wave whose fundamental frequency defines the pitch. In unvoiced speech, a noise source is produced by a turbulent airflow or an acoustic impulse. The source signal is modified by the vocal tract, which is assumed to act as a linear filter. The cavities of the vocal tract give rise to resonances, which are called the formants and are characterized by their frequency, amplitude and bandwidth. By moving the speech articulators such as the tongue, lips, and jaw, humans modify the shape of their vocal tract, which results in a change of the acoustic filter and the associated resonances. This is how the different elementary speech sounds called phonemes are produced to form syllables, words and sentences.
The power spectra and the spectral envelopes of two French vowels are displayed in Figure 1. The spectral envelopes show that the formant frequencies are different for the two vowels. In this example however, the harmonic structure of the spectra shows that the fundamental frequency is the same for the two vowels. Formant frequencies are important distinctive features of vowels. In a first approximation, they can be related to the opening of the mouth, the front/rear position of the tongue, and the rounding of the lips for the first, second, and third formant respectively. For voiced phonemes, humans are able to control the formants independently of the pitch (i.e., to change the filter independently of the source (Fant, 1970)) and of each other (MacDonald et al., 2011). The independence of the source and filter characteristics makes the speech signals an interesting material for representation learning methods, especially with deep generative latent-variable models.
In the present study, in addition to the pre-trained IS-VAE speech spectrogram model, we also assume the availability of an artificial speech synthesizer allowing for an accurate and independent control of the fundamental frequency and formants. In this work, we use Soundgen (Anikin, 2019), a parametric synthesizer based on the source-filter model of speech production. For a given speech sound, the voiced component of the source signal is generated by a sum of sine waves, the noise component by a filtered white noise, and both components are then summed and passed through a linear filter simulating the effect of the human vocal tract. Importantly, this synthesizer allows us to easily generate artificial speech data labeled with the fundamental frequency and formant frequency values.

heading: 3 ANALYZING AND CONTROLLING SOURCE-FILTER FACTORS OF SPEECH
text: VARIATION IN THE VAE
In this section, from a VAE trained on natural speech signals and a few artificially-generated labeled speech signals, we propose (i) a method to learn latent subspaces encoding source-filter factors of speech variation, (ii) a simple strategy to measure the disentanglement of the learned representation, and (iii) a weakly-supervised approach to control the continuous factors of variation in the learned subspaces and generate corresponding speech signals.
Let fi denote the speech factor of variation (in Hz) corresponding to the fundamental frequency, for i = 0, and to the formant frequencies, for i ∈ {1, 2, ...}. Let Di denote a dataset of artificiallygenerated speech vectors (more precisely short-term power spectra) synthesized by varying only fi,
all other factors {fj , j 6= i} being arbitrarily fixed. All examples inDi are labeled with the factors of variation. It would be very difficult to build such a dataset from existing corpora of unlabeled natural speech. In contrast, it is an easy task using an artificial speech synthesizer such as Soundgen, which precisely takes the fundamental frequency and formant parameters as input, and outputs waveforms from which we extract power spectra.

heading: 3.1 LEARNING LATENT SUBSPACES ENCODING SOURCE-FILTER FACTORS OF VARIATION
text: Let p̂(i)(x) denote the empirical distribution associated with Di, defined similarly as p̂(x). We also introduce the following marginal distribution over the latent vectors:
q̂ (i) φ (z) = ∫ qφ(z|x)p̂(i)(x)dx = 1
#Di ∑ xn∈Di qφ(z|xn). (3)
In the literature, this quantity is referred to as the aggregated posterior (Makhzani et al., 2016). However, qφ(z|x) is usually aggregated over the empirical distribution p̂(x) such that the aggregated posterior is expected to match with the prior p(z) (Chen et al., 2018; Dai & Wipf, 2018). In contrast, in Equation (3) we aggregate over the “biased” data distribution p̂(i)(x), where we know only one latent factor varies. This defines the explicit inductive bias (Locatello et al., 2019) that we exploit to learn the latent source-filter representation of speech in the VAE.
In the following of the paper, without loss of generality, we assume that, for each data vector in Di, the associated latent vector z has been centered by subtracting the mean vector
µφ(Di) = Eq̂(i)φ (z)[z] = 1
#Di ∑ xn∈Di µφ(xn). (4)
Because only one factor varies in Di, we expect latent vectors drawn from the “biased” aggregated posterior in Equation (3) to live on a lower-dimensional manifold embedded in the original latent space RL. We assume this manifold to be a subspace characterized by its semi-orthogonal basis matrix Ui ∈ RL×Mi , 1 ≤Mi < L. This matrix is computed by solving the following optimization problem:
min U∈RL×Mi E q̂ (i) φ (z)
[∥∥z−UU>z∥∥2 2 ] , s.t. U>U = I. (5)
The space spanned by the columns of Ui is a subspace of the original latent space RL in which the latent vectors associated with the variation of the factor fi inDi are expected to live. In Appendix A, we show that, similarly to principal component analysis (PCA) (Pearson, 1901), the solution to the optimization problem (5) is given by the Mi eigenvectors corresponding to the Mi largest eigenvalues of
Sφ(Di) = 1
#Di ∑ xn∈Di [ µφ(xn)µφ(xn) > + diag{vφ(xn)} ] − µφ(Di)µφ(Di)>. (6)
The dimension Mi of the subspace can be chosen such as to retain a certain percentage of the data variance in the latent space. Note that the only source of supervision used here is the knowledge that only the factor fi varies in the dataset Di.

heading: 3.2 DISENTANGLEMENT ANALYSIS OF THE LATENT REPRESENTATION
text: As defined by Higgins et al. (2018), a representation is disentangled if it is possible to learn orthogonal latent subspaces associated with each factor of variation, whether they are single- or multidimensional. The approach presented in the previous subsection exactly follows this definition and
offers a natural and straightforward way to objectively measure if the unsupervised VAE managed to learn a disentangled representation of the factors of variation under consideration. First, by simply looking at the eigenvalues associated with the columns of Ui ∈ RL×Mi , we can measure the amount of variance that is retained by the projection UiU>i . If a small number of components Mi represents most of the variance, it indicates that only a few intrinsic dimensions of the latent space are dedicated to the factor of variation fi and varying this factor can be done by affine transformations. Second, if for two different factors of variation fi and fj , with i 6= j, the columns of Ui are orthogonal to those of Uj , this indicates that the two factors are encoded in orthogonal subspaces and therefore disentangled. It should however be verified experimentally that applying transformations by moving onto the subspace associated with fi generalizes to values of {fj , j 6= i} different than the ones used in Di.

heading: 3.3 CONTROLLING THE FACTORS OF VARIATION FOR SPEECH TRANSFORMATION AND GENERATION
text: So far, for each factor fi, we have defined a methodology to learn a latent subspace Ui ∈ RL×Mi that encodes its variations in the dataset Di, containing a few examples of speech data generated by an artificial synthesizer. Making now use of the labels in Di, we learn a regression model gηi : R+ 7→ RMi from the factor fi, whose value is denoted by y ∈ R+, to the data coordinates in the latent subspace defined by Ui. The parameters ηi are defined as the solution of the following optimization problem:
min η { E q̂ (i) φ (z,y) [∥∥gη(y)−U>i z∥∥22] c= 1#Di ∑ (xn,yn)∈Di ∥∥gη(yn)−U>i (µφ(xn)− µφ(Di))∥∥22 }, (7)
where q̂(i)φ (z, y) = ∫ qφ(z|x)p̂(i)(x, y)dx, p̂(i)(x, y) is the empirical distribution associated with Di, considering now both the speech data vector x and the label y, and c = denotes equality up to an additive constant w.r.t. η. The dataset Di is very small with only a few hundreds examples, and as it is synthetic and labels are not provided by human annotators, the problem can be considered very weakly supervised. For simplicity and because it revealed efficient for this task, gηi is chosen as a piece-wise linear regression model learned independently for each output coordinate m ∈ {1, ...,Mi}. This choice is supported by the fact that the semi-orthogonal matrix Ui decorrelates the data (Bengio et al., 2013). Solving the optimization problem (7) then basically consists in solving a linear system of equations (Jekel & Venter, 2019).
We can now transform a speech spectrogram by analyzing it with the VAE encoder, then linearly moving in the learned subspaces using the above regression model, and finally resynthesizing it with the VAE decoder. Given a source latent vector z and a target value y for the factor fi, we apply the following affine transformation:
z̃ = z−UiU>i z+Uigηi(y). (8) This transformation consists in (i) subtracting the projection of z onto the subspace associated with the factor of variation fi; and (ii) adding the target component provided by the regression model gηi mapped from the learned subspace to the original latent space by the matrix Ui. This operation allows us to move only in the latent subspace associated with the factor fi. If this subspace is orthogonal to the latent subspaces associated with the other factors {fj , j 6= i}, the latter should remain the same between z and z̃, only fi should be modified. This process can be straightforwardly generalized to multiple factors, by subtracting and adding terms corresponding to each one of them.
Finally, as the prior p(z) and inference model qφ(z|x) are Gaussian (see Equations (1) and (2)), the transformation in Equation (8) has the following probabilistic formulation (using U>i Ui = I):
p(z̃; fi = y) = N ( z̃;Uigηi(y), I−UiU>i ) (9)
qφ(z̃|x; fi = y) = N ( z̃;Uigηi(y) + (I−UiU>i )µφ(x), (I−UiU>i ) diag{vφ(x)} ) . (10)
The prior in Equation (9) is now conditioned on the factor fi and can be used to generate speech data given input trajectories of fundamental frequency and formant frequencies. As we assumed centered latent data, the mean vector µφ(Di) defined in Equation (4) must be added to z̃ before mapping this vector through the generative model pθ(x|z).
(a) Fundamental frequency and formant transformations of the vowel /a/ uttered by a female speaker. (b) Spectrogram generated from input trajectories of the fundamental frequency and formant frequencies.
(c) Left: original spectrogram of a speech signal uttered by a female speaker; Middle: transformed spectrogram where the fundamental frequency is set constant over time; Right: transformed spectrogram where the original voiced speech signal (left) is converted into a whispered speech signal (i.e., the pitch is removed).
Figure 2: Qualitative example of modified and generated spectrograms with the proposed method. The color bar indicates the power in dB.

heading: 4 EXPERIMENTS
text: This section presents qualitative and quantitative experimental results of the proposed method for controlling the fundamental frequency and formant frequencies of speech signals with a VAE. The VAE is trained on about 25 hours of multi-speaker speech data from the Wall Street Journal (WSJ0) dataset (Garofolo et al., 1993a). The data space dimension is 513 and the latent space dimension is 16. For a given factor of variation, the corresponding latent subspace is learned (see Section 3.1) using short trajectories of speech power spectra (corresponding to a few seconds of speech) generated with Soundgen (Anikin, 2019), all other factors being arbitrarily fixed. When solving the optimization problem (5), the latent subspace dimension Mi of each factor of variation is chosen such that 80% of the data variance is retained. This leadsM0 = 4, M1 = 1 andM2 =M3 = 3. The regression models used to control the speech factors of variation in the latent space (see Section 3.3) are learned on the same trajectories, but using the labels that correspond to the input control parameters of Soundgen (i.e., fundamental frequency and formant frequencies values). More details on the experimental set-up can be found in Appendix B. Given a generated or transformed spectrogram, we use Waveglow (Prenger et al., 2019) to reconstruct the time-domain signal.

heading: 4.1 QUALITATIVE RESULTS
text: In Figure 2a, we illustrate the ability of the proposed method to modify the fundamental frequency and formant frequencies in an accurate and independent manner. The spectrogram contains five segments of equal length. The first segment corresponds to the original spectrogram of the steady vowel /a/ uttered by a female speaker. In the following segments, we vary successively each individual factor fi, for i = 0 to 3, as indicated by the black lines in the figure. Variations of f0 modify the harmonic structure of the signal while keeping the formant structure unaltered. Variations of fi, i ∈ {1, 2, 3}, modify the formant frequencies (i.e., the vocal tract resonances, as indicated by the color map) while keeping the fundamental frequency unaltered. Figure 2b represents a spectrogram generated by using the conditional prior in equation (9) (generalized to conditioning on multiple factors). We can see that the characteristics of the generated speech spectrogram match well with the input trajectories represented by the lines in the figure. In Figure 2c, from left to right we show the original spectrogram of a speech signal uttered by a female speaker (left), and the transformed spectrograms where the fundamental frequency is set constant over time (middle) and where the pitch has been removed (i.e., the original voiced speech signal is converted into a whispered speech signal) (right). This last spectrogram is simply obtained by subtracting to z its projection onto the latent
subspace corresponding to f0 (i.e., by considering only the two first terms in the right-hand side of Equation (8)). It results in a spectrogram where the harmonic component is neutralized, while preserving the original formant structure. This is remarkable considering that the VAE was not trained on whispered speech signals, and it further confirms that the proposed method dissociates the source and the filter contributions in the VAE latent space. Other qualitative results can be found in Appendix C (visualization of trajectories in the learned latent subspaces and additional examples of generated and transformed speech spectrograms) and at https://tinyurl.com/iclr2022 (including audio examples).

heading: 4.2 QUANTITATIVE RESULTS
text: Due to space limitations, we only provide in this section quantitative results related to the fundamental frequency (f0) and the first two formant frequencies (f1 and f2). Additional results can be found in Appendix D, including the performance for modifications of the third formant frequency (f3), the performance with a VAE trained on other datasets (e.g., a dataset of French speech signals), and the performance on the TIMIT dataset (Garofolo et al., 1993b), which is richer than the English vowel one in terms of phonemes but is not labeled with the fundamental and formant frequencies, making the evaluation less reliable.

heading: 4.2.1 ORTHOGONALITY OF THE LATENT SUBSPACES
text: Following the discussion in Section 3.2, we compute the dot product between all pairs of unit vectors in the matrices Ui ∈ RL×Mi , i ∈ {0, 1, 2}. Figure 3 shows that the resulting correlation matrix is mainly diagonal. Except for a correlation value of −0.23 across f1 and the first component of f2, all other values are below 0.13 (in absolute value), confirming the orthogonality of the learned subspaces and thus the disentanglement of the learned source-filter representation of speech.

heading: 4.2.2 PITCH AND FORMANT TRANSFORMATIONS OF ENGLISH VOWELS
text: Experimental set-up In this experiment, we quantitatively evaluate the performance of the proposed method regarding the modification of the fundamental frequency and formant frequencies in speech signals (see Section 3.3). We use a corpus of 12 English vowels uttered by 50 male and 50 female speakers (Hillenbrand et al., 1995). This dataset is labeled with the fundamental and formant frequencies. We transform each signal in this dataset by varying one single factor fi at a time, for i ∈ {0, 1, 2}. The ranges of transformation for the fundamental frequency, first formant frequency, and second formant frequency are respectively [100, 300] Hz, [300, 900] Hz, and [1100, 2700] Hz, with a step of 1 Hz, 10 Hz and 20 Hz respectively. For the modification of each factor fi, we measure the performance regarding three aspects: First, in terms of accuracy by comparing the target value for the factor (see Equation (8)) and its estimation computed from the modified output speech signal. Second, in terms of disentanglement, by comparing the values of fj for j 6= i, before and after modification of the factor fi. Third, in terms of speech naturalness of the transformed signal.
Metrics Accuracy and disentanglement are measured in terms of relative absolute error (in percent, the lower the better). For a given factor fi, it is defined by δfi = 100%× |ŷ− y|/y where y is
the target value of fi and ŷ its estimation from the output transformed signal. Let us take the example of a modification of f0: δf0 measures the accuracy of the transformation on f0 while δf1 and δf2 are used to assess if the other factors of variation f1 and f2 remained unchanged after modifying f0. We use CREPE (Kim et al., 2018) to estimate the fundamental frequency and Parselmouth (Jadoul et al., 2018), which is based on PRAAT, to estimate the formant frequencies. Regarding speech naturalness, we use the objective measure provided by NISQA (Mittag & Möller, 2020). This metric (the higher the better) was developed in the context of speech transformation algorithms and it was shown to highly correlate with subjective mean opinion scores (MOS) (i.e., human ratings). As a reference, the score provided by NISQA on the original dataset of English vowels (i.e., without any processing) is equal to 2.60 ± 0.53. Methods We compare the proposed approach with several methods from the literature: (i) Timedomain pitch-synchronous overlap-and-add (TD-PSOLA) (Moulines & Charpentier, 1990) performs fundamental frequency modifications through the decomposition of the signal into pitchsynchronized overlapping frames. (ii) WORLD (Morise et al., 2016) is a vocoder also used for fundamental-frequency modifications. It decomposes the speech signal into three components characterizing the fundamental frequency, the aperiodicity, and the spectral envelope. (iii) The method proposed by Hsu et al. (2017b) (here referred to as “VAE baseline”) consists in applying translations directly in the latent space of the VAE. Unlike the proposed approach, this method requires predefined latent attribute representations µsrc and µtrgt associated with the source and target values of the factor to be modified, respectively. In particular, computing µsrc requires analyzing the input speech signal, for instance to estimate the fundamental frequency, which is not the case for the proposed method. The source and target latent attribute representations are then used to perform the translation z̃ = z−µsrc +µtrgt, where z and z̃ are respectively the original and modified latent vectors. To ensure fair comparison, we build dictionaries of predefined latent attribute representations using the same artificially-generated speech data that were used in the weakly-supervised training stage of the proposed method. All the methods we compare with require a pre-processing of the input speech signal to compute the input trajectory of the factor to be modified (e.g. the fundamental frequency), which is not the case of the proposed method.
Discussion Experimental results (mean and standard deviation) are shown in Table 1. Compared to the VAE baseline, the proposed method obtains better performance in terms of accuracy, disentanglement, and naturalness. These results confirm the effectiveness of performing the transformations in the learned latent subspaces and not directly in the latent space, as well as the advantage of using regression models instead of predefined latent attribute representations. Regarding f0 transformation, WORLD obtains the best performance in terms of disentanglement, which is because the source and filter contributions are decoupled in the architecture of the vocoder. In terms of naturalness, WORLD and then TD-PSOLA obtain the best performance. This may be explained by the fact that these method operate directly in the time domain, therefore they do not suffer from phase reconstruction artifacts, unlike the proposed and VAE baseline methods. Naturalness is indeed greatly affected by phase reconstruction artificats, even from an unaltered speech spectrogram (i.e., without transformation). Phase reconstruction in a multi-speaker setting is still an open problem in speech processing. We want to emphasize that the objective of this study is not to compete with traditional signal processing methods such as TD-PSOLA and WORLD. It is rather to advance on the understanding of deep generative modeling of speech signals and to compare honestly with highly-specialized traditional systems. TD-PSOLA and WORLD exploit signal models specifically designed for the task at hand, which for instance prevents them to be used for modifying formant frequencies. In contrast, the proposed method is fully based on learning and the same methodology applies for modifying the fundamental or formant frequencies.

heading: 5 RELATED WORK
text: Variants of the VAE have recently led to considerable progress in disentangled representation learning (Kim & Mnih, 2018; Higgins et al., 2017; Chen et al., 2018). From experimental analyses on image data, these methods suggest that a vanilla VAE cannot learn a disentangled representation. In the present study, we experimentally show that learning a disentangled source-filter representation of speech using a simple VAE is possible, complying with the definition of disentanglement proposed in (Higgins et al., 2018). Locatello et al. (2019; 2020) recently showed both theoretically and from a large-scale experimental study that the unsupervised learning of disentangled representations
is impossible without inductive biases on both the models and the data. In the present study, we precisely employ a few examples of artificially-generated labeled speech data in order to disentangle the latent representation of a simple VAE, in terms of source-filter factors of speech variation.
Modifying speech signals characteristics such as the pitch or timbre is a highly covered research topic where deep generative modeling has recently emerged as a promising approach. Voice conversion consists in modifying speaker characteristics while keeping the linguistic information unchanged, e.g. (Kaneko & Kameoka, 2018; Kaneko et al., 2019; Kameoka et al., 2020). Many voice conversion methods use text information as input (Wang et al., 2018; Zhang et al., 2019; Kulkarni et al., 2021; Li et al., 2021), which greatly helps for transferring non-verbal vocal attributes from one speech signal to another, but it requires datasets labeled with text. Qian et al. (2020) and Webber et al. (2020) recently proposed speech transformation methods based on latent representation learning without resorting to textual information. The interpretability of the learned representations is enforced by the design of the model. Qian et al. (2020) proposed to use three independent encoder networks to decompose a speech signal into fundamental frequency, timbre and rhythm latent representations. Webber et al. (2020) focused on controlling source-filter parameters in speech signals, where the ability to control a given parameter (e.g. fundamental frequency) is enforced explicitly using labeled data and adversarial learning. In this approach, each parameter to be controlled requires a dedicated training of the model. Moreover, these methods are speaker-dependent, as speech generation in Qian et al. (2020) is conditioned on the speaker identity and Webber et al. (2020) used a single-speaker training dataset. This contrasts with the proposed method which is speakerindependent, and in which the source-filter representation of speech naturally emerges in the latent space of a single unsupervised VAE.
Hsu et al. (2017a;b) proposed to use VAEs for modifying the speaker identity and the phonemic content of speech signals. In (Hsu et al., 2017b), this is achieved by translations in the latent space. This is the baseline VAE approach we presented and compared with in Section 4. It requires to know predefined values of the latent representations associated with the source/target speech attributes to be modified. Conversely, Equation (8) in the proposed method does not require the knowledge of the factor fi associated with source vector z, it only requires the value associated with the target vector z̃. Moreover, Hsu et al. (2017b) did not address the control of continuous factors of speech variation in the VAE latent space, contrary to the present work.
Several methods have been recently proposed to control continuous factors of variation in deep generative models (Jahanian et al., 2019; Plumerault et al., 2020; Goetschalckx et al., 2019; Härkönen et al., 2020), focusing essentially on generative adversarial networks. They consist in identifying and then moving onto semantically meaningful directions in the latent space of the model. The present work is inspired by (Plumerault et al., 2020), which assumes that a factor of variation can be predicted from the projection of the latent vector along a specific axis, learned from artificially generated trajectories. The proposed method is however more generic, thanks to the learning of latent subspaces associated to the latent factors and to the introduction of a general formalism based on the use of “biased” aggregated posteriors. Moreover, these previous works on controlling deep generative models only allow for moving “blindly” onto semantically meaningful directions in the latent space. In the present study, we are able to generate data conditioned on a specific target value for a given factor of variation. Finally, previous work focused on image data, and to the best of our knowledge, this is the first approach to control source-filter factors of speech variation in a VAE.

heading: 6 CONCLUSION
text: In this work, using only a few seconds of artificially generated labeled speech data, we showed that the fundamental frequency and formant frequencies are encoded in orthogonal latent subspaces of an unsupervised VAE and we proposed a weakly-supervised method to control these attributes within the learned subspaces. The method generalizes well when applied to natural speech signals. To the best of our knowledge, this is the first approach that, with a single methodology, is able to extract, identify and control the source and filter low-level speech attributes within a VAE latent space. The proposed method could also be applied to other types of data such as natural or face images, provided that one can create a few synthetic images that capture variations in a single latent factor of interest, independently of others. For example, this would be possible for factors encoding the position, rotation and scale of objects in natural images, or some facial expressions.

heading: 7 REPRODUCIBILITY STATEMENT
text: To ensure reproducibility, the implementation of the method and the data generated with soundgen that are used to learn the latent subspaces and regression models will be made freely available online.

heading: A SOLUTION TO THE LATENT SUBSPACE LEARNING PROBLEM
text: In this Appendix, we show that the solution to the optimization problem (5) is given by the principal eigenvectors of Sφ(Di) in Equation (6). Without loss of generality, we formulate the problem for a centered version of the latent data
z← z− µφ(Di), (11)
where µφ(Di) is defined in Equation (4). This centering also affects the inference model originally defined in Equation (2), as follows:
qφ(z|x) = N (z;µφ(x)− µφ(Di),diag{vφ(x)) . (12)
Using Equation (3), the fact that U>U = I, and Equation (12), the cost function in the optimization problem (5) can be rewritten as follows:
E q̂ (i) φ (z) [∥∥z−UU>z∥∥2 2 ] = 1
#Di ∑ xn∈Di Eqφ(z|xn) [∥∥z−UU>z∥∥2 2 ] = tr { (I−UU>) 1
#Di ∑ xn∈Di Eqφ(z|xn)[zz >] } = tr { (I−UU>)Sφ(Di) } , (13)
where Sφ(Di) is defined in Equation (6). From this last equality, we see that the optimization problem (5) is equivalent to
max U∈RL×Mi
tr { U>Sφ(Di)U } , s.t. U>U = I. (14)
Very similarly to PCA (Pearson, 1901), the solution is given by the Mi dominant eigenvectors of Sφ(Di) (i.e., associated to the Mi largest eigenvalues) (Bishop, 2006, Section 12.1).

heading: B EXPERIMENTAL SETUP
text: B.1 VAE TRAINING
To train the IS-VAE model (Bando et al., 2018; Leglaive et al., 2018; Girin et al., 2019), we use the Wall Street Journal (WSJ0) dataset (Garofolo et al., 1993a) which contains 25 hours of speech signals sampled at 16 kHz, including 52 female and 49 male speakers. The time-domain spech signals are converted to power spectrograms using the short-time Fourier transform (STFT) with an analysis window of length 64 ms and an overlap of 75%. The architecture of the VAE is shown in Figure 4, the encoder and decoder networks each have three dense layers. A hyperbolic tangent (tanh) activation function is used at each layer, except for the output layers of the encoder and decoder where we use the identity function. The VAE input/output dimension is D = 513 (we only keep the non redundant part of the power spectrum at a given time frame) and the latent vector dimension is set to L = 16. We train the model using the Adam optimizer (Kingma & Ba, 2015) with a learning rate equal to 0.001.
B.2 ANALYZING AND CONTROLLING SOURCE-FILTER FACTORS OF SPEECH VARIATION IN THE VAE
For a given factor of variation, the corresponding latent subspace is learned (see Section 3.1) using trajectories of speech power spectra generated with Soundgen (Anikin, 2019), all other factors being arbitrarily fixed. For f0, the trajectory contains 226 points (which corresponds to 3.6 seconds of speech) evenly spaced in the range [85, 310] Hz, f1, f2 and f3 are fixed to 600 Hz, 1500 Hz, and 3200 Hz, respectively. For f1, the trajectory contains 401 points (which corresponds to 6.4 seconds of speech) evenly spaced in the range [200, 1000] Hz, f0, f2 and f3 are fixed to 140 Hz, 1600 Hz, and 3200 Hz, respectively. For f2, the trajectory contains 401 points evenly spaced in the range [800, 2800] Hz, f0, f1 and f3 are fixed to 140 Hz, 500 Hz, and 3200 Hz, respectively. For f3, the trajectory contains 241 points (which corresponds to 3.9 seconds of speech) evenly spaced in the
range [2000, 3200] Hz, f0, f1 and f2 are fixed to 140 Hz, 500 Hz, and 1200 Hz, respectively. The amplitude of the formants is set to 30 dB, and the bandwidth is automatically calculated from the fromant frequencies by Soundgen, using a formula derived from human phonetic research (Anikin, 2019). The regression models used to control the speech factors of variation in the latent space (see Sections 3.3) are learned on the same trajectories, but using the labels that correspond to the input control parameters of Soundgen.

heading: C ADDITIONAL QUALITATIVE RESULTS
text: C.1 VISUALIZATION OF THE LEARNED LATENT SUBSPACES
For i = 0, 1, 2 and 3, Figures 5a, 5b, 5c, 5d are respectively obtained by projecting the latent mean vectors µφ(x) ∈ RL, for all data vectors x ∈ Di, within the latent subspace characterized by Ui ∈ RL×Mi (i.e., we perform dimensionality reduction). In the previously reported experiments, the latent subspace dimension Mi of each factor of variation was chosen such that 80% of the data variance was retained in the latent space. It resulted in M0 = 4, M1 = 1 and M2 =M3 = 3. In this section, for visualization purposes, we set Mi = 3 for all i ∈ {0, 1, 2, 3}. However, we can see that the f1 trajectory (Figure 5b) is mainly concentrated along a single axis. Regarding f0 (Figure 5a), setting M0 = 3 retained 78% of the variance of D0 in the latent space. An important observation that we make from these figures is that two data vectors x and x′, corresponding to values of a given factor that are close, have projections of µφ(x) and µφ(x′) that are also close in the learned latent subspaces. This can be seen from the color bars which indicate the values of the factors of variation. It indicates that the learned representation preserves the notion of proximity in terms of fundamental frequency and formant frequencies.
In Figure 5e, we project three different datasets D1, defined for three different values of f2. Similarly, in Figure 5f we show the trajectories associated with the projection of three datasets D2, defined for three different values of f1. We notice that as expected, the trajectories are almost identical and only differ by a translation.
C.2 EXAMPLES OF GENERATED SPEECH SPECTRA
C.3 EXAMPLES OF TRANSFORMED SPEECH SPECTROGRAMS

heading: D ADDITIONAL QUANTITATIVE RESULTS
text: D.1 TRANSFORMATIONS OF THE THIRD FORMANT FREQUENCY ON THE ENGLISH VOWELS DATASET
Table 2 completes Table 1 by including transformations of the third formant frequency in the range [2200, 3200] Hz, with a step of 20 Hz. The new column δf3 also measures the performance in terms of disentanglement when modifying other factors fi 6= f3. These additional results do not modify the conclusions drawn in Section 4.
D.2 ROBUSTNESS WITH RESPECT TO DIFFERENT TRAINING DATASETS FOR THE VAE
This Section investigates the robustness of the proposed method with respect to different datasets used to train the VAE model. This table presents results for modifications of the fundamental frequency only, applied to the English vowels dataset. We considered the following training datasets:
• WSJ: the Wall street Journal (Garofolo et al., 1993a) dataset that was used in the previous experiments.
• SIWIS: the SIWIS French speech synthesis dataset (Honnet et al., 2017), which contains more than ten hours of French speech recordings.
• TESS: the Toronto emotional speech dataset (Dupuis & Pichora-Fuller, 2010), which contains 2800 utterances spoken by two actresses using different emotions (anger, disgust, fear, happiness, pleasant surprise, sadness, and neutral).
• LJspeech: the LJspeech dataset (Ito & Johnson, 2017), which contains 13100 short audio clips of a single speaker reading passages from 7 non-fiction books.
The artificially-generated speech dataset used for learning the latent subspaces and the regression models along with the test dataset of English vowels remain the same. It can be seen in Table 3 that the performance remains quite stable with different VAE training datasets. WSJ is the largest dataset and therefore obtains the best performance. Interestingly, the results obtained with the SIWIS dataset of French speech signals remain satisfactory, even if there is a mismatch between the training (French) and testing (English) datasets.
D.3 EVALUATION ON THE TIMIT DATASET
In this Section, we evaluate the proposed method on the TIMIT dataset (Garofolo et al., 1993b), using the VAE trained on the WSJ dataset. TIMIT is a corpus of phonemically and lexically transcribed speech of American English speakers of different sexes and dialects. We used the test corpus containing 1680 utterances. Because we are interested in studying the interaction between modifications of the fundamental frequency and formant frequencies, we only evaluate the method on the phonemes that are voiced (40 phonemes over a total of 52), which can be identified using the annotations. The ranges of transformation for the fundamental frequency, first, second and third formant frequencies are respectively [100, 300] Hz, [300, 900] Hz, [1100, 2700] Hz, and [2200, 3200] Hz with a step of 10 Hz, 50 Hz, 100 Hz and 50 Hz, respectively. The metrics and evaluated methods remain the same as those described in Section 4.2.2.
TIMIT is phonemically richer than the English vowels dataset previously used, however it is not labeled with the fundamental and formant frequencies. Therefore, we do not have the ground truth values which makes the evaluation more difficult than with the English vowels dataset. Instead of the ground truth, we use the formant frequencies and the fundamental frequency computed on the original speech utterances (i.e., before transformation) using the same tools as described in Section 4.2.2. This makes the evaluation on TIMIT less reliable than on the English vowels dataset, but it allows us to test the methods on a larger variety of phonemes.
Results are presented in Table 4. They are very consistent with the ones obtained on the English vowels dataset (see Tables 1 and 2), which further strengthens the conclusions drawn in the discussion paragraph of Section 4.2.2.

Review summaries:
The paper proposes a method for utilizing labeled synthetic data in order to characterize and control the latent space of a VAE trained on individual frames of speech spectrograms. Key properties of the data which one might want explicit control over are identified, i.e., pitch and formant frequencies, and a parametric speech synthesizer is used to generate synthetic datasets for each property, where the property in question is varied but all others are kept fixed. These labeled data are used to identify subspaces of a VAE latent which correspond to each property, essentially by a principal components analysis of the latent vectors from each point in the synthetic dataset for that property. The degree of disentanglement of the latent representation can be characterized by how orthogonal the bases are across subspaces. Furthermore, individual properties can be directly controlled in isolation by learning a simple linear regression model mapping from the quantity in question (e.g., fundamental frequency in Hertz) to the subspace basis using supervision from the corresponding synthetic data.

This paper analyzes VAE latent embeddings to extract subspaces that relate to pitch (f0) and formant frequencies (f1 through fN). This is done through first training a frame-synchronous  IS-VAE model from clean speech data. The authors pass controlled synthesized speech through the model and obtain the embeddings corresponding to the synthesized data with randomly varying one fj component and keeping others constant. Each embedding corresponds to a frame of spectral data. The set of embeddings are then analyzed by PCA to find the principal eigenvectors corresponding to each fj explaining 80% of the total energy. The mapping from f0, f1, .. fN values and the subspace coefficients is done through a linear regression mapping independently for each fj which is also learned from synthetic data. Through simple examples, the authors show that they can change individual components (f0, f1, .., fN) by modifying the embeddings correspond to a signal, to some extent. They compare their results with rule-based vocoders (WORLD, TD-PSOLA) and a VAE baseline that does not use subspaces.

This paper shows that the fundamental frequency and formant frequency information is encoded in a speech VAE model. This can be found by using artificially controlled/generated dataset. After finding how to manipulate the latent space, one can control arbitrary speech samples in a desirable way, such as controlling fundamental frequency or formant frequencies. The authors also show that the harmonic parts can be removed from the original speech and reconstructed as a whispered voice using only the spectral envelope part. Experiment results show that the proposed method can control formant frequencies and show that it can control better than the method proposed by previous work. The proposed method show on par or worse performance on fundamental frequency control experiment when compared to traditional DSP vocoders, such as TD-PSOLA or WORLD.