Learning and controlling the source-filter representation of speech with a variational autoencoder
Authors: Samir Alain Sadok
Simon Leglaive
Laurent Girin
Xavier Alameda-Pineda
Renaud Séguier
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep generative models
variational autoencoder (VAE)
speech processing
source-filter model of speech production
representation learning

Abstract: Understanding and controlling latent representations in deep generative models is a challenging yet important problem for analyzing, transforming and generating various types of data. In speech processing, inspiring from the anatomical mechanisms of phonation, the source-filter model considers that speech signals are produced from a few independent and physically meaningful continuous latent factors, among which the fundamental frequency and the formants are of primary importance. In this work, we show that the source-filter model of speech production naturally arises in the latent space of a variational autoencoder (VAE) trained in an unsupervised fashion on a dataset of natural speech signals. Using speech signals generated with an artificial speech synthesizer, we experimentally demonstrate that the fundamental frequency and formant frequencies are encoded in orthogonal subspaces of the VAE latent space and we develop a weakly-supervised method to accurately and independently control these speech factors of variation within the learned latent subspaces. Without requiring additional information such as text or human-labeled data, we propose a deep generative model of speech spectrograms that is conditioned on the fundamental frequency and formant frequencies, and which is applied to the transformation of speech signals.

Under review as a conference paper at ICLR 2022
LEARNING AND CONTROLLING THE SOURCE-FILTER REPRESENTATION OF SPEECH WITH A VARIATIONAL AUTOENCODER
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding and controlling latent representations in deep generative models is a challenging yet important problem for analyzing, transforming and generat- ing various types of data. In speech processing, inspiring from the anatomical mechanisms of phonation, the source-filter model considers that speech signals are produced from a few independent and physically meaningful continuous latent factors, among which the fundamental frequency and the formants are primary importance. In this work, tion naturally arises in the latent space a variational autoencoder (VAE) trained in an unsupervised fashion on dataset of natural speech signals. Using only few seconds of labeled speech signals generated with an artificial speech synthesizer, we experimentally demonstrate that the fundamental frequency and formant fre- quencies are encoded in orthogonal subspaces of the VAE latent space and we de- velop a weakly-supervised method to accurately and independently control these speech factors of variation within the learned latent subspaces. Without requir- ing additional information such as text or human-labeled data, we propose a deep generative model of speech spectrograms that is conditioned on the fundamental frequency and formant frequencies, and which is applied to the transformation speech signals.
show that the source-filter model of speech produe-
INTRODUCTION
High-dimensional data such as natural images or speech signals exhibit some form of regularity which prevents their dimensions from varying independently from each other. This suggests that there exists latent representation of smaller dimension from which the high-dimensional observed data were generated. Discovering the hidden properties of complex data is the goal of representa- tion learning, and deep latent-variable generative models have emerged as promising unsupervised
approaches (Goodfellow al. 2014] Kingma& & Welling 2014, Rezende et al.) 2014 Chen et al., 2016) Higgins et al.] 2017 Kim & Mnih] 2018 Chen et al.] 2018). The variational autoencoder (VAE) (Kingma & Welling 2014| Rezende et al. (2014), which is equipped with both generative and inference model, can be used not only for data generation but also for analysis and transforma- tion. As an explicit model of a probability density function (pdf), the VAE can also be used learned prior for solving inverse problems such as compressed sensing (Bora et al.] (017), speech enhancement (Bando al. 2018 Leglaive et al., 2018),, or source separation (Kameoka al. 2019] Jayaram & Thickstun| (020). Making sense of the latent representation learned by VAE and con- trolling the underlying continuous factors of variation in the data are important challenges build more expressive and interpretable generative models and probabilistic priors.
2018
Previous works on representation learning with deep generative models, in particular VAEs, have mostly focused on images (Higgins et al.) 20177; Kim & Mnih 2018: [Chen et al., 2018 Locatello ct al. 2019 2020). Yet, it is not always easy to define the ground-truth latent factors of variation involved in the generation of natural images. For speech data, the latent factors of variation can be directly related to the anatomical mechanisms of speech production. This makes speech data interesting for investigating the disentangled representation learning capabilities of VAEs, comple- mentary to studies dealing with images. A key concept for characterizing the structure of speech signals is deduced from the source-filter model proposed by Fant (1970). This model, described in

---

Under review as a conference paper at ICLR 2022
more detail in Section 2.2 implies that a speech signal is mainly characterized by a few continuous latent factors of variation corresponding to the vibration of the vocal folds (i.e., the source), which defines the fundamental frequency, and the resonances of the vocal tract (i.e., the filter), which de- fine the formants. The source-filter model is at the core of various fundamental speech processing techniques such as cepstral representations and linear predictive coding (LPC) (Rabiner & Schafer 2010). Valin & Skoglund (2019); [Wang et al. (2019) and Juvela et al. (2019) have recently shown that the efficiency of neural speech vocoders can be largely improved by leveraging the source- filter model. Other works investigating the interaction between the source-filter model and neural networks include Lee et al. (2019) and Choi et al. (202T). All these studies illustrate the interest of combining deep learning techniques with more traditional signal processing models and algo- rithms. In this work, we interpret and control the latent space of a VAE from the perspective of the source-filter model of speech production, which can be beneficial for various applications in speech analysis, transformation, and synthesis.
We first train VAE on dataset of about 25 hours of unlabeled speech signals. Then, using only few seconds of labeled speech signals generated with an artificial speech synthesizer, we propose a method to analyze and control the fundamental frequency and the formant frequencies in the latent representation of the previously trained VAE. Our contributions are the following: (i) We experi- mentally demonstrate that the fundamental frequency and the frequency of the first three formants are encoded in orthogonal subspaces of the VAE latent space. This shows that a vanilla VAE trained in an unsupervised fashion able to learn representation that is compliant with the source-filter model of speech production. (ii) We develop a weakly-supervin pendently control the source-filter continuous latent factors of speech variation within the learned subspaces. We put in evidence the orthogonality of these subspaces, which allows us to perform speech transformations
method to precisely and inde-
disentangled manner (i.e., modifying one of the factors does not affect the others). (iii) Without requiring additional information such as text or human-labeled data, we propose a deep generative model of speech spectrograms conditioned on the fundamental frequency and formant frequencies. To the best of our knowledge. this is the first study showing the link be- tween the classical source-filter model of speech production and the representation learned in the latent space of VAE. Thanks to this link, we propose principled method generate speech data controlled with interpretable trajectories (of e.g., fundamental frequency and formant frequencies).
2 BACKGROUND
2.1 VARIATIONAL AUTOENCODER
Generative modeling consists in learning a probabilistic model of observable random variable x E X' C R.D. Let D = [x1...,, XN E X} be a dataset of N = #D independent and identically distributed (i.idd) observations of x. The empirical distribution of is defined by p(x) N Ex,, 8(x - Xz), where ō is the Dirac delta function, which is null everywhere ex- cept in 0 where it takes the value 1.
The variational autoencoder (VAE) (Kingma & Welling, 2014 Rezende et al. 2014) attempts to approximate p(x) with a pdf px(x) parametrized by B. High- nigh--imensional data such as natural images or speech signals exhibit some form of regularity which prevents the D dimensions of x from varying independently from each other. We can thus assume that there exists latent variable z E RR,, with L < D. from which the observed data were generated. Accordingly, the model distribution in the VAE is defined by marginalizing the joint distribution of the latent and observed data, pe po(x) = f po (xlz))(zz))
In this work, the observed data vector x E RDD denotes the power spectrum of short frame of speech signal (i.e., a column of the short-time Fourier transform (STFT) power spectrogram). Its entries are non negative and its dimension D equals the number of frequency bins. We use the
Itakura-Saito VAE (IS- VAE) (Bando et al. 2018 Leglaive et al. 2018 Girin et al. 2019) defined by
p(z) =
(x(x = II Exp [x]a: (vo(2))7').
(1)
d=1
where / and Exp denote the densities of the multivariate Gaussian and univariate exponential distri- butions, respectively, and [v]a denotes the d-th entry of v. The inverse scale parameters of po (x|z) provided by neural network called the decoder, parametrized by 0 and taking z as input.
2

---

Under review as a conference paper at ICLR 2022
The marginal likelihood (x)) and the posterior distribution po(z)x) are intractable due to the non lincarities of the decoder, so it is necessary to introduce an inference model 4o (z(xx)) = po(z)x), which in the VAE is usually defined by
go(2|x)= - N (z; [x(x), (i), diag(v
where the mean and variance parameters are provided by neural network called the encoder network, parametrized by & and taking x as input. Then, the VAE training consists in max- imizing lower-bound of In (x)), called the evidence lower-bound (ELBO) and defined by [(6, Φ) Ex(x) [xx)( [x|z]] - DKL (ga(z)x) p(z))] During training, the generative and inference model parameters and Φ are jointly estimated by maximizing the ELBO, using (vari- ants of) stochastic gradient descent with the so-called reparameterization trick (Kingma & Welling 2014
Rezende et al. 2014).
2.2 SOURCE-FILTER MODEL OF SPEECH PRODUCTION
The source-filter model of speech production Faant 1970) is at the basis of many speech processing systems. It considers that the production of speech results from the interaction of source signal with a linear filter. In voiced speech, the source originates from the vibration of the vocal folds, which produces a quasi -eruiipp glottal sound wave whose fundamental frequency defines the pitch. In unvoiced speech, a noise source is produced by turbulent airflow an acoustic impulse. The source signal is modified by the vocal tract, which is assumed to act as a linear filter. The cavities of the vocal tract give rise to resonances, which are called the formants and characterized by their frequency, amplitude and bandwidth. By moving the speech articulators such as the tongue, lips, and jaw, humans modify the shape of their vocal tract, which results in a change of the acoustic filter and the associated resonances. This is how the different elementary speech sounds called phonemes are produced to form syllables, words and sentences.
The power spectra and the spectral envelopes of two French vowels are displayed in Figure I The spectral envelopes show that the formant frequencies are different for the two vowels. In this ex- ample however, the harmonic structure of the spectra shows that the fundamental frequency is the same for the two vowels. Formant frequencies are important distinctive features of vowels. first approximation, they can be related to the opening of the mouth, the front/rear position of the tongue, and the rounding of the lips for the first, second, and third formant respectively. For voiced phonemes. humans are able to control the formants independently of the pitch (i.e.. to change the filter independently of the source (Fant 1970)) and of cach other (Maalo et al. (01T). The in- dependence of the source and filter characteristics makes the speech signals an interesting material for representation learning methods, especially with deep generative latent-variable models.
a
In the present study, in addition to the pre-traing pre-trained IS- VAE speech spectrogram model, we also assume the availability of an artificial speech synthesizer allowing for an accurate and independent control of the fundamental frequency and formants. In this work, we use Soundgen (Anikin 2019). a parametric synthesizer based on the source-fille model of speech production. For a given speech sound, the voiced component of the source signal is generated by sum of sine waves, the noise component by filtered white noise, and both components are then summed and passed through a linear filter simulating the effect of the human vocal tract. Importantly, this synthesizer allows us to easily generate artificial speech data labeled with the fundamental frequency and formant frequency values.
ANALYZING AND CONTROLLING SOURCE-FILTED FACTORS OF SPEECH VARIATION IN THE VAE
In this section, from a VAE trained on natural speech signals and few criifiialllygeeerateee labeled speech signals, we propose (i) method to learn latent subspaces encoding source-filter factors of speech variation, (ii) a simple strategy to measure the disentanglement of the learned representation, and (iii) weakly-supervised approach to control the continuous factors of variation in the learned subspaces and generate corresponding speech signals.
Let fa denote the speech factor of variation (in Hz) corresponding to the fundamental frequency, for i = 0, and to the formant frequencies, for i E (1, Let D; denote a dataset of artificially- generated speech vectors (more precisely short-term power spectra) synthesized by varying only ft.
3

---

Under review as a conference paper at ICLR 2022
megitutte
60
500
1000
1500
2000
2500
3000
3500
4000
300
3000
1500
2000
2500
5000
35.00
4000
Irequeccy (th)
Inequerccy (hi4
Figure 1: Power spectrum (solid black line) and spectral envelop (orange dashed line) for two vowels
uttered by a male speaker.
all other factors {fz.j * i} being arbitrarily fixed. All examples in Dj are labeled with the factors of variation. would very difficult to build such dataset from existing corpora of unlabeled natural speech. In contrast, it is an easy task using an artificial speech synthesizer such as Soundgen, which precisely takes the fundamental frequency and formant parameters as input, and outputs waveforms from which we extract power spectra.
3.1 LEARNING LATENT SUBSPACES ENCODING SOURCE-FILTER FACTORS OF VARIATION
Let (x) denote the empirical distribution associated with D, defined similarly as p(x). We also introduce the following marginal distribution over the latent vectors:
2 do (2(z))))00) (x)xxx
Σ
2o (2(2x,,))
#D:
X., ED.
In the literature, this quantity is referred to as the aggregated posterior (Makhzani et al. 2016). However, 9 (z|x) is usually aggregated over the empirical distribution p(x) such that the aggregated posterior is expected to match with the prior p(z) (Chen et al., 2018 Dai & Wipf 2018). 8). In contrast, in Equation 3 we aggregate over the "biased" data distribution (x). where we know only one latent factor varies. This defines the explicit inductive bias (Locatello et 2019) that exploit to learn the latent source-filter representation of speech in the VAE.
In the following of the paper, without loss of generality, we assume that, for each data vector in Dj the associated latent vector z has been centered by subtracting the mean vector
n (D.) E
Σ Mx(xp). X,
(4)
DD
Because only one factor varies in D,, expect latent vectors drawn from the "biased" aggregated posterior in Equation (3) to live on a lower-dimensional manifold emhedded in the original latent space R4, We assume this manifold to be subspace characterized by its semi-orthogonal basis
matrix U; E RLXM, < M, < L. This matrix is computed by solving the following optimization problem:
min E (x) UER
[1 - UU-UUTZ] 2
s.t. U.UUU.L.
The space spanned by the columns of U, a subspace the original latent space R& in which the latent vectors associated with the variation of the factor fi in D, are expected to live. In Appendix A| we show that, similarly to principal component analysis (PCA) (Pearson) 1901), the solution to the optimization problem (5) is given by the M, eigenvectors corresponding the M, largest eigenval- ues of
S S #D;
Σ (Ho(xx))oxxxa) X.ED.
- - (D.))
(6)
The dimension Mj of the subspace can chosen such as to retain certain percentage of the data variance in the latent space. Note that the only source of supervision used here is the knowledge that only the factor f. varies in the dataset D,,
3.2 DISENTANGLEMENT ANALYSIS OF THE LATENT REPRESENTATION
As defined by Higgins al. 2018), representation is disentangled if it is possible to learn orthog- onal latent subspaces associated with each factor of variation, whether they are single- or multi- dimensional. The approach presented in the previous subsection exactly follows this definition and
4

---

Under review as a conference paper at ICLR 2022
offers natural and straightforward way to objectively measure if the unsupervised VAE managed to learn a disentangled representation of the factors of variation under consideration. First, by sim- ply looking at the cigenvalues associated with the columns of U, R.X,, we measure the amount of variance that is retained by the projection U.U.- a small number of components M; represents most of the variance, it indicates that only few intrinsic dimensions of the latent space are dedicated to the factor of variation fi and varying this factor can be done by affine transforma- tions. Second, if for two different factors of variation fi and fz. with * j. the columns of U; are orthogonal to those of j. this indicates that the two factors are encoded in orthogonal subspaces and therefore disentangled. It should however be verified experimentally that applying transforma- tions by moving onto the subspace associated with fa generalizes to values of {Jz. J + i} different than the ones used in D:.
3.3 CONTROLLING THE FACTORS OF VARIATION FOR SPEECH TRANSFORMATION AND GENERATION
So far, for each factor fi. we have defined a methodology to learn a latent subspace U, RLXM, that encodes its variations in the dataset D. containing a few examples of speech data generated by an artificial synthesizer. Making now use of the labels in D., we learn a regression model ga, : RR, + R from the factor ft. whose value is denoted by y E I.,, to the data coordinates in the latent subspace defined by U.. The parameters 7/s are defined as the solution of the following optimization problem:
min 0f (a.v) [1I8v(0) -U, 2
(E
(y())) - U.! (mo(xx) (D.)) (x..
Σ
#D:
(7)
vz (z. = f (o((z)))(()) (x, y)xx, p)0) (x. y) is the empirical distribution associated with where 18
D. considering now both the speech data vector x and the label y, and f denotes equality up to an additive constant w.r.t. 77. The dataset D; is very small with only few hundreds examples, and as it is synthetic and labels are not provided by human annotators, the problem can be consid- ered very weakly supervised. For simplicity and because it revealed efficient for this task, gn, is chosen as a piece-wise linear regression model learned independently for each output coordinate m {1, (1.... M,). This choice is supported by the fact that the semi-orthogonal matrix U, decorre- lates the data (Bengio et al. 2013). Solving the optimization problem (7) then basically consists in solving a linear system of equations (Jekel & Venter 2019).
We can now transform a speech spectrogram by analyzing it with the VAE encoder, then linearly moving in the learned subspaces using the above regression model, and finally resynthesizing it with the VAE decoder. Given source latent vector z and a target value y for the factor fs, we apply the following affine transformation:
(8) This transformation consists in (i) subtracting the projection of z onto the subspace associated with the factor of variation fú: and (ii) adding the target component provided by the regression model gm: mapped from the learned subspace to the original latent space by the matrix U, This operation allows us move only in the latent subspace associated with the factor ft. If this subspace is orthogonal to the latent subspaces associated with the other factors { * i}, the latter should remain the same between z and ž, only fa should be modified. This process can be straightforwardly generalized to multiple factors, by subtracting and adding terms corresponding to each one of them.
==z-U,U; z U.gn, (2)
Finally, as the prior p(z) and inference model 4s (z) x) are Gaussian (see Equations and 2 ). the transformation Equation () has the following probabilistic formulation (using U. D):
p (;; f = y/ = A
2; Unn (v), I U.U!) I.U
go (Z)x; = v) = ~ 2; U. (v) + (I -I-UU (x)) (I U,U.) diag diag(v (x)())
(10)
The prior in Equation 9 is now conditioned on the factor fo and can be used to generate speech data given input trajectories of fundamental frequency and formant frequencies. As we assumed centered latent data, the mean vector (D,) defined in Equation 4 must be added to 2 before mapping this vector through the generative model po (x)2).
5

---

Under review as a conference paper at ICLR 2022
Frequancy
Time 66)
(a) Fundamental frequency and formant transform-- (b) Spectrogram generated from input trajectories of tions of the vowel /a/ uttered by female speaker. the fundamental frequency and formant frequencies.
20.00
Frequency (Hz)
50:0
22:0
Time
Time
Time
(c) Left: original spectrogram of a speech signal uttered by female speaker; Middle: transformed spectrogram where the fundamental frequency is set constant over time; Right: transformed spectrogram where the original voiced speech signal (left) is converted into a whispered speech signal (i.c., the pitch is removed).
Qualitative example of modified and generated spectrograms with the proposed method. The color bar indicates the power in dB.
Figure 2:
EXPERIMENTS
This section presents qualitative and quantitative experimental results of the proposed method for controlling the fundamental frequency and formant frequencies of speech signals with VAE. The VAE trained on about 25 hours of multi-speaker speech data from the Wall Street Journal (WSJO) dataset (Garofolo et al., 1993a). The data space dimension is 513 and the latent space dimension is 16. For a given factor of variation, the corresponding latent subspace is learned (see Section [31] using short trajectories of speech power spectra (corresponding to a few seconds of speech) gen- erated with Soundgen (Anikin 2019), all other factors being arbitrarily fixed. When solving the optimization problem (5). the latent subspace dimension M, of each factor of variation is chosen
such that 80% of the data variance is retained. This leads Mo 4, M1 1 and M2 = M₁ = 3. The regression models used to control the speech factors of variation the latent space (see Section 3.3) are learned on the same trajectories, but using the labels that correspond to the input control parame- ters of Soundgen (i.c., fundamental frequency and formant frequencics values). More details on the experimental set-up can be found in Appendix B Given a generated or transformed spectrogram,
we use Waveglow (Prenger et al. 2019) to reconstruct the time-domain signal.
4.1 QUALITATIVE RESULTS
In Figure 2a we illustrate the ability of the proposed method to modify the fundamental frequency and formant frequencies in accurate and independent manner. The spectrogram contains five segments of equal length. The first segment corresponds to the original spectrogram of the steady vowel /a/ uttered by female speaker. In the following segments, we vary successively each indi- vidual factor St. for i = 0 to 3, as indicated by the black lines in the figure. Variations of fo modify the harmonic structure of the signal while keeping the formant structure unaltered. Variations of fr. i E {1, 2,3. modify the formant frequencies (i.c., the vocal tract resopances, as indicated by the color map) while keeping the fundamental frequency unaltered. Figure 2b represents a spectrogram generated by using the conditional prior in equation (9) (generalized to conditioning
multiple fac- the generated speech spectrogram match well with the
tors). We can see that the characteristics input trajectories represented by the lines in the figure. In Figure 2c from left to right we show the original spectrogram of a speech signal uttered by a female speaker (left), and the transformed spec- trograms where the fundamental frequency is set constant over time (middle) and where the pitch has been removed (i.e.. the original voiced speech signal is converted into a whispered speech signal) (right). This last spectrogram simply obtained by subtracting its projection onto the latent

---

Under review as a conference paper at ICLR 2022
Table 1: Performance for pitch (fo) and formant (f1, f2) transformations of English vowels.
0
0.32
0.09
310
Factor
Method
NISOA
MOSnet in 8/oon.
0 m.
325
0:34
10
TD.PSOLA
2.32
2.60 sox
38 a32
6.3 a3a
3.7 =w
3.00
WORLD
2.49
2.46
4.5 4ss
3.7
2.3
5:00
80
21
VAF haseline
zen
6.2 2nx
104
6.2 cm
25
Proposed
2.08 sem
2.55 4ear
7.2
3.6 pm
0:00
07
0
-350
201
-0.55
VAE busdine
2.40 c:0
11.3.24
15.1 135
6.0 212
3:00
-3
Proposed
2.0
2.45 200
6.
84
5.7 gm
-1.00
Figure 3: Correlation matrix of the learned latent subspaces basis vectors.
f2
VAE baseline
2.01 1c0
2.40 cm
19.5 1:22
10.7 1.0
10.9 aM
Proposed
2.03 1c0
2.12 coa
8:5
8.7
6.2 LM
subspace corresponding to fo (i.e., by considering only the two first terms in the right-hand side of Equation (8. It results in a spectrogram where the harmonic component is neutralized. while pre- serving the original formant structure. This is remarkable considering that the VAE was not trained on whispered speech signals, and further confirms that the proposed method dissociates the source and the filter contributions in the VAE latent space. Other qualitative results can be found in Ap- pendix C (visualization of trajectories the learned latent subspaces and additional examples of generated and transformed speech spectrograms) and at hbttss:/tting:: (including audio examples).
4.2 QUANTITATIVE RESULTS
Due to space limitations, we only provide in this section quantitative results related to the funda- mental frequency (o) and the first two formant frequencies (/1 and f2). Additional results can be found in Appendix D including the performance for modifications of the third formant frequency (fa), the performance with a VAE trained on other datasets (c.g., a dataset of French speech signals), and the performance vowel one in terms of phonemes but is not labeled with the fundamental and formant frequencies, making the evaluation less reliable.
the TIMIT dataset (Garofolo et al. 1993b), which is richer than the English
4.2.1 ORTHIOGONALITY OF THE LATENT SUBSPACES
Following the discussion in Section 3.2 we compute the dot product between all pairs of unit vectors in the matrices U; R RL i E 10. 1, Figure 3 shows that the resulting correlation matrix is mainly diagonal. Except for a correlation value of 0.23 across fi and the first component of f2, all other values are below 0.13 (in absolute value), confirming the orthogonality of the learned subspaces and thus the disentanglement of the learned source-filter representation of speech.
4.2.2 PITCH AND FORMANT TRANSFORMATIONS OF ENGLISH VOWELS
Experimental set-up In this experiment, we quantitatively evaluate the performance of the pro- posed method regarding the modification of the fundamental frequency and formant frequencies in speech signals (see Section 3.3). We use a corpus of 12 English vowels uttered by 50 male and 50 female speakers (Hillenbrand et al. 1995). This dataset is labeled with the fundamental and formant frequencies. We transform each signal in this dataset by varying one single factor fa time, for i E {0, 1,2). The ranges of transformation for the fundamental frequency. first formant frequency, and second formant frequency are respectively [100, 300] Hz, [300, 900] Hz, and [1100, 2700] Hz, with step of 1 Hz, 10 Hz and 20 Hz respectively. For the modification of each factor fi. we mea- sure the performance regarding three aspects: First, in terms of accuracy by comparing the target value for the factor (see Equation (8) and its estimation computed from the modified output speech signal. Second, in terms of disentanglement by comparing the values of f, for j * i, before and after modification of the factor f.- Third, in terms of speech naturalness of the transformed signal.
Metrics Accuracy and disentanglement are measured in terms of relative absolute error (in per- cent, the lower the better). For a given factor fi, it is defined by 8f; 100% vl/y where y is
7

---

Under review as a conference paper at ICLR 2022
the target value of f, and 0 its estimation from the output transformed signal. Let us take the example of a modification of fo: 5 measures the accuracy of the transformation on fo while ōf and 6 used to assess if the other factors of variation fi and fz remained unchanged after modifying fo- We use CREPE (Kim et al. 2018) to estimate the fundamental frequency and Parselmouth (Jadoul et al. 2 which is based on PRAAT, to estimate the formant frequencies. Regarding speech nat- urainess, we use the objective measure provided by NISQA (Mittag & Möller (020). This metric (the higher the better) was developed in the context of speech transformation algorithms and it was shown to highly correlate with subjective mean opinion scores (MOS) (i.e., human ratings). As reference, the score provided by NISQA on the original dataset of English vowels (i.e., without any processing) is equal to 2.60 0.53.
Methods We compare the proposed approach with several methods from the literature: (i) Time- domain pitch-synchronous andlad-and-add (TD-PSOLA) (Moulines & Charpentier, (990) per- forms fundamental frequency modifications through the decomposition of the signal into pitch- synchronized overlapping frames. (ii) WORLD (Morise et al., (016) is a vocoder also used for Lundamennall-requeense modifications. decomposes the speech signal into three components char- acterizing the fundamental frequeney, the aperiodicity, and the spectral envelope. (iii) The method proposed by HHsu et al. (2017b) (here referred to as "VAE baseline") consists in applying translations directly in the latent space of the VAE. Unlike the proposed approach, this method requires prede- fined latent attribute representations Jose and jung associated with the source and target values of the factor to be modified, respectively. In particular, computing Mere requires analyzing the input speech signal, for instance to estimate the fundamental frequency, which is not the case for the proposed method. The source and target latent attribute representations are then used to perform the transla- tion 2 = 2 Hun + Mugt, where z and ž are respectively the original and modified latent vectors. To ensure fair comparison, we build dictionaries of predefined latent attribute representations using the same
speech data that were used in the proposed method. All the methods we compare with require a pre-processing of the input speech signal to compute the input trajectory of the factor to be modified (e.g. the fundamental frequency), which is not the case of the proposed method.
training stage of the
Discussion Experimental results (mean and standard deviation) are shown in Table D Compared to the VAE baseline, the proposed method obtains better performance in terms of accuracy, disentan- glement, and naturalness. These results confirm the effectiveness of performing the transformations in the learned latent subspaces and not directly in the latent space, as well as the advantage of using regression models instead of predefined latent attribute representations. Regarding So trans- formation, WORLD obtains the best performance in terms of disentanglement, which is because the source and filter contributions are decoupled in the architecture of the vocoder. In terms of nat- urainess, WORLD and then TD-PSOLA obtain the best performance. This may be explained by the fact that these method operate directly in the time domain, therefore they do not suffer from phase reconstruction artifacts, unlike the proposed and VAE baseline methods. Naturalness is in- deed greatly affected by phase reconstruction artificats, even from an unaltered speech spectrogram (i.e., without transformation). Phase reconstruction in a multi-speaker setting is still an open prob- lem in speech processing. We want to emphasize that the objective of this study is not to compete with traditional signal processing methods such TD-PSOLA and WORLD. It is rather to advance on the understanding of deep generative modeling of speech signals and to compare honestly with highly-specialized traditional systems. TD-PSOLA and WORLD DR exploit signal models specifically designed for the task at hand, which for instance prevents them to be used for modifying formant frequencies. In contrast, the proposed method is fully based on learning and the same methodology applies for modifying the fundamental or formant frequencies.
5 RELATED WORK
Variants of the VAE have recently led to considerable progress in disentangled representation learn- ing (Kim & Mnih 2018 Higgins et al. 2017, Chen et al.) 2018). From experimental analyses on image data, these methods suggest that a vanilla VAE cannot learn a disentangled representation. In the present study, we experimentally show that learning a disentangled cource-filted representation of speech using simple VAE is possible, complying with the definition of disentanglement pro- posed in (Higgins et al., 2018). Locatello et al. (2019 2020) recently showed both theoretically and from a large-scale experimental study that the unsupervised learning of disentangled representations
8

---

Under review as a conference paper at ICLR 2022
is impossible without inductive biases both the models and the data. In the present study, pre- cisely employ a few examples of aefficially-enerrate labeled speech data in order to disentangle the latent representation of simple VAE, in terms of source-filter factors of speech variation.
Modifying speech signals characteristics such as the pitch or timbre is a highly covered research topic where deep generative modeling has recently emerged
promising approach. Voice con- version consists in modifying speaker characteristics while keeping the linguistic information un- changed, e.g. (Kaneko & Kameoka 2018 Kaneko et al., 2019: Kameoka et al. 2020). Many voice conversion methods use text information as input (Wang et al. 2018: Zhang et al. [2019] Kulkarni al. 2021, Li et al., 202I), which greatly helps for transferring non-verbal vocal attributes from one speech signal to another, but it requires datasets labeled with text. Qian et al. (2020) and Webber et al. (2020) recently proposed speech transformation methods based on latent representation learn- ing without resorting to textual information. The interpretability of the learned representations is enforced by the design of the model. Qian et al. (2020) proposed to use three independent encoder networks to decompose speech signal into fundamental frequency, timbre and rhythm latent rep- resentations. Webber et al. (2020) focused on controlling source-filter parameters in speech signals, where the ability to control given parameter (e.g. fundamental frequency) is enforced explicitly using labeled data and adversarial learning. In this approach, each parameter to be controlled re- quires a dedicated training of the model. Moreover, these methods are speaker-dependent
speech
generation in Qian et al. (2020) is conditioned on the speaker identity and Webber et al.] (2020) used a single-speaker training dataset. This contrasts with the proposed method which speaker- independent, and in which the source-filter representation of speech naturally emerges in the latent space of a single unsupervised VAE.
Hsu et al. b)) proposed to use VAES for modifying the speaker identity and the phonemic content of speech signals. In (Hsu al., 2017b), this is achieved by translations the latent space. This is the baseline VAE approach we presented and compared with in Section[4] It requires to know predefined values of the latent representations associated with the source/target speech attributes to be modified. Conversely, Equation (8) in the proposed method does not require the knowledge of the factor f. associated with source vector z, it only requires the value associated with the target vector ž. Moreover, Hsu et al. (2017b) did not address the control of continuous factors of speech variation in the VAE latent space, contrary to the present work.
Several methods have been recently proposed to control continuous factors of variation in deep gen- erative models (Jahamian et al. 2019; Plumerauut et al., 2020) Goetschalckx et al.) 2019 Härkööen let al. 2020), focusing essentially on generative adversarial networks. They consist in identifying and then moving onto semantically meaningful directions in the latent space of the model. The present work is inspired by (Plumerault al., 2020), which assumes that factor of variation can be predicted from the projection of the latent vector along a specific axis, learned from artificially generated trajectories. The proposed method is however more generic, thanks the learning of latent subspaces associated to the latent factors and to the introduction of general formalism based on the use of "biased" aggregated posteriors. Moreover, these previous works on controlling deep generative models only allow for moving "blindly" onto semantically meaningful directions in the latent space. In the present study, we are able to generate data conditioned on a specific target value for a given factor of variation. Finally, previous work focused on image data, and to the best of our knowledge, this is the first approach to control source-filter factors of speech variation in a VAE.
CONCLUSION
In this work, using only a few seconds of artificially generated labeled speech data, we showed that the fundamental frequency and formant frequencies are encoded in orthogonal latent subspaces of an unsupervised VAE and we proposed weakly-supervised method to control these attributes within the learned subspaces. The method generalizes well when applied to natural speech signals. To the best of our knowledge, this is the first approach that, with a single methodology, is able to extract, identify and control the source and filter low-level speech attributes within a VAE latent space. The proposed method could also be applied to other types of data such as natural or face images, provided that one can create few synthetic images that capture variations in single latent factor of interest, independently of others. For example, this would be possible for factors encoding the position, rotation and scale of objects in natural images, or some facial expressions.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
To ensure reproducibility, the implementation of the method and the data generated with soundgen that are used to learn the latent subspaces and regression models will be made freely available online.
REFERENCES
Andrey Anikin. Soundgen: An open-source tool for synthesizing nonverbal vocalizations. Behavior Research Methods, 51(2):778-792 2019.
Yoshiaki Bando, Masato Mimura, Katsutoshi Itoyama, Kazuyoshi Yoshii, and Tatsuya Kawahara. Statistical speech enhancement based on probabilistic integration of variational autoencoder and non-negative matrix factorization. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 716-720, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1788,887,, 2013.
Christopher M. Bishop. Pattern recognition and machine learning. Information science and statis- tics. Springer, 2006. ISBN 9780387310732.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. Compressed sensing using genera- tive models. In International Conference on Machine Learning (ICML), pp. 537-546, 2017.
Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Hycong-Seoo Choi, Juhcon Lee, Wansoo Kim, Jie Hwan Lee, Hoon Heo, and Kyogu Lee. Neu- ral analysis and synthesis: Reconstructing speech from self-supervised representations. arXiv preprint arXiv:21 1t..:2/10.1451... 14513, 2021.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. International Conference on Learning Representations (ICLR), 2018.
Kate Dupuis and M Kathleen Pichora-Fuller. Toronto emotional speech set (tess)-younger talker.angry. 2010.
Gunnar Fant. Acoustic theory of speech production. Number 2. Walter de Gruyter, 1970.
John S. Garofolo, David Graff, Doug Paul, and David Pallett. CSR-I (WSJO) Sennheiser LDC93S6B. httpSSB...tttt:////. http://///aaal.g...o 1dc upenn. edu/LDC93S6B 1993a. Philadelphia: Lin- guistic Data Consortium.
John Garofolo, Lori F. Lamel, William M. Fisher, Jonathan G. Fiscus, David S. Pallett, Nancy L. Dahlgren, and Victor Zue. TIMIT acoustic phonetic continuous speech corpus. In Linguistic data consortium, 1993b.
Laurent Girin, Fanny Roche, Thomas Hueber, and Simon Leglaive. Notes the of variational autoencoders for speech and audio spectrogram modeling. In International Conference on Digital Audio Effects (DAFx), pp. 1-8, 2019.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. GANalyze: Toward visual defi- nitions of cognitive image properties. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2014.
10

---

Under review as a conference paper at ICLR 2022
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANSpace: Discovering interpretable gan controls. In Advances in Neural Information Processing Systems (NeurIPS), pp- 9841-9850, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-vae: Learning basic visual concepts with a constrained variational framework. International Conference on Learning Representations (ICLR), 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo J. Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 2018.
James Hillenbrand, Laura A. Getty, Michael J. Clark, and Kimberlee Wheeler. Acoustic charac- teristics of American English vowels. The Journal of the Acoustical society of America, 97(5):
3099-3111, 1995.
Fierre-Edouard Honnet, Alexandros Lazaridis, Philip Garner, and Junichi Yamagishi. The siwis french speech synthesis database? design and recording of a high quality french database for speech synthesis. Technical report, Idiap, 2017.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances in Neural Information Processing Systems (NeurIPS), 2017a
Wei-Ning Hsu, Yu Zhang, and James Glass. Learning latent representations for speech generation and transformation. In Interspeech, pp. 1273-1277, 2017b.
Keith Ito and Linda Johnson. The lj speech dataset.
https://keithiito..
LJ- Speech- 2017.
Yannick Jadoul, Bill Thompson, and Bart de Boer. Introducing Parselmouth: A Python interface to Praat. Journal of Phonetics, 71:1-15, 2018
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
Vivek Jayaram and John Thickstun. Source separation with deep generative priors. In International Conference on Machine Learning (JCML). pp. 4725-4735, 2020.
Charles F. Jekel and Gerhard Venter. Pwlf: a python library for fitting ID continuous piecewise lin- linear fit py/
ear functions. URL: URL::attss:///ithuuu wwwp////wwe//eeeeeeeeeoeeeeeeeee
wtww/mastter/pppe///aa Jekel Venter v2 pdf 2019.
Lauri Juvela, Bajibabu Bollepalli, Vassilis Tsiaras, and Paavo Alku. GlotNet raw waveform model for the glottal excitation statistical parametric speech synthesis. /EEF/ACM Transactions on Audio, Speech, and Language Processing, 276)):019-11030, 2019.
Hirokazu Kameoka, Li,, Shota Inoue, and Shoji Makino. Supervised determined source separation with multichannel variational autoencoder. Neural computation, 119911189111914, 2019.
Hirokazu Kameoka, Wen-Chin Huang, Kou Tanaka, Takuhiro Kaneko, Nobukatsu Hojo, and Tomoki Toda. Many-to-many voice transformer network. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:656-670, 2020.
Takuhiro Kaneko and Hirokazu Kameoka. Cyclegan-ve: Non-parallel voice conversion using cycle- consistent adversarial networks. European Signal Processing Conference (EUSIPCO), pp- 2100-2104, 2018
Takuhiro Kancko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. Stargan-vc2: Rethinking conditional methods for strgan-basee voice conversion. arXiv preprint arXiv: 1.07:1197..1277 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning (ICMI., pp. 2649-2653 2018.

---

Under review as a conference paper at ICLR 2022
Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional represen- tation for pitch estimation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 161-165, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Confer- ence on Learning Representations (ICLR), 2014.
Ajinkya Kulkarni, Vincent Colotte, and Denis Jouvet. Improving transfer of expressivity for end- to-end multispeaker text-to-speech synthesis. In European Signal Processing Conference (EU- SIPCO), 2021.
Juheon Lee, Hyeong-Seok Choi, Chang-Bin Jeon, Junghyun Koo, and Kyogu Lee. Adversarially trained end-to-end korean singing voice synthesis system. In Interspeech, 2019.
Simon Leglaive, Laurent Girin, and Radu Horaud. A variance modeling framework based varia- tional autoencoders for speech enhancement. In IEEE International Workshop on Machine Learn- ing for Signal Processing (MLSP), pp. 1-6, 2018
Tao Li, Shan Yang, Liumeng Xue, and Lei Xie. Controllable emotion transfer for end-to-end speech synthesis. In International Symposium on Chinese Spoken Language Processing (ISCSLP), pp.
1-5, 2021.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In International Conference on Machine Learning (ICML), pp. 4114 4124, 2019.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled repre- sentations and their evaluation. Journal of Machine Learning Research, 21:1-62, 2020.
Ewen N.
MacDonald, David W. Purcell, and Kevin G. Munhall. Probing the independence of for- mant control using altered auditory feedback. The Journal of the Acoustical Society of America, 129(2):955 965,,,, 2011.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. In ICLR 2016 Workshop, International Conference Learning Representations (ICLR), 2016.
Gabriel Mittag and Sebastian Möller. Deep learning based assessment of synthetic speech natural- ness. Interspeech, 2020.
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: vocoder-based high-quality speech synthesis system for real-time applications. IEICE Transactions on Information and Systems, 99
(7):1877-1884, 2016.
Eric Moulines and Francis Charpentier. Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones. Speech communication, 9(5-6):453444 1990.
Karl Pearson. On lines and planes of closest fit to systems of points in space. The London, Edin- burgh, and Dublin philosophical magazine and journal of science, 2(11):559-572, 1901.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with con- tinuous factors of variations. In International Conference Learning Representations (ICLR), 2020.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621, 2019.
12

---

Under review as a conference paper at ICLR 2022
Kaizhi Qian, Yang Zhang, Shiyu Chang. Mark Hasegawa-Johnson, and David Cox. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning (ICML), pp. 7836-7846, 2020.
Lawrence Rabiner and Ronald Schafer. Theory and applications of digital speech processing. Pren- tice Hall Press, 2010.
Danilo J. Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. International Conference on Machine Learning (ICML), pp. 1278-1286, 2014.
Jean-Marc Valin and Jan Skoglund. Lpenet: Improving neural speech synthesis through linear pre- diction. In JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5891-5895, 2019.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter waveform models for sta- tistical parametric speech synthesis. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:402-415, 2019.
Yuxuan Wang, Daisy Stanton, Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and trans- fer end-to-end speech synthesis. In International Conference on Machine Learning (ICML), pp. 5180-51888 2018.
Jacob J Webher, Olivier Perrotin, and Simon King. Hider-finder-combiner: An adversarial architec- ture for general speech signal modification. In Interspeech, pp. 2206-3210, 2020.
Ya-Jie Zhang, Shifeng Pan, Lei He, and Zhen- Ling. Learning latent representations for style control and transfer in end-to-end speech synthesis. In IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pp.
2019.
13

---

Under review as a conference paper at ICLR 2022
A SOLUTION TO THE LATENT SUBSPACE LEARNING PROBLEM
In this Appendix, we show that the solution to the optimization problem e is given by the principal eigenvectors of (D,) in Equation 66.
Without loss of generality, we formulate the problem for a centered version of the latent data
レ 16 (Di),
(11)
where (D;) is defined in Equation (4) This centering also affects the inference model originally
defined in Equation 2 as follows:
ga(z|x) N Ho((x) (D,)), diag(v)(x)).
(12)
Using Equation the fact that UU UT-- and Equation 12 the cost function in the optimization
problem (5) can be rewritten as follows:
E 10 (x) || UU
2|
[| UUT =112]
no (x/xx..)
#D=
(I
UUT)
E Eys 22 1
#D:
En
tr ((I UU )
(13)
where (D,) is defined in Equation problem (5) equivalent to
6 From this last equality, that the optimization
max UERLXM;
tr UU (D.)U,
s.t. U
(14)
Very similarly to PCA (Pearson 1901), the solution is given by the M, dominant eigenvectors of (D:) (i.e., associated to the M, largest eigenvalues) (Bishop 2006 Section 12.1).
B EXPERIMENTAL SETUP
B.I VAE TRAINING
To train the IS-VAE model (Bando et al. 2018 Leglaive et al. 2018 Girin al. 2019). the Wall Street Journal (WSJO) dataset (Garofolo et al. 1993a) which contains 25 hours of speech signals sampled 16 kHz, including 52 female and 49 male speakers. The time-domain spech signals are converted to power spectrograms using the short-time Fourier transform (STFT) with an analysis window of length 64 ms and an overlap of 75%. The architecture of the VAE is shown in Figure 4 the encoder and decoder networks each have three dense layers. A hyperbolic tangent (tanh) activation function is used each layer, except for the output layers of the encoder and decoder where we use the identity function. The VAE input/output dimension is D 513 (we only keep the non redundant part of the power spectrum at a given time frame) and the latent vector dimension is set to L 16. We train the model using the Adam optimizer (Kingma & Ba 2015) with a learning rate equal to 0.001.
B.2
ANALYZING AND CONTROLLING SOURCE-FILTER FACTORS OF SPEECH VARIATION IN THE VAE
For given factor of variation, the corresponding latent subspace is learned (see Section 3.1 using trajectories of speech power spectra generated with Soundgen (Aníkin 2019), all other factors being arbitrarily fixed. For fo. the trajectory contains 226 points (which corresponds to 3.6 seconds of speech) evenly spaced the range (85,310) Hz, fi, f2 and fa are fixed to 600 Hz, 1500 Hz, and 3200 Hz, respectively. For fi, the trajectory contains 401 points (which corresponds to 6.4 seconds of speech) evenly spaced in the range [200, 1000] Hz, fo- fz and fa are fixed to 140 Hz, 1600 Hz, and 3200 Hz, respectively. For fz, the trajectory contains 401 points evenly spaced in the range [800, 2800] Hz, for fi and fa are fixed to 140 Hz, 500 Hz, and 3200 Hz, respectively. For fa. the trajectory contains 241 points (which corresponds to 3.9 seconds of speech) evenly spaced in the
14

---

Under review as a conference paper at ICLR 2022
Encoder
Deroder
Reparometization
Activation function tanh
FC
x
FC
FC
FC
FC 256
FC
X
256
64
FC
FC
fully connected linear layer
90 x
Po
Figure 4: Architecture of the VAE. The number of neurons for each fully-connected layer is indi- cated below "FC". within each gray box. The input/output vector dimension of the VAE is 513 and the latent vector dimension is 16.
range [2000, 3200] Hz. fo. fi and f2 are fixed to 140 Hz, 500 Hz, and 1200 Hz, respectively, The amplitude of the formants set 30 dB, and the bandwidth automatically calculated from the fromant frequencies by Soundgen, using formula derived from human phonetic research (Anikin 2019). The regression models used control the speech factors of variation the latent space (see Sections 3.3) are learned on the same trajectories, but using the labels that correspond to the input control parameters of Soundgen.
C ADDITIONAL QUALITATIVE RESULTS
C.I VISUALIZATION OF THE LEARNED LATENT SUBSPACES
respectively obtained by projecting the latent mean vectors (x(x) E RT, for all data vectors x E DD within the latent subspace characterized by U. R. (i.e., we perform dimensionality reduction). In the previously reported experiments, the latent subspace dimension M, of each factor of variation variance was retained in the latent space. It resulted Mo = 4, M₁ = and M2 = M3 = 3. In this section, for visualization purposes, we M, for all E {0, 1, 3). However, the fi trajectory (Figure 5b) is mainly concentrated along a single axis. Regarding fo (Figure (5a). setting My 3 retained 78% of the variance of Do in the latent space. An important observation that we make from these figures is that two data vectors x and x', corresponding to values of a given factor that are close, have projections of μs (x) and (x(x') that are also close in the learned latent subspaces. This can be seen from the color bars which indicate the values of the factors of variation. indicates that the learned representation preserves the notion proximity in terms of fundamental frequency and formant frequencies.
For = 0, 1,2 and 3, Figures 5b
5d
chosen such that 80% of the data
that
In Figure 5e we project three different datasets D,, defined for three different values of f2- Sim- ilarly, in Figure [i] we show the trajectories associated with the projection of three datasets D2. defined for three different values of fi. We notice that as expected, the trajectories are almost iden- tical and only differ by translation.
15

---

Under review as a conference paper at ICLR 2022
300
900
250
800
425
0.25
700
200
0.00
600
0.25
500
400
125
300
108
200
(a) trajectory corresponding to Do.
(b) fi trajectory corresponding Dj.
1.0
2380
0.0
13
18
-05
2000
1250
28
14
1.0 0.5 0.0-0.5-1.11
03.
(c) trajectory corresponding to D2.
(d) fa trajectory corresponding Dz.
1.0
2.0
1.5
1.5
1.0
0.5
0.5
0.0
0.0
05
-10
0.00..5..00551..1.....
-2
(e) Three trajectories for fi, associated with dif- ferent values f2.
(1) Three trajectories for associated with dif- ferent values f1.
Figure 5: Visualization of trajectories in the learned latent subspaces.
16

---

Under review as a conference paper at ICLR 2022
EXAMPLES OF GENERATED SPEECH SPECTRA C.2
maghtb.c (68)
Trequency (FF2)
mageitude (d8)
magrib.dc
Trequency (Ha))
nagnzude (d8)
mognntutt
frequency (ff2)
frequency
(a) Generated spectra with different values of fo-
(b) Generated spectra with different values of fr.
nagituue (3B)
frequency (H2)
magnttute (d8)
frequency (192)
Troquency (H2)
mouuteeeee (09)
frequeriyy (H2)
frequeency (ffic)
(c) Generated spectra with different values of f2-
(d) Generated spectra with different values of fs-
Figure 6: Power spectra (solid black line) and spectral envelopes (dashed orange line) obtained using the conditional prior in equation (9) (generalized conditioning multiple factors). Each subfigure contains three plots where we vary the value of one single factor at time: fo in (a), fi in (b), fa in (c) and fa in (d).
17

---

Under review as a conference paper at ICLR 2022
EXAMPLES OF TRANSFORMED SPEECH SPECTROGRAMS C.3
2300
Frequency (Hz)
2500
-0 c0
1500
2000
dB
Time
(a) Original spectrogram of the vowel /ae/ uttered by a male speaker.
Frequancy (H2)
414
-1:30
Time (5)
100
Frequency (62)
414
100
Time 15)
Time (5)
(b) Transformation of the fundamental frequency fo-
(c) Transformation of the 1st formant frequency
Frequency (H22
1808
3508
31.4
308
Time
Time (s)
Frequency (12)
31.8
334
09
Time
Time (6)
Transformation of the 2nd formant frequency f2. (e) Transformation of the 3rd formant frequency fa.
Figure 7: Figure (a) shows the spectrogram of a vowel uttered by male speaker. Figures (b), (c), (d) and (e) show transformations of this spectrogram with the proposed method, where we vary fo- fi. fz, and fs, respectively. The target value for these factors is indicated by the dashed blue line.
18

---

Under review as a conference paper at ICLR 2022
3000
2582
2003
155
15
Time (s)
Time (s)
Time (s)
Frequency (H12)
1300
1300
Time
4300
Frequency (H2)
200
Time
6300
3500
Frequency (H2)
3900
300
-70
25
26
25
Time 25
25
Time
Time (s)
3000
Frequency
2566
166
05
Time (s)
Time (s)
Time (s)
Figure 8: Each line in this figure corresponds to speech signal uttered by different speaker. Left: spectrogram of the original speech signal: Middle: transformed spectrogram where the fundamen- tal frequency is set constant over time; Right: transformed spectrogram where the original voiced speech signal (left) is converted into whispered speech signal (i.e., the fundamental frequency is removed).
19

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL QUANTITATIVE RESULTS
D.1 TRANSFORMATIONS OF THE THIRD FORMANT FREQUENCY ON THE ENGLISH VOWELS DATASET
Table 2 completes Table by including transformations of the third formant frequency in the range [2200, 3200 Hz, with a step 20 Hz. The new column ff also measures the performance in terms of disentanglement when modifying other factors fs fs- These additional results do not modify the conclusions drawn in Section
Table 2: Performance (mean and std) for the fundamental frequency (fo) and formant frequencies (f1, f2 and fa) transformations of English vowels.
Factor
Method
NISQA (t)
850 (9.
8fi (%..))
8.2
8/3 (%..4)
fo
TD-PSOLA
2.32
3.8 25
6.3
3.7
2.1
WORLD
2.49 0.6
4.5 =
3,7 1.88
2.3 1.07
1.2 +:
VAE baseline
1.94
6,21 128
10.4 2:44
6.2 1.9
4.5 0:02
Proposed
2.08 0.48
0.8 0.2
7.2 133
3.6 3.6
3.8 0.3
fi
VAE baseline
1.84 0.5
11.3 4:22
15.1 4.35
6.0 + 12
4.2 + 0.4
Proposed
1.85 0.4
6.0
8.4 3:2.
5.7
4.4
f2
VAE baseline Proposed
2.01 0.4 2.03 0.43
19.5 32
10.7 a.
10.9
5,8 0.6
8.5 =
8.7 L.
6.2 + 1.5
5.8 +: 0.2
VAE baseline
1.82 1.94 + 0.48
27.0
13.0 1.3
12.0
7.3
Proposed
8.3 = 1.0
8.6 +
4.9 0.0
2.0 = 04
D.2 ROBUSTNESS WITH RESPECT TO DIFFERENT TRAINING DATASETS FOR THE VAE
This Section investigates the robustness of the proposed method with respect to different datasets used train the VAE model. This table presents results for modifications of the fundamental fre- quency only, applied to the English vowels dataset. We considered the following training datasets:
WSJ: the Wall street Journal (Garofolo et al. 1993a dataset that was used in the previous experi- ments.
SIWIS: the SIWIS French speech synthesis dataset (Honett et al. 2017 which contains more than ten hours of French speech recordings.
TESS: the Toronto emotional speech dataset (Dupuis & Pichora-Fuller 2010 which contains 2800 utterances spoken by two actresses using different emotions (anger, disgust, fear, happiness, pleasant surprise, sadness, and neutral).
LJspeech: the IJspeech dataset (Ito & Johnson 2017). which contains 13100 short audio clips of single speaker reading passages from 7 non-fiction books.
The artificially-generatedd speech dataset used for learning the latent subspaces and the regression models along with the test dataset of English vowels remain the same. It can be seen in Table that the performance remains quite stable with different VAE training datasets. WSJ is the largest dataset and therefore obtains the best performance. Interestingly, the results obtained with the SIWIS dataset French speech signals remain satisfactory, even there is a mismatch between the training (French) and testing (English) datasets.
20

---

Under review as a conference paper at ICLR 2022
Performance (mean and std) for fundamental frequency (fo) transformations of English vowels using different datasets for training the unsupervised VAE model.
Table 3:
Dataset
NISQA (t>
f (..4)
8f1 (8.
ō (8,
0.3 (8.
WSJ
2.08 = 0.48
0.8 0.2
7.2 1.3
3.6 1
3.8 0.
SIWIS
1.93
1.2 1.5
10.0 4
2.3
14.0 =2
TESS
1.98 = 0,50
2.7 233
9.3 135
9.0 =.08
7.0 #:02
LJspeech
1.96
1.2
9.3 1.2
5.6
4.6 4:01
D.3 EVALUATION ON THE TIMIT DATASET
In this Section, evaluate the proposed method the TIMIT dataset (Garofolo et al. 1993b), using the VAE trained on the WSJ dataset. TIMIT a corpus of phonemically and lexically tran- scribed speech of American English speakers of different sexes and dialects. We used the test corpus containing 1680 utterances. Because we are interested in studying the interaction between modifi- cations of the fundamental frequency and formant frequencies, phonemes that are voiced (40 phonemes over a total of 52), which can be identified using the anno- tations. The ranges of transformation for the fundamental frequency, first, second and third formant frequencies are respectively [100, 300) Hz, [300, 900] Hz, [1100, 2700] Hz, and [2200, 3200] Hz with step of 10 Hz, 50 Hz, 100 Hz and 50 Hz, respectively. The metries and evaluated methods remain the same as those described in Section 4.2.2
only evaluate the method on the
TIMIT phonemically richer than the English vowels dataset previously used, however it is not labeled with the fundamental and formant frequencies. Therefore, we do not have the ground truth values which makes the evaluation more difficult than with the English vowels dataset. Instead of the ground truth, we use the formant frequencies and the fundamental frequency computed on the original speech utterances (i.c., before transformation) using the same tools as described in Section 4.2.2 This makes the evaluation on TIMIT less reliable than on the English vowels dataset, but it allows us to test the methods on larger variety of phonemes.
Results are presented in Table 4 They very consistent with the ones obtained on the English vowels dataset (see Tables I and 2b. which further strengthens the conclusions drawn in the discus-
sion paragraph of Section 4.2.2
Table 4: Performance (mean and std) for the fundamental frequency (fo) and formant frequencies (f1, f2 and fa) transformations on the TIMIT dataset.
Factor
Method
NISQA (t)
fo (8.4)
8,f1 (8.4)
8f2 (%.
8fs (X.
fo
TD-PSOLA
2.36 = 0.50
2.4 i 19
7.9 206
4.5 = 0.
3,9 1
WORLD
2.45 0.47
0.3 +.1
7.1 1
6.2
4.2 0
VAE baseline
1.59 0.43
16.1 =6.3
17.0 430
12.1 02
10.9 1:
Proposed
2.28 0.57
0.8 106
9.1 111
8.3
6.0 1 188
fi
VAE baseline
1.42 = 0.54
10.1 =8
16.4 +
12.4 +
11.2 22
Proposed
1.66 0.31
7.1 136
9.2 = 2.8
9.0
7.8 + L.
f2
VAE baseline Proposed
1.46 0.50
19.3 500
16.4 108
20.3 163
11.5 a.
1.49 = 0.50
9.1 222
8.3 1
4.3 =13
8.1 422
fa
VAE baseline Proposed
1.40 0:48
20.4 1.0
17.4 102
14.4 1
11.7 2:2
1.48 0.42
8.5 + 1.9
8.7 09
5.7 =11
2.5 188
21


Reviews:
[{"review_id": "zuD4vNVLIxz", "reviewer": "Reviewer_vTAw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n\nThe fact that one can control the fundamental frequency and formant frequencies of speech by manipulating the latent space is an interesting finding.\nFinding the subspaces of latent the VAE latent space using artificially controllable dataset is an interesting idea.\n\n\nweaknesses:\n\nIt's interesting work to someone who is working in the speech signal processing field.\nHowever, I wonder how much implication this work could give to general audiences.\nThe fact that the VAE latent space can be manipulated using an artificially controllable dataset is interesting but at the same time, it is a disadvantage because we cannot find the subspaces if the artificially controllable dataset cannot be acquired.\n", "summary_of_the_paper": "This paper shows that the fundamental frequency and formant frequency information is encoded in a speech VAE model.\nThis can be found by using artificially controlled/generated dataset.\nAfter finding how to manipulate the latent space, one can control arbitrary speech samples in a desirable way, such as controlling fundamental frequency or formant frequencies.\nThe authors also show that the harmonic parts can be removed from the original speech and reconstructed as a whispered voice using only the spectral envelope part.\nExperiment results show that the proposed method can control formant frequencies and show that it can control better than the method proposed by previous work.\nThe proposed method show on par or worse performance on fundamental frequency control experiment when compared to traditional DSP vocoders, such as TD-PSOLA or WORLD.\n", "summary_of_the_review": "This paper shows interesting findings on speech VAE.\nThe proposed method is straightforward and I think it can draw some attention from the speech signal processing field.\nI recommend [6: marginally above the acceptance threshold].\n\nQuestions:\n1. Why was IS-VAE chosen to model speech?\n2. On Page 3, Section 3, second paragraph, \"Di denote a dataset of artificially- generated speech vectors (more precisely short-term power spectra) synthesized by varying only fi, all other factors {fj , j ̸= i} being arbitrarily fixed\", does this mean that when making D_0, f1, f2, and f3 is fixed? If that's true what values for f1, f2, and f3 are chosen?\n3. Can we still find subspaces when we do not have the artificial generator? (e.g., by manually selecting speech samples we would like to take control of?)\n\nReferences:\nSome papers the authors can consider to add in the reference:\n[1] shows that one can disentangle source and filter in a supervised manner.\nRecently [2] showed that the source and filter parts can be disentangled even in a self-supervised setting. \n\n[1] https://arxiv.org/abs/1908.01919\n[2] https://arxiv.org/abs/2110.14513\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "z-B1OnWOtp_", "reviewer": "Reviewer_e7Cb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- This paper could be seen as an extension of Hsu et al. (2017a), both of which found that unsupervised VAEs learn to model speech properties such as formants in orthogonal subspaces. To transform speech, both consider an affine transformation that subtracts the component of the original property ($U_i U_i^T z$ in this paper and $\\mu_{src}$ in Hsu et al. (2017)) and then add the component of the target property ($U_i g_{\\eta_i} (y)$ in this paper and $\\mu_{tgt}$ in Hsu et al. (2017)). There are two methodological improvements in this paper. First, the authors find the subspace modeling for each property U using labeled data, such that the subtraction can be done for an unlabeled instance. Second, a regression model is built to predict the coefficients of the target value for each property, such that this method can in principle generalize to unseen values for a property.\n- Empirically, the proposed method demonstrates better source and filter parameter control compared to the VAE baseline (Hsu et al., 2017).\n\nWeaknesses\n- The main contribution is determining latent subspaces for F0 and formants and building a regression model to predict the coefficients of the components for each subspace. It offers limited real world applications since the performance still lags behind traditional signal-processing based methods if these attributes (source and filter parameters) were to be modified. The novelty is also limited and might have limited interest for the machine learning community. Specifically, the orthogonal subspace property has been shown in computer vision and speech modeling, and the authors have only demonstrated that the proposed method can be used to control a limited set of speech properties.\n", "summary_of_the_paper": "This paper analyzes the latent representations of speech spectrograms learned by unsupervised variational autoencoders (VAEs) and discovers that the VAE learns to model the variation of fundamental frequencies (F0/source) and formant frequencies (filters) using orthogonal subspaces. Based on the discovery, the authors further propose to build a regression model that maps target value of a property to the factor loading coefficients of the subspace corresponding to that property, which enable control of properties independently.", "summary_of_the_review": "This paper presents a nice extension from the literature. However, the novelty and applicability is limited and the generality of the proposed method for controlling other attributes is unclear. Hence, this could be of limited interest to the audience of the conference", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DmvCsvKK-C0", "reviewer": "Reviewer_pLaG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is interesting in analyzing the subspaces of the latent space that correspond to each formant and the pitch in a trained VAE. The subspaces were obtained using synthesized speech which is not ideal and it is not clear whether the method would work well on all speech data. Another possible issue is that the model performance is analyzed with very simplistic data: English vowels. It was not tried on continuous real speech utterances. Also, I am not totally convinced that this latent space post-analysis way is the best way to obtain disentangled latent information. It could be possible to have multiple embeddings that correspond to each fj from the start. How would this approach compare to the post-analysis approach? The subspaces are found to be \"mostly\" orthogonal but not exactly, so there may be some leakage between subspaces.\n\nSome itemized questions and recommendations:\n1. Dimension D of x_i is not defined before it was mentioned in line 7 of Section 2.1. Maybe instead of {\\cal X}, you can directly indicate the vector space for the features.\n2. In Section 3.3, the justification for linear regression for predicting the subspace coefficients is not clear to me. Why would the subspace coefficients be a scalar multiple of the fj value? Would it not work better if you used a multi-layer nonlinear prediction?", "summary_of_the_paper": "This paper analyzes VAE latent embeddings to extract subspaces that relate to pitch (f0) and formant frequencies (f1 through fN). This is done through first training a frame-synchronous  IS-VAE model from clean speech data. The authors pass controlled synthesized speech through the model and obtain the embeddings corresponding to the synthesized data with randomly varying one fj component and keeping others constant. Each embedding corresponds to a frame of spectral data. The set of embeddings are then analyzed by PCA to find the principal eigenvectors corresponding to each fj explaining 80% of the total energy. The mapping from f0, f1, .. fN values and the subspace coefficients is done through a linear regression mapping independently for each fj which is also learned from synthetic data. Through simple examples, the authors show that they can change individual components (f0, f1, .., fN) by modifying the embeddings correspond to a signal, to some extent. They compare their results with rule-based vocoders (WORLD, TD-PSOLA) and a VAE baseline that does not use subspaces.", "summary_of_the_review": "The analysis looks interesting, but the training and eval data used for this analysis looks too simplistic. A direct disentangled representation could also be possible with an easy way to make smaller embeddings corresponding to each fj.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "56VxUr-PO5h", "reviewer": "Reviewer_kWcG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - Clear and well written introduction to the source-filter model of speech production.\n - Clever method for identifying and controlling disentangled subspaces given supervised labels for them.\n - Convincing quantitative and qualitative demonstration in Section 4 that the proposed approach 1) learns a representation which is well disentangled and 2) can independently control each component. \n \nWeaknesses:\n - Experiments are all on a relatively simple domain of speech spectrogram frames (in fact, restricted only to vowels, speech sounds which are most easily modeled using a source-filter decomposition).  The quantitative results (sec 4.2) are further restricted to *isolated* vowels, more closely matching the domain of the labeled synthetic data and not covering the dynamic space of variation in natural speech.\n   - At the very least the paper could use a detailed discussion about the how the proposed approach might be applied to other domains of interest to the generative modeling community, e.g., to natural images or at least to more complex speech tasks.\n     Such generalizations seems difficult since a key requirement for the technique to work is the existence of datasets which capture variation in a single attribute independently of others (or a synthesizer/generator which can create such datasets), i.e., it requires data which explicitly demonstrates disentanglement to use as supervision.  This requirement seems like it would be difficult to fulfill in other less restrictive domains.\n\n - Synthetic data was generated using a synthesizer based on the same source-filter decomposition that the proposed approach is designed to identify and recover.  The learned bases $U_i$ capture the space of variation in the synthetic data, but it's unclear how representative this is of real speech spectra.\n  - Overall it is unclear how well the approach might generalize to real data which does not perfectly match the synthetic labeled datasets?  E.g., speech in the presence of background noise or overlapping speakers.\n\n - The fact that a VAE trained on clean speech spectrogram frames naturally disentangles this structure is of moderate interest to practitioners in that domain.\n   But it's also not very surprising, as the problem of decomposing speech into separate source and filter components has been well understood for decades.\n   See e.g., slides 15-18 of https://www.ee.columbia.edu/~dpwe/e6820/lectures/L05-speechmodels.pdf and references in https://en.wikipedia.org/wiki/Mel-frequency_cepstrum#History which describe how the cosine transform commonly used in cepstral analysis approximates the principal components of log-spectrum.\n   Qualitatively, the described transformations in Sec 4 (transforming to constant f0, or \"whispered\" speech) are all fairly straightforward to implement without any machine learning at all using signal models from the 1980s, e.g., LPC or cepstral analysis, by modifying the source components while keeping the filter components fixed.\n   Training a VAE and using a complex generative model such as WaveGlow for time-domain audio reconstruction is overkill for these simple transformations.\n\nSpecific comments:\n- Sec 1, top of page 2: There have been other recent approaches to speech synthesis based on the source-filter model which are worth referencing in addition to LPCNet.  For example https://ieeexplore.ieee.org/abstract/document/8915761.\n- Sec 4:  It's interesting that $M_0$  is as high as four, when the fundamental frequency should be easily captured by a scalar.  Similar for $M_2$ and $M_3$.  Is there an explanation for this? \n- Sec 4.2.2: The low average NISQA scores of 2.6 (out of 5) on the original speech signals suggests a potential mismatch between the data that model was trained on (presumably real speech?), and the isolated vowel sounds used for evaluation.  Overall the large confidence intervals for NISQA in Table 1 further call the utility of this metric for this task into question.\n- Sec 5, top of page 9: The papers cited as \"voice conversion methods\" all describe text-to-speech synthesis techniques, not approaches to pure text-independent voice conversion, an area which has been well studied for many years.  Some recent deep-learning based examples include the sequence-to-sequence-based models (e.g., https://arxiv.org/abs/1810.0686), or the CycleGAN-VC (http://www.kecl.ntt.co.jp/people/kameoka.hirokazu/publications/Kaneko2018EUSIPCO08_published.pdf) or StarGAN-VC (https://arxiv.org/abs/1907.12279) model families", "summary_of_the_paper": "The paper proposes a method for utilizing labeled synthetic data in order to characterize and control the latent space of a VAE trained on individual frames of speech spectrograms.\nKey properties of the data which one might want explicit control over are identified, i.e., pitch and formant frequencies, and a parametric speech synthesizer is used to generate synthetic datasets for each property, where the property in question is varied but all others are kept fixed.\nThese labeled data are used to identify subspaces of a VAE latent which correspond to each property, essentially by a principal components analysis of the latent vectors from each point in the synthetic dataset for that property.\nThe degree of disentanglement of the latent representation can be characterized by how orthogonal the bases are across subspaces.\nFurthermore, individual properties can be directly controlled in isolation by learning a simple linear regression model mapping from the quantity in question (e.g., fundamental frequency in Hertz) to the subspace basis using supervision from the corresponding synthetic data.", "summary_of_the_review": "Although the paper is well written, it's main findings seem likely to be of interest to only a small segment of the community.  Unless the proposed technique is more broadly applicable to more complicated domains, it seems like this work might be a better fit for a more specialized speech venue.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work shows that the source-filter model of speech production naturally arises in the latent space of a variational autoencoder (VAE). It is interesting that the fundamental frequency and formant frequencies are encoded in orthogonal subspaces of the VAE latent space -- this opens up a possible way of easily controlling these.

The key motivation/goal of the paper has caused some confusion. The abstract highlights an observation about VAE’s learned representation. In retrospection, some reviewers have not found the findings very surprising. On the other hand, the authors also do not attempt at developing and evaluating a speech generation method. As is, the paper seems to be much more suitable to a specialized workshop on speech. Alternatively, the paper could be extended to other modalities to show steerability of a representation using a synthetic dataset. However, the current scope seems to be somewhat limited hence I am not able to recommend the current manuscript for acceptance.
Decision: Reject