Learning and controlling the source-filter representation of speech with a variational autoencoder

1 INTRODUCTION . High-dimensional data such as natural images or speech signals exhibit some form of regularity which prevents their dimensions from varying independently from each other . This suggests that there exists a latent representation of smaller dimension from which the high-dimensional observed data were generated . Discovering the hidden properties of complex data is the goal of representation learning , and deep latent-variable generative models have emerged as promising unsupervised approaches ( Goodfellow et al. , 2014 ; Kingma & Welling , 2014 ; Rezende et al. , 2014 ; Chen et al. , 2016 ; Higgins et al. , 2017 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ) . The variational autoencoder ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) , which is equipped with both a generative and inference model , can be used not only for data generation but also for analysis and transformation . As an explicit model of a probability density function ( pdf ) , the VAE can also be used as a learned prior for solving inverse problems such as compressed sensing ( Bora et al. , 2017 ) , speech enhancement ( Bando et al. , 2018 ; Leglaive et al. , 2018 ) , or source separation ( Kameoka et al. , 2019 ; Jayaram & Thickstun , 2020 ) . Making sense of the latent representation learned by a VAE and controlling the underlying continuous factors of variation in the data are important challenges to build more expressive and interpretable generative models and probabilistic priors . Previous works on representation learning with deep generative models , in particular VAEs , have mostly focused on images ( Higgins et al. , 2017 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Locatello et al. , 2019 ; 2020 ) . Yet , it is not always easy to define the ground-truth latent factors of variation involved in the generation of natural images . For speech data , the latent factors of variation can be directly related to the anatomical mechanisms of speech production . This makes speech data interesting for investigating the disentangled representation learning capabilities of VAEs , complementary to studies dealing with images . A key concept for characterizing the structure of speech signals is deduced from the source-filter model proposed by Fant ( 1970 ) . This model , described in more detail in Section 2.2 , implies that a speech signal is mainly characterized by a few continuous latent factors of variation corresponding to the vibration of the vocal folds ( i.e. , the source ) , which defines the fundamental frequency , and the resonances of the vocal tract ( i.e. , the filter ) , which define the formants . The source-filter model is at the core of various fundamental speech processing techniques such as cepstral representations and linear predictive coding ( LPC ) ( Rabiner & Schafer , 2010 ) . Valin & Skoglund ( 2019 ) ; Wang et al . ( 2019 ) and Juvela et al . ( 2019 ) have recently shown that the efficiency of neural speech vocoders can be largely improved by leveraging the sourcefilter model . Other works investigating the interaction between the source-filter model and neural networks include Lee et al . ( 2019 ) and Choi et al . ( 2021 ) . All these studies illustrate the interest of combining deep learning techniques with more traditional signal processing models and algorithms . In this work , we interpret and control the latent space of a VAE from the perspective of the source-filter model of speech production , which can be beneficial for various applications in speech analysis , transformation , and synthesis . We first train a VAE on a dataset of about 25 hours of unlabeled speech signals . Then , using only a few seconds of labeled speech signals generated with an artificial speech synthesizer , we propose a method to analyze and control the fundamental frequency and the formant frequencies in the latent representation of the previously trained VAE . Our contributions are the following : ( i ) We experimentally demonstrate that the fundamental frequency and the frequency of the first three formants are encoded in orthogonal subspaces of the VAE latent space . This shows that a vanilla VAE trained in an unsupervised fashion is able to learn a representation that is compliant with the source-filter model of speech production . ( ii ) We develop a weakly-supervised method to precisely and independently control the source-filter continuous latent factors of speech variation within the learned subspaces . We put in evidence the orthogonality of these subspaces , which allows us to perform speech transformations in a disentangled manner ( i.e. , modifying one of the factors does not affect the others ) . ( iii ) Without requiring additional information such as text or human-labeled data , we propose a deep generative model of speech spectrograms conditioned on the fundamental frequency and formant frequencies . To the best of our knowledge , this is the first study showing the link between the classical source-filter model of speech production and the representation learned in the latent space of a VAE . Thanks to this link , we propose a principled method to generate speech data controlled with interpretable trajectories ( of e.g. , fundamental frequency and formant frequencies ) . 2 BACKGROUND . 2.1 VARIATIONAL AUTOENCODER . Generative modeling consists in learning a probabilistic model of an observable random variable x ∈ X ⊂ RD . Let D = { x1 , ... , xN ∈ X } be a dataset of N = # D independent and identically distributed ( i.i.d . ) observations of x . The empirical distribution of x is defined by p̂ ( x ) = 1N ∑ xn∈D δ ( x − xn ) , where δ is the Dirac delta function , which is null everywhere except in 0 where it takes the value 1 . The variational autoencoder ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) attempts to approximate p̂ ( x ) with a pdf pθ ( x ) parametrized by θ. High-dimensional data such as natural images or speech signals exhibit some form of regularity which prevents theD dimensions of x from varying independently from each other . We can thus assume that there exists a latent variable z ∈ RL , with L D , from which the observed data were generated . Accordingly , the model distribution in the VAE is defined by marginalizing the joint distribution of the latent and observed data , pθ ( x ) =∫ pθ ( x|z ) p ( z ) dz . In this work , the observed data vector x ∈ RD+ denotes the power spectrum of a short frame of speech signal ( i.e. , a column of the short-time Fourier transform ( STFT ) power spectrogram ) . Its entries are non negative and its dimension D equals the number of frequency bins . We use the Itakura-Saito VAE ( IS-VAE ) ( Bando et al. , 2018 ; Leglaive et al. , 2018 ; Girin et al. , 2019 ) defined by p ( z ) = N ( z ; 0 , I ) , pθ ( x|z ) = ∏D d=1 Exp ( [ x ] d ; [ vθ ( z ) ] −1 d ) , ( 1 ) whereN and Exp denote the densities of the multivariate Gaussian and univariate exponential distributions , respectively , and [ v ] d denotes the d-th entry of v. The inverse scale parameters of pθ ( x|z ) are provided by a neural network called the decoder , parametrized by θ and taking z as input . The marginal likelihood pθ ( x ) and the posterior distribution pθ ( z|x ) are intractable due to the non linearities of the decoder , so it is necessary to introduce an inference model qφ ( z|x ) ≈ pθ ( z|x ) , which in the VAE is usually defined by qφ ( z|x ) = N ( z ; µφ ( x ) , diag { vφ ( x ) } ) , ( 2 ) where the mean and variance parameters are provided by a neural network called the encoder network , parametrized by φ and taking x as input . Then , the VAE training consists in maximizing a lower-bound of ln pθ ( x ) , called the evidence lower-bound ( ELBO ) and defined by L ( θ , φ ) = Ep̂ ( x ) [ Eqφ ( z|x ) [ pθ ( x|z ) ] −DKL ( qφ ( z|x ) ‖ p ( z ) ) ] . During training , the generative and inference model parameters θ and φ are jointly estimated by maximizing the ELBO , using ( variants of ) stochastic gradient descent with the so-called reparameterization trick ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) . 2.2 SOURCE-FILTER MODEL OF SPEECH PRODUCTION . The source-filter model of speech production ( Fant , 1970 ) is at the basis of many speech processing systems . It considers that the production of speech results from the interaction of a source signal with a linear filter . In voiced speech , the source originates from the vibration of the vocal folds , which produces a quasi-periodic glottal sound wave whose fundamental frequency defines the pitch . In unvoiced speech , a noise source is produced by a turbulent airflow or an acoustic impulse . The source signal is modified by the vocal tract , which is assumed to act as a linear filter . The cavities of the vocal tract give rise to resonances , which are called the formants and are characterized by their frequency , amplitude and bandwidth . By moving the speech articulators such as the tongue , lips , and jaw , humans modify the shape of their vocal tract , which results in a change of the acoustic filter and the associated resonances . This is how the different elementary speech sounds called phonemes are produced to form syllables , words and sentences . The power spectra and the spectral envelopes of two French vowels are displayed in Figure 1 . The spectral envelopes show that the formant frequencies are different for the two vowels . In this example however , the harmonic structure of the spectra shows that the fundamental frequency is the same for the two vowels . Formant frequencies are important distinctive features of vowels . In a first approximation , they can be related to the opening of the mouth , the front/rear position of the tongue , and the rounding of the lips for the first , second , and third formant respectively . For voiced phonemes , humans are able to control the formants independently of the pitch ( i.e. , to change the filter independently of the source ( Fant , 1970 ) ) and of each other ( MacDonald et al. , 2011 ) . The independence of the source and filter characteristics makes the speech signals an interesting material for representation learning methods , especially with deep generative latent-variable models . In the present study , in addition to the pre-trained IS-VAE speech spectrogram model , we also assume the availability of an artificial speech synthesizer allowing for an accurate and independent control of the fundamental frequency and formants . In this work , we use Soundgen ( Anikin , 2019 ) , a parametric synthesizer based on the source-filter model of speech production . For a given speech sound , the voiced component of the source signal is generated by a sum of sine waves , the noise component by a filtered white noise , and both components are then summed and passed through a linear filter simulating the effect of the human vocal tract . Importantly , this synthesizer allows us to easily generate artificial speech data labeled with the fundamental frequency and formant frequency values .

Summary: This paper shows that the fundamental frequency and formant frequency information is encoded in a speech VAE model. This can be found by using artificially controlled/generated dataset. After finding how to manipulate the latent space, one can control arbitrary speech samples in a desirable way, such as controlling fundamental frequency or formant frequencies. The authors also show that the harmonic parts can be removed from the original speech and reconstructed as a whispered voice using only the spectral envelope part. Experiment results show that the proposed method can control formant frequencies and show that it can control better than the method proposed by previous work. The proposed method show on par or worse performance on fundamental frequency control experiment when compared to traditional DSP vocoders, such as TD-PSOLA or WORLD.