Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching

1 INTRODUCTION . Over the past years , the performance of automatic speech recognition ( ASR ) has been improved greatly and the recognition accuracy in certain scenarios could be on par with human performance ( Xiong et al. , 2016 ) . Most of the state-of-the-art ASR systems are constructed by training deep neural networks on large-scale labeled data using supervised learning ( Hinton et al. , 2012 ; Dahl et al. , 2012 ; Xiong et al. , 2016 ; Graves et al. , 2013 ; 2006 ; Graves , 2012 ) ; they rely on a large number of human labeled data to train the recognition model . In this paper , we are working towards the grand mission of training speech recognition models without any human annotated data . Such an approach could potentially save a huge amount of human labeling costs for developing ASR systems by leveraging massive unlabeled speech data . It is especially valuable when developing ASR systems for low-resource languages , where labeled data are more expensive to obtain . Specifically , we consider the phoneme recognition problem , for which we learn a sequential classifier that maps speech waveform into a sequence of phonemes . In our unsupervised learning setting , the learning algorithm can only access ( i ) the input speech acoustic features , and ( ii ) a pretrained phoneme language model ( LM ) . There is no human supervision presented to the algorithm at any level ; that is , we do not provide any ( frame-level ) label for input samples , nor do we provide any ( sentence-level ) transcription for input utterances . The language model could be trained from a separate ( text ) corpus in an unsupervised manner with the help of a pre-defined lexicon1 . There have been some recent successes in developing fully unsupervised method for neural machine translation ( Artetxe et al. , 2018 ; Lample et al. , 2018 ) and sequence classifications ( Liu et al. , ∗The work was done during an internship at Tencent AI Lab , Bellevue , WA . 1A lexicon in ASR is a pre-defined dictionary that maps word sequences into phoneme sequences . 2017 ) . However , different from these problems , speech recognition problem has segmental structures that impose unique challenges for developing unsupervised learning algorithms . First , each phoneme generally consists of a segment of consecutive input samples ( frames ) that are associated to the same phoneme label . Second , the lengths and the boundaries of these segments are usually unknown a priori . For this reason , we could not directly apply the previous techniques to develop unsupervised ASR algorithms . To address the first challenge , we develop a novel unsupervised learning cost function for ASR systems by extending the Empirical Output Distribution Matching ( Empirical-ODM ) cost in ( Liu et al. , 2017 ) to segmental structures . The key ideas of our Segmental Empirical-ODM are : ( i ) the distribution of the predicted outputs across consecutive segments shall match the phoneme language model and ( ii ) the predicted outputs within each segment should be equal to each other as they belong to the same phoneme . This cost function allows us to learn the classifier without labeled data for a given set of phoneme segmentation boundaries . To address the second challenge , we develop a novel unsupervised approach to estimate ( and refine ) the segmentation boundaries using the current classification model . Our algorithm alternates between these two steps of learning classifier and estimating the boundaries to successively improve the performance of each other . Therefore , unlike previous works in ( Liu et al. , 2018 ) , which relies on an oracle or forced alignment methods to obtain the phoneme segmentation boundaries , our method is fully unsupervised in both segmentation and classification . Furthermore , we also adapt the semisupervised HMM learning technique ( Zavaliagkos et al. , 1998 ; Kemp and Waibel , 1999 ; Nallasamy et al. , 2012 ) to our unsupervised setting to further improve the performance . In our experiments on TIMIT phoneme recognition task , our unsupervised learning method achieves a promising phone error rate ( PER ) of 41.6 % . To our best knowledge , this is the first empirical success of a fully unsupervised speech recognition that does not use any oracle segmentation or labels . Furthermore , when the oracle phoneme segmentation boundaries are given ( similar to the setting in Liu et al . ( 2018 ) ) , our method achieves a PER of 32.5 % with matching language model , which approaches supervised learning with the same model architecture , demonstrating a great potential of our method . 2 FULLY UNSUPERVISED SPEECH RECOGNITION . 2.1 PROBLEM FORMULATION . We consider the unsupervised phoneme recognition problem . Specifically , for a given sequence of input feature vectors x = ( x1 , . . . , xT ) , we want to map it into a sequence of phonemes q = ( q1 , . . . , qU ) , where xt ∈ Rm is anm-dimensional input acoustic feature vector ( e.g. , mel-frequency cepstral coefficients ( MFCC ) ) , qi ∈ Y is a categorical variable representing the phoneme class , Y denotes the set of phonemes , T is the length of the input sequence , and U is the length of the output sequence . Note that the length of the input sequence is usually much larger than that of the output sequence.2 This is because speech data have a special segmental structure where a segment of consecutive input frames are associated with one phoneme class , as shown in Figure 1 . Furthermore , the length and boundaries of each phoneme segment are generally varying and unknown a priori . We introduce a binary variable bt ∈ { 0 , 1 } to characterize the segment boundaries : bt = 1 denotes the start of a new phoneme segment ( see Figure 1 ) . Let yt ∈ Y be the frame-wise phoneme label indicating the phoneme class that the t-th input frame xt belongs to . In this work , we focus on learning a framewise phoneme classifier that maps the input sequence x1 , . . . , xT into its framewise label sequence y = ( y1 , . . . , yT ) . Once this is done , we could use a standard speech decoder to obtain the desired phoneme sequence q1 , . . . , qU from y1 , . . . , yT . We model the framewise phoneme classifier pθ ( yt|xt ) ( i.e. , the posterior probability of the frame label yt given the input xt ) by a context dependent DNN ( Dahl et al. , 2012 ) , where θ denotes the model parameter and the input feature vector xt is a concatenation of the acoustic feature vectors within a context window around time t. We may also use other model architectures such as recurrent neural network ( RNN ) , which are left as the future work . The objective of our unsupervised learning algorithm is to learn the model parameter θ from : ( i ) a training set of input sequences Dx , and ( ii ) a pretrained phoneme language model pLM ( q ) . Note that the language model could be trained from a separate ( text ) corpus in an unsupervised manner so that there is no supervision at any level . There are two main challenges for unsupervised speech recognition : ( i ) how to learn the classifier pθ ( yt|xt ) from Dx and pLM ( q ) for a given set of segmentation boundaries , and ( ii ) how to estimate 2We will use notation t to index input frames and use notation i to index segments ( or phonemes ) . the segmentation boundaries in an unsupervised manner . Unlike text where content can be broken into word units relatively easily , speech inputs are continuous and thus it is difficult to obtain the phoneme boundaries . This challenge is unique to unsupervised speech recognition and does not appear in e.g . unsupervised machine translation . In the following sections , we address the above challenges by developing a new unsupervised learning cost function by extending the EmpiricalODM cost in ( Liu et al. , 2017 ) to segmental structures . Furthermore , we develop a maximum a posteriori ( MAP ) estimator to refine the segmentation boundaries based on the current pθ ( yt|xt ) . Our algorithm alternates between these two steps , and after the iteration completes , we employ an unsupervised HMM training technique to further boost our unsupervised results . 2.2 UNSUPERVISED FRAME CLASSIFICATION WITH GIVEN SEGMENTATION BOUNDARIES . In this section , we develop an unsupervised algorithm to learn the classification model pθ ( yt|xt ) with a given set of segmentation boundaries { bt } . To this end , we define a new unsupervised learning cost function that exploits the segmental structure of the problem . Specifically , our new cost function is based on the following two observations : ( i ) the distribution of the predicted outputs across consecutive segments shall match the phoneme language model pLM ( q ) , and ( ii ) the predicted outputs within each segment should be equal to each other as they belong to the same phoneme . Accordingly , our unsupervised cost function consists of two parts , characterizing the above inter-segment and intra-segment distributions , respectively . We first define the cost function associated with the inter-segment distribution . Before that , we introduce the following terms and notations , which are also illustrated in Figure 1 . To simplify notation , we assume that all the utterances in Dx are concatenated into one long sequence . Let there be a total of K segments in the entire training set Dx and let Si be a set that includes all the time indexes in the i-th segment . We use τ = ( t1 , . . . , tK ) to denote a sequence of time indexes sampled from S1 , . . . , SK , one per segment , i.e. , ti ∈ Si . Without loss of generality , we consider N -gram phoneme language models throughout this work and define pLM ( z ) , pLM ( qi−N+1 = z1 , . . . , qi = zN ) , where z = ( z1 , . . . , zN ) ∈ YN denotes a particular N -gram . Furthermore , let τi = ( ti−N+1 , . . . , ti ) be a length-N contiguous subsequence of τ that ends at ti . We use the compact notation xτi and yτi to represent ( xti−N+1 , . . . , xti ) and ( yti−N+1 , . . . , yti ) , respectively . Then , the cost function that characterizes the inter-segment output distribution match is defined as : JODM ( θ ) = − ∑ τ∈S1×···×SK ∑ z∈YN pLM ( z ) ln p τ θ ( z ) ( 1 ) where pτθ ( z ) = 1 K ∑K i=1 pθ ( yτi = z|xτi ) is defined as the inter-segment output distribution with pθ ( yτi = z|xτi ) , ∏i j=i−N+1 pθ ( ytj = zj |xtj ) . The cost function ( 1 ) generalizes the EmpiricalODM cost in Liu et al . ( 2017 ) to the segmental structures , and it degenerates into the original Empirical-ODM cost when there is only one frame in each segment . The probability pτθ ( z ) characterizes the empirical N -gram frequency of the predicted output across N consecutive segments , and the cost function measures the cross-entropy between the pretrainedN -gram LM pLM ( z ) and pτθ ( z ) . This form of cost function enjoys several properties that are suitable for unsupervised learning of sequence classifiers , and the readers are referred to Liu et al . ( 2017 ) for more detailed discussions . Next , we define the cost function that characterizes the intra-segment distribution matching as : JFS ( θ ) = K∑ i=1 ∑ t , t+1∈Si ∑ y∈Y ( pθ ( yt = y|xt ) − pθ ( yt+1 = y|xt+1 ) ) 2 ( 2 ) where the subscript “ FS ” stands for frame-wise smoothness . The cost ( 2 ) encourages the predictions for adjacent frames within the same segment to be similar . It captures the strong intra-segment temporal structure in speech signals that complements the cost ( 1 ) . Our final unsupervised cost function combines the inter-segment and intra-segment distribution matching via : min θ { J ( θ ) , JODM ( θ ) + λJFS ( θ ) } ( 3 ) where λ is a parameter controlling the trade-off between the two parts . We call the cost function J ( θ ) Segmental Empirical-ODM as it captures the segmental structure through the inter-segment and intra-segment terms . To optimize this cost function , we sample a sequence τ at the beginning of each epoch and applies stochastic gradient descent ( SGD ) with momentum to update θ . Note that in JODM ( θ ) , there is an empirical average over all K segments in pτθ ( z ) , which is inside the logarithmic function . This makes stochastic gradient descent intrinsically biased if we also sample this empirical average by a mini-batch average . To alleviate this effect , we use a large mini-batch size to estimate the stochastic gradients . Note that our method directly optimizes the classifier pθ ( yt|xt ) that takes the raw acoustic feature vector xt ( e.g. , MFCC features ) and maps it into output space . This is different from the previous work ( Liu et al. , 2018 ) , which first performs clustering in the speech space and then maps the clusters into output space using adversarial training . This makes its performance upper-bounded by the purity of the initial clusters since input frames of different phonemes may be mapped into the same cluster . In contrast , our algorithm is end-to-end trained without using a separate clustering algorithm . This enables us to outperform the cluster purity upper bound , as shown in our experiment section .

Summary: This paper proposes a new approach to do unsupervised phoneme recognition by learning from unlabelled speech in combination with a trained phoneme language model. The proposed loss function is a combination of a term encouraging the language model of predicted phonemes to match the given language model distribution, and a term to encourage adjacent speech frames to be assigned to the same phoneme class. Phoneme boundaries are iteratively refined using a separate model. Experiments where a hidden Markov model is applied on top of the predicted phonemes are also performed.