TAL EMPIRICAL OUTPUT DISTRIBUTION MATCHING
Year: 2019

Abstract:
We consider the problem of training speech recognition systems without using any labeled data, under the assumption that the learner can only access to the input utterances and a phoneme language model estimated from a non-overlapping corpus. We propose a fully unsupervised learning algorithm that alternates between solving two sub-problems: (i) learn a phoneme classifier for a given set of phoneme segmentation boundaries, and (ii) refining the phoneme boundaries based on a given classifier. To solve the first sub-problem, we introduce a novel unsupervised cost function named Segmental Empirical Output Distribution Matching, which generalizes the work in (Liu et al., 2017) to segmental structures. For the second sub-problem, we develop an approximate MAP approach to refining the boundaries obtained from Wang et al. (2017). Experimental results on TIMIT dataset demonstrate the success of this fully unsupervised phoneme recognition system, which achieves a phone error rate (PER) of 41.6%. Although it is still far away from the state-of-the-art supervised systems, we show that with oracle boundaries and matching language model, the PER could be improved to 32.5%. This performance approaches the supervised system of the same model architecture, demonstrating the great potential of the proposed method.

heading: 1 INTRODUCTION
text: Over the past years, the performance of automatic speech recognition (ASR) has been improved greatly and the recognition accuracy in certain scenarios could be on par with human performance (Xiong et al., 2016). Most of the state-of-the-art ASR systems are constructed by training deep neural networks on large-scale labeled data using supervised learning (Hinton et al., 2012; Dahl et al., 2012; Xiong et al., 2016; Graves et al., 2013; 2006; Graves, 2012); they rely on a large number of human labeled data to train the recognition model. In this paper, we are working towards the grand mission of training speech recognition models without any human annotated data. Such an approach could potentially save a huge amount of human labeling costs for developing ASR systems by leveraging massive unlabeled speech data. It is especially valuable when developing ASR systems for low-resource languages, where labeled data are more expensive to obtain.
Specifically, we consider the phoneme recognition problem, for which we learn a sequential classifier that maps speech waveform into a sequence of phonemes. In our unsupervised learning setting, the learning algorithm can only access (i) the input speech acoustic features, and (ii) a pretrained phoneme language model (LM). There is no human supervision presented to the algorithm at any level; that is, we do not provide any (frame-level) label for input samples, nor do we provide any (sentence-level) transcription for input utterances. The language model could be trained from a separate (text) corpus in an unsupervised manner with the help of a pre-defined lexicon1.
There have been some recent successes in developing fully unsupervised method for neural machine translation (Artetxe et al., 2018; Lample et al., 2018) and sequence classifications (Liu et al.,
∗The work was done during an internship at Tencent AI Lab, Bellevue, WA. 1A lexicon in ASR is a pre-defined dictionary that maps word sequences into phoneme sequences.
2017). However, different from these problems, speech recognition problem has segmental structures that impose unique challenges for developing unsupervised learning algorithms. First, each phoneme generally consists of a segment of consecutive input samples (frames) that are associated to the same phoneme label. Second, the lengths and the boundaries of these segments are usually unknown a priori. For this reason, we could not directly apply the previous techniques to develop unsupervised ASR algorithms. To address the first challenge, we develop a novel unsupervised learning cost function for ASR systems by extending the Empirical Output Distribution Matching (Empirical-ODM) cost in (Liu et al., 2017) to segmental structures. The key ideas of our Segmental Empirical-ODM are: (i) the distribution of the predicted outputs across consecutive segments shall match the phoneme language model and (ii) the predicted outputs within each segment should be equal to each other as they belong to the same phoneme. This cost function allows us to learn the classifier without labeled data for a given set of phoneme segmentation boundaries. To address the second challenge, we develop a novel unsupervised approach to estimate (and refine) the segmentation boundaries using the current classification model. Our algorithm alternates between these two steps of learning classifier and estimating the boundaries to successively improve the performance of each other. Therefore, unlike previous works in (Liu et al., 2018), which relies on an oracle or forced alignment methods to obtain the phoneme segmentation boundaries, our method is fully unsupervised in both segmentation and classification. Furthermore, we also adapt the semisupervised HMM learning technique (Zavaliagkos et al., 1998; Kemp and Waibel, 1999; Nallasamy et al., 2012) to our unsupervised setting to further improve the performance. In our experiments on TIMIT phoneme recognition task, our unsupervised learning method achieves a promising phone error rate (PER) of 41.6%. To our best knowledge, this is the first empirical success of a fully unsupervised speech recognition that does not use any oracle segmentation or labels. Furthermore, when the oracle phoneme segmentation boundaries are given (similar to the setting in Liu et al. (2018)), our method achieves a PER of 32.5% with matching language model, which approaches supervised learning with the same model architecture, demonstrating a great potential of our method.

heading: 2 FULLY UNSUPERVISED SPEECH RECOGNITION

heading: 2.1 PROBLEM FORMULATION
text: We consider the unsupervised phoneme recognition problem. Specifically, for a given sequence of input feature vectors x = (x1, . . . , xT ), we want to map it into a sequence of phonemes q = (q1, . . . , qU ), where xt ∈ Rm is anm-dimensional input acoustic feature vector (e.g., mel-frequency cepstral coefficients (MFCC)), qi ∈ Y is a categorical variable representing the phoneme class, Y denotes the set of phonemes, T is the length of the input sequence, and U is the length of the output sequence. Note that the length of the input sequence is usually much larger than that of the output sequence.2 This is because speech data have a special segmental structure where a segment of consecutive input frames are associated with one phoneme class, as shown in Figure 1. Furthermore, the length and boundaries of each phoneme segment are generally varying and unknown a priori. We introduce a binary variable bt ∈ {0, 1} to characterize the segment boundaries: bt = 1 denotes the start of a new phoneme segment (see Figure 1). Let yt ∈ Y be the frame-wise phoneme label indicating the phoneme class that the t-th input frame xt belongs to. In this work, we focus on learning a framewise phoneme classifier that maps the input sequence x1, . . . , xT into its framewise label sequence y = (y1, . . . , yT ). Once this is done, we could use a standard speech decoder to obtain the desired phoneme sequence q1, . . . , qU from y1, . . . , yT . We model the framewise phoneme classifier pθ(yt|xt) (i.e., the posterior probability of the frame label yt given the input xt) by a context dependent DNN (Dahl et al., 2012), where θ denotes the model parameter and the input feature vector xt is a concatenation of the acoustic feature vectors within a context window around time t. We may also use other model architectures such as recurrent neural network (RNN), which are left as the future work. The objective of our unsupervised learning algorithm is to learn the model parameter θ from: (i) a training set of input sequences Dx, and (ii) a pretrained phoneme language model pLM(q). Note that the language model could be trained from a separate (text) corpus in an unsupervised manner so that there is no supervision at any level.
There are two main challenges for unsupervised speech recognition: (i) how to learn the classifier pθ(yt|xt) from Dx and pLM(q) for a given set of segmentation boundaries, and (ii) how to estimate
2We will use notation t to index input frames and use notation i to index segments (or phonemes).
the segmentation boundaries in an unsupervised manner. Unlike text where content can be broken into word units relatively easily, speech inputs are continuous and thus it is difficult to obtain the phoneme boundaries. This challenge is unique to unsupervised speech recognition and does not appear in e.g. unsupervised machine translation. In the following sections, we address the above challenges by developing a new unsupervised learning cost function by extending the EmpiricalODM cost in (Liu et al., 2017) to segmental structures. Furthermore, we develop a maximum a posteriori (MAP) estimator to refine the segmentation boundaries based on the current pθ(yt|xt). Our algorithm alternates between these two steps, and after the iteration completes, we employ an unsupervised HMM training technique to further boost our unsupervised results.

heading: 2.2 UNSUPERVISED FRAME CLASSIFICATION WITH GIVEN SEGMENTATION BOUNDARIES
text: In this section, we develop an unsupervised algorithm to learn the classification model pθ(yt|xt) with a given set of segmentation boundaries {bt}. To this end, we define a new unsupervised learning cost function that exploits the segmental structure of the problem. Specifically, our new cost function is based on the following two observations: (i) the distribution of the predicted outputs across consecutive segments shall match the phoneme language model pLM(q), and (ii) the predicted outputs within each segment should be equal to each other as they belong to the same phoneme. Accordingly, our unsupervised cost function consists of two parts, characterizing the above inter-segment and intra-segment distributions, respectively.
We first define the cost function associated with the inter-segment distribution. Before that, we introduce the following terms and notations, which are also illustrated in Figure 1. To simplify notation, we assume that all the utterances in Dx are concatenated into one long sequence. Let there be a total of K segments in the entire training set Dx and let Si be a set that includes all the time indexes in the i-th segment. We use τ = (t1, . . . , tK) to denote a sequence of time indexes sampled from S1, . . . ,SK , one per segment, i.e., ti ∈ Si. Without loss of generality, we consider N -gram phoneme language models throughout this work and define pLM(z) , pLM(qi−N+1 = z1, . . . , qi = zN ), where z = (z1, . . . , zN ) ∈ YN denotes a particular N -gram. Furthermore, let τi = (ti−N+1, . . . , ti) be a length-N contiguous subsequence of τ that ends at ti. We use the compact notation xτi and yτi to represent (xti−N+1, . . . , xti) and (yti−N+1, . . . , yti), respectively. Then, the cost function that characterizes the inter-segment output distribution match is defined as:
JODM(θ) = − ∑
τ∈S1×···×SK ∑ z∈YN pLM(z) ln p τ θ (z) (1)
where pτθ (z) = 1 K ∑K i=1 pθ(yτi = z|xτi) is defined as the inter-segment output distribution with
pθ(yτi = z|xτi) , ∏i j=i−N+1 pθ(ytj = zj |xtj ). The cost function (1) generalizes the EmpiricalODM cost in Liu et al. (2017) to the segmental structures, and it degenerates into the original Empirical-ODM cost when there is only one frame in each segment. The probability pτθ (z) characterizes the empirical N -gram frequency of the predicted output across N consecutive segments, and the cost function measures the cross-entropy between the pretrainedN -gram LM pLM(z) and pτθ (z). This form of cost function enjoys several properties that are suitable for unsupervised learning of sequence classifiers, and the readers are referred to Liu et al. (2017) for more detailed discussions.
Next, we define the cost function that characterizes the intra-segment distribution matching as:
JFS(θ) = K∑ i=1 ∑ t,t+1∈Si ∑ y∈Y ( pθ(yt = y|xt)− pθ(yt+1 = y|xt+1) )2 (2)
where the subscript “FS” stands for frame-wise smoothness. The cost (2) encourages the predictions for adjacent frames within the same segment to be similar. It captures the strong intra-segment temporal structure in speech signals that complements the cost (1). Our final unsupervised cost function combines the inter-segment and intra-segment distribution matching via:
min θ
{ J(θ) , JODM(θ) + λJFS(θ) } (3)
where λ is a parameter controlling the trade-off between the two parts. We call the cost function J(θ) Segmental Empirical-ODM as it captures the segmental structure through the inter-segment and intra-segment terms. To optimize this cost function, we sample a sequence τ at the beginning of each epoch and applies stochastic gradient descent (SGD) with momentum to update θ. Note that in JODM(θ), there is an empirical average over all K segments in pτθ (z), which is inside the logarithmic function. This makes stochastic gradient descent intrinsically biased if we also sample this empirical average by a mini-batch average. To alleviate this effect, we use a large mini-batch size to estimate the stochastic gradients.
Note that our method directly optimizes the classifier pθ(yt|xt) that takes the raw acoustic feature vector xt (e.g., MFCC features) and maps it into output space. This is different from the previous work (Liu et al., 2018), which first performs clustering in the speech space and then maps the clusters into output space using adversarial training. This makes its performance upper-bounded by the purity of the initial clusters since input frames of different phonemes may be mapped into the same cluster. In contrast, our algorithm is end-to-end trained without using a separate clustering algorithm. This enables us to outperform the cluster purity upper bound, as shown in our experiment section.

heading: 2.3 SEGMENTATION BOUNDARY REFINEMENT USING THE CLASSIFICATION MODEL
text: In this section, we develop an approach to refining the estimated segmentation boundaries {bt} using a learned framewise phoneme classifier pθ(yt|xt). More formally, for each input utterance sequence x = (x1, . . . , xT ), we would like to infer the corresponding boundary sequence b = (b1, . . . , bT ). We propose a simple yet effective MAP estimation strategy by recognizing the fact that bt = I(yt 6= yt−1). Therefore, we can perform an MAP estimate for y = (y1, . . . , yT ) and then predict the boundaries by bt = I(yt 6= yt−1). The MAP estimator of y can be expressed as (see Appendix A):
argmax y p(y|x) = argmax y T∏ t=1 p(yt|y1, . . . , yt−1) pθ(yt|xt) p(yt)
(4)
Note that p(yt|y1, . . . , yt−1) is the transition probability of the frame labels. Assuming that yt belongs to the i-th segment, we can express p(yt|y1, . . . , yt−1) as:
p(yt|y1, . . . , yt−1) = I(yt=yt−1)p(yt=yt−1)+I(yt 6=yt−1)p(yt 6=yt−1)pLM(qi=yt|q1:i−1) = I(yt=yt−1)p(bt=0)+I(yt 6=yt−1)p(bt=1)pLM(qi=yt|q1:i−1) (5)
where q1:i−1 denotes the previous i − 1 phonemes that the sequence y1, . . . , yt−1 has traversed. The first term in (5) characterizes the probability that yt stays in the same phoneme segment as yt−1 and the second term defines the probability that yt belongs to a new phoneme segment. Note that pLM(qi = yt|q1:i−1) in (5) could be obtained from the phoneme language model. It remains to estimate p(bt = 0) and p(bt = 1), which we approximate by p(bt = 0|x) and p(bt = 1|x), respectively. To obtain p(bt = 1|x), we leverage the work of Wang et al. (2017), which shows that the temporal structure of the gate signals in a gated RNN (GRNN) auto-encoder is highly correlated with phoneme boundaries. Therefore, we apply a sigmoid function to the Difference GAS value (defined as Wang et al. (2017)) to obtain p(bt = 1|x). With all these information, we substitute (5) into (4) and perform a beam search to solve (4) for an approximate MAP estimate of y. It follows that bt = I(yt 6= yt−1) and we have refined the boundaries using pθ(yt|xt).

heading: 2.4 ALTERNATING TRAINING ALGORITHM
text: Our unsupervised learning algorithm for pθ(yt|xt) alternates between the above two steps of estimating θ for a given b and refining b for a given θ. The overall algorithm is summarized in Algorithm 1. We initialize the algorithm by thresholding the temporal update gate activation in Wang et al. (2017) to obtain an initial rough estimate for b. After the training converges,3 we could apply the unsupervised HMM training technique discussed in Section 3 to further boost the performance. Note that although the training process requires boundary estimation, at testing stage, it is not necessary because the learned pθ(yt|xt) could be used in standard speech decoders just as supervised models.
Algorithm 1: Training Algorithm Input: Phoneme language model pLM(z), Training data Dx, initial boundary binit obtained by using techniques proposed in Wang et al. (2017). Output: Model parameter θ
1 Initialization for parameters θ. 2 while not converged do 3 Given a set of boundaries b, obtain a new θ by optimizing (3). 4 Given the model parameter θ, obtain a new estimate for the boundaries b by optimizing (4). 5 end

heading: 2.5 UNSUPERVISED MODEL SELECTION
text: Since there are no labeled data during the training process, we need to develop an unsupervised self-validation metric to perform model selection. We propose to use the value of the loss function (1) on a heldout validation set (including only input features) to perform model selection. This self-validation loss gives us an estimate of which model configuration is better, and is used to a) determine when to stop training, and b) to select the best hyper-parameters. To validate the effectiveness for our self-validation loss, we show the learning curves of this self-validation loss and the validation frame error rate in Figure 2(a). We observe that the self-validation loss aligns well with the true validation error. Furthermore, in Figure 2(b) we plot the self-validation loss and the validation FER for different values of λ, which shows that the two metrics are highly correlated. The results demonstrate that the self-validation loss can be effectively used to select a good model.

heading: 3 UNSUPERVISED HMM TRAINING
text: To further improve the performance of proposed unsupervised speech recognition system, we explore the semi-supervised hidden Markov model (HMM) training strategy (Zavaliagkos et al., 1998; Kemp and Waibel, 1999) that has commonly been used in speech recognition. The semi-supervised HMM training is an effective technique where a seed model trained on a relatively small amount
3We observe in our experiments that two iterations are sufficient to converge.
of labeled speech data is used for providing labels for larger amount of non-transcribed speech data for iterative model refinement. A major difference of the HMM training strategy used in this work compared to the ones used in semi-supervised learning is that we use the transcripts generated from proposed unsupervised speech recognition system (i.e., predicted labels for 3969 TIMIT training utterances) for bootstrapping the training of HMM-based models. Therefore, the training of HMM models in this work does not require any manually provided supervised information. The training of HMM based speech recognition models follows the standard recipes in Kaldi speech recognition toolkit (Povey et al., 2011). We experimented with monophone and triphone models with MFCC feature as input as well as more advanced speaker adaptive training (SAT) (Matsoukas et al., 1997) approach with feature space maximum likelihood linear regression (fMLLR) (Gales et al.) as input.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENT SETTING
text: We perform experiments on the TIMIT dataset where 6300 prompted English speech sentences are recorded. The preparation of training and test sets follow the standard protocol of the TIMIT dataset. The phoneme transcription of these utterances are manually segmented and labelled with a lexicon of 61 distinct phoneme classes. These phoneme labels are mapped to 39 phoneme classes for scoring phone error rate (Lee and Hon, 1989). We use 39 dimensional feature vectors including 13 mel-frequency cepstral coefficients (MFCC) plus its acceleration features that are extracted with 25 ms Hamming window at 10 ms interval. The classifier pθ(yt|xt) is modeled by a fully connected neural network with one hidden layer of 512 ReLU units. The input to the neural network is a concatenation of frames within a context window of size 11. We follow the default hyper-parameters in Wang et al. (2017) to estimate the phoneme boundaries, which are used to initialize our algorithm. The optimization of (3) is performed with momentum SGD with a fixed schedule of increasing batch size from 5000 to 20000. λ in (3) is chosen to be 10−5. We use both frame error rate (FER) and phone error rate (PER) as our evaluation metrics. Details of the experiment setting and other hyperparameters can be found in Appendix B.

heading: 4.2 BASELINE METHODS
text: Adversarial Mapping The first baseline we consider is the work by Liu et al. (2018), which learns an unsupervised embedding by a sequence-to-sequence autoencoder followed by k-means clustering. Each cluster is then mapped to a phoneme by adversarial training between the cluster sequences and the phoneme sequence. The phoneme boundaries are given by a supervised oracle.
Cluster Purity The accuracy of Adversarial Mapping (Liu et al., 2018) is upper-bounded by the cluster purity, which is the frame accuracy when assigning all the frames in each cluster to its most frequent phonemes. It is a supervised baseline since it relies on the phoneme labels. We show the cluster purity for 1000 clusters, which is the largest number of clusters used by Liu et al. (2018).
Supervised Neural Network We train a supervised neural network with the same architecture as our unsupervised model with standard cross-entropy loss.
Supervised RNN Transducer It is one of the state-of-the-art methods, which learns a BiLSTMRNN Transducer with supervised learning (Graves et al., 2013).

heading: 4.3 EXPERIMENT RESULTS
text: Unsupervised speech recognition with oracle boundary In our proposed unsupervised learning algorithm, we use the cost function (3) to train the classifier, which is different from the cross entropy cost in supervised learning. To examine the effect of replacing cross entropy with this new unsupervised cost function, we first conduct experiments under the oracle phoneme boundaries. This setting also allows us to compare our method to the one in Liu et al. (2018), which assumes oracle phoneme boundaries. Specifically, we consider two settings. In the first setting, we follow the standard TIMIT partition to divide the training data into a training and validation sets of 3696 and 400 utterances, respectively. We use the phoneme transcription of the 3696 utterances to train our language model pLM(z) and call this setting “matching language model”. Then we use this learned pLM(z) together with the 3696 input utterances to train our model by minimizing (3). In the second setting, we divide
the data into a training and a validation sets of 3000 and 1096 utterances, respectively. We train our language model pLM(z) on the phoneme transcription of the 1096 utterances, and use the learned pLM(z) with the other 3000 input utterances to train our model by minimizing (3). In this setting, the training corpus for pLM(z) does not overlap with the 3000 input training utterances, and we call it “non-matching language model”. Note that both settings are unsupervised since we do not use any phoneme label in our training process. The only difference is the source of the language model. The results of our algorithm and other baselines are summarized in Table 1. Although we are still far away from the state-of-the-art, in the matching LM setting, the performance of our algorithm (32.5% PER) is approaching that of the supervised system with the same model architecture (30.2% PER). This is an encouraging result, showing that replacing the supervised cross entropy loss with our unsupervised learning cost does not degrade the performance much. On the other hand, when we use non-matching LM, the gap becomes larger (40.1% vs 31.1% in PER). We think the reason is due to the discrepancy of the output distributions between the two sets and the reduced training corpus for pLM(z). We believe such a discrepancy could be alleviated by using a large-scale dataset. Other than the standard FER and PER, we additionally show the evaluation result for FER* where the starting and ending silences are removed following the setting in Liu et al. (2018). We observe that our approach significantly outperforms the unsupervised Adversarial Mapping method in Liu et al. (2018), and even outperforms the Cluster Purity (supervised upper bound in Liu et al. (2018)). This result is not surprising since the clustering does not exploit the output distribution, and may group inputs of different phonemes into the same cluster.
Fully unsupervised speech recognition We now consider the fully unsupervised setting where only input speech features and a language model is given. The phoneme boundaries are not given and has to be estimated in an unsupervised manner using our Algorithm 1. We show the quality of the learned model after each iteration of the learning process in Table 2. And we observe that our iteration process improves the results by a great margin especially in the non-matching LM case, significantly lowering the FER and PER by over 10%. This demonstrates that our boundary refining process has resulted in a better set of boundaries, which greatly improves the output distribution
matching. Moreover, we also report the results of using unsupervised HMM training where the PER can be further improved. In the matching LM setting, HMM training with monophone, triphone, and speaker adaptation training (SAT) improves the PER by a similar amount. In the non-matching LM setting, HMM training significantly improves the PER, and SAT additionally improves 3% in PER. Overall, our hybrid system with matching and non-matching LM achieved 36.5% and 41.6% PER, respectively, which is only 10% below the supervised system of the same architecture.
Unsupervised Phoneme Segmentation To understand how much our proposed boundary refinement method in Section 2.3 improves the segmentation quality, we follow the setting in previous works and report in Table 3 the recall, precision, F-score, and R-value with a 20-ms tolerance window on TIMIT’s training set (Scharenborg et al., 2010; Versteegh et al., 2016; Rasanen, 2014). We compare our results (obtained with matching LM) with several unsupervised phoneme segmentation methods (Dusan and Rabiner, 2006; Qiao et al., 2008; Lee and Glass, 2012; Rasanen, 2014; Hoang and Wang, 2015; Michel et al., 2016; Wang et al., 2017). Note that our refined segmentation significantly improves over the initial boundaries generated by Wang et al. (2017) and also outperforms other baselines. This result also confirms its contribution to the much improved phoneme recognition performance in the 2nd iteration (see “Our Model: 2nd iteration” in Table 2) to the 1st iteration. However, we emphasize that our method is designed towards unsupervised speech recognition rather than unsupervised phoneme segmentation. Estimating the segmentation boundary only serves as an auxiliary task to enable the unsupervised learning of the recognition model. And in the testing stage, there is no need to estimate the segmentation boundaries. Instead, our trained model could be directly used with a speech decoder just as any supervised recognition model would do.
Further analysis We include some further experiments and analysis in Appendix C, where we show the importance of the frame smoothness term in (3). We also compare the performance of our unsupervised algorithm to supervised learning with different amounts of labeled training data.

heading: 5 RELATED WORK
text: Unsupervised sequence-to-sequence learning Recently, unsupervised sequence-to-sequence learning has achieved great success in several problems. Liu et al. (2017) showed that it is possible to learn a sequence classifier without any labeled data by exploiting the output sequential structure using an unsupervised cost function named Empirical-ODM. Artetxe et al. (2018) and Lample et al. (2018) showed that unsupervised neural machine translation (uNMT) systems can be achieved by utilizing cross-lingual alignments and an adversarial structure without any form of parallel information. The success in the unsupervised sequence-to-sequence learning in various applications shed light on building our fully unsupervised speech recognition system. In particular, our work extends the Empirical-ODM in Liu et al. (2017) to problem with segmental structures.
Unsupervised speech segmentation One line of unsupervised segmentation methods designs robust acoustic features that are likely to remain stable within a phoneme, and capture the change of
features for phoneme boundaries (Esposito and Aversano, 2005; Hoang and Wang, 2015; Khanagha et al., 2014; Rasanen et al., 2011; Michel et al., 2016; Wang et al., 2017). Another line of research uses a simpler segmentation method as an initialization, and jointly trains the segmenting and acoustic models for phonemes or words (Kamper et al., 2015; Glass, 2003; Siu et al., 2014; Lee and Glass, 2012). Qiao et al. (2008) use dynamic programming methods in order to the derive optimal segmentation, but requires the number of segments and is not fully unsupervised. In Wang et al. (2017), the authors use the update gate of a GRNN autoencoder to discover the phoneme boundaries.
Unsupervised spoken term discovery Recently, the discovery of acoustic tokens including subword and word units has become a popular research topic (Dunbar et al. (2017); Versteegh et al. (2016); Burget et al.). The term “Spoken term discovery” includes lexicon discovery, word segmentation, and subword matching (Dunbar et al. (2017)). The standard approaches segment audio signals that are acoustically similar, and cluster the obtained segmented signals (Lee and Glass, 2012; Glass, 2012; Park and Glass, 2008; Driesen et al., 2012). Walter et al. (2013) uses the discovered unit index sequence as the transcription for the acoustic model training, similar to the HMM training in section 3. Kamper et al. (2017) iterates between the clustering and segmentation steps. Ondel et al. (2016) improves upon previous methods by replacing Gibbs sampling by variational inference, and Ondel et al. (2017) further improves the result by including a bigram language model. The effectiveness of these approaches has been demonstrated on query-by-example spoken term detection or by calculating the normalized mutual information between the self-discovered units and the actual labels. Overall, these methods differ from our method in that they segment and cluster the raw speech signals to self-discovered units, but does not recognize them into phoneme or word labels directly. More recently, Chung et al. (2018) show that unsupervised spoken word classification is possible by using adversarial cross-modal alignments similar to that in uNMT systems.
Unsupervised speech recognition with oracle segmentation There have been several attempts (Liu et al., 2018; Chen et al., 2018) on building an unsupervised speech recognition model inspired by the success of the uNMT. These methods first learn an embedding from the acoustic data, and then map the clustered embeddings to the output space by either adversarial training or iterative mapping. In contrast, our approach learns a neural network model that directly maps the raw acoustic features into the output space by optimizing the Segmental Empirical-ODM cost, and outperforms the upper bound of the above cluster-based approaches. Furthermore, all methods in Liu et al. (2018); Chen et al. (2018) assume that the phoneme boundaries are given by a supervised oracle. In contrast, our method iteratively estimates the boundaries without any labeled data, making it fully unsupervised.

heading: 6 CONCLUSION
text: We have developed a fully unsupervised learning algorithm for phoneme recognition. The algorithm alternates between two steps: (i) learn a phoneme classifier for a given set of phoneme segmentation boundaries, and (ii) refining the phoneme boundaries based on a given classifier. For the first step, we developed a novel unsupervised cost function named Segmental Empirical-ODM by generalizing the work (Liu et al., 2017) to segmental structures. For the second step, we developed an approximate MAP approach to refining the boundaries obtained from Wang et al. (2017). Our experimental results on TIMIT phoneme recognition task demonstrate the success of a fully unsupervised phoneme recognition system. Although the fully unsupervised system is still far away from the state-of-the-art supervised methods (e.g., supervised RNN transducer), we show that with oracle boundaries the performance of our algorithm could approach that of the supervised system with the same model architecture. This demonstrates the potential of our method if, in future work, we can further improve the accuracy of boundary estimation. We want to further point out that the techniques we proposed in this paper, although was evaluated in speech recognition, can be exploited to attack other similar sequence recognition problems where the source and destination sequences have different lengths and labels are not available or hard to get.

heading: A DERIVATION OF THE MAP ESTIMATE FOR THE SEGMENTATION BOUNDARIES
text: In this appendix, we derive the MAP estimate for y = (y1, . . . , yT ) given an input utterance sequence x = (y1, . . . , yT ). Specifically, we have
argmax y p(y|x) = argmax y p(y, x)
= argmax y
p(y)p(x|y)
(a) = argmax
y T∏ t=1 p(yt|y1, . . . , yt−1)pθ(xt|yt)
= argmax y T∏ t=1 p(yt|y1, . . . , yt−1) pθ(yt|xt)p(xt) p(yt)
(b) = argmax
y T∏ t=1 p(yt|y1, . . . , yt−1) pθ(yt|xt) p(yt)
(6)
where in step (a) we approximate the p(x|y) by its factored form and in step (b) we dropped the constant term that is independent of y.

heading: B DETAILED EXPERIMENT SETTING
text: We perform experiments on the TIMIT dataset where 6300 prompted English speech utterances are recorded. The phoneme transcription of these utterances are manually segmented and labelled with a lexicon of 61 distinct phoneme classes, where we compact the 61 phoneme classes into 48 phone classes and train the language model with the validation dataset, which is later used to train our main algorithm. These 48 phoneme classes are mapped to 39 phoneme classes for scoring phone error rate (Lee and Hon, 1989).
The 39 dimensional feature vectors including 13 mel-frequency cepstral coefficients (MFCC) plus its acceleration features that are extracted with 25 ms Hamming window at 10 ms interval. The classifier Pθ is modeled by a fully connected neural network with one hidden layer of 512 ReLU units. The input to the neural network is a concatenation of frames within a context window of size 11, and we repeat the starting or ending frames if the window has reached the start or end of the sentence.
The optimization of (3) is performed with momentum SGD with momentum 0.9 and learning rate of 10−3 with learning rate decay with a fixed schedule of increasing batchsize from 5000 to 20000 and decreasing temperature for softmax. The scheduler parameter is listed below: first 200 epochs with batchsize 500 and temperature 1.0, next 300 epochs with batchsize 5000 and temperature 0.9, followed by 300 epochs with batchsize 10000 and temperature 0.8, finally 300 epochs with batchsize 20000 and temperature 0.7. Whenever the batchsize is increased, we set the learning rate to the inital learning rate value. The scheduling procedure is determined by self-validation, and is not extensively tuned during the experiments.
In our experiments, we chose N = 5 for the N-gram in (1), and for computational issues we only consider the most frequent 10000 5-grams. We do not observe any noticeable performance drop by considering only the 10000 5-grams. Among the 5-gram language model PLM, 69553 5-grams (out of 485 possible 5-grams) are non-zero, and the top 10000 5-grams account for almost half of the probability. To sample τ , we use a standard truncated normal distribution for sampling the frame in every segment, with some necessary scaling and rounding. The truncated distribution is to ensure that our sampling will give us bounded frames that lie in the correct segment. This distribution can also be replaced by the uniform distribution. To obtain p(bt = 1|x), we apply a sigmoid function to the normalized Difference GAS value (defined as Wang et al. (2017)) that are local maxima. For the Difference GAS value that are not local maxima, we set it to -0.1 to get a low probability.
We randomly sample 10000 continuous frames to optimize (2), which is sampled every batch. λ in (3) is chosen to be 10−5. We use both frame error rate (FER) and phone error rate (PER) as our evaluation metrics. All phone error rate (PER) results reported has been obtained by a Kaldi decoder by considering the per-frame softmax value and the language model, and the weight between the two set to 1, which is fixed in all unsupervised setting.

heading: C ADDITIONAL EXPERIMENTS AND ANALYSIS
text: First, we examine the importance of the frame smoothness term in (3) in the fully unsupervised setting. In Figure 3(a), we show the FER of our model after the first iteration of Algorithm 1 for different values of λ. Note that when λ is close to the order of 10−5, the result does not differ a lot from the best result. However, when λ is set to zero, the performance degrades significantly. This confirms the importance of incorporating the temporal structure of speech data into the cost function, as discussed in Section 2.2. Second, we would like to study another important question regarding our unsupervised learning method: how much labeled data is it equivalent to? In Figure 3(b), we show the supervised neural network with different sizes of training data, where x-axis is the percentage of the original labeled set being used to train the model. We observe that with oracle boundary and matching LM, our algorithm is equivalent to supervised learning with 30% labeled data. With unsupervised boundary estimation, we still see a big performance loss. Therefore, it is critical to improve the boundary estimation performance in our future work.

Review summaries:
This paper proposes a new approach to do unsupervised phoneme recognition by learning from unlabelled speech in combination with a trained phoneme language model. The proposed loss function is a combination of a term encouraging the language model of predicted phonemes to match the given language model distribution, and a term to encourage adjacent speech frames to be assigned to the same phoneme class. Phoneme boundaries are iteratively refined using a separate model. Experiments where a hidden Markov model is applied on top of the predicted phonemes are also performed.

This paper presents a method to learn an acoustic model for phoneme recognition with only the training input acoustic features and a pretrained phoneme LM. This is done by matching the output phoneme sequence distribution of the training set with the phoneme LM distribution. The cost function is proposed by extending a previously proposed unsupervised cost function (Empirical-ODM) to the segmental level, and integrating an intra-segment cost function to encourage the frame-wise output distribution to be similar to each other within a segment. The authors conducted thorough experiments on TIMIT phoneme recognition and demonstrated impressive results.