Introduction
Nowadays speech processing is dominated by deep learning techniques. Deep neural network (DNN) acoustic models (AMs) for the tasks of automatic speech recognition (ASR) and speech synthesis have shown impressive performance for major languages such as English and Mandarin. Typically, training a DNN AM requires large amounts of transcribed data. For a large number of low-resource languages, for which very limited or no transcribed data are available, conventional methods of acoustic modeling are ineffective or even inapplicable.
In recent years, there has been an increasing research interest in zero-resource speech processing, i.e., only a limited amount of raw speech data (e.g. hours or tens of hours) are given while no text transcriptions or linguistic knowledge are available. The Zero Resource Speech Challenges (ZeroSpeech) 2015 BIBREF0 , 2017 BIBREF1 and 2019 BIBREF2 precisely focus on this area. One problem tackled by ZeroSpeech 2015 and 2017 is subword modeling, learning frame-level speech representation that is discriminative to subword units and robust to linguistically-irrelevant factors such as speaker change. The latest challenge ZeroSpeech 2019 goes a step further by aiming at building text-to-speech (TTS) systems without any text labels (TTS without T) or linguistic expertise. Specifically, one is required to build an unsupervised subword modeling sub-system to automatically discover phoneme-like units in the concerned language, followed by applying the learned units altogether with speech data from which the units are inferred to train a TTS. Solving this problem may partially assist psycholinguists in understanding young children's language acquisition mechanism BIBREF2 .
This study addresses unsupervised subword modeling in ZeroSpeech 2019, which is also referred to as acoustic unit discovery (AUD). It is an essential problem and forms the basis of TTS without T. The exact goal of this problem is to represent untranscribed speech utterances by discrete subword unit sequences, which is slightly different from subword modeling in the contexts of ZeroSpeech 2017 & 2015. In practice, it can be formulated as an extension to the previous two challenges. For instance, after learning the subword discriminative feature representation at frame-level, the discrete unit sequences can be inferred by applying vector quantization methods followed by collapsing consecutive repetitive symbolic patterns. In the previous two challenges, several unsupervised representation learning approaches were proposed for comparison, such as cluster posteriorgrams (PGs) BIBREF3 , BIBREF4 , BIBREF5 , DNN bottleneck features BIBREF6 , BIBREF7 , autoencoders (AEs) BIBREF8 , BIBREF9 , variational AEs (VAEs) BIBREF10 , BIBREF11 and siamese networks BIBREF12 , BIBREF13 , BIBREF14 .
One major difficulty in unsupervised subword modeling is dealing with speaker variation. The huge performance degradation caused by speaker variation reported in ZeroSpeech 2017 BIBREF1 implies that speaker-invariant representation learning is crucial and remains to be solved. In ZeroSpeech 2019, speaker-independent subword unit inventory is highly desirable in building a TTS without T system. In the literature, many works focused on improving the robustness of unsupervised feature learning towards speaker variation. One direction is to apply linear transform methods. Heck et al. BIBREF5 estimated fMLLR features in an unsupervised manner. Works in BIBREF6 , BIBREF15 estimated fMLLR using a pre-trained out-of-domain ASR. Chen et al. BIBREF7 applied vocal tract length normalization (VTLN). Another direction is to employ DNNs. Zeghidour et al. BIBREF13 proposed to train subword and speaker same-different tasks within a triamese network and untangle linguistic and speaker information. Chorowski et al. BIBREF11 defined a speaker embedding as a condition of VAE decoder to free the encoder from capturing speaker information. Tsuchiya et al. BIBREF16 applied speaker adversarial training in a task related to the zero-resource scenario but transcription for a target language was used in model training.
In this paper, we propose to extend our recent research findings BIBREF10 on applying disentangled speech representation learned from factorized hierarchical VAE (FHVAE) models BIBREF17 to improve speaker-invariant subword modeling. The contributions made in this study are in several aspects. First, the FHVAE based speaker-invariant learning is compared with speaker adversarial training in the strictly unsupervised scenario. Second, the combination of adversarial training and disentangled representation learning is studied. Third, our proposed approaches are evaluated on the latest challenge ZeroSpeech 2019, as well as on ZeroSpeech 2017 for completeness. To our best knowledge, direct comparison of the two approaches and their combination has not been studied before.
General framework
The general framework of our proposed approaches is illustrated in Figure FIGREF2 . Given untranscribed speech data, the first step is to learn speaker-invariant features to support frame labeling. The FHVAE model BIBREF17 is adopted for this purpose. FHVAEs disentangle linguistic content and speaker information encoded in speech into different latent representations. Compared with raw MFCC features, FHVAE reconstructed features conditioned on latent linguistic representation are expected to keep linguistic content unchanged and are more speaker-invariant. Details of the FHVAE structure and feature reconstruction methods are described in Section SECREF3 .
The reconstructed features are fed as inputs to Dirichlet process Gaussian mixture model (DPGMM) BIBREF18 for frame clustering, as was done in BIBREF3 . The frame-level cluster labels are regarded as pseudo phone labels to support supervised DNN training. Motivated by successful applications of adversarial training BIBREF19 in a wide range of domain invariant learning tasks BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 , this work proposes to add an auxiliary adversarial speaker classification task to explicitly target speaker-invariant feature learning. After speaker adversarial multi-task learning (AMTL) DNN training, softmax PG representation from pseudo phone classification task is used to infer subword unit sequences. The resultant unit sequences are regarded as pseudo transcriptions for subsequent TTS training.
Speaker-invariant feature learning by FHVAEs
The FHVAE model formulates the generation process of sequential data by imposing sequence-dependent and sequence-independent priors to different latent variables BIBREF17 . It consists of an inference model INLINEFORM0 and a generation model INLINEFORM1 . Let INLINEFORM2 denote a speech dataset with INLINEFORM3 sequences. Each INLINEFORM4 contains INLINEFORM5 speech segments INLINEFORM6 , where INLINEFORM7 is composed of fixed-length consecutive frames. The FHVAE model generates a sequence INLINEFORM8 from a random process as follows: (1) An s-vector INLINEFORM9 is drawn from a prior distribution INLINEFORM10 ; (2) Latent segment variables INLINEFORM11 and latent sequence variables INLINEFORM12 are drawn from INLINEFORM13 and INLINEFORM14 respectively; (3) Speech segment INLINEFORM15 is drawn from INLINEFORM16 . Here INLINEFORM17 denotes standard normal distribution, INLINEFORM18 and INLINEFORM19 are parameterized by DNNs. The joint probability for INLINEFORM20 is formulated as, DISPLAYFORM0
Since the exact posterior inference is intractable, the FHVAE introduces an inference model INLINEFORM0 to approximate the true posterior, DISPLAYFORM0
Here INLINEFORM0 and INLINEFORM1 are all diagonal Gaussian distributions. The mean and variance values of INLINEFORM2 and INLINEFORM3 are parameterized by two DNNs. For INLINEFORM4 , during FHVAE training, a trainable lookup table containing posterior mean of INLINEFORM5 for each sequence is updated. During testing, maximum a posteriori (MAP) estimation is used to infer INLINEFORM6 for unseen test sequences. FHVAEs optimize the discriminative segmental variational lower bound which was defined in BIBREF17 . It contains a discriminative objective to prevent INLINEFORM7 from being the same for all utterances.
After FHVAE training, INLINEFORM0 encodes segment-level factors e.g. linguistic information, while INLINEFORM1 encodes sequence-level factors that are relatively consistent within an utterance. By concatenating training utterances of the same speaker into a single sequence for FHVAE training, the learned INLINEFORM2 is expected to be discriminative to speaker identity. This work considers applying s-vector unification BIBREF10 to generate reconstructed feature representation that keeps linguistic content unchanged and is more speaker-invariant than the original representation. Specifically, a representative speaker with his/her s-vector (denoted as INLINEFORM3 ) is chosen from the dataset. Next, for each speech segment INLINEFORM4 of an arbitrary speaker INLINEFORM5 , its corresponding latent sequence variable INLINEFORM6 inferred from INLINEFORM7 is transformed to INLINEFORM8 , where INLINEFORM9 denotes the s-vector of speaker INLINEFORM10 . Finally the FHVAE decoder reconstructs speech segment INLINEFORM11 conditioned on INLINEFORM12 and INLINEFORM13 . The features INLINEFORM14 form our desired speaker-invariant representation.
Speaker adversarial multi-task learning
Speaker adversarial multi-task learning (AMTL) simultaneously trains a subword classification network ( INLINEFORM0 ), a speaker classification network ( INLINEFORM1 ) and a shared-hidden-layer feature extractor ( INLINEFORM2 ), where INLINEFORM3 and INLINEFORM4 are set on top of INLINEFORM5 , as illustrated in Figure FIGREF2 . In AMTL, the error is reversely propagated from INLINEFORM6 to INLINEFORM7 such that the output layer of INLINEFORM8 is forced to learn speaker-invariant features so as to confuse INLINEFORM9 , while INLINEFORM10 tries to correctly classify outputs of INLINEFORM11 into their corresponding speakers. At the same time, INLINEFORM12 learns to predict the correct DPGMM labels of input features, and back-propagate errors to INLINEFORM13 in a usual way.
Let INLINEFORM0 and INLINEFORM1 denote the network parameters of INLINEFORM2 and INLINEFORM3 , respectively. With the stochastic gradient descent (SGD) algorithm, these parameters are updated as, p p - Lpp, s s - Lss,
h h -[Lph - Lsh], where INLINEFORM0 is the learning rate, INLINEFORM1 is the adversarial weight, INLINEFORM2 and INLINEFORM3 are the loss values of subword and speaker classification tasks respectively, both in terms of cross-entropy. To implement Eqt. ( SECREF6 ), a gradient reversal layer (GRL) BIBREF19 was designed to connect INLINEFORM4 and INLINEFORM5 . The GRL acts as identity transform during forward-propagation and changes the sign of loss during back-propagation. After training, the output of INLINEFORM6 is speaker-invariant and subword discriminative bottleneck feature (BNF) representation of input speech. Besides, the softmax output representation of INLINEFORM7 is believed to carry less speaker information than that without performing speaker adversarial training.
Subword unit inference and smoothing
Subword unit sequences for the concerned untranscribed speech utterances are inferred from softmax PG representation of INLINEFORM0 in the speaker AMTL DNN. For each input frame to the DNN, the DPGMM label with the highest probability in PG representation is regarded as the subword unit assigned to this frame. These frame-level unit labels are further processed by collapsing consecutive repetitive labels to form pseudo transcriptions.
We observed non-smoothness in the inferred unit sequences by using the above methods, i.e., frame-level unit labels that are isolated without temporal repetition. Considering that ground-truth phonemes generally span at least several frames, these non-smooth labels are unwanted. This work proposes an empirical method to filter out part of the non-smooth unit labels, which is summarized in Algorithm SECREF7 .
[h] Frame-level unit labels INLINEFORM0 Pseudo transcription INLINEFORM1 INLINEFORM2 }, where INLINEFORM3 , INLINEFORM4 for INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8 ; INLINEFORM9 INLINEFORM10 Unit sequence smoothing
Dataset and evaluation metric
ZeroSpeech 2017 development dataset consists of three languages, i.e. English, French and Mandarin. Speaker information for training sets are given while unknown for test sets. The durations of training sets are INLINEFORM0 and INLINEFORM1 hours respectively. Detailed information of the dataset can be found in BIBREF1 .
The evaluation metric is ABX subword discriminability. Basically, it is to decide whether INLINEFORM0 belongs to INLINEFORM1 or INLINEFORM2 if INLINEFORM3 belongs to INLINEFORM4 and INLINEFORM5 belongs to INLINEFORM6 , where INLINEFORM7 and INLINEFORM8 are speech segments, INLINEFORM9 and INLINEFORM10 are two phonemes that differ in the central sound (e.g., “beg”-“bag”). Each pair of INLINEFORM11 and INLINEFORM12 is spoken by the same speaker. Depending on whether INLINEFORM13 and INLINEFORM14 are spoken by the same speaker, ABX error rates for across-/within-speaker are evaluated separately.
System setup
The FHVAE model is trained with merged training sets of all three target languages. Input features are fixed-length speech segments of 10 frames. Each frame is represented by a 13-dimensional MFCC with cepstral mean normalization (CMN) at speaker level. During training, speech utterances spoken by the same speaker are concatenated to a single training sequence. During the inference of hidden variables INLINEFORM0 and INLINEFORM1 , input segments are shifted by 1 frame. To match the length of latent variables with original features, the first and last frame are padded. To generate speaker-invariant reconstructed MFCCs using the s-vector unification method, a representative speaker is selected from training sets. In this work the English speaker “s4018” is chosen. The encoder and decoder networks of the FHVAE are both 2-layer LSTM with 256 neurons per layer. Latent variable dimensions for INLINEFORM2 and INLINEFORM3 are 32. FHVAE training is implemented by using an open-source tool BIBREF17 .
The FHVAE based speaker-invariant MFCC features with INLINEFORM0 and INLINEFORM1 are fed as inputs to DPGMM clustering. Training data for the three languages are clustered separately. The numbers of clustering iterations for English, French and Mandarin are INLINEFORM2 and 1400. After clustering, the numbers of clusters are INLINEFORM3 and 314. The obtained frame labels support multilingual DNN training. DNN input features are MFCC+CMVN. The layer-wise structure of INLINEFORM4 is INLINEFORM5 . Nonlinear function is sigmoid, except the linear BN layer. INLINEFORM6 contains 3 sub-networks, one for each language. The sub-network contains a GRL, a feed-forward layer (FFL) and a softmax layer. The GRL and FFL are 1024-dimensional. INLINEFORM7 also contains 3 sub-networks, each having a 1024-dimensional FFL and a softmax layer. During AMTL DNN training, the learning rate starts from INLINEFORM8 to INLINEFORM9 with exponential decay. The number of epochs is 5. Speaker adversarial weight INLINEFORM10 ranges from 0 to INLINEFORM11 . After training, BNFs extracted from INLINEFORM12 are evaluated by the ABX task. DNN is implemented using Kaldi BIBREF24 nnet3 recipe. DPGMM is implemented using tools developed by BIBREF18 .
DPGMM clustering towards raw MFCC features is also implemented to generate alternative DPGMM labels for comparison. In this case, the numbers of clustering iterations for the three languages are INLINEFORM0 and 3000. The numbers of clusters are INLINEFORM1 and 596. The DNN structure and training procedure are the same as mentioned above.
FHVAE model training and speaker-invariant MFCC reconstruction are performed following the configurations in ZeroSpeech 2017. The unit dataset is used for training. During MFCC reconstruction, a male speaker for each of the two languages is randomly selected as the representative speaker for s-vector unification. Our recent research findings BIBREF10 showed that male speakers are more suitable than females in generating speaker-invariant features. The IDs of the selected speakers are “S015” and “S002” in English and Surprise respectively. In DPGMM clustering, the numbers of clustering iterations are both 320. Input features are reconstructed MFCCs+ INLINEFORM0 + INLINEFORM1 . After clustering, the numbers of clusters are 518 and 693. The speaker AMTL DNN structure and training procedure follow configurations in ZeroSpeech 2017. One difference is the placement of adversarial sub-network INLINEFORM2 . Here INLINEFORM3 is put on top of the FFL in INLINEFORM4 instead of on top of INLINEFORM5 . Besides, the DNN is trained in a monolingual manner. After DNN training, PGs for voice and test sets are extracted. BNFs for test set are also extracted. Adversarial weights INLINEFORM6 ranging from 0 to INLINEFORM7 with a step size of INLINEFORM8 are evaluated on English test set.
The TTS model is trained with voice dataset and their subword unit sequences inferred from PGs. TTS training is implemented using tools BIBREF27 in the same way as in the baseline. The trained TTS synthesizes speech waveforms according to unit sequences inferred from test speech utterances. Algorithm SECREF7 is applied to voice set and optionally applied to test set.
Experimental results
Average ABX error rates on BNFs over three target languages with different values of INLINEFORM0 are shown in Figure FIGREF11 .
In this Figure, INLINEFORM0 denotes that speaker adversarial training is not applied. From the dashed (blue) lines, it can be observed that speaker adversarial training could reduce ABX error rates in both across- and within-speaker conditions, with absolute reductions of INLINEFORM1 and INLINEFORM2 respectively. The amount of improvement is in accordance with the findings reported in BIBREF16 , despite that BIBREF16 exploited English transcriptions during training. The dash-dotted (red) lines show that when DPGMM labels generated by reconstructed MFCCs are employed in DNN training, the positive impact of speaker adversarial training in across-speaker condition is relatively limited. Besides, negative impact is observed in within-speaker condition. From Figure FIGREF11 , it can be concluded that for the purpose of improving the robustness of subword modeling towards speaker variation, frame labeling based on disentangled speech representation learning is more prominent than speaker adversarial training.
ABX error rates on subword unit sequences, PGs and BNFs with different values of INLINEFORM0 evaluated on English test set are shown in Figure FIGREF16 .
Algorithm SECREF7 is not applied at this stage. It is observed that speaker adversarial training could achieve INLINEFORM0 and INLINEFORM1 absolute error rate reductions on PG and BNF representations. The unit sequence representation does not benefit from adversarial training. Therefore, the optimal INLINEFORM2 for unit sequences is 0. The performance gap between frame-level PGs and unit sequences measures the phoneme discriminability distortion caused by the unit inference procedure in this work.
We fix INLINEFORM0 to train the TTS model, and synthesize test speech waveforms using the trained TTS. Experimental results of our submission systems are summarized in Table TABREF17 .
In this Table, “+SM” denotes applying sequence smoothing towards test set unit labels. Compared with the official baseline, our proposed approaches could significantly improve unit quality in terms of ABX discriminability. Our system without applying SM achieves INLINEFORM0 and INLINEFORM1 absolute error rate reductions in English and Surprise sets. If SM is applied, while the ABX error rate increases, improvements in all the other evaluation metrics are observed. This implies that for the goal of speech synthesis, there is a trade off between quality and quantity of the learned subword units. Besides, our ABX performance is competitive to, or even better than the supervised topline.
Our systems do not outperform baseline in terms of synthesis quality. One possible explanation is that our learned subword units are much more fine-grained than those in the baseline AUD, making the baseline TTS less suitable for our AUD system. In the future, we plan to investigate on alternative TTS models to take full advantage of our learned subword units.
Dataset and evaluation metrics
ZeroSpeech 2019 BIBREF2 provides untranscribed speech data for two languages. English is used for development while the surprise language (Indonesian) BIBREF25 , BIBREF26 is used for test only. Each language pack consists of training and test sets. The training set consists of a unit discovery dataset for building unsupervised subword models, and a voice dataset for training the TTS system. Details of ZeroSpeech 2019 datasets are listed in Table TABREF13 .
There are two categories of evaluation metrics in ZeroSpeech 2019. The metrics for text embeddings, e.g. subword unit sequences, BNFs and PGs, are ABX discriminability and bitrate. Bitrate is defined as the amount of information provided in the inferred unit sequences. The metrics for synthesized speech waveforms are character error rate (CER), speaker similarity (SS, 1 to 5, larger is better) and mean opinion score (MOS, 1 to 5, larger is better), all evaluated by native speakers.
Conclusions
This study tackles robust unsupervised subword modeling in the zero-resource scenario. The robustness towards speaker variation is achieved by combining speaker adversarial training and FHVAE based disentangled speech representation learning. Our proposed approaches are evaluated on ZeroSpeech 2019 and ZeroSpeech 2017. Experimental results on ZeroSpeech 2017 show that both approaches are effective while the latter is more prominent, and that their combination brings further marginal improvement in across-speaker condition. Results on ZeroSpeech 2019 show that our approaches achieve significant ABX error rate reduction to the baseline system. The proposed unit sequence smoothing algorithm improves synthesis quality, at a cost of slight decrease in ABX discriminability.
Acknowledgements
This research is partially supported by the Major Program of National Social Science Fund of China (Ref:13&ZD189), a GRF project grant (Ref: CUHK 14227216) from Hong Kong Research Grants Council and a direct grant from CUHK Research Committee.

Question: What is the performance difference in performance in unsupervised feature learning between adverserial training and FHVAE-based disentangled speech represenation learning?
Answer: Unanswerable