Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis:
ZeroSpeech 2020 Challenge
Andros Tjandra1 , Sakriani Sakti1,2 , Satoshi Nakamura1,2
1

2

Nara Institute of Science and Technology, JapanRIKEN, Center for Advanced Intelligence Project AIP, Japan{andros.tjandra.ai6, ssakti, s-nakamura}@is.naist.jp

Abstract
In this paper, we report our submitted system for the ZeroSpeech 2020 challenge on Track 2019. The main theme inthis challenge is to build a speech synthesizer without any textual information or phonetic labels. In order to tackle thosechallenges, we build a system that must address two major components such as 1) given speech audio, extract subword units
in an unsupervised way and 2) re-synthesize the audio fromnovel speakers. The system also needs to balance the codebook performance between the ABX error rate and the bitratecompression rate. Our main contribution here is we proposed
Transformer-based VQ-VAE for unsupervised unit discovery
and Transformer-based inverter for the speech synthesis given
the extracted codebook. Additionally, we also explored several
regularization methods to improve performance even further.
Index Terms: unsupervised unit discovery, zero-speech, clustering, speech synthesize, deep learning

1. Introduction
Many speech technologies such as automatic speech recognition (ASR) and speech synthesis (TTS) has been used widelyaround the world. However, most such systems only cover richresource languages. For low-resource languages, using suchtechnologies remains limited since most of those systems require many labeled datasets to achieve good performance. Furthermore, some languages have more extreme limitations, including no written form or textual interpretation for their context. In the ZeroSpeech challenge [1], we addressed the latterproblem by only learning the language elements directly fromuntranscribed speech.
ZeroSpeech 2020 has three challenges: 2017 Track 1, 2017
Track 2, and 2019. In this paper, we focus on the ZeroSpeech2019 Track, where the task is called as TTS without Text. This
challenge has two major components that need to be addressed:1) given speech audio, extract subword-like units that contain
only context information with an unsupervised learning method
and 2) re-synthesize the speech into a different speaker’s voice.
To simultaneously solve these challenges, our strategy is
to develop a primary model that disentangles the speech signals into two major factors: context and speaker information.
After the disentanglement process, a secondary model predicts
the target speech representation given the context extracted by
the primary model. Following our success in a previous challenge [2] in ZeroSpeech 2019 and several prior publications on
quantization-based approaches [3–6], we used a vector quantized variational autoencoder (VQ-VAE) as the primary modeland a codebook inverter as the secondary model. To improve
our result, we introduced Transformer module [7] to capture
the long-term information from the sequential data inside the

VQ-VAE. We also explored several regularization methods toimprove the robustness and codebook discrimination scores.
Based on our experiment, all these combined methods significantly improved the ABX error rate.

2. Self-Attention and Transformer Module
The Transformer is a variant of deep learning modules that consist of several non-linear projections and a self-attention mechanism [7]. Unlike recurrent neural networks (RNNs) such as asimple RNN or long-short term memory (LSTM) [8] modules,
a Transformer module doesn’t have any recurrent connection
between the previous and current time-steps. However, a Transformer module utilizes self-attention modules to model the dependency across different time-steps.
Given input sequence X = [x1 , x2 , ..., xS ] ∈ RS×din
where S denotes the input length and din is the input dimension, a Transformer module produces hidden representation
Z = [z1 , z2 , ..., zS ] ∈ RS×din . Figure 1 shows the complete
process inside a Transformer module.
Figure 1: A Transformer module

Input sequence X is normalized by a layer-norm and processed by a multi-head self-attention module. Self-attention
module inputs are denoted as Q (query), K (key), and V (value)
where {Q, K, V } ∈ RS×dk . To calculate the output from aself-attention block, we apply the following formula:
QK T
Attention(Q, K, V ) = softmax √
V.
(1)
dk
Since a Transformer module employed multi-head selfattention instead of single head self-attention, input Q, K, V
needed to be varied to generate non-identical representation foreach head. Therefore, for each head p ∈ {1..P }, we projected
input X with different matrices WpQ , WpK , WpV and combined
the output of many single head self-attentions by concatenations
and a linear projection:
Multi-Head(Q, K, V ) = Concat(h1 , ..., hP )W O
(2)∀p ∈ {1..P }, hp = Attention(QWpQ , KWpK , V WpV )(3)where input projection matrices WpQ ∈ Rdmdl ×dk , WpK ∈Rdmdl ×dk , WpV ∈ Rdmdl ×dk , output projection matrix W O ∈RP dk ×dmdl and dimension of dk = dmdl
. After we got the outP
put from the self-attention, we applied layer normalization [9],two linear projections, and a rectified linear unit (ReLU) activation function.

3. Unsupervised Subword Discovery
Normally, a speech utterance can be factored into several bits oflatent information, including context, speaker’s speaking style,background noise, emotions, etc. Here, we assume that the
speech only contains two factors: context and the speaker’s
speaking style. In this case, the context denotes the unit that
captures the speech information itself in the discretized form,
which resembles phonemes or subwords. Therefore, to capture the context without any supervision, we used a generativemodel called a vector quantized variational autoencoder (VQVAE) [10] to extract the discrete symbols. There are some differences between a VQ-VAE with a normal autoencoder [11]and a normal variational autoencoder (VAE) [12] itself. The
VQ-VAE encoder maps the input features into a finite set of
discrete latent variables, and the standard autoencoder or VAE
encoder maps input features into continuous latent variables.
Therefore, a VQ-VAE encoder has stricter constraints due to thelimited number of codebooks, which enforces the latent variable compression explicitly via quantization. On the other hand,standard VAE encoder has a one-to-one mapping between the
input and latent variables. Due to the nature of the unsupervised subword discovery task, a VQ-VAE is more suitable forthe subword discovery task compared to a normal autoencoder
or VAE. Compared to the machine speech chain [13–16] idea,
the encoder is expected to capture only subword-like units canbe seen as pseudo-ASR and conditional decoder with speakerembedding can be seen as multi-speaker TTS.
We shows the VQ-VAE model in Fig. 2. Here, we define
E = [e1 , .., eK ] ∈ RK×De as a collection of codebook vectors and V = [v1 , .., vL ] ∈ RL×Dv as a collection of speakerembedding vectors. At the encoding step, input x denotes suchspeech features as MFCC (Mel-frequency cepstral coefficients)
or Mel-spectrogram and auxiliary input s ∈ {1, .., L} denotes
the speaker ID from speech feature x. In Fig. 3, we show the

Figure 2: VQ-VAE for unsupervised unit discovery consistsof several parts: encoder EncVθ Q (x) = qθ (y|x), decoder
DecVφ Q (y, s) = pφ (x|y, s), codebooks E = [e1 , .., eK ], andspeaker embedding V = [v1 , .., vL ].

Figure 3: Building block inside VQ-VAE encoder and decoder:a) Encoder consisted of 2x Transformer layer and 1D convolution with stride to downsample input sequence length; b) Decoder consisted of 1D convolution, followed by up-sampling torecover original input sequence shape.

details for the building block inside the encoder and decoder
modules. Encoder qθ (y|x) outputs a discrete latent variabley ∈ {1, ..K}. To transform an input in continuous variable
into a discrete variable, the encoder first produces intermediatecontinuous representation z ∈ RDe . Later on, we scan throughall codebooks to find which codebook has a minimum distance
between z and a vector in E. We define this process by thefollowing equations:(
1 if c = argmini Dist(z, ei )qθ (y = c|x) =(4)0 otherwise
ec = Eqθ (y|x) [E]
=

K
X

(5)

qθ (y = i|x) ei .

(6)

i=1

where Dist(·, ·) : RDe × RDe → R is a function that calculates the distance between two vectors. In this paper, we defineDist(a, b) = ka − bk2 as the L2-norm distance.
After the closest codebook index c ∈ {1, .., K} is found,
we substitute latent variable z with nearest codebook vector ec .
Later, decoder pφ (x|y, s) uses codebook vector ec and speakerembedding vs to reconstruct input feature x̂.
3.1. VQ-VAE objective
Given input reconstruction across all time-step ∀t ∈ [1..T ],X̂ = [x̂1 , .., x̂T ] and closest codebook index [c1 , .., cT ], we
calculate the VQ-VAE objective:
LV Q =T
X− log pφ (xt |yt , s) + γkzt − sg(ect )k22 , ,

(7)

t=1

where function sg(·) stops the gradient:x = sg(x);∂ sg(x)= 0.
∂x

The final loss is defined:
L = LV Q + λLreg ,(10)where λ denotes the coefficient for the regularization term.
3.2.2. Temporal jitter
Temporal jitter regularization [3] is used to prevent the latentvector co-adaptation and to reduce the model sensitivity near
the unit boundary. In the practice, we could apply the temporal
jitter by:jt ∼ Categorical(p, p, 1 − 2 ∗ p) ∈ {1, 2, 3}
ct−1 , if jt = 1 and t > 1cˆt = ct+1 , if jt = 2 and t < T

c ,
else
t

(11)

et = E[cˆt ]

(13)

(12)

where p is the jitter probability, ct is the closest codebook index
at time-t and cˆt is the new assigned codebook index after thejitter operation.

4. Codebook Inverter
A codebook inverter model is used to generate the speech representation given the predicted codebook from our Transformer
VQ-VAE. The input is [E[c1 ], .., E[cT ]], and the output is thefollowing speech representation sequence (here we use linearRmagnitude spectrogram): XR = [XR
1 , .., XS ].
In Fig. 4, we show our codebook inverter architecture thatconsists of two multiscale 1D convolutions and three Transformer layers with additional sinusoidal position encoding tohelp the model distinguish the duplicated codebook positions.
Depends on the VQ-VAE encoder architecture, the predicted(8)

The first term is a negative log-likelihood to measure the reconstruction loss between original input xt and reconstructionx̂t to optimize encoder parameters θ and decoder parameters
φ. The second term minimizes the distance between intermediate representation zt and nearest codebook ect , but the gradientis only back-propagated into encoder parameters θ as commitment loss. The impact from commitment loss controlled witha hyper-parameter γ (we use γ = 0.25). To update the codebook vectors, we use an exponential moving average (EMA)[17]. EMA updates the codebook E independently regardlessof the optimizer’s type and update rules, therefore the model
is more robust against different optimizer’s choice and hyperparameters (e.g., learning rate, momentum) and also avoids posterior collapse problem [18].
3.2. Model regularization
3.2.1. Temporal smoothing
Since our input datasets are sequential data, we introduced temporal smoothing between the encoded hidden vectors between
two consecutive time-steps:
Lreg =

T−1X
i=1

kzt − zt+1 k22 .

(9)Figure 4: Codebook inverter: given codebook sequence[E[y1 ], .., E[yT ]], we predict corresponding linear magnitude
R
spectrogram X̂R = [xR
1 , ..xS ]. If lengths between S and T are
different, we consecutively duplicate each codebook by r-times.

codebook sequence length T could be shorter than S because
the VQ-VAE encoder qθ (y|x) have several convolutional layerswith stride larger than one. Therefore, to re-align the codebook
sequence representation with the speech representation, we duplicate each codebook occurrences ∀t ∈ {1..T } E[ct ] into r
copies side-by-side where r = S/T . To train a codebook inverter, we set the objective function:
LIN V = kXR − X̂R k2(14)

to minimize the L2-norm between predicted spectrogram
X̂R = Invρ ([E[c1 ], ..., E[cT ]]) and groundtruth spectrogram
XR . We defined Invρ as the inverter parameterized by ρ. During the inference step, Griffin-Lim [19] is used to reconstruct
the phase from the spectrogram and applied an inverse shortterm Fourier transform (STFT) to invert it into a speech waveform.

5. Experiment
In this section, we describe our pipeline, including feature preprocessing, model settings, and hyperparameters.
5.1. Experimental Set-up
There are two datasets for two languages, English data for thedevelopment dataset, and a surprising Austronesian language
for the test dataset. Each language dataset contains subset
datasets: (1) a voice dataset for speech synthesis, (2) a unit discovery dataset, (3) an optional parallel dataset from the target
voice to another speaker’s voice, and (4) a test dataset. The
source corpora for the surprise language are described here
[20, 21], and further details can be found [1]. We only used
(1) and (2) for training the VQ-VAE codebook and inverter and(4) for evaluation.
For the speech input, we experimented with two different
feature representations, such as Mel-spectrogram (with 80 dimensions, 25-ms window size, 10-ms time-steps) and MFCC
(with 39 dimensions, 13 main +∆ + ∆2 , 25-ms window size,
10-ms time-steps). Both MFCC and Mel-spectrogram are generated by the Librosa package [22]. All models are implemented with PyTorch library [23]. We use Adam [12] optimizer
with learning rate 1e − 4 for training both models.
5.2. Experimental Results
In this subsection, we report our experiment based on different
scenarios and hyperparameters with the English dataset. First,
we use different features as the input for VQ-VAE in Table 1.
We compared the effect of using log Mel-spectrogram with 80
dimensions versus MFCC+∆ + ∆2 with 39 dimensions.
Table 1: ABX and Bitrate result between MFCC and log Melspectrogram features on Transformer VQ-VAE (TrfVQVAE with
K = 128, stride 4×)
Model
TrfVQVAE with log-Mel
TrfVQVAE with MFCC

ABX
33.79
21.91

Bitrate171.05170.42

Based on the result on Table 1, we determined that MFCC
provides much better results in terms of ABX error rate. From
this point, we use MFCC as the VQ-VAE input features. In
Table 2, we compared our proposed model with our submission from previous challenge [2] and we also explored different
codebook sizes to determine the trade-off between the ABX error rate and bitrate.

Table 2: ABX and Bitrate result between our last year submission and new proposed Transformer VQ-VAE.
ModelConv VQVAE (stride 4×, K=256) [2]
TrfVQVAE (stride 4×, K=64)
TrfVQVAE (stride 4×, K=128)
TrfVQVAE (stride 4×, K=256)
TrfVQVAE (stride 4×, K=512)

ABX
24.17
22.72
21.91
21.94
21.6

Bitrate184.32141.82
170.42194.69217.47

Based on the result on Table 2, by using Transformer VQVAE, we outperformed our previous best submission by -2.2
ABX error rate and with a similar bitrate. Later on, by using small codebook sizes K = 64, we reduce the bitrate by-30 points compared to K = 128, but we sacrifice the +0.8
ABX error rate. We explored different regularization methods
to improve transformer VQ-VAEs performance in Table 3. In
Table 3, we show the result from Transformer VQ-VAE (stride
4×, K=128) with different λ smoothing coefficient, different
jitter probability and the combination between both regularization methods. We found out that by combining both temporalsmoothing and jittering, we get -1.77 ABX error rate improvement.
Table 3: ABX and Bitrate result between different temporalsmoothing coefficient λ, jitter probability p, and their combination for Transformer VQ-VAE regularization. (Notes: F denotes our submitted system. After the submission, we continue
the experiment and we found out another hyperparameters combination brings even lower ABX than our submitted system.)
Model
TrfVQVAE (stride 4×, K=128)+ temp smooth (λ = 1e − 2)
+ temp smooth (λ = 5e − 3)
+ temp smooth (λ = 1e − 3)
+ temp jitter (p = 0.05)+ temp jitter (p = 0.075)+ temp smooth (λ = 5e − 3) F
+ temp jitter (p = 0.05)+ temp smooth (λ = 1e − 3)
+ temp jitter (p = 0.05)

ABX
21.91
21.88
21.67
21.75
21.57
21.70

Bitrate170.42169.02169.2
169.56166.19164.0820.71

171.9920.14

167.02

6. Conclusions
We described our approach for the ZeroSpeech 2020 challenge
on Track 2019. For the unsupervised unit discovery task, we
proposed a new architecture: Transformer VQ-VAE to capture
the context of the speech into a sequence of discrete latent variables. Additionally, we also use the Transformer block inside
our codebook inverter architecture. Compared to our last year’s
submission, replacing convolutional layers with Transformer
gives a significant improvement up to -2.2 ABX error rate. To
improve the result further, we investigated several regularization
methods. In the end, combining both temporal smoothing andjittering improved the Transformer VQ-VAE performance up to-1.77 ABX error rate compared to the un-regularized model.

7. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101 and JP17K00237.

8. References[1] E. Dunbar, R. Algayres, J. Karadayi, M. Bernard, J. Benjumea,
X.-N. Cao Kam, L. Miskic, C. Dugrain, L. Ondel, A. Black,L. Besacier, S. Sakti, and E. Dupoux, “The Zero Resource Speech
Challenge 2019: TTS without T,” Proc. Interspeech 2019, pp.
1088–1092, 09 2019.
[2] A. Tjandra, B. Sisman, M. Zhang, S. Sakti, H. Li, and
S. Nakamura, “VQVAE Unsupervised Unit Discovery andMulti-Scale Code2Spec Inverter for Zerospeech Challenge 2019,”
in Proc. Interspeech 2019, 2019, pp. 1118–1122. [Online].
Available: http://dx.doi.org/10.21437/Interspeech.2019-3232[3] J. Chorowski, R. J. Weiss, S. Bengio, and A. v. d. Oord, “Unsupervised speech representation learning using Wavenet autoencoders,” arXiv preprint arXiv:1901.08810, 2019.
[4] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Selfsupervised learning of discrete speech representations,” arXiv
preprint arXiv:1910.05453, 2019.
[5] R. Eloff, A. Nortje, B. van Niekerk, A. Govender, L. Nortje,
A. Pretorius, E. Van Biljon, E. van der Westhuizen, L. van Staden,and H. Kamper, “Unsupervised acoustic unit discovery for speechsynthesis using discrete latent-variable neural networks,” arXiv
preprint arXiv:1904.07556, 2019.
[6] A. Tjandra, S. Sakti, and S. Nakamura, “Speech-to-speech translation between untranscribed unknown languages,” in 2019 IEEE
Automatic Speech Recognition and Understanding Workshop(ASRU), 2019, pp. 593–600.
[7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N.
Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,”
in Advances in Neural Information Processing Systems, 2017, pp.
5998–6008.
[8] S. Hochreiter and J. Schmidhuber, “Long short-term memory,”
Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[9] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”
arXiv preprint arXiv:1607.06450, 2016.
[10] A. van den Oord, O. Vinyals et al., “Neural discrete representation
learning,” in Advances in Neural Information Processing Systems,
2017, pp. 6306–6315.
[11] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in Proceedings of the 25th international conference on
Machine learning. ACM, 2008, pp. 1096–1103.
[12] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
[13] A. Tjandra, S. Sakti, and S. Nakamura, “Listening while speaking:Speech chain by deep learning,” in 2017 IEEE Automatic Speech
Recognition and Understanding Workshop (ASRU). IEEE, 2017,
pp. 301–308.
[14] ——, “Machine speech chain with one-shot speaker adaptation,”
Proc. Interspeech 2018, pp. 887–891, 2018.
[15] A. Tjandra, S. Sakti, and S. Nakamura, “End-to-end feedback
loss in speech chain framework via straight-through estimator,” in
ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp.
6281–6285.
[16] ——, “Machine speech chain,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 976–989,
2020.
[17] L. Kaiser, A. Roy, A. Vaswani, N. J. Parmar, S. Bengio,
J. Uszkoreit, and N. Shazeer, “Fast decoding in sequence models
using discrete latent variables,” ICML, 2018. [Online]. Available:
https://arxiv.org/abs/1803.03382[18] A. Roy, A. Vaswani, A. Neelakantan, and N. Parmar,“Theory and experiments on vector quantized autoencoders,”
CoRR, vol. abs/1805.11063, 2018. [Online]. Available: http://arxiv.org/abs/1805.11063[19] D. Griffin and J. Lim, “Signal estimation from modified shorttime Fourier transform,” IEEE Transactions on Acoustics, Speech,
and Signal Processing, vol. 32, no. 2, pp. 236–243, 1984.
[20] S. Sakti, R. Maia, S. Sakai, T. Shimizu, and S. Nakamura, “Development of HMM-based Indonesian speech synthesis,” Proc. Oriental COCOSDA, pp. 215–219, 2008.
[21] S. Sakti, E. Kelana, H. Riza, S. Sakai, K. Markov, and
S. Nakamura, “Development of Indonesian large vocabularycontinuous speech recognition system within A-STAR project,”in Proceedings of the Workshop on Technologies and Corpora
for Asia-Pacific Speech Translation (TCAST), 2008. [Online].
Available: https://www.aclweb.org/anthology/I08-8004[22] B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis
in Python,” 2015, pp. 18–25.
[23] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan,
T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison,
A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch:An imperative style, high-performance deep learning library,” inAdvances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and
R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024–8035.