Transformer VQ-VAE for Unsupervised
Unit Discovery and Speech Synthesis:
ZeroSpeech 2020 Challenge
Andros Tjandra1, Sakriani Sakti1,2, Satoshi Nakamura1,2
Nara Institute of Science and Technology, JapanRIKEN AIP, Japan© 2020 Andros Tjandra

1Outline• Background• Unsupervised subword discovery• VQVAE model & objective• Self-attention & Transformers• Model regularization• Codebook inverter• Experiment• Conclusion© 2020 Andros Tjandra

2

Background• The ZeroSpeech 2020 - Track 2019 challenge confronts the problem ofconstructing a speech synthesizer without any text or phonetic labels:
TTS without T.
• Two objectives:• Find related contexts from the speech and encodes them as
efficient as possible (low-bitrate).
• Using the encoded representation, reconstruct the content back tothe speech into another speaker voice.
© 2020 Andros Tjandra

3

Vector Quantized Variational Autoencoder(VQVAE)• VQ-VAE has three main modules:• Encoder 𝑞! 𝑧 𝑥 read speech features 𝑥 ∈ ℝ" andoutput 𝑧 ∈ 1. . 𝐾• Codebook 𝐸 = 𝑒#, . . , 𝑒$ ∈ ℝ$×"!
• Decoder 𝑝& 𝑥 𝑧, 𝑥 reconstruct the speech features
conditioned by codebook 𝑧 and speaker ID 𝑠

• Using explicit speaker information for thedecoder, encoder and codebook only need tomodel the speech context without capturing
the speaking style (disentangled with speaker).
© 2020 Andros Tjandra

4

Training Objective and Inference• The discretization process in the encoder :• Training objective:

Reconstruction loss
© 2020 Andros Tjandra

Consistency loss
5

Self Attention and Transformer
Dot Product Attention

Multihead Self Attention

Transformer module

Dropout
LinearDropout
ReLU
LinearLayerNorm

Dropout
MultiheadSelf-Attn

Image reference: Attention is all you need [Vaswani et al., NIPS 2017]© 2020 Andros Tjandra

LayerNorm!

6

VQVAE Encoder & Decoder blocksa) Encoder: 2x Transformer
layer + 1D convolution (withstride to downsample input)+ batchnormb) Decoder: 1D Convolution +Upsample operation +batchnorm© 2020 Andros Tjandra

7Model regularization• Sometimes, generative model could also suffer from overfitting
especially when the amount of data is small.
• We deploy several regularization technique to improve theperformance:
1. Temporal smoothing
2. Temporal jitter© 2020 Andros Tjandra8

Temporal Smoothing• Since the encoder captures sequential data, we introduced temporalsmoothing between two consecutive time-steps:• Final loss:© 2020 Andros Tjandra

9Temporal Jitter• Temporal jitter regularization [1] is used to prevent the latent vector
co-adaptation and to reduce the model sensitivity near the unitboundary.

[1] Unsupervised speech representation learning using Wavenet autoencoders, [Chorowski et al., 2019]© 2020 Andros Tjandra

10Codebook Inverter• Codebook inverter predicts thelinear spectrogram given thepredicted codebook from VQVAE.
• We use Griffin-Lim algorithm togenerate the speech waveform• Loss: 𝐿! = ||𝑋 " − 𝑋& " ||#© 2020 Andros Tjandra

11Experimental Setup• Log mel-spectrogram (80 dims) vs MFCC (39 dims with delta & deltadelta)• For the rest of experiments, we will use MFCC features.
© 2020 Andros Tjandra

12

Conv VQVAE vs Transformer VQVAE[1]-2.2 ABX[1] VQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverterfor Zerospeech Challenge 2019 [Tjandra et al, 2019]© 2020 Andros Tjandra

13Best model + regularization-0.24 ABX
-0.34 ABX
-1.77 ABX© 2020 Andros Tjandra

14

Conclusions• We achieved significant improvement with Transformer VQ-VAE(-2.2 ABX vs Conv VQ-VAE).
• By adding regularization in the VQVAE encoder and codebook, we getfurther improvement (up to -1.77 ABX vs un-regularized model.)© 2020 Andros Tjandra

15

The end.

© 2020 Andros Tjandra

16