VQVAE Unsupervised Unit Discovery andMulti-scale Code2Spec Inverter
for Zerospeech Challenge 2019
Andros Tjandra1,2, Berrak Sisman3, Mingyang Zhang3, Sakriani Sakti1,2 ,Haizhou Li3, Satoshi Nakamura1,2
1 Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Japan
2 RIKEN, Center for Advanced Intelligence Project AIP (RIKEN AIP), Japan
3 Department of Electrical and Computer Engineering, National University of Singapore (NUS), Singapore

Copyright @ AHC Lab NAIST - 20191

Outline• Background• Previous Works• Proposed Approach:→ Vector Quantized Variational Autoencoder (VQVAE)→ Codebook-to-Spectrogram Inverter (Code2Spec)• Experiments• Conclusion

Copyright @ AHC Lab NAIST - 20192

Background• The ZeroSpeech 2019 challenge:Confronts the problem of constructing a speech synthesizer
without any text or phonetic labels: TTS without T• Two objectives:→ Discover subword units in an unsupervised way(Encodes them as efficient as possible -- low-bitrate)→ Using the encoded representation, synthesize the speech
to a different target speaker

Copyright @ AHC Lab NAIST - 20193

Previous Works• Top performance in ZeroSpeech 2015 & 2017:→ Unsupervised clustering with DPGMM [Chen et al., 2015; Heck et al. 2017]→ But, DPGMM is too sensitive to acoustic variations [Wu et al., 2018]→ It is difficult to synthesize speech from DPGMM-based unit

Achieving the best trade-off between unit discovery andspeech synthesize is necessary

Copyright @ AHC Lab NAIST - 20194

Proposed Method• Machine Speech Chain [Tjandra et al., 2017]:→ Enables ASR and TTS to assist each otherwhen they receive unpaired data
→ Optimize both models withreconstruction loss

• Inspired by a similar idea,
we propose:→ Frame-based vector quantizedvariational autoencoder (VQ-VAE)→ Multi-scale codebook-to-spectrogram
inverter (Code2Spec)

MachineSpeech Chain

Copyright @ AHC Lab NAIST - 2019

Current
Proposed Approach5Vector Quantized Variational Autoencoder (VQVAE)• Three Components of VQ-VAE [van den Oord et al. 2017]• Encoder 𝑞𝜃 𝑧 𝑥
Read speech features 𝑥 ∈ ℝ𝐷 and output latent variable 𝑧 ∈ 1. . 𝐾• Codebook 𝐸 = 𝑒1 , . . , 𝑒𝐾 ∈ ℝ𝐾×𝐷𝑒

Discretization is done by choosing the closest codebook• Decoder 𝑝𝜙 𝑥 𝑧, 𝑥 reconstruct the speech features
conditioned by codebook 𝑧 and speaker 𝑠
Copyright @ AHC Lab NAIST - 20196

Vector Quantized Variational Autoencoder (VQVAE)• Training Objective:

Reconstruction loss

Embedding loss

Consistency loss

1. Reconstruction loss between speech and generated speech To optimize encoder and decoder parameter2. Embedding Loss or the update loss of the codebook dictionary
 To optimize the move of embedding to the encoder output
3. Consistency Loss
[Source: https://www.slideshare.net/fukuabca/vqvae] As the volume of the embedding space is dimensionless, this loss
is to forces encoder to generate a representation near the codebook
Copyright @ AHC Lab NAIST - 20197

Codebook-to-Spectrogram (Code2Spec)• Code2Spec Inverter Module→ Generates magnitude linear spectrogram given the codebook using multiscale 1D convolution→ Training Objective:

1. MSE as reconstruction loss
2. GAN as auxiliary loss
WGAN [Arjovsky et al., 2017]
LSGAN [Mao et al., 2017]→ Waveform Generation:
Reconstruct the missing phase with Griffin-Lim algorithm & apply STFT
Copyright @ AHC Lab NAIST - 20198

Experimental Set-up• Dataset: Default ZeroSpeech 2019 Data on English & Surprise Languages• Feature Extraction:→ Mel-spectrogram (80 dimensions, 25-ms window size, 10-ms time-steps)→ MFCC (13 dims + Δ + Δ2 )• Feature representations:→ Directly using features (no model involves)→ K-Means→ GMM with diagonal covariances→ VQ-VAE• Stride size to reduce the time length:→ Stride size: 1, 2, 4, 8
Copyright @ AHC Lab NAIST - 20199

Results: Baseline & Experiment on Direct Features• Baseline & Topline from ZeroSpeech• Direct feature representation (ABX with DTW cosine distance)

MFCC produced better performances on the ABX metric than the Melspectrogram.
But, the bit rate still remains too high.
Copyright @ AHC Lab NAIST - 201910

Results: K-Means, GMM & Proposed VQ-VAE• K-Means continuous representation• GMM posterior representation• Proposed Approach: VQ-VAE codebook representation

VQ-VAE model has the best tradeoff between ABX and bit-ratecompared to K-Means, GMM and direct MFCC features
Copyright @ AHC Lab NAIST - 201911Conclusion• VQ-VAE model has the best tradeoff between ABX and bit-rate compared toK-Means, GMM and direct MFCC features• Things we tried but didn’t work well:→ Wavenet vocoder
The codebook every 20 or 40 ms perhaps too sparse→ GAN speech enhancement
Effective for achieving high-quality VC with clean speech, not for distorted speech• Our best submission:→ VQ-VAE+Code2Spec with 256 codebooks and 2 & 4 time-stride→ Significantly improved performance from baseline (even the topline):
the intelligibility - CER (Rank 1st), the naturalness - MOS (Rank 3rd),and the discrimination - ABX scores (Rank 4th)
Copyright @ AHC Lab NAIST - 201912

Sequel: S2ST without T

Andros Tjandra, Sakriani Sakti, Satoshi Nakamura,"SPEECH-TO-SPEECH TRANSLATION BETWEENUNTRANSCRIBED UNKNOWN LANGUAGES,"ASRU, pp. to appear, 2019

VQ-VAE & Code2Spec
Copyright @ AHC Lab NAIST - 201913

Thank you

Copyright @ AHC Lab NAIST - 201914