Speech-to-Speech Translation
without Text
Andros Tjandra1, Sakriani Sakti1,2, Satoshi Nakamura1,2
1) Nara Institute of Science & Technology, Nara, Japan
2) RIKEN AIP, Japan
30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan1

Outline• Introduction• Technical Background• Training and Inference• Experimental Setup & Results• Conclusion

30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan2

1. Introduction• Speech-to-speech translation technology
overcomes language barrier from humancommunication• Challenges:• Training requires speech-text pairs (cascade ASR-NMTTTS).
• Jia et al. 2019 proposed direct speech-to-speech,
however can’t converge without pre-training with text.
• Not all languages has written form.

30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan3

1. Our proposal …
• Direct speech-to-speech translation for unknown
languages (no prior knowledge about the language
needed).
• No transcription needed for both source and targetlanguages.

30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan

4

2. Technical Background• We utilize 3 different models:• Unsupervised unit discovery with discrete autoencoder(VQ-VAE)• Sequence-to-sequence to translate audio to codebook• Codebook-to-spectrogram inverter to re-synthesize thetranslated audio

30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan5

Unsupervised unit discovery withdiscrete autoencoder (VQ-VAE)
Speech signal can be disentangled
into {contexts, speaking style}𝐸𝑛𝑐𝜃 𝑥 = 𝑞𝜃 𝑦 𝑥
𝐷𝑒𝑐𝜙 𝑦, 𝑠 = 𝑝𝜙 𝑥 𝑦, 𝑠
Codebook 𝐸 = 𝑒1 , . . , 𝑒𝐾
Speaker vec 𝑉 = [𝑣1 , . . , 𝑣𝐿 ]Continuous speech (harder target)

Discrete symbol (easier target)[de Oord et al., 2017]30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan

6

Training VQ-VAEℒ𝑉𝑄 = − log 𝑝𝜃 𝑥 𝑦, 𝑠 + ||𝑠𝑔 𝑧 − 𝑒𝑐 ||22 + 𝛾||𝑧 − 𝑠𝑔 𝑒𝑐 ||22
Reconstruction loss

30-Mar-20

Embedding loss

Andros Tjandra @ AHCLab, NAIST, Japan

Commitement loss

7

Sequence-to-Sequence fromSpeech to Codebook

Input 𝑋 = [𝑥1 , … , 𝑥𝑆 ] as the speech from source language
Output 𝑌 = [𝑦1 , … , 𝑦𝑇 ] as the codebook from target language
Encoder consisted of Bi-LSTMs and decoder consisted of LSTMs
30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan

8

Codebook Inverter
Input is codebook embedding E𝑌 =𝐸[𝑦1 , … , 𝐸 𝑦𝑇 ]Output is linear magnitude
spectrogram- We use Griffin-Lim to recover phase
spectrogram and inverse Fouriertransform to recover the waveform.

30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan9

Training and inference30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan

10

3. Experimental Setup• Dataset: Basic Travel Expression (BTEC) corpus• Language pairs:• French to English (similar grammatical structure)• Japanese to English (distant grammatical structure)• Size:• Training 162.318 sentences pair
• Test 510 sentences pair

• Speech features:• Input: MFCC (39 dimensions)• Output: Linear spectrogram (1025 dimensions)30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan11

Evaluation• Because the large number of test samples, it is hard
to do subjective evaluation.
• How we do evaluation:
1. Train English ASR
2. Translate source language speech to target
language speech
3. Use trained ASR (step 1) to recognize translatedspeech
4. Calculate BLEU and Meteor between groundtruth
and ASR transcription
30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan

12Result• Model:• Baseline (direct spectrogram-to-spectrogram)• Proposed SP2C (C=codebook size, T=time reduce)• Topline (speech src-> text tgt*-> speech tgt, *requires texttranscription during training)

Model

30-Mar-20

BLEU4METEOR

Baseline (FR-EN & JA-EN)
SP2C FR-EN C=64, T=12Not converged25
23.2

Topline FR-EN (Cascade) *SP2C JA-EN C=128, T=8

47.4
15.3

41.2
15.3

Topline JA-EN (Cascade) *

37.4

32.8

Andros Tjandra @ AHCLab, NAIST, Japan13Additional result• Translation samples : https://sp2code-translationv1.netlify.com/
Model

Transcription

Groundtruth

how long are you going to stay

SP2C FR-EN

how long are you going to stay

SP2C JA-EN

how long will it take

Groundtruth

please tell him to call me as soon as he comes in

SP2C FR-EN

please tell him to call me back

SP2C JA-EN

please tell him that i called

incomplete
transcription

Based on the example, 1) gives quite close result
However, 2) SP2C result left out the latter part30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan

14

Conclusion• We proposed a novel approach for training speechto-speech translation w/o transcription• Experiments was performed on French-English &Japanese-English

30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan15

 Thank you for listening 

30-Mar-20

Andros Tjandra @ AHCLab, NAIST, Japan

16