Speech-to-Speech Translation without Text
Andros Tjandra1
1

Sakriani Sakti1,2Satoshi Nakamura1,2

Nara Institute of Science and Technology, Japan
2
RIKEN AIP, Japan{andros.tjandra.ai6, ssakti, s-nakamura}@.is.naist.jp

1

Introduction

tion. To the best of our knowledge, this is the firstwork that performed pure speech-to-speech transla-

The common system requires eﬀort to constructtion between untranscribed unknown languages.

several components, including automatic speech
recognition (ASR), machine translation (MT), andtext-to-speech (TTS) synthesis, all of which are2

Training and Inference

trained and tuned independently. Given speech input, ASR processes and transforms speech into text

In this section, we explain our proposed methodin the source language, MT transforms the source

in detail and step-by-step. To train our proposed

language text to corresponding text in the target

model, we setup three diﬀerent modules: VQ-VAE, alanguage, and finally TTS generates speech from thespeech-to-codebook seq2seq to generate target code-text in the target language. Significant progress has
been made and various commercial speech transla-

book given the source language speech, and a code-tion systems are already available for several lan-

get language codebook [3]. Fig. 1 shows which mod-guage pairs. However, more than 6000 languages,

ules are trained in each step. Initially, we defined

spoken by 350 million people, have not been cov-

book inverter to generate target speech given the tar-M
M{XMsrc , Xtgt } as paired parallel speech, Xsrc is theered yet. At the moment, it is diﬃcult to scale-up

MFCC (mel frequency cepstral coeﬃcients) features

the existing approach to unknown languages withoutfrom the source language, and XM
tgt is the MFCC fea-written forms or transcription data available.

tures from the target language. Ytgt is the codebook

In this paper, we take a step beyond the current

sequences generated by VQ-VAE encoder Encθ (x)framework and propose a method for training speechRgiven XM
tgt as the input. X̂tgt is the predicted lin-to speech translation tasks without any transcription
or linguistic supervision. Instead of only discoveringear spectrogram of the target language. LV Q is theVQ-VAE loss, LIN V is the raw spectogram loss, andsubword units and synthesizing them within a certain

LN LL is the negative log-likelihood of codebook inlanguage, our approach discovers subword units thattarget language given the source speech XM
src .

are directly translated to another language. Our pro-

1. First, we trained the VQ-VAE model on tar-posed method consists of two steps: (1) we train andget language MFCC XM
tgt . We also trained thegenerate discrete representation with unsupervised
term discovery, which is also based on vector quantized variational autoencoder (VQ-VAE) [4]; (2) wecodebook inverter to predict corresponding linear spectrogram XRtgt .

the source language speech to the target language

2. Second, we trained the seq2seq model from thesource language speech to the target language

discrete representation. Our proposed method cancodebook. Given a paired parallel MFCC fromtrain a sequence-to-sequence model to directly map

directly generate target speech without any auxiliary
or pre-training steps with source or target transcrip-M
source and target languages {XMsrc , Xtgt }, we ex-

tracted codebook sequence Ytgt = EncVθ Q (XMtgt )from the VQ-VAE encoder. Later, we trainedFigure 1: a) Train VQ-VAE to represent continuous MFCC vectors with codebook sequence and train codebook
inverter to generate a linear magnitude spectrogram based on generated codebook sequence; b) Train a seq2seq
model from source language MFCC to target language codebook. c) In inference stage, seq2seq model takes
source language MFCC and predicts codebook sequences, and then codebook inverter generates target language
speech representation.
the seq2seq translation model to predict Ŷtgt =Seq2Seq(XMsrc ) and minimize loss LN LL between
MM.
Xsrc and Xsrc
3. In the inference step, given source language
speech XM
src , we decoded a target language codebook index sequence Ŷtgt = Seq2Seqψ (XMsrc )and synthesized it into target language speech
X̂R
tgt = Inverter(Ŷtgt ).

2.1Evaluation

For an objective evaluation of the target speechutterances, currently there is no standard method

bound for the related translation model. We utilized
two metrics to evaluate the translation performancefrom the transcribed text: BLEU scores and METEOR with a Multeval toolkit. Our pre-trained ASR
model resulted in a 2.84% WER, a 94.9 BLEU, anda 69.1 METEOR on English speech utterances fromthe BTEC test set, and we set those scores as thegroundtruth topline scores.

3Results and Discussion
In this section, we present our experimental resultand followed by the discussion.

can be used to measure translation quality directlyon the speech utterances. Therefore, we utilized apre-trained ASR on the English BTEC dataset andthe generated transcription for our evaluation. For
the ASR architecture, the encoder module has threestacked Bi-LSTMs with 512 hidden units, and the decoder has one LSTM with 512 hidden units. For the3.1

Baseline

For the baseline translation task, we trained asequence-to-sequence model from speech-to-speech
directly. However, this approach did not convergeat all and produced no audible speech. [1] also observed a similar result with a similar scenario.

attention module, we utilized MLP attention withmultiscale location history [2]. For the output unit,
we used a word-level token from the English tran-

3.2

Topline with Cascade ASR-TTS

scription. Because there is a performance gap be-

In this paper, we set the topline performance by

tween the ASR and the ground truth cause by im-

using the cascade of ASR and TTS system. First,

perfect transcription, we assume the metric (calculated based on the ASR transcription) is the lower-

Table 1: Our experiment results based on BTEC4French-English speech-to-speech translation.
Model (FR-EN)
BLEU
Baseline
Tacotron with MFCC input
Proposed Speech2CodeTime
CodebookReduction
64
416.1
64
824.4
64
1225.0Topline47.4(Cascade ASR ->TTS)Conclusion
In this paper, we proposed a novel approach for

METEOR-training a speech-to-speech translation between twolanguages without any transcription.
French-English and Japanese-English.

16.9
22.9
23.2
41.2

5

Acknowledgements
Part of this work was supported by JSPS

KAKENHI
Table 2: Our experiment results based on BTECJapanese-English speech-to-speech translation.
Model (JA-EN)
BLEU
Baseline
Tacotron with MFCC input
Proposed Speech2CodeTime
CodebookReduction
1284
11.9
128815.3
1281214.9Topline37.4(Cascade ASR ->TTS)Our model

can perform a direct speech-to-speech translation onGrant

Numbers

JP17H06101

and

JP17K00237.

METEOR-References[1] Ye Jia, Ron J Weiss, Fadi Biadsy, Wolfgang

13.5
15.3
14.5
32.8

Macherey, Melvin Johnson, Zhifeng Chen, andYonghui Wu. Direct speech-to-speech translation with a sequence-to-sequence model. arXiv
preprint arXiv:1904.06037, 2019.
[2] Andros Tjandra, Sakriani Sakti, and Satoshi

we train the ASR system by using the source language MFCC as the input and target language character transcription. Second, we train a TTS basedon Tacotron [5] to generate a speech from the target
language characters to the target language speech

Nakamura. Multi-scale alignment and contextual
history for attention mechanism in sequence-tosequence model. In 2018 IEEE Spoken Language
Technology Workshop (SLT), pp. 648–655. IEEE,2018.

representation.
[3] Andros Tjandra, Berrak Sisman, MingyangZhang, Sakriani Sakti, Haizhou Li, and Satoshi

3.3

Speech to Codebook

Nakamura. VQVAE unsupervised unit discovery

Table 1 shows the result for French-to-English

and multi-scale code2spec inverter for zerospeech

Speech translation. Our best performance was pro-challenge 2019. CoRR, Vol. abs/1905.11449, ,duced by codebook of 64 and a time-reduction factorof 12 with a score of 25.0 BLEU and 23.2 METEOR.

2019.
[4] Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances

Table 2 shows the result for Japanese-to-EnglishSpeech translation. Our best performance was pro-in Neural Information Processing Systems, pp.
6306–6315, 2017.

duced by a codebook of 128 and a time-reduction
factor 8 with a score of 15.3 BLEU and 15.3 METEOR.

[5] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep
Jaitly, Zongheng Yang, Ying Xiao, Zhifeng
Chen, Samy Bengio, et al. Tacotron: Towards
end-to-end speech synthesis.
arXiv:1703.10135, 2017.

arXiv preprint