言語処理学会 第26回年次大会 発表論文集 (2020年3月)

Speech-to-Speech Translation without Text
Andros Tjandra1
1

Sakriani Sakti1,2Satoshi Nakamura1,2

Nara Institute of Science and Technology, Japan2
RIKEN AIP, Japan{andros.tjandra.ai6, ssakti, s-nakamura}@.is.naist.jp

1

Introduction

The common system requires eﬀort to construct
several components, including automatic speech
recognition (ASR), machine translation (MT), andtext-to-speech (TTS) synthesis, all of which aretrained and tuned independently. Given speech input, ASR processes and transforms speech into text
in the source language, MT transforms the source
language text to corresponding text in the target
language, and finally TTS generates speech from thetext in the target language. Significant progress has
been made and various commercial speech translation systems are already available for several language pairs. However, more than 6000 languages,
spoken by 350 million people, have not been covered yet. At the moment, it is diﬃcult to scale-up
the existing approach to unknown languages without
written forms or transcription data available.
In this paper, we take a step beyond the current
framework and propose a method for training speech
to speech translation tasks without any transcription
or linguistic supervision. Instead of only discovering
subword units and synthesizing them within a certain
language, our approach discovers subword units thatare directly translated to another language. Our proposed method consists of two steps: (1) we train andgenerate discrete representation with unsupervised
term discovery, which is also based on vector quantized variational autoencoder (VQ-VAE) [4]; (2) wetrain a sequence-to-sequence model to directly map
the source language speech to the target language
discrete representation. Our proposed method candirectly generate target speech without any auxiliary
or pre-training steps with source or target transcrip-tion. To the best of our knowledge, this is the firstwork that performed pure speech-to-speech translation between untranscribed unknown languages.

2Training and Inference

In this section, we explain our proposed method
in detail and step-by-step. To train our proposedmodel, we setup three diﬀerent modules: VQ-VAE, aspeech-to-codebook seq2seq to generate target codebook given the source language speech, and a codebook inverter to generate target speech given the target language codebook [3]. Fig. 1 shows which modules are trained in each step. Initially, we definedMM{XMsrc , Xtgt } as paired parallel speech, Xsrc is theMFCC (mel frequency cepstral coeﬃcients) featuresfrom the source language, and XM
tgt is the MFCC features from the target language. Ytgt is the codebooksequences generated by VQ-VAE encoder Encθ (x)Rgiven XMtgt as the input. X̂tgt is the predicted linear spectrogram of the target language. LV Q is theVQ-VAE loss, LIN V is the raw spectogram loss, andLN LL is the negative log-likelihood of codebook intarget language given the source speech XM
src .
1. First, we trained the VQ-VAE model on target language MFCC XM
tgt . We also trained thecodebook inverter to predict corresponding linear spectrogram XRtgt .
2. Second, we trained the seq2seq model from thesource language speech to the target language
codebook. Given a paired parallel MFCC fromM
source and target languages {XMsrc , Xtgt }, we extracted codebook sequence Ytgt = EncVθ Q (XMtgt )from the VQ-VAE encoder. Later, we trained― 1022 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Figure 1: a) Train VQ-VAE to represent continuous MFCC vectors with codebook sequence and train codebook
inverter to generate a linear magnitude spectrogram based on generated codebook sequence; b) Train a seq2seq
model from source language MFCC to target language codebook. c) In inference stage, seq2seq model takes
source language MFCC and predicts codebook sequences, and then codebook inverter generates target language
speech representation.
the seq2seq translation model to predict Ŷtgt =Seq2Seq(XMsrc ) and minimize loss LN LL between
MM.
Xsrc and Xsrc
3. In the inference step, given source language
speech XM
src , we decoded a target language codebook index sequence Ŷtgt = Seq2Seqψ (XMsrc )and synthesized it into target language speech
X̂Rtgt = Inverter(Ŷtgt ).

2.1bound for the related translation model. We utilized
two metrics to evaluate the translation performancefrom the transcribed text: BLEU scores and METEOR with a Multeval toolkit. Our pre-trained ASR
model resulted in a 2.84% WER, a 94.9 BLEU, anda 69.1 METEOR on English speech utterances fromthe BTEC test set, and we set those scores as thegroundtruth topline scores.

3Evaluation

In this section, we present our experimental result

For an objective evaluation of the target speechutterances, currently there is no standard method

Results and Discussionand followed by the discussion.

can be used to measure translation quality directlyon the speech utterances. Therefore, we utilized a3.1

pre-trained ASR on the English BTEC dataset andthe generated transcription for our evaluation. For
the ASR architecture, the encoder module has threestacked Bi-LSTMs with 512 hidden units, and the decoder has one LSTM with 512 hidden units. For theBaseline

For the baseline translation task, we trained asequence-to-sequence model from speech-to-speech
directly. However, this approach did not convergeat all and produced no audible speech. [1] also observed a similar result with a similar scenario.

attention module, we utilized MLP attention withmultiscale location history [2]. For the output unit,
we used a word-level token from the English tran-

3.2

Topline with Cascade ASR-TTS

scription. Because there is a performance gap be-

In this paper, we set the topline performance by

tween the ASR and the ground truth cause by im-

using the cascade of ASR and TTS system. First,

perfect transcription, we assume the metric (calculated based on the ASR transcription) is the lower-― 1023 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Table 1: Our experiment results based on BTEC

4French-English speech-to-speech translation.
Model (FR-EN)
BLEU
Baseline
Tacotron with MFCC input
Proposed Speech2CodeTime
Codebook
Reduction
64
4
16.1
64
824.4
64
1225.0Topline47.4(Cascade ASR ->TTS)Conclusion
In this paper, we proposed a novel approach forMETEOR-

training a speech-to-speech translation between twolanguages without any transcription.
French-English and Japanese-English.

16.9
22.9
23.2541.2

Acknowledgements
Part of this work was supported by JSPS

KAKENHI
Table 2: Our experiment results based on BTECJapanese-English speech-to-speech translation.
Model (JA-EN)
BLEU
Baseline
Tacotron with MFCC input
Proposed Speech2CodeTime
Codebook
Reduction
128
4
11.9
128
8
15.3
128
12
14.9
Topline37.4(Cascade ASR ->TTS)Our model

can perform a direct speech-to-speech translation on

Grant

Numbers

JP17H06101

and

JP17K00237.

METEOR

References-[1] Ye Jia, Ron J Weiss, Fadi Biadsy, Wolfgang
Macherey, Melvin Johnson, Zhifeng Chen, and13.5
15.3
14.5

Yonghui Wu. Direct speech-to-speech translation with a sequence-to-sequence model. arXiv

32.8

preprint arXiv:1904.06037, 2019.
[2] Andros Tjandra, Sakriani Sakti, and Satoshi

we train the ASR system by using the source lan-Nakamura. Multi-scale alignment and contextualguage MFCC as the input and target language char-history for attention mechanism in sequence-to-acter transcription. Second, we train a TTS based

sequence model. In 2018 IEEE Spoken Languageon Tacotron [5] to generate a speech from the target

Technology Workshop (SLT), pp. 648–655. IEEE,

language characters to the target language speech

2018.

representation.
[3] Andros Tjandra, Berrak Sisman, Mingyang
Zhang, Sakriani Sakti, Haizhou Li, and Satoshi

3.3

Speech to Codebook

Nakamura. VQVAE unsupervised unit discovery

Table 1 shows the result for French-to-English

and multi-scale code2spec inverter for zerospeech

Speech translation. Our best performance was pro-challenge 2019. CoRR, Vol. abs/1905.11449, ,duced by codebook of 64 and a time-reduction factorof 12 with a score of 25.0 BLEU and 23.2 METEOR.

2019.
[4] Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances

Table 2 shows the result for Japanese-to-English

in Neural Information Processing Systems, pp.

Speech translation. Our best performance was pro-6306–6315, 2017.

duced by a codebook of 128 and a time-reduction
factor 8 with a score of 15.3 BLEU and 15.3 ME-[5] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep

TEOR.

Jaitly, Zongheng Yang, Ying Xiao, Zhifeng
Chen, Samy Bengio, et al. Tacotron: Towards
end-to-end speech synthesis.

arXiv preprint

arXiv:1703.10135, 2017.

― 1024 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.