ISCA Archive

SLTU-2014, St. Petersburg, Russia, 14-16 May 2014

http://www.isca-speech.org/archive

Keynote Lecture - EURASIP Seminar
TOWARDS REAL-TIME MULTILINGUAL MULTIMODALSPEECH-TO-SPEECH TRANSLATION

Satoshi Nakamura
Nara Institute of Science and Technology (NAIST), Japan
http://ahclab.naist.jp/Prof.Nakamura/index_e.html

Abstract: Speech-to-speech translation technology enables natural oral communication between
different language speaking people. Many research projects have addressed speech-to-speechtranslation (S2ST) technology, such as ATR, VERBMOBIL, C-STAR, NESPOLE!, BABYLON,
GALE, and EU-bridge. The speech-to-speech translation system is normally composed ofautomatic speech recognition (ASR), machine translation (MT), and speech synthesis (TTS). All ofthe modules are corpus-based and statistical model-based systems. In this talk new challenges
toward a real-time multimodal speech-to-speech translation will be introduced.
Biography: Satoshi Nakamura is Professor of Graduate School of Information Science, Nara
Institute of Science and Technology, Japan, Honorarprofessor of Karlsruhe Institute of Technology,
Germany, and ATR Fellow. He was Director of ATR Spoken Language Communication ResearchLaboratories in 2000-2008 and Vice president of ATR in 2007-2008. He was Director General ofKeihanna Research Laboratories and the Executive Director of Knowledge CreatingCommunication Research Center, National Institute of Information and Communications
Technology, Japan in 2009-2010. He is currently Director of Augmented Human Communicationlaboratory and a full professor of Graduate School of Information Science at Nara Institute ofScience and Technology. He is interested in modeling and systems of speech-to-speech translationand speech recognition. He is one of the leaders of speech-to-speech translation research and has
been serving for various speech-to-speech translation research projects in the world including CSTAR, IWSLT and A-STAR. He was a project leader of the world first network-based commercial
speech-to-speech translation service for 3-G mobile phones in 2007 and VoiceTra Project foriPhone in 2010. He received Yamashita Research Award, Kiyasu Award from the Information
Processing Society of Japan, Telecom System Award, AAMT Nagao Award, Docomo Mobile
Science Award in 2007, ASJ Award for Distinguished Achievements in Acoustics. He received theCommendation for Science and Technology by the Minister of Education, Culture, Sports, Scienceand Technology, and the Commendation for Science and Technology in Information Technology bythe Minister of Internal Affair and Communications. He was also awarded Antonio Zampolli Prize
by ELRA Association. He organized the International Workshop of Spoken Language Translation(IWSLT 2006) and Oriental Cocosda 2008 as a general chair. He also served as the program chairof INTERSPEECH 2010. He is currently Elected Board Member of International SpeechCommunication Association (ISCA), Elected Committee Member of IEEE SPS Spoken Language
Technology Technical committee, and IEEE Signal Processing Magazine Editorial Board Member
since April 2012.

ISCA Archive

SLTU-2014, St. Petersburg, Russia, 14-16 May 2014

http://www.isca-speech.org/archive

TOWARDS REAL-TIME MULTILINGUAL MULTIMODALSPEECH-TO-SPEECH TRANSLATION
Satoshi Nakamura
Augmented Human Communication Laboratory
Graduate School of Information Science
Nara Institute of Science and Technology, Japans-nakamura@is.naist.jp
have developed a system which finds a chunk with a higher
Right Probability than a threshold and translates the chunk.
S2ST between Western languages and a non-Western
language, such as English-from/to-Japanese, or Englishfrom/to-Chinese, requires technologies to overcome thedrastic differences in linguistic structures and expressions.
Especially their word order and their coverage of words arecompletely different, among other factors. Figure 1 showsthe experimental data used from the Basic Travel Expression
Corpus (BTEC) [11] for ja-en and en-ja, and NEWS [12] forfr-en. As the BTEC sentences are relatively short comparedto NEWS, we also experiment with longer sentences that
contain at least 11 words from BTEC. For evaluation
measures, we use BLEU to measure translation accuracy
with 12 references for ja-en, and 1 reference for fr-en. We
also perform a manual evaluation using a 0-5 scale based on
acceptability [13]. We calculate translation delay D as
D = A + T. A is the ASR time per sentence, and we calculate
this using the time of each wave file in the test set. Tindicates the average MT decoding time per sentence.

ABSTRACT
Speech-to-speech translation technology enables natural oral
communication between different language speaking people.
Many research projects have addressed speech-to-speechtranslation (S2ST) technology, such as ATR [1],VERBMOBIL [2],C-STAR [3],NESPOLE! [4],BABYLON [5], GALE [6], and EU-bridge [7]. The speechto-speech translation system is normally composed ofautomatic speech recognition (ASR), machine translation(MT), and speech synthesis (TTS). All of the modules arecorpus-based and statistical model-based systems. In thistalk, new challenges toward a real-time multimodal speechto-speech translation will be introduced.
Index Terms— Speech-to-speech translation, S2ST,
multimodal processing, multilingual systems
1. SIMULTANEOUS SPEECH-TO-SPEECH
TRANSLATION
It is still not able to output translation results in simultaneousway. The reason for this is in the interaction between thethree components of conventional speech translation
systems: ASR, MT and TTS. Normally, the MT module isstarted after the ASR module finishes recognition and theTTS module is started after MT module finishes translation.
This has caused a delay between the start of the speaker’s
utterance to the end of synthesis. The longer sentences
require more time for MT decoding. In contrast, humansimultaneous interpreters generally break sentences into
smaller chunks, resulting in a lower delay (or “ear-to-voicespan”) [8].
We proposed a method for starting the translation
process before the sentence finishes, allowing the MT
module to start translation simultaneously [9]. The method
uses so-called Right Probability in the phrase table used inphrase-based MT [10]. The Right Probability represents how
much re-ordering of the phrases in MT could occur. WeFigure 1. Translation Performance vs. Delay

13SLTU-2014, St. Petersburg, Russia, 14-16 May 2014

The block diagram of the system is shown in Figure 2.
The original speakers face is first scanned and mapped to athree dimensional face model. Then the mouth part of thespeaking face images are replaced with that of the target
language created according to the translated phoneme
sequence.

Comparing ja-en and en-ja translation we confirmed thefact that both achieve similar speed accuracy curves. Inaddition, BLEU is higher overall for en-ja because Japanese
sentences are longer than English sentences, so the numberof matches with the reference is greater than when the target
language is English. Finally, we compare ja-en and fr-entranslation to investigate the effectiveness for a language
pair with small difference of word order. As can be seenfrom the graph for fr-en, by reducing the RP threshold from1.0 to 0.8 we are able to achieve a decrease in delay from
12.1s to 5.4s with an almost no drop in BLEU (19.63 to19.53 respectively). Even when we set the threshold lower,
the drop in accuracy is much smaller than ja-en or en-ja
translation, confirming that the proposed method isparticularly effective for languages with similar word order.

3. SPEECH-TO-SPEECH TRANSLATIONPRESERVING PARALINGUISTIC INFORMATION
The prominence is phenomena, which a speaker emphasizes
a part of the sentence to express their intention. The
conventional speech-to-speech translation is not able topreserve this information from the original speech into thetranslated speech in the target language. We proposed aword-based prominence preservation method [16]. The
system is based on a neural network model trained usingbilingual utterances. We recorded a bilingual speech corpus
where an English-Japanese bilingual speaker emphasizes one
word during speech in a string of digits. The lexical content
to be spoken was 500 sentences from the AURORA2 dataset, chosen to be word balanced by greedy search [17]. The
training set is 445 utterances and the test set is 55 utterances.
Experiments have been done by asking evaluators to
subjectively judge the strength of emphasis with thefollowing three degrees: 1: not emphasized, 2: slightly
emphasized, 3: emphasized. The overview of the experiment
regarding the strength of emphasis is shown in Figure 3.

2. SPEECH-TO-SPEECH TRANSLATION WITH
SPEAKING FACE MOVEMENTS
Speech includes not only linguistic information but otherinformation such as speaker individuality, speaking style,intonation, emotion, and face expression. Translation andpreservation of these kinds of information are expected to be
inevitable to realize ideal speech-to-speech translation.
Voice conversion, which is nowadays popular research
topic, was originated in early days in 1987 for a speech-tospeech system [14]. In addition to speech individuality
preservation in the speech-to-speech translation, we havedeveloped a speaking face translation system [15].

Figure 2. Speaking Face Translation

14SLTU-2014, St. Petersburg, Russia, 14-16 May 2014

Speech Translation”, In: Proc. INTERSPEECH-2013, Lyon,France, pp. 3487-3491, 2013.
[10] P. Koehn, F. J. Och, D. Marcu, “Statistical phrase-basedtranslation”, In: Proc. Human Language Technology Conference
NAACL-HLT-2003, Edmonton, Canada, pp. 48-54, 2003.
[11] T. Takezawa, E. Sumita, F. Sugaya, H. Yamamoto,S. Yamamoto, “Toward a broad-coverage bilingual corpus forspeech translation of travel conversations in the real world”, In:Proc. of The Third International Conference on Language
Resources and Evaluation LREC-2002, Las Palmas, Spain,
pp.147-152, 2002.
[12] J. Civera, A. Juan, “Domain adaptation in statistical machinetranslation with mixture modelling,” In: Proc. 2nd Workshop on
Statistical Machine Translation, Prague, Czech Republic, pp. 177–180, 2007.

Figure 3. Subjective Degree of Emphasis
This figure shows that all systems show a significant
improvement in the subjective perception of strength ofemphasis.

[13] I. Goto, B. Lu, K. P. Chow, E. Sumita, B. K. Tsou, “Overviewof the patent machine translation task at the NTCIR-9 workshop,”
In: Proc. 9th NTCIR Workshop Meeting, Tokyo, Japan, pp. 559578, 2011.

4. REFERENCES[14] M. Abe, S. Nakamura, K. Shikano, H. Kuwabara, “VoiceConversion through Vector Quantization”, In: Proc. International
Conference on Acoustics, Speech, and Signal Processing ICASSP1988, New York, USA, pp. 655-658, 1988.

[1] S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai,T. Jitsuhiro, J.-S. Zhang, H. Yamamoto, E. Sumita, S. Yamamoto,“The ATR Multilingual Speech-To-Speech translation system”,
IEEE Transactions on Audio, Speech and Language Processing,Vol. 14, No. 2, pp. 365-376, 2006.

[15] S. Ogata, K. Murai, S. Nakamura, S. Morishima, “Modelbased Lip Synchronization with Automatically Translated
Synthetic Voice toward a Multi-modal Translation System”, In:Proc. International Conference on Multimedia and Expo (ICME2001), Tokyo, Japan, pp. 28-31, 2001.

[2] W. Wahlster (Ed.), Verbmobil: Foundation of speech-to-speechtranslation, Springer Verlag, 2000.
[3] C-STAR: http://www.c-star.org[16] T. Kano, S. Sakti, S. Takamichi, G. Neubig, T. Toda,S. Nakamura, “A Method for Translation of Paralinguistic
Information”, In: Proc. International Workshop on Spoken
Language Translation IWSLT-2012, Hong Kong, China, pp. 15960, 2012.

[4] A. Lavie, F. Metze, R. Cattoni, E. Constantini, “A Multiperspective Evaluation of the NESPOLE! Speech-to-speech
Translation System”, In Proc. Workshop on Speech-to-Speech
Translation: Algorithms and Systems, Philadelphia, PA, USA,pp. 121-128, 2002.

[17] H. G. Hirsh, D. Pearce, “The AURORA experimental
framework for the performance evaluations of speech recognition
systems under noisy conditions”, In: Proc. the ISCA Tutorial andResearch Workshop (ITRW) ASR2000 - Automatic Speech
Recognition: Challenges for the new Millennium, Paris, France,
pp. 181-188, 2000.

[5] BABYLON: http://www.babylon.com[6] H.Soltau, G. Saon, B. Kingsbury, H.-K. J. Kuo, L. Mangu, D.
Povey, A. Emami, “Advances in Arabic Speech Transcription atIBM Under the DARPA GALE Program”, IEEE Transactions on
Audio, Speech and Language Processing, Vol. 17, No. 5, pp. 884894, 2009.
[7] EU-BRIDGE: http://www.eu-bridge.eu/index.php[8] F. Goldman-Eisler, “Segmentation of input in simultaneoustranslation”, Journal of Psycholinguistic Research, Kluwer
Academic Publishers-Plenum Publishers, Vol. 1, No. 2, pp. 127140, 1972.
[9] T. Fujita, G. Neubig, S. Sakti, T. Toda, S. Nakamura, “Simple,Lexicalized Choice of Translation Timing for Simultaneous

15