Multilingual Speech Translation

Shoichi Senda

Technical Expert
Standardization Promotion Office, Innovation Promotion Department
National Institute of Information and Communication Technology

1. Introduction
Speech communication is the most basic means ofcommunication used by humanity, and telephone services
implementing speech communication have been steadily enhanced
to the point where it is available “anywhere, anytime, and withanyone.” This progress has overcome almost all technical issues
related to “anywhere” and “anytime”, but implementing “withanyone” involves language differences, which are a major obstacle,and we have not yet reached the point where we can communicate
freely with anyone using speech. Actually implementing an
automatic translation telephone has been a common human
dream since the telephone was invented. Anyone would wantsuch a technology and there have been various attempts and muchresearch in this field, but only now are we beginning to see that it
could become practical soon.
With the opening of the Tokyo Olympics and Paralympics,
ever increasing numbers of foreign visitors are expected and there
is hope that, improving multilingual speech translation technology
to a sufficiently practical level, promoting it, and implementing itin society, will facilitate communication with visitors and enable
them experience Japanese hospitality directly.

R&D started in many countries, and an awareness of theneed for standard interfaces and data formats grew, to ensureintercompatibility of their work. In particular, the Asian Speech
Translation Advanced Research Consortium (A-STAR) wasorganized in 2006, centered on national research agencies fromsix Asian countries having many official languages within theirregions. The Asia-Pacific Telecommunity (APT) Standardization
Program (ASTAP) also began standardization activities.
Before long, it became clear that this standardization activity
at ASTAP should be done globally, and not be limited to theAsia-Pacific region, so activity was moved to ITU-T SG16,and standardization on a global scale began. As a result, in2010, recommendations ITU-T F.745, specifying functionalrequirements for network based Speech-to-Speech Translation(S2ST), and H.625, specifying architectural requirements,
were created. A-STAR, which had been restricted to the AsiaPacific region, was also expanded and reorganized to continue its
activities as the Universal Speech Translation Advanced Research
consortium (U-STAR). Recently, with advances in Big Data
analysis and AI, this once-limited field has begun to produce
practical products and services.

2. History and standardization in the study ofspeech translation

3. Implementation of multilingual speechtranslation

Here we look at the history over many years, of R&D
attempting to realize the dream of multilingual speech translation,
and also the part played by standardization.
At the beginning, different research agencies studied speechtranslation separately, but individual study had limitations, soin 1991, scientists from around the world, including the USA,Germany, France, Japan, Korea, and Italy, began forming
voluntary organizations such as the Consortium for Speech
Translation Advanced Research (C-STAR), and they started
collaborating by sharing independent research results for eachlanguage and integrating them together. Thus, speech translation

Generally, to realize two-way speech communication between
people speaking different languages, the audio signal expressedin the speaker’s language must be translated to an audio signal
in the language of the listener. For example, Japanese to English
translation is implemented as in Figure 1. When the speakersays “watashi wa gaxtukooni…” in Japanese, the audio signal is
automatically recognized as “わたしは学校に行く” in Japanesetext, this text is machine translated to “I go to school” in Englishthat the listener can understand, and the text is then converted toan English audio signal.

New Breeze Spring 20189

■ Figure 1: Outline of speech translation

Here, we have described speech translation as implemented
by performing Automatic Speech Recognition (ASR), MachineTranslation (MT), and Text-to-Speech (TTS) functions insequence, using the case of Japanese to English translation as anexample. This combination of functions can be used to translate
between any languages and is not limited to Japanese-to-English.
Here, ASR converts the input speech signal into text data, MT
converts the input text data into text data in a different language(but having equivalent meaning), and TTS converts the input text
data into a speech signal. The details of these conversions depend■ Figure 2: Architecture of multilingual speech translation

10New Breeze Spring 2018

on the languages and fields being translated. As such, ASR, MTand TTS generally share the basic function of using conversion
dictionaries called corpuses for each translated language andfield. When building a multi-lingual speech translation systemin this way, the engines that realize each of these functions can be
designed and developed independently, and multi-lingual speechtranslation can be achieved more easily by combining engines
appropriately, according to the languages and application area
(Figure 2).

■ Figure 3: Architecture and its implementation protocol

To make it possible to implement multilingual speechtranslation with this sort of structure, the functionality of eachfunctional element be standardized, as well as how each functional
element is deployed, what and how they exchange information,including protocols and formats. Since the main functional
elements of speech translation, namely ASR, MC, and TTS,
require significant amounts of processing, the ITU-T SG16 has
adopted a client-server model in which servers are placed on thenetwork and users use server functionality through clients that areinstalled on their various terminals. As such, protocols between
client and server, and between servers, are being standardized(Figure 3).

4. State of standardization in speechtranslation
Later in the speech translation field, in 2014, ITU-T
SG2 Q4 and ISO/IEC JTC1/SC35 WG5 proposed thatuser interfaces be studied and standardized to improve service

availability, independent of system functional and organizationalrequirements. These standardization proposals regulate userinterfaces particularly for when speaker and listener are engaged
in two-way communication with each other at a single location.
S2ST has already been standardized and has moved to a
maintenance and management phase, so the relation of these
proposals to the completed recommendations, F.745 and H.625,
as well as any particular requirements due to the face-to-faceenvironment needed to be clarified. In this area, there werealso proposals regarding the user interface for speech translationin face-to-face environments from ASTAP EG-SNLP in
Q4/2 (which was later integrated into EG-MA), which wasstudying the E.FAST draft recommendation. It was agreed
that more investigation was needed to understand various use
cases and associated issues, and that work should move fromSG2 to SG16, become Q24/16, and continue in collaboration
with Q21. On the other hand, speech translation user interfaces
being studied in ISO/IEC JTC1/SC35 WG5, IS20382-1 and -2

New Breeze Spring 201811

were standardized without necessarily coordinating sufficiently
with ITU-T speech translation standardization activities, andITU-T submitted comments identifying issues during DIS voting.
The reply given to ITU-T SG16 by JTC1/SC35 was that thesestandards apply only to the user interface setup. In response,
ITU-T SG16 reviewed and reorganized the interrelation among
S2ST related standards (F.745, H.625, E.FAST, IS 20382-1 and-2) as shown in Figure 4, and intends to reaffirm this at the JTCI/
SC35 meeting in February, 2018.
Note that while F.745 and H.625 were standardized with thegoal of realizing two-way speech translation, input and output
signals are not necessarily limited to audio signals. By skipping
the ASR and TTS functions of F.745 and H.625, and using only
the MT function directly for input and output text data, these■ Figure 4: Interrelation among S2ST related standards

12New Breeze Spring 2018

standards could be used for multilingual text translation systems.
Or, by skipping the ASR function for communication in one
direction, and the TTS function in the other direction, tools
such as Koetra (http://www.koetra.jp/) can be created to support
communication between deaf and hearing people. Still further,by applying recent advances in Big Data, AI and Deep Learning
to video recognition, the ASR function could be replaced orextended, from simple speech recognition to recognition of signlanguage video, so that multi-lingual speech translation could
evolve into general translation, including sign language. Even if
this will be difficult to realize by the time the Paralympics begin,continuing to work on implementing this sort of tool in support ofdisabled persons has the benefit of contributing to the hopes anddreams of many who live with such disabilities.