Available online at www.sciencedirect.com

Computer Speech and Language 27 (2013) 509–527

A-STAR: Toward translating Asian spoken languages夽
Sakriani Sakti a,∗,1 , Michael Paul a , Andrew Finch a , Shinsuke Sakai a , Thang Tat Vu a ,Noriyuki Kimura a , Chiori Hori a , Eiichiro Sumita a , Satoshi Nakamura a , Jun Park b ,Chai Wutiwiwatchai c , Bo Xu d , Hammam Riza e , Karunesh Arora f ,Chi Mai Luong g , Haizhou Li ha National Institute of Information and Communications Technology (NICT), Japanb Electronics and Telecommunications Research Institute (ETRI), Korea
c National Electronics and Computer Technology Center (NECTEC), Thailandd Institute of Automation, Chinese Academy of Sciences (CASIA), Chinae Agency for the Assessment and Application of Technology (BPPT), Indonesia
f Center for Development of Advance Computing (CDAC), India
g Institute of Information Technology (IOIT), Vietnam
h Institute for Infocomm Research (I2 R), Singapore

Received 14 March 2010; received in revised form 15 February 2011; accepted 6 July 2011
Available online 2 August 2011

Abstract
This paper outlines the first Asian network-based speech-to-speech translation system developed by the Asian Speech Translation
Advanced Research (A-STAR) consortium. Eight research groups comprising the A-STAR members participated in the experiments,
covering nine languages, i.e., eight Asian languages (Hindi, Indonesian, Japanese, Korean, Malay, Thai, Vietnamese, and Chinese)and English. Each A-STAR member contributed one or more of the following spoken language technologies: automatic speech
recognition, machine translation, and text-to-speech through Web servers. The system was designed to translate common spoken
utterances of travel conversations from a given source language into multiple target languages in order to facilitate multiparty travel
conversations between people speaking different Asian languages. It covers travel expressions including proper nouns that are namesof famous places or attractions in Asian countries. In this paper, we describe the issues of developing spoken language technologies
for Asian languages, and discuss the difficulties involved in connecting different heterogeneous spoken language translation systems
through Web servers. This paper also presents speech-translation results including subjective evaluation, from the first A-STAR
field testing which was carried out in July 2009.
© 2011 Elsevier Ltd. All rights reserved.
Keywords: Speech-to-speech translation; Speech recognition; Machine translation; Text-to-speech; Asian languages

夽 This paper has been recommended for acceptance by ‘Guest Editors Speech–Speech Translation.’.
∗ Corresponding author.

E-mail address: ssakti@is.naist.jp (S. Sakti).
1 Currently belongs to Nara Institute of Science and Technology, Japan.

0885-2308/$ – see front matter © 2011 Elsevier Ltd. All rights reserved.
doi:10.1016/j.csl.2011.07.001

510

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

1. Introduction
With an area of 17 million square miles, Asia is the world’s largest and most populous continent. It hosts approximately four billion people or 60% of the world’s current human population. A wide variety of societies, religions andethnicities shape the culture of Asia. There is also great linguistic diversity, and the majority of Asian countries havemore than one language that is natively spoken. In fact, from a total of more than 6800 living languages in the world,
one-third are found in Asia (Lewis, 2009).
As we enter the 21st century, information exchange among people from and into Asia is increasing. Meanwhile,the travel destinations of international travelers – whether for the purpose of tourism, emigration, or foreign study– are becoming increasingly diverse. In particular, the Asian region has witnessed a strengthening of its social andeconomic relationships. Enhancing mutual understanding and economic relations has thus become a key challenge.
Socio-economic relations in the Asian region are more vital today than ever before. These changes are increasing theneed for devising means that facilitate interaction among people in Asia speaking different languages. The language
barriers between Asian countries are critical problems to overcome.
Spoken language translation is one of the innovative technologies that enable people to communicate with eachother by speaking in their own languages. However, translating a spoken language is an extremely complex task.
This technology involves research in automatic speech recognition (ASR), machine translation (MT), text-to-speech
generation (TTS) and doing this for many languages imposes an even greater challenge. Ideally, to create effective
systems, many different countries would conduct joint research. Currently, a number of research groups are making
progress in a bid to break down the language barrier. As one of the first examples of international cooperation, theConsortium for Speech TrAnslation Research (C-STAR)2 was formed as a voluntary group of institutions committed
to building speech translation systems. Institutions in European countries have also formed TC-STAR3 (Technology
and Corpora for Speech-to-speech Translation) consortium, which has the objective of translating public speeches
and discussions in international meeting. However, these activities are still rare in the Asian community. Therefore,
the Asian Speech Translation Advanced Research (A-STAR)4 consortium was established in June 2006 as a speechtranslation consortium for creating the basic infrastructure for spoken language communication technologies andovercoming language barriers in Asia (Nakamura et al., 2007).
The A-STAR consortium was founded by the National Institute of Information and Communications Technology/Advanced Telecommunications Research (NICT/ATR), Japan, in collaboration with other research institutes in
Asia. Currently, the A-STAR consortium consists of the following members: the Electronics and Telecommunication
Research Institute (ETRI) in Korea, the National Electronics and Computer Technology Center (NECTEC) in Thailand,
the Institute of Automation, Chinese Academy of Sciences (CASIA) in China, the Agency for Assessment and Application Technology (BPPT) in Indonesia, the Center for Development of Advanced Computing (CDAC) in India, theInstitute of Information Technology (IOIT) in Vietnam, and the Institute for Infocomm Research (I2 R) in Singapore.
The consortium is working collaboratively to collect Asian language corpora, create common speech recognition andtranslation dictionaries, develop Web service speech translation modules for various Asian languages, and standardize
interfaces and data formats that facilitate international connections between the different speech translation modules
from different countries. The main objective is to create a network-based speech-to-speech translation system in the
Asian region, as illustrated in Fig. 1.
In this paper, we outline the development of the Asian network-based speech-to-speech translation system. The
system was designed to translate common spoken utterances of travel conversations from a given source language into
multiple target languages in order to facilitate multiparty travel conversations between people speaking different Asian
languages. Each A-STAR member contributed one or more of the following spoken language technologies: ASR, MT,
and TTS through Web servers. Currently, the system successfully covers nine languages: Hindi (Hi), Indonesian (Id),Japanese (Ja), Korean (Ko), Malay (Ms), Thai (Th), Vietnamese (Vi), Chinese (Zh) and English (En). The systemcovers travel expressions including proper nouns such as the names of famous places or attractions in Asian countries.

2 C-STAR – http://www.c-star.org.
3 TC-STAR – http://www.tcstar.org.
4 A-STAR – http://www.mastar.jp/AStar/.

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

511

Fig. 1. Outline of the network-based speech-to-speech translation system in the Asian region.

The rest of this paper is organized as follows. Section 2 gives an overview of Asian language processing. Section
3 describes the development of the spoken language translation engines for ASR, MT, and TTS. Issues related to
how to handle the additional proper nouns are discussed in Section 4. Next, the overall architecture of the A-STAR
speech-to-speech translation system architecture is provided in Section 5. Here, we also describe the standardized
data format and client application. Details about the first A-STAR demo experiments and speech-translation results
are reported and discussed in Section 6. Some highlights of the challenges that we faced in multiparty multilingual
speech-to-speech translation are summarized in Section 7. Finally, our conclusion is presented in Section 8.
2. Asian language processing
One of the first stages of developing a spoken language technology is the design, collection, and annotation of data
corpora. However, the processing of natural Asian languages into suitable structured data format poses a great challenge.
Table 1 summarizes the characteristics of the eight Asian languages (LNG) used in this experiment. As a reference,
English is also included. It shows that the Asian languages characteristics are very diverse, in many ways. In contrast toWestern languages that mostly use the Roman alphabet as their writing system, various scripts like Hanzi and Devanagari
are still in widespread use in Asia. The sentence structure also varies from the subject-verb-object (SVO) order used inIndonesian, Malay, Vietnamese, Thai, and Chinese, to the subject-object-verb (SOV) order used in Hindi, Japanese, andTable 1
Characteristics of Asian languages.
LNG

Sentence structure

Segmentation unit

Tonality

Writing script

En
Hi
Id
Ja
Ko
Ms
Th
Vi
Zh

SVO
SOV
SVO
SOV
SOV
SVO
SVO
SVO
SVO

Word
Word
Word
None
PhraseWord
None
Syllable
None

None
None
None
None
None
None
5 tones
6 tones
4 tonesRoman
Devanagari
Roman
Kanji-Hiragana-Katakana
HangulRoman
Thai Alphasyllabary
Roman with tone markers
Hanzi

512

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

Korean. Whitespace is not always used to separate word tokens. In Vietnamese, it is used to separate syllable tokens,
while in Korean it is used to separate word phrases. Chinese, Japanese, Thai, do not even have a proper boundaries forwords. Moreover, Thai, Vietnamese, and Chinese are also tonal languages, in which pitch contour plays an importantrole in speech communication. The tonality in these languages varies considerably. Consequently, these peculiarities
require special treatment. In the following sections, we highlight some major issues of Asian language processing.
2.1. Word segmentation
The task of word segmentation, i.e., identifying word boundaries in continuous text, is one of the fundamental
preprocessing steps of data driven natural language processing (NLP) applications. Unlike Indo-European languages
like English, many Asian languages like Chinese, Japanese, and Thai do not use a whitespace character to separate
meaningful word units.
Research into Asian word segmentation has been investigated for a long time using rule-based as well as statisticallearning methods. In NICT systems, machine learning methods are applied. For Chinese, a character-based part-ofspeech (POS) tagger and word segmentation are conducted simultaneously (Hu et al., 2009). It is trained using a
corpus, in which each character is tagged with its position in the word and the POS information of the word to which it
belongs. A maximum entropy model (ME) is adopted for implementation. We refer to this model hereafter as the ME
tagger. The output characters from the ME tagger is chunked by a conditional random field (CRF) model, which could
also be used to identify Chinese-named entity words in the text files. For Japanese, the word segmentation is producedby the Japanese morphological analyzer called ChaSen.5 The system segments Japanese text (sentences) string intomorphemes and tags those morphemes with their parts of speech (POS) and pronunciations by employing the POS
bi-gram Markov model. It also tokenizes conjugative morphemes with their base forms and conjugation types/forms(Asahara and Matsumoto, 2000).
Word segmentation in Thai has also been widely researched. However, identifying Thai word boundary is extremelycomplex because there are numerous forms of character composition that lead to ambiguous boundary alternatives. In
addition, Thai has no sentence boundary markers and the definition of sentence units is still unclear. These characteristics
obstruct the development of quality syntax parsers. A shared task project named Benchmark for Enhancing the Standardof Thai language processing (BEST)6 has been initiated to ensure the development of the best algorithm. As a result, a
guideline for Thai word segmentation, as well as an official, large word-annotated text corpus is available. Consequently,
NECTEC has proposed a novel concept that recognizes word boundary identification based on CRF (Haruechaiyasak
and Kongyoung, 2009), and this has been proven to be effective for many tasks. Recently, NECTEC also applied
statistical MT framework for word segmentation (Bangcharoensap et al., 2009), by simply viewing a string of single
characters as a source language and another string of corresponding word units as a target language.
In Korean, the writing system uses whitespace between two adjacent word-phrases, each of which generally corresponds to two or three words in English in a semantic sense. This word-phrase is basically an agglomerate of morphemes.
Although a space exists in the script, specific language-dependent word segmentation is still required. Choosing a wordphrase as a basic unit may cause a large number of out-of-vocabulary tokens in the system, while using a morpheme unit
also causes a severe inter-morpheme coarticulation problem. Here, ETRI proposed to merge several morphemes into abasic unit and define it as a word. Starting from the original morpheme units defined in the Korean morphology, pairsof short and frequent morphemes are merged into larger units by combining both rule-based and statistical methods(Kwon and Park, 2003).
2.2. Tonality
All languages use pitch to express emotion and convey emphasis, contrast, and other such features in what is called
intonation. But tonal languages, specifically use tones to distinguish lexical or grammatical meaning. So different tones
will change the meaning of the words, even if the pronunciation of the word is otherwise the same.

5 ChaSen – http://chasen.aist-nara.ac.jp/chasen/bib.html.en.
6 BEST: http://thailang.nectec.or.th/best.

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

513

Chinese is a tonal language, in which a word consists of one or more characters, with each character pronounced as amonosyllable with a pitch tone. Processing tone has always been an important topic in spoken language technology. In
the widely used traditional Chinese phoneme set of Initials/Finals (IFs), the number of non-tonal IFs is 59, while tonedependent IFs number more than 200. Since context-dependent models usually used in ASR systems, these numbers
explode into the millions, making it difficult to train and decode. NICT uses a novel way to derive an efficient phoneme
set of tone-dependent units according to the principle of minimal loss of mutual information (MI) (Zhang et al., 2008a).
The final phoneme set can be flexibly chosen based on a balance between the number of units and the information
quantities.
Thai and Vietnamese are also monosyllabic tonal languages. Five lexical tones exist in Thai, and Vietnamese has
six different tones that do not rely solely on the F0 contour. However, unlike the Chinese ASR system, the current
Vietnamese and Thai ASR systems do not yet consider tones in the phonetic symbols. For Vietnamese, IOIT simply
uses the W2P tool (Vu et al., 2009b, 2005), which consists of several rules that decompose a syllable into small parts:
Initial, Onset, Nucleus, and Coda. For Thai phonetic symbols, 74 phonemes are used where the initial consonants andfinal consonants were represented with different symbols (Kasuriya et al., 2003).
2.3. Multilingualism
The language structure of the Indonesian language is fairly simple compared to that of other Asian languages.
Difficulties arise in developing Indonesian spoken corpora since Indonesian is a “language of national unity” formed
from the hundreds of languages spoken on the Indonesian archipelago. It is reported that there exist 726 regional
languages in total, making Indonesia the world’s second most diverse country after Papua New Guinea, which has 823
local languages (Marti et al., 2005; Riza, 2008). Only 7% of the population speaks Indonesian language as their mothertongue, while the great majority of people speaking it as a second language with varying degrees of proficiency. It is
still considered too difficult to attempt to collect all existing languages and dialects of the ethnic groups recognizedin Indonesia. Similar to Indonesian, only 20% of Indians regard Hindi as their mother tongue. According to theConstitution of India, there are 22 recognized major national languages.
The Indonesia Telecom company, collaborating with ATR on the APT (Asia Pacific Telecommunity) project, sought
to develop Indonesian spoken corpora (Sakti et al., 2004, 2008a). This project successfully collected 80 h of speech
data that covered several of the major ethnic dialects spoken in Indonesia. As yet, Hindi speech corpora covering the
major ethnic dialects spoken in India are not available.
3. Spoken language technologies
The development of A-STAR spoken language technologies is described in the following section. Each A-STARmember contributes their ASR, MT, and TTS systems. All systems are allowed to be trained with any available corpora.
There is no restriction on the type of resources applied.
3.1. Automatic speech recognition
The resources and technologies used by each A-STAR for developing ASR server engines are summarized in Table 2.
The English, Japanese, Chinese, and Indonesian ASR server engines were provided by NICT. They were developedon the basis of hidden Markov network (HMnet) topology, which is obtained using a successive state splitting (SSS)
training algorithm based on the minimum description length (MDL) optimization criterion. In this case, MDL-SSS,
which is the SSS algorithm based on minimum description length, is proposed to stop the growth of the number ofstates according to the training data amount. Details about the HMnet topology based on MDL-SSS can be foundin (Jitsuhiro et al., 2004). For English, Japanese, and Chinese ASR, the composite multi-class bigram and trigram
language models were trained using the 160K sentences of the BTEC text data, with the vocabulary sizes: 44K, 66K,
and 47K words, respectively. For the Indonesian ASR, only standard bigram and trigram language models were trainedusing the 160K sentences of the BTEC text data (vocabulary size: 40K words).
The Korean ASR engines were developed by ETRI on the basis of the finite state transducers (FST) (Kwon andPark, 2003) with a 32K vocabulary and a word-based trigram LM computed from a newspaper text corpus. The mergedmorpheme units are used as words described in Section 2.1. ETRI also use pronunciation-dependent language modeling514

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

Table 2
Summarization of ASR resources and technologies.
LNG

Corpus type

Hours

Topology

Institutions

Reference

En
Id
Ja
Ko
Ms
Th
Vi
Zh

Travel expression
Daily news
Travel expression
Travel expression + monologue
Daily news
Daily news
Radio broadcast
Travel expression

202.2
79.5
270.9
330.0
100.0
124.0
40.5
249.2

HMnet (SSS)
HMnet (SSS)
HMnet (SSS)
FST
HMM
HMM
HMM
HMnet (SSS)

NICT
NICT
NICT
ETRI
I2 R
NECTECIOIT
NICTJitsuhiro et al. (2004)
Sakti et al. (2008a)
Jitsuhiro et al. (2004)
Kwon and Park (2003)
Li and Sim (2009)
Wutiwiwatchai et al. (2009)
Vu et al. (2005)
Zhang et al. (2008b)

so that a rigid Korean phonology rule could be implicitly reflected in the recognition process. The Thai ASR consistsof the ISPEECH Thai speech recognizer built at NECTEC (Wutiwiwatchai et al., 2009) based on the architecture of
the JULIUS7 speech recognizer developed by Kyoto University, Japan. The system was trained using the Thai LOTUS
corpus and the NECTEC-ATR corpus (Kasuriya et al., 2003). The word segmentation and grapheme-to-phoneme (G2P)
process are applied before the ASR conventional process.
The IOIT Vietnamese ASR was developed on the basis of hidden Markov model (HMM) (Vu et al., 2009b, 2005)
using the “Voice of Vietnam” (VOV) corpus (Mai and Duc, 2006). Force alignment is done by English–Vietnamesecross-language approach, where the English AM is used as initial Vietnamese AM through the English–Vietnamesephoneme mapping. For the Vietnamese ASR, the use of language model can improve the recognition accuracy, but its
performance is not enough to solve the tonal problem (Vu et al., 2005). Recent studies propose to apply informationfrom the F0 contour using the multi-layer perceptron (MLP) neural network (Vu et al., 2008). The I2 R Malay ASR
was a phoneme-based HMM system that was built on I2 R’s Abacus large vocabulary continuous speech recognition
platform (Li and Sim, 2009). Standard bigram and trigram language models are also applied for Thai, Vietnamese, andMalay ASR, with the vocabulary sizes: 70K, 30K, and 50K words, respectively. Preliminary experiments were also
conducted for CDAC Hindi ASR using 20 h of Hindi female speech only. However, since the Hindi ASR system was
not yet complete, it was not successfully provided during the A-STAR field testing.
3.2. Machine translation
The MT engines for all 72 combinations of language pairs are provided on NICT translation servers (Sakti et al.,2009). They were trained on the multilingual Basic Travel Expressions Corpus (BTEC)(Kikui et al., 2006; Paul et al.,2008), which is a collection of sentences that bilingual travel experts consider useful for people going to or comingfrom another country. The BTEC corpus used for training consists of 20K sentences for all of the nine languages
involved, whereby all data sets are aligned at the sentence-level.
Table 3 summarizes the characteristics of the BTEC corpus data sets used for the training (train) of the MT models,
the tuning of model weights (dev), and the evaluation of translation quality (eval). Besides the number of sentences(sen) and the vocabulary (voc), the sentence length (len) is also given, as the average number of words per sentence. To
get an idea how difficult the translation tasks may be, we calculated the language perplexity (perpl) of the respective
evaluation data sets according to the language model used by the baseline system. The perplexity in Table 3 indicatethat Chinese and Hindi are the most difficult languages (as shown in bold text), followed by Malay, Indonesian, Thai,
English, Vietnamese, Korean, and then Japanese. Word segmentation and tokenization are applied as described in
Section 2.1. For the training of the MT models, standard word alignment (Och, 2003) and language modeling (Stolcke,
2002) tools were used. Five-gram language models built with Kneser-Ney smoothing were used along with a lexicalized
distortion model. Minimum error rate training (MERT) was used to tune the decoder’s parameters, and was performedon the dev set using the technique proposed in Och (2003). For the translation, a multi-stack phrase-based Cleopatra
decoder (Finch et al., 2007) was used.

7 JULIUS – http://julius.sourceforge.jp/en index.php.

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

515

Table 3
Language characteristics of the BTEC corpus.
LNG
 of sentrain
20,000

dev
1000

eval
1000

En

voc
len
perpl

6033
7.2
–

1262
7.1
19.1

1292
7.2
19.9

Hi

voc
len
perpl

9549
7.6
–

1558
7.4
25.1

1588
7.5
25.8

Id

voc
len
perpl

6993
6.5
–

1433
6.3
24.4

1394
6.4
23.4

Ja

voc
len
perpl

7002
8.2
–

1407
8.1
15.2

1408
8.2
15.0

Ko

voc
len
perpl

6628
7.8
–

1366
7.7
16.3

1365
7.8
15.7

Ms

voc
len
perpl

7209
6.6
–

1459
6.4
24.7

1438
6.5
23.5

Th

voc
len
perpl

3471
7.4
–

1081
7.3
24.2

1053
7.4
23.4

Vi

voc
len
perpl

4373
8.7
–

1245
8.5
17.7

1267
8.6
18.0

Zh

voc
len
perpl

5996
6.6
–

1312
6.4
28.6

1301
6.5
26.3

In addition to the NICT MT engines described above, all other institutions also provide bi-directional MT systems
between English and their own languages. These are summarized in Table 4. The Korean–English bi-directional machinetranslation system provided by ETRI is based on rules and patterns or is known as rule-based machine translation(RBMT) (Lee et al., 2006; Hong et al., 2005). The major resources of translation knowledge are a probabilistic dictionary, a word dictionary, a composition word dictionary, structure analysis rules and translation patterns. To translate
conversational text, a translation memory of 300K sentences was added. NECTEC uses the MOSES tool (Koehn et al.,2007) – a statistical open source translation toolkit – to build a Thai–English MT engine (Wutiwiwatchai et al., 2009).
Table 4
Summarization of MT technologies.
LNG (source)

MT type

Decoder

Institutions

Reference

Hi
Id
Ja
Ko
Ms
Th
Vi
Zh

SMT
SMT
SMT
RBMT
RBMT
SMT
SMT
SMT

Cleopatra
Moses
Cleopatra
ParserPiramid
Moses
Moses
Cleopatra

CDAC
BPPT
NICT
ETRI
I2 R
NECTECIOIT
NICTArora and Paul (2008)
Budiono et al. (2009) and Riza and Riandi (2008)
Finch et al. (2007)
Lee et al. (2006) and Hong et al. (2005)
Aw et al. (2009) and Chen et al. (2008)
Wutiwiwatchai et al. (2009)
Vu et al. (2009b)
Finch et al. (2007)

516

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

Table 5
Summarization of TTS resources and technologies.
LNG

Voice

Hours

Topology

Institutions

Reference

En
Hi
Id
Ja
Ko
Ms
Th
Vi
Zh

Male
Male
FemaleFemaleFemaleFemaleFemaleMale
Female

16.0
1.0
2.0
20.0
3.0
2.0
14.0
3.0
20.0

Concatenative
HMM
HMM
Concatenative
HMM
HMM
Concatenative
HMM
Concatenative

NICT
CDAC
NICT
NICT
ETRI
NICT
NECTECIOIT
NICT

Kawai et al. (2004)
Arora et al. (2009)
Sakti et al. (2008b)
Kawai et al. (2004)
Oh and Kim (2006)
Sakti et al. (2008b)
Thangthai et al. (2008)
Vu et al. (2009a)
Kawai et al. (2004)

Similar to ASR, automatic word segmentation is applied before training the translation model. To improve the wordalignment, NECTEC proposed pre-processing the parallel texts by incorporating rules for word re-ordering in nounphrases. Similar to the Thai MT engine, the Vietnamese MT engine also use the MOSES framework to train the models
and decoding (Vu et al., 2009b). For the BPPT Indonesian–English bilingual translation system, various parallel text
resources were used to train the models. Among them were three different corpora: ANTARA news, BPPTPANL and
BTEC (Budiono et al., 2009). The system is also based on the MOSES framework. In contrast, with ETRI, NECTEC, andthe BPPT MT system, I2 R constructs a rule-based Malay–English translation system. This system, called Piramid (Awet al., 2009; Chen et al., 2008), utilizes rich linguistic knowledge and performs morphological analysis, as well as syntactic parsing on the source text to derive linguistic information. The word ordering models used both lexical and structuraltransformation.
3.3. Text-to-speech synthesis
For the TTS engines, Table 5 summarizes the resources and technologies used by all A-STAR members. The NICT
English, Japanese, and Chinese speech synthesis engines were developed on the basis of a waveform concatenation
algorithm (Kawai et al., 2004) in which appropriate subword units were selected from speech databases. The NECTEC
Thai speech synthesis, called VAJA (Rugchatjaroen et al., 2007), was also developed based on variable-length unit
selection. It applied phrase breaking and duration modeling to improve speech prosody generation. NECTEC also
introduce F0 analysis and synthesis algorithm called T-Tilt to accommodate syllable-unit F0 contours in tonal languages(Thangthai et al., 2008).
The NICT Indonesian/Malay (Sakti et al., 2008b), ETRI Korean (Oh and Kim, 2006), CDAC Hindi (Arora et al.,2009), and IOIT Vietnamese (Mixdorf et al., 2004) speech synthesis engines were developed using only a few hours ofspeech data. Since the training data were not large enough to build unit-concatenation speech synthesis engines, they
were developed on the basis of statistical parametric speech synthesis (Tokuda et al., 2000) in which speech waveforms
are synthesized directly from the parameters generated from the HMMs. This approach offers the ability to model
different speech styles without the need for recording very large databases. Specifically for IOIT Vietnamese TTS,
since Vietnamese is a tonal language, a tone-dependent phone sets and corresponding phonetic and prosodic question
set for the decision tree are developed. A tree-based context clustering is designed to have tone correctness, which is
crucial in Vietnamese speech.
3.4. Summary of A-STAR spoken language technologies
Table 6 summarizes the spoken language technologies provided by A-STAR members. In total, there
are ASR engines for 8 different languages, TTS engines for 9 languages, and MT engines for 72(= 9 × 8)
different language combinations. More details about the specifications of each spoken language translation
system provided by each member can be found in the reference sources listed in the last column ofTable 6.

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

517

Table 6
Speech technology servers provided by A-STAR members.
A-STAR members

Servers

System reference

ASR

TTS

MT

NICT

EnglishJapanese
ChineseIndonesianAll combination
language pairs(9 × 8) = 72 MT engines

Sakti et al. (2009)ETRI
NECTEC
BPPT
CDAC
IOIT
I2 R

KoreanThai––
Vietnamese
Malay

EnglishJapanese
ChineseIndonesian
Malay
KoreanThai–
Hindi
Vietnamese–

Korean–English
Thai–English
Indonesian–English
Hindi–English
Vietnamese–English
Malay–English

Lee et al. (2009)
Wutiwiwatchai et al. (2009)
Riza and Riandi (2008)
Arora et al. (2009)
Vu et al. (2009b)
Chen et al. (2008)

4. Multiparty dialog scenario and proper nouns
We have also collected additional multiparty dialog scenarios including famous proper nouns, as follows:• Proper nouns
The proper nouns are collected from ten Asian countries: India, Indonesia, Japan, Korea, Malaysia, Singapore,
Thailand, Vietnam, China, and the United States. They mainly comprise city names (e.g., Kyoto in Japan, Beijing
in China), tourist areas (e.g., Bulkuksa in Korea, Wat Pra Kaew in Thailand), attractions (e.g., Wayang kulit inIndonesia, Kathak in India), and similar nouns. In total, there are about 100–150 additional proper (10–15 proper
nouns from each country) which are then translated to all other eight languages. To handle the proper nouns thatnever appear or appear less frequently in the training data, we enhance the language model (LM) reliability byinterpolating both the class bigram of BTEC text and the new word bigram of the dialog scenario as follows:
P(wi |wi−1 ) = αPbtec (ci |ci−1 )P(wi |ci ) + βPdialog (wi |wi−1 ),(1)where α represents the weight of BTEC LM, and β represents the weight of dialog LM. The optimal value ofinterpolation weights are estimated using a development set other than training or using the cross-validation method(Huang et al., 2001). In this study, we estimated the interpolation weights using a development set.
• Multiparty dialog scenario
The additional multiparty dialog scenario is developed as in the following:- Step 1: Generate 25 dialog scenario (a total of 228 sentences) in English including proper nouns from the UnitedStates only.
- Step 2: Translates all sentences to other eight languages (Hindi, Indonesian, Japanese, Korean, Malay, Thai,
Vietnamese, and Chinese).
- Step 3: For each dialog scenario, generate all combination of two-to-four party conversation between different
language sentences, including the appropriate proper nouns, resulting about 6150 multiparty multilingual dialog
scenarios.
5. Integration of network-based speech-to-speech translation
The spoken language technologies, including ASR, MT, and TTS engines described in Section 3, are provided by
A-STAR members through Web servers, and form a unified multilingual network-based speech-to-speech translation
system. The overall structure is illustrated in Fig. 2, and the other component systems are described in the following
sections.

518

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

Fig. 2. Architecture of client–server interaction.

5.1. Speech Translation Markup Language (STML) servlet
To enable communication and the exchange of data between client users and speech language technology servers, all data exchanges among client users and spoken language technology servers are managed through a Web
service designed by NICT, the so-called STML servlet (Kimura, 2008). It follows a standard protocol, namely,
STML.
The communication data format consists of extensible markup language (XML) and binary data including a terminal frame, as described in Fig. 3. Each frame consists of a 4-byte header and actual data. The first frame is anFig. 3. Standardization of communication data format.

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

519

Fig. 4. The client application on a hand-held terminal device.

XML header, followed by binary data from the second frame. For the terminal frame, the data size of the header
is set to 0. The input speech format is 16-kHz, 16-bit, mono-channel, and big/little endian. This allows the raw,
distributed speech recognition (DSR), or adaptive differential pulse code modulation (ADPCM) speech format. The
recognition results are N-Best with a surface form or confusion network. MT input and output strings either have asurface form or a confusion network. As for the speech synthesis, the input string has a surface form, and the output
sound format is 16-kHz/8-kHz, 16-bit, mono-channel, and big/little endian. This allows the raw or ADPCM speechformat.
5.2. Client application
The client applications are implemented on a handheld mobile terminal device (Sony VAIO-U) shown in Fig. 4, whichallows portable speech-to-speech translation. The device is 150 mm (width) × 32 mm (thickness) × 95 mm (height). A
unidirectional microphone is used for speech recognition in noisy environments. The system uses Microsoft Windows
and a Java operating environment. The client application also supports an external TV conference feature that allows
both speech and video interaction. It was developed by NICT and supports both speech and video interaction between
client users.
5.3. Communication server
A communication server, also provided by NICT, is used to relay the speech results from one user to all other users
to enable them to perform a multiparty conversation.
For example, when a Japanese user utters sentence like “Konnichiwa,” the client application will perform a speechrecognition operation on an available Japanese ASR server through the STML servlet. To simultaneously distributethe speech recognition result to all the other clients who are connected to each other, the speech recognition results
are first sent to the communication server. Next, the communication server distributes the results to the remaining
clients. The clients, for example, an Indonesian user and a Vietnamese user, who receive the speech recognition
results from a Japanese user, will perform a translation operation on the available Indonesian and Vietnamese MT
servers using an STML servlet. In this case, the servers provide a sentence output in Indonesian, “Selamat siang,”and one in Vietnamese, “Xin cha‘o.” Finally, the TTS servers of these languages produce the speech of the resulting
sentence. In this way, the translation mechanism can be used for translating any multiparty conversation comprising
any or all of the following languages – Hindi, Indonesian, Japanese, Korean, Malay, Thai, Vietnamese, Chinese, orEnglish.

520

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

Fig. 5. Word recognition accuracy rates of the ASR engines on BTEC test set.

6. Evaluation of translation server components
The first A-STAR demo experiments for connecting spoken language technologies in the Asian region were carriedout in July 2009. Two evaluations based on basic travel expression sentences, BTEC, and dialog utterances wereconducted as described in the following section.
6.1. Evaluation on BTEC test set
Fig. 5 shows the word recognition accuracy rates of the English, Indonesian, Japanese, Korean, Malay, Thai,
Vietnamese, and Chinese ASR engines on the BTEC test set. The evaluation of the BTEC test set was performed
offline using randomly selected recorded speech files from a total of 20,400 utterances spoken by 40 different speakers(20 males, 20 females), where each speaker uttered the same 510 BTEC test sentences. Most languages have achieved
over 80% accuracy, which are encouraging figures for performing (facilitating) reliable speech translation. The best
performance was achieved by the Japanese ASR, with 94.87% word accuracy.
For the automatic evaluation of translation quality, we applied the standard BLEU metric that calculates the geometricmean of n-gram precision by system output with respect to reference translations (Papineni et al., 2002). Scores range
between 0 (worst) and 1 (best). The experimental results summarized in Table 7 show the quality of the direct translation
for all the 72(= 9 × 8) language pairs of the MT engines. This evaluation was performed offline on BTEC test set data
without ASR errors. The shortage of parallel corpus in this experiments cause the overall BLEU score to be relatively
low. However, the good performance could be achieved by Malay–Indonesian MT and Japanese–Korean MT with a
BLEU score of 60–80%, given the high similarity of both languages in syntactic and semantic structure.
Another way is to perform a pivot translation in which the translation process is done from a source language (SRC)
to a target language (TRG) through an intermediate pivot (or bridging) language (PVT). For example, a Korean–Thaitranslation is performed by cascading ETRI Korean–English and NECTEC English–Thai translation systems. Here,
English is used as a pivot language. In a real-world application, this approach is often used since identical language
resources covering three or more languages are not necessarily to be expected. To evaluate the pivot translation approach,
Table 7
Translation quality of direct translation approaches (BLEU).
SRC/TRG
En
Hi
Id
Ja
Ko
Ms
Th
Vi
Zh

En
–
40.77
47.01
29.83
27.14
48.78
42.44
48.87
28.12

Hi
29.25
–
27.49
14.86
13.76
28.22
24.49
25.43
14.47

Id
46.32
32.00
–
25.18
23.62
81.99
37.54
39.10
24.85

Ja
34.29
26.33
32.06
–
63.19
31.55
31.15
28.92
43.87

Ko
31.36
24.11
30.53
62.36
–
28.96
28.72
28.95
39.18

Ms
46.99
32.47
77.18
24.22
23.00
–
37.72
40.44
24.05

Th
42.74
31.89
40.53
29.60
28.01
40.75
–
40.54
27.78

Vi
46.23
32.52
41.04
25.59
24.75
41.42
39.32
–
24.63

Zh
26.61
20.07
26.42
38.72
35.20
25.52
25.92
23.56
–

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

521

Table 8Translation quality of pivot translation approaches SRC–English–TRG (BLEU).
SRC/TRG
Hi
Id
Ja
Ko
Ms
Th
Vi
Zh

Hi
–
23.06
12.88
11.02
23.66
19.33
21.35
12.49

Id
21.00
–
16.93
15.85
38.32
25.32
27.68
16.03

Ja
20.34
25.32
–
27.28
25.25
22.29
23.14
24.51

Ko
19.48
25.25
27.97
–
25.77
22.71
23.42
23.85

Ms
21.47
39.34
16.50
15.19
–
25.16
28.03
16.18

Th
22.22
29.80
19.39
18.24
28.69
–
28.84
18.83

Vi
23.39
31.06
18.58
17.64
30.94
28.39
–
17.99

Zh
14.77
17.92
20.00
18.98
17.67
18.36
17.63
–

the 20k sentence-aligned BTEC corpus used for the direct translation experiments was randomly split into two subsets
of 10k sentences each, whereby the first set of sentence pairs was used to train the source-to-pivot translation models(10ksp ) and the second subset of sentence pairs was used to train the pivot-to-target translation models (10kpt ). The
quality of the pivot translation for all Asian language pairs with English as pivot is shown in Table 8.
Comparing the results of the pivot translation experiments with the direct translation results, we can see that ingeneral the pivot translation approach performs more poorly than the direct translation approach due to the effect oferror chaining, i.e., translation errors in the SRC–PVT engine cause a degradation in the translation quality of the
PVT–TRG system output.
The quality of synthesized speech was also assessed by applying the mean opinion score (MOS) (CCITT, 1984)
test. Subjects listened to each presented speech and were required to rate the overall quality with regard to aspects
such as naturalness, acceptability, and clarity. A 5-point MOS scale was mostly used, with 5 denoting “excellent” and1 denoting “bad.” However, since the Japanese TTS system had been evaluated in more detail with a 7-point MOS
scale, here the Japanese MOS result was normalized to a 5-point scale. The summary of the MOS result is shown in
Fig. 6. The results imply that the overall quality of all the systems was generally fairly good. Detailed information ofeach evaluation can be found in Kawai et al. (2004); Rugchatjaroen et al. (2007), and Vu et al. (2009a).
6.2. Evaluation on dialog scenario test set
To obtain an idea of how good the overall speech-translation quality is when online conversations are performed,
we conducted an evaluation of a dialog scenario test set. It consists of 228 dialog sentences for each language (seeSection 4). Because of time limitations, we only perform the evaluation on a subset of the investigated languages, i.e.,the English, Indonesian, Japanese, Malay, Vietnamese, and Chinese languages.
The recognition results are shown in Fig. 7. The performance drops from the offline BTEC test set to the online
dialog scenario test set due to different speaking styles between read speech and conversational speech. Another reason
is that there are many proper nouns that are not familiar to the speakers. For example, native English speakers faced
more difficulties in pronouncing some specific Asian proper nouns compared with other Asian speakers. In these cases,
the English ASR displays the largest decline.

Fig. 6. MOS quality results for the TTS engines.

522

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

Fig. 7. Word recognition accuracy rates on dialog scenario test set.
Table 9Translation quality for both CRR and ASR input conditions (BLEU).
SRC–TRG

ASR

CRR

En–Ja
Ja–En
Ja–Hi
Ja–Id
Ja–Ko
Ja–Ms
Ja–Th
Ja–Vi
Ja–Zh

27.82
38.11
10.87
25.14
19.24
33.71
32.44
25.10
35.11

42.79
43.01
12.24
28.21
22.96
34.90
37.74
28.00
41.07

For machine translation, the BLEU automatic evaluation scores for both the correct recognition result (CRR) and
ASR input conditions are summarized in Table 9. The scores are given in percent (BLEU points). First, we evaluate
the translation quality in Japanese–English speech translation by translating Japanese input sentences in the dialog
scenario test set into English and vice versa. The translation of English CRR input sentences into Japanese obtained
42.79 BLEU points, while the translation of Japanese CRR input sentences into English obtained 43.01 BLEU points.
Comparing the automatic evaluation scores for the different CRR and ASR input condition, much lower scores are
obtained when translating noisy input. However, the effect of ASR errors depends largely on the recognition quality
and the target language. For language pairs having Japanese as the input, the BLEU scores decreases by 1.0–5.0 BLEU
points. The effects of English recognition errors are even more severe.
As for other languages, because of the poor performance of the English ASR performance on the dialog scenario
test set, the translation quality was only evaluated from Japanese input sentences. High scores of 35–43 BLEU points
were obtained for the translation of Japanese CRR input sentences into Chinese, Thai, and Malay. Moderate scoresof 23–29 BLEU points were achieved for the translation of Japanese into Indonesian, Korean, and Vietnamese. The
lowest automatic evaluation scores, however, were achieved for Hindi, which is a language with a rich morphology. In
Hindi, words belonging to various grammatical categories appear in lemma and inflectional forms, resulting in a large
number of inflectional word form variations, e.g., in the case of nouns, the words are inflected based on the number(singular or plural), case (direct or oblique), and gender (masculine or feminine). The automatic evaluation score will
be low, if a word variation output by the MT system cannot be exactly matched in the reference translation, althougha human might judge the respective translation as acceptable.
Automatic evaluation metrics are designed to judge the translation quality of the MT system outputs on a document
level, i.e., scores are calculated on the sets of all evaluation data sentences, but not at the sentence level. Here, we also
conducted a subjective evaluation where a native speaker of the target language had to assign one of the following
grades to each of the evaluation sentences:• “P” perfect translation• “A” translation contains errors, but is still acceptable• “I” translation is incorrect

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

523

Fig. 8. Subjective evaluation.

For each language, the evaluation was done by one expert evaluator, with some of them actively taking part in alarge-scale shared task evaluation such as the IWSLT evaluation campaign (Paul et al., 2010).
The subjective evaluations were also obtained for the translation of English input sentences into Japanese, andJapanese input sentences into all other languages. The results are shown in Fig. 8. The results reveal that, for the
majority of investigated language pairs, the systems could achieve within a range of 79–99% at grade perfect or
acceptable (P + A) given CRR input, and 62–93% at grade perfect or acceptable (P + A) given ASR input.
6.3. User satisfaction surveys
To evaluate user satisfaction, we conducted a survey questionnaire to solicit feedback from the users who taken
part in the A-STAR demo experiments. The questionnaire includes the following issues: the quality of recognition andtranslation components, feasibility of multi-party communication, system response time and system design.
In these experiments, about 20 users (14 males and 6 females), aged between 20 and 60 years old, took part in thissurvey. User responses were measured on a five-point scale. Rankings ranged from 5, indicating strong satisfaction, to
1, indicating strong dissatisfaction. Average scores for each question were summarized in Table 10. The recognition
and translation quality achieved a 3.5–3.8 user satisfaction rate, multi-party communication achieved 3.7–3.8, system
response achieved 3.6, and system design achieved 4.0. The highest rates were achieved for the use of back translationand video window which reach up to 4.4 of the user satisfaction rate. Looking at the responses in more details, morethan 85% of the users think that the back translation displayed on the screen was useful, and more than 80% of theTable 10User satisfaction surveys.
No

Questionnaire

Rating

1
2
3
4
5
6
7
8
9
10

The translation device accurately recognized the voices in the conversation.
The translation device was very fast to translate what user said or what the other party said.
The back translation displayed on the screen was very useful.
What the user wanted to say was accurately conveyed.
The user completely understood what the other party was trying to say.
The multi-party communication worked very well.
Overall rating for communication between people with different languages.
The system user interface was extremely easy to use and understand.
The video window was very helpful for improving communication.
Overall rating for system design.

3.5
3.6
4.4
3.6
3.8
3.7
3.8
3.9
4.2
4.0

524

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

users think that the video window was very helpful for improving communication. In average, the system obtained
about 3.85 of the user-satisfaction rate.
7. Challenges on multiparty multilingual speech-to-speech translation
In addition to the questionnaire described above, users were allowed to provide some feedback on problems that
occur during the A-STAR demo experiments. Here are some highlights of the challenges that we faced in multiparty
multilingual speech-to-speech translation.
7.1. Data sparsenessPhrase-based SMT systems train their statistical models using parallel corpora (Koehn et al., 2003). However,words that do not appear in the training corpus cannot be translated. Dealing with noisy translation input as well as
the translation of languages with rich morphology, while having only limited bilingual resources, makes this problem
even more severe. Given the large number of inflectional variations, many inflected words may not occur in the training
corpus. For unknown words (OOV), no translation entry is available in the statistical translation model (phrase-table).
As a result, these OOV words cannot be translated.
To overcome the problem, recent research focus on the incorporation of parapharasing techniques (Onishi et al.,2010) and lexical approximation methods (Paul et al., 2009) into the SMT framework that will help to improve theoverall system performance further. Concerning lexical approximation methods, we investigate two types of OOV, i.e.,(1) words that have not appeared in the training corpus, but for which other inflectional forms related to the given
OOV can be found in the corpus, and (2) words that appeared in the phrase-table in the context of larger phrases, but
do not have an individual phrase-table entry.
For a given OOV word, lexical approximation techniques are used to identify spelling and inflectional word variants
that occur in the training corpus. The proposed lexical approximation method applies spelling normalizers and lemmatizers to obtain word stems and generates all possible inflected word forms, whereby the variant candidates are chosenfrom the closest category sets to ensure grammatical features similar to the context of the OOV word. A vocabularyfilter is then applied to the list of potential variant candidates to select the most frequent variant word form. All OOV
words in the source sentence are replaced with appropriate word variants that can be found in the training corpus, thus
reducing the number of OOV words in the input. However, a source word can only be translated by phrase-based SMT
approaches if a corresponding target phrase is assigned in the phrase-table.
To increase the coverage of the SMT decoder, we extend the phrase-table by adding new phrase-pairs for all source
language words that do not have a single-word entry in the phrase-table but only appear in the context of larger
phrases. For each of these source language words SW, all target language phrases that are assigned to source language
phrases containing SW are extracted from the original phrase-table. For all of these target phrases, the sub-phrase
corresponding to SW is identified and the most frequent sub phrase TMAX is selected to extend the original phrase-tablewith the new lexical pair (SW, TMAX ). The extended phrase-table is then re-scored to adjust the translation probabilitiesof all phrase-table entries accordingly.
7.2. Multilingual noun entities
There are many cases in which the original script of proper nouns in one language is not available to write down aword in the other languages. Here, transliteration was used to map a one-to-one correspondence of character and be
exact, so that the original spelling of unknown transliterated words should be able to reconstructed again.
Traditional model of transliteration usually used phrase-based statistical machine translation. This technique relies
on maximum likelihood training coupled with the expectation-maximization (EM) algorithm, but has serious issues
with overfitting the training data. Because of these issues, alignment is typically performed in a one-to-many mannerfrom source-to-target and from target-to-source, and the phrase extraction process proceeds heuristically from analignment table.
To overcome the problems we investigated the novel Bayesian model for unsupervised bilingual character sequence
segmentation of corpora for transliteration. The system is based on a Dirichlet process model trained using Bayesian inference through blocked Gibbs sampling implemented using an efficient forward filtering/backward sampling

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

525

dynamic programming algorithm. The Bayesian approach seems able to overcome the overfitting problem inherent inmaximum likelihood training. The phrase-table resulting from our Bayesian segmentation model was approximately
30% smaller than that produced by the SMT system’s training procedure, and gave an increase in transliteration quality
measured in terms of word accuracy. More details about this research can be found in Finch and Sumita (2010).
7.3. Design issues on multi modal interaction
In the current A-STAR speech translation system, we mainly focus only on uni-modal speech interaction between
people speaking difference languages. However, it is often difficult to perform network-based speech-to-speech translation across different countries by relying only on speech input/output. The main reason is that diverging network
conditions in each country mean different time lengths required to finish each request. Thus, there may be a case inwhich the users could not determine whether their speech has already been successfully translated or is still in theserver queue waiting to be translated.
On the other hand, by using additional modality such as video conferencing, users no longer need to blindly wait for
their request, as they can see whether their interlocutors have received their messages or not. This is also shown fromuser feedbacks, which reveals that the use of video conferencing plays a significant role in improving communication.
In current prototype system, however, we only utilized the commercial web-based video conference, separate fromour speech translation system. This makes the start-up between two separate systems difficult. For future research,focusing on the deployment of multi-modality interaction, the integration of audio, video, and even gestures may
further improve the communication and interaction between people speaking different languages, including spoken
languages or sign languages.
8. Conclusion
The first Asian network-based speech-to-speech translation experiments were performed in July 2009. Eight researchgroups comprising the A-STAR consortium members participated in the experiments, covering eight Asian languages
and the English language. All the speech-to-speech translation engines have already been successfully implemented
into Web servers that can be accessed via client applications worldwide. This implementation has achieved the desired
objective of real-time, location-free speech-to-speech translation of Asian languages. Although the experiments have
proved quite successful, there are still many challenges to be overcome. Some of these are the need for speechto-speech translation engines that are able to handle conversational speech as well as out-of-vocabulary words, as
there are many out-of-vocabulary proper nouns. New A-STAR partners are now being sought in Asia for translation
projects involving new languages. Thus, future directions may include work and research to support other Asianspoken language technologies, the inclusion of handling a greater diversity of Asian tourism proper nouns, and multimodal interactions. The individual components of speech-to-speech translation are also expected to be employed ina wider range of applications, including speech-information retrieval, interactive navigation, dictation, and speechsummarization.
References
Arora, E.S.K., Paul, M., 2008. Handling of out-of-vocabulary words in phrase-based statistical machine translation for Hindi-Japanese. In: Proc.
NLP, Tokyo, Japan, pp. 61–64.
Arora, S., Mathur, R., Arora, K., Agrawal, S., 2009. Development of HMM-based Hindi speech synthesis system. In: Proc. TCAST Workshop,
Suntec, Singapore, pp. 31–34.
Asahara, M., Matsumoto, Y., 2000. Extended models and tools for high performance part-of-speech Tagger. In: Proc. COLING Workshop,
Saarbrucken, Germany, pp. 21–27.
Aw, A., Aljunied, S., Lee, L., Li, H., 2009. Piramid: Bahasa Indonesia and Bahasa Malaysia translation system enhanced through comparable
corpora. In: Proc. TCAST Workshop, Suntec, Singapore.
Bangcharoensap, P., Porkaew, P., Supnithi, T., 2009. A Statistical-machine-translation approach to word boundary identification: a projective analogyof bilingual translation. In: Proc. International Symposium on Natural Language Processing, Bangkok, Thailand, pp. 8–12.
Budiono, H.R., Riza, H., Hakim, C., 2009. Resource Report: Building Parallel Text Corpora for Multi-Domain Translation System. In: Workshop
on Asian Language Resources, ACL-IJCNLP, Suntec, Singapore, pp. 92–95.
CCITT, 1984. Absolute Category Rating (ACR) Method for Subjective Testing of Digital Processors. Red Book.

526

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

Chen, B., Xiong, D., Zhang, M., Aw, A., Li, H., 2008. I2R multi-pass machine translation system for IWSLT 2008. In: Proc. IWSLT, Hawaii, USA,
pp. 46–51.
Finch, A., Sumita, E., 2010. A Bayesian model of bilingual segmentation for transliteration. In: Proc. IWSLT, Paris, France, pp. 259–266.
Finch, A., Denoual, E., Okuma, H., Paul, M., Yamamoto, H., Yasuda, K., Zhang, R., Sumita, E., 2007. The NICT/ATR speech translation system
for IWSLT 2007. In: Proc. IWSLT, Trento, Italy, pp. 103–110.
Haruechaiyasak, C., Kongyoung, S., 2009. TLex: Thai Lexeme analyser based on the conditional random fields. In: Proc. International Symposium
on Natural Language Processing, Bangkok, Thailand, pp. 13–17.
Hong, M., Kim, Y., Kim, C., Yang, S., Seo, Y., Ryu, C., Park, S., 2005. Customizing a Korean–English MT system for patent translation. In: Proc.
MT Summit, Phuket, Thailand, pp. 181–187.
Hu, X., Isotani, R., Nakamura, S., 2009. Construction of Chinese segmented and POS-tagged conversational corpora and their evaluations on
spontaneous speech recognitions. In: Proc. 7th Workshop on Asian Language Resource, Annual Meeting of Association for Computational
Linguistics (ACL), Singapore, pp. 65–70.
Huang, X., Acero, A., Hon, H.-W., 2001. Spoken Language Processing. Prentice-Hall, NJ, USA.
Jitsuhiro, T., Matsui, T., Nakamura, S., 2004. Automatic generation of non-uniform HMM topologies based on the MDL criterion. IEICE Trans.
Inf. & Syst. E87-D (8), 2121–2129.
Kasuriya, S., Sornlertlamvanich, V., Cotsomrong, P., Jitsuhiro, T., Kikui, G., Sagisaka, Y., 2003. Thai speech database for speech recognition. In:
Proc. Oriental COCOSDA, Sentosa, Singapore, pp. 105–111.
Kawai, H., Toda, T., Ni, J., Tsuzaki, M., Tokuda, K., 2004. XIMERA: a new TTS from ATR based on corpus-based technologies. In: Proc. ISCA
Speech Synthesis Workshop (SSW5), Pittsburgh, USA, pp. 179–184.
Kikui, G., Yamamoto, S., Takezawa, T., Sumita, E., 2006. Comparative study on corpora for speech translation. IEEE Transactions on Audio, Speech,
and Language Processing 14 (5), 1674–1682.
Kimura, N., 2008. Speech Translation Markup Language (STML) Version 1.00, Tech. Rep., NICT, Kyoto, Japan.
Koehn, P., Och, F.J., Marcu, D., 2003. Statistical phrase-based translation. In: Proc of the Human Language Technology Conference,
pp. 127–133.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar,
O., Constantin, A., Herbst, E., 2007. Moses: open source toolkit for statistical machine translation. In: Proc. ACL, Prague, Czech Republic,
demonstration session.
Kwon, O., Park, J., 2003. Korean large vocabulary continuous speech recognition with morpheme-based recognition units. Speech Communication
39 (3-4), 287–300.
Lee, K., Park, S., Kim, H., 2006. A method for English–Korean target word selection using multiple knowledge sources. IEICE Trans. Fundamentals
E89-A (6), 1622–1629.
Lee, I., Park, J., Kim, C., Kim, Y., Kim, S., 2009. An Overview of Korean–English speech-to-speech translation system. In: Proc. TCAST Workshop,
Suntec, Singapore, pp. 6–9.
Lewis, M.P., 2009. Ethnologue: Languages of the World. SIL International, Dallas, TX, USA. http://www.ethnologue.com/.
Li, H., Sim, K.C., Singh, V., Lye, K.M., 2009. Query-by-example spoken document retrieval – the star challenge 2008. In: Proc. APSIPA Annual
Summit and Conference, Sapporo, Japan, pp. 319–323.
Mai, L.C., Duc, D.N., 2006. Design of Vietnamese speech corpus and current status. In: Proc. ISCSLP, Singapore, pp. 748–758.
Marti, F., Ortega, P., Idiazabal, I., 2005. Words and Worlds: World Languages Review. Bilingual Education and Bilingualism, Clevedon, England.
Mixdorf, H., Nguyen, D., Mai, L., Hoang, H., Vu, K., 2004. Fujisaki Model based F0 contours in Vietnamese TTS. In: Proc. ICSLP, Jeju, Korea,
pp. 177–180.
Nakamura, S., Sumita, E., Shimizu, T., Sakti, S., Sakai, S., Zhang, J., Finch, A., Kimura, N., Ashikari, Y., 2007. A-STAR: Asia speech translation
consortium. In: Proc. ASJ Autumn Meeting, Yamanashi, Japan, pp. 45–46.
Och, F.J., 2003. Minimum error rate training in statistical machine translation. In: Proc. ACL, Sapporo, Japan, pp. 160–167.
Oh, S., Kim, S., 2006. Modality-based sentence-final intonation prediction for Korean conversational-style text-to-speech systems. ETRI Journal
28 (6), 807–810.
Onishi, T., Utiyama, M., Sumita, E., 2010. Paraphrase lattice for statistical machine translation. In: Proc. ACL, Conference Short Papers, Uppsala,Sweden, pp. 1–5.
Papineni, K., Roukos, S., Ward, T., Zhu, W., 2002. BLEU: a method for automatic evaluation of machine translation. In: Proc. ACL, Philadelphia,
USA, pp. 311–318.
Paul, M., Okuma, H., Yamamoto, H., Sumita, E., Matsuda, S., Shimizu, T., Nakamura, S., 2008. Multilingual mobile-phone translation services forworld travelers. In: Proc. COLING, Companion Volume, Manchester, UK, pp. 165–168.
Paul, M., Arora, K., Sumita, E., 2009. Translation of untranslatable words – integration of lexical approximation and phrase-table extension techniques
into statistical machine translation. IEICE Transactions on Information and Systems E92-D (12), 2378–2385.
Paul, M., Federico, M., Stuecker, S., 2010. Overview of the IWSLT 2010 evaluation campaign. In: Proc. IWSLT, Paris, France, pp. 3–27.
Riza, H., Riandi, O., 2008. Toward Asian speech translation system: developing speech recognition and machine translation for Indonesian language.
In: Proc. TCAST Workshop, Hyderabad, India, pp. 30–35.
Riza, H., 2008. Indigenous languages of Indonesia: creating language resources for language preservation. In: Proc. IJCNLP Workshop on NLP for
Less Privileged Languages, Hyderabad, India.
Rugchatjaroen, A., Thangthai, A., Saychum, S., Thatphithakkul, N., Wutiwiwatchai, C., 2007. Prosody based naturalness improvement in Thai
unit-selection speech synthesis. In: Proc. ECTI-CON, Thailand.
Sakti, S., Hutagaol, P., Arman, A., Nakamura, S., 2004. Indonesian speech recognition for hearing and speaking impaired people. In: Proc. ICSLP,
Jeju, Korea, pp. 1037–1040.

S. Sakti et al. / Computer Speech and Language 27 (2013) 509–527

527

Sakti, S., Kelana, E., Riza, H., Sakai, S., Markov, K., Nakamura, S., 2008a. Development of Indonesian large vocabulary continuous speech
recognition system within A-STAR project. In: Proc. Workshop on Technologies and Corpora for Asia-Pacific Speech Translation, Hyderabad,
India, pp. 19–24.
Sakti, S., Maia, R., Sakai, S., Shimizu, T., Nakamura, S., 2008b. Development of HMM-based Indonesian speech synthesis. In: Proc. Oriental
COCOSDA Workshop, Kyoto, Japan, pp. 215–220.
Sakti, S., Vu, T., Finch, A., Paul, M., Maia, R., Sakai, S., Hayashi, T., Kimura, N., Ashikari, Y., Sumita, E., Nakamura, S., 2009. NICT/ATR Asianspoken language translation system for multi-party travel conversation. In: Proc. TCAST Workshop, Suntec, Singapore, pp. 26–30.
Stolcke, A., 2002. SRILM – an extensible language modeling toolkit. In: Proc. ICSLP, Denver, USA, pp. 901–904.
Thangthai, A., Thatphithakkul, N., Wutiwiwatchai, C., Rugchatjaroen, A., Saychum, S., 2008. T-tilt: a modified tilt model for F0 analysis andsynthesis in tonal languages. In: Proc. INTERSPEECH, Brisbane, Australia, pp. 2270–2273.
Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T., Kitamura, T., 2000. Speech parameter generation algorithms for HMM-based speech synthesis.
In: Proc. ICASSP, Istanbul, Turkey, pp. 1315–1318.
Vu, T., Dung, T., Mai, L., Hosom, J., 2005. Vietnamese Large Vocabulary Continuous Speech Recognition. In: Proc. INTERSPEECH, Lisbon,
Portugal, pp. 1689–1692.
Vu, T., Tang, K., Le, S., Mai, L., 2008. Vietnamese tone recognition based on multi-layer perceptron network. In: Proc. Oriental COCOSDA, Kyoto,
Japan, pp. 253–256.
Vu, T., Mai, L., Nakamura, S., 2009a. An HMM-based Vietnamese speech synthesis system. In: Proc. Oriental COCOSDA, Beijing, China.
Vu, T., Nguyen, K., Ha, L., Luong, M., Nakamura, S., 2009b. Toward Asian speech translation: the development of speech and text corpora for
Vietnamese language. In: Proc. TCAST Workshop, Suntec, Singapore, pp. 15–20.
Wutiwiwatchai, C., Supnithi, T., Porkaew, P., Thatphithakkul, N., 2009. Improvement issues in English–Thai speech translation. In: Proc. TCAST
Workshop, Suntec, Singapore, pp. 10–14.
Zhang, J., Hu, X., Nakamura, S., 2008a. Information criterion to design an efficient phoneme set for Chinese speech recognition. IEICE Trans. Inf.
& Syst., 508–513 (3).
Zhang, J., Jitsuhiro, T., Yamamoto, H., Hu, X., Nakamura, S., 2008b. An introduction to the Chinese speech recognition front-end of the NICT/ATR
multi-lingual speech translation system. Tsinghua Science and Technology 13 (4), 545–552.