Research Activities
for Translating
Asian Languages
Eiichiro SUMITA
Associate Director General, Universal Communication Research Institute,NICT, Japan

Coling 2016
PERIOD: 11-16 Dec. 2016, OSAKA, JAPANVENUE: OSAKA INTERNATIONAL CONVENTION CENTER (OICC)
GENERAL CHAIR: Dr. Nicoletta Calzolari (ILC CNR)
Many hotels,
including one
high-quality hotel
connected to the
OICC venue.

Japanese food: 12 restaurants
with three stars
 52 with twostars
 213 with onestar

Tons of places to visit: Traditional puppetry ■ Tunnel to Shrine
 Todaiji Temple■ Pop cultureSome important infoHomepage:
http://coling2016.anlp.jp/
Deadlines:
15 August 2016: Paper submission
30 September 2016: Notification
15 October 2016: Camera-readyversions due
10/17/2015© NICT3

Outline
1. Worldwide speech translation consortium,
Universal Speech Translation Advanced
Research (U-STAR)
2. Workshop on Asian Translation (WAT) & Asian
Language Treebank (ALT)
3. Global Communication Program (GCP) in Japan
4. Recent research topics in the National Instituteof Information Technology (NICT), Japan
5. Towards increasing collaboration

2015/10/17© NICT4

1. U-STAR (Universal SpeechTranslation AdvancedResearch) Consortium

2015/10/17© NICT5Configuration of Speech Translation

ASRConverts speech
into text

10/17/2015

駅はどこですかMTConvertsJapanese text
into English© NICT

Whereisthestation

SSConverts text
into speech

6The world’s first experiment:

1992

10/17/2015© NICT

7

Speech Translation System on a Smartphone

VoiceTra @ 2010.7

Part 1: How to use
Part 2: Don’ts
10/17/2015© NICT8translation accuracy

VoiceTra for the tourism domain
demonstrates high-quality translation
among many languages
100
90
80
70
60
50
4030
20
100

Black: VoiceTra by NICT

Gray: Other famous translator
99199 1≈

C-STAR

1.
2.
3.

200 5

2006

2007

2008

2009

A-STAR2010 2 0 1 1

2012

2013

2014

2015

2016

U-STAR

C-STAR started out over 24 years
ago with 3 organizations.
Then, A-STAR extended the activity
in Asia.
Now, with 32 institutes from 27
countries/regions, U-STAR has
grown into one of the largest
consortia in the world that conducts
research on speech-to-speechtranslation.
10

32 institutes

11

VoiceTra4U

Smartphone applications
for iOS and Android

Translates up to 30 languages including dialects

Speech Input（Available in 17languages)

Text Input / Output(Available in 30 languages)

Speech Output(Available in 14languages)

Dutch
English (US)English (UK)
FrenchGermanHindi
HungarianIndonesian
Japanese
KoreanMandarin
Malay
PolishPortuguese
Thai
TurkishVietnamese

ArabicBr-Portuguese
DanishDutch
Dzongkha
English (US)English (UK)
Filipino
FrenchGermanHindi
HungarianIndonesian
ItalianJapanese
KoreanEnglish (US)
Hindi
HungarianIndonesian
Japanese
KoreanMalay
Mandarin
Mongolian
PolishPortuguese
Thai
TurkishVietnamese

Malay
Mandarin
Mongolian
Nepali
PolishPortuguese
RussianSinhalaSpanishTw-Mandarin
Thai
TurkishUrdu
Vietnamese

12Achievements
Research collaborations among U-STAR Members
Baseline
AM adaptation（unsupervised）
AM (unsupervised)/LM adaptation
AM （supervised）/LM adaptation

Baseline
Improvements on AM using speech log data
Usage of feature value based on DNN
60

IMPROVED
21%60

WER (Word Error Rate) (%)

WER (Word Error Rate) (%)

7050403050

IMPROVED
27%4030
2020

Improved about 30% in WER (Vietnamese)

Improved about 20% in WER (Thai)（Evaluated with 803 sentences of VoiceTra4U log）

（Evaluated with 2,765 sentences of VoiceTra4U log）

3 months from July 2013: 3 researchers from NECTEC (Thailand) visited NICT to work on Thai ASR.
1 month from August 2014：one research student from UULM (Germany) visited NICT to work on Russian ASR.
2 months from September 2014：one research student from IOIT (Vietnam) visited NICT to work on Vietnamese ASR
1 year from August 2014：one researcher and two students from UCSY (Myanmar) visited NICT to work on Myanmar ASR

Baseline

LM adaptation

AM adaptation

WER (Word Error Rate) (%)

4035

IMPROVED
17%302520

Improved about 30% in WER (Russian)(Evaluated with read speech from 1 speaker)

Myanmar:ဘူတာရုံ အဘယ်မာရှိသနည််း

Japanese:「駅はどこですか？」13

Please join us @http://www.ustarconsortium.com/contactinfo.html
14

2.1 Workshop on
Asian Translation(WAT)
http://orchid.kuee.kyoto-u.ac.jp/WAT/
2015/10/17© NICT15

1st WAT (Workshop on Asian translation )• http://orchid.kuee.kyotou.ac.jp/WAT/WAT2014/index.html• Held in Tokyo in October 2014
• Had over 50 participants• Machine Translation evaluation campaign focusing on
scientific documents in Japanese-English/JapaneseChinese
2015/10/17© NICT16

2nd WAT• http://orchid.kuee.kyoto-u.ac.jp/WAT/• Will be held in Kyoto on 16 October 2015
• New Task: translation of patentdocument with bilingual JapaneseChinese/Korean corpora (1 M) provided
by Japan Patent Office (JPO)
2015/10/17© NICT17

3rd WAT

Pleasejoin us.
The third one will add new language pairs.

2015/10/17© NICT18

2.2 Asian Language
Treebank (ALT)
2015/10/17© NICT19

1. Treebank accelerates research on NLP for eachlanguage• No publicly available POS-tagged andconstituency tree corpora for many Asianlanguages.
2. Parallel corpus accelerates research on
Machine Translation between thelanguages• No big parallel corpora among many
Asian languages
2015/10/17© NICT20

In order to solvethese problems,
let’s developtreebanks ofparallel corpora
together!

2015/10/17© NICT21

4 steps for developing treebanks
1.

2.
3.

4.

2015/10/17

Translating the common English sentence
into the Asian language
Aligning words between the sentencepairs of English and the Asian language
Tagging the POS (part of speech) for thesentence in the Asian language
Building the tree of the sentence in the
Asian language© NICT22

WEB-based UI for ALT’s 4 steps

2015/10/17© NICT23

Progress so far
Alignment

POS

Tree

completion

Under
construction
Under
construction
Under
construction

Under
construction
Under
construction
Under
construction

Under
construction
Under
construction
Under
construction

March
2016

Done

PlannedPlannedPlannedDone

PlannedPlannedPlannedLanguageTranslateEnglish(-)

Japanese

Done

MyanmarDone

IndonesianVietnamese

OthersUnder consideration

10/17/2015© NICTMarch2017?

24

3. GlobalCommunication
Program
10/17/2015© NICT25

The Global Communication Program (Japan)• The Global Communication Program(GCP) is a Japanese government project
announced on 11 April 2014 to develop amulti-lingual speech translation system tobridge the language barrier during theOlympic Games in 2020.
• http://www.soumu.go.jp/main_content/00
0285578.pdf (in Japanese)
2015/07/1

4

TargetReal-time machine translation services– covers 10 languages, including Asian ones
such as Thai, Vietnamese, Indonesian andMyanmar– using National Institute of Information andCommunications Technology’s translation
technology

10/17/2015© NICT27

NICT has established a new research center(16 Sept. 2014)2015/07/14

Future vision of Japanese society in 2020
①Apakah anda
memiliki warna lain?

①お客様の鞄は、品川駅で預かっています。

③お調べします。赤色があります。

②色違いはありますか。
④Mari saya periksa. Ada merah.

①我的肚子疼
②おなかが痛いIndonesianEnglishShopping

Railway②I keep your bag atShinagawa station.

HospitalHotel②眺めのよい部屋に変えてもらえますか

ChineseVietnamese③所服用的药吃完了④いつも飲んでいる薬がなくなってしまいました

2015/07/14①Anh có thể đổi cho tôi một phòng
với khung cảnh.

8 Reasons Why the Tokyo Olympics Will Be theMost Futuristic We've Ever Seen - GIZMODO
http://gizmodo.com/8-reasons-why-the-tokyo-olympics-will-be-the-most-futur-1728007440

Meanwhile, in the private sector, Panasonic is making apalm-sized gadget worn around the neck that willtranslate Japanese into 10 languages for the thousandsof visitors set to descend on the metropolis. The
electronics giant also plans to provide visitors with asmartphone app that scans Japanese signs andtranslates them on the spot. These are services that
could be useful in countries across the globe.
https://www.youtube.com/watch?v=FluhQAXBX6E

10/17/2015© NICT30

4. Recent Research
Topics at NICT

2015/10/17© NICT31

Translation of Patent Claim
Masaru Fuji, Atsushi Fujita, Masao Utiyama, Eiichiro Sumita, Yuji Matsumoto.
Patent Claim Translation Based on Sublanguage-specific Sentence Structure. In
Proceedings of Machine Translation Summit XV (MT Summit 2015), Miami, Florida,USA, October 30-November 3, 2015.

ClaimSplit source into
patterns

Convert totarget patterns

Translatecomponents bySMT withpreordering

32

EJ

JE

baseline

23.7

22.3

proposal

28.8

27.5

Big Gain in BLEU

The actuator according to claim 1, wherein an even number of notches are formed insaid body, and the displacement of said rod in the axial direction is extracted.

PREA

the actuator according to claim 1

TRAPwherein

PURP

an even number of notches are formed in said body, and the displacementof said rod in the axial direction is extracted

PURP

an even number of notches are formed in said body, and the displacementof said rod in the axial direction is extracted

TRAP

whereinPREA

the actuator according to claim 1

PURP

偶数個の切込みが形成されている前記本体であり、前記ロッドの変位には、軸方向 を抽出するTRAPことを特徴とする

PREA

請求項１に記載のアクチュエータ

Binarized Neural Network Joint Model
Jingyi Zhang, Masao Utiyama, Eiichiro Sumita, Graham Neubig and Satoshi
Nakamura (2015). A Binarized Neural Network Joint Model for Machine Translation.
EMNLP.

Binarized Neural Network Joint Model

Neural Network Joint Model

The binarized model has achieved comparable performance and faster decoding/learning
10/17/2015© NICT33

5. Toward More and More
Collaboration
CDAC

HIT

UCSY

NIPTICT

MORE
NICT

IOIT

BPPH

NECTEC
SJTU

CASIA
10/17/2015© NICT34

Collaboration between Myanmar and NICT(1/2)• Three researchers from UCSY studied S2Stechnologies from July 2014 to June 2015 at NICT.

– They have developed the world‘s first Myanmar
Speech Recognition system and Speech Synthesissystem.
– They are also working on the development ofMachine Translation technology between
Japanese, the Myanmar language and English.

Collaboration between Myanmar and NICT (2/2)• My-En (Translation results of NICT and G***** MT)

G*****

G*****

G*****

Demonstration of speech translation fromEnglish/Japanese to MyanmarSpeech
recognition result
Reversetranslation forconfirmation

Translation result• Developed incollaboration with 3
researchers fromMyanmar.
• Applicable for travel
conversations suchas: ‘Welcome’, ‘How
about this one?’,‘What is the price?’,‘Thank you’, ‘Have asafe trip’.