15th International Workshop
on Spoken Language Translation

Bruges, Oct. 29 - 30, 2018www.iwslt.org

Proceedings

Proceedings of theInternational Workshop on
Spoken Language Translation
October 29-30, 2018
Bruges, Belgium

Edited by
Marco Turchi
Jan Niehues
Marcello Frederico

Contents
Contenti

Forewordiii

Organizersv

Acknowledgments

vii

ParticipantsviiiProgramix

Keynotes
Multimodal Machine Translation . . . . . . . . . . . . . . . . . . . . . . .

xiii
xiii

Evaluation Campaign
The IWSLT 2018 Evaluation Campaign . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Jan Niehues, Ronaldo Cattoni, Sebastian Stüker, Mauro Cettolo, Marco Turchi and Marcello Federico
Unsupervised Parallel Sentence Extraction from Comparable Corpora . . . . . . . . . . . . . . .
Viktor Hangya, Fabienne Braune, Yuliya Kalasouskaya and Alexander Fraser
Word Rewarding for Adequate Neural Machine Translation . . . . . . . . . . . . . . . . . . . .
Yuto Takebayashi, Chenhui Chu, Yuki Arase and Masaaki Nagata
Word Rewarding for Adequate Neural Machine Translation . . . . . . . . . . . . . . . . . . . .
Dakun Zhang, Josep Crego and Jean Senellart
Word-based Domain Adaptation for Neural Machine Translation . . . . . . . . . . . . . . . . .
Shen Yan, Leonard Dahlmann, Pavel Petrushkov, Sanjika Hewavitharana and Shahram Khadivi
A Machine Translation Approach for Modernizing Historical Documents Using Backtranslation .
Miguel Domingo and Francisco Casacuberta
Multi-Source Neural Machine Translation with Data Augmentation . . . . . . . . . . . . . . . .
Yuta Nishimura, Katsuhito Sudoh, Graham Neubig and Satoshi Nakamura
Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary . . . . .
Marco Turchi, Mattia Antonino Di Gangi and Nicholas Ruiz
Machine Translation Human Evaluation: an investigation of evaluation based on Post-Editing and
its relation with Direct Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Luisa Bentivogli, Mauro Cettolo, Marcello Federico and Christian Federmann
The USTC-NEL Speech Translation system at IWSLT 2018 . . . . . . . . . . . . . . . . . . . .
Dan Liu, Junhua Liu, Wu Guo, Shifu Xiong, Zhiqiang Ma, Rui Song, Chongliang Wu and Quan
Liu
The ADAPT System Description for the IWSLT 2018 Basque to English Translation Task . . . . .
Alberto Poncelas, Kepa Sarasola and Andy Way
The University of Helsinki submissions to the IWSLT 2018 low-resource translation task . . . . .
Yves Scherrer
The MeMAD Submission to the IWSLT 2018 Speech Translation Task information on submission
30 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2
2

Lucia Specia

i

7
14
23
31
39
48
54
62
70
76
83
89

Umut Sulubacak, Aku Rouhe, Jörg Tiedemann, Stig-Arne Grönroos and Mikko Kurimo
Prompsit’s Submission to the IWSLT 2018 Low Resource Machine Translation Task . . . . . . . 95
Víctor M. Sánchez-Cartagena
Neural Speech Translation at AppTek . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Evgeny Matusov, Patrick Wilken, Parnia Bahar, Julian Schamper, Pavel Golik, Albert Zeyer, Joan
Albert Silvestre-Cerda, Adria Martinez-Villaronga, Hendrik Pesch and Jan-Thorsten Peter
The Sogou-TIIC Speech Translation System for IWSLT 2018 . . . . . . . . . . . . . . . . . . . 112
Yuguang Wang, Liangliang Shi, Linyu Wei, Weifeng Zhu, Jinkun Chen, Zhichao Wang, Shixue
Wen, Wei Chen, Yanfeng Wang and Jia Jia
Samsung and University of Edinburgh’s System for the IWSLT 2018 Low Resource MT Task . . . 118
Philip Williams, Marcin Chochowski, Pawel Przybysz, Rico Sennrich, Barry Haddow and Alexandra Birch
The AFRL IWSLT 2018 Systems: What Worked, What Didn’t . . . . . . . . . . . . . . . . . . . 124
Brian Ore, Eric Hansen, Katherine Young, Grant Erdmann and Jeremy Gwinnup
KIT’s IWSLT 2018 SLT Translation System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Matthias Sperber, Ngoc Quan Pham, Thai Son Nguyen, Jan Niehues, Markus Müller, Thanh-Le
Ha, Sebastian Stüker and Alex Waibel
Alibaba Speech Translation Systems for IWSLT 2018 . . . . . . . . . . . . . . . . . . . . . . . 136
Fei Huang, Nguyen Bach and Chongjia Ni
CUNI Basque to English Submission in IWSLT18 . . . . . . . . . . . . . . . . . . . . . . . . . 142
Tom Kocmi, Dušan Variš and Ondřej Bojar
Fine-tuning on Clean Data for End-to-End Speech Translation: FBK @ IWSLT 2018 . . . . . . . 147
Mattia Antonino Di Gangi, Roberto Dessì, Roldano Cattoni, Matteo Negri and Marco Turchi
The JHU/KyotoU Speech Translation System for IWSLT 2018 . . . . . . . . . . . . . . . . . . 153
Hirofumi Inaguma, Xuan Zhang, Zhiqi Wang, Adithya Renduchintala, Shinji Watanabe and Kevin
Duh
Adapting Multilingual NMT to Extremely Low Resource Languages, FBK’s Participation of theBasque-English Low Resource MT, IWSLT 2018 . . . . . . . . . . . . . . . . . . . . . . 160
Surafel M. Lakew and Marcello Federico
Learning to Segment Inputs for NMT Favors Character-Level Processing . . . . . . . . . . . . . 166
Julia Kreutzer and Artem Sokolov.
Data Selection with Feature Decay Algorithms Using an Approximated Target Side . . . . . . . . 173
Alberto Poncelas, Gideon Maillette de Buy Wenniger and Andy Way
Multi-paraphrase Augmentation to Leverage Neural Caption Translation . . . . . . . . . . . . . 181
Johanes Effendi, Sakriani Sakti, Katsuhito Sudoh and Satoshi Nakamura
Using Spoken Word Posterior Features in Neural Machine Translation . . . . . . . . . . . . . . . 189
Kaho Osamura, Takatomo Kano, Sakriani Sakti, Katsuhito Sudoh and Satoshi Nakamura
Author Index

196

ii
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Foreword
The International Workshop on Spoken Language Translation (IWSLT) is an annuallyscientific workshop, associated with an open evaluation campaign on Spoken Language
Translation, where both scientific papers and system descriptions are presented.
Since 2004, the annual workshop has been held six times in Asia, five times in
America, and three times in Europe. This year, the 15th International Workshop on
Spoken Language Translation is hosted Europe, in awesome Bruges, Belgium from29th to 30th October 2018. The reason was to co-locate IWSLT with other two important and related conferences: the Conference on Machine Translation (WMT) and theConference on Empirical Methods for Natural Language Processing (EMNLP), bothtaking place in Brussels, after IWSLT.
The IWSLT workshop includes presentations of scientific papers in dedicated technical sessions, either in oral or poster form. Scientific papers cover theoretical andpractical issues in the field of machine translation, spoken language translation, automatic speech recognition, text-to-speech synthesis. This year, we received 26 submissions of scientific papers, which were carefully peer-reviewed by members of the
program committee. Out of them, 12 were selected for publication based on their technical merit and relevance to the workshop. The proceedings of IWSLT are publishedon the workshop website.
As in the past editions, IWSLT will assign a best student paper award for which
this year we have selected two finalists that will give an oral presentation of their work.
The workshop will also include the presentation and discussion of the outcomesof the 2018 IWSLT evaluation campaign, which this year focused on two tasks: low
resource machine translation of TED Talks from Basque to English, and speech translation of lectures from English to German.
The low resource translation task addressed a conventional bilingual text translationtask, but given the difficulty of the proposed translation direction and the scarcity ofavailable parallel data, we supported the development of deep learning models also
leveraging parallel data from related languages.
The speech translation task also introduced several novel aspects with respect tothe past, which pushed participants to go beyond the usual pipeline approach of cascading automatic speech recognition and machine translation. First, this year we only
evaluated end-to-end performance, so that every participant had to generate Germantranslations based on the English audio. Second, for participants who wanted to focus on one component of the pipeline, we provided baseline components for the otherparts. Third, we introduced a special evaluation condition to test end-to-end models,
namely deep learning models directly mapping source language speech into target language text. For this condition, we provided an aligned TED corpus of English audio
and German texts.
For each proposed task, monolingual and bilingual language resources, as needed,iii

were provided to participants in order to train their systems. Another novelty we introduced this year was to extend the evaluation period from one week to one month, sothat participants could better plan their effort. Blind test sets were released at the begin
of July, and all translation outputs produced by the participants were evaluated using
several automatic translation quality metrics. Finally, each participant was requested
to submit a paper describing his system and the utilized resources. System papers went
through a peer-review process only aiming at improving their overall quality.
The efforts of the organizers were definitely rewarded. The IWSLT evaluation campaign attracted this year 15 research teams, 8 taking part to the low-resource translationtask, and 9 to the speech translation task. The presentations of the corresponding 15
system description papers will be as usual preceded by an overview talk of the evaluation campaign given by one of the organizers.
Before opening the workshop a few words of acknowledgment.
I would like to express my gratitude to the evaluation committee, Jan Niehues, Mauro
Cettolo, Sebastian Stüker, Luisa Bentivogli, and Roldano Cattoni for preparing such agreat playground for the research community. To the program chair Marco Turchi andall the program committee members for arranging an excellent workshop program. To
Margit Rödder, for taking care of the website and all the local and financial arrangements.
Finally, I would like to thank our generous sponsors for supporting IWSLT. Our
gold sponsors Amazon Web Services and AppTek, and, our silver sponsor M*Modal.
This year sponsors will have the opportunity to give a short corporate presentation
at our workshop, which I’m sure will enrich IWSLT with an industry perspective on
spoken language translation and provide additional networking opportunities in ourcommunity.
I wish all the participants a fruitful and engaging IWSLT workshop!
Welcome to Bruges!
Marcello FedericoWorkshop Chair IWSLT 2018

iv
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Organizers
ChairsMarcello Federico (FBK, Italy /Amazon, USA): Workshop
Marco Turchi (FBK, Italy) : Program
Jan Niehues (KIT,Germany): Evaluation
Alex Waibel (CMU, USA/KIT, Germany): Steering Committee

Local and Financial Chair
Margit Röder (KIT, Germany)

Evaluation Technical Committee
Jan Niehues (KIT, Germany)
Mauro Cettolo (FBK, Italy)
Sebastian Stüker (KIT, Germany)
Luisa Bentivogli (FBK, Italy
Roldano Cattoni (FBK, Italy)
Marcello Federico (FBK, Italy/Amazon, USA)

Steering Committee
Marcello Federico, (FBK, Italy/Amazon, USA)
Masakiyo Fujimoto (NICT, Japan)
Will Lewis (Microsoft Research, USA)Chi Mai Luong (IOIT, Vietnam)
Joseph Mariani (IMMI, France)
Satoshi Nakamura (NAIST, Japan)
Hermann Ney (RWTH, Germany)
Sebastian Stüker (KIT, Germany)Alex Waibel (CMU, USA/KIT, Germany)
Francois Yvon (CNRS-LIMSI, France)

Program Committee
Marco Turchi, FBK (Italy), Program Chair
Duygu Ataman, FBK (Italy)
Loic Barrault, Université du Mans (France)
Laurent Besacier, LIG (France)v

Francisco Casacuberta, UPV (Spain)
José Guilherme Carmago de Souza, eBay Inc. (USA)Mauro Cettolo, Pervoice (Italy)
Boxing Chen, NRC-CNRC (Canada)
Mattia Di Gangi, FBK (Italy)
Orhan Firat, Google (USA)Mark Fishel, University of Tartu (Estonia)
George Foster, Google (Canada)
Paco Guzman, Facebook (USA)Barry Haddow, University of Edinburgh (UK)
Christian Hardmeier, University of Uppsala (UK)Mathias Huck, LMU (Germany)
Marcin Junczys-Dowmunt, Microsoft (USA)Kevin Kilgour, Google (Zürich)Philip Kohen, John Hopkins University (USA)Julia Kreutzer, Heildeberg University (Germany)
Roland Kuhn, NRC (Canada)
Yves Lepage, Waseda University (Japan)
Will Lewis, Microsoft (USA)Qun Liu, DCU (Ireland)
Evgeny Matusov, Apptek (Germany/USA)
Surafel Melaku Lakew, FBK (Italy)
Markus Müller, KIT(Germany)
Graham Neubig, CMU (USA)Michael Paul, ATR-Trek (Japan)
Stephan Peitz, Apple (USA)Maja Popović, DCU (Ireland)
Matt Post, Amazon Research
Elizabeth Salesky, CMU (USA)
Carolina Scarton, University of Sheffield (UK)
Lane Schwartz, U. Illinois (USA)Rico Senrich, University of Edinburgh (UK)
Matthias Sperber, KIT(Germany)
Felix Stahlberg, University of Cambridge (UK)
Ales Tamchyna, Charles University (Czech Republic)
Jörg Tiedemann, H. Helsinki (Finland)
Antonio Toral, University Groningen (The Netherlands)
Vincent Vandeghinste, KU Leuven (Belgium)
David Vilar, Amazon Research (Germany)
Andy Way, DCU (Ireland)
Marion Weller, Stuttgart University (Germany)
Philip Williams, University of Edinburgh (UK)vi
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Acknowledgments
IWSLT 2018 is proud to present its gold sponsor

and its silver sponsor

vii

Participants
USTC-NEL (C HINA)
ADAPT (I RELAND)
University of Helsinki (F INLAND)
MeMAD (F INLAND)
Prompsit (S PAIN)
AppTek (G ERMANY)
Sogou-TIIC (C HINA)SRPOL-UEDIN (UK/P OLAND)AFRL (USA)KIT (GERMANY)
Alibaba (C HINA)CUNI (C ZECH)FBK (I TALY)
JHU/KyotoU (USA/JAPAN)
SGNLP (KOREA)

Baseline
4

SLT
End-to-End
4

Low-resourced MT

4
4
4
4
4
4
4
4
4

4
4
4

4
4
4

viii

4
4

Program
Monday, October 29th, 201808:30-09:00

Workshop Registration

09:00-09:15

Welcome Remarks
Marcello Federico (Workshop Chair)
Amazon-FBK, USA, Italy
Invited talk: Multimodal Machine Translation
Prof. Lucia Specia
University of Sheffield, UK
I NDUSTRIAL S ESSION
Gold sponsor presentation: AppTek

09:15-10:05
10:05-11:05
10:05-10:20

Coffee Break (10:20-10:50)
10:50-11:05 Silver sponsor presentation: M*Modal
E VALUATION C AMPAIGN
Report on the 15th IWSLT Evaluation Campaign, IWSLT 2018
Jan Niehues, Roldano Cattoni, Sebastian Stüker, Mauro Cettolo, Marco Turchi,
Marcello Federico
KIT & FBK, Germany, Italy
Lunch (12:00-13:30)
13:30-15:00
O RAL S ESSION I
13:30-14:00 Samsung and University of Edinburgh’s System for the IWSLT 2018 Low
Resource MT Task
Philip Williams, Marcin Chochowski, Pawel Przybysz, Rico Sennrich, Barry
Haddow and Alexandra Birch
UEDIN & Samsung R&D Institute, UK, Poland
14:00-14:30 The USTC-NEL Speech Translation system at IWSLT 2018
Dan Liu, Junhua Liu, Wu Guo, Shifu Xiong, Zhiqiang Ma, Rui Song,
Chongliang Wu and Quan Liu
USTC & IFLYTEK Co. LTD, China
11:00-12:00
11:05-12:00

ix

Unsupervised Parallel Sentence Extraction from Comparable Corpora
Viktor Hangya, Fabienne Braune, Yuliya Kalasouskaya and Alexander Fraser
LMU & Volkswagen Data Lab Munich, Germany
Coffee Break (15:00-15:30)
15:30-17:30
P OSTER S ESSION
The JHU/KyotoU Speech Translation System for IWSLT 2018
Hirofumi Inaguma, Xuan Zhang, Zhiqi Wang, Adithya Renduchintala, Shinji
Watanabe and Kevin Duh
JHU & KyotoU, USA, Japan
The ADAPT System Description for the IWSLT 2018 Basque to English
Translation Task
Alberto Poncelas, Kepa Sarasola and Andy Way
DCU, Ireland
The University of Helsinki submissions to the IWSLT 2018 low-resource
translation task
Yves Scherrer
HelsinkiU, Finland
Prompsit’s Submission to IWSLT 2018 Low Resource Machine Translation Task
Víctor M. Sánchez-Cartagena
Prompsit, Spain
The AFRL IWSLT 2018 Systems: What Worked, What Didn’t
Brian Ore, Eric Hansen, Katherine Young, Grant Erdmann and Jeremy Gwinnup
AFRL, USA
Alibaba Speech Translation Systems for IWSLT 2018
Fei Huang, Nguyen Bach and Chongjia Ni
Alibaba, China
The MeMAD Submission to the IWSLT 2018 Speech Translation Task
Umut Sulubacak, Aku Rouhe, Jörg Tiedemann, Stig-Arne Grönroos and Mikko
KurimoHelsinkiU & AaltoU, Finland
Neural Speech Translation at AppTek
Evgeny Matusov, Patrick Wilken, Parnia Bahar, Julian Schamper, Pavel Golik,
Albert Zeyer, Joan Albert Silvestre-Cerda, Adria Martinez-Villaronga, Hendrik
Pesch and Jan-Thorsten Peter
AppTek, Germany
KIT’s IWSLT 2018 SLT Translation System
Matthias Sperber, Ngoc Quan Pham, Thai Son Nguyen, Jan Niehues, Markus
Müller, Thanh-Le Ha, Sebastian Stüker and Alex Waibel
KIT, Germany
CUNI Basque to English Submission in IWSLT18
Tom Kocmi, Dušan Variš and Ondřej Bojar
CUNI, Czech Republic
Adapting Multilingual NMT to Extremely Low Resource Languages,
FBK’s Participation of the Basque-English Low Resource MT, IWSLT
2018
Surafel M. Lakew and Marcello Federico
FBK, Italy
14:30-15:00

x
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

18:30-19:30
19:30-The Sogou-TIIC Speech Translation System for IWSLT 2018
Yuguang Wang, Liangliang Shi, Linyu Wei, Weifeng Zhu, Jinkun Chen, Zhichao
Wang, Shixue Wen, Wei Chen, Yanfeng Wang and Jia Jia
Sogou & TIIC, China
Fine-tuning on Clean Data for End-to-End Speech Translation: FBK @IWSLT 2018
Mattia Antonino Di Gangi, Roberto Dessì, Roldano Cattoni, Matteo Negri andMarco Turchi
FBK, Italy
Using Spoken Word Posterior Features in Neural Machine Translation
Kaho Osamura, Takatomo Kano, Sakriani Sakti, Katsuhito Sudoh and Satoshi
Nakamura
NIST & RIKEN, Japan
Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary
Surafel Melaku Lakew, Aliia Erofeeva, Matteo Negri, Marcello Federico andMarco Turchi
FBK & UniTN, Italy
Data Selection with Feature Decay Algorithms Using an ApproximatedTarget Side
Alberto Poncelas, Gideon Maillette de Buy Wenniger and Andy Way
DCU, Ireland
G UIDED TOUR THROUGH B RUGESD INNER IN THE B REWERY “D E H ALVE M AAN ”

xi
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Tuesday, October 30th, 2018
09:00-10:00
09:00-09:30
09:30-10:00
10:00-10:15
10:00-10:15

O RAL SESSION II
Multi-Source Neural Machine Translation with Data Augmentation
Yuta Nishimura, Katsuhito Sudoh, Graham Neubig and Satoshi Nakamura
NIST & CMU, Japan, USA
Word Rewarding for Adequate Neural Machine Translation
Yuto Takebayashi, Chenhui Chu, Yuki Arase and Masaaki Nagata
OsakaU & NTT Corporation, JapanI NDUSTRIAL S ESSION
Gold sponsor presentation: Amazon

Coffee Break (10:15-10:45)
10:45-12:15
O RAL SESSION III
10:45-11:15 Analyzing Knowledge Distillation in Neural Machine Translation
Dakun Zhang, Josep Crego and Jean Senellart
Systran, France
11:15-11:45 Learning to Segment Inputs for NMT Shows Preference for CharacterLevel Processing
Julia Kreutzer and Artem Sokolov
HeidelbergU & Amazon, Germany
11:45-12:15 Machine Translation Human Evaluation: a comprehensive investigationof evaluation based on Post-Editing and its relation with Direct Assessment
Luisa Bentivogli, Mauro Cettolo, Marcello Federico and Christian Federmann
FBK & Amazon & Microsoft, Italy, USA
Lunch (12:15-13:30)
13:30-15:00
O RAL SESSION IV
13:30-14:00 Multi-paraphrase Augmentation to Leverage Neural Caption Translation
Johanes Effendi, Sakriani Sakti, Katsuhito Sudoh and Satoshi Nakamura
NIST & RIKEN, Japan
14:00-14:30 A Machine Translation Approach for Modernizing Historical Documents
Using Back Translation
Miguel Domingo and Francisco Casacuberta
UPV, Spain
14:30-15:00 Word-based Domain Adaptation for Neural Machine Translation
Shen Yan, Shahram Khadivi, Leonard Dahlmann, Pavel Petrushkov and Sanjika Hewavitharana
eBay Inc, USA
Coffee Break (15:00-15:30)
15:30-16:45
PANEL S ECTION
14:00-15:00 Panel: The future of machine and speech translation
Panelists: Evgeny Matusov (AppTek), Will Lewis (Microsoft), Kevin Duh(JHU), Marcin Junczys-Dowmunt (Microsoft) and Lucia Specia (University of
Sheffield)
Moderator: Sebastian Stüker (KIT)
16:45-17:00
C LOSING REMARKS + ANNOUNCEMENTS

xii
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Keynotes
Multimodal Machine Translation

Lucia Specia, Imperial College London/Sheffield University

Abstract
Humans interact with the world through multiple modalities (hearing, vision, etc.).
This is also true when understanding and generating language. Computational models for language processing, however, are traditionally limited to exploring language
only (spoken or written). In this talk I will cover recent work in the area of multimodal machine learning for machine translation, where vision is used as additional
modality, and where the goal is to achieve structured language grounding. I will also
provide an overview on approaches that explore multimodality for language grounding in other sequence to sequence models: automatic speech recognition and spoken
language translation.

xiii

Evaluation Campaign

1The IWSLT 2018 Evaluation Campaign
J. Niehues(1)R. Cattoni(2)S. Stüker(1)M. Cettolo(2)M. Turchi(2)M. Federico(3)†KIT - Adenauerring 2, 76131 Karlsruhe, Germany
FBK - Via Sommarive 18, 38123 Trento, Italy(3)
Amazon AI - East Palo Alto, CA 94303, USA

(1)

(2)

Abstract

lower amount of rephrasing but a higher pronoun density [3].
On the other side, speech translation [4] requires the integration of MT with automatic speech recognition, which brings
with it the additional difficulty of translating content that may
have been corrupted by speech recognition errors.
Along the years, three main evaluation tracks were progressively introduced, addressing all the core technologies
involved in the spoken language translation task, namely:

The International Workshop of Spoken Language Translation (IWSLT) 2018 Evaluation Campaign featured two tasks:
low-resource machine translation and speech translation. In
the first task, manually transcribed speech had to be translated from Basque to English. Since this translation direction
is a under-resourced language pair, participants were encouraged to use additional parallel data from related languages.
In the second task, participants had to translate English audio into German text with a full speech-translation system.
In the baseline condition, participants were free to use composite architectures, while in the end-to-end condition they
were restricted to use a single model for the task.
This year, eight research groups took part in the lowresource machine translation task and nine in the speechtranslation task.

• Automatic speech recognition (ASR), i.e. the conversion of a speech signal into a transcript• Machine translation (MT), i.e. the translation of a polished transcript into another language• Spoken language translation (SLT), i.e. the conversion
and translation of a speech signal into a transcript in
another language

1. Introduction

In previous years, the ASR transcript was provided to theparticipants of the SLT task. Therefore, the SLT task main focus on investigating translation methods for automatic transcripts.
The recent development in deep learning lead to the usage of similar techniques in machine translation and automatic speech recognition. Furthermore, the success ofsequence-to-sequence model allowed the development of
end-to-end speech translation systems [5]. Therefore, in thisyears edition, we drooped the ASR task and included thetranscription of the audio into the SLT task. Hence, forthe first time, participants needed to develop a full speechtranslation pipeline and were supplied with audio-text parallel data.
The 2018 IWSLT evaluation focused on translating talks
from two sources of data: translation of TED talks corpus[6] and, for the speech translation task, university lecturescollected at KIT [7].
The TED translation task of IWSLT has become a seasoned task by now. Its introduction was motivated by its
higher complexity with respect to the previous travel tasks,
and by the availability of high quality data. In order to keep
the tasks interesting and to follow current trends in researchand industry, we expanded and developed the IWSLT tasks
further. Motivated by last years success of the multi-lingualmachine translation task, we created task on a low resources

We report here on the outcomes of the 2018 evaluation
campaign organized by the International Workshop of Spoken Language Translation (IWSLT). The IWSLT workshop
started in 2004 [1] with the purpose of enabling the exchange
of knowledge among researchers working on speech translation and creating an opportunity to develop and comparetranslation systems on a common test bed. The evaluation
campaign built on one of the outcomes of the C-STAR (Consortium for Speech Translation Advanced Research) project,
namely the BTEC (Basic Travel Expression Corpus) multilingual spoken language corpus [2], which initially served asa primary source of evaluation. Since its beginning, translation tasks of increasing difficulty were offered and new data
sets covering a large number of language pairs were sharedwith the research community. In the fifteenth editions organized from 2004 to 2018, the campaign attracted around 70
different participating teams from all over the world.
Automatic spoken language translation is particularly
challenging for a number of reasons. On one side, machinetranslation (MT) systems are required to deal with the specific features of spoken language. With respect to written
language, speech is structurally less complex, formal andfluent. It is also characterized by shorter sentences with a† Work performed while the author was at FBK, Italy.

2
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018language pair. Furthermore, we developed the speech translation task further. Participants need to build a complete
speech translation system and we encourage research on endto-end models. Unlike in previous years, we also limited thescope of the evaluation to very few languages. The main
reason for this was to avoid dispersion of participants in too
many tasks.
The translation directions considered this year were English to German for SLT task were English to German andBasque to English for low-resource MT task.
For all tasks, permissible training data sets were specified
and instructions for the submissions of test runs were given
together with the detailed evaluation schedule.
All runs submitted by participants were evaluated with
automatic metrics. In particular, for the low-resource MTtask, an evaluation server was set up so that participants
could autonomously score their runs on different dev and test
sets. This year, 15 groups participated in the evaluation (seeTable 1). In the following, we describe each task in moredetails and provide in an appendix a detailed report of theirresults.

Open Data Euskadi Repository kindly provided by the Vicomtech3 research center.
In-domain training and development data were supplied
through the website of the WIT3 ([6]), while out-of-domain
training data were made available through the workshop’s
website.
2.2. Evaluation
Automatic translation of the test2018 tst2018 evaluation
set were required to be in NIST XML format with casesensitive, detokenized and punctuated texts. Translations
quality was measured automatically by means of the threeautomatic standard metrics BLEU, NIST, and TER. Casesensitive scores were calculated with the software tools
mteval-v13a.pl3 and tercom-0.7.254, by invocating:• mteval-v13a.pl -c• java -Dfile.encoding=UTF8 -jar tercom.7.25.jar -N -s
It is worth noticing here that the two scoring scripts apply
their own internal tokenization.
In order to allow participants to evaluate their progresses
automatically and under identical conditions, an evaluation
server was developed. Participants could submit the translation of the development set to either a REST Webservice or
through a GUI on the web, receiving as output BLEU, NIST
and TER scores computed as described above. The core of
the evaluation server is a shell script wrapping the mteval
and tercom scorers. The REST service is implemented witha PHP script running over Apache HTTP Server, while the
GUI on the web is written in HTML with AJAX code. The
evaluation server was utilized also by the organizers for the
automatic evaluation of the official submissions. After theevaluation period, the evaluation on the test2018 set was enabled to all participants as well.

2. Low Resource Machine Translation
2.1. Definition
The Low Resource Translation Task addresses a conventional bilingual text translation task in the domain of the TED
talks. Participants were required to translate TED talks fromBasque to English. Given the difficulty of the proposed translation direction and the scarcity of available parallel data, additional parallel data from related languages were prepared.
Concerning Basque-English data, training set included
64 TED talks with 5.6K parallel sentences (81K Basque and
109K English tokens). Development set contained 10 talks
with 1.1K parallel sentences (17K Basque and 23K English
tokens). The valuation set tst2018 consisted of 10 talks with
1.1K parallel sentences (15K Basque and 20K English tokens).
In-domain parallel training data included also talks from
related languages: 73 talks for Basque-French, 74 forBasque-Spanish, 2595 for French-English, 2589 for SpanishFrench and 2650 for Spanish-English. Moreover, an additional archive with the original xml files of all the TED talks
available at April 2018 – excluding those in the tst2018 evaluation set – was provided. Finally, participants could download any data of the original TED talks from the TED website– excluding those in the tst2018 evaluation set.
Out-of-domain training data were restricted to paralleland monolingual corpora (including Basque data) provided
by the OPUS1 and WMT2 organizations on their respective websites. Moreover, participants were allowed to utilize Basque-Spanish parallel and monolingual data from the

2.3. Submissions
We received 15 submissions from 8 different participants (4participants sent primary submissions only).
2.4. Results
The results on the tst2018 evaluation set for each participant
are shown in Appendix A.1, sorted by the BLEU metric.

3. Speech Translation
3.1. Definition
In contrast to previous years, this year the participants needed
to build the whole speech translation systems. The organizers
did not provide any intermediate results as done in previous
years. Instead, a baseline system was provided [8]. Participants were free to used parts of this system or purely rely on
the own models.

1 http://opus.nlpl.eu/

3 http://www.vicomtech.org

2 http://www.statmt.org/wmt18/

3
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 1: List of Participants
ALIBABAA PPTEC
AFRL
ADAPT
CUNI
FBK
HY
JHU
KIT
M E MAD

Machine Intelligence Technology Lab, Alibaba Group
Applications Technology (AppTek), Aachen, Germany
Air Force Research Laboratory, United States of America
ADAPT Centre, IrelandCharles University - Institute of Formal and Applied Linguistics, Czechia
Fondazione Bruno Kessler, Italy
University of Helsinky, Finland
Johns Hopkins University, Baltimore, USA
Karlsruhe Institute of Technology, Germany
Department of Digital Humanities / HELDIG University of Helsinki, Finland
Department of Signal Processing and Acoustics Aalto University, Finland
PROMPSIT
Prompsit Language Engineering, Spain
SGNLP
NLP Laboratory in Sogang University, South Korea
SRPOL-UEDIN Samsung R&D Institute Poland and University of Edinburgh, Poland/UK
TIIC
Voice Interaction Technology Center, Sogou Inc., Beijing, China
Tiangong Institute for Intelligent Computing, Tsinghua University, Beijing, China
USTC-NEL
University of Science and Technology of China and IFLYTEK Co. LTD.
This year edition of the speech translation task contained
two different conditions. In the first condition Baseline, theparticipants could use any architecture to generate the translations in the target language. The second condition Endto-End concentrated on end-to-end models. In this condition, participants need to train one large model to performthe whole process from source language audio to target language text.
In both tasks the same test data is used. The test data is
English audio and needs to be translated into German. The
test data consisted of two related types of data. One partof the training data are TED talks. These talks are wellprepared and address a broad audience. Therefore, they contain only very few disfluencies and contain only very few
special terms. The second part of the test sets contain university talks and research presentations. Since the talks aretargeted to a small target audience, the test sets contain more
special terms.
For training the system, different data sources were provided to the participants. For training the ASR components,
the TED LIUM corpus could be used [9]. For the training of
the machine translation component, the data available form
the WMT evaluation4 was allowed. In addition, the organizers provide the WIT corpus []. Furthermore, for the firsttime, also a corpus to train the end-to-end corpus was provided. This corpus consists of English TED talks alignedwith their German transcription5 .

Therefore, in a first step of the evaluation we need to realignthe sentences of the reference and the automatic translation.
This was done by minimizing the WER between the automatic translation and reference as described in [10]. Twosegmentation were generated, one used case information forthe case-sensitive metrics and one using no case information
for the case-insensitive metrics.
Using the resegmented input, we used 4 different metrics
to evaluate the results. For BLEU [11] and TER[12], we calculated case-sensitive and case-insensitive scores. In addition, we calculated the BEER score [13] and the characTER[14].

3.2. Evaluation

We reported results of the 2018 IWSLT Evaluation Campaign which featured two tasks: the translation of TED talks
from Basque to English and the speech translation task fromEnglish to German. In the second one, the test set contains
TED talks as well as university lectures and research talks.
In this task, two tracks were offered: a baseline condition
and the end-to-end condition. In total, 14 international re-3.3. Submissions
In total we received 27 submissions from 9 partners. We
received 7 primary submissions in the baseline condition and4 primary submissions in the end-to-end submission. Twoparticipants submitted output to both conditions. The resultsof all primary submissions are summarized in Appendix A.1.
3.4. Results
The detailed results of the automatic evaluation in terms of
BLEU, TER, BEER and characTER can be found in Appendix A.1.

4. Conclusions

Since the audio was not segmented by a human into sentencelike units, the generated translation were segmented into
different sentences than reference transcript and translation.
4 https://www.statmt.org/wmt18/

5 http://i13pc106.ira.uka.de/ mmueller/iwslt-corpus.zip

4
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

search groups joined the evaluation campaign. For the firsttime, traditional pipeline approaches for speech translation
were compared to end-to-end translation models.

111, pp. 125–135, October 2018. [Online]. Available:
https://ufal.mff.cuni.cz/pbml/111/art-zenkel-et-al.pdf[9] A. Rousseau, P. Deléglise, and Y. Esteve, “Enhancing
the ted-lium corpus with selected data for language
modeling and more ted talks,” in LREC, 2014. [Online]. Available: http://www.lrec-conf.org/proceedings/
lrec2014/pdf/1104 Paper.pdf

5. Acknowledgements
6. References[1] Y. Akiba, M. Federico, N. Kando, H. Nakaiwa, M. Paul,and J. Tsujii, “Overview of the IWSLT04 Evaluation
Campaign,” in Proceedings of the International Workshop on Spoken Language Translation, Kyoto, Japan,
2004, pp. 1–12.

[10] E. Matusov, G. Leusch, O. Bender, , and H. Ney, “Evaluating Machine Translation Output with Automatic
Sentence Segmentation,” in Proceedings of the 2nd International Workshop on Spoken Language Translation(IWSLT), Pittsburgh, USA, 2005.

[2] T. Takezawa, E. Sumita, F. Sugaya, H. Yamamoto, andS. Yamamoto, “Toward a broad-coverage bilingual corpus for speech translation of travel conversations in thereal world.” in Proceedings of the International Conference on Language Resources and Evaluation (LREC),
2002, pp. 147–152.

[11] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu:a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting on
association for computational linguistics. Association
for Computational Linguistics, 2002. [Online]. Available: https://www.aclweb.org/anthology/P02-1040.pdf[3] N. Ruiz and M. Federico, “Complexity of spoken versus written language for machine translation,” in Proceedings of the 17th Annual Conference of the European Association for Machine Translation (EAMT),
Dubrovnik, Croatia, 2014, pp. 173–180.

[12] M. Snover, B. Dorr, R. Schwartz, L. Micciulla,
and J. Makhoul, “A study of translation edit ratewith targeted human annotation,” in Proceedings ofassociation for machine translation in the Americas,
2006. [Online]. Available: https://www.cs.umd.edu/∼snover/pub/amta06/ter amta.pdf[4] F. Casacuberta, M. Federico, H. Ney, and E. Vidal, “Recent efforts in spoken language processing,” IEEE Signal Processing Magazine, vol. 25, no. 3, pp. 80–88,
May 2008.

[13] M. Stanojevic and K. Sima'an, “BEER: BEtter evaluation as ranking,” in Proceedings of the Ninth Workshop
on Statistical Machine Translation. Association for
Computational Linguistics, 2014.

[5] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and
Z. Chen, “Sequence-to-sequence models can directlytranslate foreign speech,” in Interspeech 2017, 08 2017,
pp. 2625–2629.

[14] W. Wang, J.-T. Peter, H. Rosendahl, and H. Ney,“CharacTer: Translation edit rate on character level,” inProceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers. Association for
Computational Linguistics, 2016. [Online]. Available:
https://doi.org/10.18653%2Fv1%2Fw16-2342[6] M. Cettolo, C. Girardi, and M. Federico, “WIT3 : Web
Inventory of Transcribed and Translated Talks,” inProceedings of the Annual Conference of the European
Association for Machine Translation (EAMT), Trento,
Italy, May 2012. [Online]. Available: http://hltshare.
fbk.eu/EAMT2012/html/Papers/59.pdf[7] E. Cho, S. Fünfer, S. Stüker, and A. Waibel, “A corpus of spontaneous speech in lectures: The kit lecture
corpus for spoken language processing and translation,”in Proceedings of the Ninth International Conferenceon Language Resources and Evaluation (LREC’14),
N. C. C. Chair), K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and
S. Piperidis, Eds. Reykjavik, Iceland: European Language Resources Association (ELRA), may 2014.
[8] T. Zenkel, M. Sperber, J. Niehues, M. Müller,
N.-Q. Pham, S. Stüker, and A. Waibel, “OpenSource Toolkit for Speech to Text Translation,” The
Prague Bulletin of Mathematical Linguistics, vol.

5
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Appendix A. Automatic Evaluation
A.1. Official Testset (tst2018)· All the sentence IDs in the IWSLT 2018 testset were used to calculate the automatic scores for each run submission.
· MT systems are ordered according to the BLEU metrics.
· WER, BLEU and TER scores are given as percent figures (%).
Low Resource MT : Basque-English
System

BLEU

NIST

TER

SRPOL-UEDIN
HY
PROMPSIT
FBK
CUNI
ADAPT
AFRL
SGNLP

26.21
25.01
24.02
23.99
22.86
13.89
12.25
10.42

6.51
6.45
6.24
6.34
6.10
4.46
4.03
3.49

59.49
59.48
60.81
59.43
60.31
69.98
80.63
103.96

Speech Translation : English-German
System

BLEU

TER

BEER

55.74
58.03
63.03
64.12
67.94
69.10
74.51

Baseline condition
54.73
84.72
52.69
92.24
51.77
69.26
51.56
63.47
50.61
58.16
49.23
64.27
47.01
82.56

characTER

BLEU(CI)TER(CI)#Words

29.44
27.86
24.23
22.72
20.78
18.37
17.13

TIIC
USTC-NEL
ALIBABAA PPTECKIT
AFRL
M E MAD

28.09
26.47
22.36
21.45
19.44
17.24
15.8

53.73
55.98
60.22
61.69
65.52
66.78
72.00

39611
38372
39751
41210
42128
41155
41848

USTC-NEL
FBK
KIT
JHU

End-to-End condition
19.4 68.20 48.77
87.30
20.77
65.73
10.24 78.20 40.68
129.47
11.16
76.38
8.4
88.54 41.48
80.38
9.22
86.55
5.45 89.59 35.46
99.89
6.09
88.20
Speech Translation TED Only : English-German

41372
36627
44155
40932

System

BLEU

TER

BEER

characTER

BLEU(CI)TER(CI)

TIIC
USTC-NEL
ALIBABAA PPTECKIT
AFRL
M E MAD

28.18
26.79
22.77
21.05
18.84
15.46
15.57

57.31
59.89
63.66
66.31
69.05
72.23
74.83

52.74
51.28
50.62
49.96
48.73
47.26
45.35

61.06
92.50
65.54
60.96
57.97
61.02
87.54

29.36
27.89
24.57
22.17
20.02
16.51
16.8

55.65
58.23
60.96
64.20
66.92
70.06
72.56

End-to-End condition
USTC-NEL 18.32 70.50 46.65
88.73
19.58
FBK
9.75 77.57 38.98
150.35
10.57
KIT
7.99 86.68 39.55
86.36
8.82
JHU
4.51 85.84 32.71
112.77
4.97
Speech Translation Lecture Only : English-German

68.36
75.95
84.76
84.63

System

BLEU

TER

BEER

characTER

BLEU(CI)TER(CI)

TIIC
USTC-NEL
ALIBABAA PPTECKIT
AFRL
M E MAD

27.55
25.95
21.77
21.84
20.01
18.94
16.01

54.25
56.24
62.42
62.03
66.88
66.12
74.20

57.43
54.59
53.30
53.73
53.16
51.92
49.25

117.57
91.89
74.42
66.96
58.43
68.77
75.65

29.06
27.6
23.68
23.28
21.5
20.13
17.44

51.89
53.82
59.52
59.28
64.18
63.65
71.46

USTC-NEL
FBK
KIT
JHU

20.41
10.7
8.76
5.84

End-to-End condition
66.00 51.67
85.31
78.81 42.99
100.4990.32 44.11
72.07
93.18 39.24
82.03

21.87
11.7
9.58
6.58

63.22
76.79
88.26
91.60

6
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Unsupervised Parallel Sentence Extraction from Comparable Corpora
Viktor Hangya1 , Fabienne Braune1,2 , Yuliya Kalasouskaya1 , Alexander Fraser1
1Center for Information and Language Processing
LMU Munich, Germany
2
Volkswagen Data Lab Munich, Germany{hangyav, fraser}@cis.lmu.defabienne.braune@volkswagen.de

Abstract

posed in [11] which combines adversarial training with posthoc mapping [12] to learn BWEs without any bilingual signal. We show that high performance can be achieved using
no parallel sentences nor any bilingual signal.

Mining parallel sentences from comparable corpora is ofgreat interest for many downstream tasks. In the BUCC
2017 shared task, systems performed well by training ongold standard parallel sentences. However, we often want
to mine parallel sentences without bilingual supervision. We
present a simple approach relying on bilingual word embeddings trained in an unsupervised fashion. We incorporate orthographic similarity in order to handle words with similar
surface forms. In addition, we propose a dynamic threshold method to decide if a candidate sentence-pair is parallel
which eliminates the need to fine tune a static value for different datasets. Since we do not employ any language specificengineering our approach is highly generic. We show that
our approach is effective, on three language-pairs, without
the use of any bilingual signal which is important because
parallel sentence mining is most useful in low resource scenarios.

As a baseline system we produce sentence embeddings
by averaging the word embeddings in the source language
and target language sentences and compare them using cosine similarity. One difficult aspect of the task is that not all
source sentences have a parallel target sentence, thus besides
picking the most similar target sentence for a given source
sentence it has to be decided if they are actually parallel. We
propose a dynamic threshold method which calculates a minimum similarity value in an unsupervised fashion based on
the input corpus.
Taking the average of the word embeddings in a sentence
tends to give too much weight to irrelevant words [13]. Recently, various word-based sentence similarity metrics wereintroduced [14, 15]. The disadvantage of these methods iseither that they are computationally expensive or that they
do not handle irrelevant words. To overcome these issues,
we propose a simple method which efficiently pairs sourcetarget words while handling irrelevant words, thus making
it feasible to process large datasets. In addition, we consider
an important weakness of BWEs that was shown before [16],i.e., that they are poor at capturing the translations of namedentities and rare words, showing that this is an important
problem for parallel sentence extraction. We alleviate thisby combining semantic similarities taken from BWEs withorthographic cues such as Levenshtein distance.

1. Introduction
The ability to extract parallel sentences from monolingual
corpora is of great interest to the field and many approaches
have been proposed [1, 2, 3, 4]. In this paper we explore
ways to mine parallel sentences from monolingual data without bilingual supervision.
Our approach is based on bilingual word embeddings(BWEs) which represent words from different languages inthe same vector space. While many authors leverage BWEs
for parallel sentence extraction, previous work requires astrong bilingual signal to either (i) train the BWEs [5] (ii)train a classifier for sentence-pair extraction [6, 7, 8] or (iii)
for feature engineering [9]. The disadvantage of these approaches is that the required bilingual signal is not available
for many language pairs which is itself one of the reasons
why parallel sentence extraction is important. In contrast tothese approaches, our method does not need any bilingualsignal. We create BWEs using post-hoc mapping [10] whichallows us to leverage large amounts of (cheap) monolingual
data to train good monolingual word embeddings (MWEs)
which are then mapped into BWEs. We use the method pro-

In summary, our contributions are: (i) We evaluate two
approaches for parallel sentence extraction utilizing BWEs,
based on sentence embeddings and word-by-word similarities respectively, which do not need any bilingual signal, in
contrast with previous work. (ii) We introduce a dynamic
threshold method for deciding whether a candidate sentencepair is parallel. (iii) We incorporate orthographic similarity to
improve performance of parallel sentence extraction. (iv) Weshow the generality of our method on the German-English,
French-English and Russian-English comparable corpora of
the BUCC 2017 shared task [17].

7Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

2. Building Bilingual Word Embeddings

X and Y are sets of embeddings of words coming from thesource and target language. In contrast, the goal of the generator is to learn W such that it prevents the discriminator frommaking accurate predictions. After training, W is used toautomatically extract a seed lexicon of best candidate wordpairs which is used to perform post-hoc mapping with [12].
We use [11] in our fully unsupervised setup. As a contrastive experiment we report results with [12] using a seed
lexicon of 5000 word pairs, which was used as a baseline in[11] as well.

In this section we present two different scenarios to build
BWEs. In particular, we use only monolingual datasets totrain MWEs and we map them to the same bilingual spacecomparing two methods: the first only needs a small seed
lexicon while the second does not rely on any bilingual signal.
2.1. Monolingual Word Embeddings
We train MWEs for all 4 languages in our test set. For this we
used monolingual news crawls downloaded between 2011
and 2014 taken from the WMT 2014 shared task [18] containing around 80M, 117M, 31M and 45M sentences for English, German, French and Russian respectively. We used
FastText skipgram [19] to train MWEs which computes a
distributed representation of words using context and word
structure information in the form of character n-grams. Settings used are: Embedding dimension 300; Minimum occurrence frequency 5; Window size 5; Character n-gram sizes
between 3 and 6.

3. Sentence Extraction
We evaluate our model on the shared task data provided by
the BUCC Workshop at ACL 2017. We evaluate our system
on De-En, Fr-En and Ru-En language pairs. The dataset consists of comparable monolingual corpora (Wikipedia dumps)
where the BUCC organizers inserted truly parallel sentences(taken from News Commentary) into the monolingual data
for each language pair [17]. Our task is to recover the truly
parallel sentences, while minimizing false alarms.

2.2. Bilingual Word Embeddings

3.1. Sentence Embeddings

Our approach to the task of parallel sentence extraction requires BWEs, which is a common vector space for words
in two different languages. In previous research BWEs
were created either from word-aligned, sentence-aligned or
document-aligned parallel data [20, 21] or by using the crosslingual reference to optimize two monolingual spaces, so
called joint training [22, 23, 24]. Similarly to [9] we create
BWEs using post-hoc mapping. First, we explain the basic
idea of post-hoc mapping in the supervised setup and discuss
the way how supervision is eliminated in the unsupervised
method which our approach is based on.
Given two MWEs RdS and RdT post-hoc mapping is performed via a matrix W ∈ RdS ×dT which is learned using abilingual seed lexicon. Each pair of words (si , ti ) in the lexicon, with si ∈ Vs and ti ∈ Vt , is projected into x~i ∈ RdS
and y~i ∈ RdT . W can be solved by learning a linear mapping[10]:

We use a basic sentence embedding approach as a baseline.
BWEs are used to embed sentences in both languages into
the same space. Each sentence embedding is computed bydimension-wise averaging of the embeddings of words in thegiven sentence (contained in the BWEs) followed by l2 normalization. Once source and target sentences are embedded,
their similarity can be efficiently computed via cosine similarity [25]. To overcome the issue of giving too much weight
to semantically poor words, which decreases precision andmistakenly selects non-parallel sentences, we remove stopwords [26], digits and punctuation from texts before calculating sentence embeddings. Consider this erroneous example, which shows how weighting stop words like a, in, by too
highly causes an erroneous match:W∗ = arg min || XW − Y ||F

De: Inzwischen sterben mehr Frauen an Gebärmutterhalskrebs – alle zwei Minuten eine – als bei einer
Entbindung.

(1)

Gloss: Meanwhile die more women from (literal: in)ovarian cancer – every two minutes one (literal: a) –than at (literal: by) a birth.

W∈RdS ×dT

where X and Y are obtained by concatenating all projections x~i and y~i of words in the seed lexicon. The authorsof [12] showed that the mapping can be improved by enforcing an orthogonality constraint on W which can be achievedby solving the singular value decomposition of YXT . To
achieve good performance a seed lexicon of around 5000
word-pairs is used.
W can also be solved without any explicitly bilingual signal. The system of [11] uses adversarial training i.e. a generator and discriminator framework to achieve this. The aim ofthe discriminator is to distinguish mapped source language
embeddings WX and target language embeddings Y, where

En: For women in the developing world, by contrast,dying in childbirth is simply a fact of life.
3.2. Dynamic thresholding
To decide whether a candidate sentence pair, i.e., source sentence and its most similar target sentence, is parallel we introduce a method which calculates a minimum similarity valuethat the candidate has to meet. We calculate this thresholdvalue for each test set with a simple formula:

8
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

where S is a set containing the similarity values between
each source sentence in the test set and its most similar target
candidate, S̄ and std(S) are its mean and standard deviation.
We set λ = 2.0 based on the De-En development set which
worked optimally for the other setups as well. The advantageof this method is that it performed well on all our datasets,
while fine tuning a static threshold value on the development
sets did not achieve good results (see §4) due to the difference of development and test data. Note also that λ could
be quickly and easily adjusted by the user in order to balance between quality and quantity for downstream tasks (inpractice inspection of only a few samples is sufficient).

greedy fashion, with the word t ∈ T that has the highest
similarity based on our dictionary. During iteration, we ignore all t which have been already paired to overcome thehubness problem, i.e. by preventing the pairing of multiple
source words to the same target word. Then, the averaged
word-pair similarity gives the final score. We apply the same
stopword filtering as before and use dynamic thresholding
for the final decision. Although, we kept our method simple
for computational reasons we use pre-filtering as in previous
work [6]. For each source sentence we only consider the 100
most similar target sentences as candidates based on sentence
embedding similarities. Given a BWE model our method requires around 2.5 hours to process sentences from the De-En
test set (164 billion sentence pairs) on a single thread.

3.3. Bilingual dictionaries

3.3.3. Orthographic similarity

Averaging word embeddings in a sentence tends to give toomuch weight to irrelevant words. It was shown that hub
words, which are similar to a high proportion of other words,
have negative effects on performance of embedding based
methods [13]. Word Mover’s Distance was introduced [14],which is based on the minimum distance that the words in
one text need to “travel” to reach the words in the other text,
to overcome such issues. On the other hand, the approach iscomputationally intensive which is not desirable in the caseof parallel sentence extraction due to the high number of candidate sentence pairs. Furthermore, it was show that WMD
performs similarly to maximum alignment based methods onmonolingual sentence similarity tasks while the latter is computationally less intensive [15]. We propose an efficient hub
word aware maximum alignment approach based on bilingual dictionaries and show that it is more effective than simple sentence embeddings. In this method, we perform bilingual lexicon induction on the trained BWEs to generate large
n-best dictionaries, which we then use to mine parallel sentences.

As it was shown in previous work the performance of bilingual lexicon induction can be significantly improved by using orthographic cues, especially for rare words. We extend
this idea to the sentence level by using a dictionary containing orthographically similar source-target language wordpairs and their similarity1 . We define orthographic similarity
as one minus normalized Levenshtein distance. We use thisorthographic dictionary with BWE based dictionary whenmining parallel sentences by using the bigger value from thetwo dictionaries. If the given word pair is not in a dictionary
we consider their similarity as 0.0.

th = S̄ + λ ∗ std(S)(2)

4. Results
As we mentioned earlier we evaluate our system on the DeEn, Fr-En and Ru-En data of the BUCC 2017 shared task[17]. We show results based on BWEs created fully unsupervised with the method of [11] (unsup) and the lightlysupervised system of [12] (lisup) on the released training sets
as in [8]. Our systems only rely on news crawl monolingual
data and a small seed lexicon in case of the latter thus we did
not use the training set in earlier steps. We will show the performance of our final system and results of previous supervised systems on the official test set at the end2 . As baseline
we use the sentence embedding system with stopword filtering and dynamic thresholding. We report precision, recalland F1-scores.
From table 1 it can be seen that the dictionary based
approaches significantly outperform the baseline system foreach language pair. Our systems perform best on Fr-En andlowest on Ru-En which strongly correlates with the performance of the mapping approaches, that was shown in [11],on bilingual lexicon induction. Even though the baseline performs the weakest it is competitive on French-English withsimilar systems [6]. We also ran our baseline system on DeEn with lisup and static threshold value instead of dynamic.

3.3.1. Bilingual lexicon induction
Given a BWE representing two languages Vs and Vt , an nbest list of translations for each word s ∈ Vs can be inducedby taking the n words ti ∈ Vt whose representation x~t in the
BWE is closest to the representation x~s according to cosine
similarity.
From the source side of the comparable data we computea list containing the 200,000 most frequent words. For eachword in the list, we retrieve the 100-best translations usingbilingual lexicon induction on the BWEs. Each translation is
given a weight by using cosine similarity computed with theBWE.
3.3.2. Sentence extraction
Given a candidate pair of source and target sentences S and
T , the similarity score is calculated by iterating over thewords in S from left to right and pair each word s, in a

1 For speedup we only consider word pairs that have at least 0.8 similarity

2 We evaluated on the test set by sending our predictions to the sharedtask organizers.

9
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

lisupunsup

sentence-embedding
BWE dict.
BWE+ORT dict.
sentence-embedding
BWE dict.
BWE+ORT dict.

P (%)
27.86
23.05
24.19
26.53
22.67
23.71

De-En
R (%) F1 (%)
18.01
21.88
42.29
29.83
45.11
31.49
16.40
20.27
41.90
29.42
44.57
30.96

Fr-En
R (%)
14.38
52.19
52.64
14.07
52.30
52.61

P (%)
29.22
38.16
39.00
28.99
37.97
39.02

F1 (%)
19.28
44.08
44.80
18.94
44.00
44.81

P (%)
6.92
16.80
16.32
6.54
17.31
16.75

Ru-En
R (%) F1 (%)
4.42
5.39
24.77
20.02
24.05
19.45
3.84
4.84
24.97
20.44
24.20
19.80

De: Microsoft hat Nokia Milliarden von Dollar versprochen, wenn es seine Smartphones exklusiv mit
Windows Phone ausstattet.
En-: In Q1 2008 Samsung shipped 46.3 million mobile
handsets 1Q 2008.
En+: Microsoft promised to pay billions of dollars for
Nokia to use Windows Phone exclusively.
On the other hand, it did not help for Ru-En because of theirdifferent character sets. We manually analyzed the results
and saw that the use of orthographic information gave higher
similarity scores to sentence-pairs that contained named entities with the same orthography. These pairs were correctly
mined without orthography thus no performance increase
was caused. On the other hand, higher similarity scorescaused higher dynamic threshold value thus losing some correctly mined pairs. This phenomenon can be fixed by betterfine tuning λ for this setup.
Our lisup and unsup systems are on par with each other.
Regarding F1 scores, the seed lexicon caused higher performance only for De-En while the unsupervised method performed better for the rest of the language pairs. In figure 1
we show precision-recall curves comparing the two systems
on the three language pairs. This also shows that their performance is similar. There is a bigger gap between the systems

Ru-En Fr-En

By fine tuning the value on the development set (achieving high score) we got only 2.69% F1-score on the training
set showing the importance of dynamic thresholding. Furthermore, in our preliminary experiments we used a shuffledparallel dataset of parliament proceedings and news articles
for BWEs as in previous work [6]. It showed that having
strongly comparable data could give 5% performance gain
for our setups in average. On the other hand, having accessto such data is unrealistic in real life scenarios, so we don’t
use this data further in our work.
Comparing the dictionary based approaches with andwithout the orthographic dictionary it can be seen that the orthographic information helped the most for De-En and also
increased performance for Fr-En. In the following example
the incorrect En sentence is about the same topic but orthography was needed to extract the sentence with correct entities:

De-En

Table 1: Results of our proposed systems on the BUCC 2017 shared task’s training set for the 3 language-pairs. Baseline is
the sentence embedding based model with stopword filtering and dynamic tresholding. We underline the best F1 scores for alanguage-pair and BWE method and use bolding for the best overall F1 score for a given language-pair.

[27]
lisup BWE+ORT dict.
unsup BWE+ORT dict.
[27]
lisup BWE+ORT dict.
unsup BWE+ORT dict.
lisup BWE+ORT dict.
unsup BWE+ORT dict.

P (%)
88
24
24
80
39
39
16
17

R (%)
80
45
45
79
53
53
24
24

F1 (%)
84
32
31
79
45
45
19
20

Table 2: Results on the test set. We show the best performing
supervised system of the shared task [27].
in the case of Ru-En in higher precision ranges in favor of theunsupervised system. Overall, these results show that good
performance can be achieved in a fully unsupervised manner, i.e., using only monolingual data for training BWEs andusing only these for mining parallel sentence-pairs.
We show negative examples which our unsupervised system with orthographic information made on the De-En set intable 3. Examples 1-3 are incorrectly mined sentence pairs.
In the first case the meaning of the mined pair is very similar although it is not parallel while high named entity content causes the error in the next two. Although names inexample 2 are not orthographically similar they are close inembedding space which causes the error. Similarly, cardinal directions are different in example 3 but in general they
appear in similar contexts thus get represented similarly inthe word embedding space. In contrast, examples 4-6 havenot been mined by our system. The first two pairs have extra information on the source side, although they are parallel, which caused error for our system. Example 6 contains
the compound noun Entwicklungsländern (developing countries) which is not handled by our system.
Finally, we show results on the official shared task test
sets in table 23 . For comparison we also include the results ofthe best performing supervised system on De-En and Fr-En[27]. There were no submissions for Ru-En. It can be seenthat the performance of our systems on the test set are very
close to the performance on the training set which we presented in table 1. This shows that our dynamic threshold approach, with λ tuned on the De-En development set, is gen3 Results are rounded for consistency with the shared task paper.

10
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

0.8

De-En unsup
De-En lisup
Fr-En unsup
Fr-En lisup
Ru-En unsup
Ru-En lisup

0.7

0.6

recall

0.5

0.4

0.3

0.2

0.1

0
0.1

0.2

0.3

0.4

0.5

0.6

0.7

precision

Figure 1: Precision-recall curves comparing unsup and lisup systems on the three language pairs.

1.

2.

3.

4.

5.

6.

Das Werk wurde außerdem mit vier Academy Awards (Oscars) prämiert, darunter der Trophäe für
den besten fremdsprachigen Film.
In 2011, it was awarded the Academy Award for Best Documentary Feature at the 83rd Academy
Awards.
The work has also received four Academy Awards (Oscars), including the Best Foreign Language
Film Trophy.
Am 7. Juli 1957 wurde Angelas Bruder Marcus, am 19. August 1964 ihre Schwester Irene
geboren.
In April 1976 a daughter, Josina, was born, and in December 1978 a son, Malengane.
On July 7, 1957 Angela’s brother Marcus was born, on August 19, 1964 her sister Irene.
Im Osten Serbien, im Südosten Montenegro, sowie im Norden, Westen und Südwesten Kroatien.
It was in the modern Vojvodina (in northern Serbia), northern Croatia and western Hungary.
In east Serbia, southeast Montenegro, as well as in the north, west and southwest Croatia.
Aber die meisten Frauen, die Hillary Clinton wählen sollen, sind nicht Unternehmensjuristinnen
oder Staatssekretärinnen.
But most of the women sought as voters are not corporate attorneys or secretaries of state.
But most women who are to vote for Hillary Clinton are not corporate lawyers or statesecretaries.
Durch diese Kürzungen ging jedoch die Produktion weiter zurück und die wirtschaftliche Misereverschlimmerte sich nur noch mehr.
As they cut, output fell further and economic misery deepened.
As a result of these cuts, however, production continued to decline and the economic misery
deepened.
Für Frauen in den Entwicklungsländern dagegen ist es ganz normal, bei der Entbindung sterben
zu können.
For women in the developing world, by contrast, dying in childbirth is simply a fact of life.

Table 3: Samples from the manual analysis. 1-3 are incorrectly mined examples (translation of De sentences where differingfrom En pair shown in italic) while 4-6 are the missed parallel sentences.

11
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018eral enough to work well on multiple languages and datasets.
Interestingly, the supervised system performed better on DeEn comparing with Fr-En while our approach reached higher
F1 scores on the latter. One reason for this could be the better mapping quality of the word embedding space for Fr-En
which was show in [11]. Our results with the fully unsupervised system are lower comparing to the supervised method
since the latter has access to a large parallel corpus during
training. Using parallel data supervised systems can learn
features which help to decide if a sentence pair is parallel,e.g. word order of a source side phrase in the target side. In
contrast, in the unsupervised case, we can only rely on word
similarity information which can cause errors when syntax isthe deciding factor in the case of a sentence-pair with similarwords. On the other hand, our approach performed well on
this task and will serve as a strong baseline for future unsupervised methods. With our dynamic thresholding method it
is also easy to calculate a good initial threshold value which
can be changed manually by the user in order to balance between quantity and quality of the mined sentence pairs.

5. Conclusion
In this work we introduced our first steps for the task of unsupervised parallel sentence extraction. We showed the performance of a simple sentence embedding system based on
unsupervised BWEs and proposed a novel technique for dynamically setting the decision threshold. We improved uponthis baseline system by proposing a simple word pair similarity based method which is efficient for large corpora. Furthermore, we addressed the shortcomings of BWEs when
applying them for parallel sentence mining by using orthographic similarity. We showed that our system works wellfor various language pairs where BWEs could be built by
achieving good results on De-En, Fr-En and Ru-En. In addition, we showed that unsupervised BWEs perform as well
as BWEs based on a small seed lexicon. The goal of thisshort work is to provide a strong baseline for the unsupervised parallel sentence extraction task, and we are hoping toencourage more research on this important problem.

6. Acknowledgments
We thank Helmut Schmid for helpful discussions and comments. This project has received funding from the European
Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement № 640550).

7. References[1] D. S. Munteanu, A. Fraser, and D. Marcu, “Improved
Machine Translation Performance via Parallel Sentence Extraction from Comparable Corpora,” in Proc.
NAACL-HLT, 2004.
[2] J. R. Smith, C. Quirk, and K. Toutanova, “Extract-

ing Parallel Sentences from Comparable Corpora Using
Document Level Alignment,” in Proc. NAACL-HLT,
2010.
[3] C. Chu, R. Dabre, and S. Kurohashi, “Parallel Sentence
Extraction from Comparable Corpora with Neural Network Features,” in Proc. LREC, 2016.
[4] K. Krstovski and D. A. Smith, “Bootstrapping Translation Detection and Sentence Extraction from Comparable Corpora,” in Proc. NAACL-HLT, 2016.
[5] J. Grover and P. Mitra, “Bilingual Word Embeddings
with Bucketed CNN for Parallel Sentence Extraction,”in Proc. ACL, Student Research Workshop, 2017.
[6] F. Grégoire and P. Langlais, “BUCC 2017 Shared Task:a First Attempt Toward a Deep Learning Framework
for Identifying Parallel Sentences in Comparable Corpora,” in Proc. 10th Workshop on Building and Using
Comparable Corpora, 2017.
[7] H. Bouamor and H. Sajjad, “H2@ BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings,” in Proc. Workshop on Building and Using Comparable Corpora,
2018.
[8] H. Schwenk, “Filtering and Mining Parallel Data in a
Joint Multilingual Space,” in Proc. ACL, 2018.
[9] B. Marie and A. Fujita, “Efficient Extraction of PseudoParallel Sentences from Raw Monolingual Data Using
Word Embeddings,” in Proc. ACL, 2017.
[10] T. Mikolov, Q. V. Le, and I. Sutskever, “Exploiting Similarities among Languages for Machine Translation,”
CoRR, vol. abs/1309.4168, 2013.
[11] A. Conneau, G. Lample, M. Ranzato, L. Denoyer, and
H. Jégou, “Word Translation Without Parallel Data,”
CoRR, vol. abs/1710.04087, 2017.
[12] C. Xing, D. Wang, C. Liu, and Y. Lin, “Normalized
Word Embedding and Orthogonal Transform for Bilingual Word Translation,” in Proc. NAACL-HLT, 2015.
[13] G. Dinu, A. Lazaridou, and M. Baroni, “Improving
Zero-Shot Learning by Mitigating the Hubness Problem,” in Proc. workshop track at international conference on learning representation, 2015.
[14] M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger,“From Word Embeddings to Document Distances,” inProc. ICML, 2015.
[15] T. Kajiwara and M. Komachi, “Building a Monolingual
Parallel Corpus for Text Simplification Using Sentence
Similarity Based on Alignment between Word Embeddings,” in Proc. COLING, 2016.

12
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[16] F. Braune, V. Hangya, T. Eder, and A. Fraser, “Evaluating bilingual word embeddings on the long tail,” inProc. NACL-HLT, 2018.
[17] P. Zweigenbaum, S. Sharoff, and R. Rapp, “Overview
of the Second BUCC Shared Task: Spotting Parallel
Sentences in Comparable Corpora,” in Proc. BUCC,
2017.
[18] O. Bojar, C. Buck, C. Federmann, B. Haddow,
P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post,
H. Saint-Amand, R. Soricut, L. Specia, and A. Tamchyna, “Findings of the 2014 Workshop on Statistical
Machine Translation,” in Proc. 9th Workshop on Statistical Machine Translation, 2014.
[19] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov,“Enriching Word Vectors with Subword Information,”
CoRR, vol. abs/1607.04606, 2016.
[20] W. Y. Zou, R. Socher, D. M. Cer, and C. D. Manning, “Bilingual Word Embeddings for Phrase-Based
Machine Translation.” in Proc. EMNLP, 2013.
[21] K. M. Hermann and P. Blunsom, “Multilingual Distributed Representations without Word Alignment,” inProc. ICLR, 2014.
[22] S. Gouws, Y. Bengio, and G. Corrado, “BilBOWA: Fast
Bilingual Distributed Representations without Word
Alignments,” in Proc. ICML, 2015.
[23] A. Klementiev, I. Titov, and B. Bhattarai, “Inducing
Crosslingual Distributed Representations of Words,” inProc. COLING, 2012.
[24] H. Soyer, P. Stenetorp, and A. Aizawa, “Leveraging Monolingual Data for Crosslingual Compositional
Word Representations,” CoRR, vol. abs/1412.6334,
2014.
[25] J. Johnson, M. Douze, and H. Jégou, “Billionscale similarity search with GPUs,” CoRR, vol.
abs/1702.08734, 2017.
[26] E. Loper and S. Bird, “NLTK: The Natural Language
Toolkit,” in Proc. ACL Workshop on Effective Tools andMethodologies for Teaching Natural Language Processing and Computational Linguistics, 2002.
[27] A. Azpeitia, T. Etchegoyhen, and E. Martı́nez,“Weighted Set-Theoretic Alignment of Comparable
Sentences,” in Proc. 10th Workshop on Building andUsing Comparable Corpora, 2017.

13
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Word Rewarding for Adequate Neural Machine Translation
Yuto Takebayashi† , Chu Chenhui‡ , Yuki Arase† , Masaaki Nagata∗†

Graduate School of Information Science and Technology, Osaka University‡
Institute for Datability Science, Osaka University∗
NTT Communication Science Laboratories, NTT Corporation{takebayashi.yuto, arase}@ist.osaka-u.ac.jp, chu@ids.osaka-u.ac.jp, nagata.masaaki@lab.ntt.co.jp

Abstract
To improve the translation adequacy in neural machinetranslation (NMT), we propose a rewarding model with target word prediction using bilingual dictionaries inspired by
the success of decoder constraints in statistical machinetranslation. In particular, the model first predicts a set of target words promising for translation; then boosts the probabilities of the predicted words to give them better chances
to be output. Our rewarding model minimally interacts
with the decoder so that it can be easily applied to the decoder of an existing NMT system. Extensive evaluation
under both resource-rich and resource-poor settings showsthat (1) BLEU score improves more than 10 points with oracle prediction, (2) BLEU score improves about 1.0 pointwith target word prediction using bilingual dictionaries created either manually or automatically, (3) hyper-parametersof our model are relatively easy to optimize, and (4) undergeneration problem can be alleviated in exchange for increasing over-generated words.

1. Introduction
Neural machine translation (NMT) [1, 2, 3] has dramatically
improved machine translation quality compared to statisticalmachine translation (SMT). However, current NMT systems
still suffer from the adequacy problem due to inappropriate
lexical choice, under-generation, and over-generation [4]. In
SMT, bilingual dictionaries have been used to improve adequacy in translation as decoder constraints. Typical example
is the XML markup function implemented on MOSES [5].
Inspired by the decoding constraints for SMT, we propose a rewarding model using bilingual dictionaries to address the adequacy problem in NMT. Our model rewards target words that are promising to be used in correct translations
by boosting their probabilities to be output by a decoder. It
predicts such target words using bilingual dictionaries thatare created manually or automatically. By applying byte pairencoding (BPE) [6] to dictionaries, our model can benefit
from both BPE and dictionaries.
While previous studies incorporate bilingual dictionaries
into NMT for translation of rare words [7, 8] and domainspecific words [9], we do so to improve the adequacy of
NMT. Hence, dictionaries are made use of translating not

only specific types of words but also all words. In addition,these are methodologically different; our model simply biases the trained decoder while previous models change theinside NMT architectures and require training of the entire
systems. Due to this design, our model is easy to add totrained NMT systems and compatible with BPE.
Extensive evaluation on Japanese-to-English andEnglish-to-Japanese translation has been conducted using
two datasets; IWSLT (TED Talk) [10], spoken language
domain with a small set of bilingual sentences (223k), and
ASPEC [11], a scientific domain with a large set of bilingual
sentences (3M). We refer to the former as a resource-poor
domain and the latter as a resource-rich domain, hereafter.
The results show that the rewarding model with oracle
prediction of target words, where all and only target words
in references are predicted, BLEU score improves morethan 10 points on average in both of the resource-poor andresource-rich domains. When using bilingual dictionaries
created manually or automatically in the rewarding model topredict target words, BLEU scores improve about 1.0 pointon average in both domains.
Detailed analysis of our model reveals that it is relatively
insensitive to settings of its hyper-parameters and easy to optimize. In addition, it is shown that our model decreases thenumber of under-generated words while tends to increase thenumber of over-generated words.

2. Neural Machine Translation
The encoder-decoder model with attention [3, 12] is oneof the most popular architectures in NMT. It takes an input sentence X = {x1 , ..., xn } and generates its translation
Y = {y1 , ..., ym } as:p(Y |X; θ) =

m
∏

j=1

p(yj |y<j , X; θ),

where θ is a set of parameters and y<j = {y1 , · · · , yj−1 }.
Given a parallel corpus C = {(X, Y )}, the training objective
minimizes the cross-entropy loss with regard to θ:∑Lθ =− log p(Y |X).
(X,Y )∈C

14
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Rewarding Model
Df2e
$

Word
Prediction

x1 x2

yj

Softmax layer… xn

Encoder…

…

Decoder
% i-1
"

%i
"

% i+1
"

! #-1
"

!#
"

! #+1
"

xi-1

xi

xi+1

…

s0

…

…

sj

sj+1

yj-1

yj

…

Figure 1: Rewarding model at decoding step j: predicted target words Df 2e are rewarded to have better chances to be output at
each decoding time step. Note that the attention model is omitted for clarity.

The model consists of three parts, namely, an encoder, a
decoder, and an attention model. The encoder has an embedding layer and an recurrent neural network (RNN) layer. The
former converts words into their continuous space representations. Taking these embeddings, the RNN layer then computes a state that represents the input sequence till the current
time step. Specifically, we use the bi-directional long shortterm memory (LSTM) [13] that encodes the source sentence
by forward and backward directions. At time step i, the stateis represented by concatenating the forward hidden state ĥiand the backward one h̃i as hi = [ĥi ; h̃i ]. In this manner, X
can be represented as h = {h1 , ..., hn }.
The decoder remembers all the history of translationand its softmax layer computes the posterior probabilityp(yj |y<j , X) of a word yj to output as translation. In order to focus on specific parts of the input sentence necessary
for translation, the attention model is incorporated. We use
the global attention mechanism proposed in [12].

3. Rewarding ModelOn top of a decoder, our model rewards predicted words sothat they have better chances to be output as translations asshown in Figure 1. Specifically, it first predicts a set of target words Df 2e that are promising to be used in translations
using bilingual dictionaries. Then, our model rewards a target word if it is contained in Df 2e by adding weight to theposterior probability:
Q(yj |y<j , X) =log p(yj |y<j , X) + λryj ,(1)where λ is the weight of reward that will be tuned using a development set. This means that our model boosts the probabilities of predicted words that might have been slipped away
during beam search in the conventional decoder. In [14], asimilar rewarding model is proposed, but rewards are basedon remaining sequence lengths.
We use a simple binary rewarding in this paper:{
1 (yj ∈ Df 2e ),ry j =(2)0 (otherwise).

We also tried to model the rewarding function using lexical translation probabilities that can be estimated for automatically created dictionaries. However, preliminary experiments empirically showed that this simple form of rewarding worked best. This may be because these probabilities
are modeled in completely different ways, i.e., p(yj |y<j , X)
in Equation (1) is conditioned on the entire source sentence while lexical translation probabilities are conditioned
on source words. Further investigation is our future work.
Finally, a target word is output as:yj = arg max Q(yj |y<j , X).
yj

Accurate prediction of Df 2e is crucial for our rewarding
model. In the next section, we discuss practical implementations to obtain Df 2e from dictionaries.

4. Target Word Prediction with Dictionaries
In this study, we look up bilingual dictionaries created manually or automatically as word prediction, which allows to
make our model minimally interact with the original NMT
system. We will consider a sophisticated prediction modelusing an information in the encoder in future [15].
4.1. Prediction with Manually Created DictionaryThanks to the accumulated efforts by the academia and industry, bilingual dictionaries have been manually created for
language pairs of English and Japanese. Such manual bilingual dictionaries provide reliable translation knowledge, although their coverage is limited. One disadvantage of manual dictionaries is that conjugation and derivative forms are
generally not provided in such dictionaries. As a simple way
to predict the target word set, we look up source words in amanual bilingual dictionary.

15
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

4.2. Prediction with Automatically Created Dictionary
Previous studies have proposed methods to automaticallyconstruct bilingual dictionaries. Especially, word alignment
techniques for SMT [16, 5] allow us to construct a dictionary
directly from a parallel corpus. Similar alignment may bepossible using the attention model in NMT, however, reliability is not assured because the attention model is rather soft
as a constraint [17, 18].
The biggest advantage of using word alignment for dictionary construction is that the domain of the dictionary
matches that of translation targets. In addition, conjugations
are available in the dictionary. A disadvantage is that alignment errors may decrease the quality of the dictionary.
We apply the GIZA++ toolkit1 that is an implementationof the IBM alignment models [16] on a parallel corpus toautomatically create a bilingual dictionary. To control the
precision and recall of target word prediction, we introduce
a threshold δ, which is tuned on development data. Targetwords with lower translation probability than δ are discarded.
4.3. Exact and Partial Matching with BPE
Conducting translation on sub-words is effective to address
the unknown word problem [19]. We apply BPE [6] to dictionaries for word prediction to make our rewarding modelcompatible to BPE-based NMT. For both the dictionary entries and source sentences, we first apply a BPE modeltrained on a parallel corpus and then match the entries in dictionaries and source sentences.
We use two types of matching methods between an input sentence and dictionary entries: exact match and partial match. The former is precision-oriented and the latteris recall-oriented. After applying BPE, a dictionary headword (lemma) consists of multiple sub-words; a lemma w
is denoted as w = w1 , . . . , wk . Exact match regards w asmatched to a source sentence X if and only if: w1 , . . . , wk ∈X, s.t., for ∀i ∈ {1, . . . , k − 1}, wi = xj ⇔ wi+1 = xj+1 .
On the other hand, partial match regards w as matched to
X if wi ∈ X for ∃wi ∈ w. In both matching methods,
translations of w are added to the target word set as predictions. Obviously, target word predictions by partial match
subsumes those by exact match.

5. Experiment Settings
To investigate the effects of our model, we conductedJapanese-to-English and English-to-Japanese translation experiments on resource-poor and resource-rich domains.
5.1. Translation Tasks

dev 2010 and test 2010 sets for development and testing, containing 871 and 1, 549 sentences, respectively.
The resource-rich task used the Japanese-English paper
excerpt corpus (ASPEC)2 [11], which is one subtask of theworkshop on Asian translation (WAT)3 [20]. For training,
we used the first 2M parallel sentence pairs among the entire
3M pairs sentences following [21], because the remaining
1M sentences were noisy. The ASPEC task provides 1, 790,
and 1, 812 sentences for development and testing, respectively. We conducted both Japanese-to-English and Englishto-Japanese translation experiments on these two tasks, referred to as IWSLT-JE, IWSLT-EJ, ASPEC-JE, and ASPECEJ for short, hereafter.
5.2. NMT and Rewarding Model
We used the mlpnlp-nmt system4 that is an LSTM based
encoder-decoder NMT model with attention, which achieved
the best translation performance in human evaluations forboth the ASPEC-JE and ASPEC-EJ tasks at WAT 2017 [20].5
We implemented our rewarding model on top of the mlpnlpnmt system (our implementation will be public upon acceptance of the paper). We followed the hyper-parameter settings of [21]. The sizes of the source and target side embeddings, the LSTM hidden states, the attention hidden states
were all set to 512. We used 2-layer LSTMs for both the encoder and decoder with beam size of 5. Stochastic gradientdescent was used as the learning algorithm, with an initial
learning rate of 1.0, gradient clipping of 5.0, and a dropout
rate of 30% for the inter-layer dropout. The mini batch size
was 128. The training epochs for IWSLT-JE, IWSLT-EJ,ASPEC-JE, and ASPEC-EJ were all set to 20, and we chosethe model with the best development BLEU score among all
the epochs as the baseline systems.6
For the rewarding models, λ in Equation (1) was tunedon the development sets from 0.1 to 1.0 by 0.1 interval. The
threshold δ that prunes the automatically constructed dictionaries in Section 4.2 was tuned on 0, 0.0001, 0.001, 0.01 and0.1. We selected the best combination among all combinations of δ and λ on the development set for each model.
We investigate the upper-bound performance of our rewarding model using oracle target word prediction. On thisoracle model, predicted target words are all and only words
in a reference translation, i.e., precision and recall of prediction are both 100%. The best weight of λ was searched from
0.1 increasing the value by 0.1 until we observed a decreasein BLEU scores.
As preprocessing for the parallel corpora and bilingual
dictionaries, we segmented Japanese sentences/entries using MeCab,7 and tokenized and truecased the English sen2 http://lotus.kuee.kyoto-u.ac.jp/ASPEC/The resource-poor task used the IWSLT 2017 JapaneseEnglish task from the WIT project [10]. The IWSLT task
provides 223k parallel sentences for training. We used the1 http://code.google.com/p/giza-pp

3 http://orchid.kuee.kyoto-u.ac.jp/WAT/

4 https://github.com/mlpnlp/mlpnlp-nmt/

5 Experiments on other NMT models as future work.

6 Epoch #11, #20, #13 and #13 for IWSLT-JE, IWSLT-EJ, ASPEC-JE,
and ASPEC-EJ, respectively.
7 https://github.com/taku910/mecab

16
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

25

55
50

20
45
40

BLEU

BLEU

15

10

35
30

5

0
0

IWSLT-JE_Oracle

IWSLT-JE_Baseline

IWSLT-EJ_Oracle

IWSLT-EJ_Baseline

0.5

1

1.5

ASPEC-JE_Oracle
ASPEC-EJ_Oracle

25
20
2

0

0.5

ASPEC-JE_Baseline
ASPEC-EJ_Baseline

1

1.5

λ

λ

(a) IWSLT(b) ASPEC

2

2.5

3

Figure 2: BLEU scores by the oracle rewarding model when changing the λ on the development set. BLEU scores dramatically
improved on ASPEC task; 9.8 and 11.8 point improvements on ASPEC-JE and EJ, respectively.
tences/entries with the truecase.perl script in Moses8 forboth translation tasks. We further split the words into subwords using joint BPE [6] with 32, 000 merge operations.
The vocabulary sizes of the IWSLT-JE task were 21, 534 and18, 022, respectively. The vocabulary sizes of ASPEC-JE
task were 28, 852 and 22, 340, respectively.
5.3. Bilingual Dictionaries
As the manual dictionary, we used EDR,9 which is the
publicly available English and Japanese bilingual dictionary.10 The numbers of English-to-Japanese and Japaneseto-English entry pairs are 676k and 1, 052k, respectively. In
EDR, only lemmas are provided and thus inflected forms ofEnglish verbs are unavailable. To address this issue, inflected
forms of the EDR lemmas are extracted from the English
dictionary of XTAG project,11 which is used as the English
morphological analysis dictionary for TreeTagger.12 All thepossible inflected forms are added into our dictionary.
For dictionary look-up, a source sentence is first lemmatized and matched with the dictionary. We used MeCab forJapanese and TreeTagger for English to lemmatize words.
To automatically construct bilingual dictionaries,13 weused the GIZA++ toolkit on the training corpus in both
English-to-Japanese and Japanese-to-English directions.14
We applied the “grow-diag-final-and” heuristic and obtained
lexical translation probabilities using Moses. We then prunetranslation pairs with low probabilities by δ.

8 https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/
truecase.perl
9 http://www2.nict.go.jp/ipp/EDR/ENG/indexTop.html?
10 https://www.nict.go.jp/en/about/
11 https://www.cis.upenn.edu/∼xtag/
12 http://www.cis.uni-muenchen.de/ schmid/tools/TreeTagger/
13 Note that the corpora used for building the dictionaries are the same as
the one used for training each NMT systems. Other resources have not beenused to create automatic dictionaries.
14 Note that GIZA++ was applied on the parallel corpora without BPE,
which was only used for look up a source word in a dictionary.

6. Results
We first investigate the effect of λ using the development setson both the oracle target word sets and our word prediction
methods. Next, we evaluate the translation quality on thetest sets using the optimized λ. Finally, we conduct detailed
analysis of translation results by our rewarding model.
Throughout the section, the BLEU-4 score was used as
the evaluation metric, which was computed using the multibleu.perl script in Moses on tokenized and truecased English
and word-segmented Japanese sentences, respectively. The
significance tests were performed using the bootstrap resampling [22] at p < 0.01.
6.1. Effects of λFigure 2 shows the BLEU scores by the oracle word rewarding on the development sets of the IWSLT-JE, IWSLT-EJ,ASPEC-JE, and ASPEC-EJ tasks. The BLEU scores significantly improved according to the λ. The best settingsof λ improves 6.00, 8.25, 9.80, and 11.77 BLEU scores on
the IWSLT-JE, IWSLT-EJ, ASPEC-JE, and ASPEC-EJ tasks
from each baseline system, respectively.
Figure 3 shows the BLEU scores with respect to the λ andprecision/recall of word prediction on our model with word
prediction using manually or automatically created dictionaries. EDR indicates the models predicting target words using
EDR. XTAG indicates the models using EDR extended with
XTAG, which are only for the Japanese-to-English direction.
GIZA indicates the models that predict target words using
automatically constructed dictionary by GIZA++. The suffixes e and p in the legends indicate exact match and partial
match, respectively.
The results show that BLEU scores depend on precisionand recall of target word prediction by different dictionaries.
The weights of λ that achieved the best BLEU scores varied
from 0.1 to 1.0. Notice that these weights are much smaller
than the oracle prediction, which are 0.5, 0.4, 0.4, and 0.5 for
IWSLT-JE, IWSLT-EJ, ASPEC-JE, and ASPEC-EJ on GIZA
partial-match, respectively. This is because predicted words
are less reliable and too much rewarding degrades the trans-

17
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

11.5

11

11

BLEU

BLEU

11.5

10.5

EDR_e (5.43/31.89)
XTAG_e (2.40/39.51)
GIZA_e (0.29/88.02)
Baseline

10

9.5
0

0.2

10.5

EDR_p (0.30/80.20)
XTAG_p (0.23/83.78)
GIZA_p (0.27/88.02)

10

EDR_e (3.22/47.39)
GIZA_e (2.53/72.51)
Baseline

9.5

0.4

0.6

0.8

1

0

0.2

0.4

λ

27.5

40

27

39.5

26.5

25.5
0

0.2

0.4

0.8

1

(b) IWSLT-EJ
40.5

BLEU

BLEU

(a) IWSLT-JE

EDR_e (5.57/33.93)
XTAG_e (2.52/41.75)
GIZA_e (0.16/91.53)
Baseline

0.6

λ

28

26

EDR_p (0.35/68.01)
GIZA_p (0.22/88.41)

EDR_p (0.29/76.94)
XTAG_p (0.24/82.19)
GIZA_p (0.14/91.89)

39

38.5

EDR_e (3.79/48.09)

EDR_p (0.42/67.42)

GIZA_e (0.47/89.65)

GIZA_p (0.16/92.30)

Baseline
38

0.6

0.8

0

1

0.2

0.4

0.6

λ

λ

(c) ASPEC-JE(d) ASPEC-EJ

0.8

1

Figure 3: BLEU scores by our rewarding models with word prediction using bilingual dictionaries when changing the λ on thedevelopment sets. The gentle convex curves of BLEU scores show that the weight of λ is tunable by a simple grid search.
lation quality. The gentle convex curves of BLEU scores also
show that λ is easily tunable using a simple grid search.
6.2. Word Prediction and Translation Results
Table 1 shows the comparison of BLEU scores on the test
sets of the baseline and the rewarding models. We also report
the results that use a merged dictionary. We chose the XTAG
partial and GIZA partial for Japanese-to-English, EDR partial and GIZA partial for English-to-Japanese for merging because of their individual good performance. We tuned the λfor merged dictionary using the development set.
We can see that compared to the baselines, most of our
methods significantly improve BLEU scores. Overall, a word
prediction method with high recall shows a larger improvement in BLEU score as consistently shown by comparing exact matching v.s. partial matching, as well as comparing EDR
v.s. XTAG, EDR or XTAG v.s. GIZA, and GIZA v.s. merged
dictionary. However, there is still a gap between rewarding
by our target word prediction and rewarding by oracle prediction. Our GIZA and merged dictionary models achieve ahigh recall of about 90% but a very low precision of 0.1%.
Improving the precision for word prediction while keeping arecall high is our future work.
The baselines on ASPEC-JE and ASPEC-EJ are our reproduction of the state-of-the-art at WAT competition as single models, which are reported as achieved 27.62 and 39.71
BLEU scores in the paper. Compared to these scores, our rewarding model improved 0.67 and 0.36 points, respectively.

6.3. Under and Over Generation
We investigated the rate of under-generation and overgeneration that are the major adequacy problems in NMT[23] using Translation Edit Rate (TER) [24]. TER aligns areference and translation result. We counted the number ofDeletion and Insertion regarding these are caused by underand over generation, respectively. This is an approximation
to detect under and over generations, but we consider it isuseful as an automatic and handy evaluation metric.
Table 2 shows the average numbers of under and overgenerations per sentence. The under-generation decreaseson all the rewarding models in exchange of increasing overgeneration. The rewarding model with oracle target word
prediction reduces under generation about 1.2 word on average. This result shows that our rewarding model is also
effective for alleviating the under-generation problem. The
over-generation can be reduced by adding global constraint
to the rewarding model, which prohibits rewarding the same
predicted target. This is our future work.
Example translations of the baseline and our rewarding
model (GIZA partial match) are shown in the following. The
phrase of “congenital immunity” and “cancer of” were successfully translated by our model.
Source IL - 1 2 の 癌 に対する 抵抗 性 ( 先天 免疫 ) の 生
物 反応 について も 考察 し た
Reference biological response of the resistance (congenital
immunity ) to cancer of IL - 12 was also examined .

18
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

BLEU
Baseline
9.97
EDR
exact
9.99
partial
10.06
XTAG
exact
9.94
partial
10.30
GIZA
exact
10.36
partial
10.32
Merged dictionary 10.33
Oracle
17.68

IWSLT
JE
(Pre. / Rec.) BLEU(- / -)
10.26(5.19 / 30.58)
10.75(0.27 / 79.50)
10.73(2.25 / 38.10)(0.20 / 82.88)(0.27 / 85.98)
10.88(0.25 / 87.61)
10.83(0.16 / 89.25)
10.81(100 / 100)
20.26

EJ
(Pre. / Rec.) BLEU(- / -) 27.21(3.14 / 46.08) 27.82(0.33 / 67.09) 27.94(- / -) 27.73(- / -) 28.00(2.56 / 72.92) 28.29(0.21 / 87.38) 28.28(0.17 / 88.45) 28.29(100 / 100) 37.13

ASPEC
JE
(Pre. / Rec.) BLEU(- / -)
39.50(5.57 / 34.10)
39.74(0.28 / 77.15)
40.05(2.50 / 41.97)(0.23 / 82.42)(0.15 / 91.48)
39.96(0.13 / 91.80)
40.07(0.14 / 91.77)
40.05(100 / 100)
52.22

EJ
(Pre. / Rec.)(- / -)(3.70 / 48.33)(0.42 / 67.37)(- / -)(- / -)(0.46 / 89.73)(0.15 / 91.65)(0.17 / 92.12)(100 / 100)

Table 1: Comparison of BLEU scores on the test sets (The scores in bold indicate that the results are significantly better than thebaseline at p < 0.01). The best improvement in BLEU score is 1.08 point when using GIZA exact-match in ASPEC-JE.

Baseline
EDR e
EDR p
XTAG eXTAG p
GIZA e
GIZA pOracleunder-generation
IWSLT
ASPEC
JE EJ
JE EJ
3.58 3.25 3.37 3.36
3.53 2.89 3.07 2.94
3.44 3.13 2.85 2.90
3.48
- 2.923.14- 2.92
3.18 2.90 2.75 2.78
3.20 2.86 2.75 2.78
2.40 2.86 2.71 3.01

diction with bilingual dictionaries. Some studies tackle theadequacy problem in NMT, but they require an independent
SMT system [29, 30] or modification of the decoder [31].
Different from these, ours is simple and a cost-effective solution for the adequacy problem.

over-generation
IWSLT
ASPEC
JE EJ
JE
EJ
1.64 2.04 2.27 1.69
1.70 2.40 2.51 2.13
1.69 2.18 2.70 2.09
1.90
- 2.642.36- 2.642.33 2.58 2.67 2.15
2.34 2.52 2.66 2.15
4.26 2.80 3.04 3.06

Table 2: Numbers of under/over-generated words per sentence estimated by TER (The scores in bold indicate the bestscores).
Baseline the biological response of the resistance to IL - 12
is also discussed .
Our Model the biological response of the resistance(congenital immunity ) to the cancer of IL - 12 is also
discussed .

The under and over-generation problems have been recognized not only in NMT, but in other applications that
use the encoder-decoder model for natural language generation. Different solutions have been proposed. First, acoverage vector is introduced in NMT [23, 32, 33] thattracks which source words have been translated by the attention mechanism. A sparse and constrained attention has
been proposed [34], while word prediction, which are also
used to reduce computational cost of softmax function at
the decoder [35, 36], has been proposed to solve the undergeneration problem. The decoder in [37] encourages to output predicted target words by initializing the decoder throughword prediction, and the model in [38] predicts target words
and their expected frequencies to resolve the under and overgeneration problems in NMT-based summarization.

8. Conclusion

7. Related Work
Our rewarding model can be viewed as a constraint on the decoder to output desired target words. There have been studies
that aim to output predetermined words or phrases in neurallanguage generation. For this purpose, the grid beam searchin NMT is proposed [25] and the SMT lattice is combined
into NMT [26]. In neural conversation generation, Wen et al.
(2015) input a vector representing which information shouldbe generated to an encoder [27], and a decoder is designed to
explicitly control generation of emotional words [28].
Compared with these previous studies, one benefit of ourrewarding model is that the predicted words are used as soft
constraints on outputs with minimal interaction to the decoder. The most relevant study from the methodologicalpoint of view is [14] that also proposes a rewarding model ina decoder of NMT to improve the translation quality in general, such as remaining sequence lengths to output. We focus
on the adequacy problem in NMT and combine word pre-We proposed a rewarding model with word prediction to
boost the translation probabilities of the predicted targetwords that should be in correct translations. Our modelallows incorporating bilingual dictionaries on a BPE-based
NMT system. Extensive evaluation on both resource-poorand resource-rich domains showed its effectiveness.
As future work, first, we plan to improve the precisionof word prediction preserving the recall at high. Second, weplan to improve our rewarding model to effectively incorporate translation probabilities and extend the model to reward not only words but also phrases. We will also consider
a global constraint by predicting not only target words but
their frequencies, and adjust rewards when a word has beenused in translation. Finally, more experiments on datasetsof various domains and language pairs will be conducted toinvestigate the generality of our approach.

19
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

9. References[1] K. Cho, B. van Merriënboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio,“Learning phrase representations using rnn encoder–decoder for statistical machine translation,” in Proceedings of the 2014 Conference on Empirical
Methods in Natural Language Processing (EMNLP).
Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1724–1734. [Online]. Available:
http://www.aclweb.org/anthology/D14-1179[2] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence
to sequence learning with neural networks,” inProceedings of the 27th International Conference on
Neural Information Processing Systems. Cambridge,
MA, USA: MIT Press, 2014, pp. 3104–3112. [Online].
Available: http://dl.acm.org/citation.cfm?id=2969033.
2969173[3] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,” in
In Proceedings of the 3rd International Conference on
Learning Representations (ICLR 2015). San Diego,
USA: International Conference on Learning Representations, May 2015.
[4] P. Koehn and R. Knowles, “Six challenges forneural machine translation,” in Proceedings of theFirst Workshop on Neural Machine Translation.
Vancouver: Association for Computational Linguistics,
August 2017, pp. 28–39. [Online]. Available: http://www.aclweb.org/anthology/W17-3204[5] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. . Shen,
C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin,
and E. Herbst, “Moses: Open source toolkit for
statistical machine translation,” in Proceedings of the
45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedingsof the Demo and Poster Sessions. Prague, Czech
Republic: Association for Computational Linguistics, June 2007, pp. 177–180. [Online]. Available:
http://www.aclweb.org/anthology/P/P07/P07-2045[6] R. Sennrich, B. Haddow, and A. Birch, “Neuralmachine translation of rare words with subword units,”in Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume
1: Long Papers). Berlin, Germany: Association
for Computational Linguistics, August 2016, pp.
1715–1725. [Online]. Available: http://www.aclweb.
org/anthology/P16-1162[7] P. Arthur, G. Neubig, and S. Nakamura, “Incorporating discrete translation lexicons into neural machinetranslation,” in Proceedings of the 2016 Conferenceon Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational
Linguistics, November 2016, pp. 1557–1567. [Online].
Available: https://aclweb.org/anthology/D16-1162[8] J. Zhang and C. Zong, “Bridging neural machinetranslation and bilingual dictionaries,” CoRR, vol.
abs/1610.07272, 2016. [Online]. Available: http://arxiv.org/abs/1610.07272[9] M. Arcan and P. Buitelaar, “Translating domainspecific expressions in knowledge bases with neuralmachine translation,” CoRR, vol. abs/1709.02184,
2017. [Online]. Available: http://arxiv.org/abs/1709.
02184[10] M. Cettolo, C. Girardi, and M. Federico, “Wit3 : Web
inventory of transcribed and translated talks,” in Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), Trento, Italy,
May 2012, pp. 261–268.
[11] T. Nakazawa, M. Yaguchi, K. Uchimoto, M. Utiyama,
E. Sumita, S. Kurohashi, and H. Isahara, “Aspec:
Asian scientific paper excerpt corpus,” in Proceedingsof the Tenth International Conference on Language Resources and Evaluation (LREC 2016). Paris, France:
European Language Resources Association (ELRA),
May 2016.
[12] T. Luong, H. Pham, and C. D. Manning, “Effective
approaches to attention-based neural machine translation,” in Proceedings of the 2015 Conference on
Empirical Methods in Natural Language Processing.
Lisbon, Portugal: Association for Computational Linguistics, September 2015, pp. 1412–1421. [Online].
Available: http://aclweb.org/anthology/D15-1166[13] S. Hochreiter and J. Schmidhuber, “Long shortterm memory,” Neural Comput., vol. 9, no. 8,
pp. 1735–1780, Nov. 1997. [Online]. Available:
http://dx.doi.org/10.1162/neco.1997.9.8.1735[14] J. Li, W. Monroe, and D. Jurafsky, “Learning to decodefor future success,” CoRR, vol. abs/1701.06549, 2017.
[Online]. Available: http://arxiv.org/abs/1701.06549[15] S. Ma, X. SUN, Y. Wang, and J. Lin, “Bagof-words as target for neural machine translation,”in Proceedings of the 56th Annual Meeting of the
Association for Computational Linguistics (Volume 2:
Short Papers). Melbourne, Australia: Association for
Computational Linguistics, July 2018, pp. 332–338.
[Online]. Available: http://www.aclweb.org/anthology/
P18-2053[16] P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, and
R. L. Mercer, “The mathematics of statistical machinetranslation: Parameter estimation,” Computational Linguistics, vol. 19, no. 2, pp. 263–312, 1993.

20
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[17] L. Liu, M. Utiyama, A. Finch, and E. Sumita, “Neuralmachine translation with supervised attention,” in Proceedings of the 26th International Conference on Computational Linguistics (COLING2016), Osaka, Japan,
December 2016, pp. 3093–3102.
[18] H. Mi, Z. Wang, and A. Ittycheriah, “Supervised attentions for neural machine translation,” inProceedings of the 2016 Conference on Empirical
Methods in Natural Language Processing. Austin,
Texas: Association for Computational Linguistics,
November 2016, pp. 2283–2288. [Online]. Available:
https://aclweb.org/anthology/D16-1249[19] T. Luong, I. Sutskever, Q. Le, O. Vinyals, andW. Zaremba, “Addressing the rare word problem
in neural machine translation,” in Proceedings of
the 53rd Annual Meeting of the Association for
Computational Linguistics and the 7th International
Joint Conference on Natural Language Processing(Volume 1: Long Papers). Beijing, China: Association
for Computational Linguistics, July 2015, pp. 11–19.
[Online]. Available: http://www.aclweb.org/anthology/
P15-1002[20] T. Nakazawa, S. Higashiyama, C. Ding, H. Mino,
I. Goto, H. Kazawa, Y. Oda, G. Neubig, andS. Kurohashi, “Overview of the 4th workshop on
asian translation,” in Proceedings of the 4th Workshop
on Asian Translation (WAT2017). Taipei, Taiwan:
Asian Federation of Natural Language Processing,November 2017, pp. 1–54. [Online]. Available:
http://www.aclweb.org/anthology/W17-5701[21] M. Morishita, J. Suzuki, and M. Nagata, “Nttneural machine translation systems at wat 2017,” inProceedings of the 4th Workshop on Asian Translation(WAT2017). Taipei, Taiwan: Asian Federation of
Natural Language Processing, November 2017, pp.
89–94. [Online]. Available: http://www.aclweb.org/
anthology/W17-5706[22] P. Koehn, “Statistical significance tests for machinetranslation evaluation,” in Proceedings of the 2004
Conference on Empirical Methods in Natural Language
Processing. Barcelona, Spain: Association for Computational Linguistics, July 2004, pp. 388–395.
[23] Z. Tu, Z. Lu, Y. Liu, X. Liu, and H. Li,“Modeling coverage for neural machine translation,”in Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers). Berlin, Germany: Association for
Computational Linguistics, August 2016, pp. 76–85.
[Online]. Available: http://www.aclweb.org/anthology/
P16-1008[24] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, andJ. Makhoul, “A study of translation edit rate with targeted human annotation,” in Proceedings of Association for Machine Translation in the Americas, 2006.
[25] C. Hokamp and Q. Liu, “Lexically constrained
decoding for sequence generation using grid beam
search,” in Proceedings of the 55th Annual Meeting of
the Association for Computational Linguistics (Volume
1: Long Papers). Vancouver, Canada: Association
for Computational Linguistics, July 2017, pp. 1535–
1546. [Online]. Available: http://aclweb.org/anthology/
P17-1141[26] F. Stahlberg, A. de Gispert, E. Hasler, and B. Byrne,“Neural machine translation by minimising the bayesrisk with respect to syntactic translation lattices,”in Proceedings of the 15th Conference of theEuropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers.
Valencia, Spain: Association for Computational Linguistics, April 2017, pp. 362–368. [Online]. Available:
http://www.aclweb.org/anthology/E17-2058[27] T.-H. Wen, M. Gasic, N. Mrkšić, P.-H. Su, D. Vandyke,
and S. Young, “Semantically conditioned lstm-based
natural language generation for spoken dialoguesystems,” in Proceedings of the 2015 Conference on
Empirical Methods in Natural Language Processing.
Lisbon, Portugal: Association for Computational
Linguistics, September 2015, pp. 1711–1721. [Online].
Available: http://aclweb.org/anthology/D15-1199[28] H. Zhou, M. Huang, T. Zhang, X. Zhu, and B. Liu,
“Emotional chatting machine: Emotional conversation
generation with internal and external memory,” 2018.
[29] J. Zhang, M. Utiyama, E. Sumita, G. Neubig, and
S. Nakamura, “Improving neural machine translation
through phrase-based forced decoding,” in Proceedingsof the Eighth International Joint Conference on Natural
Language Processing (Volume 1: Long Papers).
Taipei, Taiwan: Asian Federation of Natural Language
Processing, November 2017, pp. 152–162. [Online].
Available: http://www.aclweb.org/anthology/I17-1016[30] L. Zhou, W. Hu, J. Zhang, and C. Zong, “Neuralsystem combination for machine translation,” inProceedings of the 55th Annual Meeting of the
Association for Computational Linguistics (Volume
2: Short Papers). Vancouver, Canada: Association
for Computational Linguistics, July 2017, pp. 378–
384. [Online]. Available: http://aclweb.org/anthology/
P17-2060[31] Z. Tu, Y. Liu, Z. Lu, X. Liu, and H. Li, “Context
gates for neural machine translation,” Transactionsof the Association for Computational Linguistics,

21
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

vol. 5, pp. 87–99, 2017. [Online]. Available: https://transacl.org/ojs/index.php/tacl/article/view/948[32] F. Meng, Z. Lu, H. Li, and Q. Liu, “Interactive attention for neural machine translation,” in Proceedings ofthe 26th International Conference on Computational
Linguistics (COLING2016), Osaka, Japan, December
2016, pp. 2174–2185.

Valencia, Spain: Association for Computational Linguistics, April 2017, pp. 291–297. [Online]. Available:
http://www.aclweb.org/anthology/E17-2047[33] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi,
W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey,
J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser,
S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens,
G. Kurian, N. Patil, W. Wang, C. Young, J. Smith,
J. Riesa, A. Rudnick, O. Vinyals, G. Corrado,
M. Hughes, and J. Dean, “Google’s neural machinetranslation system: Bridging the gap between humanand machine translation,” CoRR, vol. abs/1609.08144,
2016. [Online]. Available: http://arxiv.org/abs/1609.
08144[34] C. Malaviya, P. Ferreira, and A. F. T. Martins, “Sparseand constrained attention for neural machine translation,” in Proceedings of the 56th Annual Meeting of
the Association for Computational Linguistics (Volume
2: Short Papers). Association for Computational
Linguistics, 2018, pp. 370–376. [Online]. Available:
http://aclweb.org/anthology/P18-2059[35] X. Shi and K. Knight, “Speeding up neural machine translation decoding by shrinking run-time
vocabulary,” in Proceedings of the 55th Annual
Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Vancouver,
Canada: Association for Computational Linguistics, July 2017, pp. 574–579. [Online]. Available:
http://aclweb.org/anthology/P17-2091[36] B. Sankaran, M. Freitag, and Y. Al-Onaizan,“Attention-based vocabulary selection for NMT decoding,” CoRR, vol. abs/1706.03824, 2017. [Online].
Available: http://arxiv.org/abs/1706.03824[37] R. Weng, S. Huang, Z. Zheng, X.-Y. DAI, andJ. CHEN, “Neural machine translation with word
predictions,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics, September 2017, pp. 136–145. [Online]. Available:
https://www.aclweb.org/anthology/D17-1013[38] J. Suzuki and M. Nagata, “Cutting-off redundantrepeating generations for neural abstractive summarization,” in Proceedings of the 15th Conferenceof the European Chapter of the Association for
Computational Linguistics: Volume 2, Short Papers.

22
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Analyzing Knowledge Distillation in Neural Machine Translation
Dakun Zhang, Josep Crego, Jean Senellart
SYSTRAN / 5 rue Feydeau, 75002 Paris, France
firstname.lastname@systrangroup.com

Abstract
Knowledge distillation has recently been successfully applied to neural machine translation. It allows for building
shrunk networks while the resulting systems retain most of
the quality of the original model. Despite the fact that many
authors report on the benefits of knowledge distillation, few
have discussed the actual reasons why it works, especiallyin the context of neural MT. In this paper, we conduct several experiments aimed at understanding why and how distillation impacts accuracy on an English-German translation
task. We show that translation complexity is actually reduced
when building a distilled/synthesised bi-text when comparedto the reference bi-text. We further remove noisy data fromsynthesised translations and merge filtered synthesised data
together with original reference, thus achieving additional
gains in terms of accuracy.

1. Introduction
Neural machine translation (NMT) achieves state-of-the-art
results in several translation tasks and for multiple language
pairs [1, 2]. Equivalent to its phrase-based predecessor, neural networks directly learn from parallel bi-texts, consistingof large amounts of human created sentences with their corresponding translations. Therefore, the quality of an MT engine is heavily dependent on the amount and quality of parallel sentences.
Several techniques aimed at boosting the quality [3, 4]
and quantity [5] of training data are successfully applied toneural MT. Parallel to these techniques, knowledge distillation [6] has attracted the focus of many researchers given its
simplicity and the quality of the results. However, despite
the fact that a growing number of private entities have begun
to include distillation into their NMT systems [7, 8, 9] andthat knowledge distillation has demonstrated its performancefor multiple tasks [10, 11, 12], none of them give a detailed
analysis of the reasons why it works.
In most cases, the availability of parallel corpora is a prerequisite to build Neural MT systems. The process of compiling parallel bi-texts is usually composed of several steps:
crawling, filtering, cleaning, etc. As a result, parallel corpora
usually contain parallel sentences that are often not as parallel as one might assume. And even for parallel sentencesthat truly convey the same meaning, in some cases translations follow a more or less word-for-word pattern (more lit-

eral translations). While in many other cases, translations
show greater latitude of expression (more flexible translations) with higher degrees of variability, which humans often
judge as good. However, machine translations are usually“closer” in terms of syntactic structure and present lower levels of variability when considering word choice. It is rather
an intuitive idea that feeding more “literal” translations toa neural MT network should facilitate the training processcompared to training with less literal translations (original
bi-text).
In this paper, we report on the results of experiments
where we automatically distill a human translation bi-text
which is then used to train neural translation engines. Thus,
aiming at boosting the learning ability of neural translation
models. We show that the resulting models perform evenbetter than a neural translation engine trained on the originalreference dataset.
Our contribution is as follows:- We analyse the reason why and how distillation works
for neural machine translation.
- We analyse in detail the difference between originalreference translations and synthesised translations.
- We further filter out noise from synthetic data and measure the impact on using both synthetic and referencetranslations.
The remainder of this paper is structured as follows. Section 2 briefly surveys previous work. Section 3 outlines ourneural MT engine and details the distillation approach presented in this paper. Sections 4, 5 and 6 report training configurations and experimental results with detailed analysis.
Section 7 draws conclusions and outlines future work.

2. Related Work
Sequential knowledge distillation for neural machine translation was first detailed by [6]. The authors trained a smaller
student network to perform better by learning from a larger
teacher network allowing more compact neural MT models
to be built. [7] followed this idea and proposed a similar
language simplification method based on distillation. They
reported improvements on English to German and English to
French translations. [8] further demonstrated distillation experiments from both an ensemble teacher model and a single

23
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

model. After that, they improved training efficiency and performance by removing noisy sentences from the training corpus. As a comparison, we performed detailed experiments
and analysed the reasons why and how distillation works forneural machine translation.
Other than neural MT, [10] and [11] show that distillation
also works well when transferring knowledge from a network
ensemble or from a large highly regularised model into asmaller, distilled network for image classification and speech
recognition. [12] applied distillation based on a search based
structured prediction on dependency parsing and machinetranslation. These works demonstrate that knowledge distillation is adapted to many different tasks. In this work, we
focus on the influence of knowledge distillation to neural machine translation and suggest directions for further improvements.

3. Neural Machine Translation
We train two types of NMT systems in this work, an RNNbased model and a Transformer-based model. The RNN
model follows the architecture presented in [13]. It is implemented as an encoder-decoder network with multiple layers of an RNN with Long Short-Term Memory hidden units[14]. The Transformer model follows the work in [15]. It
encodes the representation of sentences in a way a self attention only and is reported as the current state-of-the-art inmany machine translation tasks [1, 9].
For the RNN model, the encoder is a bidirectional neuralnetwork that reads an input sequence s = (s1 , ..., sJ ) and−→−→calculates a forward sequence of hidden states (h1 , ..., hJ ),←−←−and a backward sequence (h1 , ..., hJ ). The decoder is an
RNN that predicts a target sequence t = (t1 , ..., tI ), being
J and I respectively the source and target sentence lengths.
Each word ti is predicted based on a recurrent hidden state
hi , the previously predicted word ti−1 , and a context vector ci . We employ the attentional architecture from [16] anduse the implementation of OpenNMT 1 . Additional details are
given in [17].
Unlike the RNN model, the Transformer model directlymodels the representations of each sentence with a selfattention mechanism. Hence, it reduces the number of operations related between tokens in different positions, especially for distant positions, in input and output sequence. The
Transformer model stacks a so-called multi-head self attention layer and a position-wise, fully connected layer for nonlinear conversions on both the encoder and decoder side. On
the decoder side, it uses masked self-attention to prevent positions to attend to unseen positions.
The notion of time step is encoded automatically in thesequence in the RNN model. Whereas the Transformer
model uses positional embedding to record the position information of each word in the sequence. In addition, the Transformer model is easy to be parallelized for the MLE training
1 https://github.com/OpenNMT/OpenNMT

process across multiple GPUs. This allows the benefit of accelerating the training speed compared with the RNN model.
In this work, we use the implementation of OpenNMT-tf 2 totrain our Transformer based systems.
3.1. Knowledge Distillation
Knowledge distillation is a method to train different deepneural networks on the same data. Information learned froma large teacher model with the original reference data can belearned quite well with a smaller student model with the synthesised data [10]. Thus, a compact smaller model is generated and used to replace the larger model, especially in someresource limited devices.
For machine translation, we follow the approach described by [6]. The machine translation model is trained
to minimise the Kullback-Leibler divergence, either between
the model distribution and ground-truth distribution LN LL ,or between the model distribution and synthesised data distribution LKD , which is from the teacher system, or an interpolation of both:
L = (1 − α) · LN LL + α · LKD
In [6], three distillation methods are proposed by tuning the weight (α) in sequence-level loss. We simplify thisprocess and perform experiments with α = 1 in this work.
First, we train a teacher system with the original source/target
data. Second, we train another student system with thesource/synthesised target data. The synthesised target language data is generated by running beam search (with beam
size 5) over the training set with the teacher system (forward
translation),
The objective during the training of the student systemis the same as the teacher system. The only difference isthe student system’s objective is not to maximise log likelihood toward the ground-truth reference, but toward a generalised “soft” target, which is from the teacher system. From
this point of view, the student system is directed by how theteacher system acts. Hence, in general a stronger teacher system is preferred. E.g. an ensemble teacher system is used in[8].

4. Experimental Conditions
4.1. Data
Experiments are performed using a preprocessed and tokenised version of WMT English-German translations3 . The
training set contains 4.5M sentence pairs. We use newstest2013 as validation set and both newstest2014 and newstest2015 as test sets. We applied joint byte-pair encoding(BPE) [18] with 32K merge operations. The actual training
vocabulary size is of 34K tokens after BPE tokenization. We
2 https://github.com/OpenNMT/OpenNMT-tf

3 The corpus is already tokenised and can be downloaded from https://nlp.stanford.edu/projects/nmt

24
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

tfm
rnn

Large
MiddleSmall
LargeMiddleSmall

N=6, d=512, df f =4096, h=8
N=6, d=512, df f =2048, h=8
N=4, d=256, df f =2048, h=8
Bi-LSTM, 4x1024, emb=512
Bi-LSTM, 2x1024, emb=512
Bi-LSTM, 2x512, emb=512

Table 1: Configurations of the networks used in this paper. In

the remainder of this paper we use respectively tfm.L, tfm.M, tfm.S,
rnn.L, rnn.M and rnn.S to represent systems trained with different
configurations.

limit sentence length to 100 in both source and target sides(excluding 0.31% of the training corpus). After decoding, weremove BPE joiners and evaluate the tokenised output withmulti-bleu.perl [19].
4.2. Network Configuration
In this paper, we employs several neural MT models basedon the Transformer [15] and RNN [13] models. Three different systems are used for each architecture, which differ in
network size. Details of the system configurations are given
in Table 1.
For RNN based systems, we use stochastic gradient descent, a mini-batch size of 64 in segments with dropout probability set to 0.3. We train our models during 18 epochs andevaluate the performance of the last epoch. Initial learning
rate is set to 1.0 and we start decaying after epoch 10 by afixed decay rate of 0.7. In decoding, we use a beam size of
5.
In the case of Transformer systems, we use Lazy Adam
optimiser, which starts the learning rate at 1.0. We train
the systems with a batch size of 8, 192 in tokens and savecheckpoints in every 5, 000 steps. We terminate training after 400K iterations and average the last 8 checkpoints to get
the final evaluation.

5. Analysis of Synthetic Translations
Aiming for a better understanding of the translated languages, we first conduct an elementary human analysis of theGerman hypotheses (synthesised translations) produced by
the best performing network. We observe that in many cases,
automatic translations produced by our neural MT systems
consist of paraphrases of the reference translations. Whileboth, reference and automatic translations, preserve the same
meaning and are grammatically correct, automatic translations are closer in terms of syntactic structure to the source
sentences than reference translations, which seams a key factor to train machine translation systems.
Examples in Table 2 illustrate this fact. In the first example, the English and German synthetic sentences follow avery similar structure. While in the German reference translation, the sentence: you are sure to find the nightclub you
like is expressed by clubbers (Disco-Gänger) are guaranteed

Figure 1: Histogram indicating the percentage (%) of sourcewords aligned to n (x-axis) distinct target words in the training set.
to get their money (common garantiert auf ihre Kosten). In
the second example, we can see a very similar situation. The
verb verwendet is shifted to the end of the sentence whencomparing the structure of the English and the German synthetic sentences. In contrast, the reference German translation employs a greater latitude of expression.
Next, we conduct several experiments in order to confirm the hypothesis that automatic translations are closer tothe input sentence than reference translations. We comparereference German translations (ref ) to automatic translations(syn) produced by our neural MT network over the entire
training set. In our experiments, we employ word alignments
computed using fast align4 .
5.1. Translation Fertility
First, we measure translation fertility. We identify the number of different target words aligned to each source word in
the training corpus. We regard this number as the “fertility”
between parallel sentences. Figure 1 shows a histogram indicating the percentage of source words aligned to n distinct
target words.
As it can be seen, English words are in average related
to less German words in the case of the automatic translations (syn) than for reference translations (ref ). 70.1% ofEnglish tokens are aligned to a single German token in thecase of automatic translations while this number is reduced to
48.6% in the case of reference translations. As expected, theopposite situation is also observed when considering targetwords aligned to multiple source words. In this case, reference translations show always a higher percentage of tokens.
5.2. Translation Distortion
In addition, we compare the translation distortion in order tovalidate the closeness (similarity) of syntactic structures. The
translation distortion is calculated by the number of crossedalignments on automatic (syn) and reference (ref) translations. Given a sentence pair with its set of alignments, we
4 https://github.com/clab/fast_align25
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Src:
Ref:
Syn:
Src:
Ref:
Syn:

[In Cala Ratjada]1 [you are sure to find]2 [a nightclub]3 [you like]4 .
Disco-Gänger kommen [in Cala Ratjada]1 garantiert auf ihre Kosten.
[In Cala Ratjada]1 [finden Sie sicher]2 [einen Nachtclub]3 , [den Sie mögen]4 .
[Your personal information]1 [will only be]2 [used]3 [to process your booking]4 .
Sie [werden nur]2 in dem Umfang weitergegeben , wie es [für eine Buchung]4 notwendig ist.
[Ihre persönlichen Daten]1 [werden nur zur]2 [Bearbeitung Ihrer Buchung]4 [verwendet]3 .

Table 2: Examples of English-to-German translation. Subscripts in these examples indicate the alignment between multi-words
expressions.
Configtfm.L*
tfm.M*
tfm.S
rnn.L
rnn.M
rnn.S

newstest2014
27.87
27.59
24.60
24.11
24.11
22.94

newstest2015
30.04
29.73
27.58
26.62
26.74
25.85

Table 3: BLEU scores on systems trained over the original
dataset. Systems with * are candidates for teacher systems.
Figure 2: Difference in number of crossed alignments (inpercentage (%)) between reference and synthesised translations. 0 means no crossed alignment, i.e. monotonic, between source and target sentences.
compute for each source word si the number of alignment
crossings between the given source word and the rest of thesource words. We consider that two alignments (i, j) and(i′ , j ′ ) are crossed if (i−i′ )∗(j −j ′ ) < 0. Figure 2 illustrates
the difference in number of crossed alignments between reference and synthetic translations.
As it can be seen, automatic (syn) translations show ahigher number of words with no crossed alignments (49.2%)
than reference (ref ) translations (39.8%). In contrast, when
considering larger numbers of crossings, the reference data
set shows higher ratios than the automatic data set. Thisshows the synthesised target is much “closer” to the sourcein grammatical order compared with original reference.
Note that automatic translations carry important levels ofnoise (translation errors) that cannot be neglected from theview of a human. However, since it’s the generalised output from the teacher system, it is indeed compatible to themachines. The next section evaluates the suitability of automatic translations as a training set for our neural MT systems
compared to reference translations.

6. Results
6.1. Basic systems
We first summarise translation accuracy results (BLEU
scores) of our 6 basic systems learned over the reference
training set. As shown in Table 3, systems implementing theTransformer architecture outperform RNN networks. The
best score is achieved by tfm.L, which is 27.87 for newstest2014 and 30.04 for newstest2015. The smallest transformer network (tfm.S) clearly outperforms the largest RNN
network (rnn.L) in about 0.5 BLEU points for newstest2014.
We choose the best two systems tfm.L and tfm.M as the candidates for teacher systems.
According to BLEU scores, similar performance is obtained by large and middle size versions of the Transformer
and RNN models. However, the smallest systems show aclear drop in performance for both architectures. We argue
that, when the network is big enough, the performance relies
more on the amount of training data. That is, if the training
data is fixed, there is a proper size of the neural network tolearn the information embedded inside this training data. Aneven larger network could achieve a better performance, but
not significantly.
6.2. Comparison between different teachers
For the distillation based method, the teacher system is important because its output will be used as the student’s training reference. As shown in [8], a student system trainedwith an ensemble teacher usually performs better than thattrained with a single teacher system. In our experiments, wetrain student systems based on a single teacher system. Table 3 shows that system tfm.L achieves similar accuracy compared with system tfm.M with original dataset. This meansthat there is no major difference between tfm.L and tfm.M,
and these two systems can both be used as teacher systems.
Therefore, in this section, we compare the performance between student systems trained on different teachers. We showthat a strong teacher will lead to better students. Results are

26
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Student
Systemtfm.S
rnn.S

Teacher
System
tfm.L
tfm.M
tfm.L
tfm.M

newstest
2014
26.07
25.33
24.84
24.22

newstest
2015
28.96
27.79
27.99
27.10

tfm.S

Table 4: Results of comparison between different teacher
systems.
shown in Table 4.
Both student systems outperform basic systems trainedwith original data (in Table 3). For newstest2014, systems
trained with original reference data achieve 24.60 for model
tfm.S and 22.94 for model rnn.S. When systems are trainedwith synthesised data from teacher system tfm.L, the performance improves to 26.07 (+1.47) and 24.84 (+1.90) respectively. This proves the distillation method works for neuralmachine translation.
We also notice that the difference between the twoteacher systems tfm.L and tfm.M trained with original data
is 0.28 for newstest2014. However, the difference between
same student model trained with these two teachers increases
to 0.74 for tfm.S and 0.62 for rnn.S. We argue that this isbecause of noisy data present in the synthesised target side.
We found from the training synthesised data that some target sentences are exactly the same regardless of the source
sentences. We further checked the original reference target
sentence and confirmed that it is because the original bi-text
is not parallel (noise in the original training data). During
the training of the teacher system, neural models tend to normalise all these “bad” instances into a uniformed one by minimising the log likelihood. However, during the training of
the student system, such noise is amplified and leads to thelarger gap between the different systems. We therefore analyse in detail the influence of noise in the next section.
6.3. Influence of synthesised data noise
Based on Table 4, we can conclude that a stronger teacher
is usually beneficial to train student systems. Similarly,
we compared two similar student systems rnn.M and rnn.S
based on teacher tfm.L. We found rnn.M can achieve 26.22 in
BLEU score in newstest2014, which is +1.38 BLEU points
higher than rnn.S. We therefore choose tfm.L as our teacher
system and tfm.S and rnn.M as our default student systems in
the following experiments. In this section, we train tfm.S and
rnn.M with a different proportion of the synthesised data to
see the influence of data noise for student systems.
As we showed in section 5, the synthesised data is thetranslation of the whole training set by a teacher model.
Sequences in these generated hypotheses contain noise as
they are from machine translated results. Noise includes ungrammatical sentences, wrong words selection, word ordering problems, etc. Also there are inconsistent target sen-rnn.M

synthesised
data
100%
95%
90%
100%
95%
90%

newstest
2014
26.07
26.24
26.20
26.22
26.11
25.98

newstest
2015
28.96
29.42
29.21
28.87
28.92
28.80

Table 5: Impact on different amounts of synthesised data forstudent systems by removing noisy data from the output of
the teacher system tfm.L.
tences with the source in semantic and under/over translation5 problems in the synthesised data. We regard all these
problems as noise because they are not correct translations.
We use an embedding based method proposed by [20] to
calculate the similarity between source and target sentences.
In [20], a sentence embedding was first built based on word
similarity, relying on a neural architecture, which is able toidentify several types of cross-lingual divergences. The resulting embeddings are then used to measure semantic equivalence between sentences6 . In our case, the target sentences
are synthesised data from a teacher system. We filter out sentence pairs which are not similar based on the similarity score
and train the student system with the remaining data. Table 5shows the results from distilled tfm.S and rnn.M systems.
We compare two different student systems, Transformer
based tfm.S and RNN based rnn.M. For rnn.M system, we
can not see any gains by removing different proportions ofnoisy data. While for tfm.S system, when we remove 5%
noisy data, we found an increase from 26.07 to 26.24 (+0.17)
in BLEU score for newstest2014, which is also the best performance we have achieved until now.
We argue that the Transformer based system is more sensitive to the noisy data compared with the RNN based system. When a little amount noisy data (e.g. 5%) is removed,
the performance improves because the remaining 95% isenough to train a good system. Along with the further reduction to 90%, both the Transformer based model and the
RNN based model starts to decrease because there are fewer
instances used for training. This also shows that the size of
training data is another crucial factor to the final accuracy.
Another interesting phenomenon is that the differences
between student systems with different architecture are notso big compared with systems trained with an original reference. In this experiment, rnn.M performs well comparedwith tfm.S given synthesised training data, while it is not thecase for them to be trained with original data. We speculatethat this is because the diversity in distilled bi-text is much
5 During translating, under translation is when the words/phrases inthe source sentence are missing (not translated) in the target side. Overtranslation is when there are duplicated translations for the same sourcewords/phrases present on the target side.
6 https://github.com/jmcrego/similarity

27
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

tfm.S

rnn.M

merged data(hyp+ref)95%+5%
90%+10%80%+20%50%+50%95%+5%
90%+10%80%+20%50%+50%

newstest
2014
26.27
26.20
25.76
25.58
25.94
25.52
25.66
25.04

newstest
2015
29.09
29.11
28.88
28.54
28.83
28.76
28.44
27.60

tfm.S
rnn.M

2X data(hyp+ref)100%+100%100%+100%

newstest
2014
25.95
25.78

newstest
2015
28.74
28.85

Table 7: Results of the concatenated synthesised data andoriginal reference. Twice the training cost is needed as the
corpus is doubled.
6.5. Doubled training data

Table 6: Results of merged corpus with synthesised data andoriginal data. 95%+5% means the training data is composedof 95% data from the synthesised target translations (hypothesis) according to the similarity and additional 5% data fromoriginal target data (reference).
less than in the original reference. Systems with different architecture show different sensitivity of data diversity. Thatis also to say, the distilled bi-text is consistent and compact,which is much suitable for training machine translation systems.
6.4. Replacing noisy data with the original reference
Previous experiments show that systems trained with synthesised data usually perform better than systems trained withoriginal reference data. At the same time, when we remove
some noisy data from the synthesised training set, there isfurther improvement for the student system. In this section,
we test experiments with merged synthesised data and theoriginal reference as the training corpora to see which part ismore crucial for the final performance.
First, we use a similarity score between source and target sentences calculated beforehand to rank the synthesised
data. We select top X% “similar” data and for the remaining(1 − X%) data, we replace the target side with the originalreferences to merge into a new data set. Table 6 shows theevaluation results on two student systems tfm.S and rnn.M.
Results show that synthesised data greatly contribute to
the final accuracy. Along with the increase of data from thesynthesised target side, the performance increases as well forboth tfm.S and rnn.M. However, when comparing with systems trained with 100% synthesised data or systems trainedwith 95% synthesised data, the performance is different between the Transformer and RNN based models.
For tfm.S, when training with merged 95% data, the system reaches its highest performance in newstest2014. As for
rnn.M, on the contrary, the performance starts to drop a little. It is even worse than training with the filtered 95% synthesised data. We argue this inconsistency stems from thearchitecture differences. The performance of the RNN basedmodel is difficult to be improved. However, the Transformerbased model, due to its sensitivity to data diversity, can perform quite well as long as the training data is well controlled.

Lastly, we combine the synthesised data with the original reference. This will double the training data size and lead totwice the training cost. However, based on the results shownin Table 7, we found that even though the data size was doubled, we could not achieve further improvement.
We analyse that this is because the synthesised data is
generated from the teacher system. All the information embedded in the synthesised data is already in the original data.
In other words, adding such synthesised data is somewhat
equivalent to adding the same original data. It is similar totraining the system with the same data but with a 2X data
size. Furthermore, considering noise existed in the synthesised data, systems trained with this 2X data is even worse
than the original doubled data.

7. Conclusions
We have presented distillation experiments for neural machine translation. Results indicate the suitability of usingsynthetic translations to train neural MT systems. Higher
accuracy results are obtained by the systems when trainedusing synthetic data. We show data noise present in both theoriginal translation references and synthesised translations isa key factor that influence the final performance.
Meanwhile, the Transformer-based and RNN-based systems perform differently given different amounts of synthesised and/or merged data. We further prove that much“closer” translations contribute the most to the system’s accuracy and that is also the reason why distillation works forneural machine translation.
In conclusion, we summarise that for neural machinetranslation:- Having a stronger teacher system usually helps the student systems.
- Removing noise from the synthesised data of teacher
systems also helps.
- Replacing noisy data with the original reference data
can get further improvements.
A clear drawback of distillation-based methods is the efficiency of the training process. Student systems must be
trained after the teacher system. In addition, we must also
consider the cost of translating the entire training set. As

28
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

such, one solution is to integrate this procedure during thetraining process. Since data noise is one of the key factors
during training, we believe that identifying noisy instances
during training may alleviate the time problem. We leavethat for future work.

8. Acknowledgements
We would like to thank all the anonymous reviewers for theirinsightful comments.

9. References[1] P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention
with relative position representations,” CoRR, vol.
abs/1803.02155, 2018. [Online]. Available: http://arxiv.org/abs/1803.02155[2] O. Bojar, Y. Graham, A. Kamran, and M. Stanojević,“Results of the wmt16 metrics shared task,” in Proceedings of the First Conference on Machine Translation,
Berlin, Germany, August 2016.
[3] Y. Vyas, X. Niu, and M. Carpuat, “Identifying
semantic divergences in parallel text without annotations,” in Proceedings of the 2018 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies,
Volume 1 (Long Papers). Association for Computational Linguistics, 2018, pp. 1503–1515. [Online].
Available: http://aclweb.org/anthology/N18-1136[4] H. Schwenk, “Filtering and mining parallel data in ajoint multilingual space,” in Proceedings of the 56th
Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers). Association for
Computational Linguistics, 2018, pp. 228–234. [Online]. Available: http://aclweb.org/anthology/P18-2037[5] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” in Proceedings of the 54th Annual Meeting of
the Association for Computational Linguistics (Volume
1: Long Papers). Association for Computational
Linguistics, 2016, pp. 86–96. [Online]. Available:
http://www.aclweb.org/anthology/P16-1009[6] Y. Kim and A. M. Rush, “Sequence-level knowledge
distillation,” Proceedings of the 2016 Conference on
Empirical Methods in Natural Language Processing,pp. 1317–1327, November 2016.
[7] J. M. Crego and J. Senellart, “Neural machinetranslation from simplified translations,” CoRR, vol.
abs/1612.06139, 2016. [Online]. Available: http://arxiv.org/abs/1612.06139[8] M. Freitag, Y. Al-Onaizan, and B. Sankaran, “Ensemble distillation for neural machine translation,”

CoRR, vol. abs/1702.01802, 2017. [Online]. Available:
http://arxiv.org/abs/1702.01802[9] A. Birch, A. Finch, M.-T. Luong, G. Neubig, andY. Oda, “Findings of the second workshop on neural machine translation and generation,” arXiv preprint
arXiv:1806.02940, 2018.
[10] G. E. Hinton, O. Vinyals, and J. Dean, “Distillingthe knowledge in a neural network,” CoRR, vol.
abs/1503.02531, 2015. [Online]. Available: http://arxiv.org/abs/1503.02531[11] J. H. Wong and M. J. Gales, “Sequence student-teacher
training of deep neural networks,” 2016.
[12] Y. Liu, W. Che, H. Zhao, B. Qin, and T. Liu, “Distillingknowledge for search-based structured prediction,”
CoRR, vol. abs/1805.11224, 2018. [Online]. Available:
http://arxiv.org/abs/1805.11224[13] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,”
CoRR, vol. abs/1409.0473, 2014, demoed at NIPS
2014: http://lisa.iro.umontreal.ca/mt-demo/. [Online].
Available: http://arxiv.org/abs/1409.0473[14] W. Zaremba, I. Sutskever, and O. Vinyals,“Recurrent neural network regularization,” CoRR,
vol. abs/1409.2329, 2014. [Online]. Available:
http://arxiv.org/abs/1409.2329[15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin,“Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 6000–6010.
[16] T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machinetranslation,” in Proceedings of the 2015 Conferenceon Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational
Linguistics, September 2015, pp. 1412–1421. [Online].
Available: http://aclweb.org/anthology/D15-1166[17] J. Crego, J. Kim, G. Klein, A. Rebollo, K. Yang,
J. Senellart, E. Akhanov, P. Brunelle, A. Coquard,
Y. Deng, S. Enoue, C. Geiss, J. Johanson, A. Khalsa,
R. Khiari, B. Ko, C. Kobus, J. Lorieux, L. Martins,
D. Nguyen, A. Priori, T. Riccardi, N. Segal, C. Servan,
C. Tiquet, B. Wang, J. Yang, D. Zhang, J. Zhou, andP. Zoldan, “Systran’s pure neural machine translation
systems,” CoRR, vol. abs/1610.05540, 2016. [Online].
Available: http://arxiv.org/abs/1610.05540[18] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,”
arXiv preprint arXiv:1508.07909, 2015.

29
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[19] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, et al., “Moses: Open source toolkit
for statistical machine translation,” in Proceedings ofthe 45th annual meeting of the ACL on interactive
poster and demonstration sessions. Association for
Computational Linguistics, 2007, pp. 177–180.
[20] M. Q. Pham, J. Crego, J. Senellart, and F. Yvon, “Fixing translation divergences in parallel corpora for neural mt,” Conference on Empirical Methods in Natural
Language Processing, 2018.

30
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Word-based Domain Adaptation for Neural Machine Translation
Shen Yan, Leonard Dahlmann, Pavel Petrushkov, Sanjika Hewavitharana, Shahram Khadivi
eBay Inc.
{shenyan,fdahlmann,ppetrushkov,shewavitharana,skhadivi}@ebay.com

Abstract
In this paper, we empirically investigate applying word-level
weights to adapt neural machine translation to e-commercedomains, where small e-commerce datasets and large outof-domain datasets are available. In order to mine indomain like words in the out-of-domain datasets, we compute word weights by using a domain-specific and a nondomain-specific language model followed by smoothing andbinary quantization. The baseline model is trained on mixed
in-domain and out-of-domain datasets. Experimental results on En → Zh e-commerce domain translation show thatcompared to continuing training without word weights, it
improves MT quality by up to 3.11% BLEU absolute and1.59% TER. We have also trained models using fine-tuningon the in-domain data. Pre-training a model with word
weights improves fine-tuning up to 1.24% BLEU absoluteand 1.64% TER, respectively.

1. Introduction
Domain adaptation (DA) techniques in machine translation(MT) have been widely studied. For statistical machinetranslation (SMT), several DA methods have been proposed
to overcome the lack of domain-specific data. For example, self-training [1, 2] uses a MT system trained on general
corpus to translate in-domain monolingual data as additional
training sentences. Topic-based DA [3, 4] employs topicbased translation models to adapt for different scenarios.
Data selection approaches [5, 6, 7, 8] first score the out-ofdomain data using language model trained on both domainspecific and non-domain-specific monolingual corpora, thenrank and select the out-of-domain data that are similar toin-domain data. Instance weighting methods [9, 10] scoreeach sentence/domain using statistical rules, then train the
MT models by giving sentence/domain-level scores.
Neural machine translation (NMT) has become state-ofthe-art in recent years [11, 12, 13, 14, 15]. There are several research works on NMT domain adaptation. For example, back-translation methods [16] use a NMT model trainedon the reverse direction to translate domain-specific monolingual data as additional training sentences. Fast DA approaches [13, 17] train a base model using mixed in-domain
and out-of-domain datasets, then fine-tuning on in-domain
datasets. Mixed fine-tuning [18] combines fine-tuning andmulti-domain NMT. Similar to instance weighting in SMT,

sentence/domain weighting methods [19, 20] can also be
used for NMT domain adaptation based on different objectives. DA with meta information [21] is proposed to train
topic-aware models using domain-specific tags for the decoder. Chunk weighting method [22] describes a way of selecting and integrating positive partial feedback from modelgenerated sentences into NMT training.
In this paper, we propose word-level weighting for NMT
domain adaptation. We compute the word weights in out-ofdomain datasets based on the logarithm difference of probability according to a domain-specific language model andnon-domain-specific language model followed by smoothing
and binary quantization. This gives the in-domain words inout-of-domain sentences higher weights and biases the NMT
model to generate more in-domain-like words. Thus, thework presented in this paper can be viewed as a generalization of instance weighting. To remove noise in the word
weights, we study the effectiveness of using smoothing methods. Specifically, a weighted moving average filter is proposed to apply smoothing to the computed word scores with
its nearby words.
Experiments on En → Zh e-commerce domain translations tasks show that: 1) Domain adapted model withsmoothed word weights gains significant improvement overnon-smoothed weights; 2) Continuing training the modelwith computed word weights improves translation results
significantly compared to continuing training without word
weights; and 3) Compared to directly fine-tuning on indomain datasets, fine-tuning after pre-training with word
weights results in translation performance improvement on
the in-domain e-commerce test set.
The rest of the paper is structured as follows. The approach and model we use is described in Section 2, where wefirst recap the NMT objective and then present the details of
the proposed word-level weighting approach. Experimental
results and discussions are presented in Section 3 and Section
4, followed by conclusions and outlook in Section 5.

2. Approach
We present word weighting objective on NMT before discussing how to generate the weights.

31
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

2.1. Objective
In this work we use attention-based neural machine translation model [11, 12, 14] for experiments. Given a parallelbilingual dataset D, the NMT model is trained to maximize
the conditional likelihood L of a target sequence y1T : y1 , . . .
, yT given a source sequence xN
1 : x1 , . . . , xN :
L=

X

T
X

T
t=1
(xN
1 ,y1 )∈D

log p(yt |y1t−1 , xN
1 )

(1)

Training objective (1) can be simply modified to wordlevel loss Lw with word weights wt :

Lw =

X

T
X

T
T
t=1
(xN
1 ,y1 ,w1 )∈D

wt log p(yt |y1t−1 , xN
1 )

(2)

The word weights wt for a target sequence y1T can be 0
or 1. We set wt = 1 for all in-domain sentences. For outof-domain sentences, wt = 1 means the word in the out-ofdomain sentence is related to in-domain datasets (selected),
wt = 0 means it is not.
Our training objective (2) can be seen as a generalizationof the original training objective (1) and instance weighting
methods [19, 20]. The original loss (1) sets wt = 1 for every word in all sentences. The instance-level loss can be expressed as giving a target sentence, wt = w ∀t, where w
is the weight for the sentence or the domain. Our training
objective is similar to [22], however, instead of generating
chunk-based user feedback for model predictions, we compute the word weights using language models trained on realtarget data.
2.2. Approaches to the objective
To compute discriminative word weights, we first follow thedata selection methods in SMT [5]. To state this formally,
let I be the domain-specific corpus, O be the non-domainspecific corpus, and yt be the word in out-of-domain sent−1
tences at target position t. We denote by PI (yt |yt−n) theper-word probability conditioned on previous n − 1 words,
according to a language model trained on I. Similarly, wet−1
denote by PO (yt |yt−n) the per-word probability conditioned
on previous n − 1 words according to a language modelt−1
t−1
trained on O. We can estimate PI (yt |yt−n) and PO (yt |yt−n)by training language models on I and O, separately. Therefore, the word scores st can be computed in the log domain:t−1
t−1
st = log PI (yt |yt−n) − log PO (yt |yt−n)(3)

Since the value of st is strongly correlated with the neighborhood words, it is worth investigating smoothing of theword scores before binary thresholding to remove the noise.
Hence, a weighted moving average kernel:

sˆt =

⌊L
2⌋

X

(4)

ck st+kk=⌊ −L2 ⌋is then applied to smooth word score st at each target
position t. Here L is the kernel size and ck are values of theLkernel for k ∈ [ −L
2 , 2 ]. In our experiments, we heuristically
set the values of the kernel based on mean average with ck =−k2

1
c = L1 or gaussian distribution with ck = √2πσ
e 2σ2 , where
we set σ to be the global variance of the word scores.
The special case of sentence-level weights can be expressed as sˆt = ŝ ∀t, where ŝ is the averaged smoothed wordscores for the target sentence y1T . In this case, the training objective (2) becomes equivalent to sentence weighting methodfrom [20] with appropriately modified scoring function.
After smoothing the word scores, we finally binarize thesmoothed word scores based on a threshold T :(1, if sˆt ≥ Twt =(5)0, otherwise

In our experiments we set the threshold T = 0.5 and only
keep the words above the threshold. This means we select aword if wt = 1 and do not select it if wt = 0. Considering
word weights wt are gathered in a binary form during continuing training, the selected words would be good candidates
that we want to extract from out-of-domain corpus O. In fact,
word weights wt are precomputed offline and used during thetraining. It can be set to any real value, depending on the wayof thresholding.
2.3. Chunk-based weighting
Considering that the selected words in a target sentence
might still be noisy and we select single random words,
we alternatively experimented with selecting only the part(chunk) in the target sentence that has the longest consecutive weights (LCW) with wt = 1. For each target sentence, we pick only one chunk and set all other weights tozero. See Figure 1 for an example. Then, because the surrounding context is also selected, the chunk is less likely to
be noise. If there are multiple such chunks with the samelength in the sentence, we simply randomly sample one ofthem. We found that the chunk-based approach in practiceperforms slightly better than word-level weighting.

3. Experiments
In this section, we conduct a series of experiments to study
how well NMT performs when word-level weights are given
for out-of-domain training data. We also study the effectiveness of the smoothing methods.
3.1. Datasets and data processing
We report the results on our in-house English-to-Chinese ecommerce item descriptions dataset. Item descriptions are

32
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

provided by private sellers and like any user-generated content, may contain ungrammatical sentences, spelling errors,
and other type of noise. We first segmented the Chinese sentences with Stanford Chinese word segmentation tool [23]and tokenized English sentences with the scripts provided
in Moses [24]. On both languages, we use subword units
based on byte-pair encoding (BPE) [25] with 42,000 subword symbols learned separately for each language. For
En-Zh we have 0.53M in-domain e-commerce sentence pairs
and 5.15M sampled out-of-domain sentence pairs (UN, subtitles, TAUS data collections, etc.) that have significant ngram overlap with the item description data. We validate ourmodels on an in-house development set consisting of 3173
item descriptions, and evaluate on an in-house test set of
739 item descriptions using case-insensitive character-level
BLEU [26] and TER [27] with in-house tools. For development and test sets, a single reference translation is used.
Statistics of the data sets are reported in Table 1.
To compute our word weights we train a domain-specific
4-gram language model and a non-domain specific 4-gram
language model using KenLM [28]. For the domain-specificlanguage model, we collected domain-specific monolingual
data from an e-commerce website, resulting in the numberof 15M sentences. For the non-domain-specific language
model, we use sampled LDC Chinese Gigaword (LDC Catalog No.: LDC2003T09) with 36M sentences. It should be
noted that we train our language models on the word-level.
In order to score a BPE-level corpus with such a language
model, we score its words and copy this score for each of the
subword units. After the word scores are computed, we thensmooth them via a guassian distributed kernel with window
size L = 5. We choose window size L = 5 considering thatthe language model is trained based on sequences of fourwords. We observed similar results with different window
sizes, which is discussed in Section 4. Finally, we binarize
the smoothed word scores into binary word weights by setting the threshold T = 0.5. The computed word weights areapplied to the target side of out-of-domain sentences during
the phase of continuing training. In order to get better translation results, we first trained the baseline model with mixed
in-domain and out-of-domain data according to training objective 1, where no weights are used. We start our experiments by continuing training from this baseline model.
We implemented our NMT model using Tensorflow [29]library. The encoder is a bidirectional LSTM with size of 512
and the decoder is a LSTM with 2 layers of same size. All theweight parameters are initialized uniformly in [−0.1, 0.1].
We set dropout on RNN inputs with dropping probability 0.2.
We train the networks with batch size 120 using SGD withinitial learning rate 1.0 and gradually decaying to 0.1 after
the initial 2 epochs.
3.2. Results
Statistics of the out-of-domain sentences/tokens selection after applying different types of weights are summarized in Ta-

ble 2. Before the selection, the number of out-of-domain
sentences is 5.15M and the number of tokens is 93.4M.
When sentence-level weights are used, the sentences with
wt = 0 are ignored, resulting in the number of remaining sentences/tokens around 2.63M and 26.3M, respectively.
When word-level weights are used, there are 1, 279, 927 sentences where all word weights in the sentences are equal tozero. After removing these sentences, around 3.87M sentences are preserved and the number of selected tokens withword weights wt =1 is around 36.6M. Given computed word
weights, we alternatively choose only the chunk with thelongest consecutive weights (LCW) where wt = 1, resulting
in chunk-level weights with the selected number of tokens
further reduced to 25.8M.
We train a baseline NMT model on mixed in-domain andout-of-domain data with objective defined as Eq. 1 for 6
epochs. The data is mixed completely (mixed 0.53M indomain e-commerce and 5.15M sampled out-of-domain sentence pairs) while training the baseline model. The baselinemodel initialized by a mix of in-domain/out-of-domain data
can be regarded as a kind of ”warm start”. We have also
tried training a baseline with out-of-domain data only andobserved slightly worse result after fine-tuning on in-domain
data (0.5 BLEU). Hence, we use the baseline model trainedon a mix of in-domain/out-of-domain data in the following
experiments. Given the baseline model, we then directly finetune on in-domain data for another 10 epochs or first continue training on the mixed data with sentence/chunk/word
weights for 3 epochs and then fine-tune on in-domain datafor 10 epochs. The model is saved after each epoch. We takethe model which gives the best result on our development set
for evaluation. Note that we always set word weights wt = 1
for our in-domain dataset.
In Table 3, we show the effect of different types ofweights on translation performance. First, the baseline
trained on mixed in-domain and out-of-domain datasets gives
24.37% BLEU and 61.66% TER, respectively. Directly finetuning on in-domain dataset already improves the model dueto the bias of the model towards in-domain data.
Continuing training on mixed datasets with previous objective defined in Eq. 1 shows insignificant changes in termsof BLEU and TER. However, introducing sentence-level
weights improves the model from 24.37% to 25.79% BLEU
and 61.66% to 60.82% TER, respectively. Compared to continuing training without weights, sentence-level weights are
generated as described in Section 2.2, where wt ∀t are set tothe same sentence weight w ∈ {0, 1} . We set the thresholdequal to 0.5 and keep the sentences with weights above thethreshold. The result from sentence-level feedback suggeststhat mining good out-of-domain sentences which are similar to in-domain datasets and dissimilar to out-of-domain
datasets benefits model translation towards in-domain-like
sentences even without fine-tuning on in-domain datasets.
The use of word-level weights improves the baselinemodel even better, from 24.37% to 26.14% BLEU and

33
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Data set
Language
SentencesTraining Running words
BPE vocabulary
Sentences
Dev
Running words
Sentences
Test
Running words

e-commerce + out-of-domain
EnglishChinese5,689,989
97,266,344
96,480,106
33,48445,8673173 (item descriptions)51,13048,900739 (item descriptions)
19,034
18,262

Table 1: Corpus statistics for the e-commerce English→Chinese MT tasks.
Corpusood. sentences+sent. weights+word weights+chunk weights

Sent. count5,153,191
2,633,109
3,873,264
3,873,264

Token count93,427,867
26,275,096
36,617,395
25,813,480

Table 2: Out-of-domain training corpus statistics. ood. sentences indicates the number of sentences/tokens in the outof-domain corpus. +sent. weights indicates the numberof selected out-of-domain sentences where the weights ofthe sentences are equal to 1. +word weights and +chunkweights indicate the statistics of selected out-of-domain sentences/tokens after applying word weights generation and
LCW methods as described in Section 2.2 and 2.3.
61.66% to 60.34% TER, respectively. In this approach, thenumber of selected tokens is drastically reduced to 36.6M
from 93.4M tokens, nearly 61% drop in number of tokens
with improved translation performance. Word-level weights
also outperform sentence-level weights by 0.35% in BLEU
score and 0.48% in TER. It can be explained by the fact thateach word in the sentences are given its own similarity to thein-domain datasets. Considering sentence-level weights set
all words in a sentence with the same weight, even thoughpart of the words in the sentences might not be related to thein-domain corpus, word-level weights are more accurate andeffective.
Finally, chunk-level weights are generated from ourword-level weights based on LCW. Here we aim to train
the domain-adapted model from more consecutive segments
rather than single selected words. On top of word-level
weights, it improves by another 0.28% BLEU absolute and0.24% TER, respectively. Out-of-domain sentences can be
split into chunks which can be related to the in-domain
and can be translated independently in terms of the context.
The selection of the consecutive chunk with in-domain-like
context can positively affect the training towards domainadapted model. By focusing on in-domain related and outof-domain unrelated part, word/chunk-level weights can effectively reduce the unnecessary noise in the out-of-domain
training data. Compared to continuing training without word
weights, we are able to further reduce the corpus by 72%

tokens (25.8M vs. 93.4M selected tokens), resulting in an
improvement of 2.11% BLEU absolute and 1.59% TER, respectively. It should also be noted that with similar numberof tokens (25.8M vs. 26.3M), chunk-level weights outperforms sentence-level weights by 0.63% BLEU absolute and0.72% TER.
Next, we further fine-tune the model with chunk-level
weights and obtain further improvements of 0.88% BLEU
absolute and 1.81% TER. Compared to directly fine-tuningon the baseline, continuing training the model with chunklevel weights and then fine-tuning improves translation results from 26.06% to 27.30% BLEU and 59.93% to 58.29%
TER, respectively.
Results from the study on the effect of using different
smoothing methods are shown in Table 4. The word weights
generated without using smoothing methods, where sˆt = st ,lead to poor translation quality of 21.38% from 24.37%
BLEU and 66.25% from 61.66% TER, respectively. We needto smooth the word scores before thresholding because thet−1
t−1
values of log PI (yt |yt−n) − log PO (yt |yt−n) are noisy. If
there are selected isolated words like ’,’ which have higherscores than the surrounding text, it may cause rare vocabulary problem after training.
The results from word weights computed from mean averaged filter and normal distributed filter are relatively close,
25.99% vs. 26.14% BLEU and 60.70% vs. 60.34% TER,
respectively. These results are obtained via a filter with window size L = 5. In practice, we also tried setting window
size L = 3 and L = 7, but didn’t observe different results.
We found that the surrounding word scores have to be considered for smoothing in order to make the word weights wt
less noisy as well as more precisely representing the similarity to the in-domain/out-of-domain.
Additionally, we also experimented with randomly selecting words in the out-of-domain sentences with binary
mask. However, we observed a drop in the translation accuracy.
3.3. Examples
In Table 5, we show an example for which the system trainedwith word weights produces a better translation. The English sentence is ”non-spill spout with patented valve”. The

34
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

No.
1
2
3
4
5
6
7

System description
Baseline
1 + continue training without word weights
1 + continue training with sentence weights
1 + continue training with word weights
1 + continue training with chunk weights
1 + fine-tuning on in-domain
5 + fine-tuning on in-domain

Item descriptions
BLEU [%] TER [%]
24.37
61.66
24.31
61.69
25.79
60.82
26.14
60.34
26.42
60.10
26.06
59.93
27.30
58.29

Table 3: E-commerce English → Chinese BLEU results on test set. Baseline is trained on mixed in-domain and out-of-domain
data. No. 2 is continuing training from baseline with objective defined as Eq. 1. No. 3 is continuing training from baseline withsentence-level weights and No. 4 is with word weights, as defined in Section 2.2. No. 5 refers to assigning wt using LCW method
described in Section 2.3. No. 6 is equivalent to directly fine-tuning on in-domain datasets starting from the baseline model andNo. 7 is equivalent to fine-tuning on in-domain datasets after No. 5 is finished.
SystemBaseline+w.w. without smooth.
+w.w. (mean smooth.)+w.w. (gauss. smooth.)

BLEU [%]
24.37
21.38
25.99
26.14

TER [%]
61.66
66.25
60.70
60.34

Table 4: Study on the effect of different smoothing methods for word weights generation. Baseline is the same as
before. w.w. without smoothing means the word weights(w.w.) are computed without smoothing in the log domain.
w.w. (mean smooth.) indicates smoothing the word scores
via using a mean average filter before thresholding and w.w.
(gauss. smooth.) indicates using a normal distributed filter before thresholding. The approaches regarding different
smoothing methods are described in Section 2.2.
word ”spout” is rare in our data, appearing in the out-ofdomain training sentences only once. The Chinese side ofthis training example can be seen in Figure 1 together withthe weights assigned to the individual words by our method.
When smoothing is applied, isolated Chinese words such as”空气” (”air”) are removed. With the longest consecutive
words (LCW) method, the only remaining chunk is ”防/溢出/喷口/内” (”inside the non-spills spout”), which is related
to our in-domain data. The system with word weights is thentrained only on this chunk on the target side, while the baseline model is trained on the entire sentence and generates
inappropriate translations.

4. Discussions
The domain adaptation techniques (sentence-level/chunklevel/word-level) introduced in this paper are all derived fromword weights generation. They aim to select out-of-domain
sentences/chunks/words which are more related to in-domain
corpus and unrelated to out-of-domain corpus. The word
weights are computed prior to system tuning via the logarithm difference of LM probability scoring and are then used

for tuning the sequence-to-sequence model. By measuring
domain similarity with external criteria such as LM, this kindof out-of-domain data selection is able to highlight the indomain-related and out-of-domain-unrelated parts and leads
to less variation and errors in our e-commerce domain adaptation. In addition, the selected out-of-domain segments haveto be smoothed in order to reduce noise.

5. Conclusions
In this work, we generate word-level weights by calculating the logarithm difference of the probability of two external language models for domain adaptation. This approach better selects the out-of-domain segments related to
e-commerce domain, and requires fewer tokens for training.
We experimented with continuing training models with sentence/chunk/word weights and show that they all give translation improvement in terms of BLEU and TER compared tocontinuing training without word weights. Experiments on
our in-house English-Chinese datasets also show that continuing training with word weights then fine-tuning improves
results over directly fine-tuning on baseline model.
In future, with the computed word weights as the initial parameters, we want to devise strategies to make online
domain adaptation possible by dynamically updating word
weights during training, which could in turn lead the indomain data translation to better match its references.

6. References[1] N. Ueffing and H. Ney, “Word-level confidence
estimation for machine translation using phrasebased translation models,” in Proceedings of theConference on Human Language Technology and
Empirical Methods in Natural Language Processing,ser. HLT ’05. Stroudsburg, PA, USA: Association
for Computational Linguistics, 2005, pp. 763–770.
[Online]. Available: https://doi.org/10.3115/1220575.
122067135
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

SystemBaseline+ word weights
Reference

Translation带专利阀的防溢出溅漏带专利阀的防溢出喷口带有专利阀门的防溢口

Table 5: Translations of ”non-spill spout with patented valve” produced by the baseline NMT system and the system trained withword weights. The last row shows the reference translation.

logarithm differencesmoothed score

3

2

score

1
0.5
0

-1

。

排出

通⽓口

底部

通过

则

液体

⽽

,

瓶⼦

流⼊

空⽓

允许

排⽓口

顶部

,

喷口

个

两

有

内

喷口

溢出

防

在

-2

Figure 1: Our approach for word-based domain adaptation, with the target sentence on the bottom (the source sentence is ”Inside
the non-spills spout there are two vents, where the top vent allows air to flow into the bottle while liquid smoothly pours out
through the bottom vent.”). Above are displayed the word scores and the smoothed word scores. The black bold line indicates
the threshold and the circle indicates which chunk is selected by LCW. After smoothing, the isolated random words are removed
and the red words on the bottom are selected. The red bold words on the bottom are preserved after LCW.

36
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[2] H. Schwenk, “Investigations on large-scale lightlysupervised training for statistical machine translation,”in IWSLT, 2008.
[3] Y.-C. Tam, I. R. Lane, and T. Schultz, “Bilingual-lsa
based lm adaptation for spoken language translation,”in ACL, 2007.
[4] S. Hewavitharana, D. Mehay, S. Ananthakrishnan,
and P. Natarajan, “Incremental topic-based translation
model adaptation for conversational spoken languagetranslation,” in ACL, 2013.
[5] R. C. Moore and W. Lewis, “Intelligent selection of
language model training data,” in Proceedings of the
ACL 2010 Conference Short Papers, ser. ACLShort ’10.
Stroudsburg, PA, USA: Association for Computational
Linguistics, 2010, pp. 220–224. [Online]. Available:
http://dl.acm.org/citation.cfm?id=1858842.1858883[6] A. Axelrod, X. He, and J. Gao, “Domain adaptation
via pseudo in-domain data selection,” in Proceedingsof the Conference on Empirical Methods in Natural
Language Processing, ser. EMNLP ’11. Stroudsburg,
PA, USA: Association for Computational Linguistics,
2011, pp. 355–362. [Online]. Available: http://dl.acm.
org/citation.cfm?id=2145432.2145474[7] K. Duh, G. Neubig, K. Sudoh, and H. Tsukada, “Adaptation data selection using neural language models: Experiments in machine translation,” in ACL, 2013.
[8] N. D. H. S. S. J. A. A. S. Vogel, “Using joint models fordomain adaptation in statistical machine translation,” in
MT Summit, 2015.
[9] S. Matsoukas, A.-V. I. Rosti, and B. Zhang, “Discriminative corpus weight estimation for machinetranslation,” in Proceedings of the 2009 Conferenceon Empirical Methods in Natural Language Processing: Volume 2 - Volume 2, ser. EMNLP ’09.
Stroudsburg, PA, USA: Association for Computational
Linguistics, 2009, pp. 708–717. [Online]. Available:
http://dl.acm.org/citation.cfm?id=1699571.1699605[10] G. Foster, C. Goutte, and R. Kuhn, “Discriminative
instance weighting for domain adaptation in statisticalmachine translation,” in Proceedings of the 2010
Conference on Empirical Methods in Natural Language
Processing, ser. EMNLP ’10. Stroudsburg, PA, USA:
Association for Computational Linguistics, 2010,
pp. 451–459. [Online]. Available: http://dl.acm.org/
citation.cfm?id=1870658.1870702[11] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence
to sequence learning with neural networks,” inProceedings of the 27th International Conference
on Neural Information Processing Systems - Volume
2, ser. NIPS’14. Cambridge, MA, USA: MIT

Press, 2014, pp. 3104–3112. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2969033.2969173[12] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,”
ICLR, 2015.
[13] M.-T. Luong and C. D. Manning, “Stanford neuralmachine translation systems for spoken language domains,” in Proceedings of the International Workshop
on Spoken Language Translation : December 3-4,
2015, Da Nang, Vietnam / Edited by Marcello Federico,Sebastian Stüker, Jan Niehues. International Workshop on Spoken Language Translation, Da Nang (Vietnam), 3 Dec 2015 - 4 Dec 2015, Dec 2015.
[14] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi,
W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey,
J. Klingner, A. Shah, M. Johnson, X. Liu, Łukasz
Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa,K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young,
J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado,
M. Hughes, and J. Dean, “Google’s neural machinetranslation system: Bridging the gap between humanand machine translation,” CoRR, vol. abs/1609.08144,
2016. [Online]. Available: http://arxiv.org/abs/1609.
08144[15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,“Attention is all you need,” in NIPS, 2017.
[16] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” in ACL, 2016.
[17] M. Freitag and Y. Al-Onaizan, “Fast domain adaptation for neural machine translation,” CoRR, vol.
abs/1612.06897, 2016.
[18] C. Chu, R. Dabre, and S. Kurohashi, “An empiricalcomparison of domain adaptation methods for neuralmachine translation,” in ACL, 2017.
[19] B. Chen, C. Cherry, G. Foster, and S. Larkin,“Cost weighting for neural machine translation domain
adaptation,” in Proceedings of the First Workshop
on Neural Machine Translation. Association for
Computational Linguistics, 2017, pp. 40–46. [Online].
Available: http://aclweb.org/anthology/W17-3205[20] R. Wang, M. Utiyama, L. Liu, K. Chen, andE. Sumita, “Instance weighting for neural machinetranslation domain adaptation,” in Proceedings of the2017 Conference on Empirical Methods in Natural
Language Processing. Association for Computational
Linguistics, 2017, pp. 1482–1488. [Online]. Available:
http://aclweb.org/anthology/D17-1155

37
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[21] S. Khadivi, P. Wilken, L. Dahlmann, and E. Matusov,“Neural and statistical methods for leveraging metainformation in machine translation,” in MT Summit,
2017.
[22] P. Petrushkov, S. Khadivi, and E. Matusov, “Learningfrom chunk-based feedback in neural machine translation,” in ACL, 2018.

for large-scale machine learning,” in Proceedings ofthe 12th USENIX Conference on Operating Systems
Design and Implementation, ser. OSDI’16. Berkeley,
CA, USA: USENIX Association, 2016, pp. 265–283.
[Online]. Available: http://dl.acm.org/citation.cfm?id=3026877.3026899[23] P.-C. Chang, M. Galley, and C. D. Manning,“Optimizing chinese word segmentation for machinetranslation performance,” in Proceedings of the Third
Workshop on Statistical Machine Translation, ser.
StatMT ’08. Stroudsburg, PA, USA: Association
for Computational Linguistics, 2008, pp. 224–232.
[Online]. Available: http://dl.acm.org/citation.cfm?id=
1626394.1626430[24] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin,
and E. Herbst, “Moses: Open source toolkit for
statistical machine translation,” in Proceedings ofthe 45th Annual Meeting of the ACL on Interactive
Poster and Demonstration Sessions, ser. ACL ’07.
Stroudsburg, PA, USA: Association for Computational
Linguistics, 2007, pp. 177–180. [Online]. Available:
http://dl.acm.org/citation.cfm?id=1557769.1557821[25] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,”
CoRR, vol. abs/1508.07909, 2016.
[26] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,
“Bleu: A method for automatic evaluation of machinetranslation,” in Proceedings of the 40th Annual
Meeting on Association for Computational Linguistics,
ser. ACL ’02. Stroudsburg, PA, USA: Association
for Computational Linguistics, 2002, pp. 311–318.
[Online]. Available: https://doi.org/10.3115/1073083.
1073135[27] M. Snover, B. J. Dorr, R. F. Schwartz, and L. Micciulla,“A study of translation edit rate with targeted humanannotation,” 2006.
[28] K. Heafield, “Kenlm: Faster and smaller language
model queries,” in Proceedings of the Sixth Workshop
on Statistical Machine Translation, ser. WMT ’11.
Stroudsburg, PA, USA: Association for Computational
Linguistics, 2011, pp. 187–197. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2132960.2132986[29] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis,
J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard,
M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G.
Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden,
M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: A system

38
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

A Machine Translation Approach for Modernizing Historical Documents Using
Backtranslation
Miguel Domingo, Francisco Casacuberta
PRHLT Research Center
Universitat Politècnica de València
midobal@prhlt.upv.es, fcn@prhlt.upv.es

Abstract
Human language evolves with the passage of time. This
makes historical documents to be hard to comprehendby contemporary people and, thus, limits their accessibility to scholars specialized in the time period in whicha certain document was written. Modernization aims at
breaking this language barrier and increase the accessibility of historical documents to a broader audience. To
do so, it generates a new version of a historical document, written in the modern version of the document’s
original language. In this work, we propose several
machine translation approaches for modernizing historical documents. We tested these approaches in different
scenarios, obtaining very encouraging results.

1. Introduction
Historical documents are an important part of our cultural heritage. With the aim of their preservation, there
is an increase need of digitalazing—creating a digitaltext version which can be searched and automaticallyprocessed— of historical documents [1]. However, there
is an additional difficulty created by their linguistic properties: On the one hand, human language evolves withthe passage of time. On the other hand, due to a lack of aspelling convention, orthography changes depending on
the author and time period in which a given document
was written. These problems make historical documents
harder to read, and makes it harder to digitalize them(since their digital text version needs to be searched andautomatically processed).
The orthography problem has been well researchedin the literature [2, 3, 4, 5, 6, 7]. The proposed solution that aims to solve this problem is known as spellingnormalization, and its goal is to adapt the document’s
spelling to modern standards in order to achieve an orthography consistency and increase the document’s read-ability. However, while is true that spelling normalization makes historical documents easier to read, they arestill hard to comprehend by contemporary people. This
problem limits the accessibility of historical documents
to scholars specialized in the time period in which the
document was written.
Modernization aims at breaking the language barrier,generating a new version of a historical document in
the modern version of the language in which the document was originally written (see Fig. 1 for an example).
Therefore, not only the orthography is updated. The
lexicon and grammar are also modified in order to matchthe modern use of the document’s language. The maindrawback of this solution is that part of the document’s
original intention could be lost in the process (e.g., partof the rhyme in Fig. 1 is lost for the sake of clarity).
Nonetheless, the document’s clarity is increased and,thus, its accessibility to a broader audience.
To the best of our knowledge, modernization ofhistorical documents has been less researched in theliterature. A shared task was organized in order totranslate historical text to contemporary language [9].
The shared task’s main goal was spelling normalization.
However, they also tackle modernization using a set ofrules. Finally, there was an approach to modernize historical documents using Statistical Machine Translation(SMT) [10]. In this work, we tackle modernization byusing an SMT and Neural Machine Translation (NMT)
approach. Additionally, since a frequent problem whenworking with historical documents is the scarce availability of parallel training data [5], we created two small parallel corpora (see Section 3.1) and generated synthetic
data using backtranslation [11]. Our main contributions
are the followings:• First use, to the best of our knowledge, of NMT
and backtranslation for historical documents mod-39
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Shall I compare thee to a summer’s day?
Thou art more lovely and more temperate:
Rough winds do shake the darling buds of May,
And summer’s lease hath all too short a date:
Sometime too hot the eye of heaven shines,
And often is his gold complexion dimm’d;
And every fair from fair sometime declines,
By chance or nature’s changing course untrimm’d;
But thy eternal summer shall not fade
Nor lose possession of that fair thou ow’st;
Nor shall Death brag thou wander’st in his shade,
When in eternal lines to time thou grow’st;
So long as men can breathe or eyes can see,
So long lives this, and this gives life to thee.

Shall I compare you to a summer day?
You’re lovelier and milder.
Rough winds shake the pretty buds of May,and summer doesn’t last nearly long enough.
Sometimes the sun shines too hot,and often its golden face is darkened by clouds.
And everything beautiful stops being beautiful,
either by accident or simply in the course of nature.
But your eternal summer will never fade,nor will you lose possession of your beauty,
nor shall death brag that you are wandering in the underworld,
once you’re captured in my eternal verses.
As long as men are alive and have eyes with which to see,this poem will live and keep you alive.

Figure 1: Example of modernizing a historical document. The original text is Shakespeare Sonnet 18. The modernizedversion of the Sonnet was obtained from [8].
2.2. Neural Machine Translation

ernization.
• Comparison of approaches based on SMT and
NMT.
• Experimented with three historical corpora—twoof which were created for this work—from threedifferent time periods and two different languages.
The rest of this document is structure as follows:
Section 2 introduces the different Machine Translation(MT) approaches used in our work. Then, Section 3
describes the experiments conducted in order to assess
our proposal. After that, Section 4 presents and discusses
the results of those experiments. Finally, in Section 5,
conclusions are drawn.

2. Machine Translation
In this section, we present the machine translation approaches used in our work.
2.1. Statistical Machine Translation
Given a source sentence x, SMT aims at finding its besttranslation ŷ [12]:ŷ = arg max P r(y | x)y

(1)

For years, the prevailing approach to compute thisexpression have been phrase-based models [13]. Thesemodels rely on a log-linear combination of different
models [14]: namely, phrase-based alignment models,reordering models and language models; among others [15, 16]. However, more recently, this approach has
shifted into neural models (see Section 2.2).

NMT is the neural approach to compute Eq. (1). Frequently, it relies on a Recurrent Neural Network (RNN)
encoder-decoder framework. In this framework, thesource sentence is projected into a distributed representation at the encoding step. At the decoding step, thedecoder generates its translation word by word [17].
The system’s input is a sequence of words in thesource language. Each source word is linearly projected
to a fixed-sized real-valued vector through an embedding
matrix. These word embeddings are feed into a bidirectional [18] Long Short-Term Memory (LSTM) [19] network, resulting in a sequence of annotations producedby concatenating the hidden states from the forward andbackward layers.
The model features an attention mechanism [20],which allows the decoder to focus on parts of the input
sequence, computing a weighted mean of annotations sequence. A soft alignment model computes these weights
by weighting each annotation with the previous decodingstate.
The decoder is another LSTM network, conditionedby the representation computed by the attention modeland the last word generated. Finally, a deep output
layer [21] computes a distribution over the target language vocabulary.
The model is trained by means of stochastic gradientdescend, applied jointly to maximize the log-likelihood
over a bilingual parallel corpus. At decoding time, themodel approximates the most likely target sentence withbeam-search [17].

40Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

2.3. Backtranslation
Backtranslation [11] has become the norm when building state-of-the-art NMT systems, especially in resourcepoor scenarios [22]. It is a useful technique to increase
the training data by creating synthetic text from monolingual data. Given a monolingual corpus in the target
language, and an MT system trained to translate fromthe target language to the source language, the synthetic
data is generated by translating the monolingual corpus
with the MT system. After that, the synthetic data is usedas the source part of the corpus, and the monolingual
data as the target part. Finally, this new corpus is mixed
with the available training data in order to train a new
MT system.
In this work, to generate the synthetic data, we translate the monolingual data using an ad-hoc SMT systemtrained with the corpus’ training partition. Additionally, since the datasets are considerable small, prior to
mixing the synthetic corpus with the training partition,
we replicate several times the training data in order tomatch the size of the synthetic data and avoid overfitting [23]. Finally, we trained an NMT system with thisnew corpus.

3. Experimental Framework
In this section, we present the corpora and metrics, anddescribe the MT systems used during the experimental
session.
3.1. Corpora
The first corpus used to assess our proposal was the
Dutch Bible [9]. This corpus consists in a collection ofdifferent versions of the Dutch Bible. More precisely,a version from 1637, another from 1657, another from1888 and another from 2010. All versions contain thesame text except for the 2010 version, which is missing
the last books. Moreover, the authors mentioned that thetranslation from this last version is not very reliable. Additionally, due to Dutch not evolving significantly during
this period, 1637 and 1657 versions are fairly similar.
For this reason, we decided to only use the 1637 version—considering this as the original document—andthe 1888 version—considering 19th century Dutch as
modern Dutch.
To create the synthetic corpus (see Section 2.3), wecollected all 19th century Dutch books available at the

Digitale Bibliotheek voor de Nederlandse letteren1 andused them as monolingual data.
The second corpus we used was El Quijote. We
built this corpus using a version [24] of the original 17th
century Spanish novel by Miguel de Cervantes, and a
21st century version modernized by Andrés Trapiello [25].
The first step was to split each document into sentences.
Since the 17th century version was faithful to the original manuscript (in which each document line is formed
by a very few words), we replaced line breaks by spaces
to create a single sentence, and removed empty lines.
For consistency, we did the same to the 21st century version. After that, we split each document into sentencesby adding line breaks to relevant punctuation (i.e., dots,
quotation marks, admiration marks, etc). Then, to ensure
consistency, we checked special symbols (e.g., quotation
marks) and made sure that the same character was usedin both versions. Finally, in order to create a parallel corpus, we aligned both documents using Hunalign [26].
Since the content of this corpus was a novel, we
decided that, to create the synthetic corpus, it would
be best to use monolingual data coming from Spanish
literature. For this reason and, considering that Spanish
hasn’t changed significantly over the last decades, we
decided to collect free-of-right late 20th century Spanish
novels from Project Gutenberg2 .
Finally, as a third corpus, we selected El Conde Lucanor. We built this corpus using a version of the original 14th century Spanish novel by Don Juan Manuel,and a 21st century version modernized by Luis López
Nieves [27]. To create the parallel version, we followed
the same steps than with El Quijote. However, unlikewith El Quijote, the resulting corpus was too small to beable to use for training an MT system. Therefore, we
decided to use it only as a test. Unable to find a suitabletraining corpus, we decided to test El Conde Lucanor using the systems created for El Quijote—despite the factthat the original documents were written three centuries
apart from one another.
Table 1 presents the corpora statistics.
3.2. Metrics
In order to asses our proposal, we made use of the following well-known metrics:• BiLingual Evaluation Understudy (BLEU) [28]:computes the geometric average of the modified
1
2

http://dbnl.nl/
https://www.gutenberg.org/

41
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Train

|S|
|T |
|V |

Development

|S|
|T |
|V |

Test

|S|
|T |
|V |

Monolingual

|S|
|T |
|V |

Dutch Bible

El Quijote

El Conde Lucanor

35.2K
870.4/862.4K
53.8/42.8K

10K
283.3/283.2K
31.7/31.3K-

2000
56.4/54.8K
9.1/7.8K

2000
53.2/53.2K
10.7/10.6K-

5000
145.8/140.8K
10.5/9.0K

2000
41.8/42.0K
8.9/9.0K

2252
62.0/56.7K
7.4/8.6K

4.1M
88.3M
2.0M

567.0K
9.5M
470.4K-

Table 1: Corpora statistics. |S| stands for number of sentences, |T | for number of tokens and |V | for size of the
vocabulary. Monolingual refers to the monolingual data used to create the synthetic data. M denotes million and K
thousand.
n-gram precision, multiplied by a brevity factor
that penalizes short sentences.
• Translation Error Rate (TER) [29]: computes
the number of word edit operations (insertion, substitution, deletion and swapping), normalized by
the number of words in the final translation.
Confidence intervals (p = 0.05) were computed forall metrics by means of bootstrap resampling [30].
3.3. MT Systems
We trained the SMT systems with Moses [31], following the standard procedure: we estimated a 5-gram language model—smoothed with the improved KneserNey
method—using SRILM [32], and optimized optimized
the weights of the log-lineal model with MERT [33].
Additionally, we lowercased and tokenized the corpora
using the standard scripts and, later, truecased the translated text using Moses’ truecaser.
To train the NMT systems, we used OpenNMT [34].
We used LSTM units, following the findings from [35].
We set the size of the word embedding and LSTM units
to 1024. We used Adam [36] with a learning rate of
0.0002 [37]. The beam size was set to 6. Finally, thecorpora were lowercased and tokenized—and, later, truecased and detokenized—using OpenNMT’s tools.
In order to reduce the vocabulary, we applied Byte
Pair Encoding (BPE) [38] to both SMT and NMT systems. We trained the models with a joint vocabulary of
32000 BPE units.

4. Results
In this section, we present and discuss the results of theexperiments conducted in order to assess our proposal.
Table 2 presents the experimental results.
Dutch Bible contained an additional baseline which
consisted in generating a modernized version of the text
by applying a set of rules to the original document [9].
This second baseline improved significantly (close to 40
BLEU points and 30 TER points) the standard baselineof considering the original document as the modernized version. However, the SMT approach improved
those results even more (near 30 BLEU points and 15
TER points of improvement with respect to the secondbaseline, and 70 BLEU points and 50 TER points withrespect to the standard baseline). The NMT approachyielded better results than the standard baseline (and improvement of around 25 BLEU points and 5 TER points),
but worse results than the second baseline and the SMT
approach. Most likely, this is due to the training corpus
being too small, which is a well-known problem in NMT.
Finally, the backtranslation approach yielded the worst
results. These results represent an improvement over the
standard baseline in term of BLEU (around 4 points),
and a deterioration in terms of TER (around 8 points).
Most likely, this is due to the monolingual data used forbacktranslation not being similar enough to the trainingdata.
The experiments using El Quijote behaved similarly—taking into account that the only available baseline is the standard one—to Dutch Bible: The SMT
approached yielded the best results (an improvement of

42
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

SystemBaseline
Baseline2
SMT
NMT
NMTSynthetic

Dutch Bible

El Quijote

El Conde Lucanor

BLEU

TER

BLEU

TER

BLEU

TER

13.5 ± 0.3
50.8 ± 0.4

57.0 ± 0.3
26.5 ± 0.3

36.5 ± 0.8-

43.3 ± 1.1-

5.8 ± 0.3-

89.6 ± 1.0-

80.1 ± 0.5
38.0 ± 0.6
17.4 ± 0.5

9.9 ± 0.3
51.7 ± 2.2
65.6 ± 1.7

58.9 ± 1.0
37.4 ± 1.2
45.2 ± 1.3

29.4 ± 1.2
51.5 ± 2.0
50.6 ± 3.5

8.4 ± 0.3
2.7 ± 0.2
3.1 ± 0.2

83.8 ± 1.0
99.5 ± 2.0
165.1 ± 8.2

Table 2: Experimental results. Baseline system corresponds to considering the original document as the modernizedversion. Baseline2 came with the Dutch Bible and is a modernized version of the text generated by applying a set ofrules to the original document [9]. SMT and NMT are the SMT and NMT approaches respectively. NMTSynthetic is the
NMT system trained with the synthetic data generated through backtranslation. Best results are denoted in bold.
close to 22 BLEU points and 14 TER points). The results
yielded by the NMT approach were not significantly different to the baseline in terms of BLEU, and represented
close to a 10 points deterioration in terms of TER. In
this case, however, the backtranslation approach yielded
nearly a 10 points improvement in terms of BLEU, andthe same TER results as the NMT approach.
Not being able to obtain enough suitable training
data for El Conde Lucanor, we used the same systems
than for El Quijote. However, these documents were
written three centuries apart from one another (El Conde
Lucanor is written in 14th century Spanish and El Quijote in 17th century Spanish). Therefore, the obtained
results contained a low translation quality. Nonetheless,
it is worth noting that the SMT approach yielded improvements over the baseline (around 3 BLEU points
and 6 TER points). However, the NMT and backtranslating approached yielded a deterioration of 3 BLEU points(in both cases) and 10 and 75 TER points respectively.
In general, SMT yielded the best results in all cases.
NMT was able to improve Dutch Bible’s baseline, yielding similar results to El Quijote’s baseline and worse
results than El Conde Lucanor’s baseline. Finally, despite being successfully used in resources-poor scenarios, backtranslation was only able to improve results forthe experiment using El Quijote, and these results were
worse than the ones yielded by the SMT approach.
4.1. Qualitative Analysis
Table 2 shows some examples of sentences modernizedusing the different MT approaches.
The first example is a sentence from El Quijote.
The hypothesis generated by the SMT approach is very
closed to the reference. The main differences are achange in the order of actions (the original sentencesays Y dejando de comer, se levantó, which is changedby the hypothesis into Y levantándose, dejó de comer)
and some changes in the conjugation of verbs (e.g., dejando is changed into dejó). However, the main goalof modernization is not to generate a perfect equivalentversion, but to make the document easier to comprehend—making the overall meaning more important thanthe exact choice of words. While sentences like theseare penalized by the automatic metrics, they accomplish
modernization’s goal.
The hypothesis generated by the NMT approach
follows the same structure than the SMT hypothesis (itmakes the same reordering and conjugation changes)
but contains non-existent words (e.g., ancen) and hassome errors (e.g, a los pies in stead of puesto a caballo).
Therefore, some parts are easier to comprehend than in
the original version, but the meaning of the sentence is
not clear.
Finally, the hypothesis generated by the backpropagation approach is almost the same as the one generated
by the SMT approach (the only change is fierded in steadof fiereza). While this hypothesis is less correct than the
SMT one, they are both equally easy to comprehend.
The second example is from El Conde Lucanor,
whose experiments were conducted using the systems
trained with El Quijote. As a result, all the hypothesis are hard to comprehend. While the automatic metrics heavily penalize the backtranslation approach, inthis case, is the one which is closer to modern Spanish.
Moreover, it is the only hypothesis which preserves thename of the main characters (Lucanor and Patronio).
Looking through the whole texts, the SMT and NMT
hypothesis frequently changed the characters named into
non-existent words, while the backtranslation approachrarely modified them. Finally, having trained the systems with El Quijote has a visible effect in the SMT and

43
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Original:
Modernized:
SMT:
NMT:
NMTSynthetic :Original:
Modernized:
SMT:
NMT:
NMTSynthetic :El Quijote
Y, leuantandose, dexó de comer, y fue a quitar la cubierta de la primera imagen, que mostro ser la de San Iorge puesto a
cauallo, con vna serpiente enroscada a los pies, y la lança atrauessada por la boca, con la fiereça que suele pintarse.
Y dejando de comer, se levantó y fue a quitar la cubierta de la primera imagen, que resultó ser la de san Jorge acaballo, con una serpiente enroscada a los pies y la lanza atravesándole la boca, con la fiereza que suele pintarse.
Y levantándose, dejó de comer, y fue a quitar la cubierta de la primera imagen, que mostró ser la de San Jorge puesto acaballo, con una serpiente enroscada a los pies, y la lanza atravesada por la boca, con la fiereza que suele pintarse.
Y levantándose, dejó de comer, y fue a quitar la cubierta de la primera ancen, que mostró ser la de San Marorge a los pies
y la lanza ahabesada por la boca;
Y levantándose, dejó de comer, y fue a quitar la cubierta de la primera imagen, que mostró ser la de San Jorge puesto acaballo, con una serpiente enroscada a los pies, y la lanza atravesada por la boca, con la fierded que suele pintarse.

El Conde Lucanor-Señor conde Lucanor -dixo Patronio-, vien entiendo que el mı́o consejo non vos faze grant mengua, pero vuestra
voluntad es que vos diga lo que en esto entiendo, et vos conseje sobre ello, fazerlo he luego.
-Señor Conde Lucanor -dijo Patronio-, bien sé que mi consejo no os hace mucha falta, pero, como confiáis en mı́,-– Señor conde Lucanor -dijo Patroniorosa, vien entiendo que el mı́o consejo non vos face grant mengua, pero vuestra
voluntad es que vos diga lo que en esto entiendo, et vos aconseje en ello, ferlo he luego .
Señor conde Olcanor dijo dijo Pacasos –dijo en entiendo que el mı́o consejo non os fazo felimengua y vuestra merced es
que vos diga lo que en esto entiendo.
-Señor conde Lucanor -dijo Patronio, vien entiendo que el mı́o consejo non es face grant mengua, pero vuestra voluntad
es que vos diga lo que en esto entiendo, et vos conseje sobre ello, también yo he dicho.

Table 3: Examples of modernizing a sentence using the different MT approaches. SMT and NMT are the SMT and
NMT approaches respectively. NMTSynthetic is the NMT system trained with the synthetic data generated throughbacktranslation.
NMT approaches (with verbs conjugations such as ferlo,or expression such as vuestra merced). This effect, however, is not so visible in the backtranslation hypothesis.
All in all, neither hypothesis accomplishes the goal of
improving the comprehension of the original sentence.
Nonetheless, it is worth noting that this was a tricky
example since the original sentence is modernized into amuch shorter sentence (which reflects that 14th century
Spanish used longer expressions than modern Spanish).

5. Conclusions and Future WorkIn this work we proposed several machine translation
approaches to modernize historical documents in orderto break the language barrier and increase their accessibility to a broader audience.
We tested our approaches using three historical datasets(two of which were created for this work) from threedifferent time periods and two different languages.
Our first approach was based in SMT and yielded,
for all cases, the best results. With the exception of thedataset for which there were not available any suitabletraining data, this approach yielded significant improvements of around 22 to 67 BLEU points and 14 to 48
TER points.

Since the available training data was fairly small,the approach based on NMT produced less satisfactory
results. While it was able to yield improvements for one
dataset, the rest of the experiment resulted in either not
significantly different than the baseline, or yielding adeterioration in terms of translation quality.
Finally, despite being successfully used in resourcespoor scenarios, backtranslation was only able to improve
results for one dataset, and only in terms of BLEU. Our
best hypothesis is that historical documents are very
language-specific and, therefore, choosing the monolingual corpus to use for creating the synthetic data isextremely important. While we tried to create the monolingual datasets using similar topics, the corpora’s topics
were too specific: religious texts, a cavalry novel andmedieval tales.
In a future work, we would like to research the relation between the domains of the monolingual and training corpora deeper. Additionally, we want to explore
the use of data selection techniques for constructing themonolingual corpus to use for backtranslation, and tocreate a training partition for cases in which we do nothave suitable training data available (as was the casewith El Conde Lucanor).

44
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

6. Acknowledgments
The research leading to these results has received funding from the Ministerio de Economı́a y Competitividad(MINECO) under project CoMUN-HaT (grant agreement TIN2015-70924-C2-1-R). We gratefully acknowledge the support of NVIDIA Corporation with the donation of a GPU used for part of this research, and Andrés
Trapiello and Ediciones Destino for granting us permission to use their book in our research.

7. References[1] M. Piotrowski, Natural Language Processing for
Historical Texts, ser. Synthesis Lectures on Human
Language Technologies. Morgan & Claypool,
2012, no. 17.
[2] A. Baron and P. Rayson, “VARD2: A tool for
dealing with spelling variation in historical corpora,” Postgraduate conference in corpus linguistics, 2008.
[3] J. Porta, J.-L. Sancho, and J. Gómez, “Edit transducers for spelling variation in old spanish,” inProceedings of the workshop on computational historical linguistics, 2013, pp. 70–79.
[4] Y. Scherrer and T. Erjavec, “Modernizing historicalslovene words with character-based smt,” in Proceedings of the Biennial International Workshop on
Balto-Slavic Natural Language Processing, 2013,
pp. 58–62.
[5] M. Bollmann and A. Søgaard, “Improving historical spelling normalization with bi-directional lstms
and multi-task learning,” in Proceedings of the International Conference on the Computational Linguistics, 2016, pp. 131–139.
[6] N. Ljubešic, K. Zupan, D. Fišer, and T. Erjavec,“Normalising slovene data: historical texts vs. usergenerated content,” in Proceedings of the Conference on Natural Language Processing, 2016, pp.
146–155.
[7] M. Domingo and F. Casacuberta, “Spelling normalization of historical documents by using a machinetranslation approach,” in Proceedings of the Annual Conference of the European Association for
Machine Translation, 2018, pp. 129–137.

[8] J. Crowther, No Fear Shakespeare: Sonnets. SparkNotes, 2004.
[9] E. Tjong Kim Sang, M. Bollmann, R. Boschker,
F. Casacuberta, F. Dietz, S. Dipper, M. Domingo,
R. van der Goot, M. van Koppen, N. Ljubešić,R. Östling, F. Petran, E. Pettersson, Y. Scherrer,
M. Schraagen, L. Sevens, J. Tiedemann, T. Vanallemeersch, and K. Zervanou, “The CLIN27 sharedtask: Translating historical text to contemporary
language for improving automatic linguistic annotation,” Computational Linguistics in the Netherlands Journal, vol. 7, pp. 53–64, 2017.
[10] M. Domingo, M. Chinea-Rios, and F. Casacuberta,“Historical documents modernization,” The Prague
Bulletin of Mathematical Linguistics, vol. 108, pp.
295–306, 2017.
[11] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual data,” arXiv preprint arXiv:1511.06709, 2015.
[12] P. F. Brown, V. J. D. Pietra, S. A. D. Pietra, and
R. L. Mercer, “The mathematics of statistical machine translation: Parameter estimation,” Computational Linguistics, vol. 19, no. 2, pp. 263–311,
1993.
[13] P. Koehn, Statistical Machine Translation. Cambridge University Press, 2010.
[14] F. J. Och and H. Ney, “Discriminative training andmaximum entropy models for statistical machinetranslation,” in Proceedings of the Annual Meetingof the Association for Computational Linguistics,
2002, pp. 295–302.
[15] R. Zens, F. J. Och, and H. Ney, “Phrase-based
statistical machine translation,” in Proceedings of
the Annual German Conference on Advances in
Artificial Intelligence, vol. 2479, 2002, pp. 18–32.
[16] P. Koehn, F. J. Och, and D. Marcu, “Statisticalphrase-based translation,” in Proceedings of theConference of the North American Chapter of the
Association for Computational Linguistics on Human Language Technology, 2003, pp. 48–54.
[17] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence
to sequence learning with neural networks,” pp.
3104–3112, 2014.

45
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[18] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Transactions on
Signal Processing, vol. 45, no. 11, pp. 2673–2681,
1997.
[19] S. Hochreiter and J. Schmidhuber, “Long shortterm memory,” Neural computation, vol. 9, no. 8,
pp. 1735–1780, 1997.
[20] D. Bahdanau, K. Cho, and Y. Bengio, “Neuralmachine translation by jointly learning to alignand translate,” in Proceedings of the International Conference on Learning Representations(arXiv:1409.0473), 2015.
[21] R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio,“How to construct deep recurrent neural networks,”
arXiv preprint arXiv:1312.6026, 2013.
[22] A. Poncelas, D. Shterionov, A. Way, G. Maillette de Buy Wenniger, and P. Passban, “Investigation backtranslation in neural machine translation,”in Proceedings of the Annual Conference of the European Association for Machine Translation, 2018,
pp. 249–258.
[23] R. Chatterjee, M. A. Farajian, M. Negri, M. Turchi,
A. Srivastava, and S. Pal, “Multi-source neuralautomatic post-editing: Fbk’s participation in the
wmt 2017 ape shared task,” in Proceedings of theSecond Conference on Machine Translation, 2017,
pp. 630–638.
[24] F. F. Jehle, Works of Miguel de Cervantes in Oldand Modern-spelling. Indiana University Purdue
University Fort Wayne, 2001.
[25] A. Trapiello, Don Quijote de la Mancha Puesto en
castellano actual ı́ntegra y fielmente por Andrés
Trapiello. Ediciones Destino, 2015.
[26] D. Varga, L. Németh, P. Halácsy, A. Kornai,
V. Trón, and V. Nagy, “Parallel corpora for medium
density languages,” in Proceedings of the International Conference Recent Advances in Natural
Language Processing, 2005, pp. 590–596.
[27] L. López Nieves, El Conde Lucanor.
Digital Ciudad Seva, 2002.

Biblioteca[28] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,
“BLEU: a method for automatic evaluation of machine translation,” in Proceedings of the Annual

Meeting of the Association for Computational Linguistics, 2002, pp. 311–318.
[29] M. Snover, B. Dorr, R. Schwartz, L. Micciulla,
and J. Makhoul, “A study of translation edit ratewith targeted human annotation,” in Proceedingsof the Association for Machine Translation in theAmericas, 2006, pp. 223–231.
[30] P. Koehn, “Statistical significance tests for machine translation evaluation,” in Proceedings of
the Conference on Empirical Methods in Natural
Language Processing, 2004, pp. 388–395.
[31] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, “Moses: Open source
toolkit for statistical machine translation,” in Proceedings of the Annual Meeting of the Association
for Computational Linguistics, 2007, pp. 177–180.
[32] A. Stolcke, “SRILM - an extensible language modeling toolkit,” in Proceedings of the International
Conference on Spoken Language Processing, 2002,
pp. 257–286.
[33] F. J. Och, “Minimum error rate training in statistical machine translation,” in Proceedings of theAnnual Meeting of the Association for Computational Linguistics, 2003, pp. 160–167.
[34] G. Klein, Y. Kim, Y. Deng, J. Senellart, andA. M. Rush, “OpenNMT: Open-Source Toolkit
for Neural Machine Translation,” arXiv preprint
arXiv:1701.02810, 2017.
[35] D. Britz, A. Goldie, T. Luong, and Q. Le, “Massive
exploration of neural machine translation architectures,” arXiv preprint arXiv:1703.03906, 2017.
[36] D. P. Kingma and J. Ba, “Adam: A method
for stochastic optimization,” arXiv preprint
arXiv:1412.6980, 2014.
[37] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi,
W. Macherey, M. Krikun, Y. Cao, Q. Gao,K. Macherey, et al., “Google’s neural machinetranslation system: Bridging the gap between
human and machine translation,” arXiv preprint
arXiv:1609.08144, 2016.

46
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[38] R. Sennrich, B. Haddow, and A. Birch, “Neuralmachine translation of rare words with subword
units,” in Proceedings of the Annual Meeting of the
Association for Computational Linguistics, 2016,
pp. 1715–1725.

47
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Multi-Source Neural Machine Translation with Data Augmentation
Yuta Nishimura1 , Katsuhito Sudoh1 , Graham Neubig2,1 , Satoshi Nakamura1
1

Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, Nara 630-0192, Japan
2
Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA{nishimura.yuta.nn9, sudoh, s-nakamura}@is.naist.jp
gneubig@cs.cmu.edu

Abstract

Original

Multi-source translation systems translate from multiple languages to a single target language. By using informationfrom these multiple sources, these systems achieve large
gains in accuracy. To train these systems, it is necessary tohave corpora with parallel text in multiple sources and thetarget language. However, these corpora are rarely complete
in practice due to the difficulty of providing human translations in all of the relevant languages. In this paper, we propose a data augmentation approach to fill such incomplete
parts using multi-source neural machine translation (NMT).
In our experiments, results varied over different language
combinations but significant gains were observed when using a source language similar to the target language.

1. Introduction
Machine Translation (MT) systems usually translate one
source language to one target language. However, in many
real situations, there are multiple languages in the corpus ofinterest. Examples of this situation include the multilingual
official document collections of the European parliament [1]and the United Nations [2]. These documents are manuallytranslated into all official languages of the respective organizations. Many methods have been proposed to use these multiple languages in translation systems to improve the translation accuracy [3, 4, 5, 6]. In almost all cases, multilingual machine translation systems output better translations
than one-to-one systems, as the system has access to multiple sources of information to reduce ambiguity in the target
sentence structure or word choice.
However, in contrast to the more official document collections mentioned above where it is mandated that all translations in all languages, there are also more informal multilingual captions such as those of talks [7] and movies [8]. Because these are based on voluntary translation efforts, large
portions of them are not translated, especially into languages
with a relatively small number of speakers.
Nishimura et al. [9] have recently proposed a method for
multi-source NMT that is able to deal with the case of missing source data encountered in these corpora. The implementation is simple: missing source translations are replacedwith a special symbol hNULLi as shown in Figure 1(a). ThisEn

How are you?
MissingCs

Original

Ako sa máte?

Sk<NULL>(a) Multi-source NMT with filling in a symbol [9]Original

En
Cs

How are you?
PseudoOriginal

Ako sa máte?

Sk

Jak se máte?
Original

Data Augmentation
with trained multi-source NMT{En, Sk}-to-Cs

How are you?
Original

Ako sa máte?

EnSk(b) Proposed Method: Multi-source NMT with data augmentation

Figure 1: Example of multi-source NMT with an incomplete
corpus; The language pair is {English, Czech}-to-Slovak andthe translation of Czech is missing.

method allows us to use incomplete corpora both at training
time and test time, and multi-source NMT with this method
was shown to achieve higher translation accuracy. If themodel is trained on corpora with a large number of hNULLi
symbols on the source side, a large number of training examples will be different from test time, when we actually havemultiple sources. Thus, these examples will presumably beless useful in training a model intended to do multi-source
translation. In this paper, we propose an improved method
for utilizing multi-source examples with missing data: using a pseudo-corpus whose missing translations are filled up
with machine translation outputs using a trained multi-source
NMT system as shown in Figure 1(b). Experimental results
show that the proposed method is a more effective method toincorporate incomplete multilingual corpora, achieving improvements of up to about 2 BLEU over the previous method
where each missing input sentence is replaced by hNULLi.

48
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

2. Related Work2.1. Multi-source NMT
There are two major approaches to multi-source NMT; multiencoder NMT [10] and mixture of NMT Experts [11]. In
this work, we focus on the multi-encoder NMT that showed
better performance in most cases in Nishimura et al. [9].
Multi-encoder NMT [10] is similar to the standard attentional NMT framework [12] but uses multiple encoders
corresponding to the source languages and a single decoder.
Suppose we have two LSTM-based encoders and theirhidden and cell states at the end of the inputs are h1 , h2
and c1 , c2 , respectively. Multi-encoder NMT initializes its
decoder hidden state h and cell state c using these encoder
states as follows:h = tanh(Wc [h1 ; h2 ])(1)

c = c1 + c2

(2)

Attention is then defined over encoder states at each time
step t and resulting context vectors d1t and d2t are concatenated together with the corresponding decoder hidden state
ht to calculate the final context vector h˜t .
h˜t = tanh(Wc [ht ; d1t ; d2t ])(3)

Our multi-encoder NMT implementation is basicallysimilar to the original one [10] but has a difference in
its attention mechanism. We use global attention used in
Nishimura et al. [9], while Zoph and Knight used local-p
attention. The global attention allows the decoder to look at
everywhere in the input, while the local-p attention forces tofocus on a part of the input [13].
2.2. Data Augmentation for NMT
Sennrich et al. proposed a method to use monolingual training data in the target language for training NMT systems,with no changes to the network architecture [14]. It firsttrains a seed target-to-source NMT model using a parallel
corpus and then translates the monolingual target language
sentences into the source language to create a synthetic parallel corpus. It finally trains a source-to-target NMT model using the seed and synthetic parallel corpora. This very simple
method called back-translation makes effective use of available resources, and achieves substantial gains. Imamura etal. proposed a method that enhances the encoder and attention using target monolingual corpora by generating mutliple
source sentences via sampling as an extension of the backtranslation [15].
There are also other approaches for data augmentation
other than back-translation. Wang et al. proposed a methodof randomly replacing words in both the source sentence andthe target sentence with other random words from their corresponding vocabularies [16]. Kim and Rush proposed asequence-level knowledge distillation in NMT that uses machine translation results by a large teacher model to train asmall student model as well as ground-truth translations [17].
Our work is an extension of the back-translation approach in multilingual situations by generating pseudotranslations using multi-source NMT.

3. Proposed Method
We propose three types of data augmentation for multiencoder NMT; “fill-in”, “fill-in and replace” and “fill-in andadd.” Firstly, we explain about the data requirements andoverall framework using Figure 1(b). We used three languages; English, Czech and Slovak. Our goal is to get the
Slovak translation, and to do so we take three steps. There
are not any missing data in English translations, but Slovak and Czech translations have some missing data. In thefirst step, we train a multi-encoder NMT model (Source:
English and Slovak, Target: Czech) to get Czech pseudotranslations using the baseline method, which is to replace a
missing input sentence with a special symbol hNULLi. In
the second step, we create Czech pseudo-translations using
multi-encoder NMT which was trained on the first step. We
conducted three types of augmentation, which we introduce
later. Finally in the third step, we switch the role of Czechand Slovak, in other words, we train a new multi-encoder
NMT model (Source: English and Czech, Target: Slovak).
At this time, we use Slovak pseudo-translations in the source
language side. This method is similar to back-translation but
taking advantage of the fact that we have an additional sourceof knowledge (Czech or Slovak) when trying to augment theother language (Slovak or Czech respectively).
We next introduce three types of augmentation. Figure 2
illustrates their examples in {English, Czech}-to-{Slovak}
case where one Czech sentence is missing.
(a) fill-in: where only missing parts in the corpus are filled
up with pseudo-translations.
(b) fill-in and replace: where we both augment the missing part and replace original translations with pseudotranslations in the source language except English whosetranslations has not any missing data. The motivation behind this method is not to use unreliable translation. Morishita et al. [18] demonstrated the effectiveness of applying
back-translation for an unreliable part of a provided corpus.
Translations of TED talks are from many independent volunteers, so there may be some differences between translations
other than original English, or even they may include somefree or over-simplified translations. We aim to fill such a gap
using data augmentation.
(c) fill-in and add: where we both augment the missing partand added pseudo-translations from original translations inthe source language except English. This helps prevent introduction of too much noise due to the complete replacement of original translations with pseudo-translations in thesecond method.

49
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Original

Table 1: “train” shows the number of available training sentences, and “missing” shows the number and the fraction of
missing sentences in comparison with English ones.

Hello
En

Original

Original

How are you?

Dobrý deňOriginal

Original

Ako sa máte?

Ahoj
Cs

Pair

Sk

en-hr/sr

Pseudoen-sk/cs

Jak se máte?
(a) fill-in

en-vi/id

Trg
hr
sr
sk
cs
vi
id

train
11894913355810060059918
16098482592

missing
35564 (29.9%)50203 (37.6%)
58602 (57.7%)17380 (29.0%)87816 (54.5%)
9424 (11.4%)

Original

Hello
En

Original

PseudoČauCsthree different language sets for the experiments: {English(en), Croatian (hr), Serbian (sr)}, {English (en), Slovak (sk),Czech (cs)}, and {English (en), Vietnamese (vi), Indonesian(id)}. Since the great majority of TED talks are in English,
the experiments were designed for the translation from English to another language with the help of the other languagein the language set, with no missing portions in the English
sentences. Table 1 shows the number of training sentences
for each language set. At test time, we experiment with acomplete corpus with both source sentences represented, as
this is the sort of multi-source translation setting that we areaiming to create models for.

Original

How are you?

Dobrý deňOriginalSk

Ako sa máte?

PseudoJak se máte?
(b) fill-in and replace

Original

Hello
Original
Original

Hello

En

Dobrý deňOriginal

How are you?

Dobrý deňOriginal

Ahoj
PseudoCs

4.2. Baseline Methods

Original

We compared the proposed methods with the following threebaseline methods.

Sk

Original

Ako sa máte?

One-to-one NMT: a standard NMT model from one source
language to another target language. The source language isfixed to English in the experiments. If the target language
part is missing in the parallel corpus, such sentences pairs
cannot be used in training so they are excluded from the training set.

ČauPseudoJak se máte?
(c) fill-in and add

Figure 2: Example of three types of augmentation; Language
Pair is {English, Czech}-to-{Slovak} and Czech translation
corresponding to “How are you?” is missisng. In this example, the dotted background indicates the pseudo-translation
produced from multi-source NMT and the white backgroundmeans the original translation.

Multi-encoder NMT with back-translation: a multiencoder NMT system using English-to-X NMT to fill up the
missing parts in the other source language X.1
Multi-encoder NMT with hNULLi: a multi-encoder NMT
system using a special symbol hNULLi to fill up the missing
parts in the other source language X [9].
4.3. NMT settings

4. Experiment
We conducted MT experiments to examine the performanceof the proposed method using actual multilingual corpora of
TED Talks.
4.1. Data
We used a collection of transcriptions of TED Talks andtheir multilingual translations. The numbers of these voluntary translations differs significantly by language. We chose

NMT settings are the same for all the methods in the experiments. We use bidirectional LSTM encoders [12], andglobal attention and input feeding for the NMT model [13].
The number of dimensions is set to 512 for the hidden andembedding layers. Subword segmentation was applied using
SentencePiece [19]. We trained one subword segmentation
model for English and another shared between the other two
1 This is not exactly back-translation because the pseudo-translations are
not from the target language but from the other source language (English) in
our multi-source condition. But we use this familiar term here for simplicity.

50
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018languages in the language set because the amount of training data for the languages other than English was small. For
parameter optimization, we used Adam [20] with gradientclipping of 5. We performed early stopping, saving parameter values that had the best log likelihoods on the validation
data and used them when decoding test data.

Contrary, the BLEU from “fill-in and add” was the highest
when the target language was Indonesian. We hypothesizethat this is due to much smaller fraction of the missing parts
in Indonesian corpus as shown in Table 1, so there should
be little room for improvement if we fill in only the missing parts even if the accuracy of the pseudo-translations isrelatively high.

4.4. Results
Table 2 shows the results in BLEU [21]. We can see that our
proposed methods demonstrate larger gains in BLEU thanbaseline methods in two language sets: {English, Croatian,
Serbian}, {English, Slovak, Czech}. On these pairs, we cansay that our proposed method is an effective way for using
incomplete multilingual corpora, exceeding other reasonably
strong baselines. However, in {English, Vietnamese, Indonesian}, our proposed methods obtained lower scores than thebaseline methods. We observed that the improvement by theuse of multi-encoder NMT against one-to-one NMT in thebaseline was significantly smaller than the other languagesets, so multi-encoder NMT was not as effective compared to
one-to-one NMT in the first place. Our proposed method isaffected by which languages to use, and the proposed method
is likely more effective for similar language pairs because theexpected accuracy of the pseudo-translation gets better by thehelp of lexical and syntactic similarity including shared subword entries.

5. Discussion
5.1. Different Types of Augmentation
We examined three types of augmentation: “fill-in”, “fill-in
and replace”, “fill-in and add”. In Table 2, we can see that
there were no significant differences among them, despite thefact that their training data were very different from eachother. We conducted additional experiments using incomplete corpora with lower quality augmentation by one-to-one
NMT to investigate the differences of the three types of augmentation. We created three types of pseudo-multilingual
corpora using back-translation from one-to-one NMT andtrained multi-encoder NMT models using them. Our expectation here was that the aggressive use of low quality pseudotranslations caused to contaminate the training data and to
decrease the translation accuracy.
Table 3 shows the results. In {English, Croatian, Serbian} and {English, Slovak, Czech}, we obtained significant
drop in BLEU scores with the aggressive strategies (“fill-in
and replace” and “fill-in and add”), while there are few differences in {English, Vietnamese, Indonesian}. One possible
reason is that the quality of pseudo-translations by one-toone NMT in Indonesian and Vietnamese was better than theother languages; in other words, the BLEU from one-to-one
NMT in Table 2 was sufficiently good without multi-source
NMT. Thus the translation performance for Croatian, Serbian, Slovak and Czech could not improve in the experiments
here due to noisy pseudo-translations of those languages.

5.2. Iterative Augmentation
It can be noted that if we have a better multi-source NMT
system, it can be used to produce better pseudo-translations.
This leads to a natural iterative training procedure where wealternatively update the multi-source NMT systems into thetwo target languages.
Table 4 shows the results of {English, Croatian, Serbian}. We found that this produced negative results; BLEU
decreased gradually in every step. We observed very similar
results in the other language pairs, while we omit the actual
numbers here. This indicates that the iterative training may
be introducing more noise than it is yielding improvements,
and thus may be less promising than initially hypothesized.
5.3. Non-parallelism
A problem in the use of multilingual corpora is nonparallelism. In case of TED multilingual captions, they aretranslated from English transcripts independently by many
volunteers, which may cause some differences in detailsof the translation in the various target languages. For example in {English, Croatian, Serbian}, Croatian and Serbian translations may not be completely parallel. Table 5shows such an example where the Serbian translation does
not have a phrase corresponding to “let me.” This kind of
non-parallelism may be resolved by overriding such translations with pseudo-translations with “fill-in and replace” and“fill-in and add”. Here, the Serbian pseudo-translation includes the corresponding phrase “Dozvolite mi” and can beused to compensate for the missing information. This would
be one possible reason of the improvements by “fill-up andreplace” or “fill-up and add”.

6. Conclusions
In this paper, we examined data augmentation of incomplete multilingual corpora in multi-source NMT. We proposed three types of augmentation; fill-in, fill-in and replace,fill-in and add. Our proposed methods proved better thanbaseline system using the corpus where missing part wasfilled up with “hNULLi”, although results depended on thelanguage pair. One limitation in the current experiments witha set of three languages was that missing parts in the test sets
could not be filled in. This can be resolved if we use morelanguages, and we will investigate this in future work.

51
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 2: Main results in BLEU for English-Croatian/Serbian (en-hr/sr), English-Slovak/Czech (en-sk/cs), and EnglishVietnamese/Indonesian (en-vi/id).
baseline methodPair
en-hr/sren-sk/cs
en-vi/idTrg

one-to-one(En-to-Trg)

multi-encoder NMT(fill up with symbol)

multi-encoder NMT(back translation)

fill-infill-in andreplacefill-inand add

hr
sr
sk
cs
vi
id

20.21
16.42
13.79
14.72
24.60
24.89

28.18
23.85
20.27
19.88
25.70
26.89

27.57
22.73
19.83
19.54
26.66
26.34

29.17
24.41
20.26
20.78
26.73
26.40

29.37
24.96
20.43
20.90
26.48
25.73

29.40
24.15
20.59
20.61
26.32
26.21

Table 3: The difference of three types of augmentation in BLEU for English-Croatian/Serbian (enhr/sr), English-Slovak/Czech (en-sk/cs), and EnglishVietnamese/Indonesian (en-vi/id). We used one-to-one
model to produce pseudo-translations.
multi-encoder NMT (back-translation)Pair
en-hr/sren-sk/cs
en-vi/id

proposed methodTrgfill-infill-in andreplacefill-inand add

hr
sr
sk
cs
vi
id

27.57
22.73
19.83
19.54
26.66
26.34

24.05
17.77
16.75
17.04
26.39
23.90

24.79
22.02
18.16
18.40
26.65
26.67

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics. Beijing, China:
Association for Computational Linguistics, July 2015,
pp. 1723–1732.
[4] O. Firat, K. Cho, and Y. Bengio, “Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies. San Diego, California: Association for
Computational Linguistics, June 2016, pp. 866–875.
[5] M. Johonson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu,
Z. Chen, N. Thorat, F. Vigas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean, “Google’s Multilingual
Neural Machine Translation System: Enabling ZeroShot Translation,” Transactions of the Association for
Computational Linguistics, vol. 5, pp. 339–351, 2017.

7. Acknowledgements

8. References[6] T.-L. Ha, J. Niehues, and A. Waibel, “Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder,” in Proceedings of the 13th International Workshop on Spoken Language Translation,
Seattle, Washington, December 2016.

[1] P. Koehn, “Europarl: A Parallel Corpus for Statistical
Machine Translation,” in Conference Proceedings: thetenth Machine Translation Summit, AAMT. Phuket,
Thailand: AAMT, 2005, pp. 79–86.

[7] M. Cettolo, C. Girardi, and M. Federico, “WIT3 : Web
Inventory of Transcribed and Translated Talks,” in Proceedings of the 16th EAMT Conference, May 2012, pp.
261–268.

[2] M. Ziemski, M. Junczys-Dowmunt, and B. Pouliquen,“The United Nations Parallel Corpus v1.0,” in Proceedings of the Tenth International Conference on Language
Resources and Evaluation (LREC 2016), N. C. C.
Chair), K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno,
J. Odijk, and S. Piperidis, Eds. Paris, France: European Language Resources Association (ELRA), May
2016.

[8] J. Tiedemann, “News from OPUS - A Collection ofMultilingual Parallel Corpora with Tools and Interfaces,” in Recent Advances in Natural Language Processing, N. Nicolov, K. Bontcheva, G. Angelova, and
R. Mitkov, Eds. Borovets, Bulgaria: John Benjamins,
Amsterdam/Philadelphia, 2009, vol. V, pp. 237–248.

Part of this work was supported by JSPS KAKENHI Grant
Numbers and JP16H05873 and JP17H06101.

[3] D. Dong, H. Wu, W. He, D. Yu, and H. Wang, “MultiTask Learning for Multiple Language Translation,” in[9] Y. Nishimura, K. Sudoh, G. Neubig, and S. Nakamura,“Multi-Source Neural Machine Translaion with Missing Data,” in Proceedings of the 2nd Workshop on Neural Machine Translation and Generation. Association
for Computational Linguistics, July 2018, pp. 92–99.

52
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 4: BLEU (and BLEU gains compared to step 1) in each step of iterative augmentation.
Pair
en-hr/sr

Trg
hr
sr

step 129.17 (+0.00)
24.41 (+0.00)

step 229.03 (-0.14)
24.18 (-0.23)

step 329.10 (-0.07)
24.17 (-0.24)

step 429.05 (-0.12)
23.95 (-0.46)

Table 5: Example of the Serbian pseudo-translation. This pseudo-translation is the output of {English, Croatian}-to-Serbiantranslation.
Type
Original (En)
Original (Sr)
Pseudo (Sr)Sentence
So let me conclude with just a remark to bring it back to the theme of choices.
Da zakljuim jednom konstatacijom kojom se vraam na temu izbora.
Dozvolite mi da zakljuim samo jednom opaskom, da se vratim na temu izbora.

[10] B. Zoph and K. Knight, “Multi-Source Neural Translation,” in Proceedings of the 2016 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies.
San Diego, California: Association for Computational
Linguistics, June 2016, pp. 30–34.
[11] E. Garmash and C. Monz, “Ensemble Learning forMulti-Source Neural Machine Translation,” in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. Osaka, Japan: The COLING 2016 Organizing
Committee, December 2016, pp. 1409–1418.
[12] D. Bahdanau, K. Cho, and Y. Bengio, “Neural MachineTranslation by Jointly Learning to Align and Translate,”in Proceedings of the 3rd International Conference on
Learning Representations, May 2015.
[13] T. Luong, H. Pham, and C. D. Manning, “Effective Approaches to Attention-based Neural Machine Translation,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational Linguistics, September 2015, pp. 1412–1421.
[14] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” in Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers). Association for Computational Linguistics, 2016, pp. 86–96.
[15] K. Imamura, A. Fujita, and E. Sumita, “Enhancementof encoder and attention using target monolingual
corpora in neural machine translation,” in Proceedingsof the 2nd Workshop on Neural Machine Translation
and Generation. Association for Computational
Linguistics, 2018, pp. 55–63. [Online]. Available:
http://aclweb.org/anthology/W18-2707[16] X. Wang, H. Pham, Z. Dai, and G. Neubig, “Switchout:an efficient data augmentation algorithm for neural machine translation,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
[17] Y. Kim and A. M. Rush, “Sequence-level knowledge
distillation,” in Proceedings of the 2016 Conferenceon Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational
Linguistics, November 2016, pp. 1317–1327. [Online].
Available: https://aclweb.org/anthology/D16-1139[18] M. Morishita, J. Suzuki, and M. Nagata, “NTT Neural Machine Translation Systems at WAT 2017,” inProceedings of the 4th Workshop on Asian Translation(WAT2017), 2017, pp. 89–94.
[19] T. Kudo, “Subword regularization: Improving neuralnetwork translation models with multiple subword candidates,” in Proceedings of the 56th Annual Meeting of
the Association for Computational Linguistics (Volume
1: Long Papers). Association for Computational Linguistics, 2018, pp. 66–75.
[20] D. P. K. Kingma and J. Ba, “Adam: A Method for
Stochastic Optimization,” in Proceedings of the 3rd International Conference on Learning Representations,
May 2015.
[21] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,
“BLEU: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual
Meeting of the Association for Computational Linguistics (ACL), Philadelphia, July 2002, pp. 311–318.

53
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Transfer Learning in Multilingual Neural Machine Translation
with Dynamic Vocabulary
Surafel M. Lakew†+ , Aliia Erofeeva† , Matteo Negri+ , Marcello Federico⋆ , Marco Turchi+††

University of Trento, + Fondazione Bruno Kessler, Trento, Italy⋆
Amazon AI, East Palo Alto, CA 94303, USA

name.surname@unitn.it, + surname@fbk.eu, ⋆ marcfede@amazon.com

Abstract
We propose a method to transfer knowledge across neuralmachine translation (NMT) models by means of a shared dynamic vocabulary. Our approach allows to extend an initialmodel for a given language pair to cover new languages byadapting its vocabulary as long as new data become available (i.e., introducing new vocabulary items if they are notincluded in the initial model). The parameter transfer mechanism is evaluated in two scenarios: i) to adapt a trained single
language NMT system to work with a new language pair andii) to continuously add new language pairs to grow to a multilingual NMT system. In both the scenarios our goal is toimprove the translation performance, while minimizing thetraining convergence time. Preliminary experiments spanning five languages with different training data sizes (i.e., 5k
and 50k parallel sentences) show a significant performancegain ranging from +3.85 up to +13.63 BLEU in different language directions. Moreover, when compared with training an
NMT model from scratch, our transfer-learning approach allows us to reach higher performance after training up to 4%of the total training steps.

1. Introduction
Neural Machine Translation (NMT) has shown to surpass phrase based Machine Translation approaches not
only in high-resource language settings, but also with lowresource [1] and zero-resource translation tasks [2, 3]. Although recent approaches yield promising results, training
models in low-resource settings remains a challenge for MT
research [4]. [2] have shown that a multilingual NMT (MNMT) model that utilizes a concatenation of data covering
multiple language pairs (including high-resourced ones) canresult in better performance in the low-resource translation
task. Alternatively, [5] proposed a transfer-learning approachfrom an NMT “parent-model” trained on a high-resource language to initialize a “child-model” in a low-resource setting
showing consistent translation improvements on the latter
task.
Though effective, training models on a concatenation ofdata covering multiple language pairs or initializing them by(*) Work conducted while this author was at FBK.

transferring knowledge from a parent model does not consider the dynamic nature of new language vocabularies. In
relation to how and when model vocabularies are built, there
can be two distinct scenarios. In the first one, all the training
data for all the language pairs are available since the beginning. In this case, either separate or joint sub-word segmentation models can be applied on the training material to build
vocabularies that represent all the data [6, 7]. In the second
scenario, training data covering different language directions
are not available at the same time (most real-world MT training scenarios fall in this category, in which new data or newneeds in terms of domains or language coverage emerge overtime). In such cases, either: i) new MT models are trained
from scratch with new vocabularies built from the incoming
training data, or ii) the word segmentation rules of a prior(parent) model are applied on the new data to continue thetraining as a fine-tuning task. In all the scenarios, accurate word segmentation is crucial to avoid out-of-vocabulary(OOV) tokens. However, different strategies for the different
training conditions can result in longer training time or performance degradations. More specifically, limiting the target
task with the initial model vocabulary will result in: i) a word
segmentation that is unfavorable for the new language directions and ii) a fixed vocabulary/model dimension despite thevarying language and training dataset size.
NMT models are not only data-demanding, but also require considerable time to be trained, optimized, and put into
use. In particular real-word scenarios, strict time constraints
prevent the possibility to deploy and use NMT technology(consider, for instance, emergency situations that require topromptly enable communication across languages [8]). On
top of this, when the available training corpora are limited
in size, delivering usable NMT systems (i.e., systems thatcan be used with the requirement of not making severe errors[9]) becomes prohibitive. In summary: i) on the data side,
acquiring new training material for x undefined languages iscostly and not always possible, and ii) on the model side,building an NMT system from scratch when new data become available raises efficiency and performance issues thatare particularly relevant in low-resource scenarios.
We address these issues by introducing a method to transfer knowledge across languages by means of a dynamic vo-

54
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

cabulary. Starting from an initial model, our method allows
to build new NMT models, either in a single or multiple
language translation directions, by dynamically updating theinitial vocabulary to new incoming data. For instance, given
a trained German-English NMT system (L1 ), the learned parameters can be transferred across models, while adopting
new language vocabularies. In our experimental setting wetest two transfer approaches:• progAdapt: train a chain of consecutive M-NMT
models by transferring the parameters of an initialmodel for L1 to new language pairs L2 . . . LN . In this
scenario, the goal is to maximize performance on the
new language pairs.
• progGrow: progressively introduce new language
pairs to the initial model L1 to create a growing MNMT model covering N translation directions. In this
scenario, the goal is to maximize performance on all
the language pairs.
Our experiments are carried out with Italian−English,
Romanian−English, and Dutch−English training data sets ofdifferent size, ranging from low-resource (50k) to extremely
low-resource (5k) conditions.
As such, in a rather different way from previous work [5],we show our transfer-learning approach in a multilingual
NMT model with dynamic vocabulary both in the source andtarget directions. Our contributions are as follows:• we develop a transfer-learning technique for NMT
based on a dynamic vocabulary, which adapts the parameters learned on a parent task (language direction)
to cover new target tasks;• through experiments in different scenarios, we showthat our approach improves knowledge transfer across
NMT models for different languages, particularly in
low-resource conditions;• we show that, with our transfer learning approach,
it is possible to train a faster converging model that
achieves better performance than a system trained
from scratch.

2. Related work
2.1. Transfer Learning
Recent efforts [10, 11] in natural language processing(NLP) research have shown promising results when transferlearning techniques are applied to leverage existing models tocope with the scarcity of training data in specific domains or
language settings. The advancements in NLP came following
a much larger impact of transfer-learning in computer visiontasks, such as classification and segmentation, either using
features of ImageNet [12] or by fine-tuning the last layersof a deep neural network [13]. Specific to NLP, pre-trained
word embeddings [14] used as input to the first layer of the

network have become a common practice. In a broader sense,
pre-trained models have been successfully exploited for several NLP tasks. [15] used an MT model as a pre-training step
to further contextualize word vectors for downstream tasks
like sentiment analysis, question classification, textual entailment, and question answering. In a similar way, a language
model is utilized for pre-training in sequence labeling tasks[16], question answering, textual entailment, and sentiment
analysis [17].
Close to our approach, [5] explored techniques fortransfer-learning across two NMT models. First, a “parent” model is trained with a large amount of available data.
Then the encoder-decoder components are transferred to initialize the parameters of a low-resourced “child” model. In
this parent-child setting, the decoder parameters of the childmodel are fixed at the time of fine-tuning. Later, in [18], theparent-child approach has been extended to analyze the effectof using related languages on the source side.
Although this work shares a related approach with [5], we
diverge by our hypothesis not to selectively update only theencoder, allowing all the parameters to be updated as a beneficial strategy in our setting. Our strategy is based on boththe source→target and target→source translation directions
that we consider as transferable. Moreover, our transferlearning approach relies on a dynamic vocabulary that enforces changes in the trainable parameters of the network incontrast to fixing them1 .
2.2. Multilingual NMT
In a one-to-many multilingual translation scenario, [19] proposed a multi-task learning approach that utilizes a single
encoder for the source language and separate attention mechanisms and decoders for each target language. [20] useddistinct encoder and decoder networks for modeling multiple language pairs in a many-to-many setting. Later, [21]introduced a way to share the attention mechanism acrossmultiple languages. Aimed at avoiding translation ambiguities on the decoder side, a many-to-one character level NMT
setup [22] and a two/multi-source NMT [23] were also proposed. Inspired by [24], who automatically annotated thesource side with artificial flags to manage the politeness levelof the output, other works focused on controlling the grammatical voice [25], the text domain [26, 27], and enforcing
gender agreement [28]. Simplified yet efficient multilingual
NMT approaches have been proposed by [2] and [3]. The approach in [3] applies a language-specific code to words fromdifferent languages in a mixed-language vocabulary. The approach in [2], by prepending a language flag to the input
string, greatly simplified multilingual NMT eliminating theneed of having separate encoder/decoder networks and attention mechanism for each new language pair. In this work
we follow a similar strategy by incorporating an artificial language flag.
1 In future work, we plan to further study which parameters are more

beneficial if transferred and which part of the network to selectively update.

55
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

3. Transfer Learning in M-NMT
In this work, we cast transfer-learning in a multilingual neural machine translation (M-NMT) task as the problem ofdynamically changing/updating the vocabulary of a trained
NMT system. In particular, transfer-learning across models
is assumed to: i) include a strategy to add new languagespecific items to an existing NMT vocabulary, and ii) be able
to manage a number of new translation directions in different
transfer rounds, either by covering them one at a time (i.e.,
in a chain where new languages are covered stepwise) or simultaneously (i.e., pursuing all directions at each step). Ourinvestigation focuses on two aspects. The first one is how theparameters of an existing model can be transferred to a target
one for a new language pair. The second aspect is how tolimit the impact of parameters’ transfer on the performanceof the initial model as long as new language directions areadded. For convenience, we refer to our approach as TL-DV(Transfer-Learning using Dynamic Vocabulary).
As shown in Figure 1, our transfer-learning approach isevaluated in two conditions:• progAdapt, in which progressive updates are madeon the assumption that new target NMT task data become available for one language direction at a time(i.e., new language directions are covered sequentially). In this condition, our goal is to maximize performance on the new target tasks by taking advantageof parameters learned in their parent task;• progGrow, in which progressive updates are madeon the same assumption of receiving new target task
data as in progAdapt, but with the additional goal ofpreserving the performance of the previous language
directions.
We discuss these two scenarios below in §3.2 and §3.3.
3.1. Dynamic Vocabulary
In the defined scenarios, we update the vocabulary Vp of theprevious model with the current language direction vocabulary Vc . The approach simply keeps the intersection (sameentries) between Vp and Vc , whereas replacing Vp entries
with Vc if the entries of the former vocabulary do not exist
in the latter. At training time, these new entries are randomly
initialized, while the intersecting items maintain the embeddings of the former model. The alternative approach to dynamic vocabulary in a continuous model training is to use
the initial model vocabulary Vp , which we refer to as staticvocabulary.

performed as described in §3. In this approach, the dataset of
the initial model is not included at the current training stage.
This allows the adaptation to the new language without unnecessary word segmentation that may arise by applying theinitial model’s segmentation rules. As shown in Figure 1(left), the adaptation on any of the Ln stages is language independent, though subject to the available training dataset.
We refer to the application of this approach in the experimental settings and discussion as progAdapt.
3.3. Progressive Growth of Translation Directions
In this scenario, an initial model L1 is simultaneously
adapted to an incremental number of translation directions,
under the constraint that the level of performance on L1 has
to be maintained. For a simplified experimental setup, we
will incorporate a single language pair (source→target) at atime, when adapting to Ln from Ln−1 (see Figure 1 (right)).
We refer to the application of this approach in the experimental settings and discussion as progGrow.

4. Experimental Setting
4.1. Dataset and Preprocessing
Our experimental setting includes the init model language
pair (German-English) and three additional language pairs(Italian-English, Romanian-English, and Dutch-English) for
testing the proposed approaches. We use publicly available
datasets from the WIT3 TED corpus [29]. Table 1 shows thesummary of the training, dev, and test sets. To simulate an
extremely low-resource (MELR ) and low-resource (MLR )model settings, 5K and 50K sentences are sampled from thelast three language pairs’ training data.
At the preprocessing step, we first tokenize the raw data
and remove sentences longer than 70 tokens. As in [2], weprepend a “language flag” on the source side of the corpus for
all multilingual models. For instance, if a German source ispaired with an English target, we append <2ENG> at the beginning of source segments. Next, a shared byte pair encoding (BPE) model [6] is trained using the union of the source
and target sides of each language pair. Following [30], thenumber of BPE segmentation rules is set to 8, 500 for thedata size used in our experimental setting. At different levels of training (Li ), a BPE model with respect to the language pairs is then used to segment the training, dev, and test
data into sub-word units. While, the vocabulary size of theinit is fixed, the vocabulary varies in the consecutive training stages depending on the overlap of sub-word units andlexical similarity between two language pairs.
4.2. Experimental Settings

3.2. Progressive Adaptation to New Languages
In this scenario, starting from the init model (L1 ), we perform progressive adaptation by initializing the training of amodel at each step (Ln ) with the previous model (Ln−1 ). At
time of reloading the model from Ln−1 , a TL-DV update is

All systems are trained using the Transformer [31] model implementation of the OpenNMT-tf sequence modeling framework2 [32]. At training time, to alternate between dynamic
2 https://github.com/OpenNMT/OpenNMT-tf

56
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Inputs → Input Embeddings → updating Embeddings

Vp1

Enc-Dec

Enc-Dec

Enc-Dec

Enc-Dec

EmbeddingUpdated Embedding

EmbeddingUpdated Embedding

Vp2

Vpi

Init (Ln-1) Data

Vp1

Vp2

Vp1

Vji
c

Ln progAdapt Data

Vp2

Vpi

Init (Ln-1) Data

Vp1

Vp2

Vpi

Vjc

Ln progAdapt Data
Init (Ln-1) Data

Figure 1: Transfer-Learning; (left) from an initial NMT model to a new language pair, model is applied after inserting the new
vocabulary entries, for instance, the initial model Ln−1 parameters are transfered to Ln with the updated embedding space (i.e.,
keeping Vp1 , Vp2 as overlapping entries, while replacing the non-overlapping Vpi with Vcj new language vocabularies), and (right)
from an initial model Ln−1 to Ln , but incorporating both the previous and new language pair data and vocabulary entries.
Outputs → Output Embeddings → updating Output Embeddings

Table 1: Languages and dataset sizes for train, dev, and test
sets of the init model for De-En direction and other pairs
assumed to be received progressively (It-En, Ro-En, Nl-En).
Language
German(De)-En
Italian(It)-En
Romanian(Ro)-En
Dutch(Nl)-En

Train
200k
5k/50k
5k/50k
5k/50k

Dev
1497
1501
1633
1726

Test
1138
1147
1129
1181

Received
init
L2
L3
L4

and static vocabulary, we utilized an updated version of thescript within the same framework. For all trainings, we use
LazyAdam, a variant of the Adam optimizer [33], with an
initial learning rate constant of 2 and a dropout [34, 35]of 0.3. The learning rate is increased linearly in the earlystages (warmup training steps=16, 000), and afterwards it isdecreased with an inverse square root of the training step.
To train our models using Transformer, we employ a uniform setting with 512 hidden units and embedding dimension, and 6 layers of self-attention encoder-decoder network.
The training batch size is of 4096 sub-word tokens. At inference time, we use a beam size of 4 and a batch size of
32. Following [31] and for a fair comparison, all baseline
experiments are run for 100k training steps, i.e., all models
are observed to converge within these steps. The consecutive
experiments converge in variable training steps. However, tomake sure a convergence point is reached, all restarted experiments on Li are run for additional 50K steps. All models
are trained on a GeForce-GTX-1080 machine with a single GPU. Systems are compared in terms of BLEU [36] using
the multi-bleu.perl implementation3 , on the single referencesof the official IWSLT test sets.
4.3. Baseline Models
To evaluate and compare with our approach, we train single
language pair baseline models corresponding to the newly in3 A script from the Moses SMT toolkithttp://www.statmt.org/moses

troduced language pairs at each Li training stage. The baseline models, referred to as Bi-NMT, are separately trained
from scratch in a bi-directional setting (i.e., source ↔ target).
In addition, we report scores from a multilinugal (M-NMT)
model trained with the concatenation of all available data ineach training stage. The alternative baseline are built by
fine-tuning the init model. These models use the vocabulary (word segmentation rules) of the init model, avoiding the proposed dynamic vocabulary. This fine-tuning approach is prevalent in continued model trainings, for adapting NMT models [37, 38] or improving zero-shot and lowresource translation tasks [39, 40, 41]. For the alternative
baseline where we fine-tune init with its static-vocabulary,
we observed that results were mostly analogous to Bi-NMT
models. Hence, we avoided this comparison in this work andrelied on the former baselines.

5. Results and Discussion
Experiments are performed using the progAdapt (see§3.2) and progGrow (§3.3) approaches. The experimental results with the associated discussion are presented inTable 2 for models characterized by relatively low-resource
data (MLR ), and in Table 3 for an extremely low-resource
condition (MELR ). In both dataset conditions, the performance of the proposed approaches is compared with thebaseline systems (Bi-NMT and M-NMT, see §4.3).

The init model which is trained with a data size 4X
larger than MLR and 40X the size of MELR , achieves BLEU
scores of 26.74 and 23.30, respectively, for the De-En and
En-De directions. In Table 2 and 3, the progAdapt is reported for each training stage (i.e., L2 , L3 , and L4 ), whereas
the progGrow is reported for the final stage L4 . Moreover,Table 4 analyzes the effect of language relatedness and training stage reordering in our TL-DV approach. Bold highlighted BLEU scores show the best performing approach,while the ↑↓ arrows indicate statistically significant differences of the hypothesis against the better performing baseline (M-NMT) using bootstrap resampling (p < 0.05) [42].

57
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 2: MLR models performance i) at L1 for the init
De-En direction and baseline (Bi-NMT) It-En, Ro-En, and
Nl-En directions, ii) at L2/3/4 for progAdapt, and iii) at
L4 for the progGrow approach.
Dir

De-En

It-En

Ro-En

Nl-En

Init/Bi-NMT

>
<

26.74
23.30

25.21
22.39

10.80
12.94

21.75
19.75

M-NMT

>
<

24.14
21.80

26.42
23.57

22.17
17.35

ProgAdapt

>
<

-

↑

↑

30.08
26.24

↑

24.43
20.31

↑

ProgGrow

>

26.22

↑

29.61

23.23

↑

Table 3: MELR models performance i) at L1 for the init
De-En direction and baseline (Bi-NMT) It-En, Ro-En, and
Nl-En directions, ii) at L2/3/4 for progAdapt, and iii) at
L4 for the progGrow approach.
Dir

De-En

It-En

Ro-En

Nl-En

Init/Bi-NMT

>
<

26.74
23.30

7.64
5.25

4.56
3.86

5.69
5.14

24.00
21.25

M-NMT

>
<

24.96
21.67

16.26
10.38

12.67
8.67

15.59
12.72

26.36
25.52

ProgAdapt

>
<

-

↑

15.16
14.40

↑

11.03
11.10

↓

24.78

ProgGrow

>

25.61

↓

15.02

↓

11.20

↓

↑

5.1. Low-Resource Setting
For each language pair (i.e., It-En, Ro-En, and Nl-En), the results of the baseline models Bi-NMT trained using the available 50K parallel data (MLR setting) are presented in thefirst two rows of Table 2. The progAdapt results are reported from three consecutive adaptations to new language
directions. These include the init to It-En, followed by
the adaptation to Ro-En, and then to Nl-En. Compared tothe corresponding Bi-NMT and M-NMT models, all of thethree progressive adaptations using the dynamic vocabularytechnique achieved a higher performance gain.
If we look at the specific level of adaption (Li ) against
the Bi-NMT, we observe that the It-En direction showed a+4.87 and +3.85 gain for the En and It target, respectively.
When we take this model and continue the adaptation to RoEn and Nl-En, we see a similar trend where the highest gain
is observed on L3 for the Ro-En direction with +13.63 and+7.37 points. These significant improvements over the baseline models tell us that transfer-learning using dynamic vocabulary in a multilingual setting is a viable direction. Its capability to quickly tune the representation space of the init
model to deliver improved results is an indication of the importance of using different word representations for each language pair4 .
In case of the progGrow, we observed a similar improvement trend as in the progAdapt approach. The results are reported from the final stage (L4 ) of the model
growth, but improvements are consistent throughout the L2and L3 stages. The M-NMT outperformed the Bi-NMT models except for De-En pair. However, compared to the multilingual model as an alternative method for achieving crosslingual transfer-learning, our approach shows improvementsin the consecutive training stages. Overall, our observationis that the suggested progGrow model can accommodate
new translation directions when the data are received. Most
4 We reserve the adaptation from the init model directly to all the three
new language pairs and the comparison with the current setting for future
work.

↓

↓

11.52
13.57
13.56

importantly, improvements are observed for these newly introduced languages without altering the performance of theinit model in the De-En direction.
Specific to each language direction, It-En shows acomparable performance with the progAdapt approach,whereas in case of Ro-En and Nl-En a small degradation
ranging from 0.47 (De-En) to 1.58 (Nl-En) is observed. The
loss in performance is likely due to the increased ambiguities
in the encoder side of the progGrow model, where at bothtraining and inference time there does not exist a disambiguation mechanism between languages except the prepended
language flag. This observation, which sheds a light on our
initial expectation of more data aggregation benefiting themodel performance, requires further investigation.
5.2. Extremely Low-Resource Setting
In a similar way with what we observed in the MLR experiments, the baseline models in the extremely low-resource
setting demonstrate poor performance. Looking at our approaches, we observe a relatively higher gain at the first stageof progAdapt and progGrow. For instance, for the ItEn pair there is a +7.52 improvement compared to the +4.87
in the MLR models (see Table 2) over the Bi-NMT model.
In the subsequent additional language directions (i.e., Ro-En
and Nl-En), we also observe a similar trend. However, incomparison with the M-NMT, both of our approach perform
poorly when translating to the En target. The main reason
for this could be the aggregation of all the available data fora single run in the M-NMT model, while our approaches exploit data when it becomes available in a continuous training.
Alternatively the distance between each language pair could
play a significant role when we adapt in an extremely sparsedata.
prog-Adapt/Grow with Related Languages. When related language pairs are consecutively added (Ln−1 andLn ) at each training stages, our TL-DV approach showed
the best performance. For instance, for the Nl-En experiments, we changed the sequence of the added language pair

58
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 4: MLR and MELR models performance at L1 for
progAdapt and progGrow approaches in a closely related De-En (init) and Nl-En language pairs setting.
MLR
Dir

De-En

ProgAdapt><

ProgGrow

>

Nl-En

-

↑

↑

27.23
25.51

26.62

↑

26.41

MELR
De-En

Nl-En
16.21
15.86

26.52

↑

15.52

moving from a random order to a sequence based on thesimilarity to the init model. Table 4 shows the resultsfrom progAdapt and progGrow, when the Nl-En pairis used at the L1 training stage. The MLR results confirm the trend observed in Table 2, however, with a relatively better performance when translating in to English.
Most importantly, the MELR results show a consistent andlarger gain of +4.69 (Nl-En) and +2.29 (En-Nl) with theprogAdapt, and +1.96 (Nl-En) with progGrow compared to the corresponding results in Table 3. Thus, we emphasize on the degree of language similarity as a direct influencing factor when incorporating a new language pair bothin progAdapt and progGrow approaches. .
Prog-Adapt/Grow with Faster Convergence. The othermain advantage of our TL-DV approach comes from the timea model takes to restart from the init model and reach aconvergence point with better performance. In all experiments with our TL-DV approach a converged model is foundwithin 10K steps for MELR and 20K for MLR training settings. Compared to ≈100K steps needed by a model trained
from scratch to reach good performance, our approach takes
only 4% to 20% of training steps with significantly higher
performance. For instance, taking into consideration the
MELR models, Figure 2 illustrates the steps required for thebaseline systems to converge (Table 3), in comparison with
our approach where progGrow shows to converge slightly
faster than progAdapt. However, with the relatively larger
data of the MLR models, the progAdapt approach proves
to converge much faster than progGrow, for the reason thatthe newly introduced vocabulary and training dataset sizes
are smaller compared to the concatenation of the init and
Li data.
We further analyzed the influence of shared vocabularies
between models Li and Li+1 on the performance of TL-DV.
For this discussion, we took the progAdapt MLR modelfrom all stages. Figure 3 summarizes the improvement differences from consecutive models in relation to the percentage of shared vocabularies. For instance, init and the L2(It-En) model vocabularies have a 47% overlap, whereas L3and L4 share 53% and 51% with the previous model. The interesting aspect of the shared vocabulary comes from the increase in model performance with a higher fraction of shared
vocabulary entires. Thus, a larger number of shared parame-

Figure 2: Model training steps number comparison for thethree different language pairs between the baseline (rightmost) and the proposed approaches in the MELR setting.

Figure 3: The difference in performance between the baseline and progAdapt models (Tgt→Src and Src→Tgt directions) in relation with the shared vocabulary between model
Li and new language pair model Li+1 .
ters between two consecutive models allows for a better gainin performance of the latter.
The results achieved by the transfer-learning with dynamic vocabulary approach in two different training size conditions show that: i) adapting a trained NMT model to a new
language pair improves performance on the target task significantly, and ii) it is possible to train a model faster to achieve
better performance. Overall, the capability of injecting new
vocabularies for new language pairs in the initial model is a
crucial aspect for efficient and fast adaptation steps.

6. Conclusions
In this work, we proposed a transfer-learning approachwithin a multilingual NMT. Experimental results show thatour dynamic vocabulary based transfer-learning improvesmodel performance in a significant way of up to 9.15 in
an extremely low-resource and up to 13.0 BLEU in a lowresource setting over a bilingual baseline model.
In future work, we will focus on finding the optimal wayof transferring model parameters. Moreover, we plan to test
our approach for various languages and language varieties.

59
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

7. Acknowledgments
This work has been partially supported by the EC-funded
project ModernMT (H2020 grant agreement no. 645487).
We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for thisresearch. Moreover, we thank the Erasmus Mundus European Program in Language and Communication Technology.

8. References[11] A. Radford, K. Narasimhan, T. Salimans, andI. Sutskever, “Improving language understanding bygenerative pre-training,” 2017.
[12] A. Sharif Razavian, H. Azizpour, J. Sullivan, and
S. Carlsson, “Cnn features off-the-shelf: an astounding
baseline for recognition,” in Proceedings of the IEEE
conference on computer vision and pattern recognition
workshops, 2014, pp. 806–813.

[1] O. Firat, B. Sankaran, Y. Al-Onaizan, F. T. Y. Vural, and K. Cho, “Zero-resource translation with multilingual neural machine translation,” arXiv preprint
arXiv:1606.04164, 2016.

[13] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2015, pp. 3431–3440.

[2] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu,
Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, et al., “Google’s multilingual neural machinetranslation system: Enabling zero-shot translation,”
arXiv preprint arXiv:1611.04558, 2016.

[14] Y. Qi, D. S. Sachan, M. Felix, S. J. Padmanabhan, and
G. Neubig, “When and why are pre-trained word embeddings useful for neural machine translation?” arXiv
preprint arXiv:1804.06323, 2018.

[3] T.-L. Ha, J. Niehues, and A. Waibel, “Toward multilingual neural machine translation with universal encoderand decoder,” arXiv preprint arXiv:1611.04798, 2016.
[4] P. Koehn and R. Knowles, “Six challenges
for neural machine translation,” arXiv preprint
arXiv:1706.03872, 2017.
[5] B. Zoph, D. Yuret, J. May, and K. Knight, “Transfer
learning for low-resource neural machine translation,”
arXiv preprint arXiv:1604.02201, 2016.
[6] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,”
arXiv preprint arXiv:1508.07909, 2015.
[7] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi,
W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey,
et al., “Google’s neural machine translation system:
Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016.
[8] W. Lewis, “Haitian creole: How to build and ship an mt
engine from scratch in 4 days, 17 hours, & 30 minutes,”in 14th Annual conference of the European Association
for machine translation. Citeseer, 2010.
[9] L. Bentivogli, A. Bisazza, M. Cettolo, and M. Federico,“Neural versus phrase-based mt quality: An in-depth
analysis on english-german and english-french,” Computer Speech & Language, vol. 49, pp. 52–70, 2018.
[10] J. Howard and S. Ruder, “Universal language model
fine-tuning for text classification,” in Proceedings of
the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1,
2018, pp. 328–339.

[15] B. McCann, J. Bradbury, C. Xiong, and R. Socher,“Learned in translation: Contextualized word vectors,”
in Advances in Neural Information Processing Systems,
2017, pp. 6294–6305.
[16] M. E. Peters, W. Ammar, C. Bhagavatula, and
R. Power, “Semi-supervised sequence tagging withbidirectional language models,” arXiv preprint
arXiv:1705.00108, 2017.
[17] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deepcontextualized word representations,” arXiv preprint
arXiv:1802.05365, 2018.
[18] T. Q. Nguyen and D. Chiang, “Transfer learning across
low-resource, related languages for neural machinetranslation,” arXiv preprint arXiv:1708.09803, 2017.
[19] D. Dong, H. Wu, W. He, D. Yu, and H. Wang, “Multitask learning for multiple language translation.” in ACL(1), 2015, pp. 1723–1732.
[20] M.-T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and
L. Kaiser, “Multi-task sequence to sequence learning,”
arXiv preprint arXiv:1511.06114, 2015.
[21] O. Firat, K. Cho, and Y. Bengio, “Multi-way, multilingual neural machine translation with a shared attention
mechanism,” arXiv preprint arXiv:1601.01073, 2016.
[22] J. Lee, K. Cho, and T. Hofmann, “Fully character-levelneural machine translation without explicit segmentation,” arXiv preprint arXiv:1610.03017, 2016.
[23] B. Zoph and K. Knight, “Multi-source neural translation,” arXiv preprint arXiv:1601.00710, 2016.

60
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[24] R. Sennrich, B. Haddow, and A. Birch, “Controlling
politeness in neural machine translation via side constraints,” in Proceedings of the 2016 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies,
2016, pp. 35–40.
[25] H. Yamagishi, S. Kanouchi, T. Sato, and M. Komachi,“Controlling the voice of a sentence in japanese-toenglish neural machine translation,” in Proceedings ofthe 3rd Workshop on Asian Translation (WAT2016),
2016, pp. 203–210.
[26] W. Chen, E. Matusov, S. Khadivi, and J.-T. Peter,“Guided Alignment Training for Topic-Aware Neural
Machine Translation,” in Association for MachineTranslation in the Americas (AMTA), jul 2016.
[Online]. Available: http://arxiv.org/abs/1607.01628[27] C. Chu, R. Dabre, and S. Kurohashi, “An Empirical
Comparison of Domain Adaptation Methods for Neural
Machine Translation,” in Proceedings of the 55th
Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers), 2017, pp. 385–
391. [Online]. Available: http://arxiv.org/abs/1701.
03214%0Ahttp://aclweb.org/anthology/P17-2061[28] M. Elaraby, A. Y. Tawfik, M. Khaled, and A. Osama,“Gender Aware Spoken Language Translation Applied
to English-Arabic,” in Proceedings of the Second
International Conference on Natural Language andSpeech processing, 2018. [Online]. Available: https://arxiv.org/pdf/1802.09287.pdf[29] M. Cettolo, C. Girardi, and M. Federico, “Wit3 : Web
inventory of transcribed and translated talks,” in Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), Trento, Italy,
May 2012, pp. 261–268.
[30] M. Denkowski and G. Neubig, “Stronger baselines fortrustable results in neural machine translation,” in Proceedings of the First Workshop on Neural MachineTranslation, 2017, pp. 18–27.

[34] N. Srivastava, G. E. Hinton, A. Krizhevsky,
I. Sutskever, and R. Salakhutdinov, “Dropout: asimple way to prevent neural networks from overfitting.” Journal of machine learning research, vol. 15,
no. 1, pp. 1929–1958, 2014.
[35] Y. Gal and Z. Ghahramani, “A theoretically grounded
application of dropout in recurrent neural networks,”
in Advances in neural information processing systems,
2016, pp. 1019–1027.
[36] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu:a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting on association for computational linguistics. Association
for Computational Linguistics, 2002, pp. 311–318.
[37] P. Michel and G. Neubig, “Extreme adaptation for personalized neural machine translation,” arXiv preprint
arXiv:1805.01817, 2018.
[38] D. Vilar, “Learning hidden unit contribution for adapting neural machine translation models,” in Proceedingsof the 2018 Conference of the North American Chapterof the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers),
vol. 2, 2018, pp. 500–505.
[39] S. M. Lakew, Q. F. Lotito, N. Matteo, T. Marco, andF. Marcello, “Improving zero-shot translation of lowresource languages,” in 14th International Workshop on
Spoken Language Translation, 2017.
[40] G. Lample, L. Denoyer, and M. Ranzato, “Unsupervised machine translation using monolingual corpora
only,” arXiv preprint arXiv:1711.00043, 2017.
[41] J. Gu, H. Hassan, J. Devlin, and V. O. Li, “Universal
neural machine translation for extremely low resource
languages,” arXiv preprint arXiv:1802.05368, 2018.
[42] P. Koehn, “Statistical significance tests for machinetranslation evaluation,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, vol. 4, 2004, pp. 388–395.

[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin,“Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 6000–6010.
[32] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M.
Rush, “Opennmt: Open-source toolkit for neural machine translation,” arXiv preprint arXiv:1701.02810,
2017.
[33] D. Kingma and J. Ba, “Adam: A method for stochastic
optimization,” arXiv preprint arXiv:1412.6980, 2014.

61
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Machine Translation Human Evaluation:
an investigation of evaluation based on Post-Editing
and its relation with Direct Assessment
Luisa Bentivogli(1) , Mauro Cettolo(1) , Marcello Federico(2) , Christian Federmann(3)FBK - Trento, Italy
Amazon AI - East Palo Alto, CA, USA(3)
Microsoft Cloud+AI - Redmond, WA, USA
(1)

(2)

[bentivo|cettolo]@fbk.eu
marcfede@amazon.com
chrife@microsoft.com

Abstract
In this paper we present an analysis of the two most prominent methodologies used for the human evaluation of MT
quality, namely evaluation based on Post-Editing (PE) andevaluation based on Direct Assessment (DA). To this purpose, we exploit a publicly available large dataset containing
both types of evaluations. We first focus on PE and investigate how sensitive TER-based evaluation is to the type andnumber of references used. Then, we carry out a comparative analysis of PE and DA to investigate the extent to which
the evaluation results obtained by methodologies addressing
different human perspectives are similar. This comparison
sheds light not only on PE but also on the so-called reference
bias related to monolingual DA. Also, we analyze if and how
the two methodologies can complement each other’s weaknesses.

1. Introduction
The evaluation of machine translation (MT) is of crucial importance and has a long research history. Both human andautomatic evaluation have been explored extensively within
the MT community, in the effort to find more and more suitable, efficient and reliable methods and metrics. Automatic
metrics play a central role in the progress of the field and theimprovement of MT quality over time. However, they represent a proxy for human evaluation which – despite being
costly and time-consuming – is to be considered primary.
Among the various human evaluation methods that havebeen devised and tested along the years, currently two approaches have become well-established standards in the field,
namely evaluation based on Post-Editing (PE) and evaluation
based on Direct Assessment (DA).
In the PE-based evaluation, the MT outputs are postedited, i.e. manually corrected, according to the source sentence (bilingual PE) or to an existing reference translation(2) Work conducted while this author was at FBK.

(monolingual PE). The original MT outputs are then evaluated against their post-edited versions through TER-based
automatic metrics [1]. Relying on the post-edit instead of an
independently created reference translation ensures that only
true errors in the MT output are counted, and not those differences due to linguistic variation, which are accounted for bypost-editors. PE has become the standard evaluation metricfor the yearly evaluation campaign of the International Workshop of Spoken Language Translation since 2013 (IWSLT2013) and is described in detail in [2].
The DA-based evaluation [3] consists of collecting human assessments of translation quality for single MT systems. Assessors see a candidate translation and a corresponding translation hint (e.g. the source text, a reference translation, or multimodal content) and are asked to assign a quality
score from 0 to 100. DA has become the standard evaluation metric for the yearly Conference on Machine Translation (WMT) in 2017 [4]. Following the findings of WMT17,
the main focus for DA is on semantic transfer (which corresponds to adequacy) while syntactic transfer (or fluency) has
turned out to be less relevant. Traditionally, in the DA task
MT quality is assessed according to a reference translation,
without access to the source text. This is called referencebased DA (DA-ref ). A problematic issue with DA-ref is
its inherent dependence on reference translations, which canlead to reference bias, both in the form of giving an implicit
boost to candidate translations which are very similar (e.g., insyntax or lexical choice) to the corresponding reference text,
or by penalizing good translations because of translation errors affecting the reference itself. To address the reference
bias, source-based DA (DA-src) can be used, where translation quality is assessed directly according to the source text.
DA-src has been tested on a large scale for the first time in
the IWSLT 2017 evaluation campaign [5].
DA and PE are different and complementary methodologies, not only from the point of view of their design but also
concerning their practical usage. First, the two evaluation

62
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

methods address different human perspectives. Indeed, while
DA focuses on the generic assessment of overall translation
quality, PE-based evaluation reflects a real application scenario – the integration of MT in Computer-Assisted Translation (CAT) tools – and directly measures the utility of agiven MT output to translators. Furthermore, while DA is
based only on human annotators, in PE an automatic component (i.e. TER) is applied to quantify the errors of the MT
output. Finally, in terms of data collection DA is less costly
then PE and thus more viable when used within the research
scenario; however PE has the double advantage of (i) producing a set of additional reference translations, and (ii) being
particularly suitable for performing fine-grained analyses ofthe MT systems, since it produces a set of edits pointing to
specific translation errors [6, 7, 8].
Given the importance of human evaluation for MT improvement and the specific features of these two most prominent frameworks, we present an empirical analysis of thesedifferent methodologies as a contribution to their better understanding.
The analysis is conducted on the publicly available Human Evaluation dataset created as part of the IWSLT 2017
evaluation campaign [5]. The dataset covers two language directions, namely Dutch-to-German and Romanian-to-Italian.
For each direction, it includes DA-src, DA-ref, and PE human evaluation data for nine different state-of-the art neural
MT systems on the same 603 segments. DA evaluation was
performed by linguists, while professional translators carriedout the bilingual PE task. Besides making our study possible, the size, variety and high quality of this three-way evaluation dataset ensure sound empirical analyses and generalizable outcomes.
The main investigations presented in the paper are:
• New analyses on PE data. The availability of multiple
post-edits allows us to investigate how sensitive TERbased evaluation is to the type (external versus postedit) and number of references used, both in terms of
reliability and informativeness of the evaluation;• New comparative analysis of PE and DA. In this empirical comparison we investigate the extent to which
the evaluation results obtained by methodologies addressing different human perspectives are similar. This
investigation gives us insight not only on PE but also
on the relations between DA-src and DA-ref. Also, we
analyze if and how PE and DA can complement eachother’s weaknesses.

2. Related Work
Human Evaluation has always received a lot of attention in
the field of MT and many methodologies have been devised
and tested in different scenarios. The same holds for the two
methods addressed in this paper.
PE-based evaluation was the focus of various studies [1,9, 6] and was commonly employed in large-scale evaluation

campaigns, such as IWSLT [2, 10, 11, 12, 5] and the MT
Quality Estimation Task at WMT-2015 [13].
Also research on DA has been very active since its introduction as method for human evaluation of MT [3, 14].
Large-scale evaluations were carried out through DA-ref [4]and, more recently, also through DA-src [5, 15].
As specifically regards the impact of different numbers
and types of post-edits in PE-based evaluation, a study on
multiple references was presented in [16], but it did not target
PE-based evaluation.
Concerning the issue of reference bias in DA-ref evaluation, it was examined in detail in [17], [18], and [19]. To
this aim, [17] compares directly DA-src and DA-ref but ona very small dataset, not comparable to the one used in ourinvestigation.
As regards the comparative analysis of DA and PE, correlation results between DA-ref and HTER for 9 language
directions are presented in [19]. However, the evaluation
data differs in many respects, making results not comparable. First, the dataset used in this paper includes both DA-refand DA-src. Furthermore, PE data is made of multiple bilingual post-edits created by professional translators native inthe target language and working in their professional CAT
environment. On the contrary, the post-edits used to calculate HTER in [19] were created through monolingual postediting, probably based on the same reference used to collectDA-ref judgments.

3. Evaluation Data
To perform our investigations on DA and PE we relied on the
Human Evaluation dataset created as part of the IWSLT 2017
evaluation campaign [5]. The resource is publicly available
at the WIT3 website [20], where all IWSLT data and tools
are released by the organizers of the campaign. 1
The dataset is based on TED talks2 and includes 603 sentences (around 10,000 source words), corresponding to thefirst half of ten different TED talks. It covers two language
pairs, namely Dutch-German (N lDe) and Romanian-Italian(RoIt) which – belonging to two distinct families (WestGermanic and Romance, respectively) – show rather different characteristics.
For each language direction, evaluation data were collected for nine different state-of-the-art neural MT systems:
three standard bilingual systems (i.e. a different system iscreated for each language direction) and six multilingual systems (i.e. one single system for multiple language directions), out of which three in the zero-shot condition (i.e.
tested on language pairs that are not present in the trainingdata). Furthermore, systems differ also for their architecture,since some of them implement Recurrent Neural Networks,
while others are based on the Transformer model [21].3
1 https://wit3.fbk.eu/show.php?release=201702&page=subjeval&texthead=Evaluation%20Data
2 www.ted.com
3 All details about the MT systems can be found in [22, 23, 24].

63
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The MT systems were evaluated on all the 603 dataset
sentences according to PE, source-based DA, and referencebased DA. Details on human evaluation data are given in thefollowing.
3.1. Post-Editing data
This evaluation was carried out through bilingual postediting: the outputs of the nine MT systems on the 603 test
sentences were assigned to nine professional translators to be
manually corrected directly according to the source sentence.
To ensure the soundness of the evaluation and cope withtranslators’ variability, an equal number of outputs from each
MT system was assigned randomly to each translator, in sucha way that each translator had to post-edit all the sentencesin the test set but only once.
The resulting PE data used in this study consists of nine
new reference translations for each sentence of the test set.
Each one of these references represents the targeted reference of the system output from which it was derived, while
the post-edits of the other systems are available for evaluation as additional references. All details about data preparation and post-editing can be found in [2, 5].
In addition to the PE data, an external - independently
created - reference was also available, for a total of ten references for each of the 603 sentences in the dataset.
3.2. Direct Assessment data
Both DA-src and DA-ref data were collected for all the MT
system outputs on all the 603 test sentences employing bilingual linguists. To ensure the reliability of the human assessments, part of the collected data was used for quality
control. Based on artificially degraded translation output—
which should be scored worse than the corresponding candidate translation—it is possible to identify users who randomly assign scores without paying attention to the presented
data and, thus, work unreliably. Only annotations from reliable annotators were used to compute the final system evaluation. Furthermore, as annotators may have different annotation behaviour, the collected scores (at least two for eachsentence) were standardized into z scores, which capture thenumber of standard deviations a score is different from (i.e.
better or worse than) the respective annotator’s mean score.
Then, z scores were averaged at segment and system level todetermine the overall MT system quality as observed by all
annotators.

4. Analysis of PE-based evaluation
As described in Section 1, evaluation via post-editing isbased on TER, which measures the amount of editing that ahuman would have to perform to change an automatic translation so that it exactly matches a given reference translation.
Since TER is an automatic metric that works on exact word
matching, it is unable to distinguish differences between MT
output and reference due to normal linguistic variation fromthose due to real MT errors.
For this reason the reference translations used in TERbased evaluation (as in all automatic evaluations) play a central role in determining its reliability and informativeness.
It is widely accepted that the most suitable reference toevaluate an MT system is its corresponding post-edit (targeted reference), since it is derived from that specific systemand thus should differ from the MT output only with respect
to the parts of it that are incorrect. External references areat the other hand of the spectrum, since they are manually
generated by translating the source text from scratch, independently from any MT system output. A particular case ofreference is the post-edit of an actual system output which
is not the one under evaluation. In this case the reference
represents one of the many possible translation options andcan indeed differ from the evaluated MT output due to linguistic variation. However, being created starting from an
MT output, it is possible that its peculiar features make it
more suitable to MT evaluation. This type of reference isparticularly interesting since it can be easily gathered, beinga natural by-product of professional translation in the CAT
framework. Finally, the usage of multiple references has often been investigated as a way to address the issue of acceptable linguistic variation, under the assumption that the morereferences the highest the reliability of the evaluation.
In this section we exploited the PE data – i.e. one externalreference and nine post-edits created from the nine evaluated
MT systems – to carry out different analyses aimed at understanding if and how TER-based evaluation is sensitive to thetype and number of references used.
Depending on the reference(s) used in the analysis, werelied on different variants of TER, namely: (i) Humantargeted TER (HTER), where TER is computed between themachine translation and its post-edited version (targeted reference); (ii) Multiple reference TER (mTER), where TER is
computed against the closest reference – i.e. the one which
minimizes the number of edits – among all the available ones.
We empirically analyzed the impact of references in theevaluation from two different angles: (i) for each evaluated
MT system, we investigated the specific contribution of eachof the nine available post-edits to the mTER score of the system; (ii) for each language pair, we calculated how overall
MT system performance (i.e. TER score) varies dependingon the type and number of references used.
Figure 1 shows an example of the distribution of the identity of systems which originated the post-edits that were chosen as closest reference translation in the computation ofmTER. Four NlDe systems are presented in the figure, among
which three were post-edited (BL.lab1, SD.lab2, ZS.lab3)
and one was not (SD.lab4), and is shown for comparison purposes. The same behaviour of the NlDe systems presented in
the figure was observed also for the other NlDe systems aswell as for the RoIt direction.
As expected, the peak occurs in correspondence of the
post-edit of the system under evaluation. Looking at the cor-64
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Figure 1: Frequency distribution of the closest PE selected in the computation of mTER of four NlDe systems. For eachsystem that originated the PE, in orange the number of sentences for which that PE was the closest translation to the MT underinvestigation, in blue the number of sentences where the PE was the closest together with at least another PE.

Figure 2: TERs on single references (green bars) and mTER on increasing number of references (red line).
responding column, we note however that the targeted reference is the closest to the MT output only for around onethird of the test set (orange-coloured), while for another third
there is at least one equivalent post-edit from another system (blue-coloured). Interestingly enough, for the remaining
third of the test set, the closest reference is a post-edit fromanother system.
Looking at the columns of the post-edits originated fromthe other 8 MT systems, we see that for a non-negligible
number of test sentences these references represent the closest translation (orange). This is particularly relevant whenconfronted to the results of the external reference translation,
which is not shown in the figure since it was never picked as
the closest reference translation. It is also worthwhile to notethat the post-edits of other MT systems created by the same
Lab – which are expected to have similar outputs – are notchosen as closest references significantly more often than the
post-edits of other Labs’ systems. This suggests that the advantage of the post-edits of other systems does not rely in thesimilarity of the MT systems but more generally in the factthat the reference translation is derived from an MT output.
From the point of view of the number of references usedin the evaluation, we understand from Figure 1 that a certain
degree of variability is present also in the targeted translation– since for one-third of the test set it does not ensure thelowest edit distance with the MT output. We can thus confirm
that – even when a targeted reference is available – mTER
guarantees the highest reliability of the evaluation. Finally,
the rightmost part of Figure 1 presents results for a system(SD.lab4) for which no post-edit was created. We can seethat the closest references are equally distributed among all
the available references, further confirming the importanceof having multiple references.
The same conclusions can be drawn by analyzing theoverall performances of the MT systems when using different
reference translations. For each language direction, Figure 2
shows the impact that each of the ten references at our disposal has on TER, averaged across systems. The vertical bars
provide the TER score computed using a single reference, be
it one of the external post-edits, the targeted post-edit, or theexternal reference; for each system, the PEs are considered inreverse order with respect to their overall score, that is fromthe farthest to the closest to the system output, which invariably is the targeted PE; the external reference is presented as
the last; the red line represents the mTER computed on an
incremental set of references.
The low TER results obtained using a single non-targeted
post-edit are quite interesting. Indeed evaluating a system

65
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

against a post-edit created for another system is more soundthan using an external reference. This is particularly relevant
in a real application scenario where obtaining a post-edit ofa system is easy and inexpensive. On the same line, considering the mTER cumulative score, it is interesting to see thatthe same HTER results obtained with the targeted reference(trgPE, dark green bar) can be achieved using seven external
post-edits for the NlDe direction and six for the RoIt direction.
For completeness, Table 1 gives the exact figures of themost relevant information contained in Figure 2, namely
mTER using all 9 available post-edits, HTER, and TER overthe external reference.
Indeed we can observe a considerable TER reduction
when using all collected post-edits with respect to both the
HTER obtained using the targeted post-edit and the TER
obtained using the independent reference. This reduction
clearly confirms that exploiting all the available referencetranslations allows to produce a score which is not only morereliable but also more informative about the real performanceof the systems.

NlDe
RoIt

mTER
9 PE refs
23.80
23.64

HTER
tgt PE
29.96
31.25

TER
1 ext ref
66.10
61.56

Table 1: %TERs computed on different (set of) references.

5. Comparative analysis of DA-based andPE-based evaluation
As introduced in Section 1, the DA-based and PE-based evaluation tasks focus on different aspects of automatic translation: general quality for the reader and usefulness for translator, respectively. To investigate the extent to which PE andDA lead to similar results, for each evaluated system we calculated the Pearson correlation between PE-based scores andDA-based scores for each sentence in the test set. The correlation results obtained for each system were then averaged
through the Fisher transformations suggested in [25].
Table 2 presents the average correlation results. Correlations are calculated for both DA-src and DA-ref and for
all the metrics investigated for PE-based evaluation, namely
mTER, HTER and TER.
As expected, correlation is good, that is, in general segments judged as poor by DA annotators (low DA scores) alsoneed substantial post-editing (high PE scores) or vice-versa.
Results slightly vary across language directions, but thesame trends can be observed. First, the highest correlation
is found between DA-src and mTER, confirming that these
are the two most highly reliable human evaluation measures.
As regards PE, mTER correlates better than HTER with DA,
showing once again the importance of having multiple references. As regards DA, correlation with PE is considerably

Figure 3: ZS.lab3 RoIt system: scatter plot of source-based
DA standardized scores and mTER scores.
higher for DA-src than DA-ref. This indicates that the socalled reference-bias affects not only automatic metrics but
also DA-based human evaluation. This is further confirmedby the results obtained for TER, which is calculated on thesame external reference translation used in DA-ref. Although
TER correlation scores are very low, TER correlates muchbetter with DA-ref than DA-src, showing an opposite behaviour with respect to mTER and HTER.
Given the correlation results obtained, we carried out afurther analysis to investigate whether having both evaluations can help improving the evaluation quality, i.e. whether
the two methodologies can complement each other’s weaknesses.
Figure 3 shows the scatter plot of the correlation between
mTER and DA-src for one of the investigated RoIt systems(r=–0.5812). Conflicting evaluations appear in the lower-leftand upper-right quadrants of the scatter plot. The first quadrant includes segments which resulted good according to PE
evaluation (low PE scores) but were judged as poor by DA
annotators (low DA scores); the second includes segments
which needed substantial post-editing (high PE scores) but
were judged as good by DA annotators (high DA scores).
Conflicting evaluation cases are particularly relevant
since PE is known to be more informative (see Section 1),
but DA could identify issues that PE-based evaluation cannot spot. We manually inspected a sample of the sentenceswith conflicting evaluations and we found some interesting
patterns. Examples are provided in Table 3.
When PE scores are low (i.e. few edits are needed tocorrect the MT output) but the translation is bad according
to DA, typically the sentence contains few but crucial errors,
which make it difficult to understand the meaning of the sentence (see Example 1 in the table). In these cases, the conflictis not solvable since from the point of view of DA - which is
focused on adequacy - the MT output is rightfully not good,while from the point of view of the translator who has access to the source sentence, the MT output is indeed usefulto speed-up translation.
In the opposite situation, i.e. high PE scores but goodtranslation according to DA, we have two main causes for

66
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

NlDe
RoIt
mTER HTER TERmTER HTER TER
src -0.5466 -0.4796 -0.1918 -0.5294 -0.4306 -0.2137
zDA
ref -0.4491 -0.4100 -0.3579 -0.4524 -0.3882 -0.3570
avg(r)

Table 2: Average (DA,PE) correlations across systems.

1.

2.

SRC

Nu are flapsuri, balamale, eleroane, actuatoare sau alte suprafee de control, doar o simplă elice.
It has no flaps, no hinges, no ailerons, no actuators, no other control surfaces, just a simple propeller.

MT

Non ha fiori, balconi, elenchi, attuatori o altre superfici di controllo, solo una semplice elica.
It has no flowers, no balconies, no lists, no actuators, no other control surfaces, just a simple propeller.

PE

Non ha flaps, cerniere, alettoni, attuatori o altre superfici di controllo, solo una semplice elica.

SRC

Prietenele mele, feministe convinse, au fost s, ocate.
My [female] friends, committed feminist, were aghastMT

I miei amici, femministe convinti, sono rimasti scioccati.
My [male] friends, committed feminist, were aghast.

PE

Le mie amiche, femministe convinte, sono rimasti scioccate.

mTER

DA-src (abs)

14.43%

28

47.87%

88

Table 3: RoIt language direction. Examples of conflicting DA-PE evaluation.
conflicts. First, we found very short or long sentences which
are indeed good translations but the mTER score was not correct due to tokenization (and consequently alignment) problems. These cases highlight the main weakness of PE-based
evaluation, namely the fact that it relies on automatic metrics
to compute the edit distance. The other type of conflict (seeExample 2 in the table) regards those segments that have tobe heavily post-edited for amending errors which do not alter
the overall comprehension, like in chains of morphologicalerrors. In these cases, the MT errors affect more fluency thanadequacy, to which DA-based assessment is less sensitive.

6. Conclusions
In order to shed light on the properties, strengths andweaknesses of human evaluation it is crucial to rely onhigh quality datasets. The specific characteristics of the
IWSLT-17 Human Evaluation dataset used in this investigation - size, variety and high quality of the three-way
human evaluation - ensured sound empirical analyses andgeneralizable outcomes. The main findings of this paper are
summarized in the following.
Analysis on PE evaluation data:• the targeted reference is the closest to the MT output
only for one-third of the test sentences. Thus, mTER
guarantees the highest reliability of the evaluation overHTER;
• evaluating a system against a post-edit created for another system is more sound than using an external reference, independently from the similarity of the two
MT systems;• the same results obtained with the targeted reference(HTER) can be achieved using six/seven external postedits (mTER), not including the targeted reference.
Comparative analysis of DA and PE:• the highest correlation is found between DA-src andmTER, confirming that these are the two most highly
reliable human evaluation measures;• correlation with PE is considerably stronger for DAsrc than DA-ref. This indicates that the so-calledreference-bias affects not only automatic metrics but
also DA-based human evaluation;• conflicting evaluations between DA-src and mTER exist. In some cases DA-src can help mitigate the weakness of PE which depends on its automatic component.
In other cases conflicts are caused by inherent differences due to the fact that the two evaluation methods
address different human perspectives.
To conclude, we are planning to extend our research on
both the analyses presented in this paper. First, we will further verify and generalize the results obtained on PE data by
carrying out the analyses on other publicly available IWSLT
datasets, which include multiple post-edits for other language directions such as English-German, English-French,
and Vietnamese-English. Second, we will compare moredeeply how DA-ref and DA-src behave on the same data.
Finally, we will perform the manual analysis also on NlDe
data.

7. References[1] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, andJ. Makhoul, “A study of translation edit rate with targeted human annotation,” in Proc. of AMTA, Cambridge, US-MA, 2006, pp. 223–231.

67
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[2] M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and
M. Federico, “Report on the 10th IWSLT evaluation
campaign,” in Proc. of IWSLT, Heidelberg, Germany,
2013.
[3] Y. Graham, T. Baldwin, A. Moffat, and J. Zobel,“Continuous measurement scales in human evaluationof machine translation,” in Proceedings of the 7th
Linguistic Annotation Workshop and Interoperability with Discourse. Association for Computational
Linguistics, 2013, pp. 33–41. [Online]. Available:
http://www.aclweb.org/anthology/W13-2305[4] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham,
B. Haddow, S. Huang, M. Huck, P. Koehn, Q. Liu,
V. Logacheva, C. Monz, M. Negri, M. Post, R. Rubino,
L. Specia, and M. Turchi, “Findings of the 2017 conference on machine translation (wmt17),” in Proceedingsof the Second Conference on Machine Translation,
Volume 2: Shared Task Papers. Copenhagen, Denmark: Association for Computational Linguistics,
September 2017, pp. 169–214. [Online]. Available:
http://www.aclweb.org/anthology/W17-4717[5] M. Cettolo, M. Federico, L. Bentivogli, J. Niehues,
S. Stüker, K. Sudoh, K. Yoshino, and C. Federmann,“Overview of the iwslt 2017 evaluation campaign,” inProc. of IWSLT, Tokyo, Japan, 2017.
[6] M. Denkowski and A. Lavie, “Choosing the right evaluation for machine translation: An examination of annotator and automatic metric performance on humanjudgment tasks,” in Proceedings of the 9th Conferenceof the Association of Machine Translation in the Americas (AMTA), Denver, CO, USA, 2010.
[7] L. Bentivogli, A. Bisazza, M. Cettolo, and M. Federico, “Neural versus phrase-based machine translation
quality: a case study,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language
Processing. Austin, Texas: Association for Computational Linguistics, November 2016, pp. 257–267.
[Online]. Available: https://aclweb.org/anthology/D161025[8] ——, “Neural versus phrase-based mt quality: An indepth analysis on english-german and english-french,”
Computer Speech Language, vol. 49, pp. 52 – 70, 2018.
[9] M. Snover, N. Madnani, B. J. Dorr, and R. Schwartz,“Fluency, adequacy, or hter?: exploring different human judgments with a tunable mt metric,” in Proceedings of the Fourth Workshop on Statistical MachineTranslation. Association for Computational Linguistics, 2009, pp. 259–268.
[10] M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and
M. Federico, “Report on the 11th IWSLT evaluation

campaign, IWSLT 2014,” in Proc. of IWSLT, Lake
Tahoe, US-CA, 2014.
[11] M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, R. Cattoni, and M. Federico, “The IWSLT 2015 evaluation
campaign,” in Proc. of IWSLT, Da Nang, Vietnam,
2015.
[12] ——, “The IWSLT 2016 evaluation campaign,” inProc. of IWSLT, Seattle, US-WA, 2016.
[13] O. Bojar, R. Chatterjee, C. Federmann, B. Haddow, M. Huck, C. Hokamp, P. Koehn, V. Logacheva,
C. Monz, M. Negri, M. Post, C. Scarton, L. Specia, and
M. Turchi, “Findings of the 2015 workshop on statistical machine translation,” in WMT@EMNLP, 2015.
[14] Y. Graham, T. Baldwin, A. Moffat, and J. Zobel, “Ismachine translation getting better over time?” in Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics,
2014, pp. 443–451.
[15] H. Hassan, A. Aue, C. Chen, V. Chowdhary,
J. Clark, C. Federmann, X. Huang, M. JunczysDowmunt, W. Lewis, M. Li, S. Liu, T. Liu,
R. Luo, A. Menezes, T. Qin, F. Seide, X. Tan,
F. Tian, L. Wu, S. Wu, Y. Xia, D. Zhang,
Z. Zhang, and M. Zhou, “Achieving human parity
on automatic chinese to english news translation,”
CoRR, vol. abs/1803.05567, 2018. [Online]. Available:
http://arxiv.org/abs/1803.05567[16] A. Lommel, “Blues for bleu: Reconsidering the validity of reference-based mt evaluation,” in Proceedingsof the LREC 2016 Workshop Translation EvaluationFrom Fragmented Tools and Data Sets to an Integrated
Ecosystem, 2016.
[17] M. Fomicheva and L. Specia, “Reference bias
in monolingual machine translation evaluation,” inProceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume
2: Short Papers). Association for Computational
Linguistics, 2016, pp. 77–82. [Online]. Available:
http://www.aclweb.org/anthology/P16-2013[18] Q. Ma, Y. Graham, T. Baldwin, and Q. Liu, “Further
investigation into reference bias in monolingual
evaluation of machine translation,” in Proceedings ofthe 2017 Conference on Empirical Methods in Natural
Language Processing. Association for Computational
Linguistics, 2017, pp. 2476–2485. [Online]. Available:
http://aclweb.org/anthology/D17-1262[19] Y. Graham, T. Baldwin, M. Dowling, M. Eskevich,
T. Lynn, and L. Tounsi, “Is all that glitters inmachine translation quality estimation really gold?”

68
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics:
Technical Papers. The COLING 2016 Organizing
Committee, 2016, pp. 3124–3134. [Online]. Available:
http://www.aclweb.org/anthology/C16-1294[20] M. Cettolo, C. Girardi, and M. Federico, “WIT3 : Web
inventory of transcribed and translated talks,” in Proc.
of EAMT, Trento, Italy, May 2012. [Online]. Available:
http://hltshare.fbk.eu/EAMT2012/html/Papers/59.pdf[21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in
Neural Information Processing Systems 30, I. Guyon,
U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus,
S. Vishwanathan, and R. Garnett, Eds. Curran
Associates, Inc., 2017, pp. 5998–6008. [Online].
Available: http://papers.nips.cc/paper/7181-attentionis-all-you-need.pdf[22] R. Dabre, F. Cromieres, and S. Kurohashi, “Kyoto university MT system description for IWSLT 2017,” inProc. of IWSLT, Tokyo, Japan, 2017.
[23] C. España-Bonet and J. van Genabith, “Going beyond
zero-shot MT: combining phonological, morphologicaland semantic factors. The UdS-DFKI system at IWSLT
2017,” in Proc. of IWSLT, Tokyo, Japan, 2017.
[24] S. M. Lakew, Q. F. Lotito, M. Turchi, M. Negri, and
M. Federico, “FBKs multilingual neural machine translation system for IWSLT 2017,” in Proc. of IWSLT,Tokyo, Japan, 2017.
[25] D. M. Corey, W. P. Dunlap, and M. J. Burke, “Averaging correlations: Expected values and bias in combined
Pearson rs and Fisher’s z transformations,” The Journalof General Psychology, vol. 125(3):245-261, 1998.

69
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The USTC-NEL Speech Translation system at IWSLT 2018
Dan Liu1,2 , Junhua Liu1,2 , Wu Guo1
Shifu Xiong2 , Zhiqiang Ma2 , Rui Song2 , Chongliang Wu2 , Quan Liu2
University of Science and Technology of China 1
IFLYTEK Co. LTD. 2{danliu,jhliu}@mail.ustc.edu.cn wuguo@ustc.edu.cn{danliu, jhliu, sfxiong, zqma2, ruisong, clwu4, quanliu}@iflytek.com

Abstract
This paper describes the USTC-NEL (short for ”National Engineering Laboratory for Speech and Language Information
Processing University of science and technology of china”)
system to the speech translation task of the IWSLT Evaluation 2018. The system is a conventional pipeline system which contains 3 modules: speech recognition, postprocessing and machine translation. We train a group of
hybrid-HMM models for our speech recognition, and for machine translation we train transformer based neural machinetranslation models with speech recognition output style text
as input. Experiments conducted on the IWSLT 2018 task indicate that, compared to baseline system from KIT, our system achieved 14.9 BLEU improvement.

1. Introduction
Conventional speech translation systems consist of threecomponents: source-language automatic speech recognition(ASR), post-processing over ASR outputs, and source-totarget text translation. This pipeline system suffers from error accumulation, which means speech recognition and translation models trained separately may perform well individually, but do not work well together because their error surface
do not compose well [1].
In the most recent years, end-to-end speech translation
based on encoder-decoder with attention mechanisms has
been very promising for reducing accumulated errors [2, 1,
3]. However, parallel speech data is much smaller than thoseavailable to train text-based machine translation (MT) systems, particularly neural systems that needs to learn a relatively large parameters. As a result, an end-to-end speechtranslation system can often outperform pipeline systems
with same training data, but is hard to beat pipeline systemwith dozens of training data [1].
In addition, to translate very long speech (e.g. translatea full talk), an end-to-end system must rely on voice activity
detection (VAD) method to split raw audio into sentence-likefragments, in which mis-segmented sentence fragments arevery likely to cause serious translation errors. Therefore, forpipeline systems, sentence re-segmentation based on ASR

results may be done in post-processing step, which can improve performance significantly [4].
To reduce the error accumulation of pipeline systems, we
introduce a data augmentation based solution to train translation model with ASR results as source directly, instead ofnormalize ASR results (e.g. insert punctuations, normalization for case, numerals, etc.) in post-processing. Text normalization cannot bring any new information, it just produces texts that translation system likes, and this may leadto additional errors. In our experiments, the data augmentation based solution performs significantly better than pipeline
system with text normalization and end-to-end speech translation system.
This paper is organized as follows. We first describe theprocessing for speech and text training data in Section 2, following is our full system and training details. Our experiments are presented in Section 4.

2. Data Processing
We conduct experiments on IWSLT speech translation
task [5] from English to German. All experiments were performed under requirements of IWSLT 2018 evaluation campaign speech translation task. The training data for speech
recognition and translation after filtering are listed in Table 1
and Table 2.
Table 1: speech training data.
CorpusTED LIUM2 [6]Speech Translation# of seg.
92976
171121

Speech hours
207h
272h

2.1. speech recognition training data
The speech data contains TED LIUM2 [6] and speech translation data by IWSLT evaluation campaign. In TED LIUM2,
only raw wave files and manual transcriptions (without punctuation) were offered. And in speech translation data, raw
wave files, English transcriptions and the corresponding German translations were offered, but some transcriptions is not

70
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

then we use the supervised convolutional neural network
method [8] to select monolingual training data that are close
to the TED domain. After this processes, we select 91M
monolingual English data and 43M mono-lingual German
data for language model training.

Table 2: text training data.
Corpuscommoncrawl
rapid
europalcommentary
paracrawl
opensubtitles
WIT3(in domain)

raw
2.39M
1.32M
1.92M
0.284M
36.35M
22.51M
0.209Mfiltered
1.80M
1.00M
1.81M
0.233M
12.35M
14.24M
0.207M

3. System Description
3.1. speech recognition

match to there corresponding audio. Besides this, about 166
hours of audio in speech translation data were not labeled,
we regard them as unsupervised data.
To utilize those data, we firstly train initial acoustic
model based on TED LIUM2. Using this model, we do forcealignment on IWSLT speech translation data, and discard utterances with significantly abnormal scores. After this process, the supervised data size of IWSLT has been reduced
to 246 hours from 272 hours. Meanwhile, the unsupervised
data is recognized by our initial model and filtered based on
ASR confidence to expand the training set.
To further increase the amount of data in the training set,
we perform data augmentation by noise and speed perturbations. For each speech signal, a noise version is createdinitially. Speed perturbation is then performed on the raw
signals with speed factors 0.8 and 1.2. Eventually, up to(207+246+166)*4 hours of data may be used.
2.2. speech translation training data
The speech translation training data is the same as the speech
recognition training data. The target references for LIUM2
and unsupervised data are generated by our best text machinetranslation system.
2.3. text translation training data
The text translation training data contains parallel data andmonolingual training data. As for parallel data, we use allof the allowed training data for Speech Translation Task
which includes TED corpus, data provided by WMT 2018
and OpenSubtitles2018 [7]. The data is pre-processed before
training and translation. Sentences longer than 100 words
and duplicated sentence pairs are removed. Also, numbers
are normalized in order to match the ASR outputs. NMT
systems are more vulnerable to noisy training data, rare occurrences in the data, and the training data quality in general.
So we measure the cross-lingual similarities between source
and target sentences, and then reject sentences with similarity
below a specified threshold. After filtering, we can get relevant and high quality data. The training data after filtering
are listed in Table 2.
As for monolingual training data provided by WMT2018, we clean the noisy data for English and German, andThe primary system of our speech recognition is a hybridHMM system. The acoustic model contains multiple deepneural networks based on CNN and LSTM structure. Statelevel posterior fusion technique is used for the final ASR results. The details of model structure and training criterion
are as following:
1. DenseNet [9]: DenseNet with 13 dense connection
blocks and 3 max-pooling steps with stride 2 on bothtime and frequency domain, trained with cross-entropy(CE) and sequence-discriminative training (SDT) criterion [10].
2. BiLSTM [11]: 3 layers BiLSTM network trained with
CE and SDT criterion.
3. CLDNN [12]: CNN-BiLSTM-DNN structure trainedwith CE and SDT criterion.
The language models are trained on English monolingual data described in Section 2.3. The first-pass decoding
is performed with the HMM and 3-gram LM. A 4-gram LM
is used for second-pass decoding and followed by a LSTMbased LM.
In this task we should do speech recognition on full talk,so we have to split the raw audio into sentence-like pieces forspeech recognition. We do speech segmentation with LSTM
based VAD model, which is trained on TED LIUM2 datasetwith speech/nonspeech labels extracted by force alignment
with our hybrid-HMM model.
3.2. post-processing vs data augmentation
It has been shown that post-processing is crucial for achieving good speech translation performance [4], this comes from
two aspects. First, segmentation boundaries for ASR arebased on VAD, which inevitably leads to fragments withincomplete semantics, and sentence re-segmentation basedon ASR results is needed. Second, translation models aretrained with written text as input, which means text normalization of ASR results is essential for conventional systems.
We know punctuations may contain rich semantic information, but in post-processing for speech translation, punctuations are only generated from ASR output word sequences.
In this case, these punctuations can not bring more information than words. The main goal of post-processing is just to
produce text suitable for machine translation. However, it
should be noted that errors in punctuation prediction may be
propagated in machine translation process.

71
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Here we introduce a new solution with respect to mismatch between ASR results and machine translation inputs.
Instead of transform ASR results to written text on decoding step, we transform the source text for machine translation training data into the style of ASR results on training
step. The difficulty of normalizing ASR results to written
text seems equal to the difficulty of normalizing written text
to ASR results. However, data augmentation with fake ASR
results for machine translation is more robust for errors compared to text normalization on decoding step.
We train a neural machine translation (NMT) model totranslate written text into ASR results. To build the training
data, we process the English written data by rule (remove
punctuations, lower case and translate Arabic numerals into
English words), the generated text is similar to ASR results
except for recognition errors. We also build real data with
the ASR results and source written texts provided in speechtranslation dataset. The NMT model from written text to
ASR results are trained on these two dataset and fine-tuned
on only real data. This model may generate ASR output styletext with common ASR errors. And we augment the text
machine translation dataset by translating the source written
texts into ASR output style texts. As a comparison, we also
trained an inverted NMT for text normalization.
The data augmented based solution can translate directlyfrom ASR result, which reduces errors caused by text normalization. Besides this, our model has the ability to toleratecommon recognition errors. E.g., our ASR system may mistake “two” to “to” in some special contexts, and our NMT
system may translate “top to percent” to “top zwei Prozent”.
Sentence re-segmentation are still important to speechtranslation system, because training data for machine translation are all semantically complete sentences. Data augmentation with semantic incomplete sentence fragments may sufferfrom reordering between source and target language. So wetrain a LSTM based model to re-segmented sentences based
only on text infomation. This model is trained on TED and
OpenSubtitle dataset, with one whole paragraph as input, andthe punctuation ”.!?” as sentence boundaries.

3.3.2. end-to-end speech translation

3.3. machine translation

Table 3: WER for speech recognition with different training
data on dev2010

3.3.1. text machine translation
Transformer [13] is adopted as our baseline, all experiments
use the following hyper-paramter settings based on Tensor2Tensor transformer relative big settings 1 . This corresponds to a 6-layer transformer with a model size of 1024,
a feed forward network size of 8192, and 16 heads relative
attention. Model is trained on the full dataset described in
Section 2.3 and fine-tuned on speech translation dataset. We
trained both conventional NMT model and NMT model withaugmented data described in Section 3.2.
1 https://github.com/tensorflow/tensor2tensor/tree/v1.6.3

For our end-to-end speech translation model, DenseNet described in Section 3.1 followed by one BiLSTM layer is employed as encoder, and the decoder is same as transformer
model in Section 3.3.1. It is difficult to train speech translation model from random initialization parameters, for reordering between source and target language are difficult toalign with frame based speech representations. Pre-training
with speech recognition task significantly improves the performance. And this encoder-decoder based ASR model isused for rescoring our final ASR results.
End-to-end speech translation system has no chance tore-segment sentences. We found splicing audio segments acquired by VAD may improve the translation performance, but
still has a significant gap to performance based on sentence
re-segmentation.

4. Experimental Results
In this section, we present a summary of our experiments
for the IWSLT 2018 speech translation evaluation task. We
test WER (word error rate) for our speech recognition system on dev2010, which is the only dataset with CTM format
transcriptions. And we test our speech translation systems
on IWSLT dev2010, tst2010, tst2013, tst2014 and tst2015.
Case sensitive BLEU based on realigning system outputs toreference by minimizing WER [14] is used for our speechtranslation evaluation metric.
4.1. Results of Speech Recognition
In this section, we demonstrate the results of our ASR system. The acoustic model of our primary system is the deep
CNN model, and we decode with 3-gram for first-pass decoding and 4-gram for second-pass. We test our performancein dev2010. First, we compare the impact of training data in
Table 3. Here “spv.” represents supervised data, “usv.” represents unsupervised data and “spd.” represents speed disturbed data. As show in Table 4, by training with noisy data,
the WER is relatively reduced by 7.32%.

Training Data
spv.
noisy spv.
noisy spv. usv.
noisy spd. spv. usv.

WER
9.7
8.99
8.92
8.86

Based on the above results, we train three acoustic models with different structures. Further promotion is achieved
by fusing multiple acoustic models, rescoring with RNNLM. We also test the encoder-decoder based speech recognition model described in Section 3.3.2, which performs significantly worse than our hybrid-HMM systems. But rescoring

72
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

with encoder-decoder system brings a small improvement.
Details are showed in Table 4.

found the models with text regularization and data augmentation can be combined to get better results.

Table 4: Results of fusion of different models for speech
recognition on dev2010 .

4.3.2. fusion of different models

DenseNetBiLSTM
CLDNN
Encoder-Decoder
DenseNet +BiLSTM + CLDNN+ RNN+Encoder-Decoder

8.86
8.72
8.40
14.64
8.22
7.61
7.3

We train 3 groups of different models, one for text regularization and two for data augmentation (L2R and R2L, which denotes the target order left to right and right to left). For eachgroup we train 4 models with different initialized parameters, and decoded with the ensemble models to get 80-best
hypotheses with beam size of 80. The 3 groups of hypotheses are merged and rescored by all translation models, target language model and end-to-end speech translation model.
Performances are shown in Table 7.
4.4. Submission Results

4.2. Results of End-to-end Speech Translation
In this section, we describe our experiments on end-to-end
speech translation. The average BLEU score of our baseline end-to-end speech translation system is 20.50, which issignificantly worse than our pipeline system (Tabel 6). The
degradation comes from two aspects. Firstly, our encoderdecoder speech recognition performs worse than baseline
speech recognition system (WER 7.61% to14.64%). Secondly, the end-to-end system has no chance to re-segment
sentences based on source recognition results.
To reduce the influence of incomplete sentence fragments
caused by VAD, we splice the VAD fragments to at least 10
seconds, which brings the improvements of about 1 BLEU.
For comparison, we present the performance of a system thatre-segment audio based on speech recognition results, whichbrings another 1.3 BLEU gain, but this is not a ”end-to-end”
system. At last, the ensemble of 4 different models improves
about 1 BLEU compared to corresponding single model. The
details are showed in Table 5.
4.3. Results of Pipeline Speech Translation
In this section, experiments are all based on the best ASR results described in Section 4.1. At test time, we use a beam
size of 80 and a length penalty of 0.6. All data used for training are described in Section 2. All reported scores are computed using IWSLT speech translation evaluation metric.
4.3.1. post processing
The post processing procedure.includes two parts: sentence
re-segmentation and text normalization. And we introduced
one data augmentation based solution to remove text normalization. We compare the performance for different solutionsin Table 6.
We see sentence re-segmentation has a huge impact on
performance. Since sentence-like pieces obtained by VAD
do not carry any semantic information, it is very unfavorable
for machine translation. Other than this, our data augmentation based solution achieves a average BLEU score of 28.76,
1.3 BLEU higher over system with post processing. And we

We submitted 3 systems for speech translation task. The primary system is the best fusion system demonstrated at row
7 in Table 7, and the contrastive systems are all based on
encoder-decoder model from audio features. Contrastive0 isbased on sentence re-segmentation with source speech recognition results, which is not real ”end-to-end”, while contrastive2 is real end-to-end systems with only single model.
We compared our submitted systems to KIT speech translation system (noted as ”Baseline KIT”)2 , which is the baseline system provided by KIT, performance is shown in Table
7. Our primary system achieves a average BLEU of 30.26,
which is 14.9 BLEU higher than baseline from KIT.

5. Conclusion
In this paper we presented our speech translation systems for
IWSLT 2018 evalution. Our results indicated that the end-toend system still performs significantly worse than the conventional pipeline system, and NMT with data augmentation performs better than solutions with text regularization.
Our best ensemble system achieved 14.9 BLEU improvement compared to baseline system from KIT.

6. References[1] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu,
and Z. Chen, “Sequence-to-sequence models candirectly translate foreign speech,” arXiv preprint
arXiv:1703.08581, 2017.
[2] L. Duong, A. Anastasopoulos, D. Chiang, S. Bird, andT. Cohn, “An attentional model for speech translation
without transcription,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, 2016, pp. 949–959.
[3] A. Bérard, L. Besacier, A. C. Kocabiyikoglu, and
O. Pietquin, “End-to-end automatic speech transla2 https://github.com/jniehues-kit/SLT.KIT

73
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 5: BLEU scores for end-to-end speech translation .
systemVAD
splice 10s
re-segment
ensemble(splice)ensemble(re-segment)

dev2010
21.45
22.14
23.79
23.43
24.78

tst2010
21.41
22.16
24.18
22.97
24.92

tst2013
21.76
22.76
24.18
23.58
25.41

tst2014
20.06
21.00
22.22
21.96
23.23

tst2015
17.83
19.52
20.07
20.67
21.01

average
20.50
21.52
22.89
22.52
23.87

Table 6: BLEU scores for pipeline speech translation systemre-segment
N
N
Y
Y

text regularization
Y
N
YN

data augmentation
N
Y
N
Y

dev2010
26.47
27.58
27.75
28.98

tst2010
27.71
28.26
28.90
29.98

tst2013
28.04
29.81
29.01
30.69

tst2014
25.65
26.79
26.88
28.19

tst2015
24.00
25.65
24.52
25.99

average
26.37
27.62
27.41
28.76

Table 7: BLEU scores for fusion systems
systemtext normalization
augment L2R
augment R2L
fusion+target LM+e2e model

dev2010
28.64
29.45
28.42
30.28
30.30
30.50

tst2010
29.41
30.01
29.58
31.01
31.00
31.06

tst2013
29.59
30.78
30.88
32.28
32.37
32.31

tst2014
27.37
28.37
27.98
29.38
29.44
29.35

tst2015
25.13
26.14
26.47
27.40
28.14
28.06

average
28.03
28.95
28.66
30.07
30.25
30.26

Table 8: performance of submitted systems
systemBaseline KITPRIMARY
Contrastive0
Contrastive2

end2endNN
N
Y

single modelY
NN
Y

dev2010
17.07
30.50
24.78
22.14

test2010
12.37
31.06
24.92
22.16

test2013
16.59
32.31
25.41
22.76

test2014
15.42
29.35
23.23
21.00

test2015
15.15
28.06
21.01
19.52

74
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018average
15.32
30.26
23.87
21.52

tion of audiobooks,” arXiv preprint arXiv:1802.04200,
2018.
[4] E. Cho, J. Niehues, and A. Waibel, “Nmt-based segmentation and punctuation insertion for real-time spoken language translation,” Proc. Interspeech 2017, pp.
2645–2649, 2017.
[5] C. Mauro, F. Marcello, B. Luisa, N. Jan, S. Sebastian, S. Katsuitho, Y. Koichiro, and F. Christian,“Overview of the iwslt 2017 evaluation campaign,” inInternational Workshop on Spoken Language Translation, 2017, pp. 2–14.
[6] A. Rousseau, P. Deléglise, and Y. Esteve, “Enhancing the ted-lium corpus with selected data for language
modeling and more ted talks.” in LREC, 2014, pp.
3935–3939.
[7] P. Lison, J. Tiedemann, and M. Kouylekov, “Opensubtitles 2018: Statistical rescoring of sentence alignments
in large, noisy parallel corpora,” in Proceedings of the
11th edition of the Language Resources and Evaluation
Conference (LREC 2018), Miyazaki, Japan.(accepted),
2018.
[8] Y. Kim, “Convolutional neural networks for sentenceclassification,” arXiv preprint arXiv:1408.5882, 2014.
[9] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks.” in
CVPR, vol. 1, no. 2, 2017, p. 3.
[10] K. Veselỳ, A. Ghoshal, L. Burget, and D. Povey,“Sequence-discriminative training of deep neural networks.” in Interspeech, 2013, pp. 2345–2349.
[11] A. Graves, N. Jaitly, and A.-r. Mohamed, “Hybrid speech recognition with deep bidirectional lstm,”in Automatic Speech Recognition and Understanding(ASRU), 2013 IEEE Workshop on. IEEE, 2013, pp.
273–278.
[12] T. N. Sainath, O. Vinyals, A. Senior, and H. Sak, “Convolutional, long short-term memory, fully connected
deep neural networks,” in Acoustics, Speech and Signal
Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 4580–4584.
[13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin,“Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
[14] E. Matusov, G. Leusch, O. Bender, and H. Ney, “Evaluating machine translation output with automatic sentence segmentation,” in International Workshop on
Spoken Language Translation (IWSLT) 2005, 2005.

75
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The ADAPT System Description for the IWSLT 2018 Basque to English
Translation Task
Alberto Poncelas, Andy Way
ADAPT Centre, School of Computing,Dublin City University, Dublin, Ireland{firstname.lastname}@adaptcentre.ie
Kepa Sarasola
Ixa Group (UPV/EHU), Faculty of Informatics
University of the Basque Country
kepa.sarasola@ehu.eus
Abstract
In this paper we present the ADAPT system built for theBasque to English Low Resource MT Evaluation Campaign.
Basque is a low-resourced, morphologically-rich language.
This poses a challenge for Neural Machine Translation models which usually achieve better performance when trainedwith large sets of data.
Accordingly, we used synthetic data to improve the translation quality produced by a model built using only authentic
data. Our proposal uses back-translated data to: (a) create
new sentences, so the system can be trained with more data;and (b) translate sentences that are close to the test set, so themodel can be fine-tuned to the document to be translated.

1. Introduction
We participated in the Basque to English Low Resource MT
Evaluation Campaign as part of the International Workshop
on Spoken Language Translation (IWSLT) 2018. In this task,
we aimed to build an MT model to translate subtitles of TED(Technology, Entertainment, Design) talks from Basque into
English.
Basque (or Euskera), which is mainly spoken in theBasque Country in Northern Spain and Southern France,
is considered an isolated language. Linguistically, it is anagglutinative language, and morphologically more complex
than English. Furthermore, Basque is a low resource language. Due to these characteristics, creating an MT systemthat deals with Basque is a challenging task.
As the MT Evaluation Campaign consists of translating
subtitles from TED talks, we built our MT engines mainly
from available subtitles. TED Talks1 is an event where experts in different fields, such as education, business, science,etc. give a talk of up to 18 minutes to disseminate their ideas.
The use of subtitles as training data is potentially problematic as they may not be literal translation, causing the1 https://www.ted.com

original and translated sentences not to be truly parallel. Thisis because subtitles are subjected to a great deal of adaptation. Localization strategies (adapting the text to suit consumers of a particular locale or culture), combined with therequirement to meet time constraints (where sentences inthe source and target languages which have different length
are supposed to appear on the screen within the same timeframe), results in sentences which are comparable but notnecessarily parallel [1].
Although the adaptation does not hinder human comprehension of the intended message, when these sentences areused as training data for an MT model, the translation inaccuracies become obstacles for the system to correctly learn
to translate.
The system presented in this paper aims to overcome theaforementioned problems. First, the creation of synthetic
data has two purposes: (i) it provides a new set of parallel
sentences that mitigates the problem of Basque being a low
resourced language; and (ii), artificially-created sentences
tend to be more literal than usual translated subtitles. Therefore the former may constitute better training data for an MTmodel than the latter. Secondly, as TED Talks topics cover awide variety of domains, we use data selection techniques toadapt an MT model to a particular test set.
The remainder of the paper is structured as follows. In
Section 2, we describe related work regarding MT models
that include Basque as source or target language. We also
describe previous work on the use of synthetic data and data
selection algorithms that are related to the systems described
below. Section 3 describes the two steps (hybrid corpus creation and model adaptation) performed for building the MT
system. In Section 4 we present an estimation of the performance of the models created. Finally, an overview of thesystem is described in Section 5.

76
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

2. Related Work
The system described in this paper is based on two main techniques: (a) incorporating synthetic sentences as training data
(Section 2.2), and (b) adapting the model to the test set (Section 2.3).
2.1. Basque Machine Translation
Most of the work on MT involving Basque is based on theBasque-Spanish pair. We can find multiple MT approaches
including Rule Based MT (RBMT) [2], or data-driven approaches [3] such as Example-based MT [4] or hybrid (Statistical MT and RBMT) [5] systems.
Dealing with low-resource languages is a problem for
NMT approaches as they require large amounts of data in order to generate good translations. For some language pairs,
SMT models can outperform NMT models when trained inlimited amount of data [6]. In the work of Unanue et al.
(2018) [7] they perform a comparison of Basque-English
SMT and NMT models. Their finding reveals that SMT
models trained with PaCo2-EuEn corpus in the Basque-toEnglish direction perform better than NMT models. In thereverse direction, however, NMT models can perform better
when pre-trained embeddings (which have been trained using additional sentences from Basque Wikipedia) are given
to the model.
Regarding Basque-Spanish NMT models, the most recent work is presented by Etchegoyen et al. (2018) [8] where
they explored different methods of splitting words into morphemes to improve the translation.
2.2. Addition of Back-translated Sentences
As Basque is a low-resource language, the amount of available parallel data is very limited. A technique to increase thenumber of sentences is to artificially create sentences. Sennrich et al. (2016) [9], showed that NMT models could beboosted by adding backtranslated data.
Backtranslation in this paper designates the process oftranslating monolingual sentences in the target language into
the source-side language. By doing this, a synthetic parallel
corpus is created. Adding this corpus as training data can
improve the performance of the model. In fact, models built
using solely back-translated data can even achieve comparable performance to those trained with authentic or hybrid
data [10].
2.3. Adaptation of the MT Model to the Test Set
There are several techniques for adapting a model to a particular domain [11], such as selecting relevant data (datacentric approaches), or modifying the model (model-centric
approaches).
In the case where the test set is available, it is possible
to adapt the model so it performs better in the given test. In
our work, we used a combination of data-centric and model-

centric approaches. First, we selected data that are relevant
for the test set, and then we used fine-tuning to bias the model
towards the test set.
Fine-tuning [12; 13], consists of using a pre-built NMT
model (trained on general domain data), and training the lastepochs on smaller amounts of in-domain data. An alternative
to this technique is gradual fine tuning [14], which involves
reducing the training data as the training proceeds.
While these fine-tuning techniques aim to adapt the NMT
models towards a particular domain, Li et al. (2018) [15]proposed to use fine-tuning to adapt the model to the test set,
which is closer to our approach. The main difference is thatwhile in their work the model is adapted sentence-wise (onemodel for each sentence), in ours, it is adapted documentwise (one model for the document).
In order to select sentences that are closer to the test set
we used Feature Decay Algorithms (FDA) [16; 17; 18]. This
technique has been successfully applied in both SMT [19;20; 21] and NMT [22].
FDA is a data selection method that not only aims to select sentences that are close to a seed (generally the test set),
but also to promote the variability of the training data selected.
In order to achieve that, FDA scores each sentence s in
the parallel data, and the sentence with the highest score isadded to a list of selected sentences L. The score of the sentence is based on how similar it is to the seed (counting then-grams shared with the seed), and how different it is to previously selected sentences (penalizing n-grams already contained in L), which increases the variability.
Using default values of the parameters, the score of a sentence is computed as in Equation (1):P
CL (ngr)ngr∈s 0.5
(1)score(s|L) =length(s)where CL (ngr) is the count of the n-gram ngr in the poolof selected sentences L. The more occurrences of ngr there
are in L the more penalized ngr is. The factor 0.5CL (ngr) in
Equation (1) causes the n-gram to contribute less to the total
score of the sentence.

3. System Description
The system built consists of two steps. First, (Section 3.2)
we created a basic model using authentic and synthetic data.
In the second step (Section 3.3), the model was fine-tuned tobe adapted to the test set.
3.1. Basque-English Data
The Basque-English parallel data used in this work wereobtained by combining the OpenSubtitles2016 (173K sentences), OpenSubtitles2018 [23] (805K sentences) and the
PaCo2-EuEn corpus2 [24] (130K sentences) provided in the
2 komunitatea.elhuyar.org/ig/files/2016/01/PaCo_

EuEn_corpus.tgz

77
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

IT domain MT Shared Task of WMT 2016 [25]. We randomly sampled 5000 sentences as our dev set and the rest
(1M sentences) as the training set.
In order to build the NMT models we used OpenNMTpy, which is the Pytorch port of OpenNMT [26]. All the
NMT models we built were configured with the same settings(the only difference is the training data used to build them).
The value parameters were the default ones of OpenNMTpy (i.e. 2-layer LSTM with 500 hidden units, vocabulary
size of 50000 words for each language). All the models were
executed for 13 epochs, and we also used Byte Pair Encoding(BPE) [27] with 30000 merge operations, following the workof Etchegoyen et al. (2018) [8].

model can hurt performance.
A similar effect is observed in the second row of Table 1
for the sentence “keep her steady, now.”. The Basque translation of this sentence is “ez dadila mugitu.” which uses theverb “mugitu” (“to move”), so it could be translated as “itshall not move” or “do not let it move”. In contrast, the MTmodel produced the sentence, “eutsi gogor.”, which used theverb “hold” (“eutsi”). Both translations are appropriate, butthey belong to different contexts.
Finally, we see in the third row the English sentence “asuicide?”. The corresponding sentence in Basque is “norzen?” (“who was?”). In any other context, the two sentences
have completely different meanings. The synthetic sentenceby contrast is a literal translation.

3.2. Addition of Synthetic Data
1
2
3

Authentic
Basque
joan
behar
dut?
ez
dadilamugitu.
nor zen?

Synthetic
Basquehor egon behar
dut?
eutsi gogor.
suizidioa?

Englishdo I need to be
there?
keep
hersteady, now.
a suicide?

Table 1: Examples of sentences in Basque (authentic),
Basque (synthetic) and English translation.

Figure 1: Creation of hybrid parallel corpus.
The first step in the construction of a baseline system is
to extend the parallel corpus. In Figure 1 we present a diagram of how we built the corpus. Using an initial corpus ofparallel Basque-English sentences we built an NMT model
capable of translating sentences from English into Basque.
Then, the English side of that parallel corpus was translated
into Basque using the English to Basque NMT model.
Intuitively, translating the same sentences that were fed
as training data should not be useful as it is likely to produce
very similar sentences. However, the sentences produced by
the model tend to be more literal translations, thereby avoiding the problems previously mentioned.
In Table 1 we show some examples of how synthetic data
present Basque sentences that are closer to literal translation
than a human-produced sentences. For example, in the firstrow, the translation for the English sentence “do I need to be
there?” is “joan behar dut?”, which literally means “do I haveto go?”. The artificially-created sentence is a more precisetranslation, as it uses the verb “be” (“egon”) instead of theverb “go” (“joan”). In certain contexts, the use of one or
another sentence does not affect the general understanding.
However, using the wrong translation as training data for a

Following backtranslation we obtained two parallel sets,with authentic and synthetic sentences. Next, we concatenated them as a single corpus. Note that, by doing so, thetarget-language sentences are duplicated.
Finally, we removed those sentences in which the lengthof the source and target sides differed substantially. In ourwork we kept a sentence pair (s, t) if 0.5 < len(s)len(t) > 1.5,
in order to remove the 10% outliers. In total 255K sentenceswere removed (137K sentences 118K sentences from authentic and synthetic sets, respectively). The hybrid corpus contained, therefore, 1.93M sentence pairs.
We applied these criteria to both corpus of authentic, andsynthetic sentences, so the potentially unaligned sentences
are ignored and bad translated sentences are not considered,respectively.
3.3. Adaptation to the Test Set
The second step of building the model is to adapt it to a particular test set. The work of Poncelas et al. (2018) [22]showed that when the test set is available during training time
it is possible to fine-tune a model to improve the translationof that particular test set.
In Figure 2 we show how we fine-tuned our NMT model,
which requires three phases as follows:
1. Data Selection: In this phase we aimed to retrieve English sentences that were close to the test set. As thetest set was in Basque, we first created an approximated translation using the NMT model built as ex-

78
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

base model
BLEU
NIST
TER
METEOR
CHRF3
CHRF1

0.13154.459
0.85080.142934.05
37.40

hybridmodel
0.1426*4.683
0.85760.1501*
35.92
38.67

FDA model
0.1450*4.733
0.86660.1528**
36.24
38.81

Table 2: Evaluation of the model built only with authentic
data and using both authentic and synthetic data.

Figure 2: Fine tuning with synthetic data
plained in Section 3.2. This translation can be used as
the seed for FDA and extract a set of sentences, from amonolingual English corpus, that were close to the pretranslated set and hence, to the test set. In this work we
extracted 50,000 sentences from English training data
provided in the WMT 2015 Translation Task [28].
2. Back-translation: The subset of selected English sentences were back-translated (we reused the same English-to-Basque model built as explained in Section
2.2 to create backtranslated data) in order to build aparallel corpus.
3. Fine-tuning: The synthetic parallel corpus was used tofine-tune the MT model for one epoch. In this way, themodel was tailored to the test set.

4. Experimental Results
In order to estimate the performance of the final and intermediate models described through Section 3 we evaluatedthem using the development set (containing 1K sentences extracted from subtitles of TED talks) provided by the organizers of the IWSLT Evaluation Campaign.
The models evaluated are: (a) the model built with only
authentic data (base model); (b) the model built with the
combination of authentic and synthetic data (hybrid model);and (c), the hybrid model adapted to the test set using FDAretrieved data (FDA model).
We used several evaluation metrics to compare the outputs of the three models to a human-translated reference. In
Table 2 we can see the evaluation scores for each model. The

metrics we present are BLEU [29], NIST [30], TER [31],METEOR [32] and CHRF3 [33].
We also marked in bold the scores that outperform thoseof the base model (first column of Table 2) and marked withan asterisk the scores (among BLEU, TER and METEOR)
that are statistically significant at level p=0.01. This wascomputed with multeval [34] using Bootstrap Resampling[35]. The two asterisks in column FDA model (METEORrow) indicate that it is statistically significant at p=0.01 whencompared not only to the base model but also to the hybrid
model.
As mentioned in Section 3.2, the addition of synthetic
data (even if it consists of a backtranslation of the same data
used for training the model) is helpful. This is verified withthe results in column hybrid model in Table 2. As we can see,most of the hybrid model scores of the model are better thanthe model built with authentic data only (base model column)
and according to two of the scores, the improvements are statistically significant at p=0.01. In fact, a model built using
only synthetic data (Table 3) can achieve improvements overthe base model, according to METEOR and CHRF3 evaluation metric.
synth.
model
BLEU
0.1224
NIST
4.074
TER
0.9769
METEOR 0.1481
CHRF3
36.22
CHRF1
36.40
Table 3: Evaluation of the model built with synthetic data
only.

Finally, fine-tuning the hybrid model using sentences thatare close to the test set is also beneficial. As we can see inthe column FDA model (in Table 2), most of the scores (except TER) are better than those of the base model or eventhe hybrid model, and according to METEOR metric the improvement is statistically significant at p=0.01.

79
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

5. Conclusion
In this paper we have described the ADAPT system presented
for the Low Resource MT Evaluation Campaign of IWSLT
2018. The system translates from Basque into English.
Basque is a morphologically rich language, which causes
the task of building an MT model to be more difficult thanlanguages such as Spanish or German. Furthermore, theavailable parallel Basque-English data are scarce.
Due to the limited resources of texts in Basque, we generated synthetic data that successfully boosted the performanceof the MT model trained solely with authentic sentences.
Additionally, we have used a supplementary monolingualEnglish corpus so we could retrieve sentences close to thetest set and further improve our model.

6. Acknowledgments
The research leading to these results was carried out as partof the TADEEP project (Spanish Ministry of Economy andCompetitiveness TIN2015-70214-P, with FEDER funding).
This work has been supported by the ADAPT Centre for
Digital Content Technology which is funded under the SFI
Research Centres Programme (Grant 13/RC/2106) and is cofunded under the European Regional Development Fund.

7. References[1] M. Fishel, Y. Georgakopoulou, S. Penkale,
V. Petukhova, M. Rojc, M. Volk, and A. Way,
“From subtitles to parallel corpora,” in Proceedings ofthe 16th Conference of the European Association for
Machine Translation (EAMT), Trento, Italy, 2012, pp.
3–6.
[2] A. Mayor, I. Alegria, A. D. De Ilarraza, G. Labaka,
M. Lersundi, and K. Sarasola, “Matxin, an open-source
rule-based machine translation system for Basque,”
Machine translation, vol. 25, no. 1, p. 53, 2011.
[3] G. Labaka, N. Stroppa, A. Way, and K. Sarasola,“Comparing rule-based and data-driven approaches tospanish-to-basque machine translation,” in MachineTranslation Summit XI, Copenhagen, Denmark, 2007,
pp. 297–304.
[4] N. Stroppa, D. Groves, A. Way, and K. Sarasola,“Example-based machine translation of the basque language,” pp. 232–241, 2006.
[5] G. Labaka, C. España-Bonet, L. Màrquez, and K. Sarasola, “A hybrid machine translation architecture guided
by syntax,” Machine translation, vol. 28, no. 2, pp. 91–
125, 2014.
[6] M. Dowling, T. Lynn, A. Poncelas, and A. Way, “SMT
versus NMT: Preliminary comparisons for Irish,” inTechnologies for MT of Low Resource Languages(LoResMT 2018), Boston, USA, 2018, pp. 12–20.

[7] I. J. Unanue, L. G. Arratibel, E. Z. Borzeshi, and
M. Piccardi, “English-Basque statistical and neural machine translation.” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018, pp. 880–
885.
[8] T. Etchegoyhen, E. M. Garcia, A. Azpeitia, G. Labaka,I. Alegria, I. C. Etxabe, A. J. Carrera, I. E. Santos, andM. M. eta Eusebi Calonge, “Neural machine translationof Basque,” in Proceedings of the 21st Annual Conference of the European Association for Machine Translation (EAMT), Alacant, Spain, 2018, pp. 139–148.
[9] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” in Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), Berlin, Germany, 2016, pp. 86–96.
[10] A. Poncelas, D. Shterionov, A. Way, G. M.
de Buy Wenniger, and P. Passban, “Investigating backtranslation in neural machine translation,” in 21st Annual Conference of the European Association for Machine Translation, Alacant, Spain, 2018, pp. 249–258.
[11] C. Chu and R. Wang, “A survey of domain adaptation for neural machine translation,” arXiv preprint
arXiv:1806.00258, 2018.
[12] M.-T. Luong and C. D. Manning, “Stanford neuralmachine translation systems for spoken language domains,” in Proceedings of the International Workshop
on Spoken Language Translation, Da Nang, Vietnam,
2015, pp. 76–79.
[13] M. Freitag and Y. Al-Onaizan, “Fast domain adaptation for neural machine translation,” arXiv preprint
arXiv:1612.06897, 2016.
[14] M. van der Wees, A. Bisazza, and C. Monz, “Dynamic
data selection for neural machine translation,” in Proceedings of the 2017 Conference on Empirical Methods
in Natural Language Processing, Copenhagen, Denmark, 2017, pp. 1400–1410.
[15] X. Li, J. Zhang, and C. Zong, “One Sentence One
Model for Neural Machine Translation,” in Proceedings of the Eleventh International Conference on
Language Resources and Evaluation (LREC 2018),
Miyazaki, Japan, 2018, pp. 910–917.
[16] E. Biçici and D. Yuret, “Instance selection for machine translation using feature decay algorithms,” inProceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland, 2011, pp. 272–
283.

80
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[17] E. Biçici, Q. Liu, and A. Way, “ParFDA for fast deployment of accurate statistical machine translation systems, benchmarks, and statistics,” in Proceedings of
the Tenth Workshop on Statistical Machine Translation,
Lisbon, Portugal, 2015, pp. 74–78.
[18] E. Biçici and D. Yuret, “Optimizing instance selection
for statistical machine translation with feature decay algorithms,” IEEE/ACM Transactions on Audio, Speech,
and Language Processing, vol. 23, no. 2, pp. 339–350,
2015.
[19] E. Biçici, “Feature decay algorithms for fast deployment of accurate statistical machine translation systems,” in Proceedings of the Eighth Workshop on Statistical Machine Translation, Sofia, Bulgaria, August
2013, pp. 78–84.
[20] A. Poncelas, A. Way, and A. Toral, “Extending feature
decay algorithms using alignment entropy,” in International Workshop on Future and Emerging Trends in
Language Technology, Seville, Spain, 2016, pp. 170–
182.
[21] A. Poncelas, G. M. de Buy Wenniger, and A. Way, “Applying n-gram alignment entropy to improve feature decay algorithms,” The Prague Bulletin of Mathematical
Linguistics, vol. 108, no. 1, pp. 245–256, 2017.
[22] A. Poncelas, G. M. Buy Wenniger, and A. Way, “Feature decay algorithms for neural machine translation,”in Proceedings of the 21st Annual Conference of the European Association for Machine Translation, Alacant,Spain, 2018, pp. 239–248.
[23] J. Tiedemann, “News from OPUS - A collection ofmultilingual parallel corpora with tools and interfaces,”in Recent Advances in Natural Language Processing,N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov,
Eds. Borovets, Bulgaria: John Benjamins, Amsterdam/Philadelphia, 2009, vol. V, pp. 237–248.
[24] I. San Vicente, I. Manterola, et al., “PaCo2: A fully
automated tool for gathering parallel corpora from theweb.” in Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC
2012), Istanbul, Turkey, 2012, pp. 1–6.
[25] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham,
B. Haddow, M. Huck, A. J. Yepes, P. Koehn, V. Logacheva, C. Monz, et al., “Findings of the 2016 conference on machine translation.” in ACL 2016 First Conference on Machine Translation (WMT16), Berlin, Germany, 2016, pp. 131–198.
[26] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush,“Opennmt: Open-source toolkit for neural machinetranslation,” in Proceedings of the 55th Annual Meeting of the Association for Computational LinguisticsSystem Demonstrations, Vancouver, Canada, 2017, pp.
67–72.
[27] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers), vol. 1, Berlin, Germany, 2016, pp. 1715–1725.
[28] O. Bojar, R. Chatterjee, C. Federmann, B. Haddow, M. Huck, C. Hokamp, P. Koehn, V. Logacheva,
C. Monz, M. Negri, M. Post, C. Scarton, L. Specia, and
M. Turchi, “Findings of the 2015 Workshop on Statistical Machine Translation,” in Proceedings of the TenthWorkshop on Statistical Machine Translation, Lisboa,
Portugal, September 2015, pp. 1–46.
[29] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu:a method for automatic evaluation of machine translation,” in Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia,
Pennsylvania, USA, July 2002, pp. 311–318.
[30] G. Doddington, “Automatic evaluation of machinetranslation quality using n-gram co-occurrence statistics,” in Proceedings of the second international conference on Human Language Technology Research, San
Diego, CA, 2002, pp. 138–145.
[31] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, andJ. Makhoul, “A study of translation edit rate with targeted human annotation,” in Proceedings of the 7th
Conference of the Association for Machine Translation in the Americas, Cambridge, Massachusetts, USA,2006, pp. 223–231.
[32] S. Banerjee and A. Lavie, “Meteor: An automatic metric for MT evaluation with improved correlation withhuman judgments,” in Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures formachine translation and/or summarization, Ann Arbor,
Michigan, 2005, pp. 65–72.
[33] M. Popovic, “chrF: character n-gram F-score for automatic MT evaluation,” in Proceedings of the TenthWorkshop on Statistical Machine Translation, Lisbon,
Portugal, 2015, pp. 392–395.
[34] J. H. Clark, C. Dyer, A. Lavie, and N. A. Smith, “Better hypothesis testing for statistical machine translation:
Controlling for optimizer instability,” in Proceedings ofthe 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies(Volume 2: Short Papers), Portland, Oregon, 2011, p.
176–181.

81Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[35] P. Koehn, “Statistical significance tests for machinetranslation evaluation,” in Proceedings of the 2004
Conference on Empirical Methods in Natural Language
Processing, Barcelona, Spain, 2004, pp. 388–395.

82
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The University of Helsinki submissions to the
IWSLT 2018 low-resource translation task
Yves Scherrer
Department of Digital Humanities
University of Helsinki, Finland
yves.scherrer@helsinki.fi

Abstract
This paper presents the University of Helsinki submissions
to the Basque–English low-resource translation task. Our
primary system is a standard bilingual Transformer system,
trained on the available parallel data and various types of synthetic data. We describe the creation of the synthetic datasets,some of which use a pivoting approach, in detail. One of
our contrastive submissions is a multilingual model trainedon comparable data, but without the synthesized parts. Ourbilingual model with synthetic data performed best, obtaining 25.25 BLEU on the test data.

1. Introduction
The University of Helsinki has participated in the IWSLT
low-resource translation task on Basque-to-English translation with one primary and two contrastive systems. Our experiments mainly focused on creating synthetic training datafor classical supervised neural machine translation models.
In particular, we show that a bilingual system trained on
partly synthetic data performs better than a multilingual system that includes the same data in their original, non synthetic form. Our best submitted system obtained a BLEU
score of 25.25.
Section 2 describes the available Basque–English parallel
datasets at the basis of our systems, as well as a baseline system trained on these parallel datasets alone. In Section 3, we
present additional datasets that contain either Basque or English text, but not both. We discuss several strategies for synthetically creating parallel Basque–English datasets out ofthese sources, and show the impact of these synthetic datasetson translation quality. In Section 4, we present a contrastivesystem that uses the additional datasets in their original state,
without the synthesized parts. Section 5 summarizes our submissions and details the post-processing steps carried out at
prediction time.

2. Parallel Basque–English data
The IWSLT organizers released an in-domain data set forBasque-to-English translation containing 64 TED talks fortraining and 10 TED talks for development [1]. Another 10
TED talks have been held out for testing.

The only allowed out-of-domain data source containing
parallel Basque–English datasets is OPUS [2, 3]: it contains
computer program localization files (repositories GNOME,KDE4 and Ubuntu), crowd-sourced translations (Tatoeba)
and film subtitles (OpenSubtitles2018). We only selected
OpenSubtitles2018 as the largest and most domain-similar
dataset for our experiments. Table 1 summarizes the available parallel data.1
Source

Talks

Lines

EU tokens

EN tokens

TED train
OST
TED dev

64
—
10

5623
806k
1140

97k
4.8M
20k

128k
6.5M
27k

Table 1: Basque–English parallel data.
2.1. Baseline system
We trained a baseline system using only the parallel data
mentioned in the previous section. Data were tokenized andtruecased using the Moses scripts [4]; no effort was spenton adapting the tokenization tools to Basque. Following thegood results on various typologically diverse language pairs,
we used the Transformer model setup [5] as implemented
in Marian-NMT [6] (see Appendix). We used an initial setting of 20 000 BPE units [7] shared across both languages
with tied embeddings. Training of this model converged after about 20 hours on a single-GPU node, obtaining a BLEU
score of 15.40 on the development set (see first line of Table 5).2

3. Synthetic data
Backtranslation has proven to be an effective way of improving the performance of neural machine translation systems
by taking advantage of existing monolingual datasets for thetarget language [8]. Monolingual data of the target language
is translated to the source language using a target-to-source
1 In all tables, validation and test sets are displayed in italics, whereas thetranslation output of the described system is displayed in bold (if applicable).
2 All BLEU scores were computed using the multi-bleu-detok.perl scriptof the Moses distribution.

83
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018translation system. The resulting bilingual dataset, whose
source is noisy, is then used as additional training data forthe source-to-target translation system.
In our setting, direct backtranslation would amount totranslating English data to Basque, but such an English-toBasque system would have to be trained on the same small
dataset as the baseline system presented above. Therefore,
we experimented with other ways of creating synthetic data,
exploiting the larger Spanish–Basque and Spanish–English
datasets and using Spanish as a pivot language [9].3 The
different data augmentation strategies are discussed in Sections 3.1 to 3.3, whereas the Basque-to-English systems
trained on these synthetic datasets are presented in Section 3.4 and Table 5.
3.1. Direct backtranslation of TED talks
The provided in-domain data contains a total 2683 English
TED talks. Excluding those that already have Basque translations (for training, development or testing) and excluding
those that do not have a Spanish translation (to provide comparability with the experiment below), 2576 English TED
talks can be backtranslated to Basque.
Source
TED train
OST
TED dev
TED direct-BT

Talks Lines EN tokens EU tokens
64
—
10

5623
806k
1140

128k
6.5M
27k

97k
4.8M
20k

2576

271k

6.2M

3.9M

3.2. Pivot-based backtranslation of TED talks
We hypothesize that the direct backtranslation approach
would not be particularly effective, as the system used togenerate them would suffer from the same data sparsity issues as the baseline system (trained with the same data,
but in the other direction). In order to take advantage ofthe other datasets provided by the organizers, we follow apivot-based approach along the lines of [9]: we take all
TED talks available in both English and Spanish (but not
Basque), translate the Spanish version to Basque, and alignthe Basque side with the English side to constitute additional
Basque–English data. In this setting, the backtranslation
model needs to be trained on Spanish-to-Basque data; using the same 64+10 TED talks for training and validation, as
well as the out-of-domain Open Data Euskadi (ODE) dataset
and the Basque–Spanish OpenSubtitles (OST), we create aTransformer model with the same parameters as the baselinemodel. At the end of training, this system obtained a BLEU
score of 14.52 on the Spanish-to-Basque development set.
The resulting data consists thus of the same English target sentences as above but different Basque source sentences.
Details on the setup are given in Table 3. It is striking thatthe Basque sentences translated via Spanish are considerably
longer than those translated directly from English (4501k
total tokens in Table 3 vs. 3886 total tokens in Table 2).
The experiments described below will show which of thetwo datasets improves translation most, and whether the twodatasets are complementary or not.
SourceTable 2: Basque–English data used to train the backtranslation model (above the line) and monolingual English data
backtranslated with this model (below the line, backtranslation output in bold).
In this first experiment, we train an English-to-Basquesystem analogously to the baseline system above, using thesame training data, parameter settings (20k joint BPE units)
and development set for validation, obtaining a BLEU scoreof 8.65 on the English-to-Basque development set.4 This lowscore confirmed our initial reservations about direct backtranslation. We nevertheless translate the monolingual English TED talks to Basque with this system. Table 2 summarizes the data of this experiment.
3 Note that we employ the term pivot language in the context of a data

augmentation strategy, not of a machine translation model per se. We take aparallel corpus of languages hX, Y i and translate its X side to language Z
using a X → Z machine translation system, yielding a corpus of languages
hZ, Y i. This approach is simpler than the common acceptation of pivotbased translation, where two (more or less independent) translation models
are trained, and the output of the first serves as the input of the second one.
Examples of recent work in this area include [10, 11].
4 The BLEU score of a Basque-to-English system including these backtranslations is 21.04, as shown in the second row of Table 5.

TED trainOSTODETED dev
TED pivot-BT

Talks Lines ES tokens EU tokens
64
—
—
10

5546
794k
927k
1122

124k
5.8M
23.1M
26k

98k
4.8M
17.5M
20k

2576

271k

EN 6.2M

4.5M

Table 3: Basque–Spanish data used to train the backtranslation model (above the line) and monolingual Spanish data
backtranslated to Basque and aligned with English (below
the line).
3.3. Pivot-based translation of Open Data Euskadi
Whereas backtranslation yields datasets with noisy sourcesides and clean target sides, we also wanted to explore theimpact of a corpus with clean source side and noisy target
side. This approach is not generally used in standard highresource settings, but could yield additional improvements
in low-resource settings. The Open Data Euskadi corpus isa good candidate for this approach. It is rather large andcontains Basque–Spanish parallel data. In order to create aBasque–English version of this corpus, we proceed by translating the Spanish version to English and aligning it with theexisting Basque one.

84
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The Spanish–English system is trained using most of theparallel data that was made available in WMT 2013, the lastyear in which Spanish–English featured as a WMT news
translation language pair (see Table 4) [12]. In particular,
we use the CommonCrawl, Europarl V7, NewsCommentary
V12 and UN datasets for training,5 the NewsTest 2008-2012
corpora for validation and NewsTest 2013 for testing. We
did not use OpenSubtitles as we did not find it helpful fortranslating the legal and news domain documents present inOpen Data Euskadi. Due to the larger training corpora sizes,
we increased the vocabulary to 40k joint BPE units, but kept
the same Transformer architecture and parameters otherwise.
This system obtained a BLEU score of 29.69 on the development set and 31.45 on the test set, slightly surpassing thebest systems participating in WMT 2013.6 The figures of
the resulting Basque–English Open Data Euskadi corpus areshown on the last line of Table 4.
SourceCommonCrawl
Europarl
NewsCommentary
UN
News dev
News test
ODE pivot-T

Lines

ES tokens

EN tokens

1845k
1965k
292k
11196k
13k
3k

49.5M
57.0M
8.5M
366.1M
357k
70k

46.9M
54.5M
7.5M
320.0M
336k
64k

927k

EU 17.3M

21.5M

Table 4: Spanish–English data used to train the translation
model (above the line) and monolingual Spanish data translated to English and aligned with Basque (below the line).

3.4. Bilingual systems using synthetic data
We trained various Basque-to-English systems with different
combinations of the synthetic datasets described above. All
experiments use the same Transformer model architecture,but slightly different vocabulary sizes (see below).
For some experiments, we introduce variants with domain labels [14, 15]. Tars et al. have found domain labelinguseful to teach the model about possible domain mismatches
in the training data. In our experiments, we use four labels,
distinguishing text sources (TED, OST, OPD) and methodsof corpus construction (TED-parallel and TED-BT). The validation and test instances are labeled as TED-parallel. Domain labels were included as the first tokens of each sentence. Table 5 summarizes these experiments.
Table 5 shows that any additional synthetic dataset helps
in the given low-resource setting. The direct TED backtranslations are surprisingly helpful despite their low qual5 We experimented with a reduced training set consisting of Europarl and
NewsCommentary only, but results were not quite as good as with the complete training data.
6 The best WMT 2013 submissions were the phrase-based statistical systems by the University of Edinburgh team, with BLEU scores of 31.37 in
the unconstrained setting and 30.59 in the constrained setting [13].

Training data

Domain labels

BPE BLEU

Parallel (= TED train + OST)

No

20k

15.40

+ TED direct-BT

No

20k

21.04

+ TED pivot-BT

No

20k

23.20

+ TED direct-BT + ODE pivot-T

No
Yes

30k
30k

23.20
23.84

+ TED pivot-BT + ODE pivot-T

No
Yes

30k
30k

24.22
24.52

+ TED direct-BT + TED pivot-BT+ ODE pivot-T

No
Yes

30k
30k

24.39
25.06

Table 5: Experiments with different combinations of training
data.
ity, although the pivot-based TED backtranslations are much
more useful, presumably due to the higher quality of the system that generated them. The impact of the ODE synthetic
dataset is less remarkable, but still improves BLEU scores by
2-3 absolute points. Interestingly, the direct and pivot-based
TED backtranslations are somewhat complementary, yielding slight improvements compared to using just the pivotbased ones.
On the basis of the Parallel + TED pivot-BT model (thirdline of Table 5), we performed a grid search to find the best
subword encoding scheme. We used various sizes of joint
BPE vocabularies with tied embeddings (10k, 15k, 20k, 25k,
30k, 35,) and various sizes of language-specific BPE vocabularies in conjunction with distinct embeddings (10k, 15k,
20k, 25k, 30k, 35k per language). The difference between
the worst and best setting lay at 1.5 BLEU points. The best
results were achieved with joint vocabularies and tied embeddings and a total of 25k-30k subword units. The final
submissions were made with a joint vocabulary of 30k units,
like most experiments presented in Table 5.
Domain labels show consistent improvements of about
0.5 BLEU points. As mentioned above, we labeled the validation data with TED-parallel. Additional experiments using
other domain labels at test time have shown the following results: TED-BT +0.04 BLEU, OST -2.83 BLEU, OPD -4.70
BLEU, no label -1.71 BLEU. This experiment shows that theTED-parallel and TED-BT labels yield similar results (thedifference is probably not statistically significant), suggesting that the distinction between genuinely parallel and backtranslated TED data may not have been necessary. We nevertheless kept the TED-parallel label also for the test data.

4. Multilingual system
Johnson et al. [14] have shown that multilingual translation models can be trained by using training data of various
languages and directions and prepending a target languagelabel to each source sentence. One interesting use case ofsuch multilingual models is zero-shot translation, where the85
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

System

Model type and features

BLEU

NIST

TER

Primary
Contrastive 1
Contrastive 2

Bilingual model
Bilingual modelMultilingual model

25.01
25.25
22.55

6.45
6.47
6.10

59.48
58.83
60.48

With sentence splitting
No sentence splittingWith sentence splitting

Table 6: Submitted systems and official results on the test set.
source language and target language have both been seen by
the model, but not in that particular combination. In our case,
we are not interested in zero-shot translation, as we do havea sizeable set of Basque-to-English training data. Rather, wewanted to see to what extent multilingual modelling could
supplant the creation of synthetic data. To this end, we train
a single multilingual model with the following datasets: theparallel Basque–English TED and OpenSubtitles data (as in
the baseline model), the parallel English–Spanish TED datain both directions (as used to train the pivot-based backtranslation model), and the Basque–Spanish Open Data Euskadi
data (see Table 7). In this setting, we only have English andSpanish as target languages and consequently only use thetwo target language labels TO EN and TO ES. We do not
use additional domain labels in this experiment. The modelarchitecture remains the same, but we use a joint trilingual
vocabulary consisting of 40k BPE units.
SourceLines

Source tokens

Target tokens

TED train
OST
TED train
TED trainODE
TED dev

5623
805k
277k
277k
926k
1140

97k EU
4.8M EU
6.3M EN
6.0M ES
17.5M EU
20k EU

128k EN
6.5M EN
6.0M ES
6.3M EN
23.1M ES
27k EN

Table 7: Data used to train the multilingual model.
Although we used almost the same datasets as in thesystems presented above (with the exception of the WMTEnglish–Spanish data), the multilingual model failed toachieve competitive results, with 22.55 BLEU on the validation set. There are several reasons for this lower-thanexpected performance. First, the training of the multilingual
model was stopped before convergence, after a training timeof 72 hours. Nevertheless, the learning curve does not indicate the possibility of substantial improvements if training had continued. Second, the multilingual model has tolearn three languages on the source side and two languages
on the target side instead of a one-to-one mapping. Its task
is thus inherently more complex, and it seems that the threelanguages in question (Basque, English and Spanish) are typologically too diverse for the model to generalize. Finally,[14] show that good data sampling strategies are crucial whentraining multilingual models with unbalanced data sizes. In
this regard, oversampling the Basque-to-English resources orfine-tuning the model to the target language pair might havehelped. Despite its lower performance, we base one of ourcontrastive submissions on the multilingual model.

5. Submissions
We decided to submit output from two models, the bilingual
system trained with all synthetic data and domain labels (lastline of Table 5), and the multilingual system described in
Section 4.
We have found in a different context [16] that systems
trained on single sentences may not be able to translate utterances consisting of several sentences completely. Although
there was no particular evidence of such problems occurringin the experiments at hand (since a large portion of the TED
training data already contains multi-sentence utterances), we
carried out some experiments on this issue. Concretely, weapplied a simple sentence splitter to the source text, translated each sentence separately, and merged them back together. In the validation set, 214 (of 1140) lines were split,
and sentence splitting improved the BLEU score by 0.26
points. However, qualitative inspection of the results did notshow convincing evidence in favor or against sentence splitting. Therefore, we submitted systems with and without sentence splitting.
Also, due to an error in the postprocessing script, the submitted translations were accidentally detokenized with theBasque detokenizer (and some additional rules) rather thanthe English one. The added rules minimized the adverse effect of this error, such that it only affected two tokens in thetest set, resulting in an estimated impact on BLEU score ofabout 0.01.
Table 6 summarizes the submitted systems with the official results. Sentence splitting turned out to have a slightly
negative impact on the translation of the test set, whereas thedifference between the bilingual and multilingual system iscomparable to the one that was observed with the validation
set.

6. Conclusions
The University of Helsinki submissions on Basque–English
leverage the existing parallel corpora for other language pairs
to create synthetic data of various types. In particular, wehave found pivot-based (back-)translation to be a useful approach to increase the amounts of Basque–English training
data. In this setting, one side of a parallel corpus is translated
to a third language, and this translated output is then alignedwith the other side of the original parallel corpus. By using

86
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

various synthetic datasets, we were able to increase translation performance from 14.68 BLEU to 25.06 BLEU on thedevelopment set.
Our contrastive multilingual model performed less well,although it saw almost the same data as the bilingual modeland its auxiliary models used to create the synthetic data. It
remains to be seen if better balancing of the training data,
possibly including some fine-tuning, as well as the inclusion of domain labels and additional Spanish–English training data could make this model more competitive. Also,
both approaches could be combined by training a multilingual model with added synthetic data.

7. References[1] M. Cettolo, C. Girardi, and M. Federico, “Wit3 : Web
inventory of transcribed and translated talks,” in Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), Trento, Italy,
May 2012, pp. 261–268.
[2] J. Tiedemann, “Parallel data, tools and interfaces in
OPUS,” in Proceedings of the Eight International
Conference on Language Resources and Evaluation(LREC’12), N. Calzolari, K. Choukri, T. Declerck,
M. U. Doğan, B. Maegaard, J. Mariani, A. Moreno,
J. Odijk, and S. Piperidis, Eds. Istanbul, Turkey: European Language Resources Association (ELRA), May
2012.
[3] P. Lison and J. Tiedemann, “OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC
2016), N. Calzolari, K. Choukri, T. Declerck, S. Goggi,
M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo,
A. Moreno, J. Odijk, and S. Piperidis, Eds.
Paris,
France: European Language Resources Association(ELRA), May 2016.
[4] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin,
and E. Herbst, “Moses: Open source toolkit for statistical machine translation,” in ACL’07: Proceedings ofthe 45th Annual Meeting of the Association for Computational Linguistics, Demonstration Session. Association for Computational Linguistics, 2007, pp. 177–180.
[5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin,“Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
[6] M. Junczys-Dowmunt, R. Grundkiewicz, T. Dwojak,
H. Hoang, K. Heafield, T. Neckermann, F. Seide,

U. Germann, A. Fikri Aji, N. Bogoychev, A. F. T. Martins, and A. Birch, “Marian: Fast neural machine translation in C++,” in Proceedings of ACL 2018, System
Demonstrations. Melbourne, Australia: Association
for Computational Linguistics, July 2018, pp. 116–121.
[7] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers). Berlin, Germany: Association for Computational Linguistics, August 2016, pp. 1715–1725.
[8] ——, “Improving neural machine translation models
with monolingual data,” in Proceedings of the 54th
Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics, August 2016, pp. 86–96.
[9] J. Tiedemann, “Character-based pivot translation forunder-resourced languages and domains,” in Proceedings of the 13th Conference of the European Chapter ofthe Association for Computational Linguistics. Association for Computational Linguistics, 2012, pp. 141–
151.
[10] Y. Cheng, Y. Liu, Q. Yang, M. Sun, and W. Xu, “Neuralmachine translation with pivot languages,” CoRR, vol.
abs/1611.04928, 2016.
[11] S. Ren, W. Chen, S. Liu, M. Li, M. Zhou, and S. Ma,“Triangular architecture for rare language translation,”in Proceedings of ACL 2018, Melbourne, Australia,
2018.
[12] O. Bojar, C. Buck, C. Callison-Burch, C. Federmann,
B. Haddow, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia, “Findings of the 2013 Workshop
on Statistical Machine Translation,” in Proceedings ofthe Eighth Workshop on Statistical Machine Translation. Sofia, Bulgaria: Association for Computational
Linguistics, August 2013, pp. 1–44.
[13] N. Durrani, B. Haddow, K. Heafield, and P. Koehn,“Edinburgh’s machine translation systems for European language pairs,” in Proceedings of the Eighth
Workshop on Statistical Machine Translation. Sofia,
Bulgaria: Association for Computational Linguistics,
August 2013, pp. 114–121.
[14] M. Johnson, M. Schuster, Q. Le, M. Krikun, Y. Wu,
Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean, “Google’s multilingual
neural machine translation system: Enabling zero-shottranslation,” Transactions of the Association for Computational Linguistics, vol. 5, pp. 339–351, 2017.

87
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[15] S. Tars and M. Fishel, “Multi-domain neural machinetranslation,” in Proceedings of the 21st Annual Conference of the European Association for Machine Translation (EAMT’2018), J. A. Pérez-Ortiz, F. SánchezMartı́nez, M. Esplà-Gomis, M. Popović, C. Rico,
A. Martins, J. V. den Bogaert, and M. L. Forcada, Eds.,
2018.
[16] A. Raganato, Y. Scherrer, T. Nieminen, A. Hurskainen,
and J. Tiedemann, “The University of Helsinki submissions to the WMT18 news task,” in Proceedings of theThird Conference on Machine Translation (WMT18).
Association for Computational Linguistics, 2018.

8. Appendix
All models presented in this paper were trained using the parameter settings described in https://github.com/marian-nmt/marian-examples/tree/master/transformer,
which correspondroughly to the base setup of [5].
The relevant parameters are as follows:marian --type transformer--max-length 200 --mini-batch-fit-w 10000 --maxi-batch 1000--early-stopping 10 --valid-freq 5000--valid-metrics cross-entropy
perplexity translation--valid-mini-batch 64 --beam-size 6--normalize 0.6 --enc-depth 6--dec-depth 6 --transformer-heads 8--transformer-postprocess-emb d--transformer-postprocess dan--transformer-dropout 0.1--label-smoothing 0.1--learn-rate 0.0003 --lr-warmup 16000--lr-decay-inv-sqrt 16000--optimizer-params 0.9 0.98 1e-09--clip-norm 5 --tied-embeddings-all--sync-sgd --exponential-smoothing--seed 1111

88
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The MeMAD Submission to the IWSLT 2018 Speech Translation Task
Umut Sulubacak ∗Jörg Tiedemann ∗Aku Rouhe †Stig-Arne Grönroos †Mikko Kurimo †∗ Department of Digital Humanities / HELDIG
University of Helsinki, Finland{umut.sulubacak | jorg.tiedemann}@helsinki.fi† Department of Signal Processing and Acoustics
Aalto University, Finland{aku.rouhe | stig-arne.gronroos | mikko.kurimo}@aalto.fi

Abstract
This paper describes the MeMAD project entry to the IWSLT
Speech Translation Shared Task, addressing the translationof English audio into German text. Between the pipeline andend-to-end model tracks, we participated only in the former,
with three contrastive systems. We tried also the latter, butwere not able to finish our end-to-end model in time.
All of our systems start by transcribing the audio into
text through an automatic speech recognition (ASR) modeltrained on the TED-LIUM English Speech Recognition
Corpus (TED-LIUM). Afterwards, we feed the transcripts
into English-German text-based neural machine translation (NMT) models. Our systems employ three different
translation models trained on separate training sets compiled
from the English-German part of the TED Speech Translation Corpus (TED-T RANS) and the O PEN S UBTITLES 2018section of the OPUS collection.
In this paper, we also describe the experiments leading
up to our final systems. Our experiments indicate that using O PEN S UBTITLES 2018 in training significantly improvestranslation performance. We also experimented with various
pre- and postprocessing routines for the NMT module, but
we did not have much success with these.
Our best-scoring system attains a BLEU score of 16.45
on the test set for this year’s task.

1. Introduction
The evident challenge of speech translation is the transferof implicit semantics between two different modalities. An
end-to-end solution to this task must deal with the challenge
posed by intermodality simultaneously with that of interlingual transfer. In a traditional pipeline approach, while
speech-to-text transcription is abstracted from translation,
there is then the additional risk of error transfer between
the two stages. The MeMAD project1 aims at multilingual
1 https://www.memad.eu/

description and search in audiovisual data. For this reason,
multimodal translation is of great interest to the project.
Our pipeline submission to this year’s speech translation task incorporates one ASR model and three contrastive
NMT models. For the ASR module, we trained a time-delay
neural network (TDNN) acoustic model using the Kaldi
toolkit [1] on the provided TED-LIUM speech recognition
corpus [2]. We used the transformer implementation of MarianNMT [3] to train our NMT models. For these models,
we used contrastive splits of data compiled from two different sources: The n-best decoding hypotheses of the TEDT RANS [4] in-domain speech data, and a version of the
O PEN S UBTITLES 2018 [5] out-of-domain text data (S UBS),
further “translated” to an ASR-like format (S UBS -ASR)
using a sequence-to-sequence NMT model. The primarysystem in our submission uses the NMT model trainedon the whole data including S UBS -ASR, whereas one of
the two contrastive systems uses the original S UBS before
the conversion to an ASR-like format, and the other omits
O PEN S UBTITLES 2018 altogether.
We provide further details about the ASR module in Section 2. Later, we provide a review of our experiments on
the NMT module in Section 3. The first experiment we describe involves a pre-processing step where we convert our
out-of-domain training data to an ASR-like format to avoid
mismatch between source-side training samples. Afterwards,
we report a postprocessing experiment where we retrain our
NMT models with lowercased data, and defer case restoration to a subsequent procedure, and another where we translate several ASR hypotheses at once for each source sample,
re-rank their output translations by a language model, andthen choose the best-scoring translation for that sample. We
present our results in Section 4 along with the relevant discussions.

2. Speech Recognition
The first step in our pipeline is automatic speech recognition. The organizers provide a baseline ASR implementation,

89
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

which consists of a single, end-to-end trained neural network
using a Listen, Attend and Spell (LAS) architecture [6]. The
baseline uses the XNMT toolkit [7]. However, we were notable to compile the baseline system, so we trained our own
conventional, hybrid TDNN-HMM ASR system using the
Kaldi toolkit.
2.1. Architecture
Our ASR system uses the standard Kaldi recipe for the TEDLIUM dataset (release 2), although we filter out some datafrom the training set to comply with the IWSLT restrictions.
The recipe trains a TDNN acoustic model using the latticefree maximum mutual information criterion [8]. The audio
transcripts and large amount of out-of-domain text data included with the TED-LIUM dataset are used to train a heavily pruned 4-gram language model for first-pass decoding
and less pruned 4-gram model for rescoring.
2.2. Word Error Rates
The LAS architecture has achieved state-of-the-art word error rates (WER) on a task with two orders of magnitude moretraining data than here [9], but on smaller datasets hybrid
TDNN-HMM ASR approaches are still considerably better.
Table 1 shows the results of our ASR model contrasted withthose reported by XNMT in [7], on the TED-LIUM development and test sets.
Model
TDNN + large 4-gram
LAS

Dev WER
8.24
15.83

Test WER
8.83
16.16

Table 1: Word error rates on the TED-LIUM dataset.

3. Text-Based Translation
The ASR stage of our pipeline effectively converts the taskof speech translation to text-based machine translation. For
this stage, we build a variety of NMT setups and assess theirperformances. We experiment variously with the training architecture, different compositions of the training data, andseveral pre- and postprocessing methods. We present theseexperiments in detail in the subsections to follow, and thendiscuss their results in Section 4.
3.1. Data Preparation
We used the development and test sets from 2010’s sharedtask for validation during training, and the test sets from thetasks between 2013 and 2015 for testing performance during
development. In all of our NMT models, we preprocessed
our data using the punctuation normalization and tokenization utilities from Moses [10], and applied byte-pair encoding [11] through full-cased and lowercased models as relevant, trained on the combined English and German texts in

TED-T RANS and S UBS using 37,000 merge operations tocreate the vocabulary.
We experiment with attentional sequence-to-sequencemodels using the Nematus architecture [12] with tied embeddings, layer normalization, RNN dropout of 0.2 andsource/target dropout of 0.1. Token embeddings have a dimensionality of 512 and the RNN layer units a size of 1024.
The RNNs make use of GRUs in both, encoder and decoder. We use validation data and early stopping after five cycles (1,000 updates each) of decreasing cross-entropy scores.
During training we apply dynamic mini-batch fitting with aworkspace of 3GB. We also enable length normalization.
For the experiments with the transformer architecture weapply the standard setup with six layers in encoder and decoder, eight attention heads and a dynamic mini-batch fit to
8GB of work space. We also add recommended options suchas transformer dropout of 0.1, label smoothing of 0.1, a learning rate of 0.0003, a learning-rate warmup with a linearly increasing rate during the first 16,000 steps, a decreasing learning rate starting at 16,000 steps, a gradient clip norm of 5 andexponential smoothing of parameters.
All translations are created with a beam decoder of size
12.
3.1.1. ASR Output for TED Talks
Translation models trained on standard language are not agood fit for a pipeline architecture that needs to handle noisy
output from the ASR component discussed previously in
Section 2. Therefore, we ran speech recognition on the entire
TED-T RANS corpus in order to replace the original, humanproduced English transcriptions with ASR output, which hasrealistic recognition errors.
To generate additional speech recognition errors to thetraining transcripts, we selected the top-50 decoding hypotheses. We did the same also for the development data
to test our approach. We can now sample from those ASR
hypotheses to create training data for our translation models
that use the output of English ASR as its input. We experimented with various strategies varying from a selection of the
top n ASR candidates to different mixtures of hypotheses ofdifferent ranks of confidence. Some of these are shown in Table 2. In the end, there was not a lot of variance between thescores resulting from this selection, and we decided to use
the top-10 ASR outputs in the remaining experiments to encourage some tolerance for speech recognition errors in thesystem.
3.1.2. Translating Written English to ASR-English
The training data that includes audio is very limited andmuch larger resources are available for text-only systems.
Especially useful for the translation of TED talks is the collection of movie subtitles in O PEN S UBTITLES 2018. For
English-German, there is a huge amount of movie subtitles(roughly 22 million aligned sentences with over 170 million

90
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Training data
TED-ASR-T OP -1
TED-ASR-T OP -10
TED-ASR-T OP -50
TED-ASR-T OP -1
TED-ASR-T OP -10
TED-ASR-T OP -50

Model
A MUN
A MUN
A MUN
T RANSFORMERT RANSFORMERT RANSFORMER

BLEU
16.65
16.28
15.88
18.25
17.90
18.14

Table 2: Translating the development test set with different
models and different selections of ASR output and Germantranslations from the parallel TED-T RANS training corpus.
tokens per language) that can be used to boost the performance of the NMT module.
The problem is, of course, that the subtitles come in regular language, and, again, we would see a mismatch between
the training data and the ASR output in the speech translation
pipeline. In contrast to approaches that try to normalize ASR
output to reflect standard text-based MT input such as [13],we had the idea to transform regular English into ASR-like
English using a translation model trained on a parallel corpus
of regular TED talk transcriptions and the ASR output generated for the TED talks that we described in the previous
section. We ran a number of experiments to test the performance of such a model. Some of the results are listed inTable 3.
Training data
TED-ASR-T OP -10
TED-ASR-T OP -10
TED-ASR-T OP -50

Model
A MUN
T RANSFORMER
A MUN

BLEU
61.87
61.91
61.82

what the system actually learns to do. Most of the transformations are quite straightforward. The model learns to lowercase and to remove punctuation as our ASR output does
not include it. However, it also does some other modifications that are more interesting from the viewpoint of an ASR
module. While we do not have systematic evidence, Table 4shows a few selected examples that show interesting patterns.
First of all, it learns to spell out numbers (see “2006” inthe first example). This is done consistently and quite accurately from what we have seen. Secondly, it replaces certain
tokens with variants that resemble possible confusions that
could come from a speech recognition system. The replacement of “E.U.” with “you” and “Stasi” with “stars he” inthese examples are quite plausible and rather surprising for amodel that is trained on textual examples only. However, toconclude that the model learns some kind of implicit acoustic
model would be a bit far-fetched, even though we would liketo investigate the capacity of such an approach further in the
future.
Original
ASR- REF
ASR- OUT
Original
ASR- REF
ASR- OUT

Table 3: Translating English into ASR-like English using amodel trained on TED-T RANS and tested on the development test set with original ASR output as reference.
As expected, the BLEU scores are rather high as the target language is the same as the source language, and we only
mutate certain parts of the incoming sentences. The results
show that there is not such a dramatic difference between the
different setups (with respect to the model architecture andthe data selection) and that a plain attentional sequence-tosequence model with recurrent layers (A MUN) performs aswell as a transformer model (T RANSFORMER) in this case.
This makes sense, as we do not expect many complex longdistance dependencies that influence translation quality inthis task. Therefore, we opted for the A MUN model trainedon the top-10 ASR outputs, which we can decode efficiently
in a distributed way on the CPU nodes of our computer cluster. With this we managed to successfully translate 99% of
the entire S UBS collection from standard English into ASREnglish. We refer to this set as S UBS -ASR.
We did a manual inspection on the result as well to see

Because in the summer of 2006, the E.U.
Commission tabled a directive.
because in the summer of two thousand andsix the e u commission tabled directive
because in the summer of two thousand andsix you commission tabled a directive
Stasi was the secret police in East
Germany.
what is the secret police in east germany
stars he was the secret police in east
germany

Table 4: Examples from the translations to ASR-like English.
In the first column, ASR- REF refers to the top decoding hypothesis from the ASR model, while ASR- OUT is the outputof the model translating the output to an ASR-like format.
In Section 4, we report on the effect of using synthetic
ASR-like data on the translation pipeline.
3.2. Recasing Experiments
Our first attempt at a post-processing experiment involved
using case-insensitive translation models, and deferring caserestoration to a separate process unconditioned by the sourceside that we would apply after translation. We used theMoses toolkit [10] to train a recaser model on TED-T RANS.
Afterwards, we re-trained a translation model on TEDASR-T OP -10 and S UBS -ASR after lowercasing the training and validation sets, re-translated the development test set
with this model, and then used the recaser to restore casesin the lowercase translations that we obtained. As shown in
Table 5, evaluating the translations produced through these
additional steps yielded scores that were very similar to those

91
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

obtained by the original case-sensitive translation models,
and the result of this experiment was inconclusive.
Training data
TED-ASR-T OP -10+S UBS -ASR
TED-ASR-T OP -10+S UBS -ASR- LC

BLEU
19.79
19.73

BLEU-LC
20.43
20.91

Table 5: Case-sensitive models (T RANSFORMER) versus
lowercased models with subsequent recasing. Recasing
causes a larger drop than the model gains from training onlowercased training data. BLEU-LC refers to case-insensitive
BLEU scores.
3.3. Reranking Experiments
In addition to using different subsets of the n-best lists output by the ASR model as additional training samples for thetranslation module, we also tried reranking alternatives using
KenLM [14]. We initially generated a tokenized and lowercased version of TED-T RANS with all punctuation stripped,
and then trained a language model on this set. We used thismodel to score and rerank samples in the 50-best lists, andthen generated a new top-10 subset from this reranked version. However, when we re-trained translation models fromthese alternative sets, we observed that the model trained on
the top-10 subsets before reranking exhibited a significantly
better translation performance. We suspect that this is because, while the language model is useful for assessing thesurface similarity of the ASR outputs to the source-side references, it was not uncommon for it to assign higher scores
to ASR outputs that are semantically inconsistent with thetarget-side references, causing the NMT module to produce
erroneous translations.
Similarly, we experimented with another language modeltrained on the target side of TED-T RANS, without the preprocessing. We intended this model to score and rerank outputs of the translation models, rather than the ASR module.
To measure the effect of this language model, we fed the audio of our internal test set split through the ASR module, andproduced 50-best lists for each sample. Afterwards, we used
the language model to score and rerank the alternative transcripts for each sample produced by translating this set, andthen selected the highest-scoring output for each sample. As
in the previous language model experiment, employing thisadditional procedure significantly crippled the performanceof our translation models.

the transformer model that is also visible in additional runsthat we do not list here. Secondly, we can see the importanceof additional training data even if they come from slightly
different domains. The vast amount of movie subtitles in
O PEN S UBTITLES 2018 boosts the performance by about 3
absolute BLEU points. Note that the scores in Table 6 refer
to models that do not use subtitles transformed into ASR-like
English (S UBS -ASR) and which are not fine-tuned to TED
talk translations.
Training data
TED-ASR-T OP -10
TED-ASR-T OP -10+S UBS
TED-ASR-T OP -10
TED-ASR-T OP -10+S UBS

Model
A MUN
A MUN
T RANSFORMERT RANSFORMER

BLEU
16.28
19.93
17.90
20.44

Table 6: Model performance on the development test set
when adding movie subtitles to the training data.
The effect of pre-processing by producing ASR-like English in the subtitle corpus is surprisingly negative. If welook at the scores in Table 7, we can see that the performance
actually drops in all cases when considering only the untuned
systems. We did not really expect that with the rather positive impression that we got from the manual inspection of the
English-to-ASR translation discussed earlier. However, it isinteresting to see the effect of fine-tuning. Fine-tuning here
refers to a second training procedure that continues training
with pure in-domain data (TED talks) after training the general model on the entire data set until convergence on validation data. Table 7 shows an interesting effect that may explain the difficulties of the integration of the synthetic ASR
data. The fine-tuned model actually outperforms the modeltrained on standard data, which is due to a substantial jumpfrom untuned models to the tuned version. The difference between those models with standard data is, on the other hand,
only minor.

Training data
TED-ASR-T OP -10+S UBS
TED-ASR-T OP -10+S UBS -ASR

BLEU
Untuned Tuned
20.44 20.58
19.79 20.80

4. Results

Table 7: Training with original movie subtitles versus subtitles with English transformed into ASR-like English, before and after fine-tuning on TED-ASR-T OP -10 as pure indomain training data (T RANSFORMER).

The results on development data reveal expected tendencies that we report below. First of all, as consistent with a
lot of related literature, we can see a boost in performance
when switching from a recurrent network model to the transformer model with multiple self-attention mechanisms. Table 6 shows a clear pattern of the superior performance of

The synthetic ASR data look more similar to the TEDASR data and, therefore, the model might get more confused
between in-domain and out-of-domain data than it does forthe model trained on the original subtitle data in connection
with TED-ASR. Fine-tuning to TED-ASR brings the model

92
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

back on track again and synthetic ASR data becomes modestly beneficial.
Also of note is the contrast between the evaluation scores
we obtained in development and those from the official test
set. The translations we submitted obtain the BLEU scoresshown in Table 8 on this year’s test set.
Training data
TED-ASR-T OP -10
TED-ASR-T OP -10+S UBS
TED-ASR-T OP -10+S UBS -ASR

BLEU
14.34
16.45
15.80

7. References[1] D. Povey, A. Ghoshal, G. Boulianne, L. Burget,
O. Glembek, N. Goel, M. Hannemann, P. Motlicek,
Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and
K. Vesely, “The Kaldi Speech Recognition Toolkit,” inIEEE 2011 Workshop on Automatic Speech Recognition
and Understanding. IEEE Signal Processing Society,Dec. 2011, iEEE Catalog No.: CFP11SRW-USB.
[2] A. Rousseau, P. Deléglise, and Y. Esteve, “TED-LIUM:
an automatic speech recognition dedicated corpus,” in
LREC, 2012, pp. 125–129.

Table 8:BLEU scores from our final models(T RANSFORMER)—respectively, the 2nd contrastive,
1st contrastive, and primary submission—on this year’s
test set. The scores from the two models with S UBS
in their training data were obtained after fine-tuning on
TED-ASR-T OP -10.

5. Conclusions
Apart from employing well-established practices such as normalization and byte-pair encoding as well as the benefits ofusing the transformer architecture, the only substantial boost
to translation performance came from our data selection for
the NMT module. The NMT module of our best-performing
system on this year’s test set was trained on TED-ASRT OP -10 and the raw S UBS, and later fine-tuned on TEDASR-T OP -10.
Although we ran many experiments to improve various
steps of our speech translation pipeline, their influence ontranslation performance has been marginal at best. The effects of training with different TED-ASR subsets were hardto distinguish. While using S UBS -ASR in training seemed
to provide a modest improvement in development, this effect was not carried over to the final results on the test set.
The later experiments with lowercasing and recasing had an
ambiguous effect, and those with reranking had a noticeably
negative outcome.
In future work, our aim is to further investigate what factors in a good speech translation model, and continue experimenting in relation to these on the NMT module. We will
also try to improve our TDNN-HMM ASR module by replacing the n-grams with an RNNLM, and try see how our complete end-to-end speech-to-text translation model performs
after having sufficient training time.

6. Acknowledgements
This work has been supported by the European Union’s Horizon 2020 Research and Innovation Programme under Grant
Agreement No 780069, and by the Academy of Finland in
the project 313988. In addition the Finnish IT Center for
Science (CSC) provided computational resources.

[3] M. Junczys-Dowmunt, R. Grundkiewicz, T. Dwojak,
H. Hoang, K. Heafield, T. Neckermann, F. Seide,
U. Germann, A. Fikri Aji, N. Bogoychev, A. F. T.
Martins, and A. Birch, “Marian: Fast neuralmachine translation in C++,” in Proceedings of
ACL 2018, System Demonstrations. Melbourne,
Australia: Association for Computational Linguistics, July 2018, pp. 116–121. [Online]. Available:
http://www.aclweb.org/anthology/P18-4020[4] M. Cettolo, C. Girardi, and M. Federico, “WIT3 : Web
inventory of transcribed and translated talks,” in Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), Trento, Italy,
May 2012, pp. 261–268.
[5] J. Tiedemann, “News from OPUS - A collection ofmultilingual parallel corpora with tools and interfaces,”in Recent Advances in Natural Language Processing,N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov,
Eds. Borovets, Bulgaria: John Benjamins, Amsterdam/Philadelphia, 2009, vol. V, pp. 237–248.
[6] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen,attend and spell: A neural network for large vocabulary conversational speech recognition,” in Acoustics,
Speech and Signal Processing (ICASSP), 2016 IEEE
International Conference on. IEEE, 2016, pp. 4960–
4964.
[7] G. Neubig, M. Sperber, X. Wang, M. Felix,
A. Matthews, S. Padmanabhan, Y. Qi, D. S. Sachan,
P. Arthur, P. Godard, et al., “XNMT: The extensible neural machine translation toolkit,” arXiv preprint
arXiv:1803.00188, 2018.
[8] D. Povey, V. Peddinti, D. Galvez, P. Ghahremani,
V. Manohar, X. Na, Y. Wang, and S. Khudanpur,“Purely sequence-trained neural networks for ASR
based on lattice-free MMI,” in Interspeech, 2016, pp.
2751–2755.
[9] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar,
P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao,
K. Gonina, et al., “State-of-the-art speech recognition

93
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

with sequence-to-sequence models,” arXiv preprint
arXiv:1712.01769, 2017.
[10] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, et al., “Moses: Open source toolkit
for statistical machine translation,” in Proceedings ofthe 45th annual meeting of the ACL on interactive
poster and demonstration sessions. Association for
Computational Linguistics, 2007, pp. 177–180.
[11] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in
ACL16, 2015.
[12] R. Sennrich, O. Firat, K. Cho, A. Birch, B. Haddow, J. Hitschler, M. Junczys-Dowmunt, S. Läubli,
A. V. M. Barone, J. Mokry, and M. Nadejde, “Nematus: a toolkit for neural machine translation,”
CoRR, vol. abs/1703.04357, 2017. [Online]. Available:
http://arxiv.org/abs/1703.04357[13] E. Matusov, A. Mauser, and H. Ney, “Automatic sentence segmentation and punctuation prediction for spoken language translation,” in International Workshop
on Spoken Language Translation, Kyoto, Japan, Nov
2006, pp. 158–165.
[14] K. Heafield, “KenLM: Faster and smaller language
model queries,” in Proceedings of the Sixth Workshop
on Statistical Machine Translation. Association for
Computational Linguistics, 2011, pp. 187–197.

94
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Prompsit’s Submission to the IWSLT 2018 Low Resource Machine Translation
Task
Vı́ctor M. Sánchez-Cartagena
Prompsit Language Engineering
Av. Universitat s/n. Edifici Quorum III
E-03202 Elx, Spainvmsanchez@prompsit.com

Abstract
This paper presents Prompsit Language Engineering’s submission to the IWSLT 2018 Low Resource Machine Translation task. Our submission is based on cross-lingual learning: a multilingual neural machine translation system wascreated with the sole purpose of improving translation quality on the Basque-to-English language pair. The multilingual system was trained on a combination of in-domain data,
pseudo in-domain data obtained via cross-entropy data selection and backtranslated data. We morphologically segmented
Basque text with a novel approach that only requires a dictionary such as those used by spell checkers and proved that
this segmentation approach outperforms the widespread byte
pair encoding strategy for this task.

1. Introduction
This paper presents Prompsit Language Engineering’s submission to the IWSLT 2018 Low Resource Machine Translation task. The objective of this task is building an MT system for translating TED talks from Basque to English froma very limited amount of in-domain Basque–English paralleldata. We relied on cross-lingual learning via a multilingual
approach [1] to neural machine translation (NMT), extraction
and cleaning of pseudo in-domain parallel text from out-ofdomain data, and backtranslation of Spanish text into Basquefor building our submission.
Moreover, we applied morphological segmentation to theBasque text. We took advantage of an existing spell checking dictionary and its inflection paradigms and used an automatic morphology inference model to decide between ambiguous segmentations. We proved that this method, that requires shallower linguistic information1 than other segmentation approaches based on full morphological analysis anddisambiguation [2, 3], outperforms the widespread byte pairencoding (BPE) segmentation strategy [4] in terms of translation quality for Basque-to-English NMT.
1 Neither part of speech/morphological information in the dictionary nor
a part of speech tagger/parser are needed. In principle, this approach
could be applied to any language for which a Hunspell-based (http://hunspell.github.io/) spell checker exists.

Table 1: Size of in-domain data. Processed segments arethose that remain after removing talks included in the development and test sets.
Language pair
eu-en
eu-es
eu-fr
es-en
fr-en

# raw segments
5 687
6 742
7 021
280 947
290 961# processed segments
5 687
5 610
5 878
279 737
289 722

The remainder of the paper explains the steps followed
to build the submitted NMT system. Next section explains
how the in-domain and out-of-domain parallel corpora were
processed and filtered, while Section 3 focuses on describing
and assessing the impact of the morphological segmentation
approach followed. Section 4 describes the NMT architecture and training process. Section 5 depicts the process followed to obtain the data set used to train our submission.
Finally, the most relevant related approaches are reviewed in
Section 6 and the paper ends with some concluding remarks.

2. Data acquisition and cleaning
Our submission was trained on a combination of in-domain
and out-of-domain data. The only special cleaning applied
to the in-domain training data provided by the organization is the removal of talks that are also included in the
test/development sets. Table 1 shows the number of segmentsin the in-domain data for each language pair before and after
removing such talks.
Following the shared task instructions,2 we built the outof-domain data collection by downloading all the corpora
available from the Opus [5] and WMT [6] websites.3 We
also included the Basque–Spanish parallel data from Open
2 https://sites.google.com/site/
iwsltevaluation2018/TED-tasks
3 If the same corpus was available from both websites (e.g. Europarl), we
downloaded it from WMT. If the same corpus was available from different
WMT editions, we downloaded it from the most recent one. We skippedsome corpora from Opus which were too noisy, like EUBookshop.

95
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 2: Size of out-of-domain data before and after applying
shallow cleaning.
Language pair
eu-en
eu-es
eu-fr

# raw segments
1.81M
1.64M
711K

# clean segments
928K
1.41M
375K

Table 3: Size of out-of-domain data before and after applying
aggressive cleaning.
Language pair
es-en
fr-en

# raw segments
163M
164M

# clean segments
65M
77M

3. Morphological segmentation for BasqueWord segmentation based on linguistically-informed strategies such as morphological analysis [2] or simpler alternatives based on lists of relevant prefixes and suffixes [8] haveshown to be able to outperform the popular BPE approach [4]for some agglutinative and highly inflected languages. In
this section, we present the pseudo-morphological segmentation approach based on inflection paradigms we applied toBasque text in our submission and prove that it outperforms
BPE.
3.1. Pseudo-morphological segmentation based on inflection paradigms

Data Euskadi Repository published by the task organizers.
We followed two different strategies for out-of-domain
parallel data cleaning. For language pairs with limited data
availability, namely those including Basque, we followed aconservative shallow cleaning strategy since removing correct segments can be harmful for the quality of the final system. For the remaining language pairs, since only a subsetof the data is finally used (see Section 5), we applied a moreaggressive cleaning strategy.
The shallow cleaning consisted in deduplication and removal of parallel segments that meet any of the following
conditions: they contain a low proportion of alphabetic characters, their source-language (SL) and target-language (TL)side are very similar (there is a low edit distance betweenthem), they are too long or too short (shorter than 3 tokens or
longer than 100), or they are written in another language (language is detected by means of cld24 and segments are only
discarded when the language detection is reliable according
to the cld2 algorithm). Table 2 shows the size of the out-ofdomain data for each language pair containing Basque before
and after applying shallow cleaning.5
The aggressive cleaning consisted in two steps. Firstly,
parallel segments were deduplicated and a more aggressive superset of the rules used in the shallow cleaning (implemented in the translation memory cleaning tool
Bicleaner6 ) was applied. These rules are addressed at detecting evident flaws such as encoding errors, very different
lengths in parallel segments, etc. Secondly, misaligned segments were detected and removed by means of an automaticclassifier, described in [7]. The classifier is also part of the
Bicleaner tool. Pre-trained models for the classifier wereobtained from the Paracrawl project.7 Table 3 shows the sizeof the out-of-domain data for each language pair before andafter applying the aggressive cleaning.
4 https://github.com/CLD2Owners/cld2

5 Shallow cleaning was not applied to the Basque–Spanish parallel datafrom Open Data Euskadi Repository.
6 https://github.com/bitextor/bicleaner
7 https://github.com/bitextor/bitextor-data/tree/
master/bicleaner

Inflection paradigms are commonly used in dictionaries(morphological dictionaries used in rule-based machinetranslation, spell checkers, etc.) in order to group regularities
in the inflection of a set of words.8 A paradigm is usually
defined as a collection of suffixes and, optionally, their corresponding part-of-speech/morphological information; e.g.,
the paradigm assigned to many common English verbs indicates that by adding the suffix -ing to the stem, the gerund is
obtained; by adding the suffix -ed, the past is obtained; etc.
While morphological dictionaries from rule-based machinetranslation systems contain morphological information, spell
checkers usually lack this information.
In languages with a high inflection degree, such as
Basque, a surface form can be built by sequentially appending suffixes from different paradigms to a stem. For instance, the word etxekoak can be generated from the entry
etxe + P AR240 if paradigm P AR240 contains the suffixes−ko + P AR243, −z, −rekin, etc. and paradigm P AR243
contains the suffix −ak.
As suffixes contained in inflection paradigms are usuallybased on linguistic knowledge, one can take advantage of
inflection paradigms for splitting words for training NMT
systems. In this way, words can be split in atomic units ofmeaning or morphs. For instance, in previous example, etxekoak (the plural form of “domestic”) would be split into etxe(“house”), -ko (adjectivation) and -ak (plural mark).
In order to split a corpus using inflection paradigms, there
are two types of words for which an additional strategy needs
to be devised:• Homograph words: those that can be generated bymultiple combinations of stem and suffix(es).
• Unknown words: those that are not present in the morphological dictionary/spell checking dictionary.
In order to decide the best segmentation for these words,
we took advantage of semi-supervised morphology learning
methods. In particular, we used Morfessor [9]. Morfessor is
8 Paradigms ease dictionary management by reducing the quantity of information that needs to be stored, and by simplifying revision and validation
because of the explicit encoding of regularities in the dictionary.

96
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

a family of methods for automatic learning of morphology
based on the minimum description length principle [10]: thewords in a corpus are split in morphs in such a way that thesize of the morph vocabulary and the length in tokens of the
corpus are minimized. We used a semi-supervised variantof Morfessor in which the segmentation model can be estimated from a plain corpus and a set of already segmented
words [11].
Our pseudo-morphological segmentation strategy comprises the following steps:
1. Segment words encoded in the morphological dictionary/spell checker which have only a candidate segmentation according to the inflection paradigms.
2. Train a Morfessor segmentation model in an semisupervised way [11] from the Basque corpus we want
to segment and the words segmented in the previous
step.
3. Segment homograph words by choosing the segmentation with the highest likelihood according to the previous model.
4. Segment unknown words by choosing the segmentation with highest likelihood according to the model
among those that can be generated by using solely suffixes from the inflection paradigms in the morphological dictionary/spell checker.
This approach hence allows us to segment a corpus in
atomic units of meaning using a spell checker as the only
linguistic resource. Unlike other approaches to NMT training corpus word segmentation based on linguistic information [2, 3], this approach does not require neither a fullmorphological analyzer with part-of-speech/morphologicaltags nor a part-of-speech tagger/parser for disambiguating between the different analyses of each word. Part-ofspeech/morphological information (e.g. the fact the suffix-ed represents the past tense of a verb) is not used during theprocess and disambiguation is carried out by the Morfessor
model which, in turn, controls the growth of vocabulary size.
In our submission, we used the Basque spell checker
Xuxen v5.1 as dictionary.9 Moreover, following [8], weapplied BPE splitting with a model learned on the concatenation of all training corpora after performing thepseudo-morphological segmentation. Note that applying
BPE to further split the word pieces obtained after pseudomorphological segmentation helps the system to translate
proper nouns and compounds in Basque.
3.2. Evaluation
We evaluated the pseudo-morphological segmentation approach we employed in our submission and compared it withtwo baselines: a greedy alternative in which the segmentation with the most frequent stem is chosen for unknown
9 https://xuxen.eus

Table 4: Results of the evaluation of the pseudomorphological segmentation approach proposed, a greedy
alternative, and plain BPE.
Segmentation strategy
BPE
Paradigms/Greedy+BPE
Pseudo-morph+BPE

BLEU
12.75
13.28
13.59

TER
83.68
87.80
79.73

and homograph words, and plain BPE splitting. In all cases,
BPE was applied to all the languages of the training corpus(65 000 operations) and the model was learned from theirconcatenation after carrying out pseudo-morphological segmentation (except for the plain BPE system, for which morphological segmentation was not carried out).
We trained multilingual NMT systems as described in
Section 4 on parallel corpora segmented following the threestrategies. The three multilingual NMT systems weretrained on the in-domain data and included the language
pairs Spanish–English, French–English, Basque–English,
Basque–French and Basque–Spanish.
The evaluation was carried out only on the Basque-toEnglish direction. The values of the translation evaluation metrics BLEU [12] and TER [13] computed on thedevelopment set are reported in Table 4. We can observe that our pseudo-morphological segmentation approach(Pseudo-morph+BPE) outperforms both plain BPE segmentation and segmentation based on paradigms with agreedy strategy for homograph and unknown words.
Table 5 shows several examples of words segmented by
the three alternatives evaluated. Furthermore, Table 6 depicts
three Basque sentences from the development set, how they
were segmented by the three alternatives evaluated and theirtranslation with the NMT systems built. Note that, unlike thewords in Table 5, the SL sentences in Table 6 were split with
BPE after applying the splitting strategies based on inflection
paradigms, as described previously in this section. In the firstexample, the Basque word konpartimentutan is formed by
the stem konpartimentu, which means “compartment”, plus
the inessive suffix -tan). The segmentation strategies basedon inflection paradigms are able to correctly detach the inessive suffix from the word, while the pure BPE approach fails
to do it. As a consequence, the MT system built using the latter approach is not able to produce an adequate translation bytaking advantage of the sentences in the training corpus that
contain words starting with konparti-. Similarly, in the second example, the segmentation strategies based on inflection
paradigms are able to segment estudioa into the stem estudio(that means “studio apartment”) and the suffix -a (singular
article). The pure BPE approach segments it into estudi- and-oa. Since estudi- is the stem of the verb “to study” in Spanish, the multilingual system wrongly generates that verb in
the translation into English. Finally, in the third example,
the greedy approach based on paradigms wrongly segments
Asia into as and -ia, which prevents the NMT system from

97
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 5: Examples of Basque words segmented by the three approaches evaluated. The segmentation that best splits the word in
atomic units of meaning is shown in bold.
Word

BPE

Paradigms/Greedy

Pseudo-morphadierazitako

adierazitako

adieraz itako

adierazi tako

izendatu
ebaluaketa
birgaitzeko

izendatu
ebalu aketa
bir gaitzeko

izenda tu
ebaluaket abirgaitze ko

izendatu
ebaluaketa
birgaitze ko

producing the word Asia in English.

4. Training strategy
Our submission is based on cross-lingual learning. We aimed
at improving the translation performance on the Basque-toEnglish language pair by means of the addition of training data from other language pairs. The different language
pairs were combined by means of a multilingual NMT approach [1]. A TL marker was prepended to each SL segment.
See Section 5 for more details about language pairs included
and how the data for each of them was obtained.
Our submitted NMT system follows the Transformer architecture [14]. In particular, we used the implementation in
the Marian NMT toolkit [15]. We generally used the hyperparameters of the Transformer base model [14], with the exception of warmup steps, which was set to 16 000 instead of
4 000. This parameter was increased because our minibatch
size was significantly smaller than that used in the original
paper [14]. We limited segment length to 100 tokens and let
the Marian toolkit set the batch size to fit 8 000 MiB of GPU
memory. For a vocabulary size of around 70 000 words, thenumber of TL words in a minibatch was around 3 000, while[14] report 25 000 TL words per minibatch. A checkpoint
was saved every 5 000 updates.
We used only the publicly released Basque–English
IWSLT18.TED.dev2018 corpus as a development set.10
Training ended when perplexity on the development set did
not improve in 10 consecutive checkpoints. We selected thecheckpoint that obtained the highest BLEU score on the development set.
Concerning corpora preprocessing, text was tokenizedwith the aggressive strategy11 implemented by the OpenNMT tokenizer [16]. Words were split in sub-word units as
described in Section 3. The Morfessor model was trainedon the concatenation of the Basque section of the training
data for all language pairs that contained Basque. The BPEmodel (65 000 operations), which shared by all SLs and TLs,
was learned from the concatenation of the morphologicallysegmented Basque data and the unsegmented data for the re10 It could be interesting to study whether using development data fromother language pairs has a significant impact in translation quality for
Basque–English.
11 The only multi-character tokens allowed are sequences of strictly alphabetical characters.

meaning“expressed”, built from adierazi (“to express”)
plus -tako (suffix used in relative clauses)“nominate”, atomic unit
“evaluation”, atomic unit
“rehabilitation” (birgaitze) plus genitive suffix (-ko)

maining languages and it was used to split these corpora.
Text was lowercased prior to training and the resulting English translations were recased12 with a recasing model estimated from the concatenation of the English side of the training corpora.

5. Training data
This section describes the training data from which our submission was built and the experiments carried out to selectit.
5.1. Language pairs
According to the experiments carried out by [1], including new language pairs that share either the SL or the
TL with the language pair of interest helps to increase
the translation quality for that language pair. Henceforth,
our multilingual system contains only language pairs withBasque as SL or English as TL. Moreover, we included
only language pairs for which the training set is publishedas part of this year’s data. Hence, our multilingual system
contains data from the Spanish–English, French–English,
Basque–English, Basque–French and Basque–Spanish language pairs. Preliminary experiments showed no important gains when adding data from the German–English andTurkish–English language pairs to the training collection.
Conducting more exhaustive experiments has been left as future work.
5.2. Cross-entropy data selection and oversampling
As shown in Table 3, there is a huge amount of out-ofdomain parallel data available for the Spanish–English andFrench–English language pairs. If it was just concatenatedto the in-domain data, the system would be biased towards
the out-of-domain data. In order to avoid that issue, we selected only a subset of the out-of-domain data which is similar to the in-domain one (from now on, pseudo in-domaindata) via cross-entropy difference [17].
The process was carried out as follows. Firstly, we sorted
the out-of-domain data (after cleaning it as described in Section 2) by monolingual cross-entropy difference on the English side. The in-domain language model was estimated
12 TheMoses recaser was used: http://www.statmt.org/
moses/?n=Moses.SupportTools#ntoc10.

98
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 6: Result of applying each of the three segmentation strategies evaluated in Section 3 to a three sentences extracted fromthe development set. The translation of each sentence with a multilingual NMT system trained only on the in-domain data is also
depicted. The character → at the end of a token implies that it is a sub-word unit originally attached to the token that follows it.
Words whose segmentation has a visible impact on the translation are shown in bold.
#

1

2

3

segmentation strategy
source – BPE
source – Paradigms/Greedy+BPE
source – Pseudo-morph+BPE
translation – BPE
translation – Paradigms/Greedy+BPE
translation – Pseudo-morph+BPE
referencesource – BPE
source-Paradigms/Greedy+BPE
source – Pseudo-morph+BPE
translation – BPE
translation – Paradigms/Greedy+BPE
translation – Pseudo-morph+BPE
referencesource – BPE
source – Paradigms/Greedy+BPE
source – Pseudo-morph+BPE
translation – BPE
translation – Paradigms/Greedy+BPE
translation – Pseudo-morph+BPE
reference

sentence
burmu→ ina ez dago kon→ parti→ men→ tutan ban→ atuta .
bur→ mu→ in→ a ez dago kon→ parti→ mentu→ tan bana→ tuta .
bur→ mu→ in→ a ez dago kon→ parti→ mentu→ tan bana→ tuta .
There’s no brain at all based on bias.
You don’t have a brain that’s broken up into blocks.
There’s no boundary in the brain.
The brain isn’t divided into compartments.
beraz urte batez estudi→ oa ix→ tea erabaki nuen .
bera→ z urte bat→ ez estudio→ a ix→ te→ a erabaki nu→ en .
beraz urte bat→ ez estudio→ a ix→ te→ a erabaki nuen .
So I decided to study for a year.
So one year I decided to give it a try.
So I decided to stay silent for a year.
So I decided to close it down for one year.
beraz asia aukeratu nuen .
bera→ z as→ ia aukera→ tu nu→ en .
beraz asia aukeratu nuen .
So I chose Asia.
So I decided to give it a try.
So I chose Asia.
So Asia it was.

from the English side of the parallel in-domain Spanish–
English training corpus, while the out-of-domain one wasobtained from a random sample with the same number ofsegments from the English side of all the available Spanish–
English parallel data. The same language models were used
for computing monolingual cross-entropy difference for boththe Spanish–English and French–English language pairs. As
other authors did previously [18], we split English corpora
with BPE prior to training the language models and scoring
the out-of-domain parallel segments.
Secondly, we carried out a set of experiments in orderto decide which is the most appropriate amount of pseudo
in-domain data for Spanish–English and French–English. In
these experiments, we used all the available data for Basque–English, Basque–Spanish and Basque–French, and varying
amounts of pseudo in-domain data, which was concatenatedto the real in-domain data, for Spanish–English and French–
English. In addition, we also studied the effect of oversampling the Basque–English data (concatenation of in-domain
and out-of-domain) to match the size of the Spanish–English
and French–English data.
Table 7 depicts the size of the pseudo in-domain parallel
data13 and the size of the Basque–English data included in
the training set for the different configurations evaluated, to-gether with the values of the evaluation metrics BLEU [12]and TER [13] computed on the development set. The original size of the Basque–English data is 933 356 segments.
Those rows with values higher than 0.9M imply that the data
the Basque–English has been oversampled. In other words,
it has been included as many times as necessary for reaching
the size depicted in the table. Systems were trained following
the set-up described in Section 4. For the same data configurations, Table 8 shows automatic evaluation metrics computed after finetuning the systems on the in-domain data.14
Results show no important gains when increasing the out-ofdomain data size from 2M to 5M and confirm the importanceof oversampling, in line with the results reported in [1]. Finetuning on in-domain data did not bring any positive impact.
One possible reason could be the scarce amount of in-domain
Basque–English data available (see Table 1). We chose theconfiguration with the highest BLEU score on the development set (depicted in bold in Table 7) for our submission.

13 For a given size N , the N parallel segments with the lowest cross-

14 When finetuning, the initial learning rate was set to the value employed

entropy score are selected from the out of domain data.

5.3. Backtranslation
Backtranslation, that is, the translation of additional TL
monolingual data into the SL with an MT system in order toobtain additional training material, is a widespread method
to enhance the quality of NMT systems [19].
in the last update of the main training process.

99
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 7: Results of the experiments carried out in order todetermine the best size for pseudo in-domain data and forBasque–English data (with oversampling). Unlike the experiments depicted in Table 8, these experiments did not include
finetuning on the in-domain data at the end of the training
process. The configuration highlighted in bold is the oneused in our submission.
Pseudo in-domain size
2M
2M
5M
5M
5M

eu-en size
0.9M
2M
0.9M
3M
5M

BLEU
21.05
21.72
19.47
20.46
21.10

TER
68.15
68.65
71.86
70.17
68.95

Table 8: Results of the experiments carried out in order todetermine the best size for pseudo in-domain data and forBasque–English data (with oversampling). Unlike the experiments depicted in Table 7, these experiments included
finetuning on the in-domain data at the end of the training
process.
Pseudo in-domain size
2M
2M
5M
5M
5M

eu-en size
0.9M
2M
0.9M
3M
5M

BLEU
21.15
21.68
19.96
20.88
21.68

TER
69.11
68.46
71.20
71.46
69.38

Table 9: Results of the experiments carried out in order todetermine the best size for backtranslated data. The configuration highlighted in bold is the one used in our submission.
Size of backtranslated data
0
2.2M
5.2M

BLEU
21.72
22.51
23.45

TER
68.65
67.54
66.94

as 2.2M). Results of the evaluation on the development setof the NMT systems trained with these data are depicted inTable 9. They show that using the whole backtranslated data
has a strong positive impact on the quality of the resulting
MT system. Hence, we used the 5.2M backtranslated segments in our submission.
5.4. Final submission
Our final submission was trained on the best data collection
from previous section. We experimented with finetuning andcheckpoint ensembling [21, Sec. 3.2], but translation quality
did not improve. Hence, we submitted just the result of translating the test set with the intermediate model that achieved
the highest BLEU score on the development set.

6. Related approaches

In our submission, we did not directly translate monolingual English data into Basque. Since there is high-quality
Basque–Spanish parallel data not available for Basque–English (Open Data Euskadi Repository) we opted for translating the Spanish side of Spanish–English parallel data into
Basque in order to build additional Basque–English training material. A similar approach has been successfully applied for enhancing phrase-based statistical machine translation systems [20].
In order to carry out the backtranslation, we trained an
NMT system on all the available Spanish–Basque data withthe set-up described in Section 4. Words were segmented
as described in Section 3. That system was used to backtranslate the Spanish side of the in-domain Spanish–English
training data and the top 5M segments15 from the pseudo indomain Spanish–English corpus.
We evaluated the impact of adding backtranslated data
to the best dataset from the previous section (2M pseudo indomain parallel segments, oversampling and no finetuning).
We built NMT systems after adding the full backtranslated
data (both the in-domain and the pseudo-in-domain data; rowlabeled as 5.2M), and after adding the in-domain and only
2M pseudo-in-domain backtranslated segments (row labeled
15 We could not backtranslate a larger amount of data because of time

restrictions.

Our submission is built with the help of morphological segmentation, cross-entropy data selection and cross-lingual
learning via multilingual NMT. This section reviews the most
relevant approaches in these three fields.
Morphological segmentation has been successfully applied to build a winning system [2] for the English–Finnish
language pair in the WMT 2016 news translation sharedtask [22]. Simpler alternatives based on lists of prefixes/suffixes have also been reported to bring improvements
in translation quality [8]. Morphological segmentation has
already been applied to NMT for Basque [23]. However, unlike our approach, their strategy segments homograph words
in a greedy way (longest stem). Besides morphological segmentation, there are other ways linguistic resources can beused to segment words for NMT training. For instance, TL
words can be transformed into a sequence of stem and morphological inflection tags in order to achieve better morphological generalization when translating into highly inflectedlanguages [3].
Cross-entropy data selection [17] has become a popular
approach for leveraging out-of-domain data when building
MT systems. This strategy has been used for collecting training data for phrase-based statistical machine translation systems [24] and NMT systems [18] in shared translation tasks
such as WMT [6] and IWSLT [25].
In multilingual NMT [1], a single NMT model is used
to translate between different language pairs. Some authors
proposed multilingual NMT strategies in which the underly-

100
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

ing network architecture does not need to be modified [1, 26].
That property allowed us to perform multilingual MT with aTransformer [14] model despite the fact that the multilingual
NMT approach we followed [1] was originally addressed tothe encoder-decoder with attention architecture [27]. On thecontrary, other authors [28] proposed modifying the networkarchitecture to use an independent encoder and decoder foreach language.

7. Concluding remarks
This paper presented Prompsit Language Engineering’s submission to the IWSLT 2018 Low Resource MT track. We
presented a novel method for morphological segmentation
based solely on a dictionary with inflection paradigms suchas those used by spell checkers and proved that it outperforms the widespread BPE segmentation method. Our
submission relies on cross-lingual learning via multilingual
NMT. Basque training data was segmented with the novel
method. The NMT system follows the Transformer architecture. We experimented with varying amounts of pseudo
in-domain data obtained via cross-entropy data selection andwith varying amounts of backtranslated data and submitted
the combination that maximized translation quality on thedevelopment set.
Our submission could be further improved with independent ensembles [21, Sec. 3.2]. The inclusion of additional
language pairs has not been exhaustively evaluated and thequality of the final system might be improved by adding
some more language pairs. The quality of the final system
could also improve with the addition of more backtranslated
data.

8. Acknowledgements
We would like to thank Prof. Mikel L. Forcada for the advice on Basque segmentation. Work supported by project
IADAATPA, action number 2016-EU-IA-0132, funded under the Automated Translation CEF Telecom instrument
managed by INEA at the European Commission.

9. References[1] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu,
Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, et al., “Google’s multilingual neural machinetranslation system: Enabling zero-shot translation,”Transactions of the Association of Computational Linguistics, vol. 5, no. 1, pp. 339–351, 2017.
[2] V. M. Sánchez-Cartagena and A. Toral, “Abu-matran
at WMT 2016 translation task: Deep learning, morphological segmentation and tuning on character sequences,” in Proceedings of the First Conference on
Machine Translation: Volume 2, Shared Task Papers,
vol. 2, 2016, pp. 362–370.

[3] A. Tamchyna, M. Weller-Di Marco, and A. Fraser,“Modeling target-side inflection in neural machinetranslation,” in Proceedings of the Second Conference
on Machine Translation. Copenhagen, Denmark:
Association for Computational Linguistics, September
2017, pp. 32–42. [Online]. Available: http://www.
aclweb.org/anthology/W17-4704[4] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers), vol. 1, 2016, pp. 1715–1725.
[5] J. Tiedemann, “Parallel Data, Tools and Interfacesin OPUS,” in Proceedings of the Eight International
Conference on Language Resources and Evaluation(LREC’12), N. C. C. Chair), K. Choukri, T. Declerck,
M. U. Dogan, B. Maegaard, J. Mariani, J. Odijk, and
S. Piperidis, Eds. Istanbul, Turkey: European Language Resources Association (ELRA), may 2012.
[6] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham,
B. Haddow, S. Huang, M. Huck, P. Koehn, Q. Liu,
V. Logacheva, et al., “Findings of the 2017 conferenceon machine translation (WMT17),” in Proceedings of
the Second Conference on Machine Translation, 2017,
pp. 169–214.
[7] V. M. Sánchez-Cartagena, M. Bañón, S. Ortiz-Rojas,
and G. Ramı́rez-Sánchez, “Prompsit’s submission to
WMT 2018 Parallel Corpus Filtering shared task,”in Proceedings of the Third Conference on MachineTranslation, Volume 2: Shared Task Papers. Brussels,
Belgium: Association for Computational Linguistics,
October 2018.
[8] M. Huck, S. Riess, and A. Fraser, “Target-side word
segmentation strategies for neural machine translation,”in Proceedings of the Second Conference on MachineTranslation, 2017, pp. 56–67.
[9] S. Virpioja, P. Smit, S.-A. Grönroos, and M. Kurimo,“Morfessor 2.0: Python implementation and extensions for morfessor baseline, D4 Julkaistu kehittmis- tai
tutkimusraportti tai -selvitys,” 2013. [Online]. Available: http://urn.fi/URN:ISBN:978-952-60-5501-5[10] J. Rissanen, “Modeling by shortest data description,”
Automatica, vol. 14, no. 5, pp. 465 – 471, 1978.
[Online]. Available: http://www.sciencedirect.com/
science/article/pii/0005109878900055[11] O. Kohonen, S. Virpioja, and K. Lagus, “Semisupervised learning of concatenative morphology,” inProceedings of the 11th Meeting of the ACL Special
Interest Group on Computational Morphology andPhonology, ser. SIGMORPHON ’10. Stroudsburg,

101
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

PA, USA: Association for Computational Linguistics,
2010, pp. 78–86. [Online]. Available: http://dl.acm.
org/citation.cfm?id=1870478.1870488[12] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,
“BLEU: a method for automatic evaluation of machinetranslation,” in Proceedings of the 40th annual meetingon association for computational linguistics. Association for Computational Linguistics, 2002, pp. 311–318.
[13] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, andJ. Makhoul, “A study of translation edit rate with targeted human annotation,” in Proceedings of association
for machine translation in the Americas.
[14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and
I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems
30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett,
Eds. Curran Associates, Inc., 2017, pp. 5998–
6008. [Online]. Available: http://papers.nips.cc/paper/
7181-attention-is-all-you-need.pdf[15] M. Junczys-Dowmunt, R. Grundkiewicz, T. Dwojak,
H. Hoang, K. Heafield, T. Neckermann, F. Seide,
U. Germann, A. Fikri Aji, N. Bogoychev, A. F. T.
Martins, and A. Birch, “Marian: Fast neuralmachine translation in C++,” in Proceedings of
ACL 2018, System Demonstrations. Melbourne,
Australia: Association for Computational Linguistics,
July 2018, pp. 116–121. [Online]. Available: http://www.aclweb.org/anthology/P18-4020[16] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. Rush,“Opennmt: Open-source toolkit for neural machinetranslation,” Proceedings of ACL 2017, System Demonstrations, pp. 67–72, 2017.
[17] A. Axelrod, X. He, and J. Gao, “Domain adaptation
via pseudo in-domain data selection,” in Proceedingsof the Conference on Empirical Methods in Natural
Language Processing, ser. EMNLP ’11. Stroudsburg,
PA, USA: Association for Computational Linguistics,
2011, pp. 355–362. [Online]. Available: http://dl.acm.
org/citation.cfm?id=2145432.2145474[18] M. Junczys-Dowmunt and A. Birch, “The Universityof Edinburgh’s systems submission to the MT task at
IWSLT,” in Proceedings of IWSLT 2016, 2016.
[19] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” in Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), vol. 1, 2016, pp. 86–96.

[20] M. Huck and H. Ney, “Pivot lightly-supervised training
for statistical machine translation,” in Proc. 10th Conf.
of the Association for Machine Translation in the Americas, 2012, pp. 50–57.
[21] R. Sennrich, A. Birch, A. Currey, U. Germann, B. Haddow, K. Heafield, A. V. M. Barone, and P. Williams,“The University of Edinburgh’s Neural MT Systems for
WMT17,” in Proceedings of the Second Conference on
Machine Translation, 2017, pp. 389–399.
[22] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham,
B. Haddow, M. Huck, A. J. Yepes, P. Koehn, V. Logacheva, C. Monz, et al., “Findings of the 2016 conference on machine translation,” in Proceedings of theFirst Conference on Machine Translation: Volume 2,
Shared Task Papers, vol. 2, 2016, pp. 131–198.
[23] T. Etchegoyhen, E. Martı́nez Garcia, A. Azpeitia,
G. Labaka, I. Alegria, I. Cortes Etxabe, A. Jauregi Carrera, I. Ellakuria Santos, M. Martin, and E. Calonge,“Neural Machine Translation of Basque,” in Proceedings of the 21st Annual Conference of the European
Association for Machine Translation, Alacant, Spain,
2018, pp. 139–148.
[24] R. Rubino, A. Toral, V. M. Sánchez-Cartagena,
J. Ferrández-Tordera, S. O. Rojas, G. Ramı́rezSánchez, F. Sánchez-Martı́nez, and A. Way, “AbuMaTran at WMT 2014 translation task: Two-step data
selection and RBMT-style synthetic rules,” in Proceedings of the ninth workshop on statistical machine translation, 2014, pp. 171–177.
[25] C. Mauro, F. Marcello, B. Luisa, N. Jan, S. Sebastian,S. Katsuitho, Y. Koichiro, and F. Christian, “Overview
of the IWSLT 2017 Evaluation Campaign,” in International Workshop on Spoken Language Translation,
2017, pp. 2–14.
[26] T.-L. Ha, J. Niehues, and A. Waibel, “Toward multilingual neural machine translation with universal encoderand decoder,” in Proceedings of 2016 International
Workshop on Spoken Language Translation, 2016.
[27] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi,
W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey,
J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser,
S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens,
G. Kurian, N. Patil, W. Wang, C. Young, J. Smith,
J. Riesa, A. Rudnick, O. Vinyals, G. Corrado,
M. Hughes, and J. Dean, “Google’s neural machinetranslation system: Bridging the gap between humanand machine translation,” CoRR, vol. abs/1609.08144,
2016. [Online]. Available: http://arxiv.org/abs/1609.
08144

102
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[28] O. Firat, K. Cho, and Y. Bengio, “Multi-way, multilingual neural machine translation with a shared attention mechanism,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, 2016, pp. 866–875.

103
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Neural Speech Translation at AppTek
Evgeny Matusov, Patrick Wilken, Parnia Bahar∗ , Julian Schamper∗ , Pavel Golik, Albert Zeyer∗ ,Joan Albert Silvestre-Cerdà+ , Adrià Martı́nez-Villaronga+ , Hendrik Pesch, and Jan-Thorsten Peter
Applications Technology (AppTek), Aachen, Germany{ematusov,pwilken,pbahar,jschamper,pgolik,jsilvestre,amartinez,hpesch,jtpeter}@apptek.com∗ Also RWTH Aachen University, Germany + Also Universitat Politècnica de València, Spain

Abstract
This work describes AppTek’s speech translation pipelinethat includes strong state-of-the-art automatic speech recognition (ASR) and neural machine translation (NMT) components. We show how these components can be tightly
coupled by encoding ASR confusion networks, as well as
ASR-like noise adaptation, vocabulary normalization, andimplicit punctuation prediction during translation. In anotherexperimental setup, we propose a direct speech translation
approach that can be scaled to translation tasks with large
amounts of text-only parallel training data but a limited number of hours of recorded and human-translated speech.

1. Introduction
AppTek participated in the evaluation campaign of theInternational Workshop on Spoken Language Translation(IWSLT) 2018 with the goal of obtaining best possible
speech translation quality by streamlining the interface between ASR and machine translation (MT). We tested a new
way of encoding multiple hypotheses of ASR as input to an
NMT system. We also experimented with a novel direct neural translation model that translates source language speech
into target language text, while at the same time benefitingfrom text-only parallel training data in a multi-task learning
framework. To make these experiments possible, we madesure that our NMT system can handle different types of input,
and its source language vocabulary is harmonized w.r.t. the
ASR system vocabulary. We also fine-tuned the NMT modelon ASR-like noise, making it more robust against recognition
errors. Finally, we tested different punctuation prediction approaches and found that the implicit prediction of punctuation marks by the MT component works best in our setting.
Although improving our state-of-the-art NMT model was
not our main focus, we benefited from fine-tuning the modelon the in-domain data, as well as from ensembles of models which differ in architecture – recurrent neural network(RNN) model with attention [1] or transformer architecture [2] – and/or input modality – ASR confusion network(CN) or first-best ASR output. This paper is organized asfollows. We start by reviewing related work in 2, pointing out some differences and novelties in our approach. In
Section 3, we describe our methods for data filtering, pre-processing, and punctuation prediction. Section 4 gives an
overview of our ASR system. Section 5 describes the details
of AppTek’s NMT system. Section 6 gives details of how
ASR confusion networks can be encoded as an input of the
NMT system. In Section 7, we describe our direct speechtranslation prototype. The results of our speech translation
experiments are summarized in Section 8.

2. Related Work
Theoretical background for tighter coupling of statistical
ASR and MT systems had been first published in [3]. In practice, it was realized e. g. as statistical phrase-based translation of ASR word lattices with acoustic and language modelscores [4] or confusion networks with posterior probabilities [5]. In both cases moderate improvements of translation
quality were reported when the ASR scores were includedin the log-linear model combination; the improvements werelarger when the baseline recognition quality was low.
In the first publication on word lattice translation using
a neural model [6], the proposed lattice-to-sequence model
had an encoder component with one hidden state for eachlattice node, as well as attention over all lattice nodes. Thisis a different and more computationally expensive model ascompared to what we propose in this work. In our encoder,
the number of hidden states is the same as the number of slots
in the input confusion network, which is usually only slightly
higher than the number of words in the utterance.
Adapting the NMT system to ASR-like noise was proposed by [7]. We follow the same strategy, but the noise that
we introduce is not random; it is sampled from a distributionof most common ASR errors based on statistics from recognizing the audio of the TED dataset.
Direct translation of foreign speech was proposed by [8],who used a character-level sequence-to-sequence model1 .
They report experimental results on a small (163 hoursof speech with transcriptions and translations) Spanish-toEnglish Fisher and Callhome dataset. The authors use multitask learning with a single speech encoder and two decoders,
one for English (direct translation) and one for Spanish,
which allows them to incorporate supervision from Spanish
1 A later work by [9] extends the approach of [8] to word-level models.

104
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018transcripts. In contrast, we follow the opposite approach, inwhich we have a single target language decoder and two separate encoders, one for source language speech, and one forsource language text. This approach allows us to benefit fromlarge quantities of text-only parallel MT training data, in a
multi-task learning scenario, and thus, in contrast to previous work, to potentially compete with the standard approachthat uses strong, but separate components for ASR and MT.
Punctuation prediction in MT (and especially neural MT)
context was investigated in comparative experiments in [10].
Similarly to that paper, we also confirmed experimentallythat implicit prediction of punctuation marks by the NMT
system resulted in the best BLEU and TER scores in our setting (see Section 3.3 for details).

3. Data Preparation
3.1. Parallel Data Filtering
In line with the evaluation specifications, we used the TED
corpus, the OpenSubtitles2018 corpus [11], as well as thedata provided by the WMT 2018 evaluation (Europarl,
ParaCrawl, CommonCrawl, News Commentary, and Rapid)
as the potential training data for our NMT system, amounting to 65M lines of parallel sentence-aligned text. We thenfiltered these data based on several heuristics, with the twomost important ones described next.
Since especially the crawled corpora are very noisy, theyoften contain segments in a wrong language, or even things
like programming code and XML markup. We used the
CLD2 library2 for sentence-level language identification tokeep only those sentence pairs in which the source sentence
was labeled as English and target sentence as German withthe confidence of at least 90%.
Another heuristic was based on sentence length: we only
kept sentences with at least 3 and at most 80 words (after tokenization). We also removed sentence pairs in which source
and target sentence lengths differ by a factor of 5 or more.
Overall the filtering yielded a corpus of 37.6M lines and556M words (on the English side, counted untokenized),
which we used in all of the experiments presented in thispaper. It included 256K unique lines of TED talks with 4.4M
words on the English side.
3.2. Preprocessing
We used two types of preprocessing. The first one was thestandard Moses tokenization [12] for text translation andlowercasing on the English side. The German side was truecased using a frequency-based method. The second preprocessing was used only for English with the goal of converting
text into speech transcript similar to the one produced by the
ASR system. Starting from the Moses tokenization, we removed all punctuation marks and spliced back contractions(e. g. do n’t → don’t) to match the corresponding to2 https://github.com/CLD2Owners/cld2

kens in the ASR lexicon. We also converted numbers writtenwith digits to their spoken form using a tool based on the
num2words3 python library.
The final step for both types of preprocessing wassegmentation into sub-word units with byte pair encoding(BPE) [13], separately for each language. We used 20K
merging operations. During testing, we used the option torevert BPE merge operations resulting in tokens that wereobserved less than 50 times in the segmented training data.
3.3. Punctuation Prediction
To translate a speech transcript with an NMT system trainedwith the first, standard preprocessing described above, weneed to automatically enrich it with punctuation marks. To
this end, we trained a RNN for punctuation restoration similar to the one presented in [14]. Only the words in a sentence
are used to predict punctuation marks (period, comma, andquestion mark only). The acoustic features are not used.
For the setup with the ASR-like preprocessing of English, punctuation prediction is done implicitly during translation, since the target side of the training corpus contains
punctuation marks. Thus, the output of the ASR system canbe directly used (after BPE) as input to the NMT system.

4. ASR system
The ASR system is based on a hybrid LSTM/HMM acoustic model [15, 16], trained on a total of approx. 390 hoursof transcribed speech from the TED-LIUM corpus (excluding the black-listed talks) and the IWSLT Speech-Translation
TED corpus4 . We used the pronunciation lexicon provided
with the TED-LIUM corpus. The acoustic model takes 80dim. MFCC features as input and estimates state posterior
probabilities for 5000 tied triphone states. It consists of 4 bidirectional layers with 512 LSTM units for each direction.
Frame-level alignment and state tying were obtained froma bootstrap model based on a Gaussian mixtures acoustic
model. We trained the neural network for 100 epochs by minimizing the cross-entropy using the Adam update rule [17]with Nesterov momentum and reducing the learning rate following a variant of the Newbob scheme.
The language model for the single-pass HMM decoding is a simple 4-gram count model trained with KneserNey smoothing on all allowed English text data (approx. 2.8B running words). The vocabulary consists of thesame 152k words from the training lexicon and the outof-vocabulary rate is 0.2% on TED.dev2010 and 0.5%
on TED.tst2015. The LM has a perplexity of 133 on
3 https://github.com/savoirfairelinux/num2words

4 We realized that the provided audio-to-source-sentence alignments of
the TED talks were often not correct. As this could significantly degrade theperformance of the audio encoder for the direct speech translation approachdescribed in Section 7, we had to automatically recompute these alignments
by force-aligning each TED recording to its corresponding source sentences,
and applied heuristics to overcome the problem of transcription gaps (speechsegments without a translation in the parallel data).

105
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

TED.dev2010 and 122 on TED.tst2015.
Since TED talks are a relatively simple ASR task, we
decided not to proceed with sequence training of the acoustic model or LM rescoring with LSTM models in order tohave more uncertainty in the lattices. Acoustic training of
the baseline model and the HMM decoding were performed
with the RWTH ASR toolkit [18]. We trained BLSTM models with RETURNN [19], which integrates into RWTH ASR
as an external acoustic model for decoding. Prior to constructing CNs from lattices [20], we decomposed the words
into individual arcs according to the BPE scheme describedin Section 3.2. The construction algorithm uses arcs from thefirst-best path as pivot elements to initialize arc clusters [21].

5. Neural Machine Translation System
We used the RETURNN toolkit [22] based on TensorFlow[23] for all NMT experiments. We trained two different architectures of NMT models: an attention-based RNN model
similar to [1] with additive attention and a Transformer
model [2] with multi-head attention.
In the RNN-based attention model, both the source andthe target words are projected into a 620-dimensional embedding space. The models are equipped with either 4 or 6
layers of bidirectional encoder using LSTM cells with 1000
units. A unidirectional decoder with the same number ofunits was used in all cases. We applied a layer-wise pretraining scheme that lead to both better convergence andfaster training speed during the initial pre-train epochs [22].
We also augmented our attention computations using fertility
feedback similar to [24, 25].
In the Transformer model, both the self-attentive encoder and the decoder consist of 6 stacked layers. Everylayer is composed of two sub-layers: a 8-head self-attention
layer followed by a rectified linear unit (ReLU). We applied layer normalization [26] before each sub-layer, whereas
dropout [27] and residual connection [28] were applied afterwards. Our model is very similar to “base” Transformer ofthe original paper [2], such that all projection layers and themulti-head attention layers consist of 512 nodes followed bya feedforward layer equipped with 2048 nodes.
We trained all models using the Adam optimizer [17]with a learning rate of 0.001 for the attention RNN-basedmodel and 0.0003 for the Transformer model. We applied alearning rate scheduling similar to the Newbob scheme basedon the perplexity on the validation set for a few consecutive
evaluation checkpoints. We also employed label smoothingof 0.1 [29] for all trainings. The dropout rate ranged from
0.1 to 0.3.

6. Translation of ASR Confusion Networks
To encode confusion networks as input to the NMT system,
we propose a novel, simple scheme. For a given speech utterance represented by acoustic vectors o, we treat a confusion
network C with J slots as the source sentence for the NMT.

Instead of the one-hot encoding x j ∈ {0, 1}K (where K is thesource vocabulary size) at position j within the sentence, theinput is encoded as a K-dimensional vector x̄ j ∈ RK withx̄kj := p j (wk |o), k = 1, . . . , K. Here, wk is the k-th word in
the vocabulary, and p j (wk |o) is the posterior probability of
the word wk to appear at position j in C. In practice, p j (wk )is different from 0 only for a small number of words.
In the end, following the notation of [1], we represent the
input to the RNN encoder as the vector E x̄ j where E ∈ RN×K
is the word embedding matrix and N is the dimension of theword embedding (e. g. 620). Thus, E x̄ j is a weighted combination of word embeddings for all the words in the CN slotj, with the highest weight given to the word with the highest
posterior probability. In the corner case of only one arc per
slot with the posterior probability of 1.0, we obtain a single
word sequence. Thus, we can still use normal sentence pairs(e.g. from text-only parallel data) for training, along with thepairs of source CNs and their target language translations.
The new input representation E x̄ j has the same dimensions
as Ex j and thus can be directly used to train a standard RNN
NMT model or any other model that uses word embeddings.
We kept the posterior weights fixed during back-propagation.
Word sequences of different length can be obtained froma CN because epsilon arcs can be inserted as alternatives
in some of the slots. The best solution when training an
NMT system on CNs would be to add an artificial source
language token EPS that would not appear in the originaltext-only training data. However, because we decided against
re-training the system on CN input from scratch, we mapped
all epsilon arcs to the English word “eh”, which denotes hesitation. It appears often enough in the English side of theparallel text-only corpus, but is almost always omitted in thehuman translation into German.
We also used CNs to simulate ASR word errors in text
data. Following the work of [7], we used such noisy datain the training of the NMT system to make it more robust
against similar real ASR word errors. To this end, for eachword w in the first-best ASR output for the TED training corpus, we collected all the slot alternatives w′n , n = 1, . . . , Nw to
this word in the corresponding ASR CNs with their averaged
posterior probabilities. After re-normalization of these probabilities, for each word w we obtained a confusion probability distribution pw . Then, in a given sentence, we replaced
every occurrence of the word w by one of its alternatives w′nwith probability pw (w′n ) from this distribution. One of thealternatives can also be an epsilon arc, we keep them (converted to “eh” as described above) to adapt the NMT system
to epsilon arcs in CN input, inserting up to 2 consecutive arcs
after each word with a probability e.
Finally, we used two control parameters to limit the noiselevel: probability to change a word p and probability tochange anything at all in a given sentence s. Experimentally,
we determined the settings e = 0.02, p = 0.25, s = 0.6 which
resulted in WER of the noisy text as compared to its originaltext that was similar to the WER of the baseline ASR system.

106
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 1: Results measured in B LEU [%] and T ER [%] for the individual systems for the English→German speech translationtask, translation of correct transcript vs. first-best ASR output of the TED.tst2015 set.
correct transcript
ASR output (WER of 10.9%)#
System
B LEU
T ER
B LEU
T ER
0
RWTH IWSLT 2017 best non-ensemble system
30.5
52.3
–
–
1
text translation baseline (RNN)
32.4
50.5
25.2
60.2
2
text translation baseline (Transformer)
33.0
50.5
26.3
58.7
3
speech translation baseline (RNN)
31.4
51.9
26.6
60.0
4
speech translation baseline (Transformer)
30.7
52.8
25.8
59.5

7. Direct Speech Translation
In the direct approach to speech translation, a single neuralnetwork is used to predict the target translation given the audio features of the source sentence. The amount of training
data for this setting, i.e. audio with the corresponding reference translations pairs from the TED corpus, is comparatively low. To exploit the much larger parallel text corpora,we choose a multi-task setup in which the network simultaneously learns to translate either from source audio or fromsource text. For this, we extend the RNN-based attention
model described in Section 5 with an additional audio encoder that takes MFCC features as input. It consists of 5 bidirectional LSTM layers with 512 units each. Max-poolinglayers with a pool size of 2 are inserted after each of the first 3
LSTM layers, reducing the sequence length by a factor of 8.
Also, a separate attention mechanism is added for the audio
encoder. The decoder switches between the context vector
from the text encoder ci,text and the one from the audio encoder ci,audio depending on which input is given (using notation from [1]). The remaining part of the decoder is shared
between both tasks.
To ensure that both types of input are seen frequently
enough during training, we duplicate the speech translation
corpus so that it grows to 30% the size of the parallel text
corpus (66 duplicates). The concatenation of text and audio
examples is then traversed in random order. For the direct
system, the same optimization and regularization techniques
are applied as in the NMT system described in Section 5.

8. Experimental Evaluation
We participated in the speech translation task of the
IWSLT 2018 evaluation, the translation direction was
English→German. All NMT models are trained on the filtered bilingual data as described in Section 3.1, no monolingual data was used. For the fine-tuning experiments, weused the TED talk part of the bilingual data together withthe test sets TED.tst2010,2013,2014 (which were notused for tuning or evaluation). The TED talk part was also
included in the baseline system. For the experiments withthe confusion networks, we ran the ASR system to recognize
the speech of the 170K TED training set and the test sets
TED.tst2010,2013,2014 and used the resulting CNs

with the corresponding German translations as (additional)
training data.
We shuffled the training samples before each epoch andremoved sentences longer than 75 and 100 sub-words inthe attention RNN-based and the Transformer setup, respectively. We evaluate our models almost every 10K iterations
and select the best checkpoint based on perplexity on the validation set. NMT decoding is performed using beam searchwith a beam size of 12 and the scores are normalized w.r.t thelength of the hypotheses. We used TED.dev2010 consisting of 888 sentences as our validation set and evaluated ourmodels on TED.tst2015 test set with 1080 segments. The
systems were evaluated using case-sensitive B LEU [30] andnormalized case-sensitive T ER [31].
8.1. Baselines
First we trained a model with standard preprocessing for
written text described in Section 3.2 and evaluated its quality on the correct transcript with punctuation marks of the
TED.tst2015 set, as shown in Table 1. We observed aslightly better BLEU score for the Transformer architecture(line 2) as compared to the recurrent architecture (line 1). We
also made a comparison to the best single system of RWTH
Aachen University on this set from the IWSLT 2017 evaluation. With our baseline system we improved upon that result
by 1.9% to 2.5% absolute.
We then trained a model with speech-like preprocessingof the English side of the parallel corpus as described in Section 3.2. This model not only translates English words toGerman, but also predicts punctuation marks. To match thiscondition, we applied the same preprocessing to the correct
English transcript of TED.tst2015, removing the punctuation marks. The evaluation included punctuation marks.
Because of the dual task (translation and punctuation prediction), the MT quality is lower, but only by 1% BLEU (line 3of Table 1). Here, the recurrent architecture outperforms thetransformer architecture (line 4) by a significant margin. Because of this, most of our subsequent experiments were basedon the recurrent model.
8.2. Effects of ASR errors and Punctuation Prediction
When we translate the first-best ASR output for
TED.tst2015, which has an ASR word error rate of107
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 2: Results measured in B LEU [%] and T ER [%] for the individual systems for the English→German speech translationtask, translation of ASR output.
TED.dev2010
TED.tst2015
tst2018
B LEU
T ER
B LEU
T ER
B LEU
T ER
#
System
1
2
3
4
5
6
7
8
9
10
11
12

speech translation baseline (RNN)+ fine-tuning on TED corpus+ 2 additional encoder layers+ fine-tuning on TED with noise
fine-tuning of 1) on TED CNs only
fine-tuning of 1) on TED correct + CNs
fine-tuning of 1) on TED correct+noise + CNs
speech translation baseline (Transformer)+ fine-tuning on TED corpus
Ensemble of 2, 3, 4, 9
Ensemble of 2, 3, 4, 6
Ensemble of 2, 3, 4, 5, 6, 7

26.5
27.1
27.3
27.1
26.6
26.6
26.2
26.1
27.0
27.9
27.3
27.5

10.9%, we observe a significant degradation of MT quality.
For example, the BLEU score goes down from 31.4% to26.6%, cf. line 3 of Table 1. This means that the NMT
system is sensitive to ASR errors. Otherwise, the differences
between architectures are similar when compared on the
ASR first-best output as opposed to correct transcript.
8.3. Confusion Network Translation
For the subsequent experiments, we start with the RNN
speech translation baseline. Table 2 presents the results on
the ASR output for the TED.dev2010 validation set andTED.tst2015 test set. For the lines where confusion networks are mentioned, they were used as input to the NMT
system as described in Section 6. The CNs were pruned
based on the threshold of 0.0001 for the posterior probability;
a maximum of 20 arcs per slot with highest probability were
kept. The average density of the final CNs on the training
and validation sets was 1.8 and 2.2, respectively.
Fine-tuning on the TED corpus (using the correct transcript as the English side of the parallel corpus) improves theresult on the test set by 0.9% BLEU absolute, as shown inline 2 of Table 2. We fine-tuned our models with a small
learning rate of 0.00001 for the Transformer model and themodels using CNs, which is additionally decayed by a factor
ranging from 0.8 to 0.9 after each half an epoch. For the attention RNN-based models which do not use CNs as input,
the learning rate was set to 0.0001 with decay rate of 0.9. We
also tried fine-tuning using a model with 6 encoder layers instead of 4, but have not obtained any further improvements
as compared to the fine-tuned model with 4 encoder layers.
Next, we duplicated the TED parallel corpus and introduced noise into the duplicate. The level of noise was selected to be similar to the ASR word error rate on the development set, and the noise itself was created as described in
Section 6. Line 4 of Table 2 shows that after fine-tuning on
both correct and noisy TED corpus, we obtain an improvement of 0.5% BLEU and 1.0% TER when translating the

55.2
54.2
54.7
54.1
55.7
55.5
55.9
55.6
54.4
53.7
55.6
54.2

26.6
27.5
27.6
28.0
26.9
27.0
27.0
25.8
27.0
28.3
28.0
28.3

60.0
57.5
57.5
56.5
58.3
58.5
57.9
59.5
57.7
56.7
58.1
56.7

–
–
–
21.1
–
20.3
20.2
–
–
21.4
21.2
21.5

–
–
–
64.1
–
66.5
66.7
–
–
64.2
64.4
64.1

first-best ASR output for the TED.tst2015 set, as compared to fine-tuning without the noise (line 2). Thus, to someextent the NMT system was able to learn how to cope withnoise that is based on common ASR errors.
Because we could only run ASR on the 170K sentencesfrom the TED corpus, for which the speech was well-alignedwith reference translations, we decided to use confusion networks for fine-tuning of the NMT system only. To make itpossible, we replaced the original English word embeddingsof the model with the linear combination of the embeddingsof CN slot alternatives, as described in Section 6. The finetuning was done on a random mix of correct TED talk transcripts and ASR CNs for these transcripts at the same time.
Lines 5-7 of Table 2 list the results of three fine-tuning
experiments which include CNs in the source-side training
data. We either continued training of the model on 170K CNs(and the reference translations of the corresponding transcripts), or on CNs plus correct transcripts of the same set,or on CNs plus correct transcripts and transcripts with noisethat was inserted as in the experiment in line 4 of Table 2.
In all three cases we used a lower learning rate, a smaller
batch size, and continued fine-tuning for 5-6 epochs. Unfortunately, the BLEU/TER scores go down as compared to thebest result in line 4 when translating first-best ASR output.
Detailed analysis of the NMT output from line 6 (bestresult on the validation set) showed that when translating
confusion networks, the model is able to recover from somerecognition errors. For example, the TED.tst2015 utterance you throw the ball but you’re hit rightas you throw is translated by the system in line 4 of
Table 2 as Sie werfen den Ball, aber das ist IhrThron because of the ASR error as you throw → isyour throne. The system that translates the corresponding ASR confusion network, however, is able to produce
a correct translation: Sie werfen den Ball, aber Sie
werden getroffen. In another anecdotal example there isno error in the first-best ASR output, but the NMT system is

108
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

not able to disambiguate the meaning of the word picking.
The English source is: see over there somebody iskind of picking their nose. The translation of thefirst-best ASR output is: Da drüben, da ist jemandin der Nase (Over there, someone is in the nose). Whenthe corresponding CN is translated, the translation preserves
the original meaning: Sehen Sie, da ist jemand,
der sich in die Nase bohrt. We looked at the CN
alternatives for the word picking. It had a posterior probability of 0.77, and the top competing hypotheses weretaking (0.14), making (0.06), shaking (0.02), sticking(0.004). The embeddings of these words seem to have helped
the NMT system to correctly infer the meaning of picking
in the given context.
In many cases, however, multiple alternatives, especiallyto an empty slot, sometimes confused the system to a point
where translation quality was adversely affected. In a final
contrastive experiment, we used the system from line 7 inTable 2 to translate not the CNs, but the first-best ASR output for the same test set. The result – BLEU of 28.0% andTER of 56.5% on TED.tst2015 – was nearly identical toline 4 in the table and showed that the system was fine-tuned
well to the TED domain, and the noise in the CNs made the
NMT system more robust against the errors which the ASR
system could not avoid to make. However, the system does
not generalize well to unseen CNs and may make errors byencoding meaning from alternatives to correctly recognizedwords, even if their posterior probability is low. Nevertheless, we think that the approach is promising and can benefit
from a better training strategy, more CNs in training, and abetter, adaptive weighting of CN alternatives. We plan to implement these improvements in our future work.
8.4. Direct Speech Translation
The direct translation system trained on the 170K segmentsof the TED corpus where both the audio files and their translations are available and well aligned yields a BLEU score of
15.6% when translating the speech of the TED.tst2015
set. For comparison, a standard text-only attention RNN
model trained on the same corpus using the correct English
transcript reaches the BLEU/TER scores of 18.5% on thefirst-ASR output for the same set. Although the results aremuch worse than for the NMT systems in Table 2 trained onlarge amounts of data, we see that the direct system can still
produce results only moderately worse than a system trainedon the same data, but on English text instead of speech.
When we try to improve the direct system by multi-task
learning using all of the text parallel data as described in Section 7, we obtain a BLEU score of 17.1% after many days of
training that has not converged, neither until the evaluation
nor paper submission deadline. Thus, although in preliminary tests multi-task learning seems to work correctly andbring improvements, the approach requires a faster implementation and a better training/optimization strategy.

8.5. Final Results
The Transformer model, even when fine-tuned on the TED
corpus, did not result in additional improvements over theattention based RNN model (lines 8 and 9 of Table 2). However, it contributed to the ensemble of several systems (line10). The RNN model that uses CN input can potentiallybe ensembled with the Transformer NMT model (using either first-best or CN ASR output as input). However, we did
not have time to implement the necessary changes for suchmodel combination. Therefore, for our primary evaluation
submission we combined only the RNN models which translate either first-best ASR input (models from lines 2, 3, 4 of
Table 2) or confusion networks (models from lines 5-7 of Table 2). The BLEU of this ensemble system in line 12 is only
marginally better than of the best single system from line 4.
For the 2018 evaluation data, we first used the acoustic
sentence segmentation of the ASR system. Because it was
too fine-grained and unreliable due to many pauses of the
speakers and could lead to context loss for the NMT system, we ran the punctuation prediction algorithm describedin Section 3.3 on the first-best ASR output, but used its results only to define new segment boundaries at time points
when a period or question mark was predicted after a word.
We then re-ran the recognition to generate first-best and CN
ASR output using the segmentation obtained in this way. The
performance on the 2018 evaluation data is reported in thelast column of Table 2. The BLEU and TER scores were
provided by the organizers for our primary and contrastive
submissions. We observed similar tendencies here as on the
TED.tst2015 set: unfortunately, using ASR confusion
networks as input to NMT results in worse scores (e.g. by
0.8% absolute in BLEU) as compared to the best single system translating ASR first-best output. Our primary submission from line 12 obtains the best results also on the 2018
evaluation set, but the improvement due to ensembling ofmultiple systems is not significant.

9. Conclusion
AppTek participated in the speech translation task of the
IWSLT 2018 evaluation, achieving the BLEU score of 21.5%
on the 2018 English to German evaluation data with the primary submission. Our best setup used an ensemble of attention RNN MT models which translate either first-best ASR
output or ASR confusion networks, generating target language text with punctuation marks. We proposed a novel
scheme for encoding CNs in NMT and showed that thenegative effect of some ASR errors can be reduced whenCNs are translated, although further improvements in training strategy are necessary to obtain significant improvementsin speech translation quality. Preliminary experiments withdirect speech translation with a single sequence-to-sequencemodel showed promising improvements due to a novel multitask learning scenario that allows for exploitation of text-onlyparallel MT training data.

109
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

10. References[1] D. Bahdanau, K. Cho, and Y. Bengio, “Neuralmachine translation by jointly learning to align andtranslate,” May 2015. [Online]. Available: http://arxiv.org/abs/1409.0473[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin,“Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
[3] H. Ney, “Speech translation: Coupling of recognition
and translation,” in IEEE International Conference on
Acoustics, Speech, and Signal Processing, Phoenix,
Arizona, USA, Mar. 1999, pp. 517–520.

[12] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin,
and E. Herbst, “Moses: Open source toolkit for
statistical machine translation,” in ACL 2007, Proceedings of the 45th Annual Meeting of the Association
for Computational Linguistics, June 23-30, 2007,
Prague, Czech Republic, 2007. [Online]. Available:
http://aclweb.org/anthology/P07-2045[13] R. Sennrich, B. Haddow, and A. Birch, “Neuralmachine translation of rare words with subword
units,” in Proceedings of the 54th Annual Meetingof the Association for Computational Linguistics,
ACL 2016, August 7-12, 2016, Berlin, Germany,
Volume 1: Long Papers, 2016. [Online]. Available:
http://aclweb.org/anthology/P/P16/P16-1162.pdf[4] E. Matusov, B. Hoffmeister, and H. Ney, “ASR wordlattice translation with exhaustive reordering is possible,” in Interspeech, Brisbane, Australia, Sept. 2008,
pp. 2342–2345.

[14] O. Tilk and T. Alumäe, “Bidirectional recurrent neural network with attention mechanism for punctuation
restoration.” in Interspeech, 2016, pp. 3047–3051.

[5] N. Bertoldi, R. Zens, and M. Federico, “Speech translation by confusion network decoding,” in IEEE International Conference on Acoustics, Speech, and Signal
Processing, Honululu, HI, USA, Apr. 2007, pp. 1297–
1300.

[15] H. Bourlard and C. J. Wellekens, “Links between
Markov models and multilayer perceptrons,” in Advances in Neural Information Processing Systems I,
D. Touretzky, Ed.
San Mateo, CA, USA: Morgan
Kaufmann, 1989, pp. 502–510.

[6] M. Sperber, G. Neubig, J. Niehues, and A. Waibel,“Neural lattice-to-sequence models for uncertain inputs,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017,
pp. 1380–1389.
[7] M. Sperber, J. Niehues, and A. Waibel, “Toward robustneural machine translation for noisy input sequences,”in International Workshop on Spoken Language Translation (IWSLT), 2017.
[8] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu,
and Z. Chen, “Sequence-to-sequence models candirectly translate foreign speech,” arXiv preprint
arXiv:1703.08581, 2017.
[9] S. Bansal, H. Kamper, K. Livescu, A. Lopez, and
S. Goldwater, “Low-resource speech-to-text translation,” arXiv preprint arXiv:1803.09164, 2018.
[10] V. Vandeghinste, C.-K. Leuven, J. Pelemans, L. Verwimp, and P. Wambacq, “A comparison of different
punctuation prediction approaches in a translation context,” in 21st Annual Conference of the European Association for Machine Translation, p. 269.
[11] P. Lison and J. Tiedemann, “Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles,” 2016.

[16] S. Hochreiter and J. Schmidhuber, “Long shortterm memory,” Neural Computation, vol. 9, no. 8,
pp. 1735–1780, Nov. 1997. [Online]. Available:
http://dx.doi.org/10.1162/neco.1997.9.8.1735[17] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” San Diego, CA, USA, May 2015.
[18] S. Wiesler, A. Richard, P. Golik, R. Schlüter, and
H. Ney, “RASR/NN: The RWTH neural network
toolkit for speech recognition,” in IEEE International
Conference on Acoustics, Speech, and Signal Processing, Florence, Italy, May 2014, pp. 3313–3317.
[19] P. Doetsch, A. Zeyer, P. Voigtlaender, I. Kulikov,
R. Schlüter, and H. Ney, “RETURNN: the RWTH extensible training framework for universal recurrent neural networks,” in IEEE International Conference on
Acoustics, Speech, and Signal Processing, New Orleans, LA, USA, Mar. 2017, pp. 5345–5349.
[20] L. Mangu, E. Brill, and A. Stolcke, “Finding Consensus
in Speech Recognition: Word Error Minimization andOther Applications of Confusion Networks,” Computer,
Speech and Language, vol. 14, no. 4, pp. 373–400, Oct.
2000.
[21] D. Hakkani-Tür and G. Riccardi, “A general algorithm
for word graph matrix decomposition,” in IEEE International Conference on Acoustics, Speech, and Signal
Processing, Hong Kong, Apr. 2003, pp. 596–599.

110
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[22] A. Zeyer, T. Alkhouli, and H. Ney, “RETURNN asa generic flexible neural toolkit with application totranslation and speech recognition,” in Proceedings of
ACL 2018, Melbourne, Australia, July 15-20, 2018,
System Demonstrations, 2018, pp. 128–133. [Online].
Available: https://aclanthology.info/papers/P18-4022/
p18-4022[23] M. Abadi, A. Agarwal, P. Barham, E. Brevdo,
Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean,
M. Devin, S. Ghemawat, I. Goodfellow, A. Harp,
G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser,
M. Kudlur, J. Levenberg, D. Mané, R. Monga,
S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens,B. Steiner, I. Sutskever, K. Talwar, P. Tucker,
V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals,
P. Warden, M. Wattenberg, M. Wicke, Y. Yu,
and X. Zheng, “TensorFlow: Large-scale machinelearning on heterogeneous systems,” 2015, software
available from tensorflow.org. [Online]. Available:
https://www.tensorflow.org/

Translation,” in Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics,
Philadelphia, Pennsylvania, USA, 2002, pp. 311–318.
[31] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, andJ. Makhoul, “A Study of Translation Edit Rate withTargeted Human Annotation,” in Proceedings of the
7th Conference of the Association for Machine Translation in the Americas, Cambridge, Massachusetts, USA,2006, pp. 223–231.

[24] Z. Tu, Z. Lu, Y. Liu, X. Liu, and H. Li,“Coverage-based neural machine translation,” CoRR,
vol. abs/1601.04811, 2016. [Online]. Available: http://arxiv.org/abs/1601.04811[25] P. Bahar, J. Rosendahl, N. Rossenbach, and H. Ney,
“The RWTH Aachen machine translation systems for
IWSLT 2017,” in 14th International Workshop on Spoken Language Translation, 2017, pp. 29–34.
[26] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016, version 1.
[27] N. Srivastava, G. E. Hinton, A. Krizhevsky,
I. Sutskever, and R. Salakhutdinov, “Dropout: asimple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15,
no. 1, pp. 1929–1958, 2014.
[28] K. He, X. Zhang, S. Ren, and J. Sun, “Deepresidual learning for image recognition,” in 2016
IEEE Conference on Computer Vision and Pattern
Recognition, CVPR 2016, Las Vegas, NV, USA, June
27-30, 2016, 2016, pp. 770–778. [Online]. Available:
https://doi.org/10.1109/CVPR.2016.90[29] G. Pereyra, G. Tucker, J. Chorowski, L. Kaiser,
and G. E. Hinton, “Regularizing neural networks
by penalizing confident output distributions,” CoRR,
vol. abs/1701.06548, 2017. [Online]. Available: http://arxiv.org/abs/1701.06548[30] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,
“Bleu: a Method for Automatic Evaluation of Machine

111
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The Sogou-TIIC Speech Translation System for IWSLT 2018
Yuguang Wang*, Liangliang Shi*, Linyu Wei*, Weifeng Zhu*, Jinkun Chen*
Zhichao Wang*, Shixue Wen*, Wei Chen*, Yanfeng Wang*, Jia Jia†*

Voice Interaction Technology Center, Sogou Inc., Beijing, China†{wangyuguang,shiliangliang}@sogou-inc.com

Tiangong Institute for Intelligent Computing, Tsinghua University, Beijing, China{jjia}@tsinghua.edu.cn

Abstract

2. Automatic speech recognition

This paper describes our speech translation system for the
IWSLT 2018 Speech Translation of lectures and TED talks
from English to German task. The pipeline approach is
employed in our work, which mainly includes the Automatic
Speech Recognition (ASR) system, a post-processing module,
and the Neural Machine Translation (NMT) system. Our ASR
system is an ensemble system of Deep-CNN, BLSTM, TDNN,
N-gram Language model with lattice rescoring. We report
average results on tst2013, tst2014, tst2015. Our best
combination system has an average WER of 6.73. The machinetranslation system is based on Google’s Transformer
architecture. We achieved an improvement of 3.6 BLEU overbaseline system by applying several techniques, such as
cleaning parallel corpus, fine tuning of single model, ensemble
models and re-scoring with additional features. Our final
average result on speech translation is 31.02 BLEU.

2.1. Audio Segmentation
In this evaluation, the test set is provided without manual
sentence segmentation, thus automatic segmentation of the final
test set is essential. We utilize an approach to automatic
segment audio data based on the signal energy. We set a
threshold to split the audio between 8 and 15 seconds and thenconcatenate utterances that are shorter than 8 seconds to its
neighboring utterances.
2.2. Audio Data Preparation and Feature Extraction
2.2.1.

Data Cleaning

Our acoustic data comes from two sources. The first is the TEDLIUM [5], which contains 340 hours of well transcribed data.
The second part comes from Speech-Translation TED corpus,
which is about 270 hours of data with some bad segments, e.g.
music or transcriptions not comparing the wav files. We follow
the way in kaldi toolkit [6] to do the cleaning1. This aimed to
cut the bad part off and only retrain the segments that can becompared with the transcripts. And we got about 220 hours inthis part.

1. Introduction
We have participated in the Speech Translation of lectures and
TED talks from English to German task. The goal of this task
is to translate fully un-segmented talks or lectures from English
to German.
A pipeline approach is employed in our work. It consists ofsegmentation of audio data, ASR system, punctuation
restoration and NMT system. A two pass decoding is used in
the ASR system. In the first pass, we use several different neuralnetwork, such as Deep-CNN [1] [2], BLSTM and TDNN [3] togenerate ensemble results. Then the decoding lattices of theensemble system are sent to a second pass decoder for lattice
rescoring. In order to bridge the gap between the output of ASR
system and training data of NMT system, punctuation
restoration, disfluency detection and inverse text normalization
are necessary in our pipeline. Our NMT system is based on theTransformer architecture [4], which is based solely on attention
mechanisms. Several techniques are adopted to improve oursystem, such as parallel corpus cleaning, fine tuning, model
ensembling and re-scoring with additional features.
The rest of this paper is structured as follows. Section 2
describes the details of our ASR system, and Section 3 describes
our NMT system. Our results in the speech translation task arepresented in Section 4. We conclude this paper in Section 5.

2.2.2.

Dereverbration

For speech dereverbration, we calculate the RT60 [7] of the
speech firstly. The speech whose RT60 is longer than 400ms isfiltered with the Kalman filtering algorithm [8] to dereverberate
the speech. Thus we get about 11,000 kalman filtered utterances
and add them to the original data.
2.2.3.

Speed Perturbation

Speed perturbation is done with 1.1 and 0.9 times for all the data
above. Finally we obtain about total 1700 hours acoustic data to
get robust performance in the end.
2.2.4.

Feature Extraction

Our acoustic feature engineering is not complicated. The system
is built using several different features including 39dimensional MFCC for GMM, 40-dimensional static MFCC
and 80-dimensional filter banks for neural networks. Thesefeatures can be augmented with i-vectors to train speaker1

https://github.com/kaldiasr/kaldi/blob/master/egs/ami/s5b/local/run_cleanup _segmentation.sh

112
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

For TDNN neural acoustic model, we use factored form of
TDNN [3] to design our own network. The factorized TDNN
(TDNN-F) is reported to beat common TDNN with deeper
architecture [3], in order to identify some hyper parameter
configuration, we first train TDNN-F models with only the
TED-LIUM data and decode with a relatively small n-gram
language model. We summarize the intermediate result in table
1. Here we only report average WER of tst2013, tst2014 andtst2014.
As can be seen from the table, it is beneficial to use i-vector
or fMLLR transformed feature to train speaker adapted
networks. Comparing the third row and the forth row, we findWER of fMLLR system is 15.09 which is worse than i-vector
augmented system of 14.23. As we gradually increase thenumber of layers from 16 to 26, a steady performance
improvement is obtained. TDNNs that is deeper than 26 may
decrease in performance as the 31 layer net is worse than 26
layer net. The 200 dimensional i-vector augmented 26 layer
TDNN reaches a WER of 13.93. Additional discriminative
training (DT) also help, it helps to decrease WER from 14.03 to
13.62. When adding the cleaned 220 hours of data, we lower the
WER from 14.03 to 13.35.

adapted networks. The dimension of i-vectors is chosen to be
200 which is extracted from every 50 frames. I-vectors andfeatures are combined in the input layer for TDNNs and
BLSTMs. While for CNN, input layer only contains filter bankand i-vectors are combined with the fully-connected layer
before the output. We also extracted fMLLR transformedfeature on 340hr TED-LIUM data and found fMLLR featurescontribute no significant improvement compared to i-vector
augmented system. So we only use i-vector to train our final
speaker adapted systems.
2.3. Acoustic Modeling
We use DNN-HMM hybrid acoustic model for all our ASR
experiments. All NN systems were trained using Lattice-free
MMI (LF-MMI) [9] loss function with low frame rate (LFR)
equals to 3 to predict context dependent phones (bi-phone). We
mainly used three different types of neural net architecture,including deep convolutional neural network (DCNN),
bidirectional LSTM (BLSTM) and time delayed deep neuralnetwork (TDNN). In GMM-HMM part, we use 13-dimension
mel frequency cepstral coefficient (MFCC) with first andsecond derivatives with 500 hours data without speed
perturbation. The dictionary provided in the TED-LIUM dataset
is used for our GMM training. The final GMM has totally
150,193 Gaussian mixtures, correspond to 4056 states. This
500hr GMM-HMM was used to align all the 1500h data to
generate state alignments for clustering bi-phone labels used in
LF-MMI training. After clustering we finally get 3144 biphones which equal the output nodes number of all our neuralnetworks.
2.3.1.

Table 1: TDNN results on TED-LIUM corpus#layerconfiguration
average
16
40MFCC
15.76
1640MFCC + 100ivec
15.53
2140MFCC + 100ivec
14.23
2140MFCC + fMLLR
15.09
26
40MFCC + 100ivec
14.03
26
40MFCC + 200ivec
13.93
26
40MFCC + 100ivec + DT
13.62
31
40MFCC + 100ivec
14.3
26
40MFCC + 100ivec + 220h data
13.35

Deep CNN

We were inspired by the VGG net [10] and the deep CNN
architecture used in [1] [2] to design our CNN model. We train
our DCNN model with 80 dimension filter bank feature without
first and second derivatives. We use batch normalization (BN)and ReLU nonlinear activations following each convolution
layer. We stack 31 layer of such conv-BN-ReLU block withresidual connections around every two of them. Most of the twodimensional time-frequency convolution kernels are all set to 3
x 3 with stride 1x1. We set kernel size to 5 x 5 with stride 2 x 1
in the 6th, 12th, 24th, 30th convolution layer to reduce thefrequency dimension from 80 to 5. Every time we reduce thefrequency dimension we double our kernel number. So as wego deeper, the kernel number is set to 32, 64, 128, 256, and 384.
We train such DCNN with all the 1500h data to obtain system
dcnn.
2.3.2.

Conclude from Table 1, our final TDNN use a 26 layers
TDNN architecture. We abandon fMLLR and use 200
dimensional i-vector augmented to MFCC to train speaker
adapted net. Each hidden layer contains 1024 units and 160dimension bottleneck. The input to TDNN is 5 frames of 40dimension static MFCC. The other TDNN layer has an input
context equals to 3 which has different time stride. We
constructed 3 consecutive layers with time stride 1, 4
consecutive layers with time stride 2 and 15 consecutive layerswith time stride 3. Each of these consecutive layers with thesame time stride is followed by a fully connected layer. We train
two of them with different random seeds with total data, and thethird TDNN with 80% data. After TDNN training we got tdnn1,
tdnn2 and tdnn3.

BLSTM

Our BLSTM model consists of 5 layers that has two
unidirectional LSTM with 1024 cells and 512 projections. 256of the projections are recurrent units and the other 256
projections are non-recurrent ones. 40 dimensional static
MFCC feature is extracted for BLSTM training. By
concatenating the two previous and the two following frames of
MFCC, we use 40 * 5 = 200 dimension feature to train two
BLSTM with different random seeds using all of the 1500h data.
We call the two BLSTM with blstm1 and blstm2.
2.3.3.

2.4. Language Model
2.4.1.

Data Preparation and the Vocabulary

For the data preparation, number normalization andlowercasing are adopted to formatting the all-corpora. Next,
punctuations are removed and the paragraphs are split into
sentences. We choose 152217 English words to build thevocabulary and replace all the out-of-vocabulary (OOV) words
in the corpora with the symbol “<unk>”.

TDNN

113
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

2.4.2.

N-gram Language Models

3. Neural Machine Translation

The constrained all-corpora consists of various text resources,
such as news, TED subtitles, film subtitles, Europarl dataset andsome web crawled materials. The Table 2 shows the details of
the cleaned sub-corpora and their interpolation coefficients in
n-gram language modeling. We estimate a series of sub-corpora
5-gram language models using the SRILM toolkit [11] with themodified Kneser-Ney smoothing. And then, the development
datasets are used for the perplexities and the interpolation
weights tuning. By linearly interpolating the different subcorpora 5-gram models, the final back-off language model isestimated and adopted to the speech recognition system. The
perplexities of the development datasets are listed in the Table
3.

In this section, some post-processing details of ASR output andthe architecture of our neural machine translation system aredescribed.
3.1. Punctuation Restoration
The automatic speech recognition system only generates a
stream of words without any punctuation symbols. In our work,
we model the punctuation using the sequence to sequencearchitecture. Our punctuation restoration model is based on theTransformer architecture, which is based on attention only. In
our work, given a sequence of words as our inputs, we label
each word based on the punctuation after the word. Specifically,
we label each word with comma, period, question mark,exclamation mark and non-punctuation.
The training dataset contains 41.5M sentences in total.
Sentences were encoded using byte-pair encoding [15] withsource vocabulary of about 30k tokens. We evaluate theperformance of our punctuation restoration model by precision,
recall and F1 score. We present the results in table.

Table 2: English language modelling datasets and interpolation
coefficients.
Text corpus# Words Interpolation
TED
5.747 M
0.131
OpenSubtitles
144.1 M
0.064
Para WIT
3.263 M
0.029
ParaCrawl + Common crawl
765.1 M
0.048
News discussions
4638 M
0.397
News articles
4004 M
0.331

Table 4: The result of our Punctuation Restoration model
Dev setPrecision
Recall
F1 value
tst13
88.01%
82.18%
85.00%
tst14
88.62%
84.28%
86.40%
tst15
91.51%
86.26%
88.81%
average
89.38%
84.24%
86.73%

Table 3: The perplexities (PPL) of the English dev corpora.
Dev set5-gram LM
tst2013112.31
tst2014143.11
tst2015
121.802.4.3.

3.2. Disfluency Detection and Inverse Text Normalization
Since the automatic speech recognition outputs often containvarious disfluencies. In this paper, a simple but efficient
detection approach is employed to identify and repair thesedisfluencies. At first, we remove the filled pauses, such as “uh”
and “um”. Then we define a window to identify and remove
the repetitions in the output of ASR system.
After disfluency detection, the inverse text normalization isnecessary for machine translation, because the corpus ofmachine translation are in written form, but the output of theautomatic speech recognition are generally in spoken form,
especially in figure, data and the amount of money. As shownin Figure 1, the word stream generated by ASR system istransformed into the standard form after punctuation restoration,
disfluency detection and inverse text normalization.

LSTM based Neural Language Model

To improve the computation efficiency in the neural language
model, the vocabulary needs to be downsized. We select the top
30000 frequent words from the cleaned corpora to construct a
small vocabulary, and replace the out-of-vocabulary words inthe cleaned corpora with the symbol “<oos>” according to thecustomized vocabulary.
The LSTM based language model are trained with
TensorFlow. The model contains two stacked dropout wrapped
LSTM layers [12] with the hidden size of 256. The wordembedding size is 256 and the initial learning rate is 0.1. Afterthe training, we apply the LSTM based language model in thelattice rescoring and n-best rescoring with the Kaldi toolkit [6].
The pruned lattice-rescoring algorithm in [13] helps to achieve
lower word error rate (WER) in ASR. Both in the lattice
rescoring and n-best rescoring stages, interpolating the 5-gram
language model with the LSTM based language model canfurther improve the ASR accuracies.

Figure 1: Post-processing for ASR output
The output of ASR system:and the results from the twenty twenty two uh point fivemillion sentences we selected sixteen point eight andwhich let us to throw like twenty two percent of the
corpus

2.5. System combination

After our post-process:and the results from the 22.5 million sentences, weselected 16.8 and which let us to throw like 22% of the
corpus.

In the first pass, we use 6 neural network systems described in
section 2.3, e.g. dcnn, blstm1, blstm2, tdnn1, tdnn2, tdnn3 and5-gram language model described in section 2.4.2. We combine
the system in the posterior level and generate the first passensemble results. We also select the best single network system
tdnn1 to perform discriminative training, but found no
performance gain when combine with the above 6 systems. The
decoding lattices of the ensemble system are sent to a second
pass decoder for lattice rescoring.

114
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

the output tokens’ possibilities by averaging the systems’
output possibilities.
In the final system, we choose six systems to apply the second
ensemble.

3.3. Data Preparation and Cleaning for NMT
The parallel text data consists of four parts: Speech-Translation
TED corpus, TED corpus (Web Inventory of Transcribed andTranslated Talks, WIT), WMT2018 and OpenSubtitles2018.
We tokenize both the English and Germany text data by
the Moses tokenizer2. Then the English data is transformed tolower case. To simplify the post processing of translation, we
do not transform the German data to lower case. Finally, we use
BPE subword segmentation tool to process the English data andGerman data.
We have observed some noise data, which cause a lot oftranslation errors. In order to improve the quality of parallel text
data, we have cleaned the data.
The samples whose number of tokens are over 100 will be
removed.


For one sentence pair, if the length rate of source/target isless than 1/2 or large than 2, they will be removed.



We use SRILM Toolkit [11] to train an English ngram
language model and a German ngram language model
respectively with the parallel text data. The two LMs areused to evaluate the perplexity (PPL) for the sentences. For
one source sentence (English side) and target sentence(German), they are removed if they meet the following two
conditions: (1) we use source LM to calculate PPL. The
PPL of source sentence is larger than that of target sentence;(2) we use target LM to calculate PPL. The PPL of target
sentence is larger than that of the source sentence.

3.7. Re-scoring with NMT Variants
In order to get better translation result, we test different NMT
variant models in re-scoring n-best list.
Target right-to-left NMT Model: When the target words
are decoded by the NMT system, the later words will dependon the previous words decisions in the beam search decoder.
So the word decision at time step t is much harder than that oftime step t-1[16]. In order to alleviate this imbalance problem,a variant NMT model, which decodes the target words fromright-to-left (R2L), is trained. The R2L model is used to rescore the n-best list which produced by the main NMT model.
The scores represents the conditional probabilities of thereversed translations given the source sentences.
Target-to-source NMT Model: Moreover, the translations
may be inadequate: the translations may repeat or miss out
some words [17]. In order to cope with the inadequateness, we
also test the target-to-source (T2S) model, which is trained withthe source and target swapped.
We first produce one n-best list with an ensemble of serval
models. Then we do force decoding with target right-to-left,target-to-source NMT models. We treat each models scores as
an individual feature. We use k-batched MIRA [18] to tune
weights for all the features. In order to get more diverse n-best
list, we also try to increase the size of beam from 10 to 100 for
re-scoring.

3.4. NMT Architecture

4. Results

Our model follows the Transformer architecture which is solely
based on attention mechanisms [4]. In our setup, the encoder
has six layers. Each layer is consist of two parts: multi-head
self-attention network and position-wise fully connected feedforward network. The two parts employ both residual
connection and layer-normalization. In the decoder, we employ
masking to ensure that the prediction for the current word only
depends on the previous words.
The dimension of word embedding is set to 512. The
hidden state size is set to 1024. The vocabulary sizes of English
and German are set of 60,000.
The sentences which have the similar number of tokens aregrouped together. During training, the batches of size is set by
the number of tokens which is set to 8000. We use the Adam
optimizer to train the model.

4.1. Results for ASR
Table 5 shows our systems built for the ASR submission. In thefirst pass, we use 6 neural network system described in section
2.3, e.g. dcnn, blstm1, blstm2, tdnn1, tdnn2, tdnn3 and 5-gram
language model described in section 2.4.2. We combine thesystem in the posterior level and generate the first passensemble results. The decoding lattices of the ensemble system
are sent to a second pass decoder for lattice rescoring.
Table 5: The WER result of our ASR model
Systemtst2013 tst2014 tst2015 average
dcnn
11.15
8.86
7.77
9.26
blstm1
8.65
7.84
8.02
8.17
blstm2
8.78
8.07
8.03
8.29
tdnn1
8.5
7.35
6.24
7.36
tdnn2
8.52
7.42
6.15
7.36
tdnn3
8.47
7.55
6.18
7.4
+ensemble
8.01
7.08
6.54
7.21
+rescoring
7.49
6.76
5.95
6.73

3.5. Fine-tune
A large part of the training data comes from WMT, whose
domain is news. But the test sets come from oral domain. Afterthe systems are trained, we continue to train the systems by5000 steps with the WIT parallel text data.
3.6. Ensemble
It’s common to avoid over-fitting by using ensemble of several
systems. There are two methods we have adopted. For one
system training, we always average all of parameters across thelast 20 checkpoints. For several system trainings, we compute

4.2. Results for NMT

Table 6 shows the machine translation results on validation
sets. All the results are cased BLEU evaluate by multibleu.perl script in Moses 3 . uur data cleaning techniuue
32

https://github.com/mosessmt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl

https://github.com/mosessmt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl

115
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

improves the baseline by 0.74 BLEU. Due to the domainof training data is not very consistent with that of test data,
we continue training the system with the WIT parallel text
data. This fine-tune techniuue get an improvement of 1.43
BLEU. In order to get more diverse models and better
ensemble results, we train 6 models independently withdifferent random initializations. The ensemble result gives
an improvement of 0.53 BLEU over best single system. By
increasing the beam size from 10 to 100 during decoding,we achieve another improvement of 0.05 BLEU. We add
six right-to-left and six target-to-source NMT models as rescoring features. It improved the system by 0.85 BLEU.
The test2013 set is used as development set to tune theweights of re-scoring features.

ensemble of Transformer system with n-best rescoring. And weuse various techniques in our system, such as data cleaning,fine-tune, ensemble of models and n-best rescoring. These
techniques help our system achieve 3.6 BLEU better thanbaseline. We use the outputs of the best ASR system as inputof our NMT system, and we achieved average BLEU score of
31.02 over three development sets.
How to use document-level information to improve the
ASR and NMT system performance and build a robust NMT
system will be our future work.

6. References[1] Zhang Y, Pezeshki M, Brakel P, et al. Towards end-toend speech recognition with deep convolutional neuralnetworks[J]. arXiv preprint arXiv:1701.02720, 2017.
[2] Qian, Yanmin, and Philip C. Woodland. "Very deep
convolutional neural networks for robust speech
recognition." Spoken Language Technology Workshop(SLT), 2016 IEEE. IEEE, 2016.
[3] Povey, D., Cheng, G., Wang, Y., Li, K., Xu, H.,
Yarmohamadi, M., & Khudanpur, S. (2018). Semiorthogonal low-rank matrix factorization for deep neuralnetworks. INTERSPEECH 2018.
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser,
and Illia Polosukhin. Attention is all you need. CoRR,
2017.
[5] A. Rousseau, P. Deléglise, and Y. Estève, “Enhancing the
TED-LIUM Corpus with Selected Data for Language
Modeling and More TED Talks”, in Proceedings of theNinth International Conference on Language Resources
and Evaluation (LREC’14), May 2014.
[6] Povey, D., Ghoshal, A., Boulianne, G., Burget, L.,
Glembek, O., Goel, N., ... & Silovsky, J. (2011). The Kaldi
speech recognition toolkit. In IEEE 2011 workshop on
automatic speech recognition and understanding (No.
EPFL-CONF-192584). IEEE Signal Processing Society.
[7] Nakatani, T., Yoshioka, T., Kinoshita, K., Miyoshi, M., &Juang, B. H. (2008, March). Blind speech dereverberation
with multi-channel linear prediction based on short time
Fourier transform representation. In Acoustics, Speechand Signal Processing, 2008. ICASSP 2008. IEEE
International Conference on (pp. 85-88). IEEE.
[8] Schwartz, B., Gannot, S., & Habets, E. A. (2015). Online
speech dereverberation using Kalman filter and EM
algorithm. IEEE/ACM Transactions on Audio, Speech andLanguage Processing (TASLP), 23(2), 394-406.
[9] Povey, D., Peddinti, V., Galvez, D., Ghahremani, P.,
Manohar, V., Na, X., & Khudanpur, S. (2016, September).
Purely Sequence-Trained Neural Networks for ASR Basedon Lattice-Free MMI. In Interspeech (pp. 2751-2755).
[10] Simonyan K, Zisserman A. Very deep convolutional
networks for large-scale image recognition[J]. arXiv
preprint arXiv:1409.1556, 2014.
[11] A. Stolcke, “SRILM-an extensible language modeling
toolkit,” in Proceedings of Interspeech, September 2002,
pp. 901–904.
[12] Y. Gal, and G. Zoubin, "A theoretically grounded
application of dropout in recurrent neural networks,"
in Advances in neural information processing systems,
pp. 1019-1027. 2016.
[13] H. Xu, T. Chen, D. Gao, Y. Wang, K. Li, N. Goel, Y.
Carmiel, D. Povey and S. Khudanpur, “A Pruned

Table 6: The English→Germany NMT results on threedevelopment sets. Submitted system is the last system.
systemtst2013

tst2014

tst2015

averagebaseline+data cleaning

34.73
35.4

29.09
30.03

33.02
33.62

32.28
33.02

+fine-tune+ensemble

37.13
37.79

31.28
31.56

34.93
35.58

34.45
34.98

+beam(10 → 100)
+rescore(6*R2L,6*T2S)

37.92

31.32

35.86

35.03

38.90

32.36

36.38

35.88

4.3. Results for Speech Translation
Table 7 shows the final speech translation results on three test
set. In order to tune the ASR and NMT system individually.
We first segment the full utterance, and then align the utterance
into segments with the correct English text segments andGerman translations. The transcript of the best ASR system
was then passed to disfluency detection, Punctuation
Restoration and text normalization module. Finally, ASR
outputs with punctuations were translated into German. The
average result of three test set for our Speech Translation is
31.02 BLEU.
Table 7: The English→Germany speech translation results on
three sets.
systemtst2013

tst2014

tst2015

average

final system

32.95

28.28

31.82

31.02

5. Conclusions
This paper describes our pipeline system for the IWSLT 2018
Speech Translation task from English to German. The whole
pipeline are consist of the wav utterance segmentation module,
the ASR system, the punctuation restoration and the NMT
system.
As for the ASR system, we adopted an ensemble system of
Deep-CNN, BLSTM, TDNN, n-gram Language model withlattice rescoring. According to our experiments, TDNN
achieved the lowest WER among these three acoustic modeling
network for this task. For our tdnn acoustic modeling, we foundadding layers, i-vector, cleaned data are effective. We haveachieved average WER of 6.73 over three test sets using the
combination system. For the NMT system, we also use an

116
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[16] Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro
Sumita. 2016. Agreement on Target-bidirectional Neural
Machine Translation. In NAACL HLT 16, San Diego, CA.
[17] Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, andHang Li. 2016a. Neural machine translation withreconstruction.
arXiv
URL:
https://arxiv.org/abs/1611.01874.
[18] Colin Cherry and Gorge Foster. 2012. Batch Tuning
Strategies for Statistical Machine Translation, In NAACL,2012.

RNNLM Lattice-Rescoring Algorithm for Automatic
Speech Recognition,” in Acoustics, Speech and Signal
Processing (ICASSP), 2017 IEEE International
Conference on. IEEE, 2017.

[14] E. Cho, J. Niehues, and A. Waibel, “NMT-based

segmentation and punctuation insertion for realtime spoken language translation,” Proc.
Interspeech 2017, pp. 2645–2649, 2017.
[15] Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016b. Neural machine translation of rare words withsubword units. In Proceedings of ACL 2016.

117
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Samsung and University of Edinburgh’s System for the IWSLT 2018 Low Resource
MT Task
Philip Williams2 , Marcin Chochowski1 , Pawel Przybysz1 , Rico Sennrich2 , Barry Haddow2 ,Alexandra Birch2
12

Samsung R&D Institute, Poland
School of Informatics, University of Edinburgh{m.chochowski,p.przybysz}@samsung.com{pwillia4,bhaddow}@inf.ed.ac.uk, {rico.sennrich,a.birch}@ed.ac.uk

Abstract

2. Training Data

This paper describes the joint submission to the IWSLT 2018
Low Resource MT task by Samsung R&D Institute, Poland,
and the University of Edinburgh. We focused on supplementing the very limited in-domain Basque-English training datawith out-of-domain data, with synthetic data, and with datafor other language pairs. We also experimented with a variety of model architectures and features, which included thedevelopment of extensions to the Nematus toolkit. Our submission was ultimately produced by a system combination inwhich we reranked translations from our strongest individual
system using multiple weaker systems.

1. Introduction
This paper describes the joint submission to the IWSLT 2018
Low Resource MT task by Samsung R&D Institute, Poland,
and the University of Edinburgh. We built several multilingual systems using the Tensor2Tensor1 and Nematus2 toolkits, ultimately choosing to use a system combination in which
we reranked translations from our strongest individual system using multiple weaker systems.
As there was so little in-domain Basque-English data
available, we experimented with the use of out-of-domain
data, with the addition of synthetic data via back-translation,
and with the incorporation of data for other language pairs.
To support multilingual translation, we followed the singlemodel approach of [1] and simply prepended each source
sentence with a token specifying the target language.
We experimented with a variety of model architecturesand features. This involved the development of several extensions to the Nematus toolkit, including support for multiGPU training, label smoothing, and mixtures of softmaxes.
We have contributed our code to the public Nematus repository.
1 https://github.com/tensorflow/tensor2tensor:
1.6.3
2 https://github.com/EdinburghNLP/nematus

In brief, we used all of the provided in-domain parallel training data along with parallel data from OpenSubtitles and theOpen Data Euskadil Repository. We also produced synthetic
data by back-translating from English into Basque. Table 1
lists the individual parallel corpora that made up our training
data. Note that not all of our systems used all of the data.
We will indicate differences when describing the individual
systems.
2.1. In-Domain Data
We used all of the available in-domain data, which we filteredin order to remove the TED talks covered by the devset. The
task organisers had already removed the devset talks from theBasque-English training corpus, but the talks were present in
the training data for all other language pairs. Since we were
evaluating on the devset during system development, we filtered the in-domain data to avoid being misled by artificiallystrong results. In preliminary multilingual Nematus systems
that used only the in-domain data, this filtering had a significant impact, reducing the B LEU score from 16.25 to 9.96.
For the Basque-Spanish and Spanish-English pairs, weused the excised training data to create supplementary devsets. Since the Basque-English devset only contained 1,140
sentence pairs, these additional devsets gave us greater confidence when evaluating system changes.
More generally, we noticed that the in-domain corpora
contained many of the same talks, in effect reducing theamount of available in-domain Basque data.
2.2. Out-of-Domain Data
We added out-of-domain data for the Basque-English,
Basque-Spanish, Spanish-English, and French-English language pairs. For all four, we used OpenSubtitles2018 datafrom the OPUS corpus. In order to avoid making our training data too unbalanced, we undersampled from the large
Spanish-English and French-English corpora. This was done
arbitrarily: we simply used the first N -million sentence pairs

118
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Pair

Corpuseu-en

In-domain
OpenSubtitles2018
Synthetic in-domain
Synthetic OpenSubtitles2018
Synthetic TED2013
In-domain
OpenSubtitles2018
EuskadilIn-domain
OpenSubtitles2018
In-domain
In-domain
In-domain
OpenSubtitles2018

eu-es
es-en
es-fr
eu-fr
fr-en
Total

Sentence Pairs

eu srces srcen tgtes tgt5,623
805,780
277,097
4,000,000
1,904,674
5,546
793,593926,941
277,097
10,000,000
277,278
5,815
287,137
10,000,000

5,623
805,780
277,097
4,000,000
1,904,674
5,546
793,593926,941-

277,097
10,000,000
277,278
5,815
10,000,000

5,623
805,780
277,097
4,000,000
1,904,674
277,097
10,000,000
277,278
5,815
287,137
10,000,000

5,546
793,593926,941-

29,566,581

8,719,254

20,847,327

27,840,501

1,726,080

Table 1: Statistics for the parallel training corpora used in our submission (note that not all individual systems use all of thecorpora). Since we are interested in Basque-to-English translation (primarily) as well as Basque-to-Spanish and Spanish-andEnglish, we break down the corpus sizes for those source and target languages.
occurring in the full corpora. For Basque-Spanish, we also
used the parallel data from the Open Data Euskadil Repository.
At the outset, we assumed that translation from Basqueinto Spanish would be easier than into English due to thegreater availability of in-domain data. We contemplated pivoting from Basque to English via Spanish and therefore whenselecting data from OpenSubtitles, we made an effort to include sufficient data to support high-quality Basque-Spanish
and Spanish-English translation. However, translation quality for Basque-Spanish and Basque-English (as measured by
B LEU) proved to be very similar and we therefore focusedon direct translation from Basque to English.
As with the in-domain data, we noticed that there isa high degree of content overlap between the multilingual
OpenSubtitles corpora. For OpenSubtitles2018, between
70% and 90% of the Basque side is common for BasqueEnglish, Basque-Spanish and Basque-French making the effective size of Basque data seen by the system relatively
smaller.

domain Spanish-English corpus, we back-translated the English talks from the OPUS TED2013 corpus, after filteringout the dev and test set talks. We also selected 4M pseudo
in-domain sentences from OpenSubtitles using the filtering
approach proposed in [3].

2.3. Synthetic Data

3.1. Preprocessing

Basque is a language isolate spoken by less than 1 million
people and as such there are few readily available parallel
resources. One of the simplest ways to get more parallel datais to generate it syntheticaly through back-translation. In [2]it was shown that even poor quality synthetic corpora can
improve translation quality. We used all available training
data to train an English-to-Basque back-translation system
for synthetic data generation (see Section 3.2 for details of
the back-translation system).
In addition to back-translating the English side of the in-

We relied on the preprocessing implemented in Tensor2Tensor for tokenization and wordpiece segmentation.
For each corpus configuration we defined a new T2T problem inheriting from default TranslateProblem. We set the
subword vocabulary size to 32k for all training runs, either
bilingual or multilingual. The only additional preprocessing we did was punctuation normalization using the Moses
toolkit and prepending the <2xx> tag at the beginning ofsource sentences, where xx was the code for the target language.

3. Tensor2Tensor Systems
In preliminary experiments, we tried training Transformer
models [4] using both Tensor2Tensor and Marian, eventually choosing the former as the B LEU was higher. In
all experiments we used the hyperparameters for transformer base, setting the hidden layer size to 512, filter size
to 2048, warmup steps to 16,000 and number of heads to
8. While training the back-translation model we set thelayer prepostprocess dropout parameter to 0.1, while in thebase systems it was set to 0.2. Each training was run on 8
GPUs for up to 300,000 training steps, with a batch size of
100 sentences per GPU.

119
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

System

en-eu

es-eu

en-es

System

eu-en

eu-es

es-en

EnFrEs2EuFrEs

13.26

14.89

41.92

bilingual in-domain only
EuFrEs2EnFrEs in-domain only
+ out-of-domain+ back-translation
EuEs2EnEs

11.72
18.41
22.26
23.45
25.09

-

-

22.28
17.81

42.76
43.03

Table 2: B LEU scores for the Tensor2Tensor systems on
dev2018 (eu-en) and on eu-es and es-en versions of the dev
set (extracted from the training data). Since this was a backtranslation system, we inverted the devsets to evaluate translation in the opposite direction. This system was used forback-translation of monolingual English corpora

3.2. Back-Translation System
For back-translation, we used all of the in-domain data listed
in Table 1, along with the OpenSubtitles corpora for BasqueEnglish and Basque-Spanish, and the Euskadil corpus forBasque-Spanish. We also used 1M sentence pairs of SpanishEnglish, making 5.5M sentence pairs in total.
We trained both Nematus and Tensor2Tensor systems on
the same dataset, obtaining results of 12.14 B LEU and 13.26
B LEU respectively on the inverted Basque-English devset(see Table 2). We chose the better-performing Tensor2Tensor
system for synthetic corpus generation.
3.3. Base System
For Basque-to-English translation, we experimented withdifferent language pair and corpus selections (Table 3). We
started with a bilingual model trained only on in-domain
data. Next we trained a multilingual model adding all directions for in-domain data and oversampling the BasqueEnglish data by a factor of 20 (to better balance the larger
in-domain Spanish-English corpus). This resulted in a significant improvement of almost 7 B LEU points. After adding
out-of-domain and synthetic corpora we got another 5 B LEU
points. Next we experimented with removing the Frenchdata from the multilingual setting as it had the least Basquesentences, giving little additional input for that language
and adding complexity be adding another language into themodel. We observed that removing the French parallel corpora gave significantly better results, improving BasqueEnglish translation by 1 B LEU point on dev2018. For the
final submission we used the EuEs2EnEs model for n-best
list generation.

4. Nematus Systems
Nematus [5] implements a GRU-based attentional encoderdecoder. Originally based on the model in [6], the toolkit has
been extended to support features such as deep architecturesand input factors. Our system was based on the configuration used in University of Edinburgh’s WMT17 submissions[7]. To this we added several further extensions, which we
describe below.

Table 3: B LEU scores for the Tensor2Tensor systems on
dev2018 (eu-en) and on eu-es and es-en versions of thedev set (extracted from the training data). The last system
EuEs2EnEs was used to produce the 20-best list for further
rescoring.
4.1. Preprocessing
All of our Nematus systems used a common preprocessing
pipeline, consisting of five steps: normalization, tokenization, corpus cleaning, truecasing, and BPE segmentation. [8]We used scripts from the Moses toolkit [9] to perform thefirst four steps and subword-nmt3 to perform the last.
The Moses tokenizer includes language-specific rules,
which we opted to use.4 However, we trained a shared truecasing model for all languages. The corpus cleaning script
removes empty sentences and sentence pairs with length ratios greater than 9:1.
We trained a single joint BPE model over the full multilingual corpus, using 40,000 merge operations. Character
sequences were only merged if they were observed 50 times
in the training data.
4.2. Base System
Our base Nematus system used all of the data in Table 1 except for the synthetic data (which was added later for the final systems) and the French-English OpenSubtitles data. For
Spanish-English we used 1M sentence pairs of OpenSubtitles
rather than 10M.
4.2.1. Network Configuration
We used a word embedding size of 512 and hidden layer sizeof 1024. Both the encoder and decoder used a deep transitionarchitecture [10], with 4 layers in the encoder and 8 in the
decoder. We used layer normalization [11].
We tied the weights of the target-side embedding and thetranspose of the output weight matrix [12]. Since the source
and target sides used the same vocabulary, we also tied thesource-side and target-side embeddings.
4.2.2. Training
We used the Adam [13] optimization algorithm with a learning rate of 0.0001 and a batch size of 80 (except where
3 https://github.com/rsennrich/subword-nmt

4 Moses does not include Basque-specific tokenization rules, so it fellback to generic tokenization for that language

120
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

noted). Training was stopped when the validation crossentropy failed to reach a new minimum for 10 consecutive
save-points (saving every 10,000 updates). The save-point
used in the final model was selected based on the B LEU scoreof the validation set.
To speed up training, we excluded sentences in which
either the source or target sentence contained more than 50
tokens.
In preliminary experiments, we found that it was important to use dropout (giving improvements of around 2 B LEU).
The dropout rate was set to 0.1 for source and target word tokens and to 0.2 for embedding and hidden layers.
4.3. Extensions
4.3.1. Multi-GPU Support
Training the base model was already pushing the 12GB
memory limit of our GPUs, restricting our ability to add
new features. Since we did not want to risk compromisingmodel quality by reducing the network size, we opted to implement multi-GPU training in order to reduce the per-GPU
batch size, while maintaining (or increasing) the effective total batch size.5 We added support for synchronous training
in which the batch is split between multiple GPUs (on thesame server), each running a full replica of the model, andthen the gradients of the sub-batches are averaged. Unlike
asynchronous training, this method does not affect translation quality compared to single-GPU training (assuming thebatch size is constant).
4.3.2. Source language factors
As already mentioned, our training data contains tags to indicate the target language. In preliminary experiments, wefound that it was beneficial to also specify the source language, which we did through the use of token-level factors.
Our intuition was that the factors would help to disambiguate
subword units that occur in multiple languages, but servelanguage-specific roles.
Since Nematus already included support for factors [14],this was simply a case of annotating the training and dev/test
data with language tags and adjusting the network’s wordembedding settings: of the 512 source embedding units wereserved 12 for the source language factor tag and the remaining 500 for the BPE token embedding.
A contrastive experiment showing B LEU scores on
dev2018 with and without source language factors can befound in Table 4.
4.3.3. Label smoothing
We implemented label smoothing [15], a regularization technique which has been shown to be effective for self-attentionbased translation models [4], and, more recently, for RNN5 An alternative would have been to use delayed updates on a single GPU– or of course to buy GPUs with more memory.

System

eu-en

eu-es

es-en

Base
Base + source language factors
Base + label smoothing
Base + mixture of softmaxes
Base + fine-tuning

19.99
20.12
20.46
20.02
20.75

20.45
20.86
20.65
21.02
1.86

39.74
40.16
40.16
40.14
39.94

Table 4: B LEU scores for Nematus systems on dev2018 (euen) and on eu-es and es-en versions of the dev set (extracted
from the training data). These systems use all of the parallel
training data except for the synthetic data.
based models similar to ours [16]. Following prior work, we
set the ǫ parameter to 0.1. See Table 4 for results of a contrastive experiments with and without label smoothing.
4.3.4. Mixture of Softmaxes
Like all standard neural translation models, our base model
uses a softmax function to output a probability distribution
over the target vocabulary for each timestep. For language
modelling, [17] show that performance can be improvedby using a combination of multiple softmax components.
We reimplemented their method within Nematus and experimented with using a mixture of three softmax components.
See Table 4 for results of a contrastive experiments with andwithout a mixture of softmaxes.
4.3.5. Fine-tuning
Since our system was trained on data drawn from multiple
domains and covering several language pairs, we anticipatedthat there would be a benefit to fine-tuning on in-domain
Basque-English data. After selecting the best model (according to validation set B LEU), we resumed training using only
the in-domain Basque-English data (5,623 sentence pairs).
See Table 4 for results of a contrastive experiment with andwithout fine-tuning.
4.4. Final Systems
Our final Nematus systems used all of the training data from
Table 1, with the exception of the synthetic TED2013 corpus (since training was started before the filtered corpus
was produced) and the French-English OpenSubtitles corpus. We used a 1M sentence pair version of the SpanishEnglish OpenSubtitles corpus. As in the Tensor2Tensor system, we oversampled the in-domain Basque-English corpus
by a factor of 20. We experimented with removing the Frenchtraining data but, unlike the Tensor2Tensor system, this did
not improve performance (possibly because we had used lessFrench data to start with).
We used all of the extensions just described. We trained
two such systems, one using two GPUs with a total batch
size of 80 and one using three GPUs with a total batch sizeof 160. Finally, we fine-tuned these systems giving a to-121
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

System
Nematus (batch size 80)
Nematus (batch size 80, fine-tuned)
Nematus (batch size 160)
Nematus (batch size 160, fine-tuned)Tensor2Tensor+ reranking (default length penalty)+ tuned length penalty

dev2018

tst2018

22.56
23.22
22.94
23.86
25.09
25.40
25.60

23.18
23.65
23.56
24.12
25.40
25.97
26.21

Table 5: B LEU scores on the official dev and test sets. The
first five rows show the results for the individual Nematus andTensor2Tensor systems used in the final system combination.
The bottom two rows show the results of reranking the 20best list from the Tensor2Tensor system with the Nematus
systems and then tuning the length normalization parameter.
The system in the bottom row is our submitted system.
tal of four Nematus systems. Unlike our base system, finetuning using only the in-domain data did not improve translation quality, possibly due to the oversampling of this datain the training set. Instead, we used a fine-tuning corpusthat combined the genuine in-domain data with the synthetic
in-domain data (which was back-translated from the English
side of the Spanish-English corpus). Results with and without fine-tuning are given in Table 5.

5. System Combination
Of the individual systems, we achieved the best performanceon the devset using the Tensor2Tensor EuEs2EnEs system.
We used that system to generate a 20-best list, which wethen rescored using the four final Nematus systems. After
rescoring, we renormalized the individual scores for sentence
length, optimising the length penalty (i.e., the alpha value in[18]) on dev2018, setting it to 1.5 in our submission (in all
previous systems, the length penalty was set to the default
value of 1.0). Finally, we reranked the list according to thesum of the five renormalized scores and used the resulting
1-best translations in our submission.
Table 5 gives B LEU scores on the dev and test sets for thefive component systems and the reranked system, both withand without length penalty tuning.

6. Conclusions
For this task, we focused on supplementing the very limited in-domain Basque-to-English training data with out-ofdomain data, with synthetic data, and with data for other language pairs. Through data alone, we improved translation
quality from 11.72 to 25.09 B LEU.
Although our Nematus systems underperformed the Tensor2Tensor systems, we were able to narrow the gap throughextensions to the base model, including label smoothing andsource language factors. When evaluated on tst2018, ourbest Nematus system was 1.3 B LEU behind our best Ten-sor2Tensor system.
Despite the Nematus systems being weaker, we were able
to further improve performance by reranking a 20-best list
from the Tensor2Tensor system using the four final Nematus systems. Tuning the length penalty also boosted performance slightly. Our submitted system scored 26.21 B LEU on
tst2018, outperforming the individual Tensor2Tensor system
by 0.81 B LEU.

7. References[1] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu,
Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean, “Google’s multilingual
neural machine translation system: Enabling zero-shottranslation,” Transactions of the Association for Computational Linguistics, vol. 5, pp. 339–351, 2017.
[2] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” in Proceedings of the 54th Annual Meeting of
the Association for Computational Linguistics (Volume
1: Long Papers). Berlin, Germany: Association for
Computational Linguistics, August 2016, pp. 86–96.
[3] A. Axelrod, X. He, and J. Gao, “Domain adaptation
via pseudo in-domain data selection,” in Proceedingsof the Conference on Empirical Methods in Natural
Language Processing, ser. EMNLP ’11. Stroudsburg,
PA, USA: Association for Computational Linguistics,
2011, pp. 355–362. [Online]. Available: http://dl.acm.
org/citation.cfm?id=2145432.2145474[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30, I. Guyon, U. V.
Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates,
Inc., 2017, pp. 5998–6008.
[5] R. Sennrich, O. Firat, K. Cho, A. Birch, B. Haddow,
J. Hitschler, M. Junczys-Dowmunt, S. Läubli, V. A.
Miceli Barone, J. Mokry, and M. Nadejde, “Nematus: atoolkit for neural machine translation,” in Proceedingsof the Software Demonstrations of the 15th Conferenceof the European Chapter of the Association for Computational Linguistics. Association for Computational
Linguistics, 2017, pp. 65–68.
[6] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,” inProceedings of ICLR, 2015.
[7] R. Sennrich, A. Birch, A. Currey, U. Germann, B. Haddow, K. Heafield, A. V. Miceli Barone, and P. Williams,“The university of edinburgh’s neural mt systems for
wmt17,” in Proceedings of the Second Conference on

122
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Machine Translation, Volume 2: Shared Task Papers.
Copenhagen, Denmark: Association for Computational
Linguistics, September 2017, pp. 389–399.
[8] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers). Berlin, Germany: Association for Computational Linguistics, August 2016, pp. 1715–1725.
[9] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin,
and E. Herbst, “Moses: open source toolkit for statistical machine translation,” in Proceedings of the 45th
Annual Meeting of the ACL on Interactive Poster and
Demonstration Sessions, ser. ACL ’07. Morristown,
NJ, USA: Association for Computational Linguistics,
2007, pp. 177–180.

both worlds: Combining recent advances in neural machine translation,” in Proceedings of the 56th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2018, pp. 76–86.
[17] Z. Yang, Z. Dai, R. Salakhutdinov, and W. W. Cohen,“Breaking the softmax bottleneck: A high-rank RNN
language model,” in Proceedings of ICLR, 2018.
[18] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi,
W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey,
J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser,
S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens,
G. Kurian, N. Patil, W. Wang, C. Young, J. Smith,
J. Riesa, A. Rudnick, O. Vinyals, G. Corrado,
M. Hughes, and J. Dean, “Google’s Neural MachineTranslation System: Bridging the Gap between Humanand Machine Translation,” CoRR, vol. abs/1609.08144,
2016.

[10] A. V. Miceli Barone, J. Helcl, R. Sennrich, B. Haddow,
and A. Birch, “Deep architectures for neural machinetranslation,” in Proceedings of the Second Conference
on Machine Translation.
Copenhagen, Denmark:
Association for Computational Linguistics, September
2017, pp. 99–107.
[11] L. J. Ba, R. Kiros, and G. E. Hinton, “Layer Normalization,” CoRR, vol. abs/1607.06450, 2016.
[12] O. Press and L. Wolf, “Using the output embedding to
improve language models,” in Proceedings of the 15th
Conference of the European Chapter of the Association
for Computational Linguistics: Volume 2, Short Papers.
Association for Computational Linguistics, 2017, pp.
157–163.
[13] D. P. Kingma and J. Ba, “Adam: A Method for
Stochastic Optimization,” in The International Conference on Learning Representations, San Diego, California, USA, 2015.
[14] R. Sennrich and B. Haddow, “Linguistic input features
improve neural machine translation,” in Proceedings ofthe First Conference on Machine Translation. Berlin,
Germany: Association for Computational Linguistics,
August 2016, pp. 83–91.
[15] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, andZ. Wojna, “Rethinking the inception architecture forcomputer vision,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June
2016, pp. 2818–2826.
[16] M. X. Chen, O. Firat, A. Bapna, M. Johnson,
W. Macherey, G. Foster, L. Jones, M. Schuster,N. Shazeer, N. Parmar, A. Vaswani, J. Uszkoreit,
L. Kaiser, Z. Chen, Y. Wu, and M. Hughes, “The best of123
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The AFRL IWSLT 2018 Systems: What Worked, What Didn’t
Brian Ore, Eric Hansen, Katherine Young, Grant Erdmann and Jeremy Gwinnup
Air Force Research Laboratory{brian.ore.1, eric.hansen.5, katherine.young.1.ctr, grant.erdmann, jeremy.gwinnup.1}@us.af.mil

Abstract
This report summarizes the Air Force Research Laboratory (AFRL) machine translation (MT) and automatic speech
recognition (ASR) systems submitted to the spoken languagetranslation (SLT) and low-resource MT tasks as part of the
IWSLT18 evaluation campaign.

Table 1: Kaldi WER. Decoding was performed using a trigram LM trained on TED.
Acoustic Training Data
Speech-Translation TEDTEDLIUM
Combined

dev2010
19.8
16.9
16.6

tst2010
19.6
14.8
15.1

tst2013
30.5
22.3
23.6

1. Introduction
As part of the evaluation campaign for the 2018 International
Workshop on Spoken Language Translation (IWSLT18) [1],the AFRL Human Language Technology team applied andimproved techniques from previous workshops [2] and Conference on Machine Translation efforts [3] to the Spoken Language Translation and Low-Resource Machine Translation
tasks.

2. Spoken Language Translation
2.1. Automatic Speech Recognition
This section describes the ASR systems that were developed
for the baseline condition of the Speech Translation task. We
trained two different English systems and performed system
combination to obtain the final hypothesis for translation.
Section 2.1.1 describes that language models (LMs) that wereused for decoding and rescoring. Section 2.1.2 discusses the
Kaldi ASR system, and Section 2.1.3 describes the Hidden
Markov Model ToolKit (HTK) Tensorflow system. Finally,
Section 2.1.4 describes how we segmented the test data andperformed system combination.
2.1.1. Language Models
LMs were estimated on the provided TED data and subsets
of News Crawl 2007-2017 and News Discussions versions 13. The subset of each news corpus was selected using crossentropy difference scoring [4] with TED as the in-domain
text, and selection thresholds were chosen to use 1/8 of each
corpus to train N-gram LMs, and 1/16 of each corpus to traina recurrent neural network (RNN) LM. Interpolated bigram,trigram, and 4-gram LMs were estimated using the SRILM
Toolkit,1 and a RNN maximum entropy LM was trained using
the RNNLM Toolkit.2 The RNN included 160 hidden units,
1 http://www.speech.sri.com/projects/srilm

300 classes in the output layer, 4-gram features for the direct connections, and a hash size of 109 . The LM vocabulary
included 100,000 words that were chosen using the selectvocab tool from SRILM.
2.1.2. Kaldi System
The acoustic training data available for this year’s evaluation included the Speech-Translation TED corpus and the
TEDLIUM corpus. Based on a preliminary analysis of the
Speech-Translation TED corpus, we removed all segments
longer than 15 seconds from this corpus. The devtest and offlimit talks were sequestered from TEDLIUM, and a third data
set was created by searching the Speech-Translation TED
and TEDLIUM corpora for non-overlapping time segments.
Next, an initial set of ASR systems were trained on each data
set using the Kaldi open source speech recognition toolkit[5]. All Kaldi models discussed in this paper are based on
the chain time delay neural network (TDNN)-rectified linear
unit (ReLU) setup using i-vectors.3 Standard data augmentation methods were applied during the Mel frequency cepstralcoefficient (MFCC) feature generation stage, such as speechand volume perturbation. Each system was decoded using
the same trigram LM, which was estimated from the provided
TED data using the SRILM toolkit. Table 1 shows the word
error rate (WER) obtained on dev2010, tst2010, and tst2013.
Based on the results in Table 1, a Kaldi ASR system wastrained on TEDLIUM using the interpolated bigram LM described in Section 2.1.1. This model was then used to decode
all of the audio from the Speech-Translation TED corpus (including segments longer than 15 seconds), and the ASR derived transcripts were folded in with the TEDLIUM data, asin a semi-supervised training scenario, to build the final Kaldi
ASR system. This data set is referred to as TEDLIUM+ASR
3 http://github.com/kaldi-asr/kaldi/tree/master/egs/

2 http://www.fit.vutbr.cz/$\sim$imikolov/rnnlm

swbd/s5c/local/chain

124
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 2: Kaldi WER. Decoding was performed using an interpolated bigram LM, and rescoring was applied using aninterpolated 4-gram and RNN LM.
ASR System
Kaldi TEDLIUM
Kaldi TEDLIUM+ASR

dev2010
14.0
13.5

tst2010
11.9
11.4

tst201317.7
17.0

17x40 filterbank

i-vector

5x5 conv, 64

fc 512

maxpool, /2
3x3 conv, 64
3x3 conv, 64
3x3 conv, 64

in the remainder of this paper.
The test data was decoded as follows. First, the recognition lattices from the Kaldi bigram system were rescored withthe 4-gram LM. Next, 1000-best lists were extracted fromeach lattice and rescored with the RNN LM. The final LM
scores were obtained by linearly interpolating the log probabilities from the 4-gram and RNN LM. Interpolation weights
of 0.25 for the 4-gram and 0.75 for the RNN were chosen
based on results from previous experiments. Table 2 shows
the final WER obtained with each system. Based on these results, we used the TEDLIUM+ASR system in all remaining
experiments.

3x3 conv, 64
3x3 conv, 128, /2
3x3 conv, 128
3x3 conv, 128
3x3 conv, 128
3x3 conv, 256, /2
3x3 conv, 256
3x3 conv, 256

2.1.3. HTK-Tensorflow System
A hybrid neural network hidden Markov model (HMM)
speech recognition system was developed using Tensorflow[6] and a version of HTK4 that we modified according
to the method of [7]. First, a Gaussian mixture model(GMM)-HMM system was trained on TEDLIUM. Phonemes
were modeled using word-position-dependent state-clusteredacross-word triphones, and the final HMM set included 6000
shared states with an average of 28 mixtures per state. The
feature set consisted of 12 perceptual linear prediction (PLP)
coefficients, plus the zeroth coefficient, with mean and variance normalization applied on a per talk basis. Delta, acceleration, and third differential coefficents were appended
to form a 52 dimensional vector, and heteroscedastic linear
discriminant analysis (HLDA) was used to reduce the feature dimension to 39. Speaker adaptive training (SAT) wasapplied using constrained maximum likelihood linear regression (CMLLR) transforms, and the models were discriminatively trained using the minimum phone error (MPE) criterion.
A residual network (ResNet) was trained on the
TEDLIUM+ASR data set described in Section 2.1.2. This
network is based on the 18-layer network described in [8],with the batchnorm and ReLU activations moved to utilize
full pre-activation residual units described in [9], and an additional fully connected layer for i-vector input. Figure 1
shows the ResNet structure. A context window of 17 wasapplied to the feature input, which included 40 log filterbank
outputs normalized to zero mean and unit variance on a pertalk basis. The 100 dimensional i-vectors were extracted ona per-talk basis with an i-vector extractor that was trained on
4 http://htk.eng.cam.ac.uk

3x3 conv, 256
3x3 conv, 512, /2
3x3 conv, 512
3x3 conv, 512
3x3 conv, 512
avgpoolfc 2048softmax 6000

Figure 1: ResNet architecture based on [8, 9] with convolutional (conv), max pooling (maxpool), average pooling (avgpool), and fully connected (fc) layers. H×W is the filter size
and /2 indicates that a stride of 2 was applied.
TEDLIUM using the same procedure as our IWSLT 2015 system [10]. Cross entropy training was performed using a minibatch size of 512 and an initial learning rate of 0.0005 that
was adjusted according to the QuickNet newbob algorithm.5
Recognition lattices were produced using HDecode with
the interpolated trigram LM described in Section 2.1.1, andthen rescored with the 4-gram and RNN LM using the sameprocedure as the Kaldi system. Next, confidence scores were
estimated at the acoustic frame level by aligning the 20-best
hypotheses for each utterance and counting the number ofmatching HMM states. An adapted ResNet was estimated
5 http://www.icsi.berkeley.edu/Speech/faq/nn-train.html

125
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 3: HTK-Tensorflow WER. Decoding was performed
using an interpolated trigram LM, and rescoring was applied
using an interpolated 4-gram and RNN LM.
ASR System
ResNetResNet-Adapted

dev2010
15.4
15.3

tst2010
12.9
12.6

Table 4: WER using the updated test segmentation method.
The final ASR hypothesis was obtained using N-best
ROVER.
ASR System
Kaldi TEDLIUM+ASR
ResNet-Adapted
N-best ROVER

tst2013
17.5
16.1

for each talk using frames that had a confidence score of 0.9
or higher and a single epoch of cross-entropy training with alearning rate of 0.0000625. Finally, the test set was decoded asecond time and LM rescoring was reapplied. Table 3 shows
the WER on dev2010, tst2010, and tst2013.
2.1.4. Test Segmentation and System Combination
The WER results reported in the previous sections were obtained by evaluating each ASR system on the automaticallyderived segments from the baseline implementation.6 It was
discovered that the segment boundaries did not always alignwith non-speech; therefore, we decided to use an alternative
segmentation method.
A neural network based speech activity detector (SAD)
was developed using Tensorflow. The SAD was trained on
40 hours from the TEDLIUM corpus using the automaticallygenerated phoneme alignments from the HTK GMM-HMM
system to define the speech/non-speech boundaries. The network included a context window of 41 frames on the input,
a hidden layer of 1024 neurons with rectified linear activation functions, and 2 output units corresponding to speechand non-speech. The feature set consisted of 40 log filterbank
outputs that were normalized to zero mean and unit variance.
Automatic segmentation of the test data was performed byevaluating the SAD, applying a dynamic programming algorithm to choose the best sequence of states, and defining
utterance boundaries at the midpoint of each non-speech segments longer than 0.5 seconds. Lastly, non-speech segments
longer than 1.0 second were trimmed from each utterance.
The final hypothesis was selected by applying N-best recognizer output voting error reduction (ROVER) to the output from the Kaldi TEDLIUM+ASR and HTK-Tensorflow
ResNet-Adapted system. Table 4 shows the WER obtain using the updated segmentation. Comparing Table 4 with theresults in Table 2 and 3, we can see that the updated segmentation method provided a substantial improvement in WER.

dev2010
9.5
11.2
9.5

tst2010
7.7
8.6
6.9

tst2013
12.8
11.2
9.8

Table 5: English-German cased BLEU scores for the SLT
task. For comparison purposes, this table includes the scoresobtained with the reference English source text.
English Transcripts
Reference
ASR

dev2010
27.10
18.48

tst2010
27.40
17.20

tst2013
28.83
18.40

2.3. Machine Translation
Lastly, a Marian [11] neural machine translation system was
employed to translate the repunctuated text from English into
German. This system was trained on the 41 million lines ofpreprocessed data provided by the WMT18 organizers for thenews-translation shared task[12]. The data was truecased foruniformity, then a byte-pair encoding (BPE) [13] model wastrained jointly on the source and target data with 90k merge
operations.
As described in our WMT18 news-task efforts[3], weused the same parameters in training our Marian transformer[14] model:• We used an encoding depth of 6 layers and a decoding
depth of 6 layers.
• We used 8 transformer heads.
• We held the vocabulary size constant during training to
90k entries each for source and target.
• We held the word embedding dimensionality to 512 forall models.
• We used 1024 units in the hidden layer (where appropriate).
• We exclusively used the WMT newstest2014 test set
for validation.

2.2. ASR Postprocessing

2.4. Results

We employed the provided SLT.KIT punctuator component
to re-punctuate our ASR output before applying a truecaser
model to induce the most common case for an English word
before translating with the Marian section described in thenext section.

Results of scoring our repunctuated, translated ASR output
and various references are shown in Table 5.

6 http://github.com/isl-mt/SLT.KIT

3. Low-Resource Machine Translation
For the low-resource translation task, we tried a variety of
approaches with Marian [11], and Moses[15] toolkits. We

126
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 6: Corpus size for each language pair in training corpus
Lang. Pair
Basque–English
French–English
Spanish–English
Total corpus

Lines
5,623
288,366
278,297572,286

Table 7: Corpus size for each language pair in backtranslation
training corpus
Lang. Pair
English–BasqueFrench–Basque
Spanish–Basque
Basque–English OpenSubtitles
Total corpus

Lines
5,623
6,948
6,668
458,380
477,619

tried additional approaches with stemming and morphological processing, but systems trained with data processed in thismanner were not ready in time for evaluation submission.

lated systems.

3.1. Common Training Corpus

3.2.1. Sentence-Weighted training

For many of the experiments across different toolkits and systems, we constructed a common training corpus with uniform
preprocessing in order to reduce variables when comparing
different conditions.
Using the provided parallel Basque–English, French–
English, and Spanish–English TED corpora [16], we construct a training corpus containing all three language pairs.
Sizes of each portion of the training corpus are listed in Table
6. A joint BPE model was trained with 89,500 merge operations on the combination of all languages in the training data,then applied to the unified training corpus.
A similar corpus for use in backtranslation was constructed from the provided English–Basque, French–Basque,and Spanish–Basque corpora. Due to the small size of eachof these component corpora, we also add the Basque–English
portion of the OpenSubititles Corpus7 . Sizes of each portionof this backtranslation training corpus are listed in Table 7.
The BPE model from the ‘forward’ was used to segment thesource and target data.
For some Marian experiments, we also constructedmonolingual Basque and English corpora for use in constructing pretrained word embeddings. We use 50 million
lines from the English monolingual CommonCrawl corpus
selected for use in backtranslation from our WMT17 newstask efforts [17]. Additional monolingual Basque data wastaken from the Commoncrawl website8 and language-filteredusing a modified C implementation9 of the algorithm outlinedin [18], yielding a Basque monolingual corpus of 38 million
lines. We then apply BPE to each of these corpora with thesame model as above and use word2vec [19] to generate 512dimension word embeddings compatible with our settings inMarian.

We used the “forward” corpus outlined in Section 3.1 to train
Marian systems with the same network parameters as outlined in the SLT translation system in Section 2.3. In Table 8, we note our baseline system (#1) scored 11.11 cased
BLEU on dev2018. Next, we utilize the sentence-weighting
feature of Marian that allows each sentence to be assigned a“weight” to determine how much of an effect each will haveduring training. A score of 1.0 is assigned to sentences fromthe Basque–English portion of the training corpus, French–
English and Spanish–English sentences are assigned a scoreof 0.5. The system trained with these weights (#2) shows a+2.41 increase in BLEU.
Using the same data as system #2, we train a system thatuses BEER [22] as the validation metric. While we have seen
performance gains using this tactic in other work, here theresulting system(#3) performs -0.75 BLEU worse than theprevious system.
Next, we consider averaging and ensembling of models.
We take the 4-best model checkpoints from system #2 andaverage them into a single model, resulting in system #4’s
+0.87 BLEU gain over system #2.
Lastly, we decode with an ensemble of system #4 and amodel averaged from the four best checkpoints of system #3,
resulting in a BLEU score of 15.45. This system (#5) was
then submitted as our entry to the low-resource MT task.

3.2. Marian Systems
We spent the bulk of our efforts building systems with theMarian toolkit, experimenting with a variety of settings along
two major categories: Sentence-weighting and backtrans7 http://www.opensubtitles.org
8 http://www.commoncrawl.org

9 https://github.com/saffsd/langid.c

3.2.2. Backtranslated training corpus
As a contrast, we use the “backtranslation” corpus to train ashallow “s2s” Marian system that translates English, Spanish, and French into Basque. We then translate a 2 million
line portion of the English monolingual corpus described in
3.1 into something resembling Basque and then use the combination of the two in conjunction with the small amount ofprovided Basque–English data to train two Marian “bi-deep”[20, 21] systems, both using BEER [22] as the training validation metric. These systems are listed as #6 (without pretrained word embeddings) and #7 (with pretrained word embeddings) in Table 8. We note that pretrained word embedding system scores -1.46 cased BLEU lower than the equivalent system without the pretrained embeddings, counter to127
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 8: Results for various MT systems decoding Basque–English dev2018 measured in cased BLEU. Our submission
system is highlighted in bold text.
#
1.
2.
3.
4.
5.
6.
7.
8.

Systemmarian-eseufr-transmarian-eseufr-trans-weightmarian-eseufr-trans-weight-beervalid
marian-eseufr-trans-weight-avg4
marian-eseufr-trans-weight-avg4X2
marian-bt-bideep-beervalid
marian-bt-bideep-preembed-beervalid
moses-bt-bpe

BLEU
11.11
13.52
12.77
14.39
15.45
11.23
9.77
14.06

5. References[1] M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, R. Cattoni, and M. Federico, “The IWSLT 2018 Evaluation
Campaign,” in Proceedings of the 15th International
Workshop on Spoken Language Translation (IWSLT2018), Bruges, Belgium, October 2018.
[2] M. Kazi, , E. Salesky, B. Thompson, J. Taylor, J. Gwinnup, T. Anderson, G. Erdmann, E. Hansen, B. Ore,K. Young, and M. Hutt, “The MITLL-AFRL IWSLT2016 systems,” in Proc. of the 13th International Workshop on Spoken Language Translation (IWSLT’16),
Seattle, Washington, December 2016.

our experience with our WMT18 systems.
3.3. Moses System
Using both the provided Basque–English data and the backtranslated corpus outlined in Section 3.2.2 we train a Moses
system in a similar vein to the one employed in our WMT18
submission: This system employed a hierarchical reorderingmodel [23] and 5-gram operation sequence model [24]. The
5-gram English language model was trained with KenLM on
the constrained monolingual corpus from our WMT15 [25]efforts. Our uniform BPE model used was applied to the parallel training data, but the language modelling corpus used theRussian–English joint BPE model from our WMT18 submission, possibly degrading performance due to this BPE mismatch. System weights were tuned with the Drem [26] optimizer using the “Expected Corpus BLEU” (ECB) metric.
This system, listed as #8 in Table 8 performs better than
the two other Marian-based backtranslation systems (#6 and#7).
3.4. Results
Results of various systems described in the above sections
are listed in Table 8. Our final submission system (#5) ishighlighted in bold text.

4. Conclusions
Our experimentation this year show positive results in spoken
language translation, especially our ASR component. However, for the low-resource MT task, we note that various approaches we have previously employed with great effect inhigh-resource conditions need further adaptation and refinement when scaling down to extremely low-resource conditions.
Opinions, interpretations, conclusions and recommendations are thoseof the authors and are not necessarily endorsed by the United States Government. Cleared for public release on 4 Oct 2018. Originator Reference
Number: RH-18-118975 Case Number: 88ABW-2018-4946.

[3] J. Gwinnup, T. Anderson, G. Erdmann, and K. Young,“The afrl wmt18 systems: Ensembling, continuation,
combination,” in Proceedings of the Third Conference
on Machine Translation. Brussels, Beligum: Association for Computational Linguistics, October 2018.
[4] R. Moore and W. Lewis, “Intelligent selection of language model training data,” in Association Computational Linguists 2010 Conference Short Papers, Uppsala, Sweden, July 2016.
[5] D. Povey, A. Ghoshal, G. Boulianne, L. Burget,
O. Glembek, N. Goel, M. Hannemann, P. Motlicek,
Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and
K. Vesely, “The Kaldi Speech Recognition Toolkit,” inProceedings of the Automatic Speech Recognition andUnderstanding Workshop, 2011.
[6] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,
C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin,
S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray,
C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever,K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan,
F. Viégas, O. Vinyals, P. Warden, M. Wattenberg,
M. Wicke, Y. Yu, and X. Zheng, “TensorFlow:
Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org.
[Online]. Available: https://www.tensorflow.org/[7] G. Dahl, D. Yu, L. Deng, and A. Acero, “ContextDependent Pre-trained Deep Neural Networks for Large
Vocabulary Speech Recognition,” IEEE Transactions
on Audio, Speech, and Language Processing, vol. 20,
pp. 30–42, January 2012.
[8] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual
learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, Nevada, June 2016.

128
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[9] ——, “Identity mappings in deep residual networks,” inProceedings of the 14th European Conference on Computer Vision (ECCV), Amsterdam, Netherlands, October 2016.
[10] M. Kazi, B. Thompson, E. Salesky, T. Anderson,
G. Erdmann, E. Hansen, B. Ore, K. Young, J. Gwinnup, M. Hutt, and C. May, “The MITLL-AFRL IWSLT2015 systems,” in Proc. of the 12th International Workshop on Spoken Language Translation (IWSLT’15), Da
Nang, Vietnam, December 2015.
[11] M. Junczys-Dowmunt, R. Grundkiewicz, T. Dwojak,
H. Hoang, K. Heafield, T. Neckermann, F. Seide,
U. Germann, A. F. Aji, N. Bogoychev, A. F. T.
Martins, and A. Birch, “Marian: Fast neural machinetranslation in c++,” in Proceedings of ACL 2018, System Demonstrations. Association for Computational
Linguistics, 2018, pp. 116–121. [Online]. Available:
http://aclweb.org/anthology/P18-4020[12] O. Bojar, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, P. Koehn, and C. Monz, “Findings of the2018 conference on machine translation (WMT18),”
in Proceedings of the Third Conference on MachineTranslation. Brussels, Belgium: Association for Computational Linguistics, October 2018.
[13] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers). Association for Computational Linguistics,
2016, pp. 1715–1725.
[14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and
I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems
30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett,
Eds. Curran Associates, Inc., 2017, pp. 5998–
6008. [Online]. Available: http://papers.nips.cc/paper/
7181-attention-is-all-you-need.pdf[15] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin,
and E. Herbst, “Moses: Open source toolkit for statistical machine translation,” in Proc. of the 45th Annual
Meeting of the ACL on Interactive Poster and Demonstration Sessions, ser. ACL ’07, 2007, pp. 177–180.
[16] M. Cettolo, C. Girardi, and M. Federico, “Wit3 : Web
inventory of transcribed and translated talks,” in Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), Trento, Italy,
May 2012, pp. 261–268.

[17] J. Gwinnup, T. Anderson, G. Erdmann, K. Young,
M. Kazi, E. Salesky, B. Thompson, and J. Taylor, “Theafrl-mitll wmt17 systems: Old, new, borrowed, bleu,”in Proceedings of the Second Conference on MachineTranslation, Volume 2: Shared Task Papers. Copenhagen, Denmark: Association for Computational Linguistics, September 2017, pp. 303–309. [Online]. Available: http://www.aclweb.org/anthology/W17-4728[18] M. Lui and T. Baldwin, “Cross-domain feature selection
for language identification,” in Proceedings of 5thInternational Joint Conference on Natural Language
Processing. Asian Federation of Natural Language
Processing, 2011, pp. 553–561. [Online]. Available:
http://www.aclweb.org/anthology/I11-1062[19] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector
space,” International Conference on Learning Representations (ICLR) Workshop, 2013.
[20] A. V. Miceli Barone, J. Helcl, R. Sennrich, B. Haddow, and A. Birch, “Deep architectures for neuralmachine translation,” in Proceedings of the Second
Conference on Machine Translation. Association for
Computational Linguistics, 2017, pp. 99–107. [Online].
Available: http://aclweb.org/anthology/W17-4710[21] R. Sennrich, A. Birch, A. Currey, U. Germann,
B. Haddow, K. Heafield, A. V. Miceli Barone, and
P. Williams, “The university of edinburgh’s neural mt
systems for wmt17,” in Proceedings of the Second
Conference on Machine Translation, Volume 2: Shared
Task Papers. Copenhagen, Denmark: Association
for Computational Linguistics, September 2017, pp.
389–399. [Online]. Available: http://www.aclweb.org/
anthology/W17-4739[22] M. Stanojević and K. Sima’an, “Fitting sentence leveltranslation evaluation with many dense features,” inProceedings of the 2014 Conference on Empirical
Methods in Natural Language Processing (EMNLP).
Doha, Qatar: Association for Computational Linguistics, October 2014, pp. 202–206. [Online]. Available:
http://www.aclweb.org/anthology/D14-1025[23] M. Galley and C. D. Manning, “A simple and effective hierarchical phrase reordering model,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, ser. EMNLP ’08, 2008, pp.
848–856.
[24] N. Durrani, H. Schmid, and A. Fraser, “A joint sequence translation model with integrated reordering,”
in Proc. of the 49th Annual Meeting of the Association for Computational Linguistics (ACL ’11), Portland,
Oregon, June 2011, pp. 1045–1054.

129
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[25] J. Gwinnup, T. Anderson, G. Erdmann, K. Young,
C. May, M. Kazi, E. Salesky, and B. Thompson,“The afrl-mitll wmt15 system: There’s more than
one way to decode it!” in Proceedings of the TenthWorkshop on Statistical Machine Translation. Lisbon,
Portugal: Association for Computational Linguistics,
September 2015, pp. 112–119. [Online]. Available:
http://aclweb.org/anthology/W15-3011[26] G. Erdmann and J. Gwinnup, “Drem: The AFRL submission to the WMT15 tuning task,” in Proc. of theTenth Workshop on Statistical Machine Translation,
Lisbon, Portugal, September 2015, pp. 422–427.

130
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

KIT’s IWSLT 2018 SLT Translation System
Matthias Sperber, Ngoc Quan Pham, Thai Son Nguyen, Jan Niehues,
Markus Müller, Thanh-Le Ha, Sebastian Stüker, Alex Waibel
Institute for Antrophomaics and Robotics, Karlsruhe Institut of Technology
firstname.lastname@kit.edu

Abstract
This paper describes KIT’s submission to the IWSLT2018 Translation task. We describe a system participatingin the baseline condition and a system participating in theend-to-end condition. The baseline system is a cascade of
an ASR system, a system to segment the ASR output and aneural machine translation system. We investigate the combination of different ASR systems. For the segmentation andmachine translation components, we focused on transformerbased architectures.

1. Introduction
The Karlsruhe Institute of Technology participated in
the IWSLT 2018 Evaluation Campaign with systems forEnglish→German Speech Translation task. We submitted
system to both conditions: the baseline condition and theend-to-end condition.
The submission to the baseline condition is based on the
cascaded approach described in [1]. In this evaluation campaign, we investigated the combination of different ASR systems. Furthermore, we investigated the use of transformerbased models.
This paper is structured as follows. In Section 2, we describe different speech recognition systems we employed in
the campaign and how we combined them. Afterwards, wegive a detailed description of the segmentation approach in
Section 3 and the machine translation system in Section 4.
Finally, in Section 5 we describe the end-to-end speech translation model. At the end of the paper we report the results
and finish with a conclusion.

2. Speech Recognition
In this year’s evaluation, we built three different types of automatic speech recognition systems. All the systems weretrained using the data from the TED-LIUM Corpus version 2[2].
2.1. Hybrid Model
Different from previous years, this year we built only one
single HMM-based hybrid model for the speech recognition task. The hybrid acoustic modelling is constructed bystacking 5 LSTMs layers of 320 units, a projection layerof 200 units and a softmax layer to classify 8000 contextdependence phone (CD-Phone) states. As traditional approach, we used Viterbi forced alignment to provide CDPhone state labels for the training data and the acoustic model
was trained using only cross-entropy loss function.
For model training, we use SGD with an initial learning
rate of 0.004 for 8 epochs and degrade it with a factor of 0.8
for other 8 epochs. We use a momentum term of 0.9 while
dropout is set to 10%. Only 40 features of Mel-filterbank
coefficients are fed into the LSTMs network every timestep,
we did not employ any further speaker adaptation features.
After the model was successfully trained, we performed
the traditional beam search decoding with the employmentof the 4-gram language model. We used Janus Recognition Toolkit (JRTK) [3] as the decoding framework while thelanguage model is built by Cantab research group [4] fromWMT data.
2.2. CTC Model
Our CTC-based [5] ASR model is similar to the system described by [6]. The input to the model are 40-dimensional
Mel-filterbank coefficients. We used every third speech feature of our input sequence and randomly chose the start offset during training, which has the advantage of a lower input
sequence length. We trained the model to predict Byte-Pair
Units, also referred to as Byte-Pair Encoding (BPE) [7].
The CTC-based model consists of four bidirectional
LSTM layers with 400 units in each direction followed bya softmax layer. The size of the softmax layer depends on
the number of BPE units we created. We used a dropout rateof 0.25 for all LSTM layers. We trained two models basedon BPE units with 300 (small model) and 10,000 (big model)
merges, respectively.
We used SGD with a learning rate of 0.0005 and a momentum term of 0.9 for training. The learning rate is halved
whenever the validation token error rate does not decrease by
more than 0.1%. We first trained the small model and initialized the parameters of the big model’s BiLSTM layers using
the smaller model’s ones. We decoded the model by greedily
selecting the most likely output at each time step.

131
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

2.3. Encoder-Decoder Model
Our attentional ASR model follows the listen-attend-spell [8]architecture and is similar to the system described by [9].
The model is implemented with XNMT. Compared to a conventional neural machine translation architecture, we replace
the encoder with a 4-layer bidirectional pyramidal encoderwith a total downsampling factor of 8. The layer size is set to512, the target embedding size is 64, and the attention uses an
MLP of size 128. Input to the model are Mel-filterbank features with 40 coefficients. For regularization, we apply variational dropout of rate 0.3 in all LSTMs, and word dropoutof rate 0.1 on the target side [10]. We also fix the target embedding norm to 1 [11]. For training, we use Adam [12] withinitial learning rate of 0.0003, which is decayed by factor 0.5
if no improved WER is observed. To further facilitate training, label smoothing [13] is applied. For the search, we use
beam size 20 and length normalization with the exponent set
to 1.5.
2.4. Rover
To combine the outputs of the different ASR systems, weused ROVER [14]. It operates on the final system output, theCTM-files. Multiple merging strategies exist to combine theoutputs based on a majority vote. It is, e.g. possible to takeconfidences into account to further fine-tune the merging process. The key idea of ROVER is that different systems tend
to produce different errors, but that no two systems produce
the same error. The more different the systems those outputs
are combined are, the better the result will be. Systems being
very similar on the other hand will not benefit much from thesystem combination as they are likely to generate the sameerrors.
We here combined three different architectures: a traditional HMM-based ASR system, a RNN/CTC based one andan encoder-decoder based one. Hence, combining the outputs improved the WER due to the diversity of the systemarchitectures.

3. Segmentation
Automatic speech recognition (ASR) systems typically do
not generate punctuation marks or reliable casing. Using theraw output of these systems as input to MT causes a performance drop due to mismatched train and test conditions. To
create segments and better match typical MT training conditions, we use a monolingual NMT system to add sentenceboundaries, insert proper punctuation, and add case whereappropriate before translating [15].
The idea of the monolingual machine translation system
is to translate from lower-cased, unpunctuated text into text
with case information and punctuation. Since we do not have
any information about the sentence boundaries when inserting the punctuation and case information, we also remove
them from the training data. Therefore, in the first step of
the pre-processing, we randomly segment the source corpusof the training data into chunks of 20 to 30 words. Basedon this randomly segmented corpus, we build the input andoutput data for the monolingual translation system.
For the input data, we remove all punctuation marks andlowercase all words. Since we will get lower-cased input,
we cannot use the same byte-pair encoding [7] as for the machine translation system. Therefore, we train a separate bytepair encoding on the lower-cased source data with a code sizeof 40k. To summarize, the source sequence consists of lowercased BPE units without any punctuation.
For the target side, we do not want to change the words
in the output sentence, but only add case and punctuation information. Therefore, we replace the sentence by featuresindicating case with punctuation attached. Every word is replaced by a letter U or L, whether it is upper-cased or lowercased. Furthermore, punctuation marks following the word
are directly attached to the letter.
At test time, we follow the sliding window technique described by [16]. Therefore, we created a test set with segments of length 10 starting with every word on the input
data. This means, that except for the beginning and the end of
the document, every word occurs ten times, at all positions
within the segment. This of course dramatically increases
the number of sentences in the test data. In the second step,
we generate the target features by applying the monolingualtranslation system. In a post-processing step, we case theword as it most frequently occurs in the output. We insert
punctuation marks, if there is at least one punctuation markafter the word in one of the 10 segments containing this word.
If different punctuation marks are predicted, we take the mostfrequent one. Finally, if the punctuation mark is an end ofsentence punctuation mark {”.”,”!”,”?”}, we also start a new
segment. The segmented test data with case and punctuation
information is passed on to the machine translation system.
This year, we used a transformer-based NMT system togenerated the punctuation marks.1 . For the encoder and decoder we used 12 layers each using a hidden size of 512 andan inner size of the transformer model of 1024. We applied
dropout and trained the models using adam. We first trained
the system on the source side of the parallel data. We used
the EPPS corpus, NC corpus and a filtered version of theparaCrawl corpus. In a second step, we fine-tuned the modelon the TED corpus.

4. Machine Translation
Data preprocessing Our training data, while consisting the
TED Talks provided by the evaluation campaign, also includes the following corpora: Europarl (1.8M sentences),
News Commentary(280K), Rapid (1.2M), Common crawl(2.2M), the backtranslation data from University of Edinburgh (3.M) and the Paracrawl data (30M). The Paracrawl
data is filtered by training a translation model to identify sentences with low likelihood. The final data size is around 36M
1 https://github.com/isl-mt/NMTGMinor

132
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

sentences, with basic preprocessing steps being truecasing,tokenizing, and BPE splitting with BPE size of 40K. The
development set is newstest2013 to newstest2016 from theWMT datasets for training the big models. In the adaptation
phase, we use the TED talks of dev2010 to validate our models.
Modeling Our translation models are constructed with selfattention encoders and decoders, known as the Transformer
networks, following the work of [17]. In this work, we extend the depth of the standard Tranformers, thanks for the
residual design combined with layer normalization schemes
allowing gradients to flow smoothly. Thanks to the hugeamount of data as shown above, we were able to train models
up to 32 blocks and still yield meaningful improvement.
Our hyper-parameters of the Transformer models (except
depth) follow the Base configuration of the original work.
The layer size for hidden layers is 512, while the the inner size of feed-forward network inside each block is 2048.
The attention layers (including self-attention and attention
between decoders and encoders) are multi-head attention layers with 8 heads. We also added label smoothing to regularize the cross-entropy loss. For the network depth, we trainedmodels consisting of 4, 8, 6, 12, 16 and 32 blocks (for both
encoder and decoder). Not only are deep models very demanding in terms of computation, they also consume a considerate amount of memory. In order to make training feasible, we used the checkpointing technique [18] by employ
re-computation of the network activations during the backward pass to reduce the memory cost for the models.
Training procedure We group mini-batches to fill up ourGPU’s memory depending on the network size. 12-layer
models can fit the memory with batch size containing 2048
words, while deeper models requires batch size reduction to
avoid out-of-memory. For updating the networks’ parameters, we accumulate gradients up to 25000 target words
before doing an update. The learning rate is scheduled asin [17] but we doubled the initial learning rate and extend thewarm up duration to 8000 steps. All models including the32−layer config train with 100000 updates. Each model, except the 32-layer one has an additional variation with dropout(added to the residual connection and the inner feed-forward
hidden layers).
Domain adaptation After training on all datasets, we further fine tune each model on the TED Talks specifically.
Such technique is known to improve the model’s performance greatly on the specifically adapted domain [19].
Noise adaptation Since the ASR output is fundamentallydifferent than the collected natural data, we apply a noisemodel [20] on the TED training data which randomly replacewords by sampling. The model is further fine-tuned on the
noisy data in the same fashion as domain adaptation.
Final models The output is generated from the ensemble offive models: 12-layer, 12-layer with dropout, 16-layer, 16layer with dropout and 32-layer.

5. End-to-End Models
We extend the attentional ASR described under Section 2.3
to perform translation by replacing the source-language target tokens by tokens from the target language. We use a refined encoder that performs downsampling to make memory
requirements manageable and adds depth for improved accuracy, following one of the variants described by [21]: westack several blocks consisting of a bidirectional LSTM, a
network-in-network (NiN) projection, and batch normalization. After the last block, we add a final bidirectional LSTM
layer. NiN denotes a simple linear projection applied at every
time step, performing downsampling by concatenating pairsof adjacent projection inputs.
For better results and to be able to include the TEDLIUM
corpus in the training, we devise a multi-task training strategy that trains auxiliary models on related tasks while sharing a subset of the parameters with the main ST model. Precisely, besides the main ST task we include an ASR task thatshares encoder and attention, an MT task that shares attention
and decoder, and an transcript auto-encoder task that shares
only the attention. The ASR task is trained on the TEDLIUM
corpus, whereas the other tasks are trained the respective subset of the 3-way TED corpus provided for the IWSLT 2018
evaluation. It should be noted that we did not put any efforts
into cleaning this data, despite it being relatively noisy.

6. Experiments
We evaluated the models presented in the last section on theprovided test sets. Note that for all experiments we used theaudio segmentation tool [22] provided in the IWSLT docker
container.
6.1. Cascaded
Our experiments involve different configurations regarding
three main components in the cascade. For the ASR component, we present three different setups: the ROVER combination of two CTC and one Encoder-Decoder systems(dubbed as ROVER-1) and finally the ROVER combinationof CTC,Encoder Decoder and Hybrid systems (dubbed as
ROVER-2).
For the text segmenter, we showed the models trained on
two data sizes (small and large), together with the larger
models being adapted with domains and adapted with noise.
Similarly, we showed the Translation models with additional
adaptation towards domain and noisy inputs.
ASR
ROVER-1
HYBRID
ROVER-2

WER
21.2%
17.6%
16.7%

Table 1: Word-Error Rate of different ASR configurations on
the tst2014 English set.

133
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

ASR
ROVER-1
ROVER-1
ROVER-1
ROVER-1
ROVER-1
HYBRIDHYBRID
ROVER-2
ROVER-2

SEG
Small
Small
Large
D.Adapted
N.Adapted
D.Adapted
N.Adapted
D.Adapted
N.Adapted

MT
Transformer
D. Adapted
D. Adapted
D. Adapted
N.Adapted
N.Adapted
N.Adapted
N.Adapted
N.Adapted

BLEU
13.77
16.26
18.2
19.15
19.32
21.33
21.41
22.61
21.35

man Speech Translation task.
For the cascaded approach, we see that the combinationof several ASR systems reaches the best performance. Furthermore, we get the best single performance by using a hybrid model.
For the end-to-end model, we cannot achieve the same
performance as the cascaded approach. One challenge is theintegration of the significantly larger data available for the
cascaded models.

8. Acknowledgements
Table 2: SLT English→German results. We report the BLEU
scores (after re-segmentation) on the tst2014 test data. Note:
Noise-adapted models (N.Adapted) were already adapted to
the TED domain (D.Adapted) previously.
Regarding the whole cascade performance, as can beseen from Table 2, the score dramatically increased with thehelp of domain adaptation (2.5 BLEU points improved from
adapting the translation model, and an additional 3.2 points
from having a stronger adapted segmentation model (Large
into D.Adaptation). Additional noise-adaptation on the segmentation and translation models improves the result by 0.2.
We can also see that the HYBRID ASR model is much better
than the ROVER-1 configuration, thanks to 17% improvement in word error rate on the English speech input. As aresult, the whole cascade is improved by significant 3 BLEU
points. The best ASR configuration - ROVER-2 - finalized
the best result at 22.61. Notably, we have to use the segmentation model configuration without noisy-adaptation toachieve this, the counterpart fell short by 1 BLEU point.
6.2. End-to-End
We conduct preliminary experiments on the well-establishedFisher Spanish-English Speech Translation Corpus [23] toconfirm the model’s accuracy. We obtain 35.3 BLEU points
on Fisher/Test, 2.8 points better than a cascaded model using a similar architecture and the same training data. We
then train the end-to-end model described in Section 5 on
TEDLIUM2 for the ASR task, and on IWSLT2018’s provided end-to-end TED data for the remaining tasks. We test
the resulting model on the tst2013 dataset and obtain 10.3
BLEU points without casing, and 9.3 BLEU points casesensitive scoring. This is still much worse than the cascadedmodels described above, despite the promising preliminary
results. As potential reasons we identify the rather noisy provided training data, mismatch between the manual segmentation at training time and the automatic segmentation at test
time, and the lack of additional data (beyond TED) includedin the training.

7. Conclusions
In this evaluated we build a cascaded speech translation
model as well as an end-to-end model for the English to Ger-

9. References[1] T. Zenkel, M. Sperber, J. Niehues, M. Müller,
N.-Q. Pham, S. Stüker, and A. Waibel, “OpenSource Toolkit for Speech to Text Translation,” The
Prague Bulletin of Mathematical Linguistics, vol.
111, pp. 125–135, October 2018. [Online]. Available:
https://ufal.mff.cuni.cz/pbml/111/art-zenkel-et-al.pdf[2] A. Rousseau, P. Deléglise, and Y. Esteve, “Enhancing the ted-lium corpus with selected data for language
modeling and more ted talks,” in LREC, 2014.
[3] M. Finke, P. Geutner, H. Hild, T. Kemp, K. Ries, and
M. Westphal, “The karlsruhe-verbmobil speech recognition engine,” in Acoustics, Speech, and Signal Processing, 1997. ICASSP-97., 1997 IEEE International
Conference on, vol. 1. IEEE, 1997, pp. 83–86.
[4] W. Williams, N. Prasad, D. Mrva, T. Ash, and T. Robinson, “Scaling recurrent neural network language models,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp.
5391–5395.
[5] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification,” in Proceedings of the 23rd international conference on Machine learning - ICML 2006. ACM Press, 2006.
[6] T. Zenkel, R. Sanabria, F. Metze, and A. Waibel, “Subword and crossword units for ctc acoustic models,” Interspeech, 2018.
[7] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers). Association for Computational Linguistics,
2016.
[8] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary
conversational speech recognition,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, mar 2016.

134
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[9] G. Neubig, M. Sperber, X. Wang, M. Felix,
A. Matthews, S. Padmanabhan, Y. Qi, D. S. Sachan,
P. Arthur, P. Godard, et al., “Xnmt: The extensible neural machine translation toolkit,” Boston, USA, 2018.
[10] Y. Gal and Z. Ghahramani, “A theoretically grounded
application of dropout in recurrent neural networks,”
in Neural Information Processing Systems Conference(NIPS), Barcelona, Spain, 2016.
[11] T. Nguyen and D. Chiang, “Improving lexical choice inneural machine translation,” in Proceedings of the 2018
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, 2018.
[12] D. P. Kingma and J. Ba, “Adam: A Method for
Stochastic Optimization,” in International Conference
on Learning Representations (ICLR), Banff, Canada,
2015.
[13] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, andZ. Wojna, “Rethinking the inception architecture forcomputer vision,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE,jun 2016.

Spoken Language Translation (IWSLT), Seattle, WA,
2016.
[20] M. Sperber, J. Niehues, and A. Waibel, “Toward Robust Neural Machine Translation for Noisy Input Sequences,” in International Workshop on Spoken Language Translation (IWSLT), Tokyo, Japan, 2017.
[21] Y. Zhang, W. Chan, and N. Jaitly, “Very Deep Convolutional Networks for End-to-End Speech Recognition,”in International Conference on Acoustics, Speech andSignal Processing (ICASSP), 2017.
[22] M. Rouvier, G. Dupuy, P. Gay, E. Khoury, T. Merlin, and S. Meignier, “An open-source state-of-the-art
toolbox for broadcast news diarization,” in Interspeech,
2013.
[23] M. Post, G. Kumar, A. Lopez, D. Karakos,
C. Callison-Burch, and S. Khudanpur, “Improved
Speech-to-Text Translation with the Fisher andCallhome Spanish–English Speech Translation Corpus,” in International Workshop on Spoken Language Translation (IWSLT), Heidelberg, Germany,2013. [Online]. Available: http://cs.jhu.edu/ gkumar/papers/post2013improved.pdf[14] J. Fiscus, “A post-processing system to yield reduced
word error rates: Recognizer output voting error reduction (ROVER),” in 1997 IEEE Workshop on Automatic
Speech Recognition and Understanding Proceedings.
IEEE, 1997.
[15] E. Cho, J. Niehues, and A. Waibel, “NMT-based segmentation and punctuation insertion for real-time spoken language translation,” in Interspeech 2017. ISCA,
aug 2017.
[16] ——, “Segmentation and punctuation predictionin speech language translation using a monolingual translation system,” in Proceedings of theNinth International Workshop on Spoken Language Translation (IWSLT), 2012. [Online]. Available:
https://isl.anthropomatik.kit.edu/pdf/Cho2012.pdf[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” arXiv preprint
arXiv:1706.03762, 2017.
[18] T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training
deep nets with sublinear memory cost,” arXiv preprint
arXiv:1604.06174, 2016.
[19] E. Cho, J. Niehues, T.-L. Ha, M. Sperber, M. Mediani,and A. Waibel, “Adaptation and combination of nmt
systems: The kit translation systems for iwslt 2016,”
in Proceedings of the ninth International Workshop on

135
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Alibaba Speech Translation Systems for IWSLT 2018
Nguyen Bach*, Hongjie Chen*, Kai Fan*, Cheung-Chi Leung*,
Bo Li*, Chongjia Ni*, Rong Tong*, Pei Zhang*,
Boxing Chen, Bin Ma, Fei Huang
Machine Intelligence Technology Lab, Alibaba Group
firstname.lastname@alibaba-inc.org

Abstract
This work describes the En→De Alibaba speech translation
system developed for the evaluation campaign of the International Workshop on Spoken Language Translation (IWSLT)
2018. In order to improve ASR performance, multiple ASR
models including conventional and end-to-end models arebuilt, then we apply model fusion in the final step. ASR preand post- processing techniques such as speech segmentation, punctuation insertion, and sentence splitting are foundto be very useful for MT. We also employed most techniquesthat have proven effective during the WMT 2018 evaluation,such as BPE, back translation, data selection, model ensembling and reranking. These ASR and MT techniques, combined, improve the speech translation quality significantly.

1. Introduction
In this paper we describe the Alibaba speech translation system that was built as part of the Speech Translation Task in
IWSLT 2018. The task involved translating English audio toGerman text in which English audio are from lectures and
TED talks. Our system employs a pipeline approach that includes an automatic speech recognition system (ASR) and amachine translation (MT) system.
The paper is organized as follows: Section 2 presents the
ASR system used, along with a description of conventional,
end-to-end, and fusion systems. Section 3 focuses on the MT
system in which we describe preprocessing, data augmentation, noisy input translation, ensembling, and reranking components in detail. We present our concluding remarks in Section 4.

2. Automatic Speech Recognition
In a pipeline-based speech translation system, ASR is themost front-end module. In order to get reliable translation of quality, it is critical to obtain ASR transcription
as accurate as possible. To start, we build several conventional pipeline-based ASR systems using deep neuralnetwork/hidden Markov model (DNN/HMM) framework.
In the DNN-HMM framework, we employ several DNNs

with different structures including fully-connected DNN(FDNN), time-delay deep neural network (TDNN) [1, 2],
and latency-controlled bidirectional long short-term memory(BLSTM) [3].
Our final goal is to build end-to-end speech translation
system, that is, we need to simplify the model-building process of conventional pipeline-based ASR system by constructing complicated modules with a single DNN architecture or in a data-driven learning method. Thus we also employ an end-to-end ASR system which is based on a hybrid connectionist temporal classification (CTC) [4, 5] andattention based encoder-decoder [6] architecture. Finally, we
combine the output of different ASR systems to boost the
final ASR performance.
During the development of our system, all acoustic models are trained on TED dataset together with the training
datasets provided by the organizer. We noticed that the organizer’s segmentation of the talks/lectures is not quite accurate, e.g. some sentences are not properly split. Therefore, we employ our own model-based voice activity detection (VAD) module to split the talks/lectures into utterances
before ASR decoding. For the model-based VAD, the recurrent neural network (RNN) model is used to train and classify
each frame into non-speech or speech. The RNN based VAD
model was trained by using TED and other speech corpus,and by using Alibaba’s VAD segmentation, it can get better performance when comparing with Organizer’s segmentation. Table 2 listed the comparsion.
Table 1: Configuration of DNNs in DNN-HMM acoustic
models.
SystemInput feature#Dim

Network context

FDNN
TDNN
BLSTM

FBank+Pitch
MFCC+ivector
FBank+Pitch

80
40+10080{-5, 5}{-13, 9}{-8, 8}#Dim: number of feature dimension{-L, R}: L frames in the left context and R frames in theright context* Equal contribution

136
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 2: WER (%) of different ASR systems on IWSLT 2013 and 2015 dataset using organizer’s and Alibaba’s segmentation.
SystemOrganizer’s segmentation
tst2013 tst2015 Avg.

Alibaba’s segmentation
tst2013 tst2015 Avg.

CTC (baseline)

26.1

37.6

31.9

-

-

-

1. FDNN/HMM
2. TDNN/HMM
3. BLSTM/HMM

22.9
13.6
13.6

31.6
25.5
26.3

27.3
19.6
20.0

19.4
11
10.6

28
23.1
22.3

23.7
17.1
16.5

4. CTC/Attention

26

33.1

29.6

23

29.8

26.4

1+2+3+4
2+3+4

-

-

-

10.3
8.6

22.3
21.7

16.3
15.2

2.1. Conventional ASR system

2.3. Fusion of ASR output

In the conventional DNN-HMM framework, DNNs are designated to predict the alignments derived from a GMMHMM based acoustic model, given different input features. As shown in Table 1, FDNN and BLTSM take 80dimensional filter bank feature vectors (FBank) and pitch
feature as input feature while TDNN take 40-dimensional
Mel-frequency cepstral coefficients (MFCCs) appended with
100-dimensional iVector as input feature. These DNNs takedifferent lengths of context. FDNN takes one frame together
with 5 frames from its left and right context as the input window, i.e. context span is {-5, 5}. TDNN follows the settingin [2] which takes {-13, 9} as the context span. BLSTM
takes context span {-8, 8}.

With all the aforementioned ASR systems, we have a set of
ASR output of each test set. The WER of the ASR systems
are summarized in Table. 2. We utilize ROVER [7] to fuse
the output from different ASR systems. By enumerating all
combinations of each ASR system, we select the output fromthe combination of the TDNN/HMM, BLSTM/HMM andhybrid CTC/attention based end-to-end systems, since thiscombined output gives the lowest WER. There are so big
differences between them in model structure and used features for TDNN, BLSTM and CTC/attention systems, andtherefore, after fusing between them, it can get better performance.

We train a 4-gram LM using all the allowed text. The 4gram LM obtained by linearly interpolating a number of LMs
that are trained using all the TED transcripts provided by theorganizer, all the text in the WMT18 CommonCrawl corpus,
some sentences selected from the WMT18 news, WMT18
news discussion and OpenSubtitles2018 corpora. We use
cross-entropy based data selection to select sentences fromthe corpora that are close to the TED transcripts. The LM
interpolation weights are optimized on all development data.
2.2. End-to-end ASR system
We employ a hybrid CTC/Attention architecture [6] in our
end-to-end ASR system. This system consists of a BLSTMbased encoder, a CTC output layer and an attention decoder.
The CTC output layer and the attention decoder takes the output of encoder and predicts the corresponding letters. Thatis, the end-to-end system is charactor level system. Pleaserefer to [6] for more system details. And in Table 2, there arelarge differences between CTC/attention based ASR system
and TDNN or BLSTM based ASR system. The reasons lie inthat (1) the scale of speech training data. The CTC/attention
based ASR need more large scale of traning data to get better
performance comparing with TDNN or BLSTM based approach; and (2) the weak language model for CTC/attention
based ASR system.

3. Machine Translation
3.1. MT baseline
In this section, we describe how our MT system has beendeveloped. All our models are based on the transformer architecture in [8]. We start with the TED corpus, speechtranslation TED corpus, and WMT18 data that are relevantto the speech translation domain. The total size of the bilingual corpus is 6.3 million sentence pairs. We use Marian
toolkit for all experiments [9] and our development set includes dev2010 and tst2010-2015. The baseline architectureof Marian mainly follows the default setting for transformer
NMT except for a 6-layer transformer encode-decoder, a 0.1
label smoothing, and 0.1 dropout between transformer layer.
For parameter optimization, we use synchronized ADAM
[10] with learning rate 0.0003, and set up the number ofnoam warm-up steps as 16,000.
3.2. Preprocessing
The training data is preprocessed following standard procedure. We first use the scripts in moses toolkit1 for punctuation normalization, tokenization and lowercasing. Afterwards, we jointly learn and apply the byte pair encoding2
for English and German together. Figure 1 shows a detailed
1 https://github.com/moses-smt/mosesdecoder
2 https://github.com/rsennrich/subword-nmt

137
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

pus, respectively. Then, another set of two 3-gram language
models are trained, whose training data is randomly selectedfrom the out-domain corpus, with size being similar. Thus,
each sentence pair from out-domain corpus is scored by thebilingual cross-entropy difference model. Finally, we sort all
sentence pairs and select top ranked sentences pairs. Our experiments show that with different amount of additional data,
we obtain BLEU gain on the development set between 0.4 to
1.8.
3.4. Noisy input translation

Figure 1: Performance of BPE operations including joint-bpe
16k, 32k, 48k, 64k, 80k, 96k, and shared-bpe 24K and 32K
on our development set.
comparison of different BPE operations on our developmentset. We observe that joint bpe with 32K vocabulary performs
best in this case and it is our final BPE code size. In the end,
sentences longer than 100 tokens are removed.
Generally, the ASR raw output is a long stream of words
with no punctuation, capitalization or segmentation markers.
[11] shown that using various types of text segmenters between ASR and MT modules can improve speech translation
quality. Our ASR system performed audio segmentation,
punctuation and capitalization prediction. We use spaCy3
to segment processed ASR transcription into shorter chunks.
By using text segmentation we observed BLEU score gains
between 0.2 and 1.0, depending on different ASR and MT
systems.
For sentence boundary detection and punctuation insertion model, we experiment with a 3 layers LSTM for sequence tagging, and 3-gram KenLM[12] for additional scoring. In prediction phrase, we store some token in buffer as
the foregoing context which is useful in real time prediction.
Silence time in ASR is also used for sentence boundary detection. We also trained single layer self attention sequencetagging model and a bi-directional self attention LM. And we
found it got much higher comma F1 score, but a litter lower
period F1 score. This approach was not used in the final result and we will do more experiments in our future work.
3.3. Data Augmentation
In order to obtain a high quality domain related training corpus, we exploit the algorithm described in [13, 14], aiming at selecting sentence pairs from large out-domain corpusthat are similar to the target domain. In our experiments,
the 200K TED talks data is considered as in-domain corpus,
and all the other parallel corpora provided are combined as alarge out-domain corpus. Two 3-gram language models aretrained over the source and target side of the in-domain cor3 https://spacy.io/

Though our transformer translation system is applied tothe post-processed speech recognition outputs, the insertion,
deletion and substitution errors still cannot be removed. Following the idea in [15, 16], we use the corrupted inputs totrain a robust neural machine translation model. Since weobserve that the insertion error is rare in our ASR system,
only the deletion and substitution noises are considered forthe source sentences of the regular parallel training data. In
this way, the gap between training data and testing ASR output will become potentially smaller.
Basically, we first randomly delete the token of the source
sentence with a small probability (0.01 and 0.02 are selectedin our experiments by cross-validation). Specifically, we also
pre-define a functional-words list including 120 tokens withnumber of letters less than 5, and assign a doubled deletion
rate for them. Notice that we train several deletion-noisy
models alone without any other corrupted strategy for further
ensemble. In our experiments, the single model trained withdeletion noise can increase the case insensitive (CI) BLEU
for at least 0.5 BLEU point over the baseline on the ASR
output.
Additionally, we attempt to introduce the substitution noise by randomly replacing the token with its
pronunciation-like candidates with a small probability. By
trying different substitution rate, we empirically found thatthe substitution noise model achieved no significant improvement over deletion noise model. One possible solution is to
use the adaptive substitution rate of each token, estimatedwith the maximum likelihood in the ASR model. We will
leave this as the future work.
The third strategy for noisy training is punctuation simulation. We randomly spare 30% of the regular parallel
corpus and remove all punctuations from the source side,then annotate/re-generate the same data with the tool usedin the ASR post-processing. We add the punctuation noisy
corpus back and train our model, and empirically observe
an improvement of at least 0.5 BLEU point gain comparedwith baseline as well. The punctuation simulation and deletion/substitution noise are typically not combined, since theunderlying true noise comes from the ASR system and our
two manually designed noising systems may have a largebias. Therefore, we decide to apply them separately to increase the diversity of our models in ensemble.

138
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 3: Improvement with ensemble

3.5. Refinements
3.5.1. Ensemble decoding

tst2013 + tst2015

Model ensemble is a widely used technique to boost the performance of a MT system, which is to combine the predictionof multiple models at each decode step. We adopt the ensemble method GMSE (Greedy Model Selection based Ensemble) detailed in [17].
The candidate single models are first sorted as a list according to their performance on the development dataset.
Another two model lists are maintained during the ensemble, named “keep”, “redemption”. For each iteration, a candidate model could be either drawn from the beginning or
the end of the candidates list with probability p or preserve ,or the “redemption” list with probability predempt , wherep + preserve + predempt = 1. Then, the selected model is temporarily concatenated to the “keep” list. If the evaluation of
the current model ensemble achieves a better BLEU score,
the model is permanently added to the “keep” list. Otherwise, it will be put into the “redemption” list. Notice that
one model from the “redemption” list can only be redeemed
once, after which it is withdrawn permanently from the candidates.
In order to achieve better ensemble performance, we increase the diversity of our candidate models by introducing
another 8 training schedules and further obtain about 200 single checkpoints. The greedy nature of the GMSE algorithm
makes the search feasible in an acceptable time frame. Insummary, we list the different training schedules as follows.
1. 10 million training corpus is selected by BPE level language models.
2. 8 million training corpus is selected by word level language models.
3. Adding another 3 million back-translation data to theoriginal parallel corpus.

best single model
best ensemble

22.96
23.84

3.5.2. Reranking
Besides building multiple ensemble systems with different
random initialization and configurations, we also build andoptimize the n-best list reranker. We follow the approach in[17] in which several neural machine translation models andlanguage models have been experimented with. The n-best
list reranker involves the following steps• Build and optimize neural MT models including single
models, ensemble models, and models with different
configurations such as beam size. To improve diversity, we also use the right-to-left and target-to-source
models.
• Build ngram language models from in-domain andout-of-domain data using the data selection method
similar to [17, 18].
• Apply the the greedy feature selection based reranking
method in [17] to train the reranker. To deal with overfiting, we use the tuning set that contains both manualtranscription and our ASR transcription.
Our experiments show that depending on different settings the reranker typically obtains improvements between
0.1 to 0.4 BLEU point over the best system.
Additionally, we experiment with the multi ASR inputs
for reranking. The main motivation is to directly exploit
strengths of different ASR systems into the reranking system. Out initial results show that the multi ASR inputs does
not outperform the input for the ASR fusion output.
3.5.3. Recaser

4. Training with deletion/substitution noise.
5. Training with punctuation simulation noise.
6. Fine-tuning with the 200K TED talks in-domain data.
7. Fine-tuning with the punctuation simulation data.
8. 7-layer transformer NMT model.
Due to the time limitation, we cannot do all the ablative
experiments on listed strategies. However, we can still report
the best single model and best ensemble result on our ASR
output in Table 3. Note that our development set is the combination of tst2013 and tst2015, we did not test our model on
these two dataset separately.

Since the lowercased corpus are applied to train our MT system, an additional post-processing recaser model is necessary to obtain the truecased (or capitalized) German translation output. In principle, we exploit the combination of themoses4 SMT recaser model and the Char-RNN based neural recaser model [19]. The SMT recaser model essentiallytrains a word-to-word translation model and a cased language
model without reordering. Unlike the word-level approach,the neural recaser model restores the case information at thecharacter-level, reducing the recasing problem to a sequential
binary classification task. No special treatment is required
for mixed cased words.
The final combination rule is that for every single word,
if the SMT recaser and the neural recaser reach a consensus
4 http://www.statmt.org/moses/?n=Moses.SupportTools

139
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

on the final recasing output, we will accept it; if they have adisagreement, we will always capitalize that word. This strategy will result in a combined recaser model which is slightly
better than any other one.

5. Acknowledgments

4. Conclusions
The IWSLT 2018 Speech Translation task provided the opportunity to compare different speech translation approaches
using shared datasets and standardized evaluation metrics.
Table 4 shows our submission results on the IWSLT 2018
official test set.
Table 4: IWSLT 2018 final evaluation results on contrastiveand primary submissions

BLEU
TER
BEER
CharTERBLEU(ci)TER(ci)strategy has its own limitations, and, in conjunction with thelimited availability of the labeled data, can easily produce
models that are under-performing on blind test sets.

con.1

con.2

primary

con.3

22
63.44
52.47
59.56
23.97
60.44

22.16
63.52
52.69
57.54
24.14
60.41

22.36
63.03
51.77
69.26
24.23
60.22

22.5
63.03
52.64
57.76
24.3
60.15

We thank the support from Shanbo Cheng, Jun Lu for their
help in reranking and recasing.

6. References[1] A. H. Waibel, T. Hanazawa, G. E. Hinton, K. Shikano,
and K. J. Lang, “Phoneme recognition using time-delay
neural networks,” IEEE Trans. Acoustics, Speech, andSignal Processing, vol. 37, no. 3, pp. 328–339, 1989.
[2] V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modelingof long temporal contexts,” in Proc. Interspeech, 2015.

Our participation in this task revealed three important aspects of speech translation that we regard as important for
the future.
First, our experiments indicated that the speech segmentation and transcription post processing, by themselves, canmake a big differences on transcription quality as well astranslation quality. Furthermore, these components not only
improve WER and BLEU scores, in live speech translation
scenario they also greatly improve user experience.
The second aspect relates to the importance of noisy inputs. There are many types of noises in speech translation
scenario. For example noise come from speech audio, transcription errors, and the nature of spoken language. Our experiments show that by modeling transcription errors directlyin the neural MT (NMT) training, we obtained consistent improvement. It indicates that the NMT model becomes morerobust against errors of our ASR system. Also, if we compare statistical machine translation (SMT) technique, we find
SMT is generally more robust to noises than NMT. It is probably because SMT models are built on probability distributions estimated from many occurrences of words and phrases,
therefore any unsystematic noise in the training only affects
the tail end of the distribution.
The third aspect is the importance of model engineering.
In the statistical machine learning, the best model is typically from through several rounds of feature engineering. In
the NMT context, we see that our best model is also frommany steps of model engineering and refinements. Given theavailability and good scalability of ASR and MT toolkits today, it is tempting to throw as much model configurations
as possible and let the built-in mechanisms of these learningalgorithms figure out which one is the best. However, the[3] S. Xue and Z. Yan, “Improving latency-controlled
BLSTM acoustic models for online speech recognition,” in Proc. ICASSP, 2017, pp. 5340–5344.
[Online]. Available: https://doi.org/10.1109/ICASSP.
2017.7953176[4] J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio, “End-to-end continuous speech recognition using
attention-based recurrent NN: first results,” in Proc.
NIPS, vol. abs/1412.1602, 2014.
[5] A. Graves and N. Jaitly, “Towards end-to-end speech
recognition with recurrent neural networks,” in Proc.
ICML, 2014, pp. 1764–1772.
[6] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, andT. Hayashi, “Hybrid ctc/attention architecture for endto-end speech recognition,” IEEE J. Sel. Topics Signal
Processing, vol. 11, no. 8, pp. 1240–1253, 2017.
[7] J. G. Fiscus, “A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover),” in Proc. of the IEEE Workshop
on Automatic Speech Recognition and Understanding,1997, pp. 347–352.
[8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin,“Attention is all you need,” in Proc. NIPS, 2017, pp.
5998–6008.
[9] M. Junczys-Dowmunt, R. Grundkiewicz, T. Dwojak,
H. Hoang, K. Heafield, T. Neckermann, F. Seide,
U. Germann, A. Fikri Aji, N. Bogoychev, A. F. T. Martins, and A. Birch, “Marian: Fast neural machine translation in C++,” in Proceedings of ACL 2018, System
Demonstrations. Melbourne, Australia: Association
for Computational Linguistics, July 2018, pp. 116–121.

140
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[10] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980,
2014.

Empirical Methods in Natural Language Processing).
Austin, Texas: Association for Computational Linguistics, November 2016, pp. 2090–2095.

[11] V. K. R. Sridhar, J. Chen, S. Bangalore, A. Ljolje,
and R. Chengalvarayan, “Segmentation strategies for
streaming speech translation,” in Proc. of the Human Language Technologies: Conference of the North
American Chapter of the Association of Computational
Linguistics, 2013, pp. 230–238.
[12] K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn,“Scalable modified Kneser-Ney language model estimation,” in Proceedings of the 51st Annual Meeting of
the Association for Computational Linguistics, Sofia,
Bulgaria, August 2013, pp. 690–696.
[13] A. Axelrod, X. He, and J. Gao, “Domain adaptation via
pseudo in-domain data selection,” in Proceedings of theconference on empirical methods in natural language
processing. Association for Computational Linguistics, 2011, pp. 355–362.
[14] K. Duh, G. Neubig, K. Sudoh, and H. Tsukada, “Adaptation data selection using neural language models: Experiments in machine translation,” in Proceedings of
the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), vol. 2,
2013, pp. 678–683.
[15] M. Sperber, J. Niehues, and A. Waibel, “Toward robustneural machine translation for noisy input sequences,”in Proc. of the International Workshop on Spoken Language Translation (IWSLT), Tokyo, Japan, December,
14-15 2017.
[16] N.-Q. Pham, M. Sperber, E. Salesky, T.-L. Ha,
J. Niehues, and A. Waibel, “Kit’s multilingual neuralmachine translation systems for iwslt 2017,” in Proc.
of the International Workshop on Spoken Language
Translation (IWSLT), Tokyo, Japan, December, 14-15
2017.
[17] Y. Deng, S. Cheng, J. Lu, K. Song, J. Wang, S. Wu,
L. Yao, G. Zhang, H. Zhang, P. Zhang, C. Zhu, and
B. Chen, “Alibaba’s neural machine translation systems
for wmt18,” in Proc. of the Conference on MachineTranslation. Brussels, Belgium: Association for Computational Linguistics, November 2018.
[18] B. Chen and F. Huang, “Semi-supervised convolutional
networks for translation adaptation with tiny amount of
in-domain data,” in Proc. of the Conference on Computational Natural Language Learning. Association for
Computational Linguistics, 2016, pp. 314–323.
[19] R. H. Susanto, H. L. Chieu, and W. Lu, “Learning to capitalize with character-level recurrent neural networks: An empirical study,” in Proc. of the141
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

CUNI Basque-to-English Submission in IWSLT18
Tom Kocmi

Dušan Variš

Ondřej Bojar

Charles University, Faculty of Mathematics and Physics
Institute of Formal and Applied Linguistics
Malostranské náměstı́ 25, 118 00 Prague, Czech Republic<surname>@ufal.mff.cuni.cz

Abstract
We present our submission to the IWSLT18 Low Resource task focused on the translation from Basque-toEnglish. Our submission is based on the current stateof-the-art self-attentive neural network architecture, Transformer. We further improve this strong baseline by exploiting
available monolingual data using the back-translation technique. We also present further improvements gained by atransfer learning, a technique that trains a model using ahigh-resource language pair (Czech-English) and then finetunes the model using the target low-resource language pair
(Basque-English).

1. Introduction
Despite becoming the current dominant approach in thefield of machine translation (MT), neural machine translation(NMT) [1] systems still perform poorly in certain scenarios. One of them is learning to translate between language
pairs where only a small amount of parallel data is available. Under these circumstances the NMT model quickly
overfits and its performance plummets when translating sentences not seen during training. As observed in [2], withsmall parallel data, NMT performs much worse than the previous approach of phrase-based MT.
There are situations where the ability to learn an MTmodel of a reasonable quality given only a small amount oftraining data can be crucial. For example, when a crisis occurs in a region where an under-resourced language is spoken, a quick deployment of an MT system translating fromor to that language can make a huge difference in the impactof the provided support [3].
In this paper, we describe the CUNI submission to the
IWSLT Low Resource task for translating from Basque-toEnglish in the domain of TED talks. Our submission is basedon the recently introduced self-attentive network architecture
called Transformer [4]. We improve the performance of thismodel by exploiting the English in-domain monolingual datausing the back-translation technique [5]. We achieve further
improvements via transfer learning. Transfer learning [6, 7]
consists of training a “parent” (high-resource) model first andthen continuing the training on the “child”, low-resource,
parallel data as a means of model adaptation. Furthermore,

we combine several models saved at training checkpoints bysimply averaging the weights (“model averaging”) as a substitute of model ensembling.
The structure of the paper is the following. In Section 2,
we describe the method of transfer learning followed by thedescription of back-translation in Section 3. The model description is presented in Section 4 and the dataset overview
in Section 5. Section 6 details the results achieved by oursystems. Section 7 discusses other works in the area of
low-resource translation systems. And finally Section 8 concludes the paper.

2. Transfer learning
Transfer learning is based on the observation that neural machine translation model that is first trained on the parallel
data of a high-resource language pair can be adapted to alower-resource language pair. The two languages can have alinguistic relation, however, transfer learning works even for
unrelated languages [7].
The method starts by first training the parent model until it reaches the best possible performance or until a fixed
number of gradient updates is performed. This model is thenadapted by switching the training dataset from the parent pairto the low-resource child pair. During this transition, we do
not change any hyperparameters nor the learning rate.
The transfer learning method does not need any modification of the existing NMT pipeline. The method only relies
on a single condition: the vocabulary has to be shared acrossall the languages in the parent as well as child language pairs.
We construct the shared vocabulary using subword tokens, namely wordpieces [8], instead of words. This way, we
are able to handle words not seen during training by splitting
them into subwords, which are present in the vocabulary. We
learn the subword segmentation using concatenated source
and target sides of both the parent and child language pairs.
To avoid bias in the vocabulary towards the high-resource
language pair, [7] suggest to sample a subset of the sentencesfrom the high-resource language pair that has a size similar tothe low-resource language pair dataset, calling this approach“balanced vocabulary”. They also showed, that a significant
portion of this balanced vocabulary is relevant only for thechild model, as it never appears in the parent training data.

142
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Unlike [7], we also experiment with additional vocabulary setups, using either only the parent (or only the child)training data to generate the vocabulary. We call these restricted setups “parent vocabulary” and “child vocabulary”,
respectively. The idea behind the use of “child vocabulary”
is that there will be more child-specific wordpieces which
can lead to a better performance of the child model. On theother hand, the reasoning behind the “parent vocabulary” isthat we can use only a single parent for the training of several
different child models and therefore save the time of training
parent models for each child separately.

3. Back-translation
The organizers of IWSLT 2018 provided participants witha vast amount of English monolingual data to use in theirsystem submissions, both in-domain and out-of-domain. We
exploit the English in-domain TED talks monolingual datafor creation of the synthetic data as described by [5].
The key idea is to use an MT system trained to translate in the opposite direction (English-to-Basque) and use it
to translate the monolingual data. These synthetic outputs,
when paired with the input monolingual data, can be thenused as additional parallel data for the original (Basque-toEnglish) direction. Even though the source side is noisy, theadditional training examples help the decoder to learn a more
fluent target side language model.
We use this method to back-translate only the in-domain
TED talks data because it is the target domain of the Low
Resource task.
To create the synthetic parallel data from the English
monolingual corpus, we used a Transformer model and transfer learning. We first trained on the English-to-Czech corpus
and then adapted the model using English-to-Basque corpus.
This was based on our previous experiments where transfer
learning resulted in a model with a better translation performance and therefore a better quality of synthetic data.

4. Model description
We use the self-attentive neural network architecture called
Transformer [4]. We chose this network architecture due to
its reported state-of-the-art results [9, 10],1 making it a strong
baseline for our experiments.
The architecture follows the encoder-decoder paradigm
where the encoder creates hidden representations of thesource language tokens and the decoder outputs the target sequence conditioned on that source language representations
and the representations of the already decoded tokens.
The self-attentive encoder contains several layers eachconsisting of two sublayers: the first one applies a selfattention and the second one a feed-forward network. The
decoder is similar, including an additional attention-overencoder layer between its own self-attention and the feedfor1 http://www.statmt.org/wmt18/translation-task.

html

DatasetGenuine EN-EU
Genuine EN-CS
Synthetic EN-EU

Sentences
0.9 M
40.1 M
0.3 M

Tokens EN
7.0 M
563.4 M
5.3 M

Tokens CS/EU
5.1 M
490.5 M
3.6 M

Table 1: Sizes of the parallel corpora. The “synthetic” have
Basque side back-translated from English.
ward layers. The self-attention layer is the key component of
the Transformer architecture, effectively modeling the context of each token and thus substituting other methods such as
the recurrent hidden units [1, 11] or convolutional networks[12]. The absence of recurrent units makes the training much
faster due to a possible parallelism while requiring a lower
number of layers when compared to the convolutional network.
We use the Transformer implemented in Tensor2Tensor[13],2 version 1.4.2. Our models are based on the “big single
GPU” configuration as defined in the paper. We use the default setup, only changing the batch size to 2300 and a maximum sentence length to 100 wordpieces in order to fit themodel to our GPUs (NVIDIA GeForce GTX 1080 Ti with
11 GB RAM).
We use Noam learning rate decay scheme with the starting learning rate of 0.2 and 32000 warm-up steps. The decoding uses the beam size of 8 and length normalization
penalty is set to 0.8.

5. Dataset
For Basque-English, we used all the available data that wereallowed by the organizers of IWSLT 2018. The parallel corpora consist of only around 5,600 in-domain (TED) sentencepairs and around 940,000 out-of-domain sentence pairs.
In addition to the resources suggested by the organizers,
we also used data from OPUS and WMT, which were also
allowed. Specifically, corpora PaCo2 English-Basque and
QTLeap Batches 1-3 from WMT.3
For English-Czech, we use all parallel data available for
WMT 2018 except of the Paracrawl. The majority of the datais part of the CzEng 1.7 corpus (the filtered version, [14]).4
We also created synthetic Basque-English data using
back-translation. We generate them by translating all English sentences from the TED talks data gathered across all
language pairs provided for IWSLT 2018. The data do notcontain sentences from talks in test set.
From all training sentences, we dropped sentence pairs
shorter than 4 words or longer than 75 words on either sourceor target side. This results in a speedup of the training by
allowing a larger batch size. A similar setup was used in[15] where the authors argue that in their experiments, the2 https://github.com/tensorflow/tensor2tensor

3 http://www.statmt.org/wmt16/
it-translation-task.html
4 https://ufal.mff.cuni.cz/czeng/czeng17

143
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

VocabularyChild only
Parent only
Balanced
Baseline

CS to EN (BLEU)
24.93
27.81
27.93
–

Run
Primary
Contrastive 1
Contrastive 2Contrastive 3

EU to EN (BLEU)
22.92
23.29
23.63
19.09

Table 2: The results of transfer learning. The first column
shows the performance of the parent model, the second column is the child model based on the corresponding parent.
The baseline does not use transfer learning. The results arereported on the development set. Scores are comparable only
within columns.

performance is not negatively influenced by the reduction oftraining data.
To evaluate the models during training we used the development data provided by IWSLT 2018 (Basque-English)
and development data available for WMT (Czech-English),namely WMT 2011 Newstest.

6. Results
In this section we first compare results obtained when using
the three types of vocabulary and then describe our systems
submitted to the IWSLT evaluation.
6.1. Effect of vocabulary
We experiment with three types of shared vocabulary as described in Section 2. All setups use the exact same data (anda same layout of the transfer learning); they differ only in the
vocabulary. First, we trained three models from Czech-toEnglish with different vocabularies for 1M steps and then we
continued with the transfer learning of the child Basque-toEnglish models until their performance on the validation set
stopped improving.
As seen in Table 6.1, the transfer learning for Basque-toEnglish improves the model performance significantly overthe baseline, gaining over 4 BLEU points (19.09 vs. 23.63).
When we look at the parent-only or child-only vocabulary setups, both performed worse than the balanced vocabulary. With the balanced vocabulary, we obtain the best resulton the Basque-to-English translation. We suppose that thesame holds for other language pairs too, since there is no
language specific restriction.
Still, it would be interesting to know whether the data ratio 50:50 is the best possible setup or whether other ratios
could improve the results. We plan to investigate this in our
future work. We assume that the exact ratio might be language specific, however, in general, using the balanced approach with an equal representation of the languages might
still be an effective option.

on
lati
s
n
er tra ne
nsf ck- nui
Tra Ba Ge BLEU
X X X
22.86
–
– X
16.13
X X
–
22.26
X
– X
21.11

NIST
6.01
4.98
6.00
5.84

TER
60.31
66.55
63.89
62.34

Table 3: Results of our submissions. Official evaluation on
the test set.
6.2. Final results
We submitted several contrastive models for the final IWSLT
evaluation. All our systems use the same balanced vocabulary. The synthetic data were generated by the English-toBasque system. All final models are averaged over the last 5
checkpoints.
The primary system uses the transfer learning: the parent
model is trained for 1M steps on Czech-to-English, followed
by transfer learning using only the synthetic data for 405k
steps, and completed by 60k steps on the genuine, original
parallel data.
The run labelled “Contrastive 1” in Table 3 is the baseline
trained only on the official parallel Basque-to-English data.
“Contrastive 2” uses transfer learning on the parent
model Czech-to-English trained for 1M steps, followed bytraining on only the synthetic English-to-Basque data, without the use of genuine parallel data.
Finally, “Contrastive 3” also uses transfer from Czech-toEnglish as the primary, followed by genuine parallel Englishto-Basque data, without the use of any synthetic data.
As clearly confirmed by three automatic metrics, thecombination of the back-translation and transfer learning
leads to the best performance.

7. Related work
In [16], Firat et al. propose zero-resource multi-way multilingual systems, with the main goal of reducing the total
number of parameters needed to train multiple source andtarget languages. To prevent the network from forgetting thepreviously learned language pairs, they implement a special
training schedule.
Another multilingual approach is proposed by [8] where
Johnson et al. simply use all translation pairs at the same time
and the choice of the target language happens at runtime byspecial token at the end of the input sentence. This forces themodel to learn to translate between many languages, including language pairs without available ‘direct’ parallel data.
The lack of sufficient amounts of parallel data can be also
tackled by unsupervised translation [17, 18]. The generalidea is to train monolingual embeddings using large amountsof monolingual data and finding a projection from the source
to target words that preserves the structure of embedding
vector spaces [19]. Using these shared fixed bilingual embeddings an architecture with a shared encoder [18] or both

144
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

shared encoder and decoder [17] is then trained using multiple training objectives.
Aside from the common back-translation [5], simple
copying of the target monolingual data back to the sourceside has also been shown to improve translation quality inthe low-resource setting [20].
The transfer learning we used could be also seen as a variant of the so-called “curriculum learning” [21, 22], where thetraining data are ordered from foreign out-of-domain to thein-domain training examples to speed up the training convergence.

8. Conclusion
In this paper, we presented our systems for IWSLT 2018 lowresource Basque-to-English translation task. We reached asignificant improvement using transfer learning and backtranslation. We compared three types of vocabularies used
for the transfer learning and concluded, that the balanced vocabulary is the best option.

9. Acknowledgements
This study was supported in parts by the grants SVV 260 453,
GAUK 8502/2016, GAUK 1140218, and 18-24210S of the
Czech Science Foundation. This work has been using language resources and tools stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youthand Sports of the Czech Republic (projects LM2015071 andOP VVV VI CZ.02.1.01/0.0/0.0/16 013/0001781).

10. References[1] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence
to sequence learning with neural networks,” inProceedings of the 27th International Conference
on Neural Information Processing Systems - Volume
2, ser. NIPS’14. Cambridge, MA, USA: MIT
Press, 2014, pp. 3104–3112. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2969033.2969173[2] P. Koehn and R. Knowles, “Six Challenges for
Neural Machine Translation,” in Proceedings ofthe First Workshop on Neural Machine Translation.
Vancouver: Association for Computational Linguistics,
August 2017, pp. 28–39. [Online]. Available: http://www.aclweb.org/anthology/W17-3204

I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems
30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett,
Eds. Curran Associates, Inc., 2017, pp. 6000–
6010. [Online]. Available: http://papers.nips.cc/paper/
7181-attention-is-all-you-need.pdf[5] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” in Proceedings of the 54th Annual Meeting of
the Association for Computational Linguistics (Volume
1: Long Papers). Berlin, Germany: Association for
Computational Linguistics, August 2016, pp. 86–96.
[Online]. Available: http://www.aclweb.org/anthology/
P16-1009[6] B. Zoph, D. Yuret, J. May, and K. Knight, “Transfer
learning for low-resource neural machine translation,”in Proceedings of the 2016 Conference on Empirical
Methods in Natural Language Processing. Austin,
Texas: Association for Computational Linguistics,
November 2016, pp. 1568–1575. [Online]. Available:
https://aclweb.org/anthology/D16-1163[7] T. Kocmi and O. Bojar, “Trivial Transfer Learning for
Low-Resource Neural Machine Translation,” in Proceedings of the 3rd Conference on Machine Translation(WMT), Brussels, Belgium, Nov. 2018.
[8] M. Johnson, M. Schuster, Q. Le, M. Krikun, Y. Wu,
Z. Chen, N. Thorat, F. a. Vigas, M. Wattenberg,
G. Corrado, M. Hughes, and J. Dean, “Google’s
multilingual neural machine translation system: Enabling zero-shot translation,” Transactions of the
Association for Computational Linguistics, vol. 5,
pp. 339–351, 2017. [Online]. Available: https://transacl.org/ojs/index.php/tacl/article/view/1081[9] M. Popel and O. Bojar, “Training Tips for the Transformer Model,” The Prague Bulletin of Mathematical
Linguistics, vol. 110, no. 1, pp. 43–70, 2018. [Online].
Available: https://content.sciendo.com/view/journals/
pralin/110/1/article-p43.xml[10] O. Bojar, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, P. Koehn, and C. Monz, “Findings of the2018 conference on machine translation (WMT18),”
in Proceedings of the Third Conference on MachineTranslation, Volume 2: Shared Task Papers. Brussels,
Belgium: Association for Computational Linguistics,
October 2018.

[3] W. Lewis, R. Munro, and S. Vogel, “Crisis mt:
Developing a cookbook for mt in crisis situations,”in Proceedings of the Sixth Workshop on Statistical
Machine Translation. Association for Computational
Linguistics, July 2011. [Online]. Available: https://www.microsoft.com/en-us/research/publication/
crisis-mt-developing-a-cookbook-for-mt-in-crisis-situations/ [11] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,”[4] A. Vaswani, N. Shazeer, N. Parmar, J. UszkoCoRR, vol. abs/1409.0473, 2014. [Online]. Available:
reit, L. Jones, A. N. Gomez, Ł. Kaiser, andhttp://arxiv.org/abs/1409.0473

145
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[12] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N.
Dauphin, “Convolutional sequence to sequence learning,” in ICML, ser. Proceedings of Machine Learning
Research, vol. 70. PMLR, 2017, pp. 1243–1252.

[21] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proceedings of the 26th
annual international conference on machine learning.
ACM, 2009, pp. 41–48.

[13] A. Vaswani, S. Bengio, E. Brevdo, F. Chollet,
A. Gomez, S. Gouws, L. Jones, L. Kaiser, N. Kalchbrenner, N. Parmar, R. Sepassi, N. Shazeer, and
J. Uszkoreit, “Tensor2Tensor for Neural MachineTranslation,” in Proceedings of the 13th Conferenceof the Association for Machine Translation in theAmericas (Volume 1: Research Papers). Boston, MA:
Association for Machine Translation in the Americas, March 2018, pp. 193–199. [Online]. Available:
http://www.aclweb.org/anthology/W18-1819[22] T. Kocmi and O. Bojar, “Curriculum Learning and
Minibatch Bucketing in Neural Machine Translation,”in Recent Advances in Natural Language Processing
2017, Sept. 2017.

[14] O. Bojar, O. Dušek, T. Kocmi, J. Libovický, M. Novák,
M. Popel, R. Sudarikov, and D. Variš, “Czeng 1.6: enlarged czech-english parallel corpus with processing
tools dockered,” in International Conference on Text,
Speech, and Dialogue. Springer, 2016, pp. 231–238.
[15] T. Kocmi, oman Sudarikov, and O. Bojar, “CUNI Submissions in WMT18,” in Proceedings of the 3rd Conference on Machine Translation (WMT), Brussels, Belgium, Nov. 2018.
[16] O. Firat, K. Cho, and Y. Bengio, “Multi-Way,
Multilingual Neural Machine Translation with a
Shared Attention Mechanism,” in Proceedings of the2016 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human
Language Technologies. San Diego, California:
Association for Computational Linguistics, June 2016,
pp. 866–875. [Online]. Available: http://www.aclweb.
org/anthology/N16-1101[17] M. Artetxe, G. Labaka, E. Agirre, and K. Cho, “Unsupervised neural machine translation,” in Proceedings of
the Sixth International Conference on Learning Representations, April 2018.
[18] A. Conneau, G. Lample, M. Ranzato, L. Denoyer,
and H. Jégou, “Word translation without parallel data,”
CoRR, vol. abs/1710.04087, 2017.
[19] M. Artetxe, G. Labaka, and E. Agirre, “Learningbilingual word embeddings with (almost) no bilingual
data,” in Proceedings of the 55th Annual Meeting of
the Association for Computational Linguistics (Volume
1: Long Papers). Vancouver, Canada: Association
for Computational Linguistics, July 2017, pp. 451–
462. [Online]. Available: http://aclweb.org/anthology/
P17-1042[20] A. Currey, A. V. M. Barone, and K. Heafield, “Copiedmonolingual data improves low-resource neural machine translation,” in Proceedings of the Second Conference on Machine Translation, 2017, pp. 148–156.

146
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Fine-tuning on Clean Data for End-to-End Speech Translation:
FBK @ IWSLT 2018
Mattia Antonino Di Gangi1,2 , Roberto Dessı̀ 1 *, Roldano Cattoni2 ,Matteo Negri2 , Marco Turchi2
University of Trento, Italy
Fondazione Bruno Kessler, Italy
1

2

1
2

roberto.dessi@unitn.it{digangi,cattoni,negri,turchi}@fbk.eu

Abstract
This paper describes FBK’s submission to the end-to-end
English-German speech translation task at IWSLT 2018. Our
system relies on a state-of-the-art model based on LSTMs
and CNNs, where the CNNs are used to reduce the temporal dimension of the audio input, which is in generalmuch higher than machine translation input. Our model wastrained only on the audio-to-text parallel data released forthe task, and fine-tuned on cleaned subsets of the original
training corpus. The addition of weight normalization andlabel smoothing improved the baseline system by 1.0 BLEU
point on our validation set. The final submission also featured checkpoint averaging within a training run and ensemble decoding of models trained during multiple runs. On test
data, our best single model obtained a BLEU score of 9.7,
while the ensemble obtained a BLEU score of 10.24.

1. Introduction
End-to-end speech translation (that is, the direct translationof an audio signal without intermediate transcription steps)
has recently gained increasing interest in the scientific community thanks to the recent advances of neural approaches in
the related ASR and MT fields [1, 2, 3, 4, 5]. Effective approaches to the task can become a useful solution to deal withlanguages that do not have a formal writing system [6], as it ispossible to create a collection of spoken utterances with theirrespective translations in a more common language. We canalso expect that, in the future, end-to-end speech translation
systems will overcome problems related to the cumulative
effect of speech recognition errors introduced in pipelined
architectures. FBK’s submission to the IWSLT 2018 Speech
Translation (ST) task relies on a single model that takes as
input features extracted from an English audio signal and returns as output a written translation in German. As the input is not in raw wave form, one might argue that the “endto-end” denomination does not fit in this formulation of thetask. Nevertheless, since feeding the network with the input* Work performed during an internship at FBKfeatures released by the task organizers was allowed, we adhere to the looser definition of “end-to-end” implicit in thisyear’s task formulation.1
Our system was trained using the state-of-the-art
sequence-to-sequence model based on LSTMs and CNNs
introduced in [2]. Considering the high number of experiments to run, and the high number of epochs needed totrain a speech translation model (up to 87 in the case of
our final submission), the model was implemented using
the fairseq2 [7] sequence-to-sequence learning toolkit fromFacebook AI Research. The tool, which is tailored to NMT,
was adapted to the ST task showing considerable reductions
in training time compared to the same models implemented
on other platforms (from hours to minutes in the processingof the same amount of training instances).
One of the main challenges we faced was how to maximize the usefulness of the available training data by weeding out noisy (and potentially harmful) instances. For thispurpose, we developed the two data cleaning procedures described in Section 2. The architectural choices and the main
implementation details of our system are described in Section 3. In Section 4, we report the results on our validation
set, which were obtained by using different data conditions
and hyper-parameters. Section 5 concludes the paper withfinal remarks.

2. Data Cleaning
Our submission was obtained by a model solely trained withthe data released for the speech translation task. Before
building the model, we devoted particular attention to thequality of the training material, aiming to reduce the possible impact that noise in the data can have on training timeand model convergence. Indeed, the initial training set of
171, 121 instances comprised elements featuring either a partial alignment between the audio signal and the correspond1 Our work has been pursued during a summer project with the goal ofgaining hands-on expertise in this new promising field with the simplification of a standardized data set.
2 http://github.com/facebookresearch/fairseq

147
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

ing transcription, or a skewed ratio between the number offeature frames and the characters in the transcription. To
identify and weed out such noisy and potentially harmful
training items, we applied two cleaning procedures. Both theprocedures take advantage of the available English transcriptions of the audio signals3 and were run in cascade, after theremoval of 1, 000 items to be used as our development set.
As discussed in Section 4.1, though smaller in size, the resulting subsets of the original training corpus yielded performance improvements on development data, especially whenused for fine-tuning a model trained on the original unfilteredcorpus.
2.1. Cleaning Based on Alignment
Starting from the initial training corpus of 170, 121 instances(called “Parallel” henceforth), the first cleaning step was
aimed to identify and remove the items featuring a pooralignment between the audio signal and the text. Assuming that the English and German texts are parallel, the potential noise introduced by such instances is represented bywrong transcriptions/translations (either totally inadequate orcontaining spurious words) of the original source signal. To
identify them, our approach was to align each audio signal
with the corresponding English transcription and then decide
what to retain based on the alignment quality (i.e. considering unaligned words as evidence of noise). We performed thealignment on a sentence-by-sentence basis using Gentle,4 aforced aligner based on Kaldi.5 After the alignment, we removed all the training instances in which at least one word in
the transcription was not aligned with the corresponding audio segment. This strict cleaning policy (due to time limitations, we did not experiment with less aggressive strategies)
resulted in the removal of 24, 240 instances, which reduced
the initial “Parallel” corpus to 145, 881 items. Henceforth,the corpus resulting from this first cleaning step will be referred to as “Clean 1”.
2.2. Cleaning Based on Frames/Characters Ratio
The second cleaning step was aimed to identify and remove
from “Clean 1” the training instances featuring a skewed ratio between the number of feature frames and the charactersin the transcription. In this case, the potential noise is due
to portions of the original speech that correspond to long silences, background noise (e.g. laughter and applause), orwords that are not present in the transcription/translation. To
identify such possible outliers, looking at the ratios reported
in Figure 1, we decided to cut the distribution so to retain
only the training instances belonging to ratio bins that contain at least 5, 000 items. The corresponding cutting valuesof 3.5 and 7.5 resulted in the removal of 29, 898 instances,
3 Note that data cleaning is the only phase in which we used the English
transcriptions. Being this step independent from the actual system training,
our approach is still fully end to end.
4 https://lowerquality.com/gentle/
5 http://kaldi-asr.org/index.html

Figure 1: Distribution of the training instances in terms of
the ratio between the number of feature frames and the characters in the transcription.
which further reduced our training corpus to 115, 983 items.
Henceforth, the corpus resulting from our second cleaning
step will be referred to as “Clean 2”.

3. Seq2seq Speech Translation model
We re-implemented the seq2seq ST model introduced in [2],which uses an encoder-decoder-attention architecture based
mainly on LSTMs [8]. The source-side input length is someorder of magnitudes higher than the decoder side, and thus
some reduction in the temporal dimension was performed using 2-D CNNs with stride (2, 2). The decoder is inspired by
the early deep-transition decoder used in Nematus [9], whichstacks two LSTM units in a way that the single LSTMs are
not recurrent by themselves, while the stack of the two is
globally recurrent. A schema of the model is depicted in
Figure 2.
3.1. Encoder
The input to the encoder is a variable-length audio sequence with 40 features for each time step. At first, the input sequence is processed by two time-distributed denselyconnected layers with size of 256 and 128 respectively,
each followed by a tanh activation. The output of thedensely-connected layers is then processed by two stacked
2-dimensional convolutional layers, each having a 3 × 3 kernel and stride = 2. Let n be the sequence length and f be
the number of input features to the first convolutional layer.
The output of the first convolution is of size (16, n/2, f /2)
and for the second convolution is of size (16, n/4, f /4).
The 16 filters are then flattened to obtain an output of size
(n/4, 4 × f ), which is subsequently processed by a stackof three bidirectional LSTM layers [10]. The initial state of
the LSTM is initialized as a zero vector at the beginning of
the training, but then it is optimized via back-propagation together with the rest of the network. We found that training
the initial state gives a boost in performance and speeds up

148
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

3.4. Increased Regularization

Figure 2: Schema of our end-to-end model architecture. The
numbers on the left represent the dimensionality of each encoder layer’s output. The batch size is not written.

the model convergence.
3.2. Decoder
The decoder consists of a two-layered deep-transition
LSTM [9] followed by a deep output layer [11]. The input of the first layer is the character embedding of the last
character. The output of the first layer is used as a query
vector to compute an attention over the last layer of the encoder. The attention output is then used as input to the second
LSTM layer. The hidden and cell states received as input by
the two LSTM layers are, for every time step, the last hidden and cell states produced by the other LSTM layer. The
last encoder output is averaged over the time dimension andthis new tensor is passed as input to two different denselyconnected layers with tanh non-linearity. The two functions
compute the initialization of the hidden and cell states for the
first LSTM layer. The deep output is a densely-connected
nonlinear function, which takes as input the concatenation of
the LSTM output, the attention output and the current symbol(character) embedding, and outputs a tensor of size 512. Thistensor is finally multiplied by a second character embedding
matrix to compute the scores over the whole vocabulary.
3.3. Attention
The attention layer computes a distribution of weights thatsums up to 1 for the encoder output sequence (soft attention)with no positional information (global attention). The scoresfor each encoder position are computed according to theirrelevance with respect to the decoder state. The relevance
score is computed using the general attention score proposedin [12].

Due to the small size of the training data, we found useful toapply some regularization tricks. The first and more common
technique is the dropout applied to each layer [13]. Insteadof variational dropout [14], we preferred to use the fastest
implementation of LSTMs provided by the Pytorch library,which uses regular dropout.
Besides dropout, we applied weight normalization andlabel smoothing as additional techniques for regularization.
Weight normalization [15] is a simple technique that decomposes the parameter matrices into their magnitude and direction components in order to easily produce a transformationthat scales the weights and reduces the gradient covarianceto zero. The result is a faster convergence and a limitation of
the weight space, which has a regularizing effect.
Label smoothing [16] smooths the cross-entropy costfunction by giving a weight of 0.9 to the probability of thecorrect symbol, and 0.1 to the sum of the probabilities ofthe other symbols. Label smoothing makes the model lessconfident on its predictions, producing a regularizing effect.
In NMT, it has been observed that, despite the increased
loss and perplexity usually obtained with this technique, thetranslations are usually better [17] and end up in improved
BLEU [18] scores.

4. Experiments
In this section we summarize the experiments that motivatedour choices for the final submission. Since the goal of ourparticipation was to explore the potential of a single endto-end model that can translate directly from audio signals,
we used as training data only the Speech Translation TED
Corpus that was released for the task. No pre-training has
been performed on different types of data (such a pre-training
would in fact rely on ASR data). All our models were trainedusing the Adam optimizer [19] with learning rate of 0.001,
and values for β1 and β2 of 0.9 and 0.999. We applied
dropout of 0.2 to all layers, including the input features. The
norm of the gradients was clipped to 5. All the models havebeen trained until convergence according to the loss on aheld-out set of 1, 000 sentences (see Section 2). The results
achieved by each model on the validation set are reported
Tables 1–4.
At first, we experimented with the reference implementation of the sequence-to-sequence model6 that is based on
Tensorflow [20]. However, with about 3.5 hours per epoch
on a single NVIDIA GTX-1080 GPU, its training time resulted to be incompatible with the need of quickly testing a
range of alternative solutions. To avoid this bottleneck, we
re-implemented the same model within the fairseq toolkit,
which is highly optimized to significantly reduce training
time. Our re-implementation was indeed faster, with a reduction of the training time to about 30 minutes per epoch
for the largest version of the training corpus (“Parallel”), and
6 https://github.com/eske/seq2seq

149
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Data
ParallelClean 1
Clean 2Val. BLEU
8.54
8.98
8.54

Table 1: Results of the base model over the three different
versions of the dataset.
Data
P → C1
P → C2C1 → C2
P → C1 → C2

Val. BLEU
9.55
9.85
9.89
10.14

Strategy
Adam annealing
NAG annealing

Val. BLEU
9.11
8.74

(a) Dataset fine-tuning(b) Restart strategy

Table 2: (a) Results for the base model in different finetuning conditions. P stands for Parallel, C1 for Clean 1 andC2 for Clean 2. Only the last row refers to a double step offine-tuning. (b) Results with two different restart strategies
for the model trained on Clean 2.
about 20 minutes per epoch for the smallest one (“Clean 2”).
The wall clock time of a single training run was around 30
hours, with a maximum of 10 additional hours for the finetuning.
4.1. Dataset Selection
In the first round of experiments, we were interested in understanding the impact of the data cleaning procedures described in Section 2. To this aim, we trained the base systemon the three different versions of the dataset (i.e. “Parallel”,“Clean 1” and “Clean 2”) and evaluated the resulting models on the same validation set. The results listed in Table 1
show that Clean 1 provides us with the best result, but Clean
2 leads to a result equivalent to Parallel despite using about
36% less data. Thus, we decided to use Clean 2 for the following experiments in order to have faster training cycles.
4.2. Dataset Fine-tuning and Restart Strategy
In this subsection we address two questions. The first one iswhether it is useful to fine-tune a model trained on a larger
dataset by using a smaller and cleaner subset of the same corpus. The second question is whether a restart strategy withlearning rate annealing can improve the performance.
The first question was addressed by restarting the trainingof the model by using the new, smaller dataset as training set,
but with the same training policy and hyper parameters. The
results listed in Table 2a show that fine-tuning the model on
cleaner data always helps. In particular, fine-tuning on CleanModel
AST Seq2Seq+ Weight Normalization (WN)
+ Label Smoothing (LS)+ Sigmoid Attention+ WN and LS

Val. BLEU
8.54
8.69
8.74
8.44
9.69

Table 3: Results on the data cleaned with two cleaning steps.
2 (which is smaller but of higher quality) always results inbetter performance, especially in the case of a double step offine-tuning (P → C1 → C2). Interestingly, also using only
the clean data (C1 → C2) yields better results than training
the initial model on the original Parallel corpus.
To address the second question, we used the modeltrained on Clean 2 and restarted the training on the same
training set with a policy of learning rate annealing. To thisaim, the learning rate was multiplied by 0.5 every time thevalidation loss did not improve over the best one computed so
far [21]. We experimented using both Adam with annealing
and SGD with Nesterov Accelerated Gradient (NAG) [22]with annealing. The results listed in table 2b show that,though Adam with annealing yields a better model, both the
BLEU scores are at least 0.45 points less than the worse
model with fine-tuning.
4.3. Features Exploration
In this round of experiments we trained our base model on
the Clean 2 dataset and compared its result with models thathave weight normalization, label smoothing, sigmoidal attention instead of softmax attention, and weight normalization and label smoothing together. The results on the validation set, which are listed in Table 3, show that both weightnormalization and label smoothing give a small contribution,
while the sigmoidal attention slightly decreases the translation quality. Moreover, the joint addition of label smoothing
and weight normalization gives a sensibly higher boost, suggesting that the models need high regularization. Considering the scarce amount of data, the need for high regularization was expected. However, it is interesting to note that byincreasing the dropout to 0.3 the base model converges to amuch worse point.7 From now on, we call the model withweight normalization and label smoothing “full modell”.
4.4. Experiments with Full Model
Once we found that the full model is clearly better than theothers, we replicated the experiments on all the datasets with
the new model. In the second column of Table 4a, we can seethat this model is more sensitive to noise. In fact, training itwith the “Parallel” set leads to poor performance in translation (4.66 BLEU), but this lower translation quality was
not expected by looking at only the training and validation
7 Observed in preliminary experiments, not reported here.

150
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Data
ParallelClean 1
Clean 2

BLEU
4.66
9.69
9.69

• Primary: ensemble of 4 systems (Section 4.5).

• Contrastive: Checkpoint averaging of P→C2 (Table 4b).
The result of the primary system is 11.60 BLEU score on
our validation set and 10.24 on the test set, whereas the contrastive system scored, respectively, 10.90 and 9.70 in thevalidation and the test set.

(a)

Data
P → C1C1 → C2
P → C2
P → C1 → C2
+ Adam annealing
Ensemble of 4

Best
10.26
9.71
10.63
10.41
10.50
-

Avg
10.46
10.42
10.90
10.78
10.59
11.60

Test
9.70
10.24

5. Conclusions(b)

Table 4: Results using different versions of the dataset
for training our model with weight normalization and labelsmoothing.
losses. Nonetheless, the fine-tuning of this model on cleaner
data, whose results are listed in table 4b, leads to improvements ranging from 0.57 to 0.94 BLEU points with respect
to the models trained only on the clean data.
Unfortunately, the score of 10.63 of the best model(P→C2) represents only a limited improvement when compared with the best model in the second column of Table 2a(P→C1→C2), which improved from 8.54 of the base modelto 10.14. The fifth row of Table 4b shows the results when
the last fine-tuning is performed using Adam with annealing
instead of Adam with a fixed learning rate. Based on theseresults, we submitted our single best model (P → C2 Avg) as
our contrastive submission.
4.5. Checkpoint Averaging and Ensemble Decoding

We described FBK’s participation in the end-to-end speechtranslation task at IWSLT 2018. We have shown that data
cleaning is useful in reducing the training time by discarding a good portion of the training data, while not hurtingtranslation quality. We have also observed that fine-tuninga model using a cleaner dataset can bring improvements up
to 1.6 BLEU points. Moreover, regularizing the model withnormalization and label smoothing can produce an improvement of more than 1.0 BLEU point with clean datasets, but
the same model fails to converge to a good point using all
the data. In addition, using checkpoint averaging and ensemble decoding gave us another gain of 1.0 BLEU point.
The final score on this year’s test set is of 9.70 and 10.24
BLEU respectively for our best single model and for the primary submission based on ensemble decoding. In order toimprove the competitiveness of this system, our next experiments will include ASR for pretraining the encoder [24] orfor multi-task learning [4].

6. Acknowledgements
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GPUs used for this research.

7. References

Checkpoint averaging consists in computing the average ofdifferent checkpoints of the same training. In [23], it has
been shown that, in neural machine translation, it leads toa better translation quality than using a single model. For
each model, we computed the BLEU score on the validation
set for the last 10 checkpoints, and averaged the weights ofall the models whose results are less than 0.5 BLEU points
worse than the best one. The improvement can be observedby comparing the Best and Avg columns of Table 4b.
We also performed ensemble decoding of models trainedin different runs. The ensemble involved all the Avg checkpoints listed in table 4b, except for “C1→ C2”, which wastrained using a different vocabulary. The ensemble of the 4models obtained a result of 11.60 BLEU on the validation
set.
4.6. Submitted Systems and Results
Based on the outcomes of the above experiments on development data, we opted for submitting the following systems:[1] A. Bérard, O. Pietquin, C. Servan, and L. Besacier, “Listen and translate: A proof of concept forend-to-end speech-to-text translation,” arXiv preprint
arXiv:1612.01744, 2016.
[2] A. Bérard, L. Besacier, A. C. Kocabiyikoglu, and
O. Pietquin, “End-to-end automatic speech translationof audiobooks,” in ICASSP 2018-IEEE International
Conference on Acoustics, Speech and Signal Processing, 2018.
[3] A. Anastasopoulos and D. Chiang, “Tied multitask
learning for neural speech translation,” in Proceedingsof the 2018 Conference of the North American Chapterof the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),
vol. 1, 2018, pp. 82–91.
[4] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and
Z. Chen, “Sequence-to-sequence models can directlytranslate foreign speech,” 2017.

151
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[5] A. Anastasopoulos and D. Chiang, “Leveraging translations for speech transcription in low-resource settings,”
in Proceedings of Interspeech 2018, 2018.
[6] L. Duong, A. Anastasopoulos, D. Chiang, S. Bird, andT. Cohn, “An attentional model for speech translation
without transcription,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, 2016, pp. 949–959.
[7] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N.
Dauphin, “Convolutional sequence to sequence learning,” in International Conference on Machine Learning, 2017, pp. 1243–1252.
[8] S. Hochreiter and J. Schmidhuber, “Long short-term
memory,” Neural computation, vol. 9, no. 8, pp. 1735–
1780, 1997.
[9] R. Sennrich, O. Firat, K. Cho, A. Birch, B. Haddow,
J. Hitschler, M. Junczys-Dowmunt, S. Läubli, A. V. M.
Barone, J. Mokry, et al., “Nematus: a toolkit for neural machine translation,” in Proceedings of the Software
Demonstrations of the 15th Conference of the European
Chapter of the Association for Computational Linguistics, 2017, pp. 65–68.
[10] M. Schuster and K. K. Paliwal, “Bidirectional recurrent
neural networks,” IEEE Transactions on Signal Processing, vol. 45, no. 11, pp. 2673–2681, 1997.
[11] R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” in International Conference on Machine Learning, 2013, pp.
1310–1318.
[12] T. Luong, H. Pham, and C. D. Manning, “Effective
approaches to attention-based neural machine translation,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015,
pp. 1412–1421.
[13] Y. Gal and Z. Ghahramani, “A theoretically grounded
application of dropout in recurrent neural networks,”
in Advances in neural information processing systems,
2016, pp. 1019–1027.

[16] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, andZ. Wojna, “Rethinking the inception architecture forcomputer vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
2016, pp. 2818–2826.
[17] M. X. Chen, O. Firat, A. Bapna, M. Johnson,
W. Macherey, G. Foster, L. Jones, N. Parmar, M. Schuster, Z. Chen, et al., “The best of both worlds: Combining recent advances in neural machine translation,”
arXiv preprint arXiv:1804.09849, 2018.
[18] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu:a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting on association for computational linguistics, 2002, pp. 311–
318.
[19] D. Kingma and J. Ba, “Adam: A method for stochastic
optimization,” in Proc. of ICLR 2015, 2015.
[20] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis,
J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard,
et al., “Tensorflow: a system for large-scale machinelearning.”[21] P. Bahar, T. Alkhouli, J.-T. Peter, C. J.-S. Brix, and
H. Ney, “Empirical investigation of optimization algorithms in neural machine translation,” The Prague Bulletin of Mathematical Linguistics, vol. 108, no. 1, pp.
13–25, 2017.
[22] I. Sutskever, J. Martens, G. Dahl, and G. Hinton,“On the importance of initialization and momentum indeep learning,” in International conference on machinelearning, 2013, pp. 1139–1147.
[23] M. Junczys-Dowmunt, T. Dwojak, and H. Hoang, “Isneural machine translation ready for deployment? acase study on 30 translation directions.”[24] S. Bansal, H. Kamper, K. Livescu, A. Lopez,
and S. Goldwater, “Pre-training on high-resource
speech recognition improves low-resource speech-totext translation,” arXiv preprint arXiv:1809.01431,
2018.

[14] D. P. Kingma, T. Salimans, and M. Welling, “Variational dropout and the local reparameterization trick,”in Advances in Neural Information Processing Systems,
2015, pp. 2575–2583.
[15] T. Salimans and D. P. Kingma, “Weight normalization:
A simple reparameterization to accelerate training ofdeep neural networks,” in Advances in Neural Information Processing Systems, 2016, pp. 901–909.

152
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

The JHU/KyotoU Speech Translation System for IWSLT 2018
Hirofumi Inaguma† , Xuan Zhang, Zhiqi Wang, Adithya Renduchintala, Shinji Watanabe, Kevin DuhCenter for Language and Speech Processing, Johns Hopkins University, Baltimore, USA†
Graduate School of Informatics, Kyoto University, Japan{xuanzhang,zwang132,adithya.renduchintala,shinjiw}@jhu.edu
inaguma@sap.ist.i.kyoto-u.ac.jp, kevinduh@cs.jhu.edu

Abstract

2. DataThis paper describes the Johns Hopkins University (JHU)
and Kyoto University submissions to the Speech Translation
evaluation campaign at IWSLT2018. Our end-to-end speechtranslation systems are based on ESPnet and implements anattention-based encoder-decoder model. As comparison, we
also experiment with a pipeline system that uses independentneural network systems for both the speech transcription andtext translation components. We find that a transfer learning approach that bootstraps the end-to-end speech translation system with speech transcription system’s parameters isimportant for training on small datasets.

We build our systems on the following provided corpora:

1. Introduction
We report on our efforts on the IWSLT 2018 Speech Translation task. The goal of the 2018 task is to build and evaluate English-to-German speech translation systems on the domain of lectures and TED talks. We build two systems:• Pipeline System: English (EN) speech transcription
system using a joint CTC-attention model (Section3.1), followed by a English-to-German (EN-DE) texttranslation system using a RNN-based sequence-tosequence model (Section 3.3).
• End-to-End System: English-to-German (EN-DE)
speech translation system using an RNN-based
sequence-to-sequence model transferred from the joint
CTC-attention model (Section 4).
The main challenge is to develop end-to-end neural systems that are trainable given the small amount of data (ofEnglish speech matched to German text). We find that bootstrapping the end-to-end system with the parameters of an
English-only speech transcription system (i.e. ASR of English speech to English text) was helpful.
Generally, we are interested in comparing the relative
merits of end-to-end vs. pipeline approaches. Currently,
our pipeline system outperforms the end-to-end system, evenwhen trained on the same number of utterances, suggestingthat there is much room for future work in end-to-end models.
† Work carried out as a visiting scholar at JHU.

1. Speech-Translation TED corpus (ST TED): This datacontains English speech (EN-s), the corresponding
English transcription (EN-t), as well as the Germantranslation (DE-t). We use this to train both pipeline
and end-to-end systems. In particular, (EN-s,EN-t) isused to train the pipeline’s speech transcription component (Section 3.1); (EN-t, DE-t) is used to train
the pipeline’s text translation component (Section 3.3);and (EN-s, DE-t) is used to train the end-to-end sytem(Section 4).
2. TED LIUM corpus (TEDLIUM2): This data contains
English speech (EN-s) and their English transcription (EN-t). We use this as additional data to train
the pipeline system’s speech transcription component,
which is also used to initialize the end-to-end system.
3. WMT 2018 data, filtered to the TED domain using Moore-Lewis data selection [1] (WMT-Filtered):
We trained 5-gram Engish language models on TED(LMT ED ) and a random sample of the WMT data
(LMW M T ), then selected the top 1 million WMT bitext according to the perplexity difference between
LMT ED and LMW M T . Finally, we filtered all sentences that were longer than 100 tokens or had an outof-vocabulary rate (with respect ST TED dictionary)of 10% or larger. This is used to augment the training
data for the pipeline’s text translation component.
For data preprocessing of transcriptions and translations
in all languages, we normalized punctuation and performed
tokenization using the Moses scripts1 . For both the pipeline’s
speech transcription and the end-to-end speech translation,
we used a fixed vocabulary of 5k or 10k wordpieces, whichwere composed from characters to words and generated using sentencepiece2 . We used the same dictionary including
both EN and DE wordpieces to capture the common words
in both languages.
1 normalize-punctuation.perl
andhttps://github.com/moses-smt/mosesdecoder
2 https://github.com/google/sentencepiece

153
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

tokenizer.perlinFor the pipeline’s text translation component, we experimented with different kinds of subword units. For simplicity, we did not use any truecase models for any systems andworked directly with the natural casing. Table 1 shows thedata sizes in each corpus.
For feature extraction for speech transcription and translation, we extracted 80-channel log-mel filterbank outputs
with 3-dimensional pitch features computed with a 25ms
window and shifted every 10 ms using Kaldi [2]. The features were normalized by the mean and the standard deviation on the whole training set (excluding our developmentset). We removed utterances having more than 3000 frames
or more than characters due to the GPU memory capacity.
corpusSpeech-Translation TEDTEDLIUM2
WMT-Filtered#utterance
166,214
258,943
988,697

speech datasize
271 hours
210 hours-

Table 1: Data size in each corpus.

3.1.2. Attention-based encoder-decoder
Attention-based encoder-decoder [4, 5] is another sequencelabeling model which directly predicts output sequences.
Unlike the CTC framework, this approach does not make any
conditional independence assumptions, where the model predicts each token conditioned on all previous tokens.
P (y|x) =

U
Y

u=1

P (yu |y1 , . . . , yu−1 , x)

Attention-based encoder-decoder model consists of two
modules: the encoder and decoder. The encoder network
maps input features x into high-level distributed representation h, and the decoder network picks up a portion of hwith a scoring function given encoder and decoder hidden
states, which is called the attention mechanism. We used thelocation-aware scoring function, which takes previous attention weights into account. The loss function is designed as
the negative log probabilities as follows:
Latt = − ln P (y|x)

3. Pipeline System
3.1. Speech Transcription Component

3.1.3. Joint CTC-attention

In this section, we briefly describe the joint CTC-attention
framework for the speech transcription (i.e. ASR) component. Let x = (x1 , . . . , xT ) be acoustic features andy = (y1 , . . . , yU ) be the corresponding target sentence inthe same language as x.

We introduce the multitask learning (MTL) framework with
the CTC objective in the training of the attention-based
encoder-decoder model [6]. This approach has two advantages: 1) it encourages monotonic alignments in the encoder network, which leads to fast convergence and removes
inappropriate alignments in long sequences, 2) it leads tosequence-level optimization. The loss function of the joint
CTC-attention framework is designed as an interporation of
Lctc and Latt with a tunable parameter λ (0 ≤ λ ≤ 1):

3.1.1. Connectionist Temporal Classification (CTC)
Connectionist Temporal Classification (CTC) [3] is a latentvariable model which directly maps the input sequence into
the output sequence of shorter length. To compensate the differences of sequence lengths, CTC introduces an additional”blank” symbol. The CTC loss function is defined as thesummation of negative log probabilities of all possible paths
mapped from ground truth labels interleaved with blank labels.
Lctc = − ln P (y|x)
X
P (π|x)= − lnπ∈B−1 (y)where π represents a CTC path, and B represents a collapse function which maps all the CTC paths into the uniqueground truth labels y by removing all blank labels. Basedon the conditional independence assumption, posterior probabilities P (π|x) is factorized frame by frame as follows:
P (π|x) =

T
Y

t=1

P (πt |ht )where ht represents an activation of the top layer of theencoder. P (π|x) is effectively calculated by the forwardbackward algorithm.

Lmtl = λLctc (y|x) + (1 − λ)Latt (y|x)
In addition, scores from CTC outputs are taken into account in the beam search decoding of the attention-basedmodel during the inference stage [7, 8]. Because CTC isframe-synchronous, hyper-parameters tuning such as length
penalty and coverage penalty are not neccesary any more inorder to prune inappropriate hypotheses.
3.2. Evaluation of Speech Transcription Component
Preprocessing: We used the Speech-Translation TED
corpus augmented with TED LIUM corpus, totaling 481h.
With regard to the official development sets provided by
the IWSLT organizers (dev2010, tst2013 etc.), there isno segmentation information of the start and end time ofutterances. Therefore, we sampled 4k utterances from the
Speech-Translation TED corpus as the validation set, andremoved them from the original training data. For evaluation, we segmented each audio file in the development setswith the LIUM SpkDiarization tool [9] first, then performed
MWER segmentation with the toolkit from RWTH [10] as

154
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

in the baseline implementation provided by organizers3 .
Architecture: We built end-to-end ASR models with
the ESPnet toolkit [11] with a pytorch backend [12]. For
the encoder part, we used 2-layers CNN layers with maxpooling layers followed by 3 or 5-layers of 1024 dimensional
bidirectional LSTM [13], resulting in 4-fold time reduction.
For the decoder part, we used 2-layers of 1024 dimensional
LSTM. We did not conduct regularization such as dropout,label smoothing [14, 15], scheduled sampling [16] forspeech transcription.
Optimization: Our systems were optimized with the
AdaDelta algorithm with epsilon annealing for 15 epochs.
The weight for CTC loss λ was empirically set to 0.5.
Decoding: We conducted beam search decoding withbeam width 20. LSTM language model of 2 layers with 650
hidden units trained on the same parallel corpus was used.
Results: Results for the TEDLIUM2 corpus are shownin Table 2. 5k wordpiece units are always better than 10k in
this corpus. We also conformed the consistent improvementswith deeper encoders (3 layers → 5 layers). Results forthe official development sets are shown in Table 3. As in
Table 2, 5k units are better than 10k units, but we cannot see
improvements by adding encoder layers. We suspect that
this is due to the quality of audio segmentation by the LIUM
SpkDiarization tool and utterance matching by the RWTH
MWER tool.
#unit
10k
5k
10k
5k

#layer
3
3
5
5

dev
13.8
12.1
13.3
11.6

test
12.3
11.1
12.5
10.7

Table 2: Word error rate (WER) evaluated on the
TEDLIUM2 corpus. #unit represents the number of untis in
the softmax layer. #layer represents the number of BiLSTM
layers following CNN layers in the encoder network.
#unit
10k
5k
10k
5k

#layer3
3
5
5

dev2010
28.2
25.6
28.8
27.4

test2010
29.5
27.7
31.2
30.8

test2013
31.9
30.6
33.1
32.0

test2014
32.6
31.1
34.5
33.7

test2015
46.5
44.4
45.6
47.9

Table 3: Word error rate (WER) evaluated on the official development sets.

3.3. Text Translation Component
Preprocessing: We built neural machine translation (NMT)
systems for the English-German text translation componentof our pipeline system. These systems were trained on the
3 https://github.com/isl-mt/SLT.KITST TED corpus, with English manual transcript for speech
recognition on the source side and corresponding Germantranslation on the target. Training data were tokenized andsplit into subwords using Byte Pair Encoding (BPE) [17].
We set the number of BPE merge operations to be 20k forthe source side — same for the target side. The validation
set used for early stopping consists of around 4k utterances,and they were randomly sampled from the corpus.
Architecture: The attention-based NMT models consist of two components: an encoder network, which is arecurrent neural network (RNN), that provides a representation of the input sentence, and a decoder network, whichis also a RNN, that generates translation based on the input
context with attention mechanism [5, 18] applied.
We trained our NMT systems with Sockeye [19]. In
the model we used based on hyper-parameter tuning, theencoder and decoder both had 2 layers with 512 LSTM
hidden units on each layer and we applied dot productattention for RNN decoders. Both the source and target
embedding vectors were set to 512. We used word-countbased batch of size 4096 words and maximum sequencelength 100. For regularization, the RNN inputs and states
dropout rates for both the encoder and the decoder were set
to 0.1.
Optimization: Our systems employed the Adam optimizer to reduce the cross-entropy loss with an initial
learning rate 0.0005. We made a checkpoint after every 2000
batch updates, and if the model had not improved in perplexity on the validation data for more than 8 checkpoints,
we would perform early stopping for the training process. In
general, it takes around 50 epochs (about 10 hours) for themodel to converge.
#BPE merge ops
avg dev BLEU

20k
23.83

30k
24.18

40k
24.28

50k
23.77

Table 4: Effect of different number of BPE merge operations on average BLEU score on development sets. The initial learning rate was set to 0.0007.
Hyper-parameter Tuning: Hyper-parameters weretuned based on systems’ average decoding performance(BLEU score) on dev2010, tst2010, tst2013, tst2014 andtst2015 set. We searched the number of BPE merge operations from 20k, 30k, 40k and 50k, word embedding size andthe number of RNN hidden units from 512 and 1024, batch
size from 4096 and 6000, initial learning rate from 0.0002 to
0.0007, dropout probability from 0.1 and 0.24 .
When searching for a good hyper-parameter configuration, we found that a more complex model, in terms of RNN
hidden size, was not necessarily needed to get better performance on this corpus: when we increased the number of
4 Due to time and computational resources limitation, we only tried asubset of all the possible combinations.

155
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

systemManual: NMT (ST TED) on EN reference
Pipeline: NMT (ST TED) on ASR output
Manual: NMT (ST TED + WMT-Filtered) on EN reference

dev2010
23.96
15.47
28.07

tst2010
27.54
20.54
30.59

tst2013
26.23
16.68
29.47

tst2014
22.30
14.35
26.04

tst2015
25.07
16.21
26.70

Table 5: BLEU comparison of NMT translating English reference (Manual) or ASR ouput (Pipeline). BLEU scores are evaluatedon development sets using multi-bleu.perl with the Moses tokenization.
RNN hidden units from 512 to 1024, the average BLEU score
dropped from 24.71 to 24. Another interesting finding wasthat with other hyper-parameters fixed, when the number of
BPE operations increased, the BLEU score on the development data tended to first go up and then decrease (see Table4). Finally, the initial learning rate turned out to be an important hyper-parameter to tune. For example, we got 23.44
BLEU with initial learning rate 0.0003, but 24.18 BLEU with
0.0007.
3.4. Evaluation of Pipeline System
We show the main results of our pipeline systems in Table 5.
For the purpose of comparison, we provided the NMT systems with either the manual transcripts (Manual) or the output of our ASR system (Pipeline), which is described in Section 3.1. As expected for error cascading in pipeline systems,
BLEU scores drop substantially, by up to 36.4%, when translating noisy ASR outputs compared to translating the clean
English transcript.
A paired permutation test shows that Manual outperforms Pipeline statistically significantly with p-value < 1%.
NMT systems trained with good manual transcripts might beintolerant to various ASR errors, and it is very likely they
will propagate the errors during decoding.
Additionally, the final row in Table 5 we show the BLEU
scores of the NMT system trained with additional WMTFiltered data. There is a large improvement, for example
from 23.96 to 28.07 on the dev2010. This confirms that
adding more bitext helps.
While the corpus-level BLEU score of the pipeline islower than the manual system, we did observe some interesting variances at the level of individual sentences: it is not
the case that translations of ASR outputs are always worse
than translations of manual, clean transcripts. Figure 1 compares the sentence-level BLEU scores in three different scatter plots. For each sentence in tst2010, we have the English
transcript (EN-ref) and the resulting translation (DE-manual)by our NMT system; we also have the English ASR output(EN-ASR) and the resulting translation (DE-pipeline). Finally we have the correct German reference (DE-ref). We
then computed three sentence-level BLEU scores (with addone smoothing) as follows:• Manual BLEU: BLEU of DE-manual vs. DE-ref• Pipeline BLEU: BLEU of DE-pipeline vs. DE-ref• ASR BLEU: BLEU of EN-ASR vs. EN-ref

Our goal is to compare ASR BLEU (which measures
whether the English sentence was difficult to transcribe) with
Manual/Pipeline BLEU. Our original hypothesis is that sentences with low ASR BLEU should result in a larger difference in Manual BLEU minus Pipeline BLEU.
Interestingly, as seen in Figure 1 (c), there are individual
sentences where Manual BLEU is less than Pipeline BLEU.
An example is shown in Table 6. The difference between the
English reference and the ASR output is ”where it gets” vs”what gets”, which are arguably both correct. However, theNMT result is very different, one translating perfectly andthe other not. It appears that since NMT output has highvariance, i.e. it can output very different translations evenwhen the inputs are semantically similar.

4. End-to-End System
In this section, we describe our end-to-end speech translation
model and transfer learning from pre-trained ASR model.
4.1. Model for End-to-End Speech Translation
We used an attention-based encoder-decoder model for theend-to-end speech translation model. The architecture of
the encoder is exactly the same as that in the ASR modelin Section 3.1 (VGG-like CNN layers followed by stacked
BiLSTM layers). The decoder includes two modificationfrom the ASR decoder: (1) adopting input-feeding mechanism [18], and (2) adding scheduled sampling [16].
It is possible to integrate a language model during thedecoding stage (i.e. shallow fusion [20]) and also training
stage (i.e. deep fusion [21] and cold fusion [22]), but we did
not use any language models for the speech translation task
in this paper. We’ll leave them to the future work.
4.2. Transfer learning from ASR
In our preliminary experiments, it took too much time to train
end-to-end speech translation models from scratch, i.e. manyepochs are required for convergence. Therefore, we exploredmethods to better initialize our end-to-end model.
Speech translation can be viewed as a combination of
ASR and MT tasks, so we can treat the encoder and decoder networks as having roles in ASR and MT, respectively.
Therefore, it is a natural choice to initialize the encoder withthat of a pre-trained ASR model. Initialing the decoder withpre-trained MT model will be left to the future work.
Weiss et al. [23] shows improvements of BLEU scores

156
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

English Reference
ASR Output

NMT Result (Manual)
NMT Result (Pipeline)

But here’s where it gets interesting.
But here’s what gets interesting.

Aber hier ist das, was interessant wird.
Aber hier wird es interessant.

Table 6: An example where Pipeline system outperforms the Manual system (100 sentBLEU vs. 6.5 sentBLEU). The German
reference for the utterance is Aber hier wird es interessant .
100

100

Slope: 0.27
Intercept: 1.1
r_value: 0.39
p_value: 9e-50

60

40

20

60

40

20

0
20

40

60

ASR BLEU

80

100

60

40

20

0
0

Slope: 0.81
Intercept: 3.4
r-value: 0.8
p_value: 1.5e-302

80

Pipeline BLEU

80

Pipeline BLEU

Manual BLEU

80

100

Slope: 0.2
Intercept: 5.5
r-value: 0.29
p_value: 1.5e-26

0
0

20

40

60

ASR BLEU

(a)

80

100

0

20

40

60

Manual BLEU

(b)

80

100

(c)

Figure 1: Sentence-level BLEU of manual, pipeline and ASR system on tst2010. A linear least-squares regression is calculated
for each pair of systems.
by multi-task learning (MTL) with ASR task by sharing theencoder. Berard et al. [24] also shows both MTL and pretraining strategies lead to fast convergence and better resultsfor end-to-end speech translation. Here we chose the pretraining strategy and transfer the weights of the encoder inthe ASR model to the end-to-end model prior to training.
4.3. Experiments
We did the same data preprocessing as speech transcription
in Section 3.2. We also built end-to-end speech translation
models with the ESPnet toolkit with a pytorch backend. The
differences of the architecture, optimization, and decoding
from speech transcription models are as follows:• We did not use the CTC framework due to its monotone assumption• We used scheduled sampling with probability 0.2• We ran for 30 epochs• We did not perform beam search decoding (i.e. greedy
decoding)• We did not use any language models (due to time constraints)
We use the official scripts from the organizer and calculatedcase-sensitive BLEU scores with multi-bleu-detok.perl in theMoses toolkit after detokenization. We report BLEU scoresin Table 7 for both pipeline systems and end-to-end speechtranslation models (E2E). There are several observations:First, we can confirm that better ASR models led to better BLEU scores in the pipeline systems when comparing

Table 3 and Table 7. The two ASR models with 5k units
have the lowest WER scores, and the resulting two pipeline
systems (b) and (d) also achieved the best BLEU scores. Second, it seems challenging to train an E2E speech translation
model from scratch. Transfer learning with parameters froman existing ASR model gave consistent gains.
Finally, there are large differences between pipeline andend-to-end systems. For example, on dev2010, E2E trained
from scratch achieved a BLEU of 4.44, E2E with transferfrom ASR achieved a BLEU of 6.71, and the pipeline systems achieved BLEU in the range of 14. This may be due
to data sparseness. Perhaps the explicit intermediate representation of transcripts in the language of the speech input
is important for constraining the model complexity. Further,
10k wordpieces is a relatively large unit size for speech models and the data needs of an end-to-end model may be larger
than that of a pipeline model.
We show some examples of the end-to-end speech translation model transferred from pre-trained ASR (system (f) in
Table 7) in Table 8. Despite the low BLEU scores in general,
the end-to-end model sometimes do generate reasonable sentences and correctly predicts keywords such as proper nouns
and numbers. Our system was robust to misspelling because
we used 10k units for the vocabulary. The official development sets include many long sentences, and it appears that
our E2E model may be doing relatively worse compared to
Pipeline systems on long sentences.

5. Discussion
We described our pipeline and end-to-end speech translation
systems for IWSLT 2018. For the official evaluations, we
submitted the pipeline system (a) in Table 7 as a contrastive

157
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

System(a) Pipeline(b) Pipeline(c) Pipeline(d) Pipeline(e) E2E(f) E2E

Configuration
ASR (10k unit, 3 layer); NMT (ST TED)
ASR (5k unit, 3 layer); NMT (ST TED)
ASR (10k unit, 5 layer); NMT (ST TED)
ASR (5k unit, 5 layer); NMT (ST TED)
train from scratchtransfer learning from ASR parametersdev2010
14.22
14.68
14.54
14.87
4.44
6.71

test2010
13.62
14.70
13.05
13.76
4.10
6.21

test2013
14.21
15.08
14.60
14.75
3.57
6.01

test2014
11.73
12.23
11.73
11.58
3.52
5.08

test2015
10.68
11.59
11.16
10.96
2.42
4.51

Table 7: BLEU evaluated on the development sets using the official scripts provided by organizers. Note the results here are notcomparable to Table 5 due to differences in the tokenization and evaluation scripts.
EN(Ref)
DE (Ref)
DE (Hyp)
EN(Ref)
DE (Ref)
DE (Hyp)

In the last five years we’ve added 70 million tons of CO2 every 24 hours – 25 million tons every day to the oceans.
In den letzten 5 Jahren haben wir 70 Millionen Tonnen an CO2 produziert alle 24 Stunden – 25 Millionen Tonnen jeden Tag in die Ozeane.
In den letzten fnf Jahren haben wir die 70 Millionen Tonnen CO2 / h. Wir haben die Ostkste
But not just any mission, it’s a mission that is perfectly matched with your current level in the game.
Aber nicht nur irgendeine Mission, sondern eine Mission, die perfekt zu Ihrem aktuellen Level im Spiel passt, richtig?
Aber nicht nur die Mission, sondern nur eine Mission, die sich perfekt antreibt. Mit dem deren auf dem Spiel.

Table 8: Examples of the end-to-end speech translation model (system (f) in Table 7)
system and the E2E system (f) in Table 7 as the primary system; they were our best systems at the time of submission.
Our main findings are that (1) pipeline systems can be verystrong systems, and that (2) more work is needed to train
end-to-end systems effectively, especially in small datasets.
For the official development sets, we had to use othertools to segment audio files before decoding and then matchthe number of references and hypotheses after decoding. We
found that this affected WER and BLEU scores seriously due
to misalignment. Therefore, the exact segmentation information for acoustic features is desired for the future evaluationin speech translation.

6. Acknowledgements
We would like to thank Pamela Shapiro for providing theMoore-Lewis data selection toolkit.

7. References[1] R. C. Moore and W. Lewis, “Intelligent selectionof language model training data,” in Proceedings of
the ACL 2010 Conference Short Papers. Uppsala,Sweden: Association for Computational Linguistics, July 2010, pp. 220–224. [Online]. Available:
http://www.aclweb.org/anthology/P10-2041[2] D. Povey, A. Ghoshal, G. Boulianne, L. Burget,
O. Glembek, N. Goel, M. Hannemann, P. Motlicek,
Y. Qian, P. Schwarz, et al., “The kaldi speech recognition toolkit,” in Proc. of ASRU, 2011.
[3] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling
unsegmented sequence data with recurrent neural networks,” in Proc. of ICML, 2006, pp. 369–376.
[4] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho,and Y. Bengio, “Attention-based models for speech
recognition,” in Proc. of NIPS, 2015, pp. 577–585.
[5] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,” inProc. of ICLR, 2015.
[6] S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention
based end-to-end speech recognition using multi-task
learning,” in Preceedings of ICASSP. IEEE, 2017, pp.
4835–4839.
[7] T. Hori, S. Watanabe, Y. Zhang, and W. Chan, “Advances in joint ctc-attention based end-to-end speech
recognition with a deep cnn encoder and rnn-lm,” arXiv
preprint arXiv:1706.02737, 2017.
[8] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, andT. Hayashi, “Hybrid ctc/attention architecture for endto-end speech recognition,” IEEE Journal of Selected
Topics in Signal Processing, vol. 11, no. 8, pp. 1240–
1253, 2017.
[9] S. Meignier and T. Merlin, “Lium spkdiarization: an
open source toolkit for diarization,” in CMU SPUD
Workshop, 2010.
[10] O. Bender, R. Zens, E. Matusov, and H. Ney, “Alignment templates: the rwth smt system.” in IWSLT, 2004,
pp. 79–84.
[11] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, et al., “Espnet: End-to-end speechprocessing toolkit,” arXiv preprint arXiv:1804.00015,
2018.
[12] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang,
Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, andA. Lerer, “Automatic differentiation in pytorch,” 2017.

158
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[13] S. Hochreiter and J. Schmidhuber, “Long short-term
memory,” Neural Computation, vol. 9, no. 8, pp. 1735–
1780, 1997.
[14] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, andZ. Wojna, “Rethinking the inception architecture forcomputer vision,” in Proc. of CVPR, 2016, pp. 2818–
2826.
[15] J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” in Proc. of Interspeech, 2017.
[16] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer,“Scheduled sampling for sequence prediction with recurrent neural networks,” in Preceedings of NIPS,
2015, pp. 1171–1179.
[17] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,”
arXiv preprint arXiv:1508.07909, 2015.
[18] M.-T. Luong, H. Pham, and C. D. Manning, “Effective
approaches to attention-based neural machine translation,” arXiv preprint arXiv:1508.04025, 2015.
[19] F. Hieber, T. Domhan, M. Denkowski, D. Vilar,
A. Sokolov, A. Clifton, and M. Post, “Sockeye: A
toolkit for neural machine translation,” arXiv preprint
arXiv:1712.05690, 2017.
[20] A. Kannan, Y. Wu, P. Nguyen, T. N. Sainath, Z. Chen,
and R. Prabhavalkar, “An analysis of incorporating an
external language model into a sequence-to-sequencemodel,” 2017.
[21] C. Gulcehre, O. Firat, K. Xu, K. Cho, L. Barrault, H.C. Lin, F. Bougares, H. Schwenk, and Y. Bengio, “On
using monolingual corpora in neural machine translation,” arXiv preprint arXiv:1503.03535, 2015.
[22] A. Sriram, H. Jun, S. Satheesh, and A. Coates,“Cold fusion: Training seq2seq models together withlanguage models,” arXiv preprint arXiv:1708.06426,
2017.
[23] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu,
and Z. Chen, “Sequence-to-sequence models candirectly translate foreign speech,” arXiv preprint
arXiv:1703.08581, 2017.
[24] A. Bérard, L. Besacier, A. C. Kocabiyikoglu, and
O. Pietquin, “End-to-end automatic speech translation of audiobooks,” arXiv preprint arXiv:1802.04200,
2018.

159
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Adapting Multilingual NMT to Extremely Low Resource Languages
FBK’s Participation in the Basque-English Low-Resource MT Task, IWSLT 2018
Surafel M. Lakew†+ , Marcello Federico⋆†

University of Trento, + Fondazione Bruno Kessler, Trento, Italy⋆
Amazon AI, East Palo Alto, CA 94303, USA
lastname@fbk.eu, ⋆ marcfede@amazon.com

Abstract
Multilingual neural machine translation (M-NMT) has recently shown to improve performance of machine translationof low-resource languages. Thanks to its implicit transferlearning mechanism, the availability of a highly resourced
language pair can be leveraged to learn useful representation for a lower resourced language. This work investigates
how a low-resource translation task can be improved within amultilingual setting. First, we adapt a system trained on multiple language directions to a specific language pair. Then,
we utilize the adapted model to apply an iterative traininginference scheme [1] using monolingual data. In the experimental setting, an extremely low-resourced Basque-English
language pair (i.e., ≈ 5.6K in-domain training data) is ourtarget translation task, where we considered a closely related French/Spanish-English parallel data to build the multilingual model. Experimental results from an i) in-domain
and ii) an out-of-domain setting with additional training data,
show improvements with our approach. We report a translation performance of 15.89 with the former and 23.99 BLEU
with the latter on the official IWSLT 2018 Basque-English
test set.

1. Introduction
The amount and diversity of model training data have beenshown to affect the performance of Neural Machine Translation (NMT) system [2]. The direct relation between dataset
size and performance of NMT [3], calls for alternative approaches to improve low-resource language translation.
Multilingual models that constitute more than one language pair has been shown to improve the translation performance of the low-resources language direction [4, 5]. In it’s
simplified and most effective setting, building an M-NMT
system requires only an additional “language-flag” on thedata level. Then, the attentional encoder-decoder based NMT
model can be trained with the aggregation of several language pairs. The flag functions as a mechanism to trigger
and direct the generation of target tokens in a specific target
language. Thus, when the training set is constructed withthe merge of several language directions, the latent transferlearning across languages within the conventional NMT architecture showed to improve low-resourced language pairs.
However, M-NMT training mechanism is biased towards
generating the language pair with the largest portion of training data [1]. This bias will consequently limit the expectedlevel of improvement in translating low-resource language
pairs.
In this work, we propose a progressive adaptation ofa multilingual model to a single language pair. We cast
the adaptation stage in iterative training-inference operations
that utilize monolingual data. Assuming, the availabilityof a low-resource language pair and a high resource related/language pairs data, we specifically explore the following two mechanisms:• Adapting a multilingual model trained with several
language directions to a specific low-resourced language pair, with the aim to avoid ambiguities at thetime of inference.
• Then, applying an iterative training-inference using
monolingual data of the low-resourced pair, with theaim to acquire a more cleaner pseudo-parallel corpus
for the next adaptation stage.
In our experimental setting, we apply the above two mechanisms for improving the extremely low-resourced (ELR)
Basque(EU)-English(EN) language pair. Then, with the experimental results and discussion we present our participation of the IWSLT-20181 shared task on Low Resource MT
of TED2 talks from Basque to English direction. We evaluated our approach with the i) ELR training condition ina constrained in-domain data, and ii) by adding an out-ofdomain training data in addition to the in-domain. We train
both models in a similar language setting (i.e., the additional/related language pairs are French/Spanish-English).
For comparing our approach, we train a bilingual(Basque-English) baseline and multilingual baseline modelby adding more data from the related language pairs. Morespecifically, to build the M-NMT model Basque-French and
1 https://sites.google.com/site/iwsltevaluation2018/TED-tasks
2 https://wit3.fbk.eu/(*) Work conducted while this author was at FBK.

160
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Basque-Spanish with a similar ELR condition, and FrenchEnglish and Spanish-English with the relatively high resource data size are added to the bilingual model.
Allmodels share common configurations at training and inference time, unless stated differently. Models are trained following [4], preprocessing and training procedures using theTransformer model [6].
In the following sections, we begin by introducing NMT(§2). Following, we review the related work in multilingual
models and transfer-learning (§3). In Section 4, we describe
our model training approach, followed by dataset and preprocessing, experimental settings, and baseline models (§5).
Finally, we give further analysis on the experimental resultsin Section 6.

2. Neural Machine Translation
A standard state-of-the-art NMT system comprises an encoder, a decoder and an attention mechanism, which are
all trained with maximum likelihood in an end-to-end fashion [7]. Although there are different variants of the encoderattention-decoder based approach, Recurrent variants being
the predominant until recently [8], this work utilizes the“Transformer” model [6]. The encoder is purposed to encode a source sentence into hidden state vectors, whereas the
decoder uses the last representation of the encoder to predictsymbols in the target language. In a broad sense, the attention mechanism improves the prediction process by deciding
which portion of the source sentence to emphasize at a time[9]. Nevertheless, in the Transformer architecture, the application of attention spans to the representation of encoder
latent and decoder latent space.
The Transformer architecture works by relying on a selfattention (intra-attention) mechanism, removing all the recurrent operations that are found in the RNN approach. In
other words, the attention mechanism is repurposed to compute the latent space representation of both the encoder andthe decoder sides. However, with the absence of recurrence,
positional-encoding is added to the input and output embeddings. Similarly, as the time-step in a recurrent network,
the positional information provides the Transformer networkwith the order of input and output sequences.
In our work, we use the absolute positional encoding, butvery recently the use of the relative positional information
has been shown to improve performance [10]. The model isorganized as a stack of encoder-decoder networks that works
in an auto-regressive way, using the previously generated
symbol as input for the next prediction. Both the decoderand encoder can be composed of uniform layers, each built of
sub-layers, i.e., a multi-head self-attention layer and a position wise feed-forward network (FFN) layer. The multi-head
sub-layer enables the use of multiple attention functions witha similar cost of utilizing attention, while the FFN sub-layer
is a fully connected network used to process the attention
sublayers; as such, FFN applies two linear transformations
on each position and a ReLU [6].

3. Related Works
3.1. Multilingual NMT
Prior to the introduction of a shared attention mechanism[11], early works in multilingual NMT utilizes separate encoder, decoder and an attention mechanism to support thetranslation of either many-to-one [12], or one-to-many [13]language directions. Moreover, Firat et al. [11] introduced a many-to-many system, however, relying on separate
encoder-decoder setup. In a simplified yet delivering better
performance [4] and [5] introduced a “language-flag” based
approach that shares the attention mechanism and a single
encoder-decoder networks to enable multilingual models. In
this work, we follow the Johnson et al. [4] approach forprepending a language-specific flag at the source side of the
training and inference examples.
3.2. Transfer Learning and Model Adaptation
Zoph et al., (2016) [14], proposed how transfer-learning between two NMT models can improve a low-resourced MT
task. In their approach, a language pair with the relatively
large amount of data is first utilized to train a parent model,
then the encoder-decoder parameters are transferred to initialize a child model for a low-resourced language pair. Afterinitializing, in the fine-tunning stage, the parameters of thechild decoder network is fixed. The main motivation behind
updating only the encoder parameters is that the decoder language across the parent-child models stays the same. Similarly, the parent-child approach has been extended to analyze
the effect of using related languages on the source side of theencoder-decoder network [15].
In a related way to benefit the low-resource language
from the high resourced pair [16] proposed an alternativetransfer-learning approach built on a component that allows
to share lexical and sentence level representations of multiple source language to a single target language. In a prior
work, a multi-source approach where two or more encoders
shares an attention mechanism has been suggested in [17], toaddress the ambiguities of translating a source token to a single target language. Unlike [18] where a single multilingual
model is used for several language translations [19] showed
how adapting the multilingual model on a specific language
pair improves performance. Recently [20] explored the advantage of initializing a low-resource language pair training
using a pre-trained multilingual model showing a significant
improvement over baseline approaches.

4. Adaptation from a Multilingual Model
This work aims to exploit the transfer-learning across languages, however, instead of the parent-child strategy [14],we rely on using a multilingual model as in [4] that allows
to abstract the representation of several languages in a single
attentional encoder-decoder model. We hypothesis if data
is received both for the low and high resourced language

161
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

pairs, training a single model with the concatenation of all thedata and progressively fine tunning it with the low-resource(target-task) language pair can avoid possible ambiguities
between languages at the time of inference.
First, we train a model with all the available language
pairs (including the target-task). Second, we adapt the best
performing model to the target-task language pair. Unlike
the recently proposed approach [20], we adapt using thesame target-task data that has been utilized for training thebaseline multilingual model. The main reason behind thisis that the target-task data is already received at time oftraining the multilingual model. Then, the (latest) adapted
model is used to perform back-translation [21] in a target →source direction or an iterative dual-inference in a source ↔
target directions [1]. However, both inference approaches
are used to create a source side synthetic data, the dualinference requires an available monolingual data both fromthe source and target language. More importantly, the factthat we adapt from the multilingual to a bi-directional modelallows us to avoid the use of auxiliary models (i.e., a separate model trained in a target → source direction) to performthe inference operations. After the inference stage, we continue training the model by combining the target-task and the
newly formed source (synthetic) → target parallel data, consequently creating a progressive adaptation stages.
In the experimental section, the adaptation and progressive update of the multilingual model to the single language
pair (Basque-English) target-task are evaluated in two settings:• iELR, an extremely low-resource language pairtrained and evaluated using an in-domain parallel andmonolingual data.
• oELR, an extension of the iELR training condition
with an additional out-of-domain parallel and monolingual data, as described in Section 5
In the following Section, the details of the experimental setup
are given for the two evaluation scenarios.

EU-EN
EU-FR
EU-ES
FR-EN
ES-EN
EU-MonoEN-Mono

TED
5623
5815
5546
287134277093242831

Opus16/18
856314689358840458503970PaCo130359-OpenData
926203741254-

Table 1: Languages and dataset size of the training set. TED
represents the in-domain data, whereas the Opus from the2016 and 2018 (excluding the FR-EN and ES-EN pairs),
PaCO for the EU-EN pair, and OpenData for the EU-ES pair
represent the out-of-domain pairs.
sources and preprocessed by removing the overlapping segments with the parallel data. Note; EN is the only available
in-domain monolingual data, whereas the rest is collectedfrom the out-of-domain sources based on availability. Table 1 summarizes the source and data size of each language
direction.
For evaluating the target-task (EU-EN) a development setof 1140 segments and for reporting the official submission
results, the 2018 test set constituting 1051 source side segments are used from the TED talks in-domain data.
5.2. Preprocessing
We first tokenize the raw data and remove sentences longer
than 70 tokens. As in [4], we prepend a “language-flag”
on the source side of the corpus for all multilingual models. The internal sub-word segmentation [26] provided by
the Tensor2Tensor library4 is used before each training andinference. Note that prepending the “language-flag” on thesource side of the corpus is specific to the multilingual models. Following the recommendation in [27], the number ofsegmentation rules is set to 16K for the in-domain data and
32K for the out-of-domain data.
5.3. Experimental Settings

5. Experiments
5.1. Dataset
The experimental setting covers the Basque (EU), English(EN), French (FR), and Spanish (ES) languages. The ELR
language pair (EU-EN) and the related language pairs (FREU/EN, and ES-EU/EN) are categorized into the in-domain
and out-of-domain settings. The in-domain data are extracted from the publicly released shared task dataset, WIT3
TED corpus [22]. Where as the the out-of-domain dataset
is collected from the WMT evaluation campaign PaCo corpus [23, 24], Opus corpus [25], and the Open Data Euskadi
Repository (OpenData)3 . Monolingual datasets for the EuEN pair are extracted from the TED, Opus18, and OpenData
3 http://hltshare.fbk.eu/IWSLT2018/OpendataBasqueSpanish.tgz

All systems are trained using the Transformer [6] model implementation in the Tensor2Tensor library. For all trainings, we use the Adam optimizer [28], with an initial
learning rate constant of 2 and a dropout [29, 30] of 0.2.
The learning rate is increased linearly in the early stages(warmup training steps=16, 000) and afterward it is decreased with an inverse square root of the training step.
Considering the two training scenario (i.e., iELR andoELR), we utilize two model configurations; i) for the indomain data a 512 embedding and hidden units dimension,and 6 layers of self-attention encoder-decoder network, andii) for an out-of-domain scenario the dimension is set to1024. The training batch size is of 4, 096 sub-word tokens.
4 https://github.com/tensorflow/tensor2tensor/tree/v1.6.2/tensor2tensor

162
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Eu-En

Round

NMT

M-NMT

iELR

oELR

I
II
III

3.10
-

13.37
-

12.96
15.65
15.15

22.48
22.72
23.14

Basque-English

Table 2: BLEU results on the dev2018 using the EU-EN single language pair NMT and the multilingual M-NMT baselinemodels, as compared to the in-domain iELR and the out-ofdomain oELR adapted multilingual models from three training rounds. The bold highlight shows the best performing
training rounds.
At inference time, we employ a beam size of 4 and a batch
size of 32.
Following [6], iELR experiments are run upto 100k
training steps, whereas oELR experiments are run upto 400K
steps, i.e., all models are observed to converge within thesesteps. The consecutive adaptation converged in a variable
training steps, however, to make sure a convergence point isreached, all restarted experiments are run for additional 50K
steps. Then, the best performing checkpoint on the dev set
is used in the next training stage. All models are trained ona Tesla V100-pcie-16gb with a single GPU for iELR
and 4 GPU’s for oELR.
5.4. Baseline Models
Baseline: models are trained as a term of comparison in two
settings, i) using only the available in-domain EU-EN data,
refereed to as NMT, and ii) by adding the related language(EU-FR/ES and FR/ES-EN) in-domain data on the EU-EN
target-task. The latter forms a multilingual (M-NMT) baselinemodel.
The following section, discusses the results andthe comparison between the baselines and the adapted modeltypes.

6. Results and Discussion
The baseline models (NMT and M-NMT) compared to against
the adapted multilingual (iELR and oELR) models are reported in Table 2. The single language pair model trainedwith the in-domain (≈5.6K) training data showed a performance of 3.10 BLEU. As we expected, the poor performanceis directly related to the small amount of training data. In
case of the M-NMT, we observed an improvement of +10.27
over the NMT with a performance of 13.37 BLEU. As discussed in Section 1, the transfer-learning across languages,that arise from the additional EU-ES/FR and FR/ES-EN indomain language pairs highly contributed for the observed
improvement. Moreover, the experiments with our suggestion have been run for two consecutive rounds.
In-Domain Setting. In the first adaptation stage, the iELR
model showed no significant difference with the baseline
M-NMT. However, the adaptation stage helps to narrow thetranslation direction to the target task and avoid possible am-

NMT

M-NMT

iELR

oELR

-

-

15.89

23.99

Table 3: Official BLEU results of tst2018 evaluated using thein-domain iELR and the out-of-domain oELR best performing adapted multilingual models.

biguities for the inference stage. In the second round theiELR model showed a +2.28 BLEU improvement over the
M-NMT (13.37 BLEU) baseline. The improvement is expected for the reason that the model is trained with the additional pseudo-parallel corpus from the back-translation stepof the EN in-domain monolingual data to the EU target. In
the consecutive round, however, the model performance degrades after the back-translation stage. This is likely causedby poorly generated source side synthetic EU from the EN
monolingual data. Thus, for the final evaluation we take thebest performing model from the second training round.
Out-of-Domain Setting: oELR models are trained in a similar training strategy with iELR, except the availability of additional parallel and monolingual (both for EU and EN, see
Table 1) data. The relatively higher amount of training data,
contributed for the larger gain of the oELR model over the indomain training condition. Compared to the baseline models(NMT and M-NMT), oELR showed the highest performancewith 23.14 BLEU score at the third training round. Unlike
the performance degradation observed in the iELR setting,the availability of monolingual data both for EU and EN
benefits each training-inference stage. However, with only a0.66 BLEU gain over the initial model after three rounds, weobserved that the domain mismatch between parallel (EUEN) and the monolingual data disadvantages the expected
improvement using the training-inference approach.
In case of, the official evaluation campaign, this work
focused on a primary submission using the oELR model anda contrastive submission using the in-domain iELR model.
Table 3, shows the performance of the two models on test2018.
An interesting aspect from the multilingual adaptationand the iterative training-inference stages is that improvements are observed within 6k-20k steps. Meaning, the continued training approach from the latest adapted model showsa faster convergence than training a model from scratch.
Overall, our approach aimed at training a baseline multilingual model for a progressive adaptation to a target-task (i.e.,
EU-EN), and applying an iterative training-inference scheme
using monolingual corpora showed to improve over the baseline model. Our results suggest that the progressive adaptation is critical when the target-task language pair has new additional data at each stage. The experimental findings havebrought our attention for a further study on how to adapt amultilingual model and what type of monolingual data to utilize in the training-inference stages.

163
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

7. Conclusions
In this work, we showed how progressively adapting a multilingual model to an extremely low-resourced (EU-EN) language pair improves the translation performance, with anadditional training-inference stage that utilizes monolingual
data. To evaluate the approach, the experimental setting
is carried out in an in-domain (iELR) and out-of-domain(oELR) scenarios. Results show a significant improvement
over a single language pair model (NMT), as well as a 2.28
BLEU increase over the baseline M-NMT model in an indomain setting. As future work, we will focus on improving
the joint iterative training-inference and progressive adaptation stages.

8. Acknowledgements
This work has been partially supported by the EC-funded
project ModernMT (H2020 grant agreement no. 645487).
We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for thisresearch.

9. References[1] S. M. Lakew, Q. F. Lotito, N. Matteo, T. Marco, andF. Marcello, “Improving zero-shot translation of lowresource languages,” in 14th International Workshop on
Spoken Language Translation, 2017.
[2] P. Koehn and R. Knowles, “Six challenges
for neural machine translation,” arXiv preprint
arXiv:1706.03872, 2017.
[3] L. Bentivogli, A. Bisazza, M. Cettolo, and M. Federico,“Neural versus phrase-based mt quality: An in-depth
analysis on english-german and english-french,” Computer Speech & Language, vol. 49, pp. 52–70, 2018.

[8] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence tosequence learning with neural networks,” in Advancesin neural information processing systems, 2014, pp.
3104–3112.
[9] M.-T. Luong, H. Pham, and C. D. Manning, “Effective
approaches to attention-based neural machine translation,” arXiv preprint arXiv:1508.04025, 2015.
[10] P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention
with relative position representations,” arXiv preprint
arXiv:1803.02155, 2018.
[11] O. Firat, K. Cho, and Y. Bengio, “Multi-way, multilingual neural machine translation with a shared attention
mechanism,” arXiv preprint arXiv:1601.01073, 2016.
[12] D. Dong, H. Wu, W. He, D. Yu, and H. Wang, “Multitask learning for multiple language translation.” in ACL(1), 2015, pp. 1723–1732.
[13] M.-T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and
L. Kaiser, “Multi-task sequence to sequence learning,”
arXiv preprint arXiv:1511.06114, 2015.
[14] B. Zoph, D. Yuret, J. May, and K. Knight, “Transfer
learning for low-resource neural machine translation,”
arXiv preprint arXiv:1604.02201, 2016.
[15] T. Q. Nguyen and D. Chiang, “Transfer learning across
low-resource, related languages for neural machinetranslation,” arXiv preprint arXiv:1708.09803, 2017.
[16] J. Gu, H. Hassan, J. Devlin, and V. O. Li, “Universal
neural machine translation for extremely low resource
languages,” arXiv preprint arXiv:1802.05368, 2018.
[17] B. Zoph and K. Knight, “Multi-source neural translation,” arXiv preprint arXiv:1601.00710, 2016.

[4] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu,
Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, et al., “Google’s multilingual neural machinetranslation system: Enabling zero-shot translation,”
arXiv preprint arXiv:1611.04558, 2016.

[18] S. M. Lakew, Q. F. Lotito, T. Marco, N. Matteo, andF. Marcello, “Fbks multilingual neural machine translation system for iwslt 2017,” in 14th International Workshop on Spoken Language Translation (IWSLT 2017),
2017, pp. 35–41.

[5] T.-L. Ha, J. Niehues, and A. Waibel, “Toward multilingual neural machine translation with universal encoderand decoder,” arXiv preprint arXiv:1611.04798, 2016.

[19] N.-Q. Pham, M. Sperber, E. Salesky, T.-L. Ha,
J. Niehues, and A. Waibel, “Kits multilingual neuralmachine translation systems for iwslt 2017.” IWSLT,
2017.

[6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin,“Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 6000–6010.
[7] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,”
arXiv preprint arXiv:1409.0473, 2014.

[20] G. Neubig and J. Hu, “Rapid adaptation of neural machine translation to new languages,” arXiv preprint
arXiv:1808.04189, 2018.
[21] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” arXiv preprint arXiv:1511.06709, 2015.

164
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[22] M. Cettolo, C. Girardi, and M. Federico, “Wit3 : Web
inventory of transcribed and translated talks,” in Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), Trento, Italy,
May 2012, pp. 261–268.
[23] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham,
B. Haddow, M. Huck, A. J. Yepes, P. Koehn, V. Logacheva, C. Monz, et al., “Findings of the 2016 conference on machine translation,” in Proceedings of theFirst Conference on Machine Translation: Volume 2,
Shared Task Papers, vol. 2, 2016, pp. 131–198.
[24] I. San Vicente, I. Manterola, et al., “Paco2: A fully
automated tool for gathering parallel corpora from theweb.” in LREC, 2012, pp. 1–6.
[25] J. Tiedemann, “Parallel data, tools and interfaces inopus.” in Lrec, vol. 2012, 2012, pp. 2214–2218.
[26] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi,
W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey,
et al., “Google’s neural machine translation system:
Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016.
[27] M. Denkowski and G. Neubig, “Stronger baselines fortrustable results in neural machine translation,” in Proceedings of the First Workshop on Neural MachineTranslation, 2017, pp. 18–27.
[28] D. Kingma and J. Ba, “Adam: A method for stochastic
optimization,” arXiv preprint arXiv:1412.6980, 2014.
[29] N. Srivastava, G. E. Hinton, A. Krizhevsky,
I. Sutskever, and R. Salakhutdinov, “Dropout: asimple way to prevent neural networks from overfitting.” Journal of machine learning research, vol. 15,
no. 1, pp. 1929–1958, 2014.
[30] Y. Gal and Z. Ghahramani, “A theoretically grounded
application of dropout in recurrent neural networks,”
in Advances in neural information processing systems,
2016, pp. 1019–1027.

165
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Learning to Segment Inputs for NMT Favors Character-Level Processing
Julia Kreutzer∗ , Artem Sokolov⋄,∗∗

Computational Linguistics, Heidelberg University, Germany⋄
Amazon Research, Germany
kreutzer@cl.uni-heidelberg.de, artemsok@amazon.com

Abstract
Most modern neural machine translation (NMT) systems
rely on presegmented inputs. Segmentation granularity importantly determines the input and output sequence lengths,
hence the modeling depth, and source and target vocabularies, which in turn determine model size, computationalcosts of softmax normalization, and handling of out-ofvocabulary words. However, the current practice is to use
static, heuristic-based segmentations that are fixed before
NMT training. This begs the question whether the chosensegmentation is optimal for the translation task. To overcome suboptimal segmentation choices, we present an algorithm for dynamic segmentation, that is trainable end-to-end
and driven by the NMT objective. In an evaluation on fourtranslation tasks we found that, given the freedom to navigate between different segmentation levels, the model prefers
to operate on (almost) character level, providing support forpurely character-level NMT models from a novel angle.

1. Introduction
Segmentation of input sequences is an essential preprocessing step for neural machine translation (NMT) and has beenfound to have a high positive impact on translation quality inrecent WMT shared task evaluations [1, 2]. This success canbe explained statistically, since shorter segments are beneficial for reducing sparsity: They lower the type-to-token ratio,
decrease the number of out-of-vocabulary (OOV) tokens andsingletons, which improves the coverage of unseen inputs.
Two subword segmentation methods are presently thestate-of-the-art in NMT: the byte-pair encoding (BPE), thatstarts with a dictionary of single characters and iteratively
creates a new entry from the two currently most frequent entries [3, 4], and a similar wordpiece (WP) model [5].
While being empirically more successful than wordbased NMT, both BPE and WP are preprocessing heuristics,
they do not account for the translation task or the language
pairs at hand (unless applied to both sides jointly), and require additional preprocessing for languages that lack explicit word separation in writing. Being used in a pipeline
fashion, they make it impossible for an NMT system to resegment an unfavorably presplit input and require consistentWork was done while the first author was interning at Amazon, Berlin.

application of the same segmentation model during testing,which adds an integration overhead and contributes to the‘pipeline jungles’ in production environments [6].
On the other extreme from word-based NMT models lie
purely character models. Their advantages are smaller vocabularies, thus smaller embedding and output layers, allowing for more learning iterations within a training time budget
to improve generalization [7], and no preprocessing requirements. At the same time, longer input sequences aggravate
known optimization problems with very large depths of timeunrolled RNNs [8] and may require additional memory for
tracking gradients along the unrolling steps.
In this work1 , we pose the following question: what
would the input segmentations look like if the NMT
model could decide on them dynamically? Instead ofheuristically committing to a fixed (sub)word- or charactersegmentation level prior to NMT training, this would allow
segmentation for each input to be driven by the training objective and avoid solving the trade-offs of different levels
by trial and error. To answer this question, we endow an
NMT model with the capacity of adaptive segmentation byreplacing the conventional lookup embedding layer with a‘smart embedding’ layer that sequentially reads input characters and dynamically decides to group a block of theminto an output embedding vector, feeding it to the upstream
NMT encoder before continuing with the next block (withan optional reverse process on the target side). To signalthat a block of characters, encoded as an embedding vector, is ready to be fed upstream, we use accumulated values of a scalar halting unit [9], which learns when to output
this block’s embedding. It simultaneously affects weighting probabilities of intermediate output vectors that compose the output embedding. Thanks to this weighting, ourmodel is fully differentiable and can be trained end-to-end.
Similarly to BPE, it has a hyper-parameter that influences
segmentation granularity, but in contrast to BPE this hyperparameter does not affect the model size. While we evaluate our on-the-fly segmentation algorithm on RNN-based
NMT systems, it is transferable to other NMT architectures(CNN [10] or Transformer [11]), since it only replaces the
input embedding layer. Empirically, we find a strong preference of such NMT models to operate on segments that are
1 Extended report: https://arxiv.org/abs/1810.01480.

166
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

only one to a few characters long. This turns out to be a reasonable choice, as in our experiments character-level NMT
systems of smaller or comparable size were able to outperform word- and subword-based systems, which corroborates
results of [12, 13]. Given this finding and the unique advantages of character-level processing (no pipelining, no tokenization, no additional hyperparameters, tiny vocabulary
and memory, and robustness to spelling errors [14]), we
hope that character-level NMT, and in general character-level
sequence-to-sequence learning, will receive more attention
from researchers.
Note that, although our character-based models outperform (sub)word-based ones with similar architectures onsome datasets, we are not seeking to establish a new state-ofthe-art in NMT with our model. Our goal is to isolate the effects of segmentation on quality by introducing a flexibilityenhancing research tool. Therefore, in the comparisons between (sub)word- and character-based models we purposely
avoided introducing changes to our baseline RNN NMT architecture beyond upgrading the embedding layer.

2. Related Work
To tackle the OOV problem in word-level models, [15] proposed a hybrid model that composes unknown words fromcharacters both on encoder and decoder side. While their approach relies on given word boundaries, they report a purely
character-based baseline performing as well as a word-basedmodel with unknown word replacement, but taking 3 months
to train, which seems to have cooled off the NMT community in investigating fully character-based models as an alternative to (sub)word-based ones. Unlike [15], we found that
despite the training speed being slower than for (sub)word
vocabularies, it is possible to train reasonable character-level
models within a few weeks. To combine the best of bothworlds, [16] proposed hierarchical en-/decoders that receive
inputs on both word- and character-level. The encoder learnsa weighted recurrent representation of each word’s characters and the decoder receives the previous target word andpredicts characters until a delimiter is produced. Similar to
our work, they find improvements over BPE models. The
idea to learn composite representations of blocks of characters is similar to ours, but their approach requires given wordboundaries, which our model learns on-the-fly. [12] combined a standard subword-level encoder with a two-layer, hierarchical character-level decoder. The decoder has gating
units that regulate the influence of the lower-level layer tothe higher-level one, hence fulfilling a similar purpose as ourhalting unit. This model outperforms a subword-level NMT
system, and achieves state-of-the-art on a subset of WMTevaluation tasks. While not requiring explicit segmentationon the target side, the model still relies on given source segmentations. Finally, [14] proposed a fully character-level
NMT model. They mainly address training speed, which [15]identified as a problem, and introduce a low-level convolutional layer over character embeddings to extract informationfrom variable-length character n-grams for higher-level processing with standard RNN layers. Thus, overlapping segments are modelled with a length depending on the filters.
Perhaps closest to our work is [13], where each layer of ahierarchical RNN encoder is updated at different rates, withthe first layer modelling character-level structures, the following modelling sub(word)-level structures. They introduce
a binary boundary detector, similar to our halting unit, thattriggers feeding of a representation to the next level, so that
latent hierarchical structures without explicit boundary information are learnt. Unlike our fully-differentiable model, suchdiscrete decisions of the boundary detector prohibit end-toend differentiability, forcing a recourse to the biased straightthrough estimator [17]. On the other hand, while our model
relies on a to-be-tuned computation time penalty, [13] do not
impose constraints on the number of boundaries.

3. Jointly Learning to Segment and Translate
Instead of committing to a single segmentation before NMT
model training, we propose to learn the segmentationgoverning parameters along with the usual network parameters in a end-to-end differentiable manner. With this approach, we get rid of pipelining and pre-/postprocessing,and can adaptively segment arbitrary inputs we encounter
during training or testing. Our segmentations are contextdependent, i.e. the same substring can be segmented into different parts in different contexts. Being able to smoothlyinterpolate between word-based and character-based models
we allow the model to find a sweet spot in between.
We extend the Adaptive Computation Time (ACT)paradigm [9], where a general RNN model is augmented
with a scalar halting unit that decides how many recurrent
computations are spent on each input. For segmentation, we
use the halting unit to decide how many inputs (characters)a segment consists of. The output of the ACT module canthus be thought of as an ‘embedding’ vector for a segmentthat replaces the classic lookup embedding for (sub)words instandard NMT models. While our model can in principle use
larger units as elementary inputs, we will focus on character inputs to be able to model the composition of arbitrary
segments. That means that we only add a small amount ofparameters to a basic character-based model, but explicitly
model higher-level merges of characters into subwords.
3.1. ACT for Dynamic Depth
Here we summarize the ACT model [9]. It is applicable toany recurrent architecture that transforms an input sequencex = (x1 , . . . , xT ) into outputs o = (ō1 , . . . , ōT ) via computing a sequence of states s = (s1 , . . . , sT ) through a statetransition function S on an embedded input Ext and a linear
output projection defined by matrix Wo and bias bo :st = S(st−1 , Ext ),o t = W o s t + bo

(1)

Instead of stacking multiple RNN layers in S to achieve
increased complexity of an RNN network, the ACT model

167
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

ō1
o1

o2

s1

s2

x1

x2

ō2

s̄1

o3

o4

o5

s3

s4

s5

x3

x4

x5

Figure 1: Diagram of the ACT-ENC encoder. Note the differences to the original ACT model: An input is here read on
every internal recurrent iteration (gray arrows) and the halting unit (red stop sign) is repurposed to trigger feeding ofan encoded embedding vector of a block of characters to the
upstream NMT layers.
dynamically decides on the number of necessary recurrent
steps (layers) for every input xt . This saves computation oneasy inputs, while still being able to use all of the processing
power on hard inputs before emitting outputs. Concretely, an
ACT cell performs an arbitrary number of internal recurrent
applications of S for each input xt :(S(s̄t−1 , Ext ), if n = 1
n
st =
(2)
S(stn−1 , Ext ), otherwise
The total number of internal steps is N (t) = min{n′ :P n′nnn=1 ht ≥ 1 − ǫ}, where ǫ ≪ 1 and ht is the scalar output
of sigmoid halting unit,
hnt = σ(Wh snt + bh ).

(3)

Once halted, the final output ōt and state s̄t (which is fed to
the next ACT step in (2)) are computed as weighted meansof intermediate outputs and states:
N (t)

N (t)

s̄t =

X

pnt snt ,

ōt =

n=1
where probabilities pnt are defined asX

pnt ont

(4)

n=1

(

R(t), if n = N (t)(5)
hnt , otherwise
PN (t)−1 ′and remainders R(t) = 1 − n′ =1 hnt . Finally, to preventthe network from pondering on an input for too long, theremainder R(t) is added as a penalty to the RNN training
loss (usually cross-entropy (XENT)) with a weight τ :pnt =LACT = LXENT + τ R(t).

(6)

Thanks to (4), the model is deterministic and differentiable.
3.2. ACT for Dynamic Segmentation
We now describe how to use the ACT paradigm to enhance
an encoder for dynamic segmentation on the source side
(ACT-ENC). We reuse the idea of halting units, mean field
updates and τ -penalized training objective, but instead oflearning how much computation is needed for each atomic
input, we learn how much computation to allow for an ag-

gregation of atomic inputs, i.e. one segment.
The input to an ACT-ENC cell is a sequence of one-hotencoded characters x = (x1 , . . . , xTx ). The ACT-ENC, depicted in Figure 1, receives one input xt at a time and decides whether to halt or not. In the case of no halting, the cellproceeds reading more inputs; if it halts, it produces an output ‘embedding’ ō of a block of characters read so far, andthe cell resets for reading the next block. The sequence ofthe output embeddings o = (ō1 , . . . , ōTo ) is then fed to upstream standard (possibly bidirectional) NMT encoder layers, replacing the usual, one-hot encoded, (sub)word lookup
embeddings. The length of o is variable: The more frequently ACT-ENC halts, the more embeddings are generated. In extreme cases, it can generate one embedding per
input (To = Tx ) or just one embedding for the full sequenceof inputs (To = 1).
Algorithm 1 ACT-ENC
Input: Weights Wo , bo , Wh , bh , transition function S, embeddings Esrc , inputs x = (x1 , . . . , xTx ), threshold ǫ.
Output: Outputs o = (ō1 , . . . , ōTo ), remainder R.
1: o = [ ]⊲ empty sequence
2: R = 0, H = 0⊲ init remainder and halting sum
3: s̄ = 0, ō = 0, s0 = 0⊲ init mean state and output
4: for t = 1 . . . Tx do⊲ loop over inputs
5:st = S(st−1 , Esrc xt )⊲ new state6:o t = W o s t + bo
⊲ new output
7:
ht = σ(Wh st + bh )⊲ halting score8:
f = [[H + ht ≥ 1 − ǫ]]⊲ halting flag
9:pt = (1 − f ) ht + f (1 − H) ⊲ halting probability
10:
H = H + ht
⊲ update halting sum
11:s̄ = s̄ + pt st , ō = ō + pt ot ⊲ mean state and output
12:
R = R + (1 − f ) ht
⊲ increment remainder
13:
if f then14:o = o⌢ [ō]⊲ append output
15:
st = s̄⊲ overwrite for next step16:s̄ = 0, ō = 0, H = 0
17: R = (1 − R)/t
⊲ normalize remainder
In more detail, ACT-ENC implements the pseudocode
given in Algorithm 1. Let S(st−1 , it ) be any recursive computation function (in this work we use GRUs) of an RNN thatreceives a hidden state st−1 and an input vector it at time
step t and computes the new hidden state st . In line 5 thisfunction is computed on the regular previous state or, if there
was a halt in the previous step (line 13), on the mean statevector s̄ that summarizes the states of the previous segment(line 15, cf. (4), 1st eq.). Per-step outputs ot are computedfrom the hidden states st with a feed-forward layer (line 6,
cf. (1), 2nd eq.). A sigmoid halting unit computes a haltingscore in each step (line 7, cf. (3)). The halting probability
for step t is either the halting score ht or the current valueof remainder 1 − H to ensure that all halting probabilities
within one segment form a distribution (line 9, cf. (5)). ǫ
is set to a small number to allow halting after a single step.

168
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Whenever the model decides to halt, an output embedding ō
is computed as a weighted mean of the intermediate outputs of the current segment (line 14, cf. (4), 2nd eq.). The
weighted mean on the one hand serves the purpose of circumventing stochastic sampling, on the other hand can beinterpreted as a type of intra-attention summarizing the intermediate states and outputs of the segment. The haltingscores from each step are accumulated (line 12) to penalize computation time as in (6). The hyperparameter τ here
controls the segment length: The higher its value, the more
preference will be given to smaller remainders, i.e. shorter
segments. We introduce an additional normalization by input
length (line 17), such that longer sequences will be allowed
more segments than shorter sequences. This implementation
exploits the fact that ACT-ENC outputs are weighted meansover time steps and updates them incrementally. The algorithm allows efficient minibatch processing by maintaining ahalting counter that indicates which embedding each current
intermediate output in the batch contributes to. Incremental
updates of embeddings and states are achieved with masks
depending on the halting position.

4. Experiments
We reimplemented the Groundhog RNN encoder-decoder
model with attention [18] in MxNet Gluon to allow for
dynamic computation graphs. We report results on four
language directions and domains, for word-, subword-,character-level and ACT-ENC segmentation: German-toEnglish TED talks, Chinese-to-English web pages, Japaneseto-English scientific abstracts and French-to-English news.
Table 1 gives a data overview.
The IWSLT data is split and processed as in [19]; since
it comes pretokenized and lowercased, models are evaluatedwith tokenized, lowercased BLEU (using sacrebleu [20])and chrF scores on character bigrams [21]. For WMT, weused the 2014 dataset prepared for [18], additionally filtering
the training data to include only sequences of a lengths 1
to 60, and models are evaluated with cased BLEU and chrF
(sacrebleu, with the “13a” tokenizer).
The CASIA and ASPEC data are, respectively, from the2015 China Workshop on MT (CWMT), used without preprocessing and with sampled dev/test sets, and from the
WAT 2017 SmallNMT shared task, pretokenized with WP.
Both datasets have BPE and WP vocabularies of around 16k
for each side, and we report cased BLEU and chrF on them.
Hyperparameters.
All models are trained with
Adam [22] and a learning rate of 0.0003, halved wheneverthe validation score (tokenized BLEU) has not increased for
Data

Domain

Lang

Train

Dev

Test

IWSLT
CASIA
ASPEC
WMT

TED talkswebsci. abstracts
news

de-en
zh-en
ja-en
fr-en

153,352
1,045,000
2,000,000
12,075,604

6,970
2,500
1,790
6,003

6,750
2,500
1,812
3,003

Table 1: Data statistics (number of parallel sentences).

Data

Model

BLEU

chrF

Param

SegLen

IWSLT
de-en

Word
BPE
Char
ACT-ENC

22.11
25.38
22.63
22.67

0.44
0.49
0.46
0.46

80.5M
46.5M
13.4M
13.5M

4.66
4.09
1.00
1.88

TrainTime
23h
20h
1d22h
9d21h

CASIA
zh-en

BPE
Char
ACT-ENC

10.59
12.60
9.87

0.37
0.40
0.36

49.9M
21.0M
21.3M

1.72
1.00
1.006

18h
10d6h
3d13h

ASPEC
ja-en

WP
Char
ACT-ENC

21.05
22.75
15.82

0.53
0.55
0.46

50.0M
15.6M
15.6M

2.07
1.00
1.00074d4h
24d15h
15d4h

WMT
fr-en

Word
BPE
Char
ACT-ENC

20.32
27.02
24.25
13.74

0.49
0.55
0.53
0.42

80.5M
86.0M
14.1M
14.2M

5.19
4.05
1.00
1.82

4d9h
3d23h
9d
13d8h

Table 2: Results on test sets for 1-layer models, and number of parameters and average source segment lengths on dev
sets. Time to reach stopping criterion is in (d)ays and (h)ours.
3 validations. Training stopped when the learning rate has
been decreased 10 times in a row. All models use recurrent
cells of size 1,000 for the decoder, with a bidirectional encoder of size 500 for each direction, input and output embedding of size 620, and the attention MLP of size 1,000,
all following [18]. When multiple encoders layers are used,
they are all bidirectional [23] with attention on the uppermost layer. The ACT layer for ACT-ENC models has size 50
for IWSLT, CASIA and ASPEC, and 25 for WMT (pickedfrom {25, 50, 75, 100, 150}). The word-based models on
IWSLT and WMT have a vocabulary of 30k for each side,
the BPE models have separate 15k vocabularies for IWSLT
and a joint 32k vocabulary for WMT. For IWSLT, CASIA
and ASPEC all characters from the training data were included in the vocabularies, resulting in vocabulary sizes of
117 (de) and 97 (en), 7,284 (zh) and 166 (en), and 3,212 (ja)and 233 (en), respectively. For WMT the vocabularies included the 400 most frequent characters on each side. Wordand BPE-based models are trained with minibatches of size
80, character-based models with 40. The maximum sequencelength during training is 60 for word- and BPE-based models, 200 for character-based models and 150 for ACT-ENC,
to fit into available memory. τ = 1.0 delivered the highest
BLEU score for IWSLT and CASIA, τ = 0.8 for WMT andτ = 0.7 for ASPEC. Following [9], we fixed ǫ = 0.01 in all
the experiments. During inference, we use beam-search witha beam size of 5 and length-normalization.
Evaluation Results. Table 2 lists the results for the mostcomparable, 1-layer, configuration. BPE/WP models expectedly outperform word-based models, however word-basedmodels are also outperformed by character-based models.
The picture is similar w.r.t. the chrF with even smaller relative differences. The ACT-ENC model with one unidirectional ACT layer manages to match the 1-layer bidirectional
character-based model on IWSLT. But it does not reach theresults of other models on CASIA and ASPEC, which canbe explained by increased complexity of doing simultaneous segmentation during training on sentences longer than
the average sentence length in IWSLT. However, the main

169
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Data

Model

BLEU

chrF

Param

SegLen

IWSLT
de-en

Word, 4L
BPE, 1L
Char, 5L
ACT-ENC, 3L

24.54
25.38
28.19
25.10

0.45
0.49
0.51
0.49

97.0M
46.5M
26.9M
25.6M

4.66
4.09
1.00
1.31

1d8h
20h
3d10h
9d7h

CASIA
zh-en

BPE, 3L
Char, 3L
ACT-ENC, 2L

11.01
13.43
10.35

0.38
0.42
0.37

58.9M
30.0M
21.3M

1.72
1.00
1.00

24h
5d6h
10d

ASPEC
ja-en

WP, 3L
Char, 1L
ACT-ENC, 1L

22.02
22.75
15.82

0.55
0.55
0.46

61.4M
15.6M
15.6M

2.07
1.00
1.00074d2h
24d15h
15d4h

WMT
fr-en

Word, 2L
BPE, 3L
Char, 6L
ACT-ENC, 2L

21.04
27.93
27.23
14.01

0.48
0.56
0.55
0.43

94.0M
98.0M
27.6M
21.7M

5.19
4.05
1.00
1.0001

4d16h
5d3h
18d13h9d10houtperformed by character-based models with multiple encoder layers, achieving gains of 2.8 BLEU points on IWSLT,
0.7 on ASPEC, and losing only 0.7 on WMT (with a minor
decrease in chrF), despite having at least 3.5 times fewer parameters. Such ranking of character- and BPE-based models
on WMT might be explained by much longer sentences in
the corpus, compared other corpora, since the ability of character and ACT-based models to cover unseen input is limitedby the maximum training sequence length limit (here 200
characters), which on WMT data crops 30.5% of sentences.

TrainTime

Table 3: Results on respective test sets after tuning the number of encoder (L)ayers (from 1 to 6) on the dev set.
finding here is that ACT-ENC recovers an almost characterlevel segmentation (“SegLen” column in Table 2). On the
IWSLT dev set, the average segment length is only 1.88, witha maximum of 5 characters per segment. For CASIA andASPEC domains, and with the larger datasets than IWSLT,
the ACT-ENC segmentations becomes more fine-grained:
The average segment length is, respectively, just 1.006 and1.0007 on the dev set (max. 2 chars per segment). Given
that the character model outperforms the BPE/WP models,
it is not surprising that ACT-ENC converged to the character segmentation. We hypothesize that ACT-ENC could not
improve over the 1-layer bidirectional character model because of complexity of identifying segments in Chinese andJapanese, unidirectionality of its initial layer, and increased
hardness of optimization of character-based models with extra non-linearities [24], that causes earlier convergence topoorer minima in many runs. Similarly for WMT, failingto match the performance of the character model could becaused by harder optimization task on particularly long sentences in the WMT data, and unidirectionality of ACT-ENC.
The ACT-ENC’s segment length is 1.82 (max. 6 chars), again
close on average to a purely character segmentation.
Inspired by the ACT-ENC’s recovery of almost character segmentation and by the competitive performance of purecharacter-based models, we decided to verify if the advantage of character-level processing carries over to multiple
layers. Since the character models are much smaller thantheir word-/BPE-based counterparts, one should allow multiple layers (consuming the same or less memory) to make up
for the difference in number of parameters for fairer comparison. This also aimed to verify whether an increased numberof non-linearities (one of ACT’s benefits [25]) plays a role.
Table 3 shows the test results after tuning the number ofbidirectional encoder layers, from 1 to 6, on dev sets. First,
we observe the modest parameter number of character models even with multiple layers, that allows them to take advantage of deeper cascades of non-linearities while staying
well below the memory budget of (sub)word-based 1-layer
models. Second, comparing to Table 2, we again confirm
the negative correlation of quality and segment length for
ACT-ENC. Finally, we discover that BPE/WP models are

Analysis of Segmentation and Outputs. Randomly selected translation examples from the IWSLT dev set and theirsegmented sources are given in Table 4. In general, when
encountering rare inputs, word-based models fail by producing the unknown word token (<unk>), and the BPE-model
is able to translate only a more common part of German
compounds (e.g. ‘tiere’ → ‘animals’). The character-basedmodels invent words (‘altients’, ‘jes lag’) that are similar tostrings that they saw during training and the source. In a
few cases they fallback to a language-modeling regime having attended to the first characters of a corresponding source
word: e.g., instead of translating ‘reisen’ to ‘journeys’, the
ACT-ENC model translates it to ‘rows’ (confusing ‘reisen’
to a similarly spelled German ‘reihen’), or ‘layering’ insteadof ‘shift work’ (confusing ‘schichten’ to the prefix-sharing‘schichtarbeit’). This is confirmed when inspecting attention
scores: The model frequently attends to the correct source
word, but mainly to the first characters only. Note that
ACT-ENC segmentations are context-dependent, e.g. occurrences of ’tiere’ are segmented differently.
Table 5 lists the most frequent segments produced by
1-layer ACT-ENC. For IWSLT, we observe that many segments make sense statistically (frequent or rare patterns) andlinguistically to some extent: Many of the frequent segments include whitespace (itself a frequent symbol); 2-gramsegments amongst others include frequent word suffixes(‘en’, ‘in’, ‘er’), but also frequent diphthongs (‘ei’ and ‘ie’);
3-grams start with rare characters like ‘x’ and ‘y’ or single
dashes; 4-grams combine single characters with whitespaces
and double dashes; 5-grams cover numbers, in particular,
years. Importantly, though, since the best test BLEU scoresfor IWSLT were obtained by a multi-layer character-basedmodel, the ACT-ENC model has done a reasonable job in improving over the already well-performant strategy, one character per segment, despite having only a single NMT layer.
For CASIA and ASPEC, ACT-ENC converged to a segmentation even closer to pure characters: for CASIA, the mostfrequent 2-grams are punctuation combined with frequent
pronoun 他 or preposition 的, or with the hieroglyph 明 froma common phrase ‘[smth.] shows, [that]’ (all 4-10k in train),and parts of rare English words; for ASPEC, it is mostly theHiragana letter き that starts the segments. While this letter also occurs as singleton (183× in the dev set, vs. 52× as
part of a learned segment), and is frequent in the training set(239k), it is not the most frequent letter. For WMT, charac-

170
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Ref

in social groups of animals , the juveniles always look different than the adults .

Word

in groups of social animals , the children are always different from the other than the <unk>.

BPE

in gruppen sozialer tiere sehen die jung@@ tiere immer anders aus als die alt@@ tiere .
in groups , in groups , the juveniles are seeing the same animals as well as the animals .

ACT-ENC

in| g|ru|pp|en| s|oz|ia|le|r |ti|er|e |se|he|n |d|ie| j|un|gt|ie|re| i|m|m|er| a|nd|er|s |au|s |al|s |d|ie| a|lt|ti|er|e |.|in groups , the juvenile seems to see the different approach than the algaes .

Charin groups of social animals , the juveniles are still in the alite of the altients .

Ref

we &apos;re living in a culture of jet lag , global travel , 24-hour business , shift work .

Word

we live in a civilization with <unk> , global travel , <unk> and <unk> .

BPE

wir leben in einer zivilisation mit jet@@ -@@ lag , weltweiten reisen , non@@ sto@@ p-@@ business und sch@@ icht@@ arbeit .
we live in a civilization with a single , a variety of global travel , presidential labor and checking .

ACT-ENC

w|ir| l|eb|en| i|n |ei|ne|r |z|iv|il|is|at|io|n |m|it| j|et|-la|g |,| w|el|tw|ei|te|n |re|is|en| ,| n|on|st|op|-bu|si|ne|ss| u|nd| s|ch|ic|ht|ar|be|it| .|we live in a civilization with jes lag , worldwide rows , nonstop business and failing .

Char

we live in a civilization with jet walk , global journeys , nonstop-business and layering

Table 4: Greedy translations from the IWSLT dev set. Explicit segmentations are given for the ACT-ENC and BPE models.
Data

Len

Segments

IWSLT

2
3
4
5

en; n ; er; d; ie; e ; ei; in; s; w . . .
yst; - d; xtr; - u; 100; xpe; - w; xis; - e; -ge . . .
– d; – w; – s; – i; – e; – u; – g; – m; – a; – k . . .
1965 ; 969 ,; 1987 ; 1938 ; 1621 ; 1994 ; 1985 . . .

CASIA

2

”。; ”，; er; ”他; --; ”的; le; 明，; li; ut;

ASPEC

2

きる; きた; きな; きに; りん; きは; き，; きて

WMT

2
3
4

e ; s ; d; t ; l; es; on; a; de; en . . .
übe; Rüc; rüb; öve; ürs; Köp; üslümov; ölln; rüng; Jürg; ülle; Müsl Müni; üric; üdig; . . .

...
...

Table 5: Most frequent ACT-ENC segments.
ter 2-grams are all very frequent in the training data (8-11M
occurrences) while longer segments are very rare (max. 1koccurrences). Longer segments all include umlauts (ü, ö),which are atypical for French and should be treated as one
unit semantically since they are loan words or proper names.
Gating Behavior of Char-GRUs. To investigate thereasons for success of the deep character-based encoders
and their better or on-par performance with the segmenting ACT-ENC model, we analyzed average activations of
GRU gates. A GRU cell computes the next state as: st =z ⊙ tanh(xt Wh + (ht−1 ⊙ r)Wg ) + (1 − z) ⊙ st−1 , where z
is the update gate and r the reset gate, both being outputs ofsigmoid layers receiving xt and ht−1 [26]. Taking a closer
look at the average values of these gates, we find patterns ofsegmentation as depicted in Figure 2 for a 5-layer charactermodel. Most of the time, a whitespace character triggers avisible change of gate behavior: Forward reset gates close(reset) one character after a whitespace and backward resetgates close at whitespaces and then both open at the subsequent character. The update gates show similar regularities, but here the average gate values are less extreme. For
longer words all gate activations progressively decay with thelength. In addition, the block-wise processing of the compound ‘schreibtisch’ (German: ‘writing table’) that was correctly split into ‘schreib’ and ‘tisch’, points to decompounding abilities that pure character-level models possesses beyond simple whitespace tokenization.
Overall, this illustrates that the recurrent gates equip pure

Figure 2: Mean activations for reset and update forward(FW) and backward (BW) GRU gates for an IWSLT sentence
as produced by the 5-layer char model. Layers are stacked
from bottom to top. Blue: values ≃ 0, yellow: values ≃ 1.
character models with the capacity to implicitly model input
segmentations, which would explain why ACT-ENC could
not find a radically different or advantageous segmentation.

5. Summary & Conclusion
We proposed an approach to learning (dynamic and adaptive)
input segmentation for NMT based on the Adaptive Computation Time paradigm [9]. Experiments on four translations
tasks showed that our model prefers to operate closely to thecharacter level. This is echoed by the quantitative success ofpure character-level models (without dynamic segmentation)
and a qualitative analysis of gating mechanisms, suggestingthat our adaptive model rediscovers the segmenting capacity
already present in gated recurrent, pure character-based models. Given this and the absence of many development hurdles
with character-based models, their lower memory consumption and higher robustness, the presented dynamic segmentation capacity, being primarily a diagnostic research tool, does
not seem to be necessary to be modelled explicitly. We hopethese insights can serve as justification for intensification ofresearch in pure character-level NMT models.

171
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

6. References[1] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham,
B. Haddow, et al., “Findings of the 2016 conferenceon machine translation,” in WMT, 2016.
[2] ——, “Findings of the 2017 conference on machinetranslation,” in WMT, 2017.
[3] P. Gage, “A new algorithm for data compression,” TheC Users Journal, vol. 12, no. 2, pp. 23–38, 1994.
[4] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in
ACL, 2016.
[5] M. Schuster and K. Nakajima, “Japanese and Korean
voice search,” in ICASSP, 2012.

[15] M.-T. Luong and C. D. Manning, “Achieving open vocabulary neural machine translation with hybrid wordcharacter models,” in ACL, 2016.
[16] S. Zhao and Z. Zhang, “Deep character-level neural machine translation by learning morphology,” inarXiv:1608.04738, 2016.
[17] Y. Bengio, N. Léonard, and A. C. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” in arXiv:1308.3432,
2013.
[18] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,” in
ICLR, 2015.

[6] D. Sculley, G. Holt, D. Golovin, E. Davydov,
T. Phillips, et al., “Hidden technical debt in machinelearning systems,” in NIPS, 2015.

[19] D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe,et al., “An actor-critic algorithm for sequence prediction,” in ICLR, 2017.

[7] E. Hoffer, I. Hubara, and D. Soudry, “Train longer, generalize better: closing the generalization gap in large
batch training of neural networks,” in NIPS, 2017.

[20] M. Post, “A call for clarity in reporting BLEU scores,”in arXiv:1804.08771, 2018.

[8] S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber, “Gradient flow in recurrent nets: the difficultyof learning long-term dependencies,” in A field guideto dynamical recurrent neural networks. IEEE Press,
2001.
[9] A. Graves, “Adaptive computation time for recurrent
neural networks,” in arXiv:1603.08983, 2016.

[21] M. Popovic, “chrF: character n-gram F-score for automatic MT evaluation,” in WMT, 2015.
[22] D. Kingma and J. Ba, “Adam: A method for stochastic
optimization,” in ICLR, 2015.

[10] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N.
Dauphin, “Convolutional sequence to sequence learning,” in ICML, 2017.

[23] M. X. Chen, O. Firat, A. Bapna, M. Johnson,
W. Macherey, et al., “The best of both worlds: Combining recent advances in neural machine translation,”in arXiv:1804.09849, 2018.

[11] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, et al., “Attention is all you need,” in NIPS,
2017.

[24] W. Ling, I. Trancoso, C. Dyer, and A. W. Black,“Character-based neural machine translation,” inarXiv:1511.04586, 2015.

[12] J. Chung, K. Cho, and Y. Bengio, “A character-level decoder without explicit segmentation for neural machinetranslation,” in ACL, 2016.
[13] J. Chung, S. Ahn, and Y. Bengio, “Hierarchical multiscale recurrent neural networks,” ICLR, 2017.
[14] J. Lee, K. Cho, and T. Hofmann, “Fully character-levelneural machine translation without explicit segmentation,” TACL, vol. 5, pp. 365–378, 2017.

[25] D. Fojo, V. Campos, and X. Giró-i Nieto, “Comparing
fixed and adaptive computation time for recurrent neural networks,” in Workshop Track of ICLR, 2018.
[26] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, et al., “Learning phrase representations using RNN encoder–decoder for Statistical Machine Translation,” in EMNLP, 2014.

172
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Data Selection with Feature Decay Algorithms Using an Approximated Target Side
Alberto Poncelas, Gideon Maillette de Buy Wenniger, Andy Way
ADAPT Centre, School of Computing,Dublin City University, Dublin, Ireland{firstname.lastname}@adaptcentre.ie
Abstract
Data selection techniques applied to neural machine translation (NMT) aim to increase the performance of a model by
retrieving a subset of sentences for use as training data.
One of the possible data selection techniques are transductive learning methods, which select the data based on thetest set, i.e. the document to be translated. A limitation ofthese methods to date is that using the source-side test set
does not by itself guarantee that sentences are selected withcorrect translations, or translations that are suitable given the
test-set domain. Some corpora, such as subtitle corpora, may
contain parallel sentences with inaccurate translations caused
by localization or length restrictions.
In order to try to fix this problem, in this paper we propose to use an approximated target-side in addition to thesource-side when selecting suitable sentence-pairs for training a model. This approximated target-side is built by pretranslating the source-side.
In this work, we explore the performance of this generalidea for one specific data selection approach called Feature
Decay Algorithms (FDA).
We train German-English NMT models on data selectedby using the test set (source), the approximated target side,
and a mixture of both. Our findings reveal that models built
using a combination of outputs of FDA (using the test set
and an approximated target side) perform better than thosesolely using the test set. We obtain a statistically significant
improvement of more than 1.5 BLEU points over a modeltrained with all data, and more than 0.5 BLEU points over astrong FDA baseline that uses source-side information only.

1. Introduction
Supervised machine learning aims to learn predictive models
from a set of labeled examples (training data) so that it canaccurately predict the labels of new, unlabeled, examples.
Having more data may seem at first glance to be beneficial
to building more accurate models, but upon closer inspection this is not necessarily always the case. Machine learningmodels by design have an inductive bias that forces them togeneralize over the training examples rather than just memorizing them without generalization. This means, however,that if the size of the training set is increased, this may leadto optimizing the model for predicting the labels of more examples, but which on average are less relevant at test timethan would be the case for a more focused, smaller training set. The intuition of the importance of using a highly
relevant set of training examples is captured well by the Knearest neighbour model, which essentially computes at test
time on-the-fly a very localized density estimate for every
test example, based on the K training examples closest to thetest example. It then uses this density estimate for classification. For the K-nearest neighbour model, increasing K toomuch is at the expense of basing predictions on an increasing number of less relevant examples. Furthermore similar
to the K-nearest neighbour model, other predictive models
which typically discard the original training examples andkeep only a learned generalization over these examples, cansuffer if the training data becomes bigger but on average lessrelevant to the test set.
In Machine Translation (MT), the data used to build themodels are parallel sentences (pairs of sentences in two languages, which are translations of each other) and we encounter the same problem when the amounts of data become
excessively large. Too much training data may cause themodel to be too generic, and not perform well if testsrc (thedocument to be translated, i.e. the test set), belongs to a specific domain.
Data selection techniques aim to solve that problem byselecting a subset of training data. Models that are trainedon a small set of parallel sentences can perform better thanthose trained on all training data [1; 2].
Within the data selection field we can find several approaches to reduce the data: select sentences of good translation quality (data quality), select sentences relevant for a particular domain (domain adaptation), or select sentences thatare relevant for testsrc (transductive learning). We focus on
this last type, and so in this paper we propose new methods
to build Neural Machine Translation (NMT) models that are
tailored towards a testsrc .
Transductive learning [3] aims to find the best training
instances given an unlabeled example. In MT this means
finding the best parallel sentences given a document testsrc
to be translated. In our work, the transductive data-selection
method that we explore is Feature Decay Algorithms (FDA)[4; 5; 6]. Standard FDA uses the n-grams of testsrc to retrieve training sentence pairs with source-side most similar
to testsrc . FDA has demonstrated good performance in Statistical Machine Translation (SMT) and NMT [2].
In most cases, FDA is used as a single step in the pipeline

173
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

of building a model, using testsrc to extract a subset of parallel sentences. In this paper, we propose a different configuration of use of FDA for building NMT models (see left
side of Figure 1). In particular, we propose executing FDA
not only using the testsrc (source-side language), as is common, but additionally on a pre-translated test set (approximated target-side). In order to avoid confusion, in this work
we use testsrc to indicate the test set (in the source-side language) and testtrg to indicate the pre-translation of the testset (in the target-side language). The outputs of these two
executions can be combined into one training set to build amodel that produces better translations than models built using FDA having only testsrc as input.
Considering both the source side and target side of theparallel sentences as selection criteria is especially usefulwhen using a corpus that includes sentences from subtitles
in different languages. There are particular problems concerning parallel sentences comprising subtitles. For example, both sentences in the source and target side are limited
to be displayed in the same time window (assuming they aresynchronized). As the length of the same sentences in different languages can be different, this may causes the longest
one to be rephrased, split in two, or have words omitted so itmeets the time requirement.
In our work we use an approximated, synthetic targetside using a technique we call pre-translation. One way tolook at this is as a form of synthetic-data generation. As
such it is somewhat reminiscent of synthetic source-data generation using a target-to-source translation model, a technique known as back-translation introduced by Sennrich etal.(2016) [7].

2. Related Work
Data selection techniques aim to select a subset of data suchthat the models trained on that subset perform better. There
are multiple approaches to achieve those improvements, suchas domain adaptation or noise reduction approaches [8].
Methods based on domain adaptation include the work of
Moore and Lewis (2010) [9], who propose to use language
models (LM) to select data. An LM is a distribution oversequences of words in a monolingual text, and is often usedby SMT systems to model the fluency of the outputs. Givena string s and a language model LMd , Hd (s) is the entropyof the distribution of s according to LMd .
Moore and Lewis build an in-domain language model
LMI and an out-of-domain language model LMO , and determine how likely each sentence s is to be in-domain bycomputing the entropy difference [HI (s) − HO (s)]. Axelrod et al. (2011) [10] extend the method by using LMs inboth the source-side and target-side languages, defining thebilingual cross-entropy difference.
Another method, proposed by van der Wees et al. (2017)[1], is to gradually remove out-of-domain sentences each ηepochs when training the NMT model.
In our work, we select data that is similar to testsrc (andso, more relevant for use as training data). Previous researchon selecting data considering the test set includes the work of
Li et. al. (2018) [11] where they fine-tune a pre-built NMT
model using training data selected based on testsrc . Theyuse similarity measures, such as Levenshtein distance [12] orthe cosine similarity of the average of the word embeddings,[13].
The method that we use to select data is FDA [4; 5; 6],
which has already proven to be useful in SMT [14; 15; 16]
and NMT [2]. Selecting a small subset of sentences from aparallel corpus using FDA is enough to train SMT systems
that perform better than systems trained using the whole parallel corpus.
FDA takes as input a set of parallel sentences U and aseed (generally the testsrc ). Given U and the seed, FDA
retrieves an ordered sequence of sentences L from U . Sentences are ordered based on the amount of n-grams they sharewith the seed, with more shared n-grams meaning higher
preference, while also considering the variability of the ngrams in the selected sentences.
The algorithm initializes L as a void sequence and iteratively selects one sentence s ∈ U − L and appends it to L.
The sentence s to select at each step is the one most relevant
to testsrc , based on the number of n-grams that s shares withthe testsrc . The score of the relevance is computed as in (1):P
CL (f )
f ∈Fs 0.5
(1)score(s) =# words in s
where Fs is the set of n-grams present in s and testsrc (bydefault the order of the n-grams ranges from 1 to 3). CL (f )is the count of the n-gram f in the sequence L of selected
sentences. Including CL (f ) in the computation of the scorecauses the algorithm to penalize n-grams that have been selected several times, and hence favouring the selection ofsentences that contain new n-grams.

3. Using an Approximated Test Target-side
FDA uses testsrc as seed to retrieve a subset from a set ofparallel sentences. In order to retrieve the sentences it scores
the n-grams of testsrc (source-side language). We show the
pipeline of usage of FDA on the left side of Figure 1. Here,the files testsrc and parallel text are used as input, and FDA
retrieves a subset of the sentences to be used for building amodel that is adapted to testsrc .
We propose to use both the test source-side testsrc andthe approximated test target-side testtrg as features in FDA,
when selecting the set of sentences from the parallel text.
We show the pipeline of our approach on the right side of
Figure 1. First, testsrc is translated (translate step). Then,
using FDA, we select a subset of parallel sentences given: (a)testsrc as seed (F DAsrc ), and (b) testtrg as seed (F DAtrg ).
These two sets can be combined into one set which serves as
training data to build an MT model.
In the following subsections we explain in more detail
two issues that are yet unanswered in the pipeline : (1) how

174
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Figure 1: Pipeline of the traditional usage of FDA (left) and pipeline of our proposal, using the target-side (right).
to build testtrg (addressed in Section 3.1), and (2) how to
combine the outputs of FDA (addressed in Section 3.2).
3.1. Pre-Translation of testsrc
The first step in our approach consists of building testtrg(translate step on the right side of Figure 1) so it can be usedas the seed to extract parallel sentences using the target side.
In order to perform this pre-translation we need to build amodel, which we refer to as the initial model.
There are several approaches to build the initial model,such as using SMT or NMT. These models can be trained using the full training data or subsets (such as randomly sampled, selected according to a particular domain, etc.). In thiswork we use an NMT model built with the full training data.
3.2. Combining FDA outputs
In order to combine the sentences of F DAsrc and F DAtrg
into one training set of N sentences, various strategies are
possible such as retrieving the intersection or the union ofsentences. In this work we explore the strategy of concatenating both outputs (allowing the repetition of sentences) andpropose as future work alternative methods for merging bothparallel datasets.
The outputs of F DAsrc and F DAtrg can be seen as anordered sequence of sentences as in equation (2) and equation (3):(src)

F DAsrc = (s1

(src)

, s2

(src)

, s3

(src)

, ...sN

)

(2)

(trg)

F DAtrg = (s1

(trg)

, s2

(trg)

, s3

(trg)

, ...sN)(3)

In order to obtain a training set that combines the outputsof F DAsrc and F DAtrg , we concatenate the top sentencesof each subset to obtain a new list of sentences of size N , asin equation (4)(src)

F DA = (s1

(src)

(trg)

, ...sN ∗α , s1

(trg)

, ...sN ∗(1−α) )(4)where 0 ≤ α ≤ 1 indicates the proportion of sentences thatare selected from F DAsrc and F DAtrg .
Note that some of the sentences may be replicated; it
(src)
(trg)
= sj , i.e. those that have been
may happen that si
retrieved by both executions FDA. In this work we decided
to keep the duplicates as it may be beneficial to oversample those sentences in which there is an agreement of both
executions of FDA. However, we propose as future work toinvestigate the effect of removing those duplicate sentences.
The core of our approach is combining the outputs of thetwo executions of FDA (using the test and translated sets).
Given the concatenation method presented in this section, theoutputs can be classified as one of the three options:• Source-side only: use only the output of F DAsrc forbuilding the model. It is the configuration where α = 1
in Equation (4), which is equivalent to the traditional
procedure of using FDA (left side of Figure 1, so we
use this approach as the baseline.
• Target-side only: use the output of F DAtrg for building the model, which is the configuration where α = 0
in Equation (4).

175
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018• Source-and-target-side: combine F DAsrc and F DAtrg .
This is the configuration where different values of α inequation (4) are set. In our work we explore the values
α = 0.25, α = 0.50 and α = 0.75.

4. Experiments
4.1. Experimental Settings
We experiment with models for German-to-English direction. The parallel data used for the experiments is the training
data provided in the WMT 2015 [17] (4.5M sentence pairs,
225M words). The dev set of the NMT models (both the initial model and those trained using the selected datasets) are
5K randomly sampled sentences from development sets fromprevious years. All the models presented here are evaluatedusing the same test set which comprises documents provided
in WMT 2015 translation task as testsrc .
In order to build the NMT models we use OpenNMT-py,
which is the Pytorch port of OpenNMT [18]. All the NMT
models we build use the same settings (we only change thetraining data used to build them). The value parameters are
the default ones of OpenNMT-py (i.e. 2-layer LSTM with
500 hidden units, vocabulary size of 50000 words for eachlanguage). All the models are executed for 13 Epochs.
In the experiments we build models with the data selectedby using F DAsrc and F DAtrg . We explore selecting different sizes of selected data: 500K, 1M and 2M sentence pairs.

5. Results
BLEU
TER
METEOR
CHRF3

baseline
0.24740.55250.279848.9473

Table 1: Results of the model trained with all available training data; also the no-FDA baseline.

First, we show in Table 1 the quality of the pre-translated
testtrg . This has been produced by the initial model, an
NMT model trained with all training data. This result also
serves as a no-FDA baseline to asses the benefit of using FDA
in general with.
The evaluation metrics presented in Table 1 give an estimation of the similarity between the model output and ahuman-translated reference. The evaluation metrics we use
are: BLEU [19], TER [20], METEOR [21] and CHRF [22].
The results of the models are shown in Table 2. The
columns show the different configurations used to build theset of selected sentences (i.e. the value of α in equation (4)used). This means that the column α = 0.75 shows the results of the model trained with the sentences from the top750K sentences of F DAsrc and the top-250K sentences of

F DAtrg .
First, one may wonder whether FDA data selection is at
all helpful? Comparing the scores in Table 2 to the baseline
system trained on all data in Table 1, we see that all FDA
systems outperform it, with the best one obtaining more than
1.5 BLEU points improvement (a relative improvement of
6%).
We have marked in bold the scores that outperform thesecond baseline: FDA applied using testsrc only (i.e the configuration using F DAsrc and α = 1), as proposed in [2], andcomputed the statistical significance (marked with an asterisk) with multeval [23] for BLEU, TER and METEOR whencompared to the baseline at level p=0.01 using bootstrap resampling [24].
5.1. Ratio of data obtained using source and target side
Intuitively, models built using the data selected based on testtrg
might perform worse than using testsrc only. testtrg may
contain errors produced by the machine-generated text, so analgorithm that bases the decision on that text may not selectthe best sentences. Indeed, this can be seen in the column
α = 0 of Table 2, where most of the scores are worse thanthose in column α = 1.
On the other hand, using only testsrc as a selection criterion also has its limitations. While it guarantees the selected source sentences to be similar to testsrc , it does notprovide any information about the target side of the selected
sentences. Therefore, it may still select sentences with targetside translations that are wrong or not suitable given the domain of the test-set, thereby hurting the final translation accuracy.
Using training data containing parallel sentences that are
not an accurate translation of each other is a problem that canbe encountered when using parallel sentences obtained from
subtitles. Often, translation of subtitles needs to be adapted
to meet length requirements (due to the restriction of time itis displayed on screen). We present some examples of sentences that are not accurately translated in Table 3.
We find that selecting sentences based both on testsrc
and on testtrg works better than using one selection criterion alone. Thus, using an approximated target side, even
if imperfect, can help. The best performance is obtainedusing configurations that combine outputs of F DAsrc and
F DAtrg (α = 0.75, α = 0.50 and α = 0.25 columns).
The best results are obtained for α = 0.75 using 1 million
sentences for selection. This setting improves 1.53 BLEU
points over the no-FDA baseline (model trained with all data)
and 0.67 BLEU points over the baseline that uses only thesource side for selection in FDA.
In Table 3 we show examples of sentences that are exclusive outputs of F DAsrc or F DAtrg . These examples givean indication about how including the output of F DAtrg canbenefit (or hurt) the quality of the selected data.
In the first row we see that the sentence “nun gibt es kein
Zurück mehr .” has been selected by F DAsrc as it matches

176
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

1M lines 500K lines
2M lines

BLEU
TER
METEOR
CHRF3
BLEU
TER
METEOR
CHRF3
BLEU
TER
METEOR
CHRF3

α=1
0.25170.56010.288649.83140.256
0.54970.288650.0932
0.25850.54540.289450.095α = 0.75
0.25420.5521*0.289550.0915
0.2627*0.5455*0.2920*50.6273
0.26100.54290.2923*50.5582

α = 0.50
0.25430.55630.288249.8898
0.25950.54620.2921*50.5226
0.25800.54650.290350.2431

α = 0.25
0.25340.55440.288849.90740.2600*0.5493*0.2918*50.5682
0.26140.54370.2927*50.5487α=00.24410.56280.278948.77960.2496
0.55340.283349.51920.25470.54960.285249.7838

Table 2: Results of the models using different sizes of F DAsrc and F DAtrg .

GermanEnglishpos
F DAsrcpos
F DAtrg

nun gibt es kein Zurück mehr .
diese Zahl ist mehr als doppelt so viel , als vor 10
Jahren .

there is no going back now .
famous pieces from the 19th century include thoseby Delacroix , Gauguin , Monet , Renoir andCorot .
and I could continue .
all data can be downloaded free of charge .

12
50

-

-

63
92

diese Aufzählung ließe sich beliebig fortführen .
bitte beachten Sie , dass Sie sich registrieren
lassen müssen , um einen Zugang zu den detaillierten Außenhandelsdaten zu erhalten .

Table 3: Examples of sentences retrieved by F DAsrc and F DAtrg

177
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

“kein Zurück mehr” in the input. According to this sentence,
this n-gram should be translated as “no going back”. The
translation found for “kein Zurück mehr” in testtrg is “point
where there is no return” (which, in addition, is closer to thereference “point of no return”) and hence F DAtrg will use
n-grams such as “point” or “no return” to retrieve sentences.
In the second row, we find an example of a sentence retrieved by F DAsrc whose translation is not accurate (this iseasily noticeable as the names “Delacroix, Gauguin, Monet,
Renoir and Corot” are not present in the English-side sentence). Including this sentence in the training data causes thequality to decrease and the models to perform worse. This
problem is not exclusive of F DAsrc , as in rows 3 and 4 we
see the same problem happening in the output of F DAtrg .
Combining the outputs of F DAsrc and F DAtrg causes
the training data to be reinforced with sentences with relevanttranslations. Note that mixing the outputs of the two executions of FDA cause some sentence pairs to be replicated, as
there is an overlap of the outputs.
In Table 4 we indicate the amount of unique lines contained in the training data of the models (those presented inTable 2). In the table we observe that the number of uniquelines is high in all training sets. The proportion of uniquelines ranges from 82% to 94%, which shows how F DAsrc
and F DAtrg retrieve different sentences mostly.

500K
1M
2M

α = 0.75
471753 (94%)
918506 (92%)
1749015(87%)α = 0.50
460993 (92%)
886685 (89%)
1648727(82%)

In our experiments, we built models using training setscontaining replicated instances of sentence pairs (as the output of the two runs of FDA, on the source-side and targetside, may overlap). This opens the door to exploring data
selection algorithms allowing the repetition of selected instances.
In the future, we want to consider other procedures for
combining the outputs of FDA, as we believe that other merging strategies may achieve better results. For example, considering both n-grams on the source and target side in combination (rather than two separate executions of FDA) may
achieve better performance.
In addition, we want to explore the performance whenusing a different initial model. Changing the initial model toproduce the testtrg causes F DAtrg to have a different performance. We believe that using another dataset to build theinitial NMT model (or even using different paradigms suchas SMT or rule-based MT) or choosing an initial model thatis also closer to testsrc (e.g. using FDA to build the initialmodel) should boost the performance. Moreover, the use of
several initial models allow us to perform concatenations ofseveral outputs of FDA using different seeds.
Finally, we want to explore how data selection algorithms
may improve when allowing the algorithm to select the samesentence pairs several times.

7. Acknowledgements

α = 0.25
471174 (94%)
917087 (92%)
1745142(87%)

This research has been supported by the ADAPT Centre for
Digital Content Technology which is funded under the SFI
Research Centres Programme (Grant 13/RC/2106) and is cofunded under the European Regional Development Fund.

Table 4: Number of unique sentences in the training data.
When performing a column-wise comparison in Table 4,
we see how the number of unique lines is larger when the
output of one of the FDA models dominates the training data
(α = 0.25 or α = 0.75 columns) compared to those setsthat contain the same amount of sentences extracted fromF DAsrc and F DAtrg (column α = 0.50).
We also see that the larger the amount of selected data,
the more overlap exists between the two outputs (the proportional amount of unique lines is smaller). For example,
in column α = 0.50, when 500K lines are selected, there
are 92% non-repeated lines, and this decreases to 82% whenselecting 2M lines. The same can be observed in the othercolumns. This indicates how the selected data using F DAsrc
and F DAtrg tend to be more similar the more sentences are
retrieved.

6. Conclusion and Future WorkIn this work, we explored a different pipeline in which FDA
can be used. We discovered that using testtrg (which ismachine-generated) as the seed of FDA can improve the performance.

This work has also received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement
No 713567.

8. References[1] M. van der Wees, A. Bisazza, and C. Monz, “Dynamic
data selection for neural machine translation,” in Proceedings of the 2017 Conference on Empirical Methods
in Natural Language Processing, Copenhagen, Denmark, 2017, pp. 1400–1410.
[2] A. Poncelas, G. M. de Buy Wenniger, and A. Way,
“Feature decay algorithms for neural machine translation,” in Proceedings of the 21st Annual Conferenceof the European Association for Machine Translation,
Alacant, Spain, 2018, pp. 239–248.
[3] V. N. Vapnik, Statistical Learning Theory.
Interscience, 1998.

Wiley-[4] E. Biçici and D. Yuret, “Instance selection for machine translation using feature decay algorithms,” in178
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland, 2011, pp. 272–
283.
[5] E. Biçici, Q. Liu, and A. Way, “ParFDA for fast deployment of accurate statistical machine translation systems, benchmarks, and statistics,” in Proceedings of
the Tenth Workshop on Statistical Machine Translation,
Lisbon, Portugal, 2015, pp. 74–78.
[6] E. Biçici and D. Yuret, “Optimizing instance selection
for statistical machine translation with feature decay algorithms,” IEEE/ACM Transactions on Audio, Speech,
and Language Processing, vol. 23, no. 2, pp. 339–350,
2015.
[7] R. Sennrich, B. Haddow, and A. Birch, “Improving
neural machine translation models with monolingual
data,” in Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), Berlin, Germany, 2016, pp. 86–96.
[8] S. Eetemadi, W. Lewis, K. Toutanova, and H. Radha,“Survey of data-selection methods in statistical machine translation,” Machine Translation, vol. 29, no. 34, pp. 189–223, 2015.
[9] R. C. Moore and W. Lewis, “Intelligent selection of language model training data,” in Proceedings of the ACL2010 conference short papers, Uppsala, Sweden, 2010,
pp. 220–224.
[10] A. Axelrod, X. He, and J. Gao, “Domain adaptation
via pseudo in-domain data selection,” in Proceedings ofthe 2011 Conference on Empirical Methods in Natural
Language Processing, Edinburgh, Scotland, UK., 2011,
pp. 355–362.
[11] X. Li, J. Zhang, and C. Zong, “One Sentence One
Model for Neural Machine Translation,” in Proceedings of the Eleventh International Conference on
Language Resources and Evaluation (LREC 2018),
Miyazaki, Japan, 2018, pp. 910–917.
[12] V. Levenshtein, “Binary codes capable of correcting
deletions, insertions and reversals,” in Soviet Physics
Doklady, vol. 10, no. 8, 1966, pp. 707–710.
[13] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado,and J. Dean, “Distributed representations of words
and phrases and their compositionality,” in Advancesin neural information processing systems, 2013, pp.
3111–3119.
[14] E. Biçici, “Feature decay algorithms for fast deployment of accurate statistical machine translation systems,” in Proceedings of the Eighth Workshop on Statistical Machine Translation, Sofia, Bulgaria, August
2013, pp. 78–84.

[15] A. Poncelas, A. Way, and A. Toral, “Extending feature
decay algorithms using alignment entropy,” in International Workshop on Future and Emerging Trends in
Language Technology, Seville, Spain, 2016, pp. 170–
182.
[16] A. Poncelas, G. M. de Buy Wenniger, and A. Way, “Applying n-gram alignment entropy to improve feature decay algorithms,” The Prague Bulletin of Mathematical
Linguistics, vol. 108, no. 1, pp. 245–256, 2017.
[17] O. Bojar, R. Chatterjee, C. Federmann, B. Haddow, M. Huck, C. Hokamp, P. Koehn, V. Logacheva,
C. Monz, M. Negri, M. Post, C. Scarton, L. Specia, and
M. Turchi, “Findings of the 2015 Workshop on Statistical Machine Translation,” in Proceedings of the TenthWorkshop on Statistical Machine Translation, Lisboa,
Portugal, September 2015, pp. 1–46.
[18] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush,“Opennmt: Open-source toolkit for neural machinetranslation,” in Proceedings of the 55th Annual Meeting of the Association for Computational LinguisticsSystem Demonstrations, Vancouver, Canada, 2017, pp.
67–72.
[19] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu:a method for automatic evaluation of machine translation,” in Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia,
Pennsylvania, USA, July 2002, pp. 311–318.
[20] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, andJ. Makhoul, “A study of translation edit rate with targeted human annotation,” in Proceedings of the 7th
Conference of the Association for Machine Translation in the Americas, Cambridge, Massachusetts, USA,2006, pp. 223–231.
[21] S. Banerjee and A. Lavie, “Meteor: An automatic metric for MT evaluation with improved correlation withhuman judgments,” in Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures formachine translation and/or summarization, Ann Arbor,
Michigan, 2005, pp. 65–72.
[22] M. Popovic, “chrF: character n-gram F-score for automatic MT evaluation,” in Proceedings of the TenthWorkshop on Statistical Machine Translation, Lisbon,
Portugal, 2015, pp. 392–395.
[23] J. H. Clark, C. Dyer, A. Lavie, and N. A. Smith, “Better hypothesis testing for statistical machine translation:
Controlling for optimizer instability,” in Proceedings ofthe 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies(Volume 2: Short Papers), Portland, Oregon, 2011, p.
176–181.

179
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[24] P. Koehn, “Statistical significance tests for machinetranslation evaluation,” in Proceedings of the 2004
Conference on Empirical Methods in Natural Language
Processing, Barcelona, Spain, 2004, pp. 388–395.

180
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Multi-paraphrase Augmentation to Leverage Neural Caption Translation
Johanes Effendi1 , Sakriani Sakti1,2 , Katsuhito Sudoh1,2 , Satoshi Nakamura1,2
1

2

Nara Institute of Science and Technology, JapanRIKEN, Center for Advanced Intelligence Project AIP, Japan{johanes.effendi.ix4,ssakti,sudoh,s-nakamura}@is.naist.jp

Abstract
Paraphrasing has been proven to improve translation
quality in machine translation (MT) and has been widely
studied alongside with the development of statistical MT(SMT). In this paper, we investigate and utilize neural paraphrasing to improve translation quality in neural MT (NMT),
which has not yet been much explored. Our first contribution
is to propose a new way of creating a multi-paraphrase corpus through visual description. After that, we also proposed
to construct neural paraphrase models which initiate expert
models and utilize them to leverage NMT. Here, we diffuse
the image information by using image-based paraphrasing
without using the image itself. Our proposed image-based
multi-paraphrase augmentation strategies showed improvement against a vanilla NMT baseline.

1. Introduction
In general, sentence paraphrasing is a way to restate a concept with different vocabulary, style, and level of detail.
As defined by De Beaugrande and Dressler, a paraphrase
is an approximate conceptual equivalence among outwardly
different material [1]. In many language generation tasks,
paraphrasing plays a critical role for enrichment and adding
flexibility. In the MT system, paraphrases are often used
for multi-reference evaluation [1], pre-editing of source sentences [2, 3, 4] and automatic post-editing [5, 6, 7].
Moreover, since the development of SMT, there havebeen a lot of approaches for using paraphrasing to elaborate
the source language data. Such method have been concluded
as a convenient way to handle out-of-vocabulary (OOV) andrare words problem [8]. A study by Madnani and Dorr also
showed that by using targeted paraphrases, unfair penalization of translation hypotheses could be avoided [9]. Paraphrasing could also be used to augment the dataset size,
which correlates positively with translation result in SMT[4, 10].
However, despite a wide range of existing works of paraphrasing, MT studies usually use a strict definition of paraphrase which accepts only word substitution and reordering. The reason is that we cannot grasp a tangible concept
about the idea of the sentence being translated. On the other
hand, Hirst argues that paraphrases don’t necessarily need to
be fully synonymous. It is sufficient for them to be quasi-

synonymous, as a mutually replaceable form of truth applicable in some contexts [11]. By taking further this idea, as
long as the semantics of the mutual paraphrase sentence canbe determined, we actually can widen the paraphrase definition to some extent.
In this research, we treat an image as a symbolic form ofsentence idea, regarded as the basis of paraphrasing. We consider two sentences as paraphrase as long as both of them aretalking about the same image. This means that the word orphrase insertion and deletion based on the same picture as aconcept may now be accepted as one of the paraphrase variations. Slightly different from the usual use case, this definition can be called image-based paraphrasing. Furthermore,as paraphrasing to enable multi-source information in NMT
is not much investigated yet, in this study we explore the useof image-based paraphrasing to leverage NMT quality.
Recently, the Second Conference on Machine Translation (WMT17) accelerated a “Multimodal Machine Translation” shared task that aimed to translate the image descriptions into the target language. Most approaches focus on utilizing image features in addition to the information from asingle caption of the source language. However, the resultsfrom most submitted systems reveal that the additional image features could only slightly contribute to system performance. As pointed out by Calixto et al. [12] the image-text
latent representation combination approach has not yieldedsignificant improvement on WMT 2017 Multimodal sharedtask dataset testing. Here, we attempt to go in another direction in which we diffuse the image information by using image-based paraphrasing without using the image itself.
The resulting paraphrase captions are then utilized within amulti-source and multi-expert NMT model.
In summary, the contributions of this work include:
1. Introduce a new way of creating a multi-paraphrase
corpus through image captions so-called image-based
paraphrasing.
2. Generate multi-paraphrase sentences of the WMT17
Multimodal Translation Task dataset through crowdsourcing, which can be used by the community1
3. Develop automatic paraphrase generation in a semisupervised manner;
1 The data will be soon available publicly.

181
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

4. Utilize multi-expert translation in neural machinetranslation using our proposed paraphrase; and5. Improve the baseline used at WMT17 with a 13.2
BLEU score margin, which is close to the top scorethat used a multimodal model.

2. Multi-paraphrase Generation
2.1. Defining Paraphrase Elementary Operation
To train an NMT model with our image-based multiparaphrases, firstly we need to build a set of paraphrased
source sentences with images as the basis of paraphrasing.
However, the process of manually collecting paraphrases is
expensive and time-consuming. On the other hand, Resnik etal. (2013) proposed that corpus creation with a crowdsourcing platform provides such advantages as low cost, effectiveness, and reasonable quality [13].

Figure 1: Reference image for captioning and paraphrasing
shown in Table 1.
Furthermore, the requirement to have an image and several captions, are similar with an image captioning dataset
such as Microsoft Common Object in Context (MSCOCO)
dataset [14]. The caption of this dataset can be regarded asparaphrase, such as done by Prakash et al. for their neuralparaphrase generation study [15]. They stated that the annotators described the most obvious things in an image andconcluded that several captions of an image can be counted
as paraphrases. While this may be true, we cannot define
what kind of operation has been done from the original sentence to the paraphrase. Consequently, the arbitrary natureof the corpus distribution might cause the paraphrases to become noise to each other.
To prevent this, a set of paraphrase operation which
covers all possible paraphrase variations needs to be defined. Bhagat and Hovy categorized the variations of how
humans paraphrase [16] and argued that “although the logical definition of paraphrases requires strict semantic equivalence, linguistics accept a broader, approximate, equivalence.” Based on this idea, they analyzed paraphrase characteristics in various studies and in corpora and established 25
quasi-paraphrase classes, such as change in tenses, metaphor
substitution, and, function-word variations.

Given some quasi-paraphrases have very small frequency
in the MTC and MSRP corpora as reported by them, wegrouped these into 4 elementary paraphrase operations: deletion, insertion, reordering, and substitution. Then, we constructed a paraphrase corpus based on these four operations.
The paraphrase collection was done through a crowdsourcing platform on the partial WMT17 Multimodal Translation
Task dataset [17]. After that, we constructed our automaticneural paraphrase model based on partial data to generate theparaphrase sentences of the full WMT17 dataset. The details
are described below.
2.2. Crowdsourcing Paraphrases on Partial WMT17
DatasetThe WMT17 Multimodal Translation Task dataset [17] contains a set of images with triplets of captions in English,
German, and French. The dataset was created from the
Flickr30K Entities dataset of image captions in English [18]that was extended to also contain manually translated German and French captions. The data consists of 29000, 1014,
and 1000 triplets respectively for the training, development
and testing. An out-of-domain dataset consisting 461 images
taken from the MSCOCO dataset [14] was also introduced,
which contains ambiguous verbs [19].
We focused on paraphrasing the English sentences which
are considered as source language. Table 1 shows an example of a paraphrased image caption based on four elementary
operations (deletion, insertion, reordering, and substitution)
and Figure 1 shows the reference image. As paraphrasing
the whole 29k triplet training dataset (29k training dataset)using crowdsourcing would not be efficient in terms of cost
and time, we crowdsourced only 10k triplets of this dataset(10k training dataset), along with the whole development andtesting datasets.
We used Crowdflower2 (now Figure Eight) as the crowdsourcing platform. Each crowdworker was instructed to paraphrase at least two image captions for one session. We
limited the task to English speakers, or at least those who
spoke English as their second language, to maintain quality. We discarded sentences that were not valid such as
randomly inputted character, empty string, or captions thataren’t English. The crowdsourcing process took about 3
months and 201 workers participated from 16 countries suchas the United States, Philippines, and Malaysia. Each workers created 50.1 quintuplets of paraphrases on average.
2.3. Semi-supervised Paraphrase Generation on Full
WMT17 Dataset
Furthermore, to complete the paraphrasing on the full
WMT17 dataset, we then used 10k quintuplets of crowdsourced paraphrases and constructed neural paraphrase
model using four encoder-decoder long short-term memory(LSTM) models with attention [20] for each paraphrase oper2 http://www.figure-eight.com

182
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 1: Image caption and example paraphrases

Operation
Image Caption
Deletion
Insertion
Paraphrase
Substitution
Reordering

Sentence
A little gray dog jumps over a small hurdle.
A little gray dog jumps over a hurdle.
A little gray dog jumps over a small hurdle successfully.
A little gray dog pass over a small hurdle.
Over a small hurdle, a little gray dog jumps.

ation. We tuned and tested our automatic neural paraphrase
model using these crowdsourced paraphrases of the development and testing datasets, respectively. With these fourparaphrasing models, we generated multi-paraphrases on theremaining 19k image captions.
The generated 19k dataset was combined with the original crowdsourced 10k training dataset. Finally, these 29k
paraphrased dataset are combined with original dataset resulting 58k-triplet training dataset for each operation. In conclusion, the 29k paraphrased training dataset is working as
the regularizer for the original dataset. These are the final
data that will be used to train a mixture-of-experts translation model, which is described in the next section. The data
will be publicly available to augment the WMT17 dataset.
Based on our empirical observation, using paraphrased
data on development and testing dataset will reduce the performance of the overall system. When using paraphrased
data on development, the training objective becomes unclear,and the loss returned will not represent the real loss. Given
that, we emphasize that the use of paraphrased dataset intranslation step was done on training step, in combination
with original dataset. In this stage, the paraphrases were acting as regularizer and the means of ensembling, improving
robustness of the ensembled model as a whole.

3. Neural Caption Translation
This section describes several approaches on using our proposed multi-paraphrase operations to improve NMT. The
score of these approaches will then be compared with WMTbaseline and our encoder-decoder LSTM NMT baseline.
3.1. Combining All Data in a Single Model
This method was done by just using the paraphrase as ameans for data augmentation in source side, such as reported
by Nichols et al. (2010) to leverage SMT system [10]. All
paraphrases and its original sentence were combined, andthe target sentence was duplicated to the number of multiple
paraphrases. This approach was done to measure the baseline
performance with augmented data.
3.2. Multi-source Model
We implemented Zoph and Knight (2016) multi-source NMT
to incorporate various paraphrase inputs with one output[21]. For this model, the encoded representation and attention were combined by concatenation. They reported that

this model has the advantage of information triangulation to
reduce ambiguity. In their paper, they used several translation pairs such as {French, German} to English in which this
triplet of language has similar language structure. However,given this advantages, the use of this model to monolingual
input has never been investigated.
3.3. Uniform-weighted Ensemble Model
For this uniform weighted ensemble model, we trained NMT
models which source sentence has been paraphrased basedon each elementary operations and another one that uses
original source sentence, resulting five expert NMT models. After that, these five models are ensembled by averaging each output layer probability distribution, so that every
model was weighted uniformly. This model is used to compare the performance with mixture-of-experts model listed in
the next subsection, where each expert model have different
weight.
The training of this translation model consists of two
steps. The first step is to train five translation models basedon each paraphrase as the source sentence using the 56k
dataset (the combination of original and paraphrased source
sentences). Five of those models are trained against the sametarget sentence. Each model is then regarded as an expert
model. Each of the expert models operates on subword level,
tokenized by Sentence Piece with 3000 vocabulary unit3 .
3.4. Mixture-of-experts Model
Next, we adopted the mixture-of-experts model proposed byGarmash and Monz (2016). Here, instead of linear layer proposed in their study [22], the expert model is implemented
into a single LSTM layer hid that receives the concatenateddecoder hidden state output hn .
ct = tanh(LST Mhid ([h0 , h1 , ..., hn ]))g0:i = sof tmax(Wgate D(ct ) + bgate ).
A sof tmax function is then applied to obtain the weightsof each expert model’s output layer on . Assuming Wn is theweight of the output layer from expert n. Then, the aggregated weight Wagg is a linear combination function of eachof those weights:
Wagg = g0 W0 + g1 W1 + ... + gn Wn .
3 https://github.com/google/sentencepiece

183
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Table 2: Paraphrasing model result in BLEU and METEOR
Operation
Deletion
Insertion
Reordering
Substitution

BLEU
53.0
56.1
47.2
59.6

METEOR
42.2
40.5
42.0
44.8

For this model, a 50% dropout D will be applied on thehidden representation after tanh nonlinearity was applied.
The regularized representation was further transformed by
the gate layer which has the same output size with the number of expert.
A diagram of mixture-of-experts neural caption translation model using our proposed approach is shown in Fig. 2.
First, the source sentence is paraphrased into four different
paraphrases used to train each of the expert model. Then,each expert will pass their abstract decoding state into mixture model which will produce weights as many as the number of expert. The resulting weight distribution is the linear combination function between each expert’s output probability distribution and gating weight produced by mixture
model.

4. Experiments
The purpose of this experiment is to choose the best type ofmodel suitable for our multi-paraphrase, by comparing score
between Bahdanau et al. NMT baseline and several popular
multi-source NMT.
4.1. Setup
We followed the training, development, and test set-up of
WMT17 shared task. All result were scored using multeval [23] with lowercased and tokenized sentences. We used
BLEU [24] and METEOR [25] as evaluation metrics.
The multi-source NMT has five single-depth encoders
with 512 hidden size trained with Adam [26]. The mixtureof-experts model was trained using RMSprop optimizer with
0.0001 learning rate [27]. In every increase of development
loss, the learning rate is decayed by half into maximum 5
decays. The results are decoded with beam size of 5.
4.2. Evaluation of Neural Paraphrase Model
We constructed four encoder-decoder LSTM models with attention [20] for each elementary paraphrase operation. Eachmodel has a bidirectional encoder and attentional decoderwith one layer, 50% dropout ratio, and 512 hidden layer size.
Implementation was done using Chainer framework version
3.0 [28] and ran on GTX Titan X GPU. We used Adam [26]as the optimizer with decaying alpha into half in every development loss increase with maximum of 7 decays for training
early stopping. After stopping the training, model with thelowest development was selected and used for decoding.
Table 2 lists the scores of the paraphrases produced with
our automatic paraphrasing model. The substitution opera-tion produced the highest BLEU score while the reordering
operation producing the lowest BLEU score. This was expected because the reordering operation sometimes includes
the changing of the active/passive properties of a sentence.
Overall, we believe this score is high enough to paraphrase
the remaining 19k WMT dataset.
4.3. Translation Model Results
Table 3 shows the performance of our proposed neural caption translation. All results using our multi-paraphrase outperformed the NMT baseline. There are no improvements
gained from combining all data, which is the simplest form ofdata augmentation. This simple combination of data breaks
the relation existed between each paraphrases that mention
the same image. Furthermore, we cannot be sure that eachsource sentence has the same amount of paraphrase. By
considering these factors, we utilized multi-source NMT and
multi-expert NMT, which yield better BLEU and METEOR
score.
This performance increase indicates that each expert
model is slightly different between each other, and worked
well in uniform-weighted ensemble and mixture-of-experts
scenario. This model also performed better than uniformweighted NMT in three cases. Moreover, the mixture-ofexperts model performed better in out-of-domain ambiguous
MSCOCO test dataset, implying that overfitting did not occur. This also proves the argument that adding additional
knowledge will improve model performance on disambiguating inputs. From applying to these several models, we canconclude that our elementary operation paraphrase is suitable
to be used as a means for ensembling.
Table 4 shows the current submission systems in the official WMT17 shared task which submissions consist of one
textual model [29] and several multimodal models. Our proposed approach outperformed the baseline in WMT17 with a
13.2 BLEU score margin. Our proposed model, although it
is textual, could produce competitive result with other multimodal models. The mixture-of-experts model outperformed
several multimodal models such as other WMT submission[30, 31, 32, 33]. Even in the out-of-domain dataset of COCO
2017, the mixture-of-experts model also performed reasonably high with a 28.0 BLEU score. Nevertheless, our score
was close to that best score. This proved that the paraphrasing of the source side also helped our model to work with
unseen data and prevent overfitting.
4.4. Discussion
To further analyze the contribution between the experts
trained on the original data and that trained on paraphrased
data, we compared the translation process step-by-step inour proposed approach. This source sentence shown in Table 5 was translated using each baseline model (an expert),
resulting five different translation hypotheses. Each expert
has been trained with slightly different paraphrased source

184
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Figure 2: Diagram of proposed mixture-of-experts neural caption translation model
Table 3: The performance of proposed neural caption translation in comparison with the baseline.
Textual Model
Our NMT Baseline
Combine all data
Multi-source NMT
Uniform weighted ensemble
mixture-of-experts ensemble

Test 2016
BLEU METEOR
37.7
55.6
36.7
53.9
37.6
55.4
39.6
56.9
40.5
57.6

Test 2017
BLEU METEOR
30.1
49.7
29.6
47.7
30.1
49.4
31.4
50.7
32.5
51.3

Test COCO 2017
BLEU METEOR
25.0
44.6
25.1
43.7
24.4
44.3
26.7
46.0
28.0
46.8

Table 4: Existing submission systems in official WMT17 shared task.
Textual Model
Official WMT Baseline
Zhang et al. (2017)
Multimodal Model
Madhyastha et al. (2017)
Calixto et al. (2017)
Ma et al. (2017)
Helcl and Libovicky (2017)
Caglayan et al. (2017)

Test 2016
BLEU METEOR
32.5
52.5
-

Test 2017
BLEU METEOR
19.3
41.9
31.9
53.9

Test COCO 2017
BLEU METEOR
18.7
37.6
28.1
48.5

Test 2016
BLEU METEOR
41.3
59.2
36.8
53.1
41.0
60.4

Test 2017
BLEU METEOR
25.0
44.5
29.8
50.5
31.0
50.6
31.1
51.0
33.4
54.0

Test COCO 2017
BLEU METEOR
21.4
40.7
26.4
45.8
27.4
46.5
26.6
46.0
28.5
48.8

Table 5: Examples of resulting sentences in mixture-of-experts model.

Translation Model(Data)
Baseline

Single Paraphrase Model
Uniform Weight
mixture-of-experts(Data)

Type
Original
Original
Deletion
Insertion
Reordering
Substitution
Ensemble
Ensemble
TargetSentences
BLEU-1
two motorcycles drive on a road along the river .
zwei motorradfahrer fahren auf einer straße entlang .
0.75
zwei motorräder fahren auf einer straße am fluss .
0.87
zwei motorradfahrer fahren auf einer straße am fluss .
0.84
zwei motorradfahrer fahren auf einer straße am fluss entlang .
0.95
zwei motorradfahrer fahren auf einer straße am flussufer .
0.82
zwei motorradfahrer fahren auf einer straße am fluss .
0.84
zwei motorräder fahren auf einer straße am fluss entlang .
0.97
zwei motorräder fahren auf einer straße dem fluss entlang .

sentence. We calculated BLEU-1 scores for each hypothesis against the target, resulting the source-reordered expert

model yielded the best result between all experts.
The aim of proposed mixture-of-experts model task is to185
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

make sure the best part of each model is kept, and leavingout any noise or error that might occur in each model result.
As can be seen from the German result from the mixture-ofexperts model compared with the target sentence, the only
difference is the word “am” in which the correct one shouldbe “dem”.
In this example, in deletion translation result, the word
“motorräder” is decoded instead of “motorradfahrer”. Another example is the phrase “fluss entlang” which can only befound in reordering translation result. This goodness on eachexpert model however, should be kept by the mixture modelby distributing right word in every word being decoded. In
conclusion, the final result of the ensemble of expert model
combines every goodness in each expert model.
Quantitatively, the mixture-of-experts model successfully kept the good feature of best performing 0.87 and0.95 BLEU-1 score yielded in source-deleted and sourcereordered model results respectively, resulting 0.97 BLEU-1
score. This is a significant improvement compared with the
BLEU-1 score of the uniform weighted model that was only
increased into 0.84.

5. Conclusions and Future Works
A single caption cannot represent all the information of theimage to which it refers to. In this study, we elaborated animage by various paraphrase operations. This enables us toincorporate additional knowledge from image to the translation process, without using the image itself, but diffused in aform of paraphrase.
We successfully generated multi-paraphrase sentences ofthe WMT17 Multimodal Translation Task dataset throughcrowdsourcing which will be publicly available. We constructed an automatic paraphrase generation model, and used
it with the multi-expert approach within NMT.
The results indicate that our proposed paraphrase elementary operations are best to be used for ensembling, especially on multi-expert ensembling settings. The hypothesis of regularizing models by paraphrasing on the source
sentence was proven to be effective. In the future, we willfurther investigate various methods of incorporating visual
information into NMT models.

6. Acknowledgement
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101 and JP17K00237.

7. References[1] R. De Beaugrande and W. Dressler, Introduction
to text linguistics, ser. Longman linguistics library.
Longman, 1981. [Online]. Available: https://books.
google.co.jp/books?id=mvJsAAAAIAAJ[2] A. Barreiro, SPIDER: A System for Paraphrasing in Document Editing and Revision — Ap-

plicability in Machine Translation Pre-editing.
Berlin, Heidelberg:
Springer Berlin Heidelberg, 2011, pp. 365–376. [Online]. Available:
https://doi.org/10.1007/978-3-642-19437-5 30[3] C. Callison-Burch, P. Koehn, and M. Osborne,“Improved statistical machine translation using paraphrases,” in Proceedings of the Main Conferenceon Human Language Technology Conference of theNorth American Chapter of the Association of Computational Linguistics (HLT-NAACL ’06), Stroudsburg,
PA, USA, 2006, pp. 17–24. [Online]. Available:
http://dx.doi.org/10.3115/1220835.1220838[4] W. He, S. Zhao, H. Wang, and T. Liu, “Enriching smt
training data via paraphrasing,” in Proceedings of IJCNLP, 2011.
[5] M. Simard, N. Ueffing, P. Isabelle, and R. Kuhn,“Rule-based translation with statistical phrase-based
post-editing,” in Proceedings of the Second Workshop
on Statistical Machine Translation (StatMT ’07),
Stroudsburg, PA, USA, 2007, pp. 203–206. [Online].
Available: http://dl.acm.org/citation.cfm?id=1626355.
1626383[6] M. Simard, C. Goutte, and P. Isabelle, “Statisticalphrase-based post-editing,” in Proceedings of the Main
Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL ’07),
Rochester, NY, USA, 2007, pp. 508–515.
[7] A.-L. Lagarda, V. Alabau, F. Casacuberta, R. Silva,
and E. Dı́az-de Liaño, “Statistical post-editing of arule-based machine translation system,” in Proceedingsof Human Language Technologies: The 2009 Annual
Conference of the North American Chapter of the
Association for Computational Linguistics, Companion
Volume: Short Papers, ser. NAACL-Short ’09,
Stroudsburg, PA, USA, 2009, pp. 217–220. [Online].
Available: http://dl.acm.org/citation.cfm?id=1620853.
1620913[8] S. Pal, P. Lohar, and S. K. Naskar, “Role ofparaphrases in pb-smt,” in Proceedings of the 15th
International Conference on Computational Linguistics
and Intelligent Text Processing - Volume 8404,
ser. CICLing 2014. Berlin, Heidelberg: SpringerVerlag, 2014, pp. 242–253. [Online]. Available:
https://doi.org/10.1007/978-3-642-54903-8 21[9] N. Madnani and B. J. Dorr, “Generating targeted
paraphrases for improved translation,” ACM Trans.
Intell. Syst. Technol., vol. 4, no. 3, pp. 40:1–40:25, July
2013. [Online]. Available: http://doi.acm.org/10.1145/
2483669.2483673

186
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[10] E. Nichols, F. Bond, D. S. Appling, and Y. Matsumoto, “Paraphrasing training data for statistical machine translation,” Journal of Natural Language Processing, vol. 17, no. 3, pp. 3 101–3 122, 2010.
[11] G. Hirst, “Paraphrasing paraphrased,” Invited talk at the
ACL International Workshop on Paraphrasing, 2003.
[12] I. Calixto, Q. Liu, and N. Campbell, “Doubly-attentive
decoder for multi-modal neural machine translation,” in
ACL, 2017.
[13] P. Resnik, O. Buzek, Y. Kronrod, C. Hu, A. J. Quinn,
and B. B. Bederson, “Using targeted paraphrasing andmonolingual crowdsourcing to improve translation,”
ACM Transactions on Intelligent Systems and Technology (TIST), vol. 4, no. 3, p. 38, 2013.
[14] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona,
D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft
coco: Common objects in context,” in European conference on computer vision. Springer, 2014, pp. 740–
755.
[15] A. Prakash, S. A. Hasan, K. Lee, V. Datla, A. Qadir,
J. Liu, and O. Farri, “Neural paraphrase generation
with stacked residual lstm networks,” in Proceedings of
COLING 2016, the 26th International Conference on
Computational Linguistics: Technical Papers. Osaka,
Japan: The COLING 2016 Organizing Committee,
December 2016, pp. 2923–2934. [Online]. Available:
http://aclweb.org/anthology/C16-1275[16] R. Bhagat and E. Hovy, “What is a paraphrase?” Computational Linguistics, vol. 39, no. 3, pp. 463–472,
2013.
[17] D. Elliott, S. Frank, K. Sima’an, and L. Specia,“Multi30k: Multilingual english-german image descriptions,” in Proceedings of the 5th Workshop on Vision and Language, 2016, pp. 70–74.
[18] B. A. Plummer, L. Wang, C. M. Cervantes, J. C.
Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k
entities: Collecting region-to-phrase correspondences
for richer image-to-sentence models,” CoRR, vol.
abs/1505.04870, 2015. [Online]. Available: http://arxiv.org/abs/1505.04870[19] D. Elliott, S. Frank, L. Barrault, F. Bougares, and
L. Specia, “Findings of the Second Shared Task on
Multimodal Machine Translation and Multilingual Image Description,” in Proceedings of the Second Conference on Machine Translation, Copenhagen, Denmark,September 2017.
[20] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,”
CoRR, vol. abs/1409.0473, 2014. [Online]. Available:
http://arxiv.org/abs/1409.0473[21] B. Zoph and K. Knight, “Multi-source neural translation,” CoRR, vol. abs/1601.00710, 2016. [Online].
Available: http://arxiv.org/abs/1601.00710[22] E. Garmash and C. Monz, “Ensemble learning for
multi-source neural machine translation,” in COLING,
2016.
[23] A. L. Jonathan Clark, Chris Dyer and N. Smith, “Better hypothesis testing for statistical machine translation:
Controlling for optimizer instability,” in Proceedings of
the Association for Computational Lingustics, 2011.
[24] K. Papineni, S. Roukos, T. Ward, and W. jing Zhu,
“Bleu: a method for automatic evaluation of machinetranslation,” 2002, pp. 311–318.
[25] A. Lavie and A. Agarwal, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” 2005, pp. 65–72.
[26] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014.
[27] A. Graves, “Generating sequences with recurrent
neural networks,” CoRR, vol. abs/1308.0850, 2013.
[Online]. Available: http://arxiv.org/abs/1308.0850[28] S. Tokui, K. Oono, S. Hido, and J. Clayton, “Chainer: anext-generation open source framework for deep learning,” in Proceedings of Workshop on Machine Learning
Systems (LearningSys) in The Twenty-ninth Annual
Conference on Neural Information Processing Systems(NIPS), 2015. [Online]. Available: http://learningsys.
org/papers/LearningSys 2015 paper 33.pdf[29] J. Zhang, M. Utiyama, E. Sumita, G. Neubig, and
S. Nakamura, “Nict-naist system for wmt17 multimodal translation task,” in WMT, 2017.
[30] P. S. Madhyastha, J. Wang, and L. Specia, “Sheffield
multimt: Using object posterior predictions for
multimodal machine translation,” in Proceedings of
the Second Conference on Machine Translation,
Volume 2: Shared Task Papers. Copenhagen,Denmark: Association for Computational Linguistics,
September 2017, pp. 470–476. [Online]. Available:
http://www.aclweb.org/anthology/W17-4752[31] I. Calixto, K. D. Chowdhury, and Q. Liu, “Dcu system
report on the wmt 2017 multi-modal machine translation task,” in Proceedings of the Conference of MachineTranslation (WMT), vol. 2, 2017.
[32] M. Ma, D. Li, K. Zhao, and L. Huang, “Osu multimodal
machine translation system report,” in Proceedings of
the Second Conference on Machine Translation,
Volume 2: Shared Task Papers. Copenhagen,Denmark: Association for Computational Linguistics,
September 2017, pp. 465–469. [Online]. Available:
http://www.aclweb.org/anthology/W17-4751

187
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[33] J. Helcl and J. Libovický, “CUNI system for the
WMT17 multimodal translation task,” CoRR, vol.
abs/1707.04550, 2017. [Online]. Available: http://arxiv.org/abs/1707.04550

188
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Using Spoken Word Posterior Features in Neural Machine Translation
Kaho Osamura1 , Takatomo Kano1 , Sakriani Sakti1,2 , Katsuhito Sudoh1,2 , Satoshi Nakamura1,2
Nara Institute of Science and Technology, JapanRIKEN, Center for Advanced Intelligence Project AIP, Japan1

2{osamura.kaho.oe5, kano.takatomo.km0, ssakti, sudoh, s-nakamura}@is.naist.jp

Abstract
A spoken language translation (ST) system consists of atleast two modules: an automatic speech recognition (ASR)
system and a machine translation (MT) system. In mostcases, an MT is only trained and optimized using error-freetext data. If the ASR makes errors, the translation accuracy
will be greatly reduced. Existing studies have shown thattraining MT systems with ASR parameters or word lattices
can improve the translation quality. However, such an extension requires a large change in standard MT systems, resulting in a complicated model that is hard to train. In thispaper, a neural sequence-to-sequence ASR is used as feature processing that is trained to produce word posterior features given spoken utterances. The resulting probabilisticfeatures are used to train a neural MT (NMT) with only aslight modification. Experimental results reveal that the proposed method improved up to 5.8 BLEU scores with synthesized speech or 4.3 BLEU scores with the natural speech incomparison with a conventional cascaded-based ST systemthat translates from the 1-best ASR candidates.

1. Introduction
Spoken language translation is one innovative technologythat allows people to communicate by speaking in their native languages. However, translating a spoken language, inother words, recognizing speech and then translating words
into another language, is incredibly complex. A standard approach in speech-to-text translation systems requires effort toconstruct automatic speech recognition (ASR) and machinetranslation (MT), both of which are trained and tuned independently.
ASR systems, which aim for the perfect transcription ofutterances, are trained and tuned by minimizing the word
error rate (WER) [1]. MT outputs are optimized and automatically measured based on a wide variety of metrics. Oneof the standard methods is the BLEU metric. However, all
the errors from the words in ASR outputs are treated uniformly without considering their syntactic roles, which areoften critical for MT. Many studies have investigated the effectiveness of the WER metric of ASR on the whole speechtranslation pipeline [2, 3, 4] and verified that ASR errors thatcompose the WER metric do not contribute equally to the
BLEU score of translation quality.

Furthermore, most MT systems are only trained and optimized using error-free text data. Despite the fact that ASR
technologies and their recognition rates have continued to
improve, the occurrence of speech recognition errors remains
inevitable. This is because there are many ambiguities due toa wide variety of acoustic and linguistic patterns produced
by different speakers with various speaking styles and background noises. If the ASR engine makes mistakes, the translation accuracy will be significantly reduced. Thus, ignoring the existence of ASR errors while constructing a speechtranslation system is practically impossible.
Previous research on traditional phrase-based MTs has
attempted to train the ASR and MT parameters of the loglinear model to directly optimize the BLEU score of thetranslation metric of full speech translation systems [3]. It allows the model to directly select recognition candidates thatare easy to translate and improve the translation accuracy
given an imperfect speech recognition. Ohgushi et al. [5]further elaborated various techniques in the context of thejoint optimization of ASR and MT, including minimum error rate training (MERT) [6], pair-wise ranking optimization(PRO) [7], and the batch margin infused relaxed algorithm(MIRA) [8]. Other studies directly performed translation on
the lattice representations of the ASR output [9, 10, 11]. The
results showed that a better translation can be achieved bytranslating the lattices rather than with the standard cascadesystem that translated the single best ASR output.
Recently, deep learning has shown great promise in manytasks. A sequence-to-sequence attention-based neural network is one type of architecture that offers a powerful model
for machine translation and speech recognition [12, 13]. Several studies revisited similar problems and proposed handlinglattice inputs by replacing the encoder part with a lattice encoder to obtain a lattice-to-sequence model [14, 15]. With
these methods, robust translation to speech recognition errors
became possible. However, this approach requires a large
modification to standard NMT systems, resulting in a complicated model that is hard to train. Also, as the NMT takes
word lattices as input, it might be difficult to backpropagate
a translation error to the ASR part.
An extreme case is to train the encoder-decoder architecture for end-to-end speech translation (ST) tasks, which directly translates speech in one language into text in another.
Duong et al. [16] directly trained attentional models on par-189
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

allel speech data. But their work focused only on alignment
performance. The works by Berard et al. [17] might be thefirst attempts that successfully build a full-fledged end-to-end
attentional-based speech-to-text translation system. But they
only performed with a small parallel French-English BTEC
corpus, and their best results were behind the cascade baseline model. Later on, Weiss et al. [18] proposed a similar
approach and conducted experiments on the Spanish Fisher
and Callhome corpora of telephone conversations augmented
with English translations. However, most of these works
were only done for language pairs with similar syntax andword order (SVO-SVO), such as Spanish-English or FrenchEnglish. For such languages, only local movements are sufficient for translation. Kano et al. [19] showed that direct attentional ST approach failed to handle English-Japanese language pairs with SVO versus SOV word order.
In this research, we also focus on English-Japaneseand we aim for a neural speech translation that is robust
against speech recognition errors without requiring significant changes in the NMT structure. This can be considered
as a simplified version of the one that directly performed
translation on the lattice representations. But, instead of providing full lattice outputs, we perform a neural sequence-tosequence ASR as feature processing that is trained to produce word posterior features given spoken utterances. Thismight resemble the word confusion networks (WCNs) [20]that can directly express the ambiguity of the word hypotheses at each time point. The resulting probabilistic features areused to train NMT with just a slight modification. Such vectors are expected to express the ambiguity of speech recognition output candidates better than the standard way using
the 1-best ASR outputs while also providing a simpler structure than the lattice outputs. During training, the approachalso allows backpropagating the errors from NMT to ASR
and performs join training. Here, we evaluate our proposed
English-Japanese speech translation model using both synthesized and natural speech with various degrees of ASR errors.

2. Overview of Attention-based Speech
Translation

The forward LSTM reads the input sequence from x1 to xN−−→
and estimates forward henc , and the backward LSTM reads
the input sequence in reverse order from xN to x1 and es←−−timates backward henc . Thus, for each input xn , we obtain
−−
→
←−−
enc
and backward henc :hencn by summation forward h−−→ ←enc−−
henc= hencnn + hn .

(1)

The decoder, on the other hand, predicts target sequencey = [y0 , y1 , y2 , ..., yM ] with length M by estimating conditional probability p(y|x). Here, we use uni-directional
LSTM (forward only). Conditional probability p(y|x) is estimated based on the whole sequence of the previous output:p(ym |y<m , x) = softmax(Wy h̃dec
m ).

(2)

Decoder hidden activation vector h̃dec
m is computed by applying linear layer Wc over context information cm and current
hidden state hdec
m :
dec
h̃dec
m = tanh(Wc [cm ; hm ]).

(3)

Here, cm is in the context information of the input sequence when generating current output at time m. It is estimated by the attention module over encoder hidden states
henc
n :
N
X
am (n) ∗ henc
(4)
cm =
n ,
n=1

where variable-length alignment vector am is computed
whose size equals length of input sequence x:
am

=

dec
align(hencn , hm )=dec
softmax(dot(hencn , hm )).

(5)

This step assists the decoder to find relevant information on
the encoder side based on the current decoder hidden states.
dec
Several variations calculate align(hencn , hm ). Here, we simply use the dot product between the encoder and decoder hidden states.
2.2. Automatic Speech Recognition

Our English-Japanese end-to-end speech translation system
consists of ASR and MT modules that were constructedon standard attention-based, encoder-decoder neural networkarchitecture [21, 22].
2.1. Basic Attentional Encoder-Decoder model
An attentional encoder-decoder model consists of an encoder, a decoder, and attention modules. Given input
sequence x = [x1 , x2 , ..., xN ] with length N , the encoder produces a sequence of vector representation henc =
enc
enc
(henc
1 , h2 , ..., hN ). Here, we used a bidirectional recurrent neural network with long short-term memory (bi-LSTM)
units [23], which consist of forward and backward LSTMs.

Speech recognition tasks estimate a word sequence givena sequence of speech features. Input sequence x =[x1 , ..., xN ] is the input speech filter bank feature sequenceof the source language, and target sequence y = [y1 , ..., yM ]is the predicted corresponding word sequence in the source
language.
2.3. Machine Translation
Machine translation tasks estimate a word sequence of a target language given a word sequence of a source language.
Input sequence x = [x1 , ..., xN ] is the word sequence ofthe source language, and target sequence y = [y1 , ..., yM ]is the predicted corresponding word sequence in the target

190
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018language. Here, xn is a one-hot vector in the baseline orposterior vector in the proposed method, ym is the index representation of the words, and y0 is an index representation of
the target sequence’s start.
2.4. Speech-to-text Translation
Speech-to-text translation tasks estimate a word sequence ofa target language given a sequence of speech features. Here,
we use both the sequence-to-sequence ASR and MT systems.
Output sequence y from ASR becomes input sequence x in
an MT system.

3. Proposed method: NMT using Spoken
Word Posterior Features

Figure 2: Proposed NMT architecture

4. Experiments
Figure 1: Construction of spoken word posterior features
Fig. 1 illustrates the construction of spoken word posterior features. Here, we train an end-to-end ASR using thestandard attention-based encoder-decoder neural network architecture described in the previous section. But instead ofproviding 1-best outputs of the most probable word sequence
to the translation system,ŷm = argmax p(ym |y<m , x),ym(6)

we utilize the posterior probability vectors before theargmax function:p(ym |y<m , x).

(7)This way the vectors can still express the ambiguity of the
speech recognition output candidates with probabilities.
The resulting probabilistic features are then used to train
the NMT with only a slight modification. We train theend-to-end NMT using the standard attention-based encoderdecoder neural network architecture described in the previous section. The only difference is in the input features. Instead of training the model with the one-hot vector of themost probable words, we utilize the posterior vectors obtained from the ASR. However, the dimension of input vector
representation used in a standard one-hot vector and the proposed posterior vectors is the same. The overall architecture
is illustrated in Fig. 2.

We evaluated the performance of the proposed method on
an English-Japanese translation task. To simulate the effectof various ASR errors, we first assessed it on synthesized
speech and later applied it to natural speech.
4.1. Data set
The experiments were conducted using a basic travel expression corpus (BTEC) [24]:• Text corpus
We used a BTEC English-Japanese parallel text corpusthat consists of about 460k (BTEC1-4) training sentences and 500 sentences in the test set.
• Synthesized speech corpus
Since corresponding speech utterances for the BTEC
parallel text corpus are not available, we used Google
text-to-speech synthesis [25] to generate a speech corpus of the BTEC1 source language (about 160k utterances). We used about 500 speech utterances in thetest set.
• Natural speech corpus
We also evaluated with natural speech. In this case,
we used the ATR English speech corpus [26] in our
experiments. The text material was based on the basic
travel expression domain. The speech corpus we usedconsisted of American, British, and Australian (AUS)

191
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

English accents with about 120k utterances spoken by
100 speakers (50 males, 50 females) for each accent.
The speech utterances were segmented into multiple
frames with a 25-ms window size and a 10-ms step size.
Then we extracted 23-dimension filter bank features using
Kaldi’s feature extractor [27] and normalized them to havezero mean and unit variance.
4.2. Models
We further used the data to build a speech translation systemwith attention-based ASR and MT systems. The ASR and
NMT share the same vocabulary (16,745 words). The dimensions of the distributed vector representation are smaller
than vocabulary size (the size depends on the model settings).
The hidden encoder and decoder layer consists of 500 nodes.
A batch size of 32 and a dropout of 0.1 were also applied.
For all systems, we used a learning rate of 0.0001 for the encoder and 0.0005 for the decoder and adopted Adam [28] toall the models.
As we aim to have a neural speech translation that is robust against speech recognition errors without requiring significant changes in the NMT structure. We constructed three
types of models that fit those requirements:

5. Result5.1. Speech Recognition System
To simulate different degrees of ASR errors, we constructed
an ASR model using synthesized speech with different numbers of training epochs, resulting in four different models
with the following WERs: (1) System 1 (WER=15.17%),
System 2 (WER=12.34%), System 3 (WER=11.05%), andSystem 4 (WER=8.82%). As a model that is trained withnatural speech, our performance achieved a 24.98% WER.
5.2. Translation System
As mentioned earlier, we compare three translation system:
one for standard text-based machine translation, one forbaseline speech translation with the cascade model, and one
for our proposed speech translation.

• Text-based machine translation system (upperbound)This is a text-to-text translation model from the source
language to the target language. Here the BTEC
English-Japanese parallel text corpus is used to train
the model.
• Baseline speech translation
This speech-to-text translation model was created bycascading the ASR (speech-to-text) in the source language with a text-to-text MT module using 1-best
ASR outputs. First, we pre-trained the NMT with the
BTEC English-Japanese parallel text corpus and thenfine-tuned the NMT model with a one-hot vector provided from the ASR.

Figure 3: Translation quality given synthesized speech input• Proposed speech translation
This speech-to-text translation model was created bycascading ASR (speech-to-text) in the source languagewith the text-to-text MT module using the ASR posterior vectors. First, we pre-trained the ASR with the
speech of the source language and the NMT with the
BTEC English-Japanese parallel text corpus. After
that, we fine-tuned the parameter of both models by
jointly training, where the posterior vector of ASR output is used as the NMT input.
Note that the ASR systems used for the baseline and the
proposed systems are the same. Also, all translation systems were tuned adequately, and the best model from training
epochs was selected for each system.

Figure 4: Translation quality given natural speech input
The quality of those translation systems with the inputof synthesized speech was evaluated using BLEU [29] and192
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

Figure 5: Attention matrix of text translation

Figure 6: Attention matrix of proposed method

shown in Fig. 3. Here System 1-4 represent of using different
ASR systems (1-4), respectively. The results show that thebetter the ASR performance, the stronger the baseline cascade model. Nevertheless, our proposed approach stable outperformed the cascade model in all cases. The BLEU score
improved from 4.8 to 5.8 compared to the baseline model.
The proposed methods (System 3 and 4) exceed the texttranslation because the recognition candidates included in theposterior vector made it possible to correctly distinguish confusing words in the word embedding of the text translation.
We will scrutinize this result in the next section.
Next, the quality of the speech translation systems using
natural speech was also evaluated using BLEU and shown in
Fig. 4. For the text translation, we provided the transcriptionof the natural speech, which is different than the text usedin Fig. 3. This system used the ASR model where WER is
24.98%. Importantly, unlike several published ASR systems
using BTEC dataset, our ASR system only used the text transcription of the training set for the language model. Therefore, the ASR results reported in the paper could not reachstate-of-the-art ASR performance. Nevertheless, the translation results are still convincing as evidence of the proposed
framework’s effectiveness. The proposed method improved
the 4.3 BLEU score of the baseline model, confirming thatthe proposed method is also effective for natural speech.

6. Discussion
Table 1 shows the sentence output examples in EnglishJapanese translation: (1) with ASR error, and (2) without
ASR error. In the first example, to analyze the effect of ASR
error, we compare the sentence output of the proposed modeland the baseline (the cascase model). Here, ASR misrecognized “shoe” as “station”. This error impacted the baseline(cascade system), where it translated “station” as “eki” (thecorrect translation for “shoe store” is “kutsuya”). How-

Table 1: Examples of sentences output: (1) with ASR error,and (2) without ASR error.
ASR reference
ASR result
Baseline
Proposed
MT reference
ASR reference
ASR result
Text translation
Proposed
MT reference

Example 1: With ASR error
Excuse me where is the closest shoe store?
Excuse me where is the closest station store?
Sumimasen ichiban chikai eki wa doko desuka?
Sumimasen ichiban chikai kutsuya wa doko desuka?
Sumimasen ichiban chikai kutsuya wa doko desuka?
Example 2: Without ASR error
i d like to have a perm and a haircut pleasei d like to have a perm and a haircut please
Paama to paama o onegai shitai nodesuga
Paama to katto o onegaishimasu
Paama to katto o onegaishimasu

Table 2: Posterior vector
RecognizedstationshoechangecashieralwaysPosterior
0.439
0.321
0.086
0.036
0.016

ever, in the proposed method, it was still able to translate it
to “kutsuya”. This might be because the ASR provided aposterior vector in which the recognition candidate and eacha posteriori probability are weighted (Table 2). Here, “shoe”
information was still contained in the posterior vector with
only slightly lower probability than “station,” and based on
the context information, the machine translation translated
the word as “kutsuya.”
In the second example, ASR provided a correct sentence.
Here, we compare the sentence output of the proposed modeland the text translation. Since the contexts of “perm” and“haircut” are close, the text translation mistakenly translated

193
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

both “perm” and “haircut” into “paama” (Fig. 5 illustrates
the text translation’s alignment matrix). On the other hand,
having a posterior vector as the input in the proposed model(see the attention matrix in Fig. 6) allowed NMT to correctly
distinguish confusing words by the word embedding of thetext translation.

7. Conclusions
In this research, a speech translation system that is robust
against speech recognition errors is obtained by using a posterior vector, which is a normalized vector that expresses theambiguity of the speech recognition candidates, as the inputof an NMT engine. The lower the WER of the ASR model is,
the weaker the tendency of translation error becomes. Nevertheless, the whole test’s accuracy surpassed the baseline. As
a result, the posterior vector improved the BLEU score by 4.8
to 5.8 points over the baseline in the simulation experimentand improved it by 4.3 BLEU points over the baseline in theexperiment using natural voice. By providing the probabilityof the speech recognition output candidates in speech translation, an optimal input selection for NMT was made. In the
future, we will directly perform join training from ASR to
NMT.

8. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101 and JP17K00237. We also thankPhillip Arthur, Do Quoc Truong and Andros Tjandra for their
feedback and insightful discussions.

9. References[1] X. He, L. Deng, and W. Chou, “Discriminative learning in sequential pattern recognition,” IEEE Signal Processing Magazine, vol. 25, no. 5, pp. 14–36, 2008.
[2] P. Dixon, A. Finch, C. Hori, and H. Kashioka, “Investigation on the effects of ASR tuning on speech translation performance,” in IWSLT, San Francisco, USA,2011.
[3] X. He, L. Deng, and A. Acero, “Why word error rate
is not a good metric for speech recognizer training forthe speech translation task?” in ICASSP, Prague, Czech
Republic, 2011.
[4] N. Ruiz and M. Federico, “Assessing the impact ofspeech recognition errors on machine translation quality,” in Association for Machine Translation in theAmericas (AMTA), 2014, pp. 261–274.
[5] M. Ohgushi, G. Neubig, S. Sakti, T. Toda, and S. Nakamura, “An empirical comparison of joint optimization
techniques for speech translation,” in INTERSPEECH,
2013, pp. 2619–2623.

[6] F. J. Och, “Minimum error rate training in statisticalmachine translation,” in ACL, Sapporo, Japan, 2003.
[7] M. Hopkins and J. May, “Tuning as ranking,” in
EMNLP, Edinburgh, UK, 2011, pp. 1352–1362.
[8] C. Cherry and G. Foster, “Batch tuning strategies forstatistical machine translation,” in NAACL, Montreal,
Canada, 2012, pp. 34–35.
[9] S. Saleem, S.-C. Jou, S. Vogel, and T. Schultz, “Using word lattice information for a tighter coupling inspeech translation systems,” in ICSLP, Jeju Island, Korea, 2004, pp. 41–44.
[10] R. Zhang, G. Kikui, H. Yamamoto, and W.-K. Lo,“A decoding algorithm for word lattice translation in
speech translation,” in IWSLT, Pittsburgh, USA, 2005,
pp. 23–29.
[11] E. Matusov, B. Hoffmeister, and H. Ney, “ASR wordlattice translation with exhaustive reordering is possible,” in Interspeech, Brisbane, Australia, 2008, pp.
2342–2345.
[12] j. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and
Y. Bengio, “Attention-based models for speech recognition,” arXiv preprint arXiv:1506.07503, 2015.
[13] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machinetranslation by jointly learning to align and translate,”
arXiv preprint arXiv:1409.0473, 2014.
[14] J. Su, Z. Tan, D. Xiong, R. Ji, X. Shi, and Y. Liu,
“Lattice-based recurrent neural network encoders forneural machine translation,” in AAAI, 2017, pp. 3302–
3308.
[15] M. Sperber, G. Neubig, J. Niehues, and A. Waibel,“Neural lattice-to-sequence models for uncertain inputs,” in EMNLP, Copenhagen, Denmark, 2017, p.
13801389.
[16] L. Duong, A. Anastasopoulos, D. Chiang, S. Bird, andT. Cohn, “An attentional model for speech translation
without transcription,” in HLT-NAACL, 2016.
[17] A. Berard and O. Pietquin, “Listen and translate: A
proof of concept for end-to-end speech-to-text translation,” in 30th Conference on Neural Information Processing Systems, 2016.
[18] R. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z. Chen,“Sequence-to-sequence models can directly translateforeign speech,” arXiv preprint arXiv:1703.08581,
2017.
[19] T. Kano, S. Sakti, and S. Nakamura, “Structured-based
curriculum learning for end-to-end English-Japanese
speech translation,” in INTERSPEECH, 2017.

194
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

[20] L. Mangu, E. Brill, and A. Stolcke, “Finding consensus in speech recognition: word error minimization andother applications of confusion networks,” Computer
Speech & Language, vol. 14, no. 4, pp. 373–400, 2000.
[21] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence tosequence learning with neural networks,” in Advancesin neural information processing systems, 2014, pp.
3104–3112.
[22] M.-T. Luong, H. Pham, and C. D. Manning, “Effective
approaches to attention-based neural machine translation,” arXiv preprint arXiv:1508.04025, 2015.
[23] S. Hochreiter and J. Schmidhuber, “Long short-term
memory,” Neural computation, vol. 9, no. 8, pp. 1735–
1780, 1997.
[24] T. Takezawa, E. Sumita, F. Sugaya, H. Yamamoto, andS. Yamamoto, “Toward a broad-coverage bilingual corpus for speech translation of travel conversations in thereal world,” in LREC, 2002, pp. 147–152.
[25] “Google
Text
toSpeech
https://github.com/pndurette/gTTS.

API,”[26] S. Sakti, M. Paul, A. Finch, X. Hu, J. Ni, N. Kimura,
S. Matsuda, C. Hori, Y. Ashikari, H. Kawai, H. Kashioka, E. Sumita, and S. Nakamura, “Distributed speechtranslation technologies for multiparty multilingual
communication,” TSLP, vol. 9, pp. 4:1–4:27, 2012.
[27] D. Povey, A. Ghoshal, G. Boulianne, L. Burget,
O. Glembek, N. Goel, M. Hannemann, P. Motlicek,
Y. Qian, P. Schwarz, et al., “The Kaldi speech recognition toolkit,” in IEEE 2011 workshop on automatic
speech recognition and understanding, no. EPFLCONF-192584.
IEEE Signal Processing Society,2011.
[28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980,
2014.
[29] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,
“BLEU: a method for automatic evaluation of machinetranslation,” in Proceedings of the 40th annual meetingon association for computational linguistics. Association for Computational Linguistics, 2002, pp. 311–318.

195
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

IndexAlbert, Joan, 104
Antonino, Mattia, 54, 147
Arase, Yuki, 14

Huang, Fei, 136

Bach, Nguyen, 136
Bahar, Parnia, 104
Bentivogli, Luisa, 62
Birch, Alexandra, 118
Bojar, Ondřej, 142
Braune, Fabienne, 7

Jia, Jia, 112

Casacuberta, Francisco, 39
Cattoni, Roldano, 147
Cattoni, Ronaldo, 2
Cettolo, Mauro, 2, 62
Chen, Jinkun, 112
Chen, Wei, 112
Chochowski, Marcin, 118
Chu, Chenhui, 14
Dahlmann, Leonard, 31Dessì, Roberto, 147
Domingo, Miguel, 39Duh, Kevin, 153
Effendi, Johanes, 181
Erdmann, Grant, 124Federico, Marcello, 2, 62, 160
Federmann, Christian, 62
Fraser, Alexander, 7
Golik, Pavel, 104
Grönroos, Stig-Arne, 89
Guo, Wu, 70
Gwinnup, Jeremy, 124
Ha, Thanh-Le, 131
Haddow, Barry, 118
Hangya, Viktor, 7Hansen, Eric, 124
Hewavitharana, Sanjika, 31

Inaguma, Hirofumi, 153

Kalasouskaya, Yuliya, 7Kano, Takatomo, 189
Khadivi, Shahram, 31
Kocmi, Tom, 142
Kreutzer, Julia, 166
Kurimo, Mikko, 89
Liu, Dan, 70
Liu, Junhua, 70
Liu, Quan, 70
M., Surafel, 160
M., Víctor, 95
Müller, Markus, 131
Ma, Zhiqiang, 70
Martinez-Villaronga, Adria, 104
Matusov, Evgeny, 104
Nagata, Masaaki, 14
Nakamura, Satoshi, 48, 181, 189
Negri, Matteo, 147
Neubig, Graham, 48
Ni, Chongjia, 136
Niehues, Jan, 2, 131
Nishimura, Yuta, 48
Ore, Brian, 124
Osamura, Kaho, 189
Pesch, Hendrik, 104
Peter, Jan-Thorsten, 104
Petrushkov, Pavel, 31
Poncelas, Alberto, 76
Poncelas„ Alberto, 173
Przybysz, Pawel, 118
Quan, Ngoc, 131

196

Renduchintala, Adithya, 153
Rouhe, Aku, 89
Ruiz, Nicholas, 54
Sakti, Sakriani, 181, 189
Sarasola, Kepa, 76
Schamper, Julian, 104
Scherrer, Yves, 83
Senellart, Jean, 23
Sennrich, Rico, 118
Shi, Liangliang, 112
Sokolov., Artem, 166
Son, Thai, 131
Song, Rui, 70
Sperber, Matthias, 131
Stüker, Sebastian, 2, 131
Sudoh, Katsuhito, 48, 181, 189
Sulubacak, Umut, 89
Takebayashi, Yuto, 14
Tiedemann, Jörg, 89
Turchi, Marco, 2, 54, 147
Variš, Dušan, 142
Waibel, Alex, 131
Wang, Yanfeng, 112
Wang, Yuguang, 112
Wang, Zhichao, 112
Wang, Zhiqi, 153
Watanabe, Shinji, 153
Way, Andy, 76, 173
Wei, Linyu, 112
Wen, Shixue, 112
Wilken, Patrick, 104
Williams, Philip, 118
Wu, Chongliang, 70
Xiong, Shifu, 70
Yan, Shen, 31
Young, Katherine, 124
Zeyer, Albert, 104
Zhang, Xuan, 153
Zhang„ Dakun, 23
Zhu, Weifeng, 112
Zong,Chengqing, xiii

197
Proceedings of the 15th International Workshop on Spoken Language Translation
Bruges, Belgium, October 29-30, 2018

www.iwslt.org
2018