12th Dec., 2016

The 3rd Workshop on Asian Language Translation (WAT2016), Japan
collocated with COLING 2016

1IIT Bombay’s English-Indonesian submission at
WAT: Integrating neural language models
with SMT
Sandhya Singh, Anoop Kunchukuttan,
Pushpak Bhattacharyya{sandhya, anoopk, pb}@cse.iitb.ac.in
Center for Indian Language Technology
IIT Bombay

12th Dec., 2016

WAT2016

2Motivation• At CFILT, English-Indonesian language pair is being

experimented as a part of a Project.
• Relatively new language pair among Asian languageTranslations.

12th Dec., 2016

WAT 2016

3About English-Indonesian Language pair
• Script is Latin for both English and Indonesian.
• Sentence structure followed is SVO (Subject Verb Object).
• Not much structural divergence between English andIndonesian.
• Indonesian is highly agglutinative and morphologically rich ascompared to English language.
• Indonesian is considered as resource poor language.

12th Dec., 2016

WAT 2016

4

Experiment Description (1/4)
Four different systems were trained for both directions oflanguage pair:
1. Phrase Based SMT system (Moses baseline )• MGIZA++ for word alignment• grow-diag-final-end heuristic• Lexicalized Reordering• Batch MIRA tuning• 5-gram LM with Kneser-Ney smoothing using SRILM• Data Statistics
Language

Training Set

Tuning Set

Test Set

For LMEnglish

44939 sentences

400 sentences

400 sentences

50000 sentences

Indonesian44939 sentences

400 sentences

400 sentences

50000 sentences

12th Dec., 2016

WAT 2016

5Experiment Description (2/4)
2. System using Neural Language Model as a feature fortranslation(NPLM)• Neural Language model with default NPLM settings (Vaswani et al. (2013))• Word embedding size as 700, 750, 800 for 5 epochs• One hidden layer• Integrated as a feature in PBSMT system• Data statistics
Language

Training Set

Tuning Set

Test Set

English44939 sentences

400 sentences

400 sentences

Indonesian44939 sentences

400 sentences

400 sentences

For LM
50000 sentences + 2Msentences (Europarl)
50000 sentences + 2Msentences (CommonCrawl)

12th Dec., 2016

WAT 2016

6

Experiment Description (3/4)
3. System using Bilingual Neural Language Model as afeature for translation(NNJM)• Neural network joint LM with Parallel data (Devlin et al. (2014))• 5-gram LM with 9 source context word
• One hidden layer• Integrated as a feature in PBSMT system• Data Statistics
Language

Training Set

Tuning Set

Test Set

For LM

English44939 sentences

400 sentences

400 sentences

50000 sentences

Indonesian44939 sentences

400 sentences

400 sentences

50000 sentences

12th Dec., 2016

WAT 2016

7

Experiment Description (4/4)
4. System using Operation Sequence Model fortranslation(OSM)• Integrates 5-gram-based reordering and translation in a single generative

process (Durrani et al. (2013))• Deals with words along with context of source & target.
• Data Statistics
Language

Training Set

Tuning Set

Test Set

For LM

English44939 sentences

400 sentences

400 sentences

50000 sentences

Indonesian44939 sentences

400 sentences

400 sentences

50000 sentences

12th Dec., 2016

Evaluation Process1.

Automatic Evaluation metrics• BLEU points• RIBES Scores• AMFM Scores

2.

Pairwise Crowdsourcing Evaluation• Against the shared task baseline

3.

JPO Adequacy Evaluation• For content transmission

WAT 2016

812th Dec., 2016

WAT 2016

9

English-Indonesian MT system12th Dec., 2016

WAT 2016

10Automatic Evaluation of English – Indonesian MT system
Approach Used

BLEU score

Phrase based SMT

21.74

Operation Sequence
Model

21.70

0.806182

22.120.804933

0.552821.64

0.806033

0.555

22.08

0.806697

0.551880.808943

0.55597

Neural LM with OE = 700
Neural LM with OE =750
Neural LM with OE = 800
Joint neural LM*

22.35

RIBES score

AMFM score0.804986

0.55095
0.552480• Increase in BLEU score with NNJM by 0.61 points over PBSMT system* WAT Submission, OE: Output Embedding

12th Dec., 2016

WAT 2016

11Pairwise Crowdsourcing Analysis of EI system(1/2)
Crowdsourcing Evaluation method—• 5 Evaluators scored the sentence translations against the shared taskbaseline translation as :
Ø Better than baseline :1
Ø Tie with baseline :0Ø Worse than baseline : -1
• All 5 scores were added and converted to :
Ø 1 if >= 2
Ø -1 if <= -2
Ø 0 if between 2 & -2

12th Dec., 2016

WAT 2016

12Pairwise Crowdsourcing Analysis of EI system(2/2)• Scores received from pairwise evaluations
Experiment
EnglishIndonesianApproach
FollowedBetter thanBaseline

Comparable
to Baseline

Worse thanBaseline

Scores

NNJM

23%44.75%

32.25%-9.0250• Observations•

For worse sentences, sentence length is found to be >= 25 words.

•Words not getting translated is the most visible error.

12th Dec., 2016

WAT 2016

13

JPO Adequacy Scores of EI system• Adequacy evaluation method –Ø 2 Annotators evaluated 200 translations for adequacy scores from 1 – 5
Ø Frequency of each score is used to compare.

• Scores :Experiment
EnglishIndonesianApproach
Followed
NNJM

Adequacy distribution
543217.75%

25.25%

23.25%

16.5%

117.25%Adequacy
Score
3.10

12th Dec., 2016

WAT 2016

14

Summary of all evaluations for EI system (NNJM)• Our systems adequacy scores suggests that the sentences areable to convey the meaning well.

12th Dec., 2016

WAT 2016

Indonesian-English MT system15

12th Dec., 2016

WAT 2016

16

Results for Indonesian – English MT system
Approach Used

BLEU score

Phrase based SMT

22.03

Operation Sequence
Model*
Neural LM with
OE= 700
Neural LM with OE= 750
Neural LM with OE= 800
Joint Neural LM

22.24
22.58
21.99
22.1522.05

RIBES score

AMFM score0.78032

0.564580

0.781430

0.566950

0.781983

0.569330

0.780901

0.563400.782302

0.566470

0.781268

0.565860• Increase in BLEU score with NPLM by 0.55 points over PBSMT system* WAT Submission, OE: Output Embedding

12th Dec., 2016

WAT 2016

17

Pairwise Crowdsourcing Analysis of IE system• Scores of crowdsourcing evaluation(refer to slide-11 for evaluation method)
Experiment
IndonesianEnglish

Approach
Followed

Better thanBaseline

Comparable toBaseline

Worse thanBaseline

ScoresOSMapproach

20%

34%

46%

-26.00• Observations
Ø For worse sentences, Sentence length is found to be >= 25 words

12th Dec., 2016

WAT 2016

18JPO Adequacy Scores of IE system• Scores (refer to slide-13 for evaluation method ):Experiment
IndonesianEnglish

Approach
Followed
OSM
approach

Adequacy distribution
512%

432

18.75%

31.75%

30.5%

17%Adequacy
Score

2.98

• Observation:-From adequacy distribution, it can be observed that > 50% of translations
are adequate enough to convey the meaning.

12th Dec., 2016

WAT 201619

Summary of all evaluations for Indonesian-English system(OSM)• Our systems scores with OSM approach are not very promisingagainst the baseline system.

12th Dec., 2016

WAT 2016

20

Output Analysis of Indonesian-English System
Reference Sentence

Translated Sentence

Moreover, syariah banking has yet In addition, the banking industry
to become a national agenda, had not so national agenda, said
Riawan said.
Riawan who also director of themain BMI.
Of course, we will adhere to the We will certainly patuhi regulations,rules, Bimo said.
Bimo said.
The Indonesian government last The government has cancel foreign
year canceled 11 foreign-funded loans from various creditors to 11
projects across the country for projects in 2006 because various
various reasons, the Finance reasons.
Ministry said.
As the second largest Islamic bank As the second largest bank of thewith a 29% market share of the market by 29 percent of the total
Islamic banking industry's total assets syariah banking loans at theassets at end-2007 albeit only 0.5% end of December 2007 although theof overall banking industry's total market only 0.5 percent of the total
assets, net financing margin NFM assets banking industry as a whole,on Muamalat's financing operations financing profit margin Muamalatincreased to 7.9% in 2007 from rose to 7.9 percent in 2007 from 6.4
6.4% in 2004 due to better funding percent in 2004 thanks to fundingstructure.
structure.

* Text in blue represents error

Error Analysis
Phrase insertion

All words not translated
Phrase dropped

Phrase dropped

12th Dec., 2016

WAT 2016

21

Observations by Language Experts
Output analysis of Indonesian-English system• The Sentences were adequate and fluent to some extent.
• The major error was of dropping and insertion of phrases.
• Some Indonesian words could not be translated to English

due to lack of vocabulary learnt.
ØThough OOV word percentage was found to be only 5% of thetotal words in the test set.
• Error in choice of function words used for English language.
ØRequire some linguistic insight on the Indonesian side ofthelanguage to understand the usage of function words in the sourcelanguage.

12th Dec., 2016

WAT 2016

22Conclusion• Due to structural similarity, translation outputs are adequate tounderstand.
• Integrating Neural Probabilistic LM (NPLM) with additional

data as a feature in PBSMT system improves the translationquality.
• Integrating Neural Network Joint Model (Bilingual LM) trainedon parallel data as a feature in PBSMT system improvestranslation quality.

12th Dec., 2016

WAT 2016

23

Future Work
• Investigate the hyperparameters for the neural language model.
• Experiment with pure neural MT system for English-Indonesianlanguage pair.

12th Dec., 2016

WAT 2016

24References• Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. "Neural machine translation byjointly learning to align and translate." In ICLR.
• Devlin, Jacob, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard M. Schwartz, and John
Makhoul. 2014. "Fast and Robust Neural Network Joint Models for Statistical Machine Translation."
In conference of the Association of Computational Linguistics.
• Durrani, Nadir, Helmut Schmid, and Alexander Fraser. 2011. "A joint sequence translation model withintegrated reordering." Proceedings of the 49th Annual Meeting of the Association for ComputationalLinguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics.
• Durrani, Nadir, Alexander M. Fraser, and Helmut Schmid. 2013. "Model With Minimal Translation
Units, But Decode With Phrases." HLT-NAACL.
• Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola
Bertoldi, Brooke Cowan. 2007. "Moses: Open source toolkit for statistical machine translation."
In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration
sessions. Association for Computational Linguistics.
• Nakazawa, Toshiaki and Mino, Hideya and Ding, Chenchen and Goto, Isao and Neubig, Graham andKurohashi, Sadao and Sumita, Eiichiro. 2016. “Overview of the 3rd Workshop on Asian Translation.”
Proceedings of the 3rd Workshop on Asian Translation (WAT2016), October.
• Niehues, Jan, Teresa Herrmann, Stephan Vogel, and Alex Waibel. 2011. "Wider context by usingbilingual language models in machine translation." InProceedings of the Sixth Workshop on
Statistical Machine Translation. Association for Computational Linguistics.
• Vaswani, Ashish, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. "Decoding with LargeScale Neural Language Models Improves Translation." In EMNLP.

12th Dec., 2016

WAT 2016

Thank You!

25