Neural Reranking Improves Subjective Quality of Machine Translation

Neural Reranking Improves Subjective
Quality of Machine Translation:
NAIST at WAT 2015

Graham Neubig, Makoto Morishita, ○Satoshi Nakamura
Nara Institute of Science and Technology (NAIST)
2015-10-16

1

Neural Reranking Improves Subjective Quality of Machine Translation

Statistical Translation Frameworks
Symbolic Models
Phrase-based MT [Koehn+ 03]
hehas

a

Continuous-space (Neural) Models
Encoder-Decoder [Sutskever+ 14]
he hascold

acold 
 彼は 風邪 を引いている

hehasa cold
彼 は 引いている 風邪 を
hea coldhas
彼 は 風邪 を 引いている
彼 は 風邪 を 引いている

Tree-to-String MT [Liu+ 06]
S

彼は 風邪 を 引いて
いるAttentional [Bahdanau+ 15]
he hasacold

VPNPPRPVBZDETNN

hehasacold

彼 は

g1,...,g4
a1

a2
hi-1

a3

a4
hi

風邪 を 引いている
ri-1

P(ei|F,e1,...,ei-12)

Neural Reranking Improves Subjective Quality of Machine Translation

Relative Merits/Demerits●Symbolic Models
Inner workings well understood✔
Better at translating low-frequency words✔

●Continuous-space Models
Easier to implement✔
Produce more fluent output✔
Probabilistic model – can score output of other systems!
✔

3

Neural Reranking Improves Subjective Quality of Machine Translation

Reranking with Neural MT Models
Input
he hasa cold

T2S/PBMT

N-best w/MT Features

Neural Features

1. 彼は寒さを持っている t=-0.5 l=-5.6 | -6.1

nmt=-5.8

2. 彼は風邪を持っている t=-0.9 l=-5.8 | -6.7

nmt=-5.5

t=-1.5 l=-5.3 | -6.8

nmt=-3.4

t=-1.9 l=-5.4 | -7.3

nmt=-5.2

3. 彼は風邪を引いた
4. 彼は風邪がある

NeuralModelReranking

Rescored/Reranked N-best
1. 彼は風邪を引いた

t=-1.5 l=-5.3 nmt=-3.4 | -9.2

2. 彼は寒さを持っている t=-0.5 l=-5.6 nmt=-5.8 | -10.9
3. 彼は風邪を持っている t=-0.9 l=-5.8 nmt=-5.5 | -11.2
4. 彼は風邪がある

t=-1.9 l=-5.4 nmt=-5.2 | -12.5

4

Neural Reranking Improves Subjective Quality of Machine Translation

What Do We Know About Reranking?
●Reranking greatly improves BLEU score, even overstrong baseline systems:
Sutskever+ 2014

en-fr
BLEU
Base
33.3
Rerank36.5

Alkhouli+ 2015

Baseline
Rerankedde-en ar-en
BLEU BLEU
30.6 26.4
32.3 27.05

Neural Reranking Improves Subjective Quality of Machine Translation

What Don't We Know About Reranking?
●●Does reranking improve subjective impressions ofresults?
What are the qualitative differences before/afterreranking with neural MT models?

6

Neural Reranking Improves Subjective Quality of Machine Translation

Experiments

7

Neural Reranking Improves Subjective Quality of Machine Translation

Experimental Setup●

Data: ASPEC Scientific Abstracts●●Japanese ↔ English, Chinese

Baseline: NAIST WAT2014 Tree-to-String SystemStrong baseline achieving high scores● Implemented using Travatar (http://phontron.com/travatar)●●

Neural MT Model: Attentional model●Trained ~500k sent., 256 hidden nodes, 2 model ensemble

Use words occurring 3+ times (vocab 50,000~80,000)● Trained w/ lamtram (http://github.com/neubig/lamtram)●●●

Automatic Evaluation: BLEU, RIBESManual Evaluation: WAT 2015 HUMAN Score

8Neural Reranking Improves Subjective Quality of Machine Translation

40
BLEU

+2.5+1.6+1.5+2.8

30

90

20
100en-ja

ja-enzh-ja

ja-zh+1.4+1.885
RIBES50Results80+1.8+2.77570

BaseReranken-ja ja-en zh-ja ja-zh

HUMANConfirm what we know: Neural reranking helps automatic evaluation.
7060
5040
30
20
100+12.5+23.7 +10.0+4.2
en-ja ja-en zh-ja ja-zh

BaseRerank

9Show what we didn't know: Also help manual evaluation.

Neural Reranking Improves Subjective Quality of Machine Translation

What is Getting Better?
●Perform detailed categorization of the changes inJapanese-English results:
1. Is the sentence better/worse after ranking?
2. What is the main error corrected: insertion, deletion,substitution, reordering, or conjugation?
3. What is the detailed subcategory?

10

Neural Reranking Improves Subjective Quality of Machine Translation

Main Types of Errors Corrected/Caused
TypeReordering
Deletion
Insertion
Substitution
Conjugation
Total

Improved
55
20
19158117

Degraded% Impr.
9102
11
1
33

86%
67%
90%
58%
89%
78%

Overall improvements re-confirmed
In particular fixing reordering, insertion, andconjugation errors
11Neural Reranking Improves Subjective Quality of Machine Translation#1 Detailed Improvement Category:Phrasal Reordering (+26, -4)Source
症例2においては、直腸がんの肝転移に対する化学療法中に、発赤、硬結、皮膚潰ようを生じた。

Ref
In case 2, reddening, induration, and skin ulcer appeared duringchemical therapy for liver metastasis of rectal cancer.

Base
In case 2, occurred during chemotherapy for liver metastasis ofrectal cancer, flare, induration, skin ulcer.

RerankIn case 2, the flare, induration, skin ulcer was produced during the12chemotherapy for hepatic metastasis of rectal cancer.

Neural Reranking Improves Subjective Quality of Machine Translation#2 Detailed Improvement Category:
Auxiliary Verb Ins./Del. (+15, -0)
Source
これにより得られる支配方程式は壁面乱流のようなせん断乱流にも有用である。

RefGoverning equation derived by this method is useful for turbulent
shear flow like turbulent flow near wall.

BaseThe governing equation is obtained by this is also useful for suchas wall turbulence shear flow.

RerankThe governing equation obtained by this is also useful for shear
13
flow such as wall turbulence.

Neural Reranking Improves Subjective Quality of Machine Translation#3 Detailed Improvement Category:
Coordinate Structures (+13, -2)
Sourceレーザー加工は高密度光束による局所的な加熱とアブレーションにより行う。

RefLaser work is done by local heating and ablation with high density
light flux.

Base
The laser processing is carried out by local heating by highdensity luminous flux and ablation.

RerankThe laser processing is carried out by local heating and ablation
14
by high-density flux.

Neural Reranking Improves Subjective Quality of Machine Translation#4 Detailed Improvement Category:
Verb Agreement (+6, 0)
Sourceラングミュア‐ブロジェット法や包接化にも触れた。

Ref
Langmuir-Blodgett method and inclusion compounds arementioned.

Base
Langmuir-Blodgett method and inclusion is also discussed.

RerankLangmuir-Blodgett method and inclusion are also mentioned.
15

Neural Reranking Improves Subjective Quality of Machine Translation

What Wasn't Helped:
Terminology (+2, -4)
Source
放射熱を利用する赤外線応用計測が応力解析に役立っている

Ref
Infrared ray applied measurement using radiant heat is useful forstress analysis.

Base
The infrared application measurement using radiant heat is usefulin the stress analysis.

RerankInfrared ray application measurement using radiation heat isuseful for stress analysis.

16Neural Reranking Improves Subjective Quality of Machine Translation

Conclusion

17

Neural Reranking Improves Subjective Quality of Machine Translation

What Do We Know Now?
●

●Neural reranking improves subjective quality ofmachine translation output.
Main gains are from grammatical factors, and not
lexical selection.

18

Neural Reranking Improves Subjective Quality of Machine Translation

What Do We Still Not Know Yet?
●●

How do neural translation models compare with neurallanguage models?
How does reranking compare with pure neural MT?

19

Neural Reranking Improves Subjective Quality of Machine Translation

Thank You!

20