Neural Reranking Helps Subjective Quality of Machine Translation: NAIST at WAT 2015
Graham Neubig, Makoto Morishita, Satoshi Nakamura
Nara Institute of Science and Technology (NAIST), Japan

T2S40+1.5+2.830
20100en-jaja-en

Neural Features

1. 彼は寒さを持っている t=-0.5 l=-5.6 | -6.1
2. 彼は風邪を持っている t=-0.9 l=-5.8 | -6.7
3. 彼は風邪を引いた
t=-1.5 l=-5.3 | -6.8
4. 彼は風邪がある
t=-1.9 l=-5.4 | -7.3

NeuralModel

HUMAN

he hasa cold+1.6zh-ja868482807876747270

ja-zh+1.4#1 Improvement: Phrasal Reordering (+26, -4)+1.8+1.8+2.7

en-ja

ja-en

BaseRerankzh-ja

ja-zhConfirm what we know: Neural reranking helps BLEU.

Neural Reranking
N-best w/MT Features

+2.5

RIBES

1. Does reranking with neural MT models improve
subjective impressions of translation results?
2. If so, what are the qualitative differences
between reranked and non-reranked output?
3. How big of an n-best list do we need?

50BLEU

Research Questions

Input

Detailed Analysis

Quantitative Results

nmt=-5.8
nmt=-5.5
nmt=-3.4nmt=-5.2

RerankingRescored/Reranked N-best

70
60
50
4030
20
100+12.5+23.7 +10.0+4.2
en-ja

ja-enzh-ja

BaseRerankja-zh

Show what we didn't know: Also helps manual eval.

Reranking and N-best Size

1. 彼は風邪を引いた
t=-1.5 l=-5.3 nmt=-3.4 | -9.2
2. 彼は寒さを持っている t=-0.5 l=-5.6 nmt=-5.8 | -10.9
3. 彼は風邪を持っている t=-0.9 l=-5.8 nmt=-5.5 | -11.2
4. 彼は風邪がある
t=-1.9 l=-5.4 nmt=-5.2 | -12.5●Baseline: NAIST WAT2014 Tree-to-String System– Strong baseline achieving high scores– Implemented using Travatar(http://phontron.com/travatar)●Neural MT Model: Attentional model– Trained ~500k sent., 256 hidden nodes, 2 model

ensemble– Use words occurring 3+ times (vocab50,000~80,000)– Trained w/ lamtram(http://github.com/neubig/lamtram)

Automatic Evaluation: BLEU, RIBES● Manual Evaluation: WAT 2015 HUMAN Score●Source これにより得られる支配方程式は壁面乱流のようなせん断乱流にも有用である。
Ref
Governing equation derived by this method is useful forturbulent shear flow like turbulent flow near wall.
BaseThe governing equation is obtained by this is also useful forsuch as wall turbulence shear flow.
Rerank The governing equation obtained by this is also useful for shear
flow such as wall turbulence.

#3 Improvement: Coordinate Structures (+13, -2)

Source レーザー加工は高密度光束による局所的な加熱とアブレーションにより行う。
Ref
Laser work is done by local heating and ablation with highdensity light flux.
Base
The laser processing is carried out by local heating by highdensity luminous flux and ablation.
Rerank The laser processing is carried out by local heating and ablation
by high-density flux.
Source ラングミュア‐ブロジェット法や包接化にも触れた。
Ref
Langmuir-Blodgett method and inclusion compounds arementioned.
Base
Langmuir-Blodgett method and inclusion is also discussed.

Data: ASPEC Scientific Abstracts– Japanese ↔ English, Chinese#2 Improvement: Auxiliary Verb Ins./Del. (+15, -0)#4 Improvement: Verb Agreement (+6, -0)

Experimental Setup●Source 症例2においては、直腸がんの肝転移に対する化学療法中に、発赤、硬結、皮膚潰ようを生じた。
Ref
In case 2, reddening, induration, and skin ulcer appeared duringchemical therapy for liver metastasis of rectal cancer.
BaseIn case 2, occurred during chemotherapy for liver metastasis ofrectal cancer, flare, induration, skin ulcer.
Rerank In case 2, the flare, induration, skin ulcer was produced during
the chemotherapy for hepatic metastasis of rectal cancer.

All data sets improve approximately log-linear● Little saturation even at 100-best●

Qualitative Analysis
Type
Reordering
Deletion
Insertion
Substitution
Conjugation
Total

Improved Degraded % Impr.
55
986%
20
1067%
19290%
15
1158%
8189%
117
3378%

Overall improvements re-confirmed
Particularly reordering, insertion, and conjugation errors

Rerank Langmuir-Blodgett method and inclusion are also mentioned.

What Didn't Work: Terminology (+2, -4)

Source 放射熱を利用する赤外線応用計測が応力解析に役立っている。
RefInfrared ray applied measurement using radiant heat is usefulfor stress analysis.
BaseThe infrared application measurement using radiant heat isuseful in the stress analysis.
Rerank Infrared ray application measurement using radiation heat isuseful for stress analysis.

Conclusion
Reranking with neural MT models leads tosubjective improvements in MT quality● Future work includes comparisons with neurallanguage models or neural MT w/o reranking●