Neural Reranking Helps Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham Neubig, Makoto Morishita, Satoshi Nakamura Nara Institute of Science and Technology (NAIST), Japan
Research Questions
1. Does reranking with neural MT models improve subjective impressions of translation results? 2. If so, what are the qualitative differences between reranked and non-reranked output? 3. How big of an n-best list do we need?
Neural Reranking
Input N-best w/MT Features he has a cold .彼は寒さを持っているt=-0.5 l=-5.6 l -6.1 2.彼は風邪を持っているt=-0.9 l=-5.8 | -6.7彼は風邪を引いたt=-1.5 l=-5.3 | -6.8彼は風邪があるt=-1.9 l=-5.4 l -7.3 nmt=-5.8 nmt=-5.5 nmt=-3.4 nmt=-5.2 Rescored/Reranked N-best 1.彼は風邪を引いたt=-1.5 l=-5.3 nmt=-3.4 | -9.2 2.彼は寒さを持っているt=-0.5 l=-5.6 nmt=-5.8 | -10.9 3.彼は風邪を持っているt=-0.9 l=-5.8 nmt=-5.5 | -11.2 4.彼は風邪があるt=-1.9 l=-5.4 nmt=-5.2 | -12.5 Experimental Setup • Data: ASPEC Scientific Abstracts – Japanese   English, Chinese • Baseline: NAIST WAT2014 Tree-to-String System – Strong baseline achieving high scores – Implemented using Travatar (http://phontron.com/travatar) • Neural MT Model: Attentional model – Trained ~500k sent., 256 hidden nodes, 2 model ensemble – Use words occurring 3+ times (vocab 50,000~80,000) – Trained w/ lamtram (http://github.com/neubig/lamtram) • Automatic Evaluation: BLEU, RIBES • Manual Evaluation: WAT 2015 HUMAN Score Quantitative Results ow: Neurai reranking helps BLEU. | Show what we didn't know: Also helps manual eval.
Reranking and N-best Size
• All data sets improve approximately log-linear • Little saturation even at 100-best
Qualitative Analysis

Particularly reordering, insertion, and conjugation errors Jetailed Analysis nt: Pnrasal Reorderıng (+26. -4) Rerank  In case 2. the tlare. induration, skin ulcer was produc provement: Auxılıary Verb Ins./Del. (+15, -0)これにより得られる支配方桿式は壁面乱流のようなせん断乱流にも有力でのる。 Ref Governing equation derived by this method is useful for turbulent shear flow like turbulent flow near wall. Base The governing equation is obtained by this is also useful for such as wall turbulence shear flow. Rerank  The governing equation obtained by this is also useful for shear flow such as wall turbulence. #3 Improvement: Coordinate Structures (+13, -2)レーザー加工は高密度光束による局所的な加熱とアブレーションSourceにより行う。 Ref Laser work is done by local heating and ablation with high density light flux. The laser processing is carried out by local heating by high- density luminous flux and ablation. Rerank  The laser processing is carried out by local heating and ablation by high-density flux. #4 Improvement: Verb Agreement (+6, -0) Sourceラングミュア-ブロジェット法や包接化にも触れた。 Ref Langmuir-Blodgett method and inclusion compounds are mentioned. Base Langmuir-Blodgett method and inclusion is also discussed. Rerank  Langmuir-Blodgett method and inclusion are also mentioned. | What Didn't Work: Terminology (+2, -4) Source 放射熱を利用する赤外線応用計測が応力解析に役立っている。 Ref Base Infrared ray applied measurement using radiant heat is useful for stress analysis. The infrared application measurement using radiant heat is useful in the stress analysis. Rerank  Infrared ray application measurement using radiation heat is useful for stress analysis. Conclusion • Reranking with neural MT models leads to subjective improvements in MT quality • Future work includes comparisons with neural language models or neural MT w/o reranking Neural Features Base