Question: What conclusions can be drawn from the experiments?
Evidence: Tables TABREF15 and TABREF16 show the translation accuracy in BLEU scores, the INLINEFORM0 -value of the significance test by bootstrap resampling BIBREF4 and training time in hours until convergence. The encoder-decoder-reconstructor BIBREF3 requires slightly longer time to train than the baseline NMT, but we emphasize that decoding time remains the same with the encoder-decoder-reconstructor and baseline-NMT. The results show that the encoder-decoder-reconstructor BIBREF3 significantly improves translation accuracy by 1.01 points on ASPEC and 1.37 points on NTCIR in English-Japanese translation ( INLINEFORM1 ). However, it does not significantly improve translation accuracy in Japanese-English translation. In addition, it is proved that the encoder-decoder-reconstructor without pre-training worsens rather than improves translation accuracy.
Table TABREF24 shows examples of outputs of English-Japanese translations. In Example 1, “UTF8min乱 流 粘性 と 数値 粘性 の 大小 関係 により ,” (on the basis of the relation between turbulent viscosity and numerical viscosity in size) is missing in the output of baseline-NMT, but “UTF8min乱 流 粘性 と 数値 的 粘性 の 関係 を 基 に” (on the basis of the relation between turbulent viscosity and numerical viscosity) is present in the output of encoder-decoder-reconstructor. In Example 2, “UTF8min新生児” (newborn infant) and “UTF8min30歳以上の” (of 30 ‐ year ‐ old or more) are repeated in the output of baseline-NMT, but they appear only once in the output of encoder-decoder-reconstructor.
In addition, Figures FIGREF25 and FIGREF25 show the attention layer on baseline-NMT and encoder-decoder-reconstructor in each example. In Figure FIGREF25 , although the attention layer of baseline NMT attends input word “turbulent”, the decoder does not output “UTF8min乱流” (turbulent) but “UTF8min検討” (examined) at the 13th word. Thus, under-translation may be resulted from the hidden layer or the embedding layer instead of the attention layer. In Figure FIGREF25 , it is found that the attention layer of baseline-NMT repeatedly attends input words “newborn infant” and “30 ‐ year ‐ old or more”. Consequently, the decoder repeatedly outputs “UTF8min新生児” (newborn infant) and “UTF8min30歳以上の” (of 30 ‐ year ‐ old or more). On the other hand, the attention layer of encoder-decoder-reconstructor almost correctly attends input words.
Table TABREF28 shows a comparison of the number of word occurrences for each corpus and model. The columns show (i) the number of words that appear more frequently than the counterparts in the reference, and (ii) the number of words that appear more than once but are not included in the reference. Note that these numbers do not include unknown words, so (iii) shows the number of unknown words. In all the cases, the number of occurrence of redundant words is reduced in encoder-decoder-reconstructor. Thus, we confirmed that encoder-decoder-reconstructor achieves reduction of repeating and missing words while maintaining the quality of translation.
The results show that the encoder-decoder-reconstructor significantly improves translation accuracy by 1.01 points on ASPEC and 1.37 points on NTCIR in English-Japanese translation (p < 0.05). However, it does not significantly improve translation accuracy in Japanese-English translation. In addition, it is proved that the encoder-decoderreconstructor without pre-training worsens rather than improves translation accuracy.
Answer: The results show that the encoder-decoder-reconstructor significantly improves translation accuracy by 1.01 points on ASPEC and 1.37 points on NTCIR in English-Japanese translation (p < 0.05). However, it does not significantly improve translation accuracy in Japanese-English translation. In addition, it is proved that the encoder-decoderreconstructor without pre-training worsens rather than improves translation accuracy.