SlowGuess/ACL_2020_OCR huggingface dataset:
Table 4: Character-level BLEU-4 scores  $(\%)$  on ASPEC  $\mathrm{Ja}\rightarrow \mathrm{Zh}$  task. The result of our model is statistically significant compared to the other baselines  $(p < 0.01)$ .
belines and outperforms the Transformer baseline by 1.14 BLEU point on average, which shows that the template could effectively improve the performance. More specifically, our model outperforms the Transformer model by 0.76 BLEU on NIST2003, 1.52 BLEU on NIST 2005, 0.91 BLEU on NIST 2008, and 1.39 BLEU on NIST 2012.
We further demonstrate the effectiveness of our model on WMT14 English-German translation tasks, and we also compare our model with other competitive models, including ABD-NMT (Zhang et al., 2018), Deliberation Network (Xia et al., 2017), SoftPrototype (Wang et al., 2019b), SB-NMT (Zhou et al., 2019a) and SBSG (Zhou et al., 2019b). As shown in Table 3, our model also significantly outperforms others and gets an improvement of 0.43 BLEU points than a strong Transformer model.
To investigate the effect of our approach on the different language pairs, we also evaluate
Figure 4: The effect of the multiple templates. We feed the top-K results of the beam search as multiple templates and source sentence to generate the target translation.
our model on the Japanese-Chinese translation task. According to Table 4, ST-NMT outperforms GNMT by 3.72 BLEU points, ConvS2S by 2.52 BLEU points, and the Transformer model by 0.82 BLEU points, which demonstrates that the soft template extracted by constituency-based parse tree can also bring strong positive effects.
3.6 Multiple Templates
Because of the diversity of the templates, we investigate the performance with the different numbers of the templates. On top of the original parallel training data  $D = \{(x^{(i)},y^{(i)})\}_{i = 1}^{N}$ , we construct the training data from the source text to the soft target template  $D_{X\rightarrow T} = \{(x^{(i)},t^{(i)})\}_{i = 1}^{N}$  by the model  $P_{\theta_{X\rightarrow T}}$ . Through this construction procedure, we could use the top-K results of the beam search as multiple templates by model  $P_{\theta_{X\rightarrow T}}$ . We could expand the training data of the source text to the target template as  $D_{X\rightarrow T} = \{(x^{(1)},t_{top_1}^{(1)}),\ldots ,(x^{(1)},t_{top_K}^{(1)}),\ldots ,(x^{(N)},t_{top_1}^{(N)}),\ldots ,(x^{(N)},t_{top_K}^{(N)})\}$ . As shown in Figure 4, our model gains the best performance only using the single template. When the number of templates is 8, our model gains the worst BLEU score of 29.22. We can summarize that our model can be more robust but maybe get worse performance with the number of templates rising. Besides, in order to further improve the stability of our model, we expand the dataset by selecting random templates for the source sentence. The different templates confuse our model, although it can make our model more robust.
Figure 5: We use two objective to update our model parameters. One objective only use source sentence and another utilizes source and soft template sentences together to generate the final translation. The hyperparameter  $\alpha$  is used to balance the two objectives,
3.7 Balance of Two Objectives
To further control how much our model leverages templates for translation, we tune the hyperparameter  $\alpha$ . With the value rising, the contribution of template information gradually decreases. We study the influence of the ratio  $\alpha$ . To investigate the effect of this hyperparameter, we set the discrete value  $\alpha = \{10\%, 20\%, 30\%, 40\%, 50\%, 60\%, 70\%, 80\%, 90\%, 100\}$ . According to Figure 5, when the  $\alpha$  switches from 0.4 to 0.9, our model can get the better performance which is greater than or equal to 29.3 BLEU. The results show that we can set the hyper-parameter  $\alpha$  in a reasonable interval  $(0.4 \leq \alpha \leq 0.9)$  to keep the balance between source text and template.
3.8 Depth of Parsing Tree
Considering that the template derived from the specific depth can lead to the divergent performance, our model is examined with the different depth. The effect of the template extraction which is described as Section 3 is decided by the sub-tree which is controlled by the depth of sub-tree. For the same constituency-based parse tree, the different sub-tree can be obtained based on the different chosen depth  $d$ . When we get the sub-tree, the template could be derived from it. The depth of the constituency-based parse tree is decided by a simple but effective strategy as formula:
$$
d = \min  \left(\max  \left(L \times \lambda , \gamma_ {1}\right), \gamma_ {2}\right) \tag {8}
$$
where  $L$  is the length of the input sentence,  $\gamma_{1}$  is the lower bound,  $\gamma_{2}$  is the upper bound depth
λ	MT03	MT05	MT08	MT12
0.10	45.92	45.01	36.55	35.34
0.15	46.56	46.04	37.53	35.99
0.20	46.02	45.20	37.08	35.82
0.25	46.27	44.83	36.88	35.64
0.30	46.08	45.02	36.72	35.54
0.35	46.22	44.92	36.84	35.51
0.40	46.32	45.40	36.94	35.61
Table 5: The results of the different depth on NIST2003, NIST2005, NIST2008 and NIST2012.
λ	MT03	MT05	MT08	MT12
0.15	79.4	81.6	78.6	77.6
Table 6: The ratio(%) of overlapping words between the predicted soft target template and the translation on NIST2003, NIST2005, NIST2008 and NIST2012.
of the sub-tree and  $\lambda$  is the ratio of the length of source sentence. When the  $\lambda$  approximates 1.0, the template contains more target tokens and less tags. In addition, we tune the depth on the LDC training data and list the results. According to the Table 5, the soft templates of the specific depth provide helpful information to the translation procedure when the  $\lambda = 0.15$  in the LDC dataset.
3.9 Ratio of Overlapping Words
To measure contribution of the predicted soft target template for final translation, we calculate the overlapping words between the template and the translation. Table 6 gives the specific overlapping words ratio on the different test sets including NIST2003, NIST2005, NIST2008 and NIST2012. The overlapping ratio is calculated by the following formula:
$$
\text {r a t i o} = \frac {\sum_ {w \in T} \min  \left(\operatorname {C o u n t} _ {y} (w) , \operatorname {C o u n t} _ {t} (w)\right)}{\sum_ {w \in T} \operatorname {C o u n t} _ {t} (w)} \tag {9}
$$
where  $\text{Count}_y(\cdot)$  and  $\text{Count}_t(\cdot)$  denote the number of  $w$  in the target translation  $Y$  and the template  $T$ , and  $w$  is the words in the target language. The overlapping ratio represents the correlation between the predicted template  $T$  and the target translation  $Y$ . According to Table 6, the correlation between the template  $T$  and the translation  $Y$  is highly relevant which demonstrates the contribution of our template to the final translation.
5986
Source	另一方面，如果我们反应过度，将会被他们欺骗。
Reference	on the other hand, if we overreact, we will be deceived by their trick.
Template	on the other hand, if NP VP, we will VP.
Ours	on the other hand, if we react too much, we will be hit by them.
Table 7: A Chinese-English translation example of our proposed method. VP and NP represent non-terminal nodes in the constituency-based parse tree.
3.10 Example Study
To further illustrate which aspects of NMT are improved by the target soft template, we provide a Chinese-English translation example shown in 7. Templates provide the structural and grammatical information of the target sentence. For instance, Chinese source sentence "另一方面，如果我们反应过度，将会被他们欺骗"，our model first predicts the target template "on the other hand, if NP VP, we will VP", and then generate the final translation "on the other hand, if we react too much, we will be hit by them". Our target template provides the sentence pattern "If sb. do sth, sb. will be done". Our method introduces the constituency-based parse tree and utilizes the constituency grammar to distinguish terminal and non-terminal nodes. Therefore, our model can automatically learn sentence patterns, including grammatical and structural information.
Many types of encoder-decoder architecture (Bahdanau et al., 2015; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018) have been proposed in the past few years. Furthermore, Transformer enhances the capability of NMT in capturing long-distance dependencies based on these backbone models, including CNN-based, RNN-based, and Transformer based architecture.
To improve the quality of the translation, many authors have endeavored to adopt multi-pass generation decoding method, their models first predict the rough translation and then generate the final translation based on the previous draft (Niehues et al., 2016; Chatterjee et al., 2016; Junczys-Dowmunt and Grundkiewicz, 2017; Xia et al., 2017; Geng et al., 2018; Wang et al., 2019b).
Besides, some works (Liu et al., 2016; Zhang et al., 2018; Zhou et al., 2019b,a) use the right-to-left (R2L) and left-to-right (L2R) to improve the quality of machine translation. Non-Autoregressive
decoding (Ghazvininejad et al., 2019) first predicts the target tokens and masked tokens, which will be filled in the next iterations. Then, the model predicts the unmasked tokens on top of the source text and a mixed translation consisting of the masked and unmasked tokens. Semi-autoregressive also (Akoury et al., 2019) predicts chunked fragments or the unmasked tokens based on the tree structure before the final translation. In addition, there are many existing works (Eriguchi et al., 2016; Aharoni and Goldberg, 2017; Wu et al., 2017; Wang et al., 2018; Dong and Lapata, 2018; Wang et al., 2018; Gu et al., 2018) which incorporate syntax information or the tree structure into NMT to improve the quality of translation results.
In this work, we propose a novel approach that utilizes source text and additional soft templates. More specifically, our approach can extract the templates from the sub-tree, which derives from the specific depth of the constituency-based parse tree. Then, we use a Transformer model to predict the soft target templates conditioned on the source text. On top of soft templates and source text, we incorporate the template information to guide the translation procedure. We compare our soft-template neural machine translation (ST-NMT) with other baselines on four benchmarks and multiple language pairs. Experimental results show that our ST-NMT significantly improves performance on these datasets.
This work was supported in part by the National Natural Science Foundation of China (Grant Nos.U1636211, 61672081,61370126), the Beijing Advanced Innovation Center for Imaging Technology (Grant No.BAICIT2016001), and the Fund of the State Key Laboratory of Software Development Environment (Grant No.SKLSDE2019ZX-17).
5987
Roee Aharoni and Yoav Goldberg. 2017. Towards string-to-tree neural machine translation. In ACL 2017, pages 132-140.
Nader Akoury, Kalpesh Krishna, and Mohit Iyyer. 2019. Syntactically supervised transformers for faster neural machine translation. In ACL 2019, pages 1269-1281.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015.
Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018. Retrieve, rerank and rewrite: Soft template based neural summarization. In ACL 2018, pages 152-161.
Rajen Chatterjee, José G. C. de Souza, Matteo Negri, and Marco Turchi. 2016. The FBK participation in the WMT 2016 automatic post-editing shared task. In WMT 2016, pages 745-750.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In ACL 2018, pages 76-86.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In ACL 2018, pages 731-742.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In EMNLP 2017, pages 866-874.
Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In ACL 2016.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In ICML 2017, pages 1243-1252.
Xinwei Geng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2018. Adaptive multi-pass decoder for neural machine translation. In EMNLP 2018, pages 523-532.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP 2019, pages 6111-6120.
Jetic Gu, Hassan S. Shavarani, and Anoop Sarkar. 2018. Top-down tree structured decoding with syntactic connections for neural machine translation and parsing. In EMNLP 2018, pages 401-413.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. TACL, 6:437-450.
Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2017. An exploration of neural sequence-to-sequence architectures for automatic post-editing. In IJCNLP 2017, pages 120-129.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR 2015.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL 2007, pages 177-180.
Lemao Liu, Masao Utiyama, Andrew M. Finch, and Eiichiro Sumita. 2016. Agreement on target-bidirectional neural machine translation. In *NAACL* 2016, pages 411-416.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In ACL 2014, pages 55-60.
Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016. Aspec: Asian scientific paper excerpt corpus. In LREC 2016.
Jan Niehues, Eunah Cho, Thanh-Le Ha, and Alex Waibel. 2016. Pre-translation for neural machine translation. In COLING 2016, pages 1828-1836.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In WMT 2018, pages 1-9.
Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In ACL 2018, pages 1329-1338.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL 2002, pages 311-318.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL 2016, pages 1715-1725.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. In ICML 2019, pages 5976-5985.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS 2017, pages 5998-6008.
5988
Kai Wang, Xiaojun Quan, and Rui Wang. 2019a. Biset: Bi-directional selective encoding with template for abstractive summarization. In ACL 2019, pages 2153-2162.
Xinyi Wang, Hieu Pham, Pengcheng Yin, and Graham Neubig. 2018. A tree-based decoder for neural machine translation. In EMNLP 2018, pages 4772-4777.
Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Cheng Xiang Zhai, and Tie-Yan Liu. 2019b. Neural machine translation with soft prototype. In NIPS 2019, pages 6313-6322.
Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2018. Learning neural templates for text generation. In EMNLP 2018, pages 3174-3187.
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In ICLR 2019.
Shuangzhi Wu, Dongdong Zhang, Nan Yang, Mu Li, and Ming Zhou. 2017. Sequence-to-dependency neural machine translation. In ACL 2017, pages 698-707.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In NIPS 2017, pages 1784-1794.
Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. 2018. Asynchronous bidirectional decoding for neural machine translation. In AAAI 2018, pages 5698-5705.
Long Zhou, Jiajun Zhang, and Chengqing Zong. 2019a. Synchronous bidirectional neural machine translation. TACL, 7:91-105.
Long Zhou, Jiajun Zhang, Chengqing Zong, and Heng Yu. 2019b. Sequence generation: From both sides to the middle. In *IJCAI* 2019, pages 5471-5477.