SlowGuess/ACL_2020_OCR huggingface dataset:
We first compare our model, i.e., IRN + KNN with all those strong baselines metioned above. Figure 2 shows that the proposed model significantly outperforms previous baselines on both BLEU score and ERR. Compared with current state-of-the-art model, RALSTM, it achieves reductions of 1.45, 1.38, 1.45 and 1.80 times for SF Restaurant, SF Hotel, Laptop, and Television datasets, respectively. Furthermore, it improves  $3.59\%$ ,  $1.45\%$ ,  $2.29\%$  and  $3.33\%$  of BLEU scores on these datasets, respectively. This improvements of BLEU score can be contributed from language fluency reward  $r^{\mathrm{LM}}$ .
To verify whether IRN helps improve slot consistency of general NLG models, we further equip strong baselines, including HLSTM, TGen and RALSTM, with IRN. We evaluate their performances on SF Restaurant and Television datasets. As shown in Table 3, the methods consistently reduce ERRs and also improve BLEU scores for all
Table 4: Ablation study of rewards (upper part) and training data algorithms (lower part).
Model	Television
Informative	Natural
TGen	4.49	3.41
TGen + IRN	4.72	3.52
RALSTM	4.63	4.01
RALSTM + IRN	4.86	4.07
Table 5: Real user trial for generation quality evaluation on both informativeness and naturalness.
basielines on both datasets.
In conclusion, our model, IRN (+ KNN), not only has achieved the state-of-the-art performances but also can contribute to improvements of slot consistency for general NLG systems.
We perform a set of ablation experiments on the SCLSTM+IRN models on Laptop dataset to understand the relative contribution of data aggregation algorithms in Sec. 3.2 and rewards in Sec. 5.1.
6.3.1 Effect of Reward Designs
The results in Table 4 show that removal of slot consistency reward  $r^{\mathrm{SC}}$  or distant supervision reward  $r^{\mathrm{DS}}$  from advantage function dramatically degrades SER performance. Language fluency related information from baseline  $BLEU$  and reward  $r^{\mathrm{LM}}$  also have positive impact on BLEU and SER, though they are smaller than using  $r^{\mathrm{SC}}$  or  $r^{\mathrm{DS}}$ .
6.3.2 Effect of Data Algorithms
Using only candidates from baselines degrades performance to approximately that of the baseline SCLSTM. This shows that incorporating candidates from IRN is important. The model without bootstrapping, even including candidates from IRN, has worse performance than SCLSTM in Table 3. This shows that bootstrapping to include generic samples from templates database is critical.
6.4 Human Evaluation
We evaluate IRN and some strong baselines on TV dataset. Given an input DAs, we ask human eval
$^{1}$ https://github.com/shawnwun/RNNLG.
$^{2}$ https://github.com/UFAL-DSG/tgen.
Input DA	recommend(NAME = crios 93, FAMILY = l1, AUDIO = nicam stereo, SIZE = large)
Reference Text	the large crios 93 television in the l1 family features nicam stereo
Mistaken Generation	the $NAME$ is in $FAMILY$ with $SIZE$ screen and cost about $PRICE$ [AUDIO, PRICE]
1-st IRN Revision	the $NAME$ is a nice television in $FAMILY$ with a $SIZE$ screen [AUDIO]
2-st IRN Revision	the $NAME$ is very nice in $FAMILY$ with a $SIZE$ screen size [AUDIO]
3-st IRN Revision	the $NAME$ is very nice in the $FAMILY$ family with a $SIZE$ screen size and $AUDIO$
Lexicalized Form	the crios 93 is very nice in the l1 family with a large screen size and nicam stereo
Table 6: A DA from Television dataset and a candidate from HLSTM on the DA. The output template from each iteration of IRN. Slot errors are marked in colors (missing, misplaced).
uator to score generated surface realizations from our model and other baselines in terms of informativeness and naturalness. Here informativeness measures whether output utterance contains all the information specified in the DA without insertion of extra slots or missing an input slot. The naturalness is defined as whether it mimics a response from a human (both ratings are out of 5).
Table 5 shows that RALSTM + IRN outperforms RALSTM notably in informativeness relatively by  $4.97\%$ , from 4.63 to 4.86. In terms of naturalness, the improvement is from 4.01 to 4.07, relative by  $1.50\%$ . Meanwhile, IRN helps to improve the performances of TGen by  $5.12\%$  on informativeness and  $3.23\%$  on naturalness.
These subjective assessments are consistent to the observations in Table 3, which both have verified the effectiveness of proposed method.
6.5 Case Study
Table 6 presents a sample on TV dataset and shows a progress made by IRN. Given an input DA, the baseline HLSTM outputs in the third row a template that misses slot $AUDIO$ but inserts slot $PRICE$. The output template from the first iteration of IRN has a removal of the inserted $PRICE$ slot. The second iteration has improved language fluency but no progress in slot-inconsistency. The third iteration achieves slot consistency, after which a natural language, though slightly different from the reference text, is generated via lexicalization.
Conventional approaches for solving NLG task are mostly pipeline-based, dividing it into sentence planning and surface realisation (Dethlefs et al., 2013; Stent et al., 2004; Walker et al., 2002). Oh and Rudnicky (2000) introduce a class-based n-gram language model and a rule-based reranker. Ratnaparkhi (2002) address the limitations of n
gram language models by using more sophisticated syntactic dependency trees. Mairesse and Young (2014) employ a phrase-based generator that learn from a semantically aligned corpus. Despite their robustness, these models are costly to create and maintain as they heavily rely on handcrafted rules.
Recent works (Wen et al., 2015b; Dusek and Jurcicek, 2016; Tran and Nguyen, 2017a) build data-driven models based on end-to-end learning. Wen et al. (2015a) combine two recurrent neural network (RNN) based models with a CNN reranker to generate required utterances. Wen et al. (2015b) introduce a novel SC-LSTM with an additional reading cell to jointly learn gating mechanism and language model. Dusek and Jurcicek (2016) present an attentive neural generator to apply attention mechanism over input DA. Tran and Nguyen (2017b,a) employ a refiner component to select and aggregate the semantic elements produced by the encoder. More recently, domain adaptation (Wen et al., 2016) and unsupervised learning (Bahuleyan et al., 2018) for NLG also receive much attention.
We are also inspired by the post-edit paradigm (Xia et al., 2017), which uses a second-pass decoder to improve the translation quality.
A recent method in (Wu et al., 2019) defines an auxiliary loss that checks if the object words exist in the expected system response of a task-oriented dialogue system. It would be interesting to apply this auxiliary loss in the proposed method. On the other hand, the REINFORCE (Williams, 1992) algorithm applied in this paper is more general than (Wu et al., 2019) to incorporate other metrics, such as BLEU.
Nevertheless, end-to-end neural-based generators suffer from hallucination problem and are hard to avoid generating slot-inconsistent utterance (Balakrishnan et al., 2019). Balakrishnan et al. (2019) attempts to alleviate this issue by employing a tree-structured meaning representation and constrained
decoding technique. However, the tree-shaped structure requires additional human annotation.
We have proposed Iterative Rectification Network (IRN) to improve slot consistency of general NLG systems. In this method, a retrieval-based bootstrapping is introduced to sample pseudo mistaken cases from training corpus to enrich the original training data. We also employ policy-based reinforcement learning to enable training the models with discrete rewards that are consistent to evaluation metrics. Extensive experiments show that the proposed model significantly outperforms previous methods. These improvements include both of correctness measured with slot error rates and naturalness measured with BLEU scores. Human evaluation and case study also confirm the effectiveness of the proposed method.
This work was supported by the National Natural Science Foundation of China (NSFC) via grant 61976072, 61632011 and 61772153. This work was done while the first author did internship at Ant Financial. We thank anonymous reviewers for valuable suggestions.
Hareesh Bahuleyan, Lili Mou, Kartik Vamaraju, Hao Zhou, and Olga Vechtomova. 2018. Probabilistic natural language generation with Wasserstein autoencoders. arXiv preprint arXiv:1806.08462.
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. Constrained decoding for neural NLG from compositional representations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. To appear.
Nina Dethlefs, Helen Hastie, Heriberto Cuayáhuitl, and Oliver Lemon. 2013. Conditional random fields for responsive surface realisation using global features. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1254-1263, Sofia, Bulgaria. Association for Computational Linguistics.
Ondrej Dusek and Filip Jurcicek. 2016. Sequenceto-sequence generation for spoken dialogue via deep syntax trees and strings. arXiv preprint arXiv:1606.05491.
Juraj Juraska, Panagiotis Karagiannis, Kevin K. Bowden, and Marilyn A. Walker. 2018. A deep ensemble model with slot alignment for sequence-to-sequence natural language generation. In Proceedings of NAACL-HLT, pages 152-162.
Ioannis Konstas and Mirella Lapata. 2013. A global model for concept-to-text generation. Journal of Artificial Intelligence Research, 48:305-346.
François Mairesse and Steve Young. 2014. Stochastic language generation in dialogue using factored language models. Computational Linguistics, 40(4):763-799.
Feng Nie, Jin-Ge Yao, Jinpeng Wang, Rong Pan1, and Chin-Yew Lin. 2019. A simple recipe towards reducing hallucination in neural surface realisation. In Proceedings of the 57nd Annual Meeting of the Association for Computational Linguistics (ACL-19), page 2673–2679, Florence, Italy.
Alice H Oh and Alexander I Rudnicky. 2000. Stochastic language generation for spoken dialogue systems. In ANLP-NAACL 2000 Workshop: Conversational Systems.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.
Adwait Ratnaparkhi. 2002. Trainable approaches to surface natural language generation and their application to conversational dialog systems. Computer Speech & Language, 16(3-4):435-455.
Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentations in spoken dialog systems. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 79-86, Barcelona, Spain.
Van-Khanh Tran and Le-Minh Nguyen. 2017a. Natural language generation for spoken dialogue system using rnn encoder-decoder networks. arXiv preprint arXiv:1706.00139.
Van-Khanh Tran and Le-Minh Nguyen. 2017b. Neural-based natural language generation in dialogue using rnn encoder-decoder with semantic aggregation. arXiv preprint arXiv:1706.06714.
Marilyn A Walker, Owen C Rambow, and Monica Rogati. 2002. Training a sentence planner for spoken dialogue using boosting. Computer Speech & Language, 16(3-4):409-433.
Lex Weaver and Nigel Tao. 2001. The optimal reward baseline for gradient-based reinforcement learning. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pages 538-545. Morgan Kaufmann Publishers Inc.
Tsung-Hsien Wen, Milica Gasic, Dongho Kim, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015a. Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking. arXiv preprint arXiv:1508.01755.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. arXiv preprint arXiv:1603.01232.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, PeiHao Su, David Vandyke, and Steve Young. 2015b. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256.
Yuk Wah Wong and Raymond Mooney. 2007. Generation by inverting a semantic parser that uses statistical machine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 172-179.
Chien-Sheng Wu, Richard Socher, and Caiming Xiong. 2019. Global-to-local memory pointer networks for task-oriented dialogue. In International Conference on Learning Representations.
Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In Advances in Neural Information Processing Systems, pages 1784-1794.