SlowGuess/ACL_2020_OCR huggingface dataset:
Yangming Li $^{1,2}$ , Kaisheng Yao $^{1}$ , Libo Qin $^{2}$ , Wangxiang Che $^{2}$ , Xiaolong Li $^{1}$ , Ting Liu $^{2}$
<sup>1</sup>Ant Financial Services Group, Alibaba Group
$^{2}$ Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology
{pangmao.lym, kaisheng.yao, xl.li}@antfin.com
{lbqin,car,tliu}@ir.hit.edu.cn
Data-driven approaches using neural networks have achieved promising performances in natural language generation (NLG). However, neural generators are prone to make mistakes, e.g., neglecting an input slot value and generating a redundant slot value. Prior works refer this to hallucination phenomenon. In this paper, we study slot consistency for building reliable NLG systems with all slot values of input dialogue act (DA) properly generated in output sentences. We propose Iterative Rectification Network (IRN) for improving general NLG systems to produce both correct and fluent responses. It applies a bootstrapping algorithm to sample training candidates and uses reinforcement learning to incorporate discrete reward related to slot inconsistency into training. Comprehensive studies have been conducted on multiple benchmark datasets, showing that the proposed methods have significantly reduced the slot error rate (ERR) for all strong baselines. Human evaluations also have confirmed its effectiveness.
Natural Language Generation (NLG), as a critical component of task-oriented dialogue systems, converts a meaning representation, i.e., dialogue act (DA), into natural language sentences. Traditional methods (Stent et al., 2004; Konstas and Lapata, 2013; Wong and Mooney, 2007) are mostly pipeline-based, dividing the generation process into sentence planing and surface realization. Despite their robustness, they heavily rely on handcrafted rules and domain-specific knowledge. In addition, the generated sentences of rule-based approaches are rather rigid, without the variance of human language. More recently, neural network based models (Wen et al., 2015a,b; Dusek and Jurcicek, 2016;
Input DA	inform(NAME = pickwick hotel, PRICERANGE = moderate)
Reference	the hotel named pickwick hotel is in a moderate price range
Missing	this is a moderate hotel [NAME]
Misplace	the pickwick hotel in fort mason is a moderate price range [AREA]
Table 1: An example (including mistaken generations) extracted from SF Hotel (Wen et al., 2015b) dataset. Errors are marked in colors (missing, misplaced).
Tran and Nguyen, 2017a) have attracted much attention. They implicitly learn sentence planning and surface realisation end-to-end with cross entropy objectives. For example, Dusek and Jurcicek (2016) employ an attentive encoder-decoder model, which applies attention mechanism over input slot value pairs.
Although neural generators can be trained end-to-end, they suffer from hallucination phenomenon (Balakrishnan et al., 2019). Examples in Table 1 show a misplacement error of an unseen slot AREA and a missing error of slot NAME by an end-to-end trained model, when compared against its input DA. Motivated by this observation, in this paper, we define slot consistency of NLG systems as all slot values of input DAs shall appear in output sentences without misplacement. We also observe that, for task-oriented dialogue systems, input DAs are mostly with simple logic forms, therefore enabling retrieval-based methods e.g. K-Nearest Neighbour (KNN) to handle the majority of test cases. Furthermore, there exists a discrepancy between the training criterion of cross entropy loss and evaluation metric of slot error rate (ERR), similarly to that observed in neural machine translation (Ranzato et al., 2015). Therefore, it is beneficial to use training methods that integrate the evaluation metrics in their objectives.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 97-106 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In this paper, we propose Iterative Rectification Network (IRN) to improve slot consistency for general NLG systems. IRN consists of a pointer rewriter and an experience replay buffer. Pointer rewriter iteratively rectifies slot-inconsistent generations from KNN or data-driven NLG systems. Experience replay buffer of a fixed size collects candidates, which consist of mistaken cases, for training IRN. Leveraging the above observations, we further introduce a retrieval-based bootstrapping to sample pseudo mistaken cases as candidates for enriching the training data. To foster consistency between training objective and evaluation metrics, we use REINFORCE (Williams, 1992) to incorporate slot consistency and other discrete rewards into training objectives.
Extensive experiments show that, the proposed model, KNN + IRN, significantly outperforms all previous strong approaches. When applying IRN to improve slot consistency of prior NLG baselines, we notice large reductions of their slot error rates. Finally, the effectiveness of the proposed methods are further confirmed using BLEU scores, case analysis and human evaluations.
2.1 Delexicalization
Inputs to NLG are structured meaning representations, i.e., DA, which consists of an act type and a list of slot value pairs. Each slot value pair represents the type of information and its content while the act type control the style of sentence. To improve generalization capability of DA, delexicalization technique (Wen et al., 2015a,b; Dušek and Jurčíček, 2016; Tran and Nguyen, 2017a) is widely used to replace all values in reference sentence by their corresponding slot in DA, creating pairs of delexicalized input DAs and output templates.
Hence the most important step in NLG is to generate templates correctly given an input DA. However, this step can introduce missing and misplaced slots, because of modeling errors or unaligned training data (Balakrishnan et al., 2019; Nie et al., 2019; Juraska et al., 2018). Lexicalization is followed after a template is generated, replacing slots in template with corresponding values in DA.
Formally, we denote a delexicalized input DA as a set  $\mathbf{x} = \{x_1,x_2,\dots ,x_N\}$  that consists of an act type and some slots. Universal set S con
tains all possible slots. The output template  $\mathbf{y} = [y_1, y_2, \dots, y_M]$  from NLG systems  $f(\mathbf{x})$  is a sequence of tokens (words and slots).
We define a slot extraction function  $g$  as
$$
g (\mathbf {z}) = \{t \mid t \in \mathbf {z}; t \in \mathbf {S} \}. \tag {1}
$$
where  $\mathbf{z}$  consists of the DA  $\mathbf{x}$  and elements of the template  $\mathbf{y}$ .
A slot-consistent NLG system  $f(\mathbf{x})$  satisfies the following constraint:
$$
g (f (\mathbf {x})) = g (\mathbf {x}). \tag {2}
$$
To avoid trivial solutions, we require that  $f(\mathbf{x}) \neq \mathbf{x}$ .
However, due to the hallucination phenomenon, it is possible to miss or misplace slot value in generated templates (Wen et al., 2015a), which is hard to avoid in neural-based approaches.
2.3 KNN-based NLG System
A KNN-based NLG system  $f^{\mathrm{KNN}}$  is composed of a distant function  $\rho$  and a template set  $\mathbf{Y} = \{\mathbf{y}_1, \mathbf{y}_2, \dots, \mathbf{y}_Q\}$  which is collected from  $Q$  delexicalized sentences in training corpus.
Given input DA  $\mathbf{x}$ , the distance is defined as
$$
\rho (\mathbf {x}, \mathbf {y} _ {i}) = \# \left(\{s \mid s = t; t \in \mathbf {y} _ {i}; s \in \mathbf {x} \}\right), \tag {3}
$$
where function  $\#$  computes the size of a set. During evaluation, system  $f^{\mathrm{KNN}}$  first ranks the templates in set  $\mathbf{Y}$  by distant function  $\rho$  and then selects the top  $k$  (beam size) templates.
Figure 1 shows the architecture of Iterative Rectification Network. It consists of two components: a pointer rewriter to produce templates with improved performance metrics and an experience replay buffer to gather and sample training data.
The improvements on slot consistency are obtained via an iterative rewriting process. Assume, at iteration  $k$ , we have a template  $\mathbf{y}^{(k)}$  that is not slot consistent with input DA, i.e.,  $g(\mathbf{y}^{(k)}) \neq g(\mathbf{x})$ . Then, a pointer rewrite iteratively rewrites it as
$$
\mathbf {y} ^ {(k + 1)} = \phi^ {\mathrm {P R}} (\mathbf {x}, \mathbf {y} ^ {(k)}). \tag {4}
$$
Above recursion ends once  $g(\mathbf{y}^{(k)}) = g(\mathbf{x})$  or a certain number of iterations is reached.
Figure 1: IRN consists of two modules: an experience replay buffer and a pointer rewritten. The experience replay buffer collects mistaken cases from NLG baseline, template and IRN itself (the red dashed arrow) whereas the pointer network outputs templates with improved performance metrics. In each epoch of rectification, IRN obtains samples of cases for training from the buffer and trains a pointer rewritten with metrics such as slot consistency using a policy-based reinforcement learning technique. We omit some trivial connections for brevity.
3.1 Pointer Rewriter
The pointer rewritten  $\phi^{\mathrm{PR}}$  is trained to iteratively correct the candidate  $\mathbf{y}^{(k)}$  given a DA  $\mathbf{x}$ . This correction operation is conducted time-recurrently. At each position  $j$  of rewriting a template, there is a state  $\mathbf{h}_j$  to represent the past history of the pointer rewritten and an action  $a_j$  to take according to a policy  $\pi$ .
State We use an autoregressive model, in particular LSTM to compute state  $\mathbf{h}_j$ , given its past state  $\mathbf{h}_{j-1}$ , input  $\mathbf{x}$  and its past output  $y_{j-1}^{(k)}$
$$
\mathbf {h} _ {j} = \phi^ {\mathrm {L S T M}} \left(\mathbf {h} _ {j - 1}, \left[ \mathbf {x}; y _ {j - 1} ^ {(k)}; \mathbf {c} _ {j} \right]\right), \tag {5}
$$
where DA  $\mathbf{x}$  is represented by one-hot representation (Wen et al., 2015a,b).  $\mathbf{c}_j$  is a context representation over input template  $\mathbf{y}^{(k)}$ , to be described in Eq. (6). The operation  $[\cdot]$  means vector concatenation.
Action For position  $j$  in the output template  $\mathbf{y}^{(k)}$ , its action  $a_{j}$  is in a space consisting of two categories: template copy,  $c(i)$ , to copy a token from the template  $\mathbf{y}^{(k)}$  at  $i$ , and word and slot generation,  $w$ , to generate a word or a slot at the position. For a length- $M$  input template  $\mathbf{y}^{(k)}$ , the action  $a_{j}$  is therefore in a set of  $\{w, c(1), \dots, c(M)\}$ . The action sequence  $\mathbf{a}$  for a length- $N$  output template is  $[a_{1}, \dots, a_{N}]$ .
Template Copy The model  $\phi^{\mathrm{PR}}$  for template copy uses attentive pointer to decide, for position  $j$ , what token to copy from the candidate  $\mathbf{y}^{(k)}$ . Each token  $y_{i}^{(k)}$  in candidate  $\mathbf{y}^{(k)}$  is represented using an embedding  $\mathbf{y}_{i}^{(k)}$ . For position  $j$  in the output template, this model utilizes the above hidden state  $\mathbf{h}_j$  and computes attentive weights to all of the tokens in  $\mathbf{y}^{(k)}$ , with weight to token embedding  $\mathbf{y}_{i}^{(k)}$  as follows:
$$
\left\{ \begin{array}{c} \phi^ {\mathrm {P R}} \left(\mathbf {h} _ {j}, \mathbf {y} _ {i} ^ {(k)}\right) = \mathbf {v} _ {a} ^ {T} \sigma \left(\mathbf {W} _ {h} * \mathbf {h} _ {j} + \mathbf {W} _ {y} * \mathbf {y} _ {i} ^ {(k)}\right) \\ \mathbf {p} _ {i j} ^ {\mathrm {P R}} = \operatorname {S o f t m a x} \left(\phi^ {\mathrm {P R}} \left(\mathbf {h} _ {j}, \mathbf {y} _ {i} ^ {(k)}\right)\right) \\ \mathbf {c} _ {j} = \sum_ {1 \leq i \leq M} \mathbf {p} _ {i j} ^ {\mathrm {P R}} \mathbf {y} _ {i} \end{array} , \right. \tag {6}
$$
where  $\mathbf{v}_a, \mathbf{W}_h, \mathbf{W}_y$  are learnable parameters.
Word and Slot Generation Another candidate for position  $j$  is a word or a slot key from a predefined vocabulary. The action  $w$  computes a distribution of words and slot keys below
$$
\mathbf {p} _ {j} ^ {\text {V o c a b}} = \operatorname {S o f t m a x} \left(\mathbf {W} _ {v} * \mathbf {h} _ {j}\right), \tag {7}
$$
where this distribution is dependent on the state  $\mathbf{h}_j$  and matrix  $\mathbf{W}_v$  is learnable.
Policy The probabilities for the above actions can be computed as follows
$$
\left\{ \begin{array}{c} \pi (c (i) | \mathbf {h} _ {j}) = \lambda_ {j} * \mathbf {p} _ {j} ^ {\mathrm {P R}} (i) \\ \pi (w | \mathbf {h} _ {j}) = (1 - \lambda_ {j}) * \mathbf {p} _ {j} ^ {\mathrm {V o c a b}}, \end{array} \right. \tag {8}
$$
where  $\pi (c(i)|\mathbf{h}_j)$  is the probability of copying the  $i$  -th token from input template  $\mathbf{y}^{(k)}$  to position  $j$ $\pi (w|\mathbf{h}_j)$  is the probability to use words or slot keys predicted from the distribution  $\mathbf{p}_j^{\mathrm{Vocab}}$  in Eq. (7). The weight  $\lambda_{j}$  is a real value between 0 and 1. It is computed from a Sigmoid operation as  $\lambda_{j} = \mathrm{Sigmoid}(\mathbf{v}_{h}*\mathbf{h}_{j})$  .With the policy, the pointer rewriter does greedy search to decide whether copying or generating a token.
3.2 Experience Replay Buffer
The experience replay buffer aims at providing training samples for IRN. It has three sources of samples. The first is from off-the-shelf NLG systems. The second is from the pointer rewriter in the last iteration. Both of them are real mistaken
samples. They are stored in a case set  $C$  in the buffer. These samples are off-policy as the case set  $C$  can contain samples from many iterations before. The third source is sampled from a bootstrapping algorithm. They are stored in a set  $\Omega$ .
Iterative Data Aggregation The replay experiences should be progressive, reflecting improvements in the iterative training of IRN. Therefore, we design an iterative data aggregation algorithm in Algorithm 1. In the algorithm, the experience replay buffer  $B$  is defined as a fixed size set of  $B = C + \Omega$ . For a total epoch number of  $E$ , it randomly provides mistaken samples for training pointer rewrite  $\phi^{\mathrm{PR}}$  at each epoch. Importantly, both content of  $C$  and  $\Omega$  are varying from each epoch. For  $C$ , it initially consists of real mistaken samples from the baseline system (line 3-th to line 8-th). Later on, it's gradually filled by the samples from the IRN (line 14-th to line 19-th). For  $\Omega$ , its samples reflect a general distribution of training samples from a template database  $T$  (line 10-th). Finally, the algorithm aggregates these two groups of mistaken samples (line 11-th) and use them to train the model  $\phi^{\mathrm{PR}}$  (line 12-th).
Bootstrapping via Retrieval Relying solely on the real mistaken samples exposes the system to data scarcity problem. It is easy to observe that real samples are heavily biased towards certain slots, and the number of real mistaken samples can be small. To address this problem, we introduce a
Figure 2: Correcting a candidate given a reference template.  $d^{c}$ ,  $d^{l}$ , and  $d^{\pi}$  are inferred by simple rules.
bootstrapping algorithm, described in Algorithm 2. It uses a template database  $T$ , built from delexicalized NLG training corpus and organized by pairs of DA and reference template  $(\mathbf{x},\mathbf{z})$ .
At each turn of the algorithm, it first randomly samples (line 3-th) a pair  $(\mathbf{x},\mathbf{z})$ , from training template data base of  $T$ . Then for every pair  $(\hat{\mathbf{x}},\hat{\mathbf{z}})$  in  $T$ , it measures if the pair  $(\hat{\mathbf{x}},\hat{\mathbf{z}})$  is slot-inconsistent with respect to  $(\mathbf{x},\mathbf{z})$ , and adds the pair that is within a certain distance  $\epsilon$  (a hyper parameter) to a set  $Z$  (line 5-th to 11-th).  $\epsilon$  is usually set to a small number so that the selected samples are close enough to  $(x,z)$ . In practice, we set it to 2. Finally, it does a random sampling (line 12-th) on  $Z$  and insert its return into the output set  $\Omega$ . Such bootstrapping process stops when the number of generated samples reaches a certain limit  $K$ .
These samples, which we refer them as pseudo samples in the following, represent a wider coverage of training samples than the real mistaken samples. Because they are sampled from general distribution of the templates, some of semantics are not seen in the real mistaken cases. We will demonstrate through experiments that it effectively addresses data scarcity problem.
4 Training with Supervised Learning and Distant Supervision
One key idea behind the proposed IRN model is to conduct distant supervision on the actions of template copy and generation. We diagram its motivation in Figure 2. During training, only candidate  $\mathbf{y}$  and its reference  $\mathbf{z}$  are given. The exact actions that convert template  $\mathbf{y}$  to  $\mathbf{z}$  have to be inferred from the two templates. Here we use simple rules for the inference. Firstly, the rules check if reference token  $\mathbf{z}_j$  exists in the candidate  $\mathbf{y}$ . The output is a label  $d^c$  consisting of 1s and 0s, representing whether tokens in the reference template are existent/absent in the candidate. Secondly, the rules locate the orig
inal position  $d_j^l$  in the candidate for each token  $j$  in the reference template if  $d^{c} = 1$  and use -1 for  $d^{c} = 0$ . Finally, the action label  $d^{\pi}$  for policy is inferred, with  $w$  for  $d_j^l = -1$  and  $c(i)$  for  $d_j^l = i$ .
We may use the extracted tags to do supervised learning. The loss to be minimized is as follows
$$
J ^ {\mathrm {S L}} = - \sum_ {j = 1} ^ {L} \log \pi \left(d _ {j} ^ {\pi} | \mathbf {h} _ {j}\right), \tag {9}
$$
where  $L$  is the length of ground truth.  $\pi (d_j^\pi |\mathbf{h}_j)$  computes the likelihood of action  $d_j^\pi$  at position  $j$  given state  $\mathbf{h}_j$ .
However, there are following issues when attempting to utilize the labels produced by distant supervision for training. Firstly, the importance of every token in candidate is different. For example, noun phrase (colored in blue) is critical and should be copied. Function words (colored in red) is of little relevance and can be generated by IRN itself. However, distant supervision treats them the same. Secondly, rule-based matching may cause semantic ambiguity (dashed line colored in black). Lastly, the training criterion of cross entropy is not directly relevant to the evaluation metric using slot error rate. To address these issues, we use reinforcement learning to obtain the optimal actions.
5 Training with Policy-based Reinforcement Learning
In this section, we describe another method to train IRN. We apply policy gradient (Williams, 1992) to optimize models with discrete rewards.
5.1 Rewards
Slot Consistency This reward is related to the correctness of output templates. Given the set of slot-value pairs  $g(\mathbf{y})$  from the output template generated by IRN and the set of slot-value pairs  $g(\mathbf{x})$  extracted from input DA, the reward is zero when
Model	SF Restaurant	SF Hotel	Laptop	Television
BLEU	ERR	BLEU	ERR	BLEU	ERR	BLEU	ERR
HLSTM (Wen et al., 2015a)	0.747	0.74%	0.850	2.67%	0.513	1.10%	0.525	2.50%
SCLSTM (Wen et al., 2015b)	0.753	0.38%	0.848	3.07%	0.512	0.79%	0.527	2.31%
TGen (Dušek and Jurčićek, 2016)	0.751	0.84%	0.853	4.14%	0.515	0.87%	0.521	2.32%
ARoA (Tran and Nguyen, 2017b)	0.776	0.30%	0.892	1.13%	0.522	0.50%	0.539	0.60%
RALSTM (Tran and Nguyen, 2017a)	0.779	0.16%	0.898	0.43%	0.525	0.42%	0.541	0.63%
IRN (+ KNN)	0.807	0.11%	0.911	0.32%	0.537	0.29%	0.559	0.35%
Table 2: Experiment results on four datasets for all baselines and our model. Meanwhile, the improvements over all prior methods are statistically significant with  $p < 0.01$  under t-test.
they are equal; otherwise, it is negative with value set to the cardinality of the difference between the two sets as follows
$$
r ^ {\mathrm {S C}} = - | g (\mathbf {y}) - g (\mathbf {x}) |. \tag {10}
$$
Language Fluency This reward is related to the naturalness of the realized surface form from a response generation method. Following (Wen et al., 2015a,b), we first train a backward language model on the reference texts from training data. Then, the perplexity (PPL) of the surface form after lexicalization of the output template  $\hat{\mathbf{y}}$  is measured using the language model. This PPL is used for the reward for language fluency as follows:
$$
r ^ {\mathrm {L M}} = - \operatorname {P P L} (\mathbf {y}). \qquad (1 1)
$$
Distant Supervision We also measure the reward from using distant supervision in Section 4. For a length- $N$  reference template, the reward is given as follows:
$$
r ^ {\mathrm {D S}} = - \sum_ {j = 1} ^ {L} \log \pi \left(d _ {j} ^ {\pi} | \mathbf {h} _ {j}\right), \tag {12}
$$
where  $d_j^\pi$  is the inferred action label.
The final reward for action  $\mathbf{a}$  is a weighted sum of the rewards discussed above:
$$
r (\mathbf {a}) = \gamma^ {\mathrm {S C}} r ^ {\mathrm {S C}} + \gamma^ {\mathrm {L M}} r ^ {\mathrm {L M}} + \gamma^ {\mathrm {D S}} r ^ {\mathrm {D S}} \quad (1 3)
$$
where  $\gamma^{\mathrm{SC}} + \gamma^{\mathrm{LM}} + \gamma^{\mathrm{DS}} = 1$ . We set them to equal value in this work. A reward is observed after the last token of the utterance is generated.
5.2 Policy Gradient
We utilize supervised learning in Eq. (9) to initialize our model with the labels extracted from distant supervision. After its convergence, we continuously tune the model using policy gradient described in this section. The policy model in  $\phi^{\mathrm{PR}}$
itself generates a sequence of actions  $\mathbf{a}$ , that are not necessarily the same as  $d^{\pi}$ , and this produces an output template  $\mathbf{y}$  to compute slot consistency reward in Eq. (10) and language fluency reward in Eq. (11). With these rewards, the final reward is computed in (13). The gradient to back propagate is estimated using REINFORCE as
$$
\nabla J ^ {\mathrm {R L}} (\theta) = (r (\mathbf {a}) - b) * \sum_ {j = 1} ^ {N} \nabla \log \pi \left(a _ {j} \mid \mathbf {h} _ {j}\right), \tag {14}
$$
where  $\theta$  denotes model parameters.  $r(\mathbf{a}) - b$  is the advantage function per REINFORCE.  $b$  is a baseline. Through experiments, we find that  $b = \mathrm{BLEU}(\mathbf{y},\mathbf{z})$  performs better (Weaver and Tao, 2001) than tricks such as simple averaging of the likelihood  $\frac{1}{N}\sum_{j=1}^{N}\log \pi(a_j|\mathbf{h}_j)$ .
We assess the model performances on four NLG datasets of different domains. The SF Hotel and SF Restaurant benchmarks are collected in (Wen et al., 2015a) while Laptop and TV benchmarks are released by (Wen et al., 2016). Each dataset is evaluated with five strong baseline methods, including HLSTM (Wen et al., 2015a), SC-LSTM (Wen et al., 2015b), TGen (Dusek and Jurcicek, 2016), ARoA (Tran and Nguyen, 2017b) and RALSTM (Tran and Nguyen, 2017a). Following these prior works, the evaluation metrics consist of BLEU and slot error rate (ERR), which is computed as
$$
E R R = \frac {p + q}{N}, \tag {15}
$$
where  $N$  is the total number of slots in the DA, and  $p, q$  is the number of missing and redundant slots in the generated template, respectively.
Model	SF Restaurant	Television
BLEU	ERR	BLEU	ERR
HLSTM (Wen et al., 2015a) w/ IRN	0.060 ↑	0.66% ↓	0.040 ↑	2.29% ↓
TGen (Dušek and Jurčićek, 2016) w/ IRN	0.002 ↑	0.73% ↓	0.005 ↑	1.99% ↓
RALSTM (Tran and Nguyen, 2017a) w/ IRN	0.007 ↑	0.11% ↓	0.004 ↑	0.36% ↓
Table 3: The up and down arrows emphasize the absolutely improved performances contributed by IRN.
Method	Laptop
BLEU	SER
IRN (+KNN)	0.537	0.29%
w/o IRN	0.414	0.88%
w/o reward rSC	0.526	0.75%
w/o reward rDS	0.527	0.66%
w/o reward rLM	0.529	0.49%
w/o baseline BLEU	0.531	0.37%
w/o Aggregation	0.515	0.48%
w/o Bootstrapping	0.464	0.83%
We follow all baseline performances reported in (Tran and Nguyen, 2017b) and use open source toolkits, RNNLG<sup>1</sup> and Tgen<sup>2</sup> to build NLG systems, HLSTM, SCLSTM and TGen. We re-implement the baselines ARoA and RALSTM since their source codes are not available.