SlowGuess/ACL_2020_OCR huggingface dataset:
Chao Zhao†, Marilyn Walker‡ and Snigdha Chaturvedi†
† Department of Computer Science, University of North Carolina at Chapel Hill ‡ Natural Language and Dialog Systems Lab, University of California, Santa Cruz {zhaochao, snigdha}@cs.unc.edu mawalker@ucsc.edu
Generating sequential natural language descriptions from graph-structured data (e.g., knowledge graph) is challenging, partly because of the structural differences between the input graph and the output text. Hence, popular sequence-to-sequence models, which require serialized input, are not a natural fit for this task. Graph neural networks, on the other hand, can better encode the input graph but broaden the structural gap between the encoder and decoder, making faithful generation difficult. To narrow this gap, we propose DUALENC, a dual encoding model that can not only incorporate the graph structure, but can also cater to the linear structure of the output text. Empirical comparisons with strong single-encoder baselines demonstrate that dual encoding can significantly improve the quality of the generated text.
Data-to-text generation aims to create natural language text to describe the input data (Reiter and Dale, 2000). Here we focus on structured text input in a particular form such as a tree or a graph. Figure 1 shows an example where the input data is a mini knowledge graph, and the output text is its corresponding natural language description. Generating text from such data is helpful for many NLP tasks, such as question answering and dialogue (He et al., 2017; Liu et al., 2018; Moon et al., 2019).
During generation, the structure of the data as well as the content inside the structure jointly determine the generated text. For example, the direction of the edge "capital" in Figure 1 determines that "London is the capital of U.K." is an accurate description, but not vice versa. Current generation methods are based on sequence-to-sequence (Seq2Seq) encoder-decoder architecture (Sutskever et al., 2014), which requires the input data to be
Figure 1: Illustration of the WebNLG challenge: the source data is an RDF graph and the target output is a text description of the graph.
serialized as a sequence, resulting in a loss of structural information.
Recent research has shown the utility of incorporating structural information during generation. By replacing the sequential encoder with a structure-aware graph encoder, such as a graph convolutional network (GCNs) (Kipf and Welling, 2017) or graph-state LSTMs (Song et al., 2018), the resulting graph-to-sequence (Graph2Seq) methods can encode the structural information of the input and thus outperform Seq2Seq models on certain tasks. However, these architectures broaden the structural gap between the encoder and decoder. That is, while the encoder receives the input data as a graph, the decoder has to create the output text as a linear chain structure.
This structural gap increases the difficulty of establishing alignments between source and target, which is believed to play a key role in text generation. For example, in machine translation, pre-reordering the source words into a word order that is close to that of the target sentence can yield significant improvements in translation quality (Bisazza and Federico, 2016). This suggests a need for an intermediate "planning" stage (Reiter
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2481-2491 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
and Dale, 2000; Puduppully et al., 2019) to help with organizing the output.
In this work, we present a dual encoding model that is not only aware of the input graph structure but also incorporates a content planning stage. To encode the structural information in the input graph, we use a GCN based graph encoder. To narrow the ensuing structural gap, we use another GCN-based neural planner to create a sequential content plan of this graph, which is represented as a re-ordered sequence of its nodes. The plan is then encoded by an LSTM based sequential encoder. During generation, an LSTM based decoder simultaneously conditions on the two encoders, which helps it in capturing both the graph structure of the input data and the linear structure of the plan. We expect such a dual encoding (DUALENC) structure can integrate the advantages of both graph and sequential encoders while narrowing the structural gap present in single-encoder methods.
We evaluate the proposed planning and generation models on the WebNLG dataset (Colin et al., 2016; Gardent et al., 2017) – a widely used benchmark for data-to-text generation. Experimental results show that our neural planner achieves a  $15\%$  absolute improvement on accuracy compared to the previous best planning method. Furthermore, DUALENC significantly outperforms the previous start-of-the-art on the generation task. The human evaluation confirms that the texts generated by our model are preferred over strong baselines.
The contributions of this paper are three-fold:
- We propose a dual encoding method to narrow the structural gap between data encoder and text decoder for data-to-text generation;
- We propose a neural planner, which is more efficient and effective than previous methods;
- Experiments show that our method outperforms all baselines on a variety of measures.
This work is inspired by two lines of research: Seq2Seq generation and Graph2Seq generation.
2.1 Seq2Seq Generation
Traditional data-to-text generation follows a planning and realization pipeline (Reiter and Dale, 2000; Stent et al., 2004). More recent methods use Seq2Seq architecture (Sutskever et al., 2014) to combine planning and realization into an end-to-end network and have achieved the state-of-the-art
on a variety of generation tasks (Lebret et al., 2016; Trisedya et al., 2018; Juraska et al., 2018; Reed et al., 2018). Despite the fair fluency and grammatical correctness, the generated text suffers from several problems such as repetition, omission, and unfaithfulness, which are less likely to happen in traditional planning-and-realization frameworks.
Recent work has shown that neural models can also benefit from an explicit planning step to alleviate the above-mentioned problems. The input of these planners ranges from unstructured keyphrases (Hua and Wang, 2019) to structured tables (Puduppully et al., 2019) and graphs (Ferreira et al., 2019; Moryossef et al., 2019a). Our work also focuses on planning from graph data. Compared with previous methods, we show that our neural planning method is more feasible and accurate. More importantly, rather than serializing the planning and realization stages in a pipeline, our dual encoding method simultaneously captures information from the original data and the corresponding plan.
2.2 Graph2Seq Generation
Graph neural networks (GNN) (Scarselli et al., 2009) aim to learn a latent state representation for each node in a graph by aggregating local information from its neighbors and the connected edges. Previous work has explored different ways of aggregating this local information, such as in GCNs (Kipf and Welling, 2017), gated graph neural networks (GGNNs) (Li et al., 2016), and Graph attention networks (GANs) (Veličković et al., 2018)
Several works have applied GNNs instead of Seq2Seq models for text generation (Beck et al., 2018; Marcheggiani and Perez-Beltrachini, 2018; Guo et al., 2019; Li et al., 2019), and some of them outperform Seq2Seq models. However, Damonte and Cohen (2019) use both types of encoders and show that GCN can help LSTM capture reentrant structures and long-range dependencies, albeit on a different problem than ours. Our method also uses the two types of encoders but instead of using one to assist the other, it combines them simultaneously to capture their complementary effects.
In this work we focus on text generation from RDF data. The input for this task is a set of RDF triples, where each triple  $(s,p,o)$  contains a subject, a predicate, and an object. For example, ("U.K.", "cap
<https://www.w3.org/TR/rdf-concepts/>
Figure 2: The architecture of the proposed DUALENC model. The input triples are converted as a graph and then fed to two GCN encoders for plan and text generation (Planner and Graph Encoder, top center). The plan is then encoded by an LSTM network (Plan Encoder, bottom center). Finally an LSTM decoder combines the hidden states from both the encoders to generate the text (Text Decoder, middle right).
ital”, “London”) is a RDF triple. The output is a natural language text with one or more sentences to describe the facts represented by this graph. Figure 1 shows an example of this task.
4 Dual Encoding Model
For a given input RDF graph, the aim of our method is not only to capture its structural information, but also to facilitate the information alignment between the input and output. The first goal can be achieved by employing a GCN encoder. To achieve the second goal, we first serialize and re-order the nodes of the graph as an intermediate plan using another GCN, and then feed the plan into an LSTM encoder. Finally, an LSTM decoder is used to generate the output by incorporating the context representations of both encoders. Notice that the graph and the plan are dual representations of the same input data. We encode them with two independent encoders, which can provide complementary information for decoding. The architecture of our dual encoding method is shown in Figure 2. We describe the two encoders and the decoder in the following three subsections.
4.1 Graph Representation and Encoding
To make it easier for GCNs to encode information from both entities and predicates, we reconstruct the input graph by regarding both entities and predicates as nodes, which is different from Figure 1.
Formally, for each RDF triple  $(s,p,o)$ , we regard the  $s$ ,  $p$ , and  $o$  as three kinds of nodes.  $s$  and  $o$  are identified by their entity mentions, and  $p$  is identified by a unique ID. That is, two entities from different triples that have the same mentions will
be regarded as the same node. However, since we want to use predicates to distinguish between different triples, two predicates with the same mentions will be regarded as separate nodes.[2]
Figure 3: The graph obtained from an RDF triple.
We use the same edge structure as Beck et al. (2018). As Figure 3 shows, a triple contains four directed edges to connect its nodes:  $s \to p$ ,  $p \to s$ ,  $o \to p$ , and  $p \to o$ . These edges help in information exchange between arbitrary neighbor pairs. There is also a special self-loop edge  $n \to n$  for each node  $n$  to enable information flow between adjacent iterations during feature aggregation.
After building the graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  from the RDF data, we use a relational GCN (R-GCN) (Schlichtkrull et al., 2018) to encode the graph and learn a state representation  $\mathbf{h}_v\in \mathbb{R}^d$  for each node  $v\in \mathcal{V}$  using the following iterative method:
$$
\mathbf {h} _ {v} ^ {t} = \rho \left(\sum_ {r \in \mathcal {R}} \sum_ {u \in \mathcal {N} _ {v} ^ {r}} \frac {1}{c _ {v , r}} \mathbf {W} _ {r} \mathbf {h} _ {u} ^ {(t - 1)} + \mathbf {b} _ {r}\right) \tag {1}
$$
where  $\mathbf{h}_v^0 = \mathbf{x}_v$  is the input embedding of the node  $v$ , and  $\mathbf{h}_v^t$  is its hidden state at time-step  $t$ . We use the average embedding of the node mentions as  $\mathbf{x}_v$ .  $\mathcal{R}$  is the set of all possible edge types, and  $\mathcal{N}_v^r$  is the set of in-neighbors of node  $v$  with the edge
For example, 'capital's in (U.K., capital, London) and (U.S., capital, Washington D.C.) are different nodes.
Figure 4: The sequential decision-making process of the planning stage.
type as  $r$ .  $\mathbf{W}_r$  and  $\mathbf{b}_r$  are parameters for each edge type, which allow transformations of message to become relational-specific.  $c_{v,r} = 1 / |\mathcal{N}_v^r|$  is a normalization term and  $\rho (\cdot)$  is an activation function.
4.2 Planning Creation and Encoding
In the planning stage, we determine the content plan or order of triples (identified by their predicates) for text realization. For example, the content plan for the text in Figure 1 is: "assembly  $\rightarrow$  capital  $\rightarrow$  successor  $\rightarrow$  manufacturer".<sup>3</sup>
Learning a plan can be naturally regarded as a sequential decision-making process. That is, given a set of triples, we first determine which triple to mention/visit first, and then select the second triple from the remaining triples that have not been visited so far. This process continues until all the triples have been visited. During each decision step, the selection of the next triple can be regarded as a classification task, where the output space is all the remaining unvisited triples.
Figure 4 shows how our model implements this process. We first utilize the GCN encoder described in Section 4.1 to get the state representation of each node. However, while obtaining a predicate's representation, we concatenate two extra bits to the input feature  $\mathbf{X}^t$ . One is to indicate whether or not the predicate has been visited, the other to indicate the last predicate that has been visited. After the encoding, we get the final hidden state  $\mathbf{h}_{r_i} = \mathbf{h}_{r_i}^{(T)}$  for each predicate  $r_i \in \mathcal{R}$  as its representation, and calculate its probability of being selected as
$$
P \left(r _ {i}\right) = \operatorname {s o f t m a x} \left(\mathbf {h} _ {r _ {i}} ^ {T} \mathbf {W} \bar {\mathbf {h}} _ {\mathcal {R}}\right) \tag {2}
$$
where  $\bar{\mathbf{h}}_{\mathcal{R}}$  is the average pooling of all the predicate embeddings. For obtaining a plan, we select the predicate with the highest probability, append it onto the plan sequence, and then repeat the above process until all the predicates have been visited.
After determining an order of input predicates, we complete the plan's triples by adding the corresponding subjects and objects. To better help the plan encoder (described below) capture the semantic roles of each entity and predicate, we add special tokens before Subjects, Predicates, and Objects as delimiters. For example, the plan of the example in Figure 1 will be:
$\langle S\rangle$  Aston Martin V8  $\langle \mathbb{P}\rangle$  assembly  $\langle 0\rangle$  United Kingdom  $\langle S\rangle$  United Kingdom  $\langle \mathbb{P}\rangle$  capital  $\langle 0\rangle$  London  $\langle S\rangle$  Aston Martin V8  $\langle \mathbb{P}\rangle$  successor  $\langle 0\rangle$  Aston Martin Virage  $\langle S\rangle$  Aston Martin Virage  $\langle \mathbb{P}\rangle$  manufacturer  $\langle 0\rangle$  Aston Martin
Finally, we use an LSTM to encode the plan obtained above. We choose LSTM because it excels at capturing sequential information.
4.3 Decoding
During decoding, we adopt an LSTM-based decoder with an attention and copy mechanism. Since we have two representations of the input triple-set: the original graph and the serialized plan, we adopt two strategies for inputting context to the decoder.
The first strategy is to only use hidden states of the plan encoder as context. We refer to this strategy as PLANENC.
While the serialized plan may contain some structural information, it cannot preserve all the information of the original graph. We therefore propose a second strategy, DUALENC, to incorporate the information from both the graph and the plan. More concretely, when calculating the context state  $\mathbf{m}_t$  of the LSTM decoder at time step  $t$ , we concatenate the previous hidden state  $\mathbf{z}_{t-1}$  and the two context vectors  $\mathbf{c}_t^1$  and  $\mathbf{c}_t^2$ , and then update the current hidden state,  $\mathbf{z}_t$  as:
$$
\mathbf {m} _ {t} = \mathrm {M L P} ([ \mathbf {z} _ {t - 1}; \mathbf {c} _ {t} ^ {1}; \mathbf {c} _ {t} ^ {2} ]), \qquad (3)
$$
$$
\mathbf {z} _ {t} = \mathrm {L S T M} \left(\mathbf {z} _ {t - 1}, \left[ \left(\mathbf {y} _ {t - 1}; \mathbf {m} _ {t} \right]\right), \right. \quad (4)
$$
where  $\mathbf{c}_t^1$  and  $\mathbf{c}_t^2$  are the attention-based weighted sum of the context memories from GCN and RNN encoders, respectively, and  $\mathbf{y}_{t - 1}$  is the embedding of the previously generated token. The initial hidden state  $\mathbf{z}_0$  is the summation of the final states from the two encoders. For the plan encoder, we use the final state  $\mathbf{H}^T$  of LSTM as the context representation. For the graph encoder, we use an average of all the hidden states following a two-layer perceptron to produce the final state.
3Here we only consider the order of triples. Future plans could explore ordering of subjects and/or objects.
We conduct experiments to evaluate our Planner (Section 5.2) and the overall generation system (Section 5.3).<sup>4</sup>
We conduct experiments on the WebNLG dataset (Gardent et al., 2017; Castro Ferreira et al., 2018) used in the WebNLG challenge. For each instance, the input is a set of up to 7 RDF triples from DBpedia, and the output is their text descriptions. Each triple-set is paired with a set of (up to three) human-generated reference texts. Each reference is also paired with the order of triples it realized. We use them to train and evaluate our Planner. Overall, the dataset contains 9,674 unique triple-sets and 25,298 text references, and is divided into training, development, and test set. The test set contains two subsets, the SEEN part where the instances belong to one of the nine domains that are seen in the training and development set (such as Astronaut and Food), and the UNSEEN part where the instances are from the other five unseen domains. The UNSEEN part is designed to evaluate models' generalizability to out-of-domain instances.
5.2 Experiments on Plan Generation
As previous work suggests, planning plays a crucial role in text generation. We, therefore, first investigate the performance of our planner.
During the graph encoding, we initialize the node embeddings with 100-dimensional random vectors. Our GCN model has two layers, with the hidden size of each layer as 100. The activation function is ReLU (Nair and Hinton, 2010). We optimize the training objective using Adam (Kingma and Ba, 2015) with a learning rate of 0.001 and an early stopping on the development set. The batch size is 100. We compare our results with the following six baseline planners:
- Random: returns a random permutation of the input triples as a plan;
- Structure-Random: returns a random traversal over the input graph. We report the highest score among three random strategies: random walk, random BFS, and random DFS;
- Step-By-Step (Moryossef et al., 2019a): a transition-based statistical ranking method;
- Step-By-Step II (Moryossef et al., 2019b): a DFS-based method with a neural controller;
- GRU & Transformer (Ferreira et al., 2019): two neural Seq2Seq methods with attention;
We report the performance on three test sets: SEEN, UNSEEN, and ALL (SEEN & UNSEEN). We remove all one-triple instances for planner's evaluation since the planning for these instances is trivial. Results are evaluated with accuracy and BLEU-n (Papineni et al., 2002). For accuracy, we regard a plan as correct only if it exactly matches one of the human-generated plans. BLEU-n is more forgiving than accuracy. It is also adopted in Yao et al. (2019) for plan evaluation. Here we choose  $n = 2$ .
Table 1 shows results of the planning experiments. Our GCN method significantly outperforms all the baselines (approximate randomization (Noreen, 1989; Chinchor, 1992),  $p < 0.05$ ) by a large margin on all the test sets and both measures, indicating the effectiveness of our planner. The most competitive baseline on ALL and UNSEEN sets is Step-By-Step, but our method is more time-efficient. For example, Step-By-Step needs 250 seconds to solve one 7-triple instance, but our method solves all 4928 instances in less than 10 seconds. For the SEEN set, the most competitive models are GRU and Transformer. However, while their accuracies drop by 0.46 on UNSEEN test set, our method drops only slightly by 0.02, indicating our method's better generalization power.
We believe that this superior generalization capacity comes from the modeling of the graph structure. While the surface forms of triples in UNSEEN set do not overlap with those in the training data, the graph-level structural features are still shared, making it a key factor for generalization. GRU and Transformer linearize the graph as a sequential input, making them miss the structural information and resulting in poorer generalization capacity. Step-By-Step II also considers graph structure, but our model achieves better performance because we use GCN to encode the node representation, which can aggregate richer information from both the graph structure and the surface information.
We also investigated the effect of the graph size on the plan quality. In Figure 5, we separate the ALL test set into six subsets according to the size of input triple-sets, to reflect the model's capacity
$^{4}$ Code is available on https://github.com/zhaochaocs/DualEnc
http://webnlg.loria.fr/pages/index.html
Accuracy	BLEU-2
SEEN	UNSEEN	ALL	SEEN	UNSEEN	ALL
Random	0.28	0.34	0.31	54.1	62.1	57.9
Structure-random	0.32	0.38	0.34	56.6	62.9	59.5
Transformer (Ferreira et al., 2019)	0.56	0.09	0.34	74.3	20.9	49.3
GRU (Ferreira et al., 2019)	0.56	0.10	0.35	75.8	25.4	52.2
Step-By-Step II (Moryossef et al., 2019b)	0.45	0.44	0.44	67.7	67.3	67.5
Step-By-Step (Moryossef et al., 2019a)	0.49	0.44	0.47	73.2	68.0	70.8
GCN	0.63	0.61	0.62	80.8	79.3	80.1
Table 1: Planning results of three test sets evaluated by accuracy and BLEU-2.
Figure 5: Fine-grained planning results for the ALL test set. Our method outperforms all the baselines regardless of the triple size.
at a fine-grained level. Fewer input triples make the planning task easier, while the 7-triple case is the most difficult one. The accuracy of seven out of eight baselines drops to around 0 in this case, while our method achieves an accuracy of 0.19. Besides this, our method consistently outperforms all the baselines for all the triple-set sizes.
5.3 Experiments on Text Generation
This section investigates the ability of our models to improve the generation quality.
5.3.1 Setup
We implement the generator based on the OpenNMT toolkit. For the graph encoder, we use a similar setting as above. Since the generation task is more complicated than planning, we increase the dimension of the input and the hidden states to 256. The plan encoder is a 2-layer bidirectional LSTM with the same dimension setting of the GCN to ease the information fusion. During encoding, for UNSEEN test set, we adopt delexicalization (Gardent et al., 2017) to enhance the model's generalizability to unseen domains.
We use Adam with a batch size of 64. The initial learning rate is set to 0.001 and is decayed with a rate of 0.7 after the eighth epoch. We continue the
training until the perplexity of the development set does not decrease. We also apply dropout on the decoding output layer with a rate of 0.3.
The quality of the generated text (as well as those of the baselines) is evaluated through a variety of automatic measures, such as BLEU, METEOR, and TER, which are strictly the same as those applied in the official challenge. Following Marcheggiani and Perez-Beltrachini (2018), we report averaged performances over ten runs of the models.
We compare our method with the top systems of the WebNLG challenge and published state-of-the-art systems. The WebNLG systems are:
- ADAPT: a neural system with sub-word representations to deal with rare words and sparsity.
- TILB-SMT: a statistical machine translation method using Moses and delexicalization.
- MELBOURNE: a Seq2Seq model with enriched delexicalization from DBPedia.
The published research models are:
- GTR-LSTM (Trisedya et al., 2018): a graph-based triple encoder;
- GCN-EC (Marcheggiani and Perez-Beltrachini, 2018): a GCN-based triple encoder with glove embedding and copy;
- GRU & Transformer (Ferreira et al., 2019): two pipeline methods with 5 sequential steps and GRU or Transformer as the encoder;
- STEP-BY-STEP (Moryossef et al., 2019a): a pipeline method that generates the text from plans with OpenNMT and a copy mechanism.
5.3.2 Qualitative Results