SlowGuess/ACL_2020_OCR huggingface dataset:
Linfeng Song $^{1}$ , Ante Wang $^{2}$ , Jinsong Su $^{2*}$ , Yue Zhang $^{3}$ , Kun Xu $^{1}$ , Yubin Ge $^{4}$  and Dong Yu $^{1}$
$^{2}$ Xiamen University, Xiamen, China
$^{3}$ School of Engineering, Westlake University, China
<sup>4</sup>University of Illinois at Urbana-Champaign, USA
The task of graph-to-text generation aims at producing sentences that preserve the meaning of input graphs. As a crucial defect, the current state-of-the-art models may mess up or even drop the core structural information of input graphs when generating outputs. We propose to tackle this problem by leveraging richer training signals that can guide our model for preserving input information. In particular, we introduce two types of autoencoding losses, each individually focusing on different aspects (a.k.a. views) of input graphs. The losses are then back-propagated to better calibrate our model via multi-task training. Experiments on two benchmarks for graph-to-text generation show the effectiveness of our approach over a state-of-the-art baseline. Our code is available at http://github.com/Soistesimmer/AMR-multiview.
Many text generation tasks take graph structures as their inputs, such as Abstract Meaning Representation (AMR) (Banarescu et al., 2013), Knowledge Graph (KG) and database tables. For example, as shown in Figure 1(a), AMR-to-text generation is to generate a sentence that preserves the meaning of an input AMR graph, which is composed by a set of concepts (such as "boy" and "want-01") and their relations (such as ".ARG0" and ".ARG1"). Similarly, as shown in Figure 1(b), KG-to-text generation is to produce a sentence representing a KG, which contains worldwide factoid information of entities (such as "Australia" and "Above the Veil") and their relations (such as "followedBy").
Recent efforts on graph-to-text generation tasks mainly focus on how to effectively represent input graphs, so that an attention mechanism can better transfer input knowledge to the decoder when
Figure 1: (a) An AMR graph meaning "The boy wants the beautiful girl to eat lunch with him.", and (b) A knowledge graph carrying the meaning "Above the Veil is an Australian novel and the sequel to Aenir. It was followed by Into the Battle."
generating sentences. Taking AMR-to-text generation as an example, different graph neural networks (GNNs) (Beck et al., 2018; Song et al., 2018; Guo et al., 2019; Ribeiro et al., 2019) have been introduced to better represent input AMRs than a sequence-to-sequence model (Konstas et al., 2017), and later work (Zhu et al., 2019; Cai and Lam, 2019; Wang et al., 2020) showed that relation-aware Transformers can achieve even better results than GNNs. These advances for encoding have largely pushed the state-of-the-art performance.
Existing models are optimized by maximizing the conditional word probabilities of a reference sentence, a common signal for training language models. As a result, these models can learn to produce fluent sentences, but some crucial input concepts and relations may be messed up or even dropped. Taking the AMR in Figure 1(a) as an example, a model may produce "the girl wants the boy to go", which conveys an opposite meaning to the AMR graph. In particular, this can be very likely if "the girl wants" appears much more frequent than "the boy wants" in the training corpus. This is a very important issue, because of its wide existence across many neural graph-to-text
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7987-7998 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
generation models, hindering the usability of these models for real-world applications (Dušek et al., 2018, 2019; Balakrishnan et al., 2019).
A potential solution for this issue is improving the training signal to enhance preserving of structural information. However, little work has been done to explore this direction so far, probably because designing such signals is non-trivial. As a first step towards this goal, we propose to enrich the training signal with additional autoencoding losses (Rei, 2017). Standard autoencoding for graph-to-sequence tasks requires reconstructing (Parsing into) input graphs, while the parsing algorithm for one type of graphs (such as knowledge graphs) may not generalize to other graph types or may not even exist. To make our approach general across different types of graphs, we propose to reconstruct different views of each input graph (rather than the original graph), where each view highlights one aspect of the graph and is easy to produce. Then through multi-task learning, the autoencoding losses of all views are back-propagated to the whole model so that the model can better follow the input semantic constraints.
Specifically, we break each input graph into a set of triples to form our first view, where each triple (such as "want-01 :ARG0 boy" in Figure 1(a)) contains a pair of entities and their relation. As the next step, the alignments between graph nodes and target words are generated to ground this view into the target sentence for reconstruction. Our second view is the linearization of each input graph produced by depth-first graph traversal, and this view is reconstructed token-by-token from the last decoder state. Overall the first view highlights the local information of each triple relation, the second view focuses on the global semantic information of the entire graph.
Experiments on AMR-to-text generation and WebNLG (Gardent et al., 2017) show that our graph-based multi-view autoencoding loss improves the performance of a state-of-the-art baseline by more than 2 BLEU points without introducing any parameter during decoding. Besides, human studies show that our approach is indeed beneficial for preserving more concepts and relations from input graphs.
Previous work for neural graph-to-text generation (Konstas et al., 2017; Song et al., 2018; Beck
et al., 2018; Trisedya et al., 2018; Marcheggiani and Perez-Beltrachini, 2018; Xu et al., 2018; Cao and Clark, 2019; Damonte and Cohen, 2019; Hajduk et al., 2019; Koncel-Kedziorski et al., 2019; Hong et al., 2019; Song et al., 2019; Su et al., 2017) mainly studied how to effectively represent input graphs, and all these models are trained only with the standard language modeling loss. As the most similar one to our work, Tu et al. (2017) proposed an encoder-decoder-reconstructor model for machine translation, which is trained not only to translate each source sentence into its target reference, but also to translate the target reference back into the source text (reconstruction). Wiseman et al. (2017) extended the reconstruction loss of Tu et al. (2017) on table-to-text generation, where a table contains multiple records that fit into several fields. We study a more challenging topic on how to reconstruct a complex graph structure rather than a sentence or a table, and we propose two general and effective methods that reconstruct different complementary views of each input graph. Besides, we propose methods to breakdown the whole (graph, sentence) pair into smaller pieces of (edge, word) pairs with alignments, before training our model to reconstruct each edge given the corresponding word. On the other hand, neither of the previous efforts tried to leverage this valuable information.
Our work is remotely related to the previous efforts on string-to-tree neural machine translation (NMT) (Aharoni and Goldberg, 2017; Wu et al., 2017; Wang et al., 2018), which aims at generating target sentences with their syntactic trees. One major difference is that their goal is producing grammatical outputs, while ours is preserving input structures. Besides, our multi-view reconstruction framework is a detachable component on top of the decoder states for training, so no extra error propagation (for structure prediction) can be introduced. Conversely, their models generate trees together with target sentences, thus extra efforts (Wu et al., 2017) are introduced to alleviate error propagation. Finally, there exist transition-based algorithms (Nivre, 2003) to convert tree parsing into the prediction of transition actions, while we study reconstructing graphs, where there is no common parsing algorithm for all graph types.
Autoencoding loss by input reconstruction was mainly adopted on sequence labeling tasks, such as named entity recognition (NER) (Rei, 2017; Liu et al., 2018a; Jia et al., 2019), simile detection
7988
(Liu et al., 2018b) and sentiment analysis (Rei and Søgaard, 2019). Since input reconstruction is not intuitively related to these tasks, the autoencoding loss only serves as more training signals. Different from these efforts, we leverage autoencoding loss as a means to preserve input knowledge. Besides, we study reconstructing complex graphs, proposing a general multi-view approach for this goal.
3 Base: Structure-Aware Transformer
Formally, an input for graph-to-text generation can be represented as  $\pmb{G} = \langle \pmb{V}, \pmb{E} \rangle$ , where  $\pmb{V}$  is the set of graph nodes and  $\pmb{E}$  corresponds to all graph edges. Each edge  $e \in \pmb{E}$  is a triple  $(v_i, l, v_j)$ , showing labelled relation between two connected nodes  $v_i$  and  $v_j$ . Given a graph, we choose a recent relation-aware transformer model (Zhu et al., 2019) as our baseline to generate the ground-truth sentence  $\pmb{y} = (y_1, \dots, y_N)$  that contain the same meaning as the input graph. It exhibits the state-of-the-art performance for AMR-to-text generation.
3.1 Structure-aware Transformer Encoder
Similar to the standard model (Vaswani et al., 2017), the structure-aware Transformer encoder stacks multiple self-attention layers on top of an embedding layer to encode linearized graph nodes. Taking the  $l$ -th layer for example, it consumes the states of its preceding layer  $(h_1^{l - 1}\dots h_N^{l - 1})$  or the embedding layer when  $l$  is 1) and its states are then updated by a weighted sum:
$$
\boldsymbol {h} _ {i} ^ {l} = \sum_ {j \in [ 1.. N ]} \alpha_ {i j} \left(\boldsymbol {h} _ {j} ^ {l - 1} \boldsymbol {W} ^ {P} + \boldsymbol {\gamma} _ {i j} \boldsymbol {W} ^ {R _ {1}}\right), \tag {1}
$$
where  $\gamma_{ij}$  is the vector representation of the relation between nodes  $v_{i}$  and  $v_{j}$ , and  $\mathbf{W}^{P}$  and  $\mathbf{W}^{R_1}$  are model parameters. The weights, such as  $\alpha_{ij}$ , are obtained by relation-aware self-attention:
$$
\alpha_ {i j} = \frac {\exp \left(e _ {i j}\right)}{\sum_ {k \in [ 1.. N ]} \exp \left(e _ {i k}\right)} \tag {2}
$$
$$
e _ {i j} = \frac {\left(\boldsymbol {h} _ {i} ^ {l - 1} \boldsymbol {W} ^ {Q}\right) \left(\boldsymbol {h} _ {j} ^ {l - 1} \boldsymbol {W} ^ {K} + \boldsymbol {\gamma} _ {i j} \boldsymbol {W} ^ {R _ {2}}\right) ^ {\intercal}}{\sqrt {d _ {h}}} \tag {3}
$$
where  $\pmb{W}^{Q},\pmb{W}^{K}$  and  $\pmb{W}^{R_2}$  are model parameters, and  $d_h$  denotes the encoder-state dimension. The encoder adopts  $L$  self-attention layers and  $\pmb{H}^{L} = (h_{1}^{L}\dots h_{|\pmb{V}|}^{L})$  represents the concatenated top-layer hidden states of the encoder, which will be used in attention-based decoding.
Compared with the standard model, this encoder introduces the vectorized structural information (such as  $\gamma_{ij}$ ) for all node pairs. Given a node pair  $v_i$  and  $v_j$ , generating such information involves two main steps. First, a sequence of graph edge labels along the path from  $v_i$  to  $v_j$  are obtained, where a direction symbol is added to each label to distinguish the edge direction. For instance, the label sequence from "boy" to "girl" in Figure 1(a) is " $ARG0\uparrow:ARG1\downarrow:ARG0\downarrow$ . As the next step, the label sequence is treated as a single (feature) token and represented by the corresponding embedding vector, and this vector is taken as the vectorized structural information  $\gamma_{ij}$  from  $v_i$  to  $v_j$ . Since there are a large number of features, only the most frequent 20K are kept, while the rest are mapped into a special UNK feature. $^1$
3.2 Standard Transformer Decoder
The decoder is the same as the standard Transformer architecture, which stacks an embedding layer, multiple  $(L)$  self-attention layers and a linear layer with softmax activation to generate target sentences in a word-by-word manner. Each target word  $y_{i}$  and decoder state  $s_i$  are generated sequentially by the self-attention decoder:
$$
y _ {i}, \boldsymbol {s} _ {i} = \operatorname {S A D e c o d e r} \left(\left[ \boldsymbol {H} ^ {L}; \boldsymbol {s} _ {1} \dots \boldsymbol {s} _ {i - 1} \right], y _ {i - 1}\right), \tag {4}
$$
where SADecoder() is the function of decoding one step with the self-attention-based decoder.
3.3 Training with Language Modeling Loss
Same as most previous work, this model is trained with the standard language modeling loss that minimizes the negative log-likelihood of conditional word probabilities:
$$
\begin{array}{l} l _ {b a s e} = - \sum_ {i \in [ 1.. N ]} \log p (y _ {i} | y _ {1}, \dots , y _ {i - 1}; \boldsymbol {G}) \\ = - \sum_ {i \in [ 1.. N ]} p (y _ {i} | \boldsymbol {s} _ {i}; \boldsymbol {\theta}), \\ \end{array}
$$
where  $\theta$  represents all model parameters.
4 Multi-View Autoencoding Losses
Figure 2 visualizes the training framework using our multi-view autoencoding losses, where the
<sup>1</sup>Zhu et al. (2019) also mentions other (such as CNN-based or self-attention-based) alternatives to calculate  $\gamma_{ij}$ . While the GPU memory consumption of these alternatives is a few times more than our baseline, ours actually shows a comparable performance.
7989
Figure 2: The training framework using multi-view autoencoding losses.
attention-based encoder-decoder model with the language modeling loss is the baseline. Our losses are produced by reconstructing the two proposed views (surrounded by slashed or dotted box) of the input graph, where each view represents a different aspect of the input. With the proposed losses, we expect to better refine our model for preserving the structural information of input graphs.
4.1 Loss 1: Reconstructing Triple Relations with Biaffine Attention
Our first view breaks each input graph into a set of triples, where each triple (such as "want-01 :ARG0 boy" in Figure 1(a)) contains a pair of nodes and their labeled relation. Next, we use pre-generated alignments between graph nodes and target words to ground the graph triples onto the target sentence. As illustrated in the slashed blue box of Figure 2, the result contains several labeled arcs, each connecting a word pair (such as "wants" and "boy"). While each arc represents a local relation, their combination implies the global input structure.
For certain types of graphs, a node can have multiple words. To deal with this situation, we use the first word of both associated graph nodes when grounding a graph edge onto the target sentence. Next, we also connect the first word of each grounded entity with the other words of the entity in order to represent the whole-entity information in the sentence. Taking the edge "followedBy" in Figure 1(b) as an example, we first ground it onto the target sentence to connect words "Above" and "Into". Next, we create edges with label "compound" from "Above" to words "the" and "Veil", and from "Into" to words "the" and "Battle" to indicate the two associated entity mentions.
For many tasks on graph-to-text generation, the node-to-word alignments can be easily generated from off-the-shell toolkits. For example, in AMR-to-text generation, there have been several aligners (Pourdamghani et al., 2014; Flanigan et al., 2016; Wang and Xue, 2017; Liu et al., 2018c; Szubert et al., 2018) available for linking AMR nodes to words. For knowledge graphs, the alignments can be produced by simple rule-based matching or an entity-linking system.
The resulting structure with labeled arcs connecting word pairs resembles a dependency tree, and thus we employ a deep biaffine model (Dozat and Manning, 2017) to predict this structure from the decoder states. More specifically, the model factorizes the probability for making each arc into two parts: an unlabeled factor and a labeled one. Given the decoder states  $s_1, \ldots, s_N$ , the representation for each word  $y_i$  as the head or the modifier of any unlabeled factor is calculated by passing its hidden state  $s_i$  through the corresponding multi-layer perceptrons (MLPs):
$$
\boldsymbol {r} _ {i} ^ {\text {a r c - h}} = \operatorname {M L P} ^ {\text {a r c - h e a d}} \left(\boldsymbol {s} _ {i}\right) \tag {6}
$$
$$
\boldsymbol {r} _ {i} ^ {\text {a r c - m}} = \operatorname {M L P} ^ {\text {a r c - m o d}} \left(\boldsymbol {s} _ {i}\right), \tag {7}
$$
The (unnormized) scores for the unlabeled factors with any possible head word given the modifier  $y_{i}$  are calculated as:
$$
\phi_ {i} ^ {\operatorname {a r c}} = \boldsymbol {R} ^ {\operatorname {a r c - h}} \boldsymbol {U} _ {a} ^ {\top} \boldsymbol {r} _ {i} ^ {\operatorname {a r c - m}} + \boldsymbol {R} ^ {\operatorname {a r c - h}} \boldsymbol {v} _ {a}, \tag {8}
$$
where  $R^{\mathrm{arc - h}}$  is the concatenation of all  $r_i^{\mathrm{arc - h}}$ , and  $U_{a}$  and  $v_{a}$  are model parameters. Similarly, the representations for word  $y_{i}$  being the head or the modifier of a labeled factor are calculated by
two additional MLPs:
$$
\boldsymbol {r} _ {i} ^ {\text {l a b e l - h}} = \mathrm {M L P} ^ {\text {l a b e l - h e a d}} (\boldsymbol {s} _ {i}) \tag {9}
$$
$$
\boldsymbol {r} _ {i} ^ {\text {l a b e l - m}} = \mathbb {M L P} ^ {\text {l a b e l - m o d}} (\boldsymbol {s} _ {i}), \tag {10}
$$
and the (unnormized) scores for all relation labels given the head word  $y_{j}$  and the modifier  $y_{i}$  are calculated as:
$$
\begin{array}{l} \boldsymbol {\phi} _ {i, j} ^ {\text {l a b e l}} = \boldsymbol {r} _ {j} ^ {\text {l a b e l - h}} \boldsymbol {U} _ {l} \boldsymbol {r} _ {i} ^ {\text {l a b e l - m}} + \\ \left(\boldsymbol {r} _ {j} ^ {\text {l a b e l - h}} \oplus \boldsymbol {r} _ {i} ^ {\text {l a b e l - m}}\right) ^ {\intercal} \boldsymbol {V} _ {l} + \boldsymbol {b} _ {l}, \tag {11} \\ \end{array}
$$
where  $\pmb{U}_l$ ,  $\pmb{V}_l$  and  $\pmb{b}_l$  are model parameters. The overall conditional probability of a labeled arc with label  $l$ , head word  $y_j$  and modifier  $y_i$  is calculated by the following chain rule:
$$
\begin{array}{l} p \left(y _ {j}, l \mid y _ {i}\right) = p \left(l \mid y _ {j}, y _ {i}\right) \cdot p \left(y _ {j} \mid y _ {i}\right) \\ = \operatorname {s o f t m a x} \left(\phi_ {i, j} ^ {\text {l a b e l}}\right) _ {[ l ]} \cdot \operatorname {s o f t m a x} \left(\phi_ {i} ^ {\mathrm {a r c}}\right) _ {[ j ]}, \tag {12} \\ \end{array}
$$
where  $[x]$  in the subscript represents choosing the  $x$ -th item from the corresponding vector.
As the final step, the loss for reconstructing this view is defined as the negative log-likelihood of all target arcs  $\pmb{E}^{\prime}$  (the grounded triples from  $\pmb{E}$ ):
$$
l _ {\text {a u t o 1}} = \sum_ {\left(y _ {j}, l, y _ {i}\right) \in \boldsymbol {E} ^ {\prime}} - \log p \left(y _ {j}, l \mid y _ {i}\right) \tag {13}
$$
4.2 Loss 2: Reconstructing Linearized Graphs with a Transformer Decoder
As a supplement to our first loss for reconstructing the local information of each grounded triple, we introduce the second loss for predicting the whole graph as a linearized sequence. To minimize the loss of the graph structural information caused by linearization, we adopt an algorithm based on depth-first traversal (Konstas et al., 2017), which inserts brackets to preserve graph scopes. One linearized AMR graph is shown in the red dotted box of Figure 2, where the node suffixes (such as "-01") representing word senses are removed.
One may argue that we could directly predict the original graph so that no structural information would be lost. However, each type of graphs can have their own parsing algorithm due to their unique properties (such as directed vs undirected, rooted vs unrooted, etc). Such an exact prediction will hurt the generality of the proposed approach. Conversely, our solution is general, as linearization works for most types of graphs. From Figure 2 we can observe that the inserted brackets clearly
infer the original graph structure. Besides, previous work (Iyer et al., 2017; Konstas et al., 2017) has shown the effectiveness of generating linearized graphs as sequences for graph parsing, which also confirms our observation.
Given a linearized graph represented as a sequence of tokens  $x_{1}, \ldots, x_{M}$ , where each token  $x_{i}$  can be a graph node, a edge label or a inserted bracket, we adopt another standard Transformer decoder (SADecoder  $g$ ) to produce the sequence:
$$
x _ {i}, \boldsymbol {t} _ {i} = \operatorname {S A D e c o d e r} _ {g} ([ \boldsymbol {S}; \boldsymbol {t} _ {1} \dots \boldsymbol {t} _ {i - 1} ], x _ {i - 1}), \tag {14}
$$
where  $S = (s_1 \dots s_N)$  denotes the concatenated states for the target sentence (Equation 4), and the loss for reconstructing this view is defined as the negative log-likelihood for the linearized graph:
$$
l _ {\text {a u t o 2}} = - \sum_ {i \in [ 1.. M ]} \log p \left(x _ {i} \mid \boldsymbol {t} _ {i}; \boldsymbol {\theta}\right), \tag {15}
$$
where  $\theta$  represents model parameters.
4.3 Discussion and Comparison
Our autoencoding modules function as detachable components based on the target-side decoder states, and thus this brings two main benefits. First, our approaches are not only orthogonal to the recent advances (Li et al., 2016; Kipf and Welling, 2017; Velicković et al., 2018) on the encoder side for representing graphs, but also flexible with other decoders based on multi-layer LSTM (Hochreiter and Schmidhuber, 1997) or GRU (Cho et al., 2014). Second, no extra error propagation is introduced, as our approach does not affect the normal sentence-decoding process.
In addition to the different aspects both losses focus on, each has some merits and disadvantages over the other. In terms of training speed, calculating Loss 1 can be faster than Loss 2, because predicting the triple relations can be done in parallel, while it is not feasible for generating a linearized graph. Besides, calculating Loss 1 suffers from less variances, as the triple relations are agnostic to the token order determined by input files. Conversely, graph linearization is highly sensitive to the input order. One major merit for Loss 2 is the generality, as node-to-word alignments may not be easily obtained, especially for multi-lingual tasks.
4.4 Training with Autoencoding Losses
The final training signal with both proposed autoencoding losses is formalized as:
$$
l _ {\text {f i n a l}} = l _ {\text {b a s e}} + \alpha l _ {\text {a u t o 1}} + \beta l _ {\text {a u t o 2}}, \tag {16}
$$
7991
where  $\alpha$  and  $\beta$  are coefficients for our proposed losses. Both coefficient values are selected by a development experiment.
We study the effectiveness of our autoencoding training framework on AMR-to-text generation and KG-to-text generation. BLEU (Papineni et al., 2002) and Meteor (Denkowski and Lavie, 2014) scores are reported for comparison. Following previous work, we use the multi-bleu.perl from Moses $^2$  for BLEU evaluation.
AMR datasets<sup>3</sup> We take LDC2015E86 that contains 16,833, 1,368 and 1,371 instances for training, development and testing, respectively. Each instance contains a sentence and an AMR graph. Following previous work, we use a standard AMR simplifier (Konstas et al., 2017) to preprocess our AMR graphs, and take the PTB-based Stanford tokenizer<sup>4</sup> to tokenize the sentences. The node-to-word alignments are produced by the ISI aligner (Pourdamghani et al., 2014). We use this dataset for our primary experiments. We also report our numbers on LDC2017T10, a later version of AMR dataset that has 36521, 1,368 and 1,371 instances for training, development and testing, respectively.