Venue: NAACL 2021

# Text Generation from Discourse Representation Structures

 Jiangming Liu   Shay B. Cohen     Mirella Lapata

Institute for Language, Cognition and Computation

School of Informatics, University of Edinburgh

jiangming.liu@ed.ac.uk, {scohen,mlap}@inf.edu.ac.uk

###### Abstract

We propose neural models to generate text from formal meaning representations based on Discourse Representation Structures (DRSs). DRSs are _document-level_ representations which encode rich semantic detail pertaining to rhetorical relations, presupposition, and co-reference _within_ and _across_ sentences. We formalize the task of neural DRS-to-text generation and provide modeling solutions for the problems of condition ordering and variable naming which render generation from DRSs non-trivial. Our generator relies on a novel sibling treeLSTM model which is able to accurately represent DRS structures and is more generally suited to trees with wide branches. We achieve competitive performance (59.48 BLEU) on the GMB benchmark against several strong baselines.

+
Footnote †: The main is going to play the piano. Tom may stop him.

## 1 Introduction

It is not uncommon for text generation systems to produce natural language output from intermediate semantic representations Yao et al. (2012); Takase et al. (2016). The literature presents several examples of generating text from logical forms underlying various grammar formalisms Wang (1980); Shieber et al. (1990); Carroll and Oepen (2005); White et al. (2007), typed lambda calculus Lu Ng (2011), Abstract Meaning Representations AMR; Flanigan et al. (2016); Konstas et al. (2017); Song et al. (2018); Beck et al. (2018); Damonte and Cohen (2019); Ribeiro et al. (2019); Zhu et al. (2019); Cai and Lam (2020); Wang et al. (2020), Discourse Representation Theory DRT; Basile and Bos (2011); Basile (2015), and Minimal Recursion Semantics MRS; Horvat et al. (2015); Hajdik et al. (2019).

In this work, we propose neural models to generate high-quality text from semantic representations based on Discourse Representation Structures (DRSs). DRSs are the basic meaning-carrying units in Discourse Representation Theory (DRT; Kamp 1981; Kamp and Reyle (1993); Asher and Lascarides (2003), a formal semantic theory designed to handle a variety of linguistic phenomena, including anaphora, presuppositions Van der Sandt (1992); Venhuizen et al. (2018), and temporal expressions within and across sentences. DRSs are scoped meaning representations, they capture the semantics of negation, modals, and quantification.

Figure 1 displays in box format the meaning representation for a discourse consisting of two sentences. The outermost box is a _segmented_ DRS expressing the rhetorical relation CONTRAST between box \(b_{1}\) representing the first sentence and box \(b_{4}\) representing the second sentence. Boxes \(b_{1}\) and \(b_{2}\) are DRSs, the top layers contain variables (e.g., \(x_{1}\), \(x_{2}\)) indicating discourse referents and the bottom layers contain conditions (e.g., Named(\(x_{3}\), "tom")) representing information about discourse referents. Variables and conditions have pointers (denoted by \(b\) in the figure) pointing to the boxes where they should be interpreted.1 Predicates are disambiguated to their Wordnet Fellbaum (1998) senses (e.g., male.n.02 and play.v.03).

Footnote 1: In Figure 1, \(b_{6}\) is a presuppositional box for the interpretation of the man in the context of the two-sentence discourse.

Although there has been considerable activity recently in developing models which analyze text in the style of DRT van Noord et al. (2018, 2019); Liu et al. (2019, 2018); Fancellu et al. (2019), attempts

to _generate_ text from DRSs have been few and far between (however see Basile 2015 and Narayan and Gardent 2014 for notable exceptions). This is primarily due to two properties of DRS-based semantic representations which render generation from them challenging. Firstly, DRS conditions are _unordered_ representing a _set_ (rather than a list).2 A hypothetical generator would have to produce the same output text for any DRSs which convey the same meaning but appear different due to their conditions having a different order (see Figures 1 and 2a which are otherwise identical but the order of conditions in boxes \(b_{1}\) and \(b_{4}\) varies). The second challenge concerns _variables_ and their prominent status in DRSs. Variables identify objects in discourse (such as entities and predicates), and are commonly used to model semantic phenomena including coreference, control constructions, and scope. In Figure 1, variables \(x,e,s,t,p\), and \(b\) denote entities, events, states, time, propositions and boxes, respectively. Variable names themselves are arbitrary and meaningless posing a challenge for learning. Our generator must verbalize different variable names to the same surface form. The meaning representations in Figures 1 and 2b are identical and both correspond to the same discourse except that the variables have been given different names (\(b_{5}\) in Figure 1 has been named \(b_{1}\) in Figure 2b, \(b_{1}\) is now \(b_{2}\), \(x_{1}\) is \(x_{2}\), \(e_{1}\) is \(e_{9}\), and so on).

Footnote 2: An exception are conditions in segmented DRSs whose order can be retrieved deterministically based on the arguments of theoretical relations. For example, given the relation BECAUSE(\(b_{1}\), \(b_{3}\)), we can assume that box \(b_{1}\) precedes \(b_{3}\).

These two problems are further compounded by the way DRSs are displayed, in a box-like format which is intuitive and easy to read but not convenient for modeling purposes. As a result, DRSs are often post-processed in a format that can be handled more easily by modern neural network models. For example, DRS variables and conditions are converted to clauses van Noord et al. (2018) or DRSs are modified to trees where each box is a subtree and conditions within the box correspond to children of the subtree Liu et al. (2019, 2018).

In this paper we propose novel solutions to condition ordering and variable naming. We argue that even though DRS conditions appear unordered, they have a _latent_ order due to biases in the way the training data is created. To give a concrete example, the Groningen Meaning Bank (GMB; Bos et al. 2017) provides the largest collection to date of English texts annotated with DRSs. These annotations were generated with the aid of a CCG parser Clark and Curran (2007); atomic DRS conditions were associated with CCG supertags and then semantically combined following the syntactic CCG derivations. Even annotators creating DRSs manually would be prone to follow a canonical order (e.g., listing named entities first, then verbal predicates and their thematic roles, and finally temporal conditions). We propose a graph-based model which learns to recover the latent order of conditions without explicitly enumerating all possible orders which can be prohibitive. We also handle variable names with a method which rewrites arbitrary indices to _relative_ ones which are in turn determined by the order of conditions.

Following previous work, we convert DRSs to a more amenable format. Specifically, we consider Discourse Representation Tree Structures (DRTSs; Liu et al. 2019) as the semantic representation input to our document generation task, and generate a sequence of words autoregressively. We adopt an encoder-decoder framework with a treeLSTM Tai et al. (2015) encoder and a standard LSTM Hochreiter and Schmidhuber (1997) decoder. Problematically, DRS trees are wide and the number of children for a given node can be as many as 180. It therefore becomes memory-consuming and sparse to assign a forget gate for each child as in the case of conventional (\(N\)-ary) treeLSTM Tai et al. (2015). We propose a variant which we call _Sibling_ treeLSTM that replaces \(N\) forget gates with a _parent_ gate and a _sibling_ gate. As a result, it reduces

Figure 2: DRS from Figure 1 with (a) shuffled conditions and (b) different variable names.

 

memory usage from \(O(N)\) to \(O(2)\), and is more suitable for modeling wide and flat trees.

Our contributions can be summarized as follows: (1) we formalize the task of neural DRS-to-text generation; (2) we provide solutions for the problems of condition ordering and variable naming, which render generation from DRS-based meaning representations non-trivial; and (3) propose a novel sibling treeLSTM model that can be also generally used to model wide tree structures. We make our code and datasets publicly available.3

Footnote 3: [https://github.com/LeonCrashCode/Discourse-Representation-Tree-Structure/tree/main/gmb/DRS-to-text](https://github.com/LeonCrashCode/Discourse-Representation-Tree-Structure/tree/main/gmb/DRS-to-text)

## 2 Problem Formulation

Let \(S\) denote a DRS-based meaning representation. The aim of DRS-to-text generation is to produce text \(T\) that verbalizes input meaning \(S\):

\[T^{*}=\operatorname*{arg\,max}_{T\in\mathbb{T}}P(T|S,\Theta),\]

where \(\mathbb{T}\) is the set of all possible texts, \(S\) has an arbitrary order of conditions and indexing of variables, and \(\Theta\) is the set of model parameters.

Our generation model is based on the encoder-decoder framework Bahdanau et al. (2015) and operates over tree structures. Moreover, prior to training, variable names are rewritten so that their (arbitrary) indices denote relative order of appearance. We propose a novel sibling TreeLSTM for encoding tree structures. The decoder is a sequential LSTM equipped with an attention mechanism generating word sequence \(T=[t_{0},t_{1},...,t_{m-1}]\), where \(m\) is the length of the text. At test time, DRS conditions are normalized, i.e., they are reordered following a canonical order learned from data, and used as input to our generation model.

We first describe our DRS-to-tree conversion and variable renaming procedures (Sections 2.1 and 2.2). We next present our tree-to-sequence generation model (Section 2.3), and explain how DRS conditions are ordered (Section 2.4).

### DRS-to-Tree Conversion

The algorithm of Liu et al. (2018) renders DRSs in a tree-style format. It constructs trees based on DRS conditions in the bottom box layers, without considering variables in the top layer. This results in oversimplified semantic representations and information loss (e.g., presuppositions cannot be handled). We improve upon their approach by merging variables in the top layer with variables in the bottom layer via introducing _special_ conditions.

We collect variables in top layers of DRS boxes to construct a dictionary \(d=\{v:b\}\), where \(v\) denotes a variable and \(b\) is a presupposition box label (e.g., \(x_{1}:b_{1}\)). We then move variables from the top to the bottom layer by expressing them as special conditions \(b:\) Ref(\(v\)) and placing them before conditions on variable \(v\). For example, \(b_{6}:x_{1}\) in Figure 1 becomes special condition \(b_{6}:\) Ref(\(x_{1}\)) and is placed before condition \(b_{6}:\) Pred(\(x_{1}\), male.n.02) in Figure 3(a).

Once top variables have been rewritten as special conditions, the resulting DRSs are converted into trees as shown in Figure3(b). Box variables (e.g., \(b_{1}\), \(b_{5}\)) become parent nodes, while conditions, which are also subtrees, become children.

Figure 4: DRSs with relative variables.

Figure 3: (a) Box-style DRSs; (b) Tree-style DRSs.

 



### Relative Variables

We rename variables with regard to their relative position in a given DRS following a predefined traversal order.

We obtain the sequence of _box variables_ by traversing DRSs in an outer-to-inner and left-to-right manner, e.g., \([b_{5},b_{1},b_{4},b_{3}]\) in Figure 1. For SDRSs, we replace variables in discourse relations with \(k_{i}\), where \(i\) denotes th \(i\)th box from left to right. For example "CONTRAST(\(b_{1},b_{4}\))" in Figure 1 is rewritten to "CONTRAST(\(k_{0},k_{1}\)). Variables and conditions within presupposition boxes are rewritten to \(B_{i}\), where \(i\in\mathbb{Z}\) denotes the distance of the current box to the presupposition box. For example, \(b_{1}:\text{Agent}(e_{1},x_{1})\) is rewritten to \(B_{0}:\text{Agent}(e_{1},x_{1})\) because it is in the current box \(b_{1}\), while \(b_{1}:\text{Pred}(x_{1}\), "male.n.02") is rewritten to \(B_{-2}:\text{Pred}(x_{1}\), "male.n.02") because it is in box \(b_{3}\) and two hops away from presupposition box \(b_{1}\). We use special label \(O\) for presupposition boxes pertaining to semantic content outwith the current DRS. For example, \(b_{6}:\text{Ref}(x_{1})\) is rewritten to \(O:\text{Ref}(x_{1})\) because it introduces a new presupposition box, and \(b_{6}:\text{Pred}(x_{1}\), male.n.02) is rewritten to \(O_{0}:\text{Pred}(x_{1}\), male.n.02) because the condition can only be interpreted in this new presupposition box (now \(O_{0}\) and previously \(b_{6}\)).

We obtain a sequence of _general variables_ by traversing conditions as they appear in the DRS. Variables introduced for the first time are denoted by their type (going from left-to-right), while subsequent mentions of the same variables are rewritten with relative indices denoting their distance from the position where they were first introduced. Take Figure 3(a) as an example. The sequence of general variables is [\(x_{1}\), \(x_{1}\), \(e_{1}\), \(e_{1}\), \(x_{1}\), \(x_{2}\), \(e_{1}\), \(x_{2}\), \(x_{2}\), \(t_{1}\), \(t_{1}\), \(e_{1}\), \(t_{1}\), \(x_{3}\), \(x_{3}\), \(e_{2}\), \(e_{2}\), \(x_{2}\), \(x_{3}\), \(x_{1}\), \(e_{2}\), \(x_{1}\), \(e_{2}\), \(e_{1}\)], and is rewritten to [\(X\), \(X_{0}\), \(E\), \(E_{0}\), \(E_{0}\), \(X_{0}\), \(X\), \(E_{0}\), \(X_{0}\), \(X_{0}\), \(T\), \(D_{0}\), \(E_{0}\), \(T_{0}\), \(X\), \(X_{0}\), \(E\), \(E_{0}\), \(E_{0}\), \(X_{0}\), \(X_{-2}\), \(E_{0}\), \(X_{-2}\), \(E_{0}\), \(E_{-1}\)]. The DRS from Figure 3(a) is shown in Figure 4 with relative variables.

### Generation Model

Our generation model is based on the encoder-decoder framework, where an encoder is used to encode input DRS trees and a decoder outputs a sequence of words. A limitation of sequential encoders is that they only allow sequential information propagation without considering the structure of the input (Ta et al., 2015; Wang et al., 2019). In our case, DRS tree structures are additionally wide (the longer a document, the wider the tree) and relatively flat (see Figure 3(b)). To better model these aspects, we propose a treeLSTM encoder which takes _sibling_ information into account.

As shown in Figure 5, the hidden representations of the sibling TreeLSTM cells are updated from preceding sibling and child nodes. More formally, the hidden representation for node \(j\) is given by:

\[u_{j}=\tanh(g^{u}([x_{j};h_{js};h_{jp}])) \tag{1}\] \[i_{j},o_{j}=\sigma(g^{io}([x_{j};h_{js};h_{jp}]))\] (2) \[f_{js}=\sigma(g^{f}_{s}([x_{j};h_{js}]))\] (3) \[f_{jp}=\sigma(g^{f}_{p}([x_{j};h_{jp}]))\] (4) \[c_{j}=i_{j}\cdot u_{j}+f_{js}\cdot c_{js}+f_{jp}\cdot c_{jp}\] (5) \[h_{j}=o_{j}\cdot\tanh(c_{j}), \tag{6}\]

where \(x_{j}\) is the token input representation, \(h_{js}\) is the hidden representation of the sibling node preceding \(j\), \(h_{jp}\) is the hidden representation of the last child of node \(j\) (Equation (1)), \(g^{*}\) are linear functions, and \(\sigma\) is a sigmoid function (Equations (2)-(4)). For each node \(j\), we obtain its cell input representation \(u_{j}\) (Equation (1)), its input gate \(i_{j}\) and output gate \(o_{j}\) (Equation (2)), and two forget gates \(f_{js}\) (Equation (3)) and \(f_{jp}\) (Equation (4)) for its neighbor cell and the last child cell, respectively. The memory of the current cell \(c_{j}\) (Equation (5)) is updated by the gated sum of its cell input representation and the memories of its neighbor and child cells. The hidden representation of current node \(h_{j}\) is computed with its output gate \(o_{j}\) (Equation (6)).

Figure 5: The sibling treeLSTM; grey boxes are hidden representations of nodes, black/red arrows represent sibling/parent information flow.



Finally, a DRS tree is represented by the hidden representations of its nodes \([h_{0},h_{1},...,h_{n^{\prime}-1}]\) as computed by the sibling treeLSTM (\(n^{\prime}\) denotes the number of nodes). The decoder is a standard LSTM with global attention Bahdanau et al. (2015).

### Condition Ordering

As discussed previously, DRSs at test time may exhibit an arbitrary order of conditions, which our model should be able to handle. Our solution is to to reorder conditions prior to generation by learning a latent canonical order from training data (e.g., to recover boxes \(b_{1}\) and \(b_{3}\) in Figure 1 from boxes \(b_{1}\) and \(b_{3}\) in Figure 2). More formally, given a set of conditions \(R_{set}\), we obtain an optimal ordering \(R=[r_{0},r_{1},...,r_{n-1}]\) such that:

\[R^{*}=\operatorname*{arg\,max}_{R\in\pi(R_{set})}\textsc{Score}_{\mathbb{K}}( R|R_{set}), \tag{7}\]

where \(\pi(R_{set})\) are all permutations of \(R_{set}\), and \(R^{*}\) is the order with the highest likelihood according to Score\({}_{\mathbb{K}}\). Here, \(\mathbb{K}\) parametrizes Score as "knowledge" we collect from our training data by observing canonical orders of conditions. Unfortunately, the time complexity of calculating Equation (7) is \(O(n!)\), we must enumerate all possible permutations for a set of conditions with \(n\) as large as \(180\). Since this is prohibitive, we resort to _graph ordering_ which allows us to recover the order of the conditions without enumeration.

Graph ConstructionWe construct a graph from the set of DRS conditions which we break down into graph nodes and edges. Conditions in DRSs can be _simple_ or _complex_ according to their type of arguments. A simple condition might have a relation name with two arguments (e.g., Named(\(x_{3}\), "tom") and Agent(\(e_{1}\), \(x_{3}\))), while a complex condition has a scoped name (e.g., possibility \(\diamond\)) and takes one or more DRSs as arguments. Simple conditions are denoted by a 3-tuple \((l_{s},a_{0},a_{1})\), where \(l_{s}\) is the condition name (e.g., Named and Agent) and \(a_{0}\) and \(a_{1}\) are its first and the second argument, respectively, which could be a variable or constant (e.g., \(e_{1}\), \(x_{3}\) and "piano.n.01"). Complex conditions are a 2-tuple (\(l_{c}\), \(V_{r}\)), where \(l_{c}\) is the scope name, and \(V_{r}\) the set of arguments scoped by the condition. For example, the set of arguments for the possibility scope (\(\diamond\)) in Figure 1 is \(\{e_{1}\), \(e_{2}\), \(x_{1}\), \(x_{3}\), "tom", "stop.v.05", "male.n."\(\}\).

Condition names become nodes in our graph. Simple conditions are further divided into constant and thematic nodes. Constant nodes are constructed by concatenating the relation name in the condition with the constant argument (e.g., condition Pred(\(x_{1}\), "male.n.02") becomes node Pred "male.n.02"). Thematic nodes correspond to the relation name of the thematic condition (e.g., Agent(\(e_{1}\), \(x_{1}\)) becomes the node "Agent"). Complex nodes correspond to the name of complex conditions (e.g., possibility \(\diamond\)).

We insert edges between graph nodes if these share arguments. For example, in Figure 6(b), there is an edge connecting node Pred "male.n.02" with Agent as they share argument \(x_{1}\). We label this edge with \(a_{1}\) to denote the fact that it is the second argument of Agent. Another edge is drawn between Pred "play.v.03" and Agent (as they share argument \(e_{1}\)) with label \(a_{0}\) denoting that this is the first argument of Agent. Edges between nodes are bidirectional, with inverse edges bearing the suffix "-of". Edges drawn between constant and complex nodes bear the label "Related", while edges between two constant nodes (with the same variables) bear the label "Equal" (we provide a more formal description in the Appendix).

Ordering ModelGiven graph \(G=(R_{set},E)\), where \(R_{set}=\{r_{0},r_{1},...,r_{n-1}\}\) is the set of nodes and \(E\) is the set of edges in \(G\), our model outputs \(R^{*}\) as the optimal order of \(R_{set}\).

As shown Figure 6(a), each node is a sequence of words. A BiLSTM is applied to obtain representation \(x_{i}\) of each node \(r_{i}=[w_{0}^{i},...,w_{m-1}^{i}]\):

\[x_{i}=\text{BiLSTM}([w_{0}^{i},...,w_{m-1}^{i}]). \tag{8}\]

We encode the graph with a Graph Convolutional Recurrent network (GCRN; Seo et al. 2018). For

Figure 6: (a) Conditions and their corresponding graph nodes (b) graph with inverse edges (shown in red).

 

each node \(r_{i}\), we collect information from neighbor hidden representations with a gate controling the information flow from neighbors to current nodes:

\[h^{\prime k}_{i}=\sum_{j}g^{k}_{j}\cdot h^{k-1}_{j}; \tag{9}\] \[g^{k}_{j}=\sigma(f([e_{ji},h^{k-1}_{i},h^{k-1}_{j}])), \tag{10}\]

where \(e_{ji}\) is the embedding of edges from node \(r_{j}\) to \(r_{i}\), and \(k\) is the recurrent step in the GRU. The node hidden representations are updated as:

\[h^{k}_{i}=\text{GRUCell}([x_{i};g^{k-1}_{G}],h^{k-1}_{i}) \tag{11}\] \[g^{k}_{G}=\text{GRUCell}(\frac{1}{n}\sum_{i}h^{k}_{i},g^{k-1}_{G}) \tag{12}\]

where \(g_{G}\) represents the hidden representation of the graph as the average of (hidden) node representations, and GRUCell denotes the gated recurrent cell function. We obtain the hidden representations of nodes in the final recurrent step (\(K\)) as \(H^{K}=\{h^{K}_{0},h^{K}_{1},...,h^{K}_{n-1}\}\).

Our _decoder_ obtains the orders with the highest probability. We avoid enumerating all possible permutations for a set of nodes by generating their order autoregressively with an LSTM-based Pointer Network (PN; Vinyals et al., 2015):

\[\textsc{Score}_{\mathbb{K}}(R|R_{set})= \text{PN}(R|R_{set},H^{K},\theta) \tag{13}\] \[\text{PN}(R|R_{set},H^{K},\theta)= \prod_{i}P(r_{i}|r_{<i},H^{K})\] (14) \[P(r_{i}|r_{<i},H^{K})=\operatorname*{softmax}(v^{\mathrm{T}} \tanh(W[h^{d}_{i};H^{K}])) \tag{15}\]

where \(\theta\) are the parameters of the Pointer Network, \(h^{d}_{i}\) is the \(i\)th step hidden representation of the Pointer Network, and \(v\), and \(W\) are parameters. Hidden representation \(h^{d}_{i}\) is updated by the input representation of the \((i-1)\)th ordered node: \(h^{d}_{i}=\text{LSTMCell}(x_{r_{i-1}},h^{d}_{i-1})\). All parameters are optimized with standard back-propagation.

## 3 Experiments

Our experiments were carried out on the Groningen Meaning Bank (GMB; Bos et al., 2017) which provides a large collection of English documents annotated with DRSs. We used the standard training, development, and test splits that come with the distribution of the corpus. All DRSs in the GMB were preprocessed into the tree-based format discussed in Section 2.1. We also extracted from the training data conditions and their order for training our graph ordering model. Dataset statistics are shown in Table 1.

### Condition Ordering

Models and SettingsBefore evaluating our generator per se, we assess the effectiveness of the proposed condition ordering model (see Section 2.4). Specifically we compare four kinds of graphs: **NoEdges**, is a graph without edges; **FullEdges**, is a complete graph where each pair of nodes has edges; **SiGraph**, is the proposed graph without bidirectional edges; and **BiGraph**, is the proposed graph with bidirectional edges (see Figure 6). We also consider **Counting**, a baseline model which greedily orders pairs of conditions according to their frequency of appearance in the training data (see the Appendix for details).

For all neural models the embedding dimension was 50 and the hidden dimension 300. The bidirectional LSTM used for representing the graph nodes has a single layer, and the recurrent step in the GCRN is 2 (\(K=2\)). We applied the Adam optimizer (Kingma and Ba, 2014). We use accuracy to measure the percentage of absolute orders which are predicted correctly and Kendall's \(\tau\) coefficient to measure the relationship between two lists of ordered items; \(\tau\) ranges from \(-1\) to \(1\), where \(-1\) means perfect inversion and 1 means perfect agreement.

ResultsTable 2 summarizes our results. SiGraph performs better than NoEdges (+14.83% accuracy), showing that edge information is helpful for the representation of nodes which are used to order conditions. FullEdges performs worse than SiGraph (\(-13.68\%\) accuracy), underlying the fact that graph structure matters (i.e., edges are helpful when connecting certain pairs of nodes). BiGraph achieves the best ordering performance by a large margin compared to SiGraph (+9.63 % accuracy). One possible reason is that bidirectionality ensures all nodes have incoming edges, which can be used to update the node representations.

\begin{table}
\begin{tabular}{l r r r} \hline Task & train & dev & test \\ \hline Generation & 7,970 & 992 & 1,038 \\ Condition Ordering & 133,332 & 16,493 & 17,624 \\ \hline \end{tabular}
\end{table}
Table 1: GMB dataset statistics; number of documents (generation) and number of different sequences of conditions (ordering).

 



### Ideal-World Generation

Models and SettingsWe first examine generation performance in an ideal setting where (gold standard) condition orders are given and the indices of variables are fixed.

We compared the proposed treeLSTM against **Seq**, a baseline sequence-to-sequence model which adopts a bidirectional LSTM as its encoder.4 Trees were linearized in a top-down and left-to-right fashion, \(X=[x_{0},x_{1},...x_{n-1}]\), where \(n\) is the tree length. We obtained hidden representations \(H=[h_{0},h_{1},...,h_{n-1}]\) of the input with:

Footnote 4: The length of the input tokens can be around 4,000.

\[[h_{0},h_{1},...,h_{n-1}]=\text{BiLSTM}([x_{0},x_{1},...,x_{n}])\]

In addition, we included various models with tree-based encoders: **ChildSum**, is the bidirectional childsum-treeLSTM encoder of Tai et al. (2015); it operates over right-branch binarized trees; **Nary**, is the bidirectional Nary-TreeLSTM of Tai et al. (2015), again over right-branch binarized trees;5 and **Sibling** is our bidirectional sibling-TreeLSTM. All models were equipped with the same LSTM decoder, global attention Bahdanau et al. (2015), and the copy strategy of See et al. (2017).

Footnote 5: We experimented with \(n\)-ary \((n>2)\) trees, but found that binary trees perform best. Right-branch binary trees are also empirically better than left-branch ones.

The embedding dimension was 300 and the hidden dimension 512. All encoders and decoders have 2 layers. The detailed settings are shown in the Appendix. We measure generation quality with case-insensitive BLEU Papineni et al. (2002).

ResultsTable 3 shows our results on the development dataset. Overall, treeLSTM models performs better (average \(+1.69\) BLEU) than sequence models. Navy performs better (\(+0.26\) BLEU) than ChildSum because the latter cannot model the order of children. Sibling performs best (74.22 BLEU), because it it not only encodes the tree structure but also keeps track of sequential information.

### Real-World Generation Models and Settings

Models and SettingsWe finally, present our results in a more realistic setting where both problems of condition ordering and variable naming must be addressed. We recover condition order using four approaches: a **Naive** method which has no special-purpose ordering mechanism; the order of conditions is random in the development/test sets and fixed in the training set; **Random**, the order of conditions is random in the training, development, and test sets; **Counting**, the order of conditions is recovered by the Counting method; **GraphOrder** recovers the order of conditions with **BiGraph**. All comparison systems employ variable renaming as introduced in Section 2.2. We report experiments with a sequence-to-sequence generator and our sibling-TreeLSTM.

\begin{table}
\begin{tabular}{l l} \hline \hline Models & BLEU \\ \hline Seq+Naive & 4.61 \\ Seq+Random & 24.34 (16.77) \\ Seq+Counting & 45.17 \\ Seq+GraphOrder & 55.57 \\ \hline Sibling+Naive & 6.98 \\ Sibling+Random & 43.43 (0.26) \\ Sibling+Counting & 49.54 \\ Sibling+GraphOrder & **58.73** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Real-world generation (dev set). For Random, we report average results after shuffling 5 times (variance shown in parentheses).

\begin{table}
\begin{tabular}{c c c c} \hline \hline Models & Acc (%) & \(\tau\) & Parameters \\ \hline Counting & 14.38 & 0.57 & — \\ \hline NoEdges & 44.64 & 0.75 & 6.1M \\ FullEdges & 45.79 & 0.75 & 6.1M \\ SiGraph & 59.47 & 0.85 & 6.1M \\ BiGraph & 69.10 & 0.89 & 6.1M \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results for condition ordering (dev set).

\begin{table}
\begin{tabular}{l l l} \hline \hline Models & BLEU & Parameters \\ \hline Seq & 71.79 & 47.4M \\ ChildSum & 72.98 (+1.19) & 47.1M \\ Narry & 73.24 (+1.45) & 49.2M \\ Sibling & **74.22 (+2.43)** & 49.2M \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ideal-world generation (dev set); improvements compared to Seq shown in parentheses.

\begin{table}
\begin{tabular}{l l l} \hline \hline Models & BLEU & Parameters \\ \hline Graph & 45.72 & 30.1M \\ Seq+GraphOrder & 55.28 & 32.4M + 6.1M \\ Sibling+GraphOrder & 59.26 & 34.5M + 6.1M \\ \hline \hline \end{tabular}
\end{table}
Table 5: Real-world generation (test set).



ResultsTable 4 summarizes our results on the development set. Naive performs poorly, indicating that both Seq and Sibling models are sensitive to the order of conditions. Random, has higher variance with Seq (\(+16.51\)) compared to Sibling. Hidden representations for each timestep in Seq are heavily influenced by all previous steps, which are sequentially encoded; subtrees are encoded as a unit in Sibling, which is a more global representation for capturing patterns. Overall, we observe that the order of conditions plays a key role in the generation: both Seq and Sibling models improve when ordering of conditions is explicitly incorporated (either with Counting or GraphOrder). We observe that the combination of Sibling with GraphOrder achieves the best results (\(58.73\) BLEU).

Table 5 presents our results on the test set. We compare our Sibling encoder against a sequential one. Both models are interfaced with GraphOrder. We also compare to a previous graph-to-text model Song et al. (2018); Damonte and Cohen (2019) which has been used for generating from AMRs. We converted DRSs to graphs following the method of Liu et al. (2020); graphs were encoded with a GCRN Seo et al. (2018) and decoded with an LSTM. As can be seen, Sibling+GraphOrder outperforms all comparison systems achieving a BLEU of \(59.26\). However, compared to ideal-world generation (see Table 3) there is still considerable room for improvement.

### Analysis

Figure 7 shows model performance on test set against DRS size (i.e., the number of nodes in a DRS tree). Perhaps unsurprisingly, we see that generation quality deteriorates with bigger DRSs (i.e., with >1,600 nodes).

While BLEU is frequently adopted as an automatic evaluation metric for genration tasks, it is somewhat problematic in our case as it merely calculates word overlap between generated and goldstandard text without assessing whether model output is faithful to the semantics of the input (i.e., the DRS meaning representations). To this effect, we present examples of text generated by our model, demonstrating how the DRS input constrains and affects the output text.

Figure 8 shows examples of text generation from the test set. In the first example, the model generates the word because from the rhetorical relation, BECAUSE(\(b_{10}\), \(b_{12}\)). Temporal information (highlighted in blue in the figure) is also accurately reflected in the generated text (self is inflected to its present tense form). In addition, the model tends to over-generate (e.g., the word dollar is mentioned twice) and sometimes misses out on important terminers (e.g., some). In the second example, the model generates the word themselves referring to the entities mentioned before, e.g., \(x_{29}\) equals to \(x_{27}\) which refers to nimates, resolving the coreference. In the third example, the model generates the modal verb must in accordance with the scope operator NEC (a shorthand for Necessity, \(\square\)). Also, the model generates all for food and goods corresponding to the Implication (IMP) condition (i.e., \(~{}\forall x(P(x)\to Q(x))\)).

## 4 Related Work

Much previous work has focused on text generation from formal representations of meaning focusing exclusively on isolated sentences or queries. The literature offers a collection of approaches to generating from AMRs most of which employ neural models and structured encoders Song et al. (2018); Beck et al. (2018); Damonte and Cohen (2019); Ribeiro et al. (2019); Zhu et al. (2019); Cai and Lam (2020); Wang et al. (2020). Other work generates text from structured query language (SQL) adopting either sequence-to-sequence Iyer et al. (2016) or graph-to-sequence models Xu et al. (2018).

Basile (2015) was the first to attempt generation from DRT-based meaning representations. He proposes a pipeline system which operates over graphs and consists of three components: an alignment module learns the correspondence between surface text and DRS structure, an ordering module determines the relative position of words and phrases in the surface form and a realizer generates the final text. Narayan and Gardent (2014) simplify complex sentences with a two-stage model which

Figure 7: BLEU score against DRS size (test set).

 

first performs sentence splitting and deletion operations over DRSs and then uses a phrase-based machine translation model for surface realization.

Our work is closest to Basile (2015); we share the same goal of generating from DRSs, however, our model is trained end-to-end and can perform long-form generation for documents and sentences alike. We also adopt an ordering component, but we order DRS conditions rather than lexical items, and propose a model capable of inferring a global order. There has been long-standing interest in information ordering within NLP Lapata (2003); Abend et al. (2015); Chen et al. (2016); Gong et al. (2016); Logeswaran et al. (2018); Cui et al. (2018); Yin et al. (2019); Honovich et al. (2020). Our innovation lies in conceptualizing ordering as a graph scoring task which can be further realized with graph neural network models Wu et al. (2020).

## 5 Conclusions

In this paper, we have focused on _document-level_ generation from formal meaning representations. We have adopted DRT as our formalism of choice and highlighted various challenges associated with the generation task. We have introduced a novel sibling treeLSTM for encoding DRSs rendered as trees and shown it is particularly suited to trees with wide branches. We have experimentally demonstrated that our encoder coupled with a graph-based condition ordering model outperforms strong comparison systems. In the future, we would like to embed our generator in practical applications such as summarization and question answering.

## Acknowledgments

We thank the anonymous reviewers for their feedback. We gratefully acknowledge the support of the European Research Council (Lapata, Liu; award number 681760), the EU H2020 project SUMMA (Cohen, Liu; grant agreement 688139) and Bloomberg (Cohen, Liu).

## References

* Ahend et al. (2015) Omri Abend, Shay B. Cohen, and Mark Steedman. 2015. Lexical event ordering with an edge-factored model. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech

Figure 8: DRS example from test set with gold and automatically generated text by ours (Sibling+GraphOrder). Temporal information marked in blue, rhetorical relations marked in red, co-reference marked as green, and scope marked in brown.

 

nologies (NAACL-HLT)_, pages 1161-1171, Denver, Colorado, USA.
* Asher and Lascarides [2003] Nicholas Asher and Alex Lascarides. 2003. _Logics of conversation_. Cambridge University Press.
* Bahdanau et al. [2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of the 4th International Conference on Learning Representations (ICLR)_, San Diego, California.
* Basile [2015] Valerio Basile. 2015. _From Logic to Language: Natural Language Generation from Logical Forms_. Ph.D. thesis, University of Groningen, Netherlands.
* Basile and Bos [2011] Valerio Basile and Johan Bos. 2011. Towards generating text from discourse representation structures. In _Proceedings of the 13th European Workshop on Natural Language Generation_, pages 145-150, Nancy, France.
* Beck et al. [2018] Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 273-283, Melbourne, Australia.
* Bos et al. [2017] Johan Bos, Valerio Basile, Kilian Evang, Noortje J Venhuizen, and Johannes Bjerva. 2017. The Groningen meaning bank. In _Handbook of Linguistic Annotation_, pages 463-496. Springer.
* Cai and Lam [2020] Deng Cai and Wai Lam. 2020. Graph transformer for graph-to-sequence learning. In _Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI)_, pages 7464-7471, New York, USA.
* Carroll and Oepen [2005] John Carroll and Stephan Oepen. 2005. Highefficiency realization for a wide-coverage unification grammar. In Robert Dale and Kam-Fai Wong, editors, _Proceedings of the 2nd International Joint Conference on Natural Language Processing (IJCNLP)_, volume 3651, pages 165-176. Springer, Jeju, Korea.
* Chen et al. [2016] Xinichi Chen, Xipeng Qiu, and Xuanjing Huang. 2016. Neural sentence ordering. _arXiv preprint arXiv:1607.06952_.
* Clark and Curran [2007] Stephen Clark and James Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. _Computational Linguistics_, 33(4):493-552.
* Cui et al. [2018] Baiyun Cui, Yingming Li, Ming Chen, and Zhongfei Zhang. 2018. Deep attentive sentence ordering network. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4340-4349, Brussels, Belgium.
* Damonte and Cohen [2019] Marco Damonte and Shay B. Cohen. 2019. Structural neural encoders for AMR-to-text generation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 3649-3658, Minneapolis, USA.
* Fancelllu et al. [2019] Federico Fancelltu, Sorcha Gilroy, Adam Lopez, and Mirella Lapata. 2019. Semantic graph parsing with recurrent neural network DAG grammars. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JUNLP)_, pages 2769-2778, Hong Kong, China.
* Fellbaum [1998] Christiane Fellbaum, editor. 1998. _WordNet: An Electronic Database_. MIT Press, Cambridge, MA.
* Flanigan et al. [2016] Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime G. Carbonnell. 2016. Generation from abstract meaning representation using tree transducers. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 731-739, San Diego, California, USA.
* Gong et al. [2016] Jingjing Gong, Xinchi Chen, Xipeng Qiu, and Xuaming Huang. 2016. End-to-end neural sentence ordering using pointer network. _arXiv preprint arXiv:1611.04953_.
* Hajdijk et al. [2019] Valerie Hajdijk, Jan Buys, Michael Wayne Goodman, and Emily M. Bender. 2019. Neural text generation from rich semantic representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 2259-2266, Minneapolis, Minnesota.
* Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Honovich et al. [2020] Or Honovich, Lucas Torroba Hennigen, Omri Abend, and Shay B. Cohen. 2020. Machine reading of historical events. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7486-7497, Online. Association for Computational Linguistics.
* Horvat et al. [2015] Matic Horvat, Ann Copestake, and Bill Byrne. 2015. Hierarchical statistical semantic realization for Minimal Recursion Semantics. In _Proceedings of the 11th International Conference on Computational Semantics (IWCS)_, pages 107-117, London, UK. Association for Computational Linguistics.
* Iyer et al. [2016] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 2073-2083, Berlin, Germany.

 

* Kamp (1981) Hans Kamp. 1981. A theory of truth and semantic representation. In J. A. G. Groenendijk, T. M. V. Janssen, and M. B. J. Stokhof, editors, _Formal Methods in the Study of Language_, volume 1, pages 277-322. Mathematisch Centrum, Amsterdam.
* Kamp and Reyle (1993) Hans Kamp and Uwe Reyle. 1993. _From Discourse to Logic; An Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory_. Kluwer, Dordrecht.
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_, Banff, Canada.
* Konstas et al. (2017) Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 146-157, Vancouver, Canada.
* Lapata (2003) Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In _Proceedings of the 41st Annual Meeting on Association for Computational Linguistics (ACL)_, pages 545-552, Sapporo, Japan.
* Liu et al. (2018) Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2018. Discourse representation structure parsing. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 429-439, Melbourne, Australia.
* Liu et al. (2019a) Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2019a. Discourse representation parsing for sentences and documents. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 6248-6262, Florence, Italy.
* Liu et al. (2019b) Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2019b. Discourse representation structure parsing with recurrent neural networks and the transformer model. In _Proceedings of the IWCS Shared Task on Semantic Parsing_, Gothenburg, Sweden. Association for Computational Linguistics.
* Liu et al. (2020) Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2020. Dscorer: A fast evaluation metric for discourse representation structure parsing. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 4547-4554.
* Logeswaran et al. (2018) Lajangen Logeswaran, Honglak Lee, and Dragomir Radev. 2018. Sentence ordering and coherence modeling using recurrent neural networks. In _Proceedings of the 32nd AAAI Conference on Artificial Intelligence_, pages 5285-5292, New Orleans, Louisiana, USA.
* Lu and Ng (2011) Wei Lu and Hwee Tou Ng. 2011. A probabilistic forest-to-string model for language generation from typed lambda calculus expressions. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1611-1622, Edinburgh, Scotland, UK. Association for Computational Linguistics.
* Narayan and Gardent (2014) Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 435-445, Baltimore, Maryland. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL)_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Ribeiro et al. (2019) Leonardo F.R. Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing AMR-to-text generation with dual graph representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langauge Processing (EMNLP-IJCNLP)_, pages 3174-3185, Hong Kong, China.
* See et al. (2017) Agigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1073-1083, Vancouver, Canada.
* Seo et al. (2018) Youngjoo Seo, Michael Defferrard, Pierre Vandergheynst, and Xavier Bresson. 2018. Structured sequence modeling with graph convolutional recurrent networks. In _Neural Information Processing Systems (NeurIPS)_, pages 362-373, Montreal, Canada. Springer.
* Shieber et al. (1990) Stuart M. Shieber, Gertijan van Noord, Fernando C. N. Pereira, and Robert C. Moore. 1990. Semantic-head-driven generation. _Computational Linguistics_, 16(1):30-42.
* Song et al. (2018) Linfeng Song, Yue Zhang, Zhiguuo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1616-1626, Melbourne, Australia.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP)_, pages 1556-1566, Beijing, China 

Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hirao, and Masaaki Nagata. 2016. Neural headline generation on Abstract Meaning Representation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), Austin, Texas. Association for Computational Linguistics.
* Van der Sandt (1992) Rob A. Van der Sandt. 1992. Presupposition projection as anaphora resolution. Journal of Semantics9(4):333-377.
* van Noord et al. (2018) Rik van Noord, Lasha Abzianidze, Antonio Toral, and Johan Bos. 2018. Exploring neural methods for parsing discourse representation structures. Transactions of the Association for Computational Linguistics, 6:619-633.
* van Noord et al. (2019) Rik van Noord, Antonio Toral, and Johan Bos. 2019. Linguistic information in neural semantic parsing with multiple encoders. In Proceedings of the 13th International Conference on Computational Semantics (IWCS), Gothenburg, Sweden.
* Venhuizen et al. (2018) Noortje J. Venhuizen, Johan Bos, Petra Hendriks, and Harn Brouwer. 2018. Discourse semantics with information structure. Journal of Semantics35(1):127-169.
* Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Neural Information Processing Systems (NeurIPS), Montreal, Canada.
* Wang (1980) Juen-tin Wang. 1980. On computational sentence generation from logical form. In Proceedings of the 8th International Conference on Computational Linguistics (COLING), Tokyo, Japan.
* Wang et al. (2020) Tianming Wang, Xiaojun Wan, and Hanqi Jin. 2020. AMR-to-text generation with graph transformer. Transactions of the Association for Computational Linguistics, 8:19-33.
* Wang et al. (2019) Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019. Tree transformer: Integrating tree structures into self-attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.
* White et al. (2007) Michael White, Rajakrishnan Rajkumar, and Scott Martin. 2007. Towards broad coverage surface realization with CCG. In Proceedings of the Workshop on Using Corpora for NLG: Language Generation and Machine Translation (UCNLG+MT), Copenhagen, Denmark.
* Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems.
* Xu et al. (2018) Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, and Vidam Sheinin. 2018. SQL-to-text generation with graph-to-sequence model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium.
* Yao et al. (2012) Xuchen Yao, Gosse Bouma, and Yi Zhang. 2012. Semantics-based question generation and implementation. Dialogue and Discourse2(3):11-42.
* Yin et al. (2019) Yongjing Yin, Linfeng Song, Jinsong Su, Jiali Zeng, Chulun Zhou, and Jiebo Luo. 2019. Graph-based neural sentence ordering. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), Macao, China.
* Zhu et al. (2019) Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019. Modeling graph structure in transformer for better AMR-to-text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.

 

Counting Method

We count how frequently condition \(r_{i}\) appears before \(r_{j}\) with type \(t\). Type \(t\) is identified according to the overlap between the arguments of the two conditions6. For example, \(r_{i}\) = (Named, \(x_{3}\), "tom") and \(r_{j}\) = (Agent, \(e_{1}\), \(x_{3}\)) has the type "\(a_{0}\to a_{1}\)", showing that the first argument in \(r_{i}\) equals to the second argument in \(r_{j}\). We score the order of two conditions using the following function:

Footnote 6: We only consider (and count) conditions with overlapping rguments, i.e., their type \(t\) is not None.

\[\begin{split}\textsc{Score}_{\mathbb{K}}(R|R_{set})=& \sum_{i}(\sum_{i<j}\textsc{Count}_{\mathbb{K}}(r_{i},r_{j}))\\ -&\sum_{j<i}\textsc{Count}_{\mathbb{K}}(r_{i},r_{ j})\end{split} \tag{16}\]

where Count returns the frequency of a pair of conditions subject to a dataset or corpus \(\mathbb{K}\); the score increases with \(r_{j}\) following \(r_{i}\) more frequently than preceding it.

### Types

We define different types of relations between two conditions based on argument overlap (i.e., two simple conditions, a simple and a complex condition, and two complex conditions).

Simple and SimpleGiven two simple conditions, \(r_{i}=(l_{i},a_{0i},a_{1i})\) and \(r_{j}=(l_{j},a_{0j},a_{1j})\) (with different arguments), we define their types as:

* \(t=a_{0}\to a_{0}\) if \(a_{0i}=a_{0j}\) and \(a_{1i}\neq a_{1j}\).
* \(t=a_{1}\to a_{1}\) if \(a_{0i}\neq a_{0j}\) and \(a_{1i}=a_{1j}\).
* \(t=a_{0}\to a_{0},a_{1}\to a_{1}\) if \(a_{0i}=a_{0j}\) and \(a_{1i}=a_{1j}\).
* \(t=a_{0}\to a_{1}\) if \(a_{0i}=a_{1j}\) and \(a_{1i}\neq a_{0j}\).
* \(t=a_{1}\to a_{0}\) if \(a_{1i}=a_{0j}\) and \(a_{0i}\neq a_{1j}\).
* \(t=a_{0}\to a_{1},a_{1}\to a_{0}\) if \(a_{1i}=a_{0j}\) and \(a_{0i}=a_{1j}\).
* \(t=\textit{None}\) if others

Simple and ComplexGiven a simple conditions \(r_{i}=(l_{i},a_{0i},a_{1i})\) and a complex condition \(r_{j}=(l_{j},V_{j})\), the types are defined as:

* \(t=1\) if \(a_{0i}\in V_{j}\) and \(a_{1i}\not\in V_{j}\).
* \(t=2\) if \(a_{0i}\in V_{j}\) and \(a_{1i}\in V_{j}\).
* \(t=\textit{None}\) if others

Complex and ComplexGiven two complex conditions, \(r_{i}=(l_{i},V_{i})\) and \(r_{j}=(l_{j},V_{j})\), the types are defined as:

* \(t=\textit{intersection}\) if \(V_{i}\cap V_{j}\neq\O\).
* \(t=\textit{None}\) otherwise

### Greedy Algorithm

We generate an ordering of conditions following greedy Algorithm 1 which identifies the highest scoring pair in \(R_{set}\), appends it to partial ordering \(R^{*}\), and keeps going until \(R_{set}\) is empty. The score is given by the function:

\[\begin{split}\textsc{Partial}_{\mathbb{K}}(r)=& \sum_{r^{\prime}\in R_{set}}\textsc{Count}(r,r^{\prime})_{ \mathbb{K}}\\ &-\sum_{r^{\prime\prime}\in R}\textsc{Count}(r,r^{\prime\prime}) _{\mathbb{K}},\end{split} \tag{17}\]

## Appendix B Edge Construction Algorithm

Algorithm 2 shows how edges are created for graph ordering.

```
Input:\(R_{set}\), set of conditions Output:\(R^{*}\), list of ordered conditions
1:\(R^{*}=[]\)
2:while\(R_{set}\) is not empty do
3:\(r^{*}\mathrel{\mathop{\arg\max}\limits_{F\in R_{set}}}\textsc{Partial}(r)\)
4:\(R_{set}\) removes \(r^{*}\)
5:\(R^{*}\) appends \(r^{*}\)
6:end while
```

**Algorithm 1**Greedy Ordering Algorithm

## Appendix C Model Settings

In the following, we report the best experimental settings for our condition ordering and text generation models.

### Condition Ordering Models

Model hyperparameters are shown in Table 6. The size of the token embeddings (in the graph nodes) and edge embeddings is 100. The node hidden dimension is the same as the hidden dimension of the BILSTM which is used to encode a sequence of input words for each node and the hidden dimension of PointerNet. The BILSTM and PointerNet have one layer. Training hyperparameters are shown in Table 7 



### Text Generation Models

Model hyperparameters are shown in Table 8. The size of the input embeddings in the encoder and the decoder is 300. The hidden dimensions of the encoder and decoder are 512. Both the encoder and decoder have two layers. The hyperparameters of the training are shown in Table 9.

## Appendix D Hyperparameter Tuning

We show below model performance with various hyperparameters. Best hyperparameters were manually chosen after monitoring model accuracy on the development set.

We take **BiGraph** as our final condition ordering model (see Table 10) and **Sibling** with **BiGraph** as the DRS-to-text generation model (see Table 11).

\begin{table}
\begin{tabular}{l c} \hline \hline hyperparameters & value \\ \hline embedding dim & 50 \\ hidden dim & 300 \\ LSTM layer & 1 \\ graph step & 2 \\ beam size & 64 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Hyperparameters of condition ordering model.

\begin{table}
\begin{tabular}{l c} \hline \hline hyperparameters & value \\ \hline optimizer & adam \\ learning rate & 0.001 \\ \(\beta\) & (0.9, 0.999) \\ dropout rate & 0.5 \\ batch size & 32 \\ max gradient norm & 5 \\ training steps & 30,000 \\ learning rate decay & 0.5 \\ start decay steps & 8,000 \\ decay steps & 1,000 \\ warmup steps & 4,000 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Training hyperparameters for the condition ordering models.

\begin{table}
\begin{tabular}{l c} \hline \hline hyperparameters & value \\ \hline optimization & adam \\ learning rate & 0.001 \\ \(\beta\) & (0.9, 0.999) \\ dropout rate & 0.5 \\ batch size & 32 \\ max gradient norm & 5 \\ training steps & 400,000 \\ learning rate decay & 0.3 \\ start decay steps & 200,000 \\ decay steps & 20,000 \\ warmup steps & 40,000 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Hyperparameters of text generation model.

\begin{table}
\begin{tabular}{l c} \hline \hline hyperparameters & value \\ \hline embedding dim & 300 \\ hidden dim & 512 \\ layer & 2 \\ attention & general \\ beam size & 5 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Training hyperparameters for text generation model.





## Appendix E Examples

We provide example output of our final model (**Sibling+GraphOrder**) on the GMB test dataset. The DRS in tree format with condition ordering given by **GraphOrder** is shown in Figures 9-11. Figure 10 expands nonterminal \(b_{33}\) in Figure 9, and Figure 11 expands nonterminal \(b_{28}\) in Figure 10. The corresponding document generated by **Sibling** is:

the u.s. dollar hit a record low against the euro tuesday . it took a dollar , but 48 cents to buy one euro , and a series of problems including the key of the u.s. housing sector in which has been battered by the slowing n.01 , including continuing the u.s. economy , the dollar . speculator are selling dollars because they expect that the u.s. central bank will try to stimulate the economy by cutting interest rates soon . u.s. lower interest rates can cut the return on investments . the falling dollar is prompting oil-rich nations around the persian gulf to consider ending the practice of linking the value of its currency to those of the dollar and instead supplement the u.s. currency . such a move would reduce demand for dollars and weaken the u.s. currency .





\begin{table}
\begin{tabular}{c c c c c|l} \hline \hline embedding & hidden & layer & dropout & beam & BLEU \\ \hline
300 & 512 & 2 & 0.5 & 5 & 58.97 \\ \hline
100 & — & — & — & — & 58.19 \\
200 & — & — & — & — & 58.81 \\ \hline — & 128 & — & — & — & 43.36 \\ — & 256 & — & — & — & 55.06 \\ \hline — & — & 1 & — & — & 52.58 \\ — & — & 3 & — & — & 58.94 \\ \hline — & — & — & 0.1 & — & 47.43 \\ — & — & — & 0.2 & — & 58.24 \\ — & — & — & 0.3 & — & 58.97 \\ — & — & — & 0.4 & — & 58.96 \\ — & — & — & 0.6 & — & 58.22 \\ \hline — & — & — & — & 1 & 58.73 \\ — & — & — & — & 3 & 58.91 \\ — & — & — & — & 10 & 58.97 \\ \hline \end{tabular}
\end{table}
Table 11: Performance of text generation model (development set; various hyperparameters). Increasing model size leads to out-of-memory problems when training on a single GPU.



Figure 9: A partial DRS in tree format with condition ordering recovered by GraphOrder.

 

Figure 10: A partial DRS in tree format with condition ordering recovered by **GraphOrder**.

 

Figure 11: A partial DRS in tree format with condition ordering recovered by **GraphOrder**.