Venue: NAACL 2016

# Generation from Abstract Meaning Representation using Tree Transducers

Jeffrey Flanigan\({}^{\spadesuit}\) Chris Dyer\({}^{\clubsuit}\) Noah A. Smith\({}^{\heartsuit}\) Jaime Carbonell\({}^{\spadesuit}\)

\({}^{\spadesuit}\)School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA

\({}^{\heartsuit}\)Computer Science & Engineering, University of Washington, Seattle, WA, USA

{jflanigan,cdyer,jgc}@cs.cmu.edu, nasmith@cs.washington.edu

###### Abstract

Language generation from purely semantic representations is a challenging task. This paper addresses generating English from the Abstract Meaning Representation (AMR), consisting of re-entrant graphs whose nodes are concepts and edges are relations. The new method is trained statistically from AMR-annotated English and consists of two major steps: (i) generating an appropriate spanning tree for the AMR, and (ii) applying tree-to-string transducers to generate English. The method relies on discriminative learning and an argument realization model to overcome data sparsity. Initial tests on held-out data show good promise despite the complexity of the task. The system is available open-source as part of JAMR at: [http://github.com/jlanigan/jamr](http://github.com/jlanigan/jamr)

## 1 Introduction

We consider natural language generation from the Abstract Meaning Representation (AMR; Banarescu et al., 2013). AMR encodes the meaning of a sentence as a rooted, directed, acyclic graph, where concepts are nodes, and edges are relationships among the concepts.

Because AMR models propositional meaning1 while abstracting away from surface syntactic realizations, and is designed with human annotation in mind, it suggests a separation of (i) engineering the application-specific propositions that need to be communicated about from (ii) general-purpose realization details, modeled by a generator shareable across many applications. The latter is our focus here.

Footnote 1: In essence, AMR handles semantic roles, entity types, within-sentence coreference, discourse connectives, modality, negation, and some other phenomena.

Because any AMR graph has numerous valid realizations, and leaves underspecified many important details--including tense, number, definiteness, whether a concept should be referred to nominally or verbally, and more--transforming an AMR graph into an English sentence is a nontrivial problem.

To our knowledge, our system is the first for generating English from AMR. The approach is a statistical natural language generation (NLG) system, trained discriminatively using sentences in the AMR bank Banarescu et al. (2013). It first transforms the graph into a tree, then decodes into a string using a weighted tree-to-string transducer and a language model Graehl and Knight (2004). The decoder bears a strong similarity to state-of-the-art machine translation systems Koehn et al. (2007); Dyer et al. (2010), but with a rule extraction approach tailored to the NLG problem.

## 2 Overview

Generation of English from AMR graphs is accomplished as follows: the input graph is converted to a tree, which is input into the weighted intersection of a tree-to-string transducer (SS4) with a language model. The output English sentence is the (approximately) highest-scoring sentence according to a feature-rich discriminatively trained linear model. After discussing notation (SS3), we describe our approach in SS4. The transducer's rules are extracted from the limited AMR corpus and learned general

izations; they are of four types: **basic rules** (SS5), **synthetic rules** created using a specialized model (SS6), **abstract rules** (SS7), and a small number of **handwritten rules** (SS8).

## 3 Notation and Definitions

AMR graphs are directed, weakly connected graphs with node labels from the set of **concepts**\(L_{N}\) and edge labels from the set of **relations**\(L_{E}\).

AMR graphs are transformed to eliminate cycles (details in SS4); we refer to the resulting tree as a **transducer input representation (TI representation)**. For a node \(n\) with label \(\mathcal{C}\) and outgoing edges \(n\xrightarrow{L_{1}}n_{1},\ldots,n\xrightarrow{L_{m}}n_{m}\) sorted lexicographically by \(L_{i}\) (each an element of \(L_{E}\)), the TI representation of the tree rooted at \(n\) is denoted:2

Footnote 2: If there are duplicate child edge labels, then the conversion process is ambiguous and any of the conversions can be used. The ordering ambiguity will be handled later in the tree-transducer rules.

\[(X~{}\mathcal{C}~{}(L_{1}~{}T_{1})\dots(L_{m}~{}T_{m})) \tag{1}\]

where each \(T_{i}\) is the TI representation of the tree rooted at \(n_{i}\). See Fig. 1 for an example. A LISP-like textual formatting of the TI representation in Fig. 1 is:

\[(X~{}\text{want-01}~{}(\textit{ARG0}~{}(X~{}\text{boy}))~{}( \textit{ARG1}~{}(X~{}\text{ride-01}\\ (\textit{ARGO}~{}(X~{}\text{bicycle}~{}(\textit{mod}~{}(X~{} \text{red})))))))\]

To ease notation, we use the function \(sort[]\) to lexicographically sort edge labels in a TI representation. Using this function, an equivalent way of representing the TI representation in Eq. 1, if the \(L_{i}\) are unsorted, is:

\[(\mathrm{X}~{}\mathcal{C}~{}sort[(L_{1}~{}T_{1})\dots(L_{m}~{}T_{m})])\]

The TI representation is converted into a word sequence using a tree-to-string transducer. The tree transducer formalism we use is one-state extended linear, non-deleting tree-to-string (**1-xRLNs**) transducers [9, 10].3

Footnote 3: Multiple states would be useful for modeling dependencies in the output, but we do not use them here.

**Definition 1**.: _(From Huang et al., 2006.) A **I-xRLNs transducer** is a tuple \((N,\Sigma,W,\mathcal{R})\) where \(N\) is the set of nonterminals (relation labels and \(X\)), \(\Sigma\) is the input alphabet (concept labels), \(W\) is the output alphabet (words), and \(\mathcal{R}\) is the set of rules. A rule in \(\mathcal{R}\) is a tuple \((t,s,\phi)\) where:_

1. \(t\) _is the LHS tree, whose internal nodes are labeled by nonterminal symbols, and whose frontier nodes are labeled terminals from_ \(\Sigma\) _or variables from a set_ \(\mathcal{X}=\{X_{1},X_{2},\dotsc\}\)_;_

Figure 1: The generation pipeline. An AMR graph (top), with a deleted re-entrancy (dashed), is converted into a transducer input representation (TI representation, middle), which is transduced to a string using a tree-to-string transducer (bottom).

 

2. \(s\in({\cal X}\cup W)^{*}\) _is the RHS string;_
3. \(\phi\) _is a mapping from_ \({\cal X}\) _to nonterminals_ \(N\)_._

A rule is a **purely lexical rule** if it has no variables.

As an example, the tree-to-string transducer rules which produce the output sentence from the TI representation in Fig. 1 are:

\[(X\mbox{ want-01 $(ARG0~{}X_{1})$ (ARG1~{}X_{2})$})\rightarrow\] \[\qquad\quad\mbox{The $X_{1}$ wants to $X_{2}$ }.\] \[(X\mbox{ ride-01 $(ARG1~{}X_{1})$})\rightarrow\mathrm{ride~{}the $X_{1}$}\] \[(X\mbox{ bicycle $(mod~{}X_{1})$})\to X_{1}\;\mathrm{bicycle}\] \[(X\mbox{ red})\rightarrow\mathrm{red}\] \[(X\mbox{ boy})\rightarrow\mathrm{boy} \tag{2}\]

Here, all \(X_{i}\) are mapped by a trivial \(\phi\) to the nonterminal \(X\).

The output string of the transducer is the target projection of the derivation, defined as follows:

**Definition 2**.: _(From Huang et al., 2006.) A **derivation**\(d\), its **source and target projections**, denoted \({\cal S}(d)\) and \({\cal E}(d)\) respectively, are recursively defined as follows:_

1. _If_ \(r=(t,s,\phi)\) _is a purely lexical rule, then_ \(d=r\) _is a derivation, where_ \({\cal S}(d)=t\) _and_ \({\cal E}(d)=s\)_;_
2. _If_ \(r=(t,s,\phi)\) _is a rule, and_ \(d_{i}\) _is a (sub)-derivation with the root symbol of its source projection maching the corresponding substitution node in_ \(r\)_, i.e.,_ \(\text{root}({\cal S}(d_{i}))=\phi(x_{i})\)_, then_ \(d=r(d_{1},\ldots,d_{m})\) _is also a derivation, where_ \({\cal S}(d)=[x_{i}\mapsto{\cal S}(d_{i})]t\) _and_ \({\cal E}(d)=[x_{i}\mapsto{\cal E}(d_{i})]s\)_._

The notation \([x_{i}\mapsto y_{i}]t\) is shorthand for the result of substituting \(y_{i}\) for each \(x_{i}\) in \(t\), where \(x_{i}\) ranges over all variables in \(t\).

The set of all derivations of a target string \(e\) with a transducer \(T\) is denoted

\[{\cal D}(e,T)=\{d\mid{\cal E}(d)=e\}\]

where \(d\) is a derivation in \(T\).

We use a shorthand notation for the transducer rules that will be useful when discussing rule extraction and synthetic rules. Let \(f_{i}\) be a TI representation. The TI representation has the form

\[f_{i}=(X\;{\cal C}\;(L_{1}\;T_{1})\ldots(L_{m}\;T_{m}))\]

where \(L_{i}\in L_{E}\) and \(T_{1},\ldots,T_{m}\) are TI representations.4 Let \(A_{1},\ldots A_{n}\in L_{E}\). We use

Footnote 4: If \(f_{i}\) is just a single concept with no children, then \(m=0\) and \(f_{i}=(X\;{\cal C})\).

\[(f_{i},A_{1},\ldots,A_{n})\to r \tag{3}\]

as shorthand for the rule:

\[(X\;{\cal C}\;sort[(L_{1}\;T_{1})\;\ldots\;(L_{m}\;T_{m})\] \[(A_{1}\;X_{1})\;\ldots\;(A_{n}\;X_{n})])\to r \tag{4}\]

Note \(r\) must contain the variables \(X_{1}\;\ldots\;X_{n}\). In (3) and (4), argument slots with relation labels \(A_{i}\) have been added as children to the root node of the TI representation \(f_{i}\).

For example, the shorthand for the transducer rules in (2) is:

\[((X\mbox{ want-01}),\textit{ARG0},\textit{ARGI})\rightarrow\] \[\qquad\quad\mbox{The $X_{1}$ wants to $X_{2}$ }.\] \[((X\mbox{ ride-01}),\textit{ARG1})\rightarrow\mathrm{ride~{}the $X_{1}$}\] \[((X\mbox{ bicycle}),\textit{mod})\to X_{1}\;\mathrm{bicycle}\] \[((X\mbox{ red}))\rightarrow\mathrm{red} \tag{5}\]

## 4 Generation

To generate a sentence \(e\) from an input AMR graph \(G\), a spanning tree \(G^{\prime}\) of \(G\) is computed, then transformed into a string using a tree-to-string transducer.

Spanning tree.The choice of the graph \(G\)'s spanning tree \(G^{\prime}\) could have a big effect on the output, since the transducer's output will always be a projective reordering of the tree's leaves. Our spanning tree results from a breadth-first-search traversal, visiting child nodes in lexicographic order of the relation label (inverse relations are visited last). The edges traversed are included in the tree. This simple heuristic is a baseline which can potentially be improved in future work.

Decoding.Let \(T=(N,\Sigma,W,{\cal R})\) be a tree-to-string transducer. The output sentence is the highest scoring transduction of \(G^{\prime}\):

\[e={\cal E}\left(\operatorname*{arg\,max}_{d\in{\cal D}(G^{\prime}, T)}score(d;\boldsymbol{\theta})\right)\] (6) 

Eq. 6 is solved approximately using the cdec decoder for machine translation Dyer et al. (2010). The score of the transduction is a linear function (with coefficients \({\mathbf{\theta}}\)) of a vector of features including the output sequence's language model log-probability and features associated with the rules in the derivation (denoted **f**; Table 1):

\[\mathit{score}(d;{\mathbf{\theta}})=\theta_{\textit{LM}}\log(p_{\textit{LM}}( \mathcal{E}(d)))+\sum_{r\in d}{\mathbf{\theta}^{\top}\mathbf{f}(r)}\]

The feature weights are trained on a development dataset using MERT Och (2003).

In the next four sections, we describe the rules extracted and generalized from the training corpus.

## 5 Inducing Basic Rules

The basic rules, denoted \(\mathcal{R}_{B}\), are extracted from the training AMR data using an algorithm similar to extracting tree transucers from tree-string aligned parallel corpora Galley et al. (2004). Informally, the rules are extracted from a sentence \({\mathbf{w}}=\langle w_{1},\ldots,w_{n}\rangle\) with AMR graph \(G\) as follows:

1. The AMR graph and the sentence are aligned; we use the JAMR aligner from Flanigan et al. Flanigan et al. (2014), which aligns non-overlapping subgraphs of the graph to spans of words. The subgraphs that JAMR aligns are called **fragments**. In JAMR's aligner, all fragments are trees.
2. \(G\) is replaced by its spanning tree by deleting relations that use a variable in the AMR annotation.
3. In the spanning tree, for each node \(i\), we keep track of the word indices \(b(i)\) and \(e(i)\) in the original sentence that trap all of \(i\)'s descendants. (This is calculated using a simple bottom-up propagation from the leaves to the root.)
4. For each aligned fragment \(i\), a rule is extracted by taking the subsequence \(\langle w_{b(i)}\dots w_{e(i)}\rangle\) and "punching out" the spans of the child nodes (and their descendants) and replacing them with argument slots.

See Fig. 2 for examples.

More formally, assume the nodes in \(G\) are numbered \(1,\ldots,N\) and the fragments are numbered \(1,\ldots,F\). Let \(\mathrm{nodes}:\{1,\ldots,F\}\to 2^{\{1,\ldots,N\}}\) and \(\mathrm{root}:\{1,\ldots,F\}\rightarrow\{1,\ldots,N\}\) be functions that return the nodes in a fragment and the root of a fragment, respectively, and let \(\operatorname{children}:\{1,\ldots,N\}\to 2^{\{1,\ldots,N\}}\) return the child nodes of a node. We consider a node aligned if it belongs to an aligned fragment. Let the span of an aligned node \(i\) be denoted by endpoints \(a_{i}\) and \(a^{\prime}_{i}\); for unaligned nodes, \(a_{i}=\infty\) and \(a^{\prime}_{i}=-\infty\) (depicted with superscripts in Fig. 2). The node alignments are propagated by defining \(b(\cdot)\) and \(e(\cdot)\) recursively, bottom up:

\[b(i)=\min(a_{j},\min_{j\in\operatorname{children}(i)}b(j))\] \[e(i)=\max(a^{\prime}_{j},\max_{j\in\operatorname{children}(i)} e(j))\]

Also define functions \(\tilde{b}\) and \(\tilde{e}\), from fragment indices to integers, as:

\[\tilde{b}(i)=b(\mathrm{root}(i))\] \[\tilde{e}(i)=e(\mathrm{root}(i))\]

For fragment \(i\), let \(C_{i}=\operatorname{children}(\mathrm{root}(i))-\mathrm{nodes}(i)\), which is the children of the fragment's root concept that are not included in the fragment. Let \(f_{i}\) be the TI representation for fragment \(i\).5 If \(C_{i}\) is empty, then the rule extracted for fragment \(i\) is:

Footnote 5: I.e., the nodes in fragment \(i\), with the edges between them, represented as a TI representation.

\[r_{i}:(f_{i})\to w_{\tilde{b}(i):\tilde{e}(i)} \tag{7}\]

Otherwise, let \(m=|C_{i}|\), and denote the edge labels from \(\mathrm{root}(i)\) to elements of \(C_{i}\) as \(A_{1}(i)\ldots A_{m}(i)\). For \(j\in\{1,\ldots,m\}\), let \(k_{j}\) select the elements \(c_{k_{j}}\) of \(C_{i}\) in ascending order of \(b(k_{j})\). Then the rule extracted for fragment \(i\) is:

\[r_{i}:(f_{i},A_{k_{1}}(i),\ldots A_{k_{m}}(i))\rightarrow\] \[w_{\tilde{b}(i):\tilde{b}(k_{1})}~{}X_{1}~{}w_{\tilde{e}(k_{1}): \tilde{b}(k_{2})}~{}X_{2}\ldots\] \[\ldots w_{\tilde{e}(k_{m-1}):\tilde{b}(k_{m})}X_{m}~{}w_{\tilde{e} (k_{m}):\tilde{e}(i)} \tag{8}\]

A rule is only extracted if the fragment \(i\) is aligned and the child spans do not overlap. Fig. 2 gives an example of a tree annotated with alignments, \(b\) and \(e\), and the extracted rules.

 



## 6 Modeling Synthetic Rules

The synthetic rules, denoted \({\cal R}_{S}(G)\), are created to generalize the basic rules and overcome data sparseness resulting from our relatively small training dataset. Our synthetic rule model considers an AMR graph \(G\) and generates a set of rules for each node in \(G\). S synthetic rule's LHS is a TI representation \(f\) with argument slots \(A_{1}\ldots A_{m}\) (this is the same form as the LHS for basic rules). For each node in \(G\), one or more LHS are created (we will discuss this further below), and for each LHS, a set of \(k\)-best synthetic rules are produced. The simplest case of a LHS is just a concept and argument slots corresponding to each of its children.

For a given LHS, the synthetic rule model creates a RHS by concatenating together a string in \(W^{*}\) (called a **concept realization** and corresponding to the concept fragment) with strings in \(W^{*}{\cal X}W^{*}\) (called an **argument realization** and corresponding to the argument slots). See the top of Fig. 3 for a synthetic rule with concept and argument realizations highlighted.

Synthetic rules have the form:

\[r:(f,A_{1},\ldots A_{m})\rightarrow \tag{9}\] \[{\bf l}_{k_{1}}X_{k_{1}}{\bf r}_{k_{1}}\ldots{\bf l}_{k_{c}}X_{ k_{c}}{\bf r}_{k_{c}}~{}{\bf c}\] \[{\bf l}_{k_{c+1}}X_{k_{c+1}}{\bf r}_{k_{c+1}}\ldots{\bf l}_{k_{m} }X_{k_{m}}{\bf r}_{k_{m}}\]

where:

* \(f\) is a TI representation.
* Each \(A_{i}\in L_{E}\).
* \(\langle k_{1},\ldots,k_{m}\rangle\) is a permutation of \(\langle 1,\ldots,m\rangle\)
* \({\bf c}\in W^{*}\) is the **realization** of TI representation \(f\).
* Each \({\bf l}_{i},{\bf r}_{i}\in W^{*}\) and \(X_{i}\in{\cal X}\). Let \(R_{i}=\langle{\bf l}_{i},{\bf r}_{i}\rangle\) denote the **realization of argument \(i\)**.
* \(c\in[0,m]\) is the position of \({\bf c}\) among the realizations of the arguments.

Let \({\cal F}\) be the space of all possible TI representations. Synthetic rules make use of three lookup tables (which are partial functions) to provide candidate realizations for concepts and arguments: a table for concept realizations \(\mathit{lex}:{\cal F}\to 2^{W^{*}}\), a table for argument realizations when the argument is on the left \(\mathit{left}_{\mathit{lex}}:{\cal F}\times L_{E}\to 2^{W^{*}}\), and a table for argument realizations when the argument is on the right \(\mathit{right}_{\mathit{lex}}:{\cal F}\times L_{E}\to 2^{W^{*}}\). These tables are constructed during basic rule extraction, the details of which are discussed below .

Synthetic rules are selected using a linear model with features \({\bf g}\) and coefficients \(\boldsymbol{\phi}\), which scores each RHS for a given LHS. For LHS = \((f,A_{1},\ldots A_{m})\), the RHS is specified completely by \({\bf c},c,R_{1},\ldots,R_{m}\) and a permutation \(k_{1},\ldots,k_{m}\). For each node in \(G\), and for each TI representation \(f\) in the domain of \(\mathit{lex}\) that matches the node, a LHS is created, and a set of \(K\) synthetic rules is produced for each \({\bf c}\in\mathit{lex}(f)\). The rules produced are the

\begin{table}
\begin{tabular}{|l|p{85.4pt}|} \hline
**Name** & **Description** \\ \hline Rule & \(1\) for every rule \\ Basic & \(1\) for basic rules, else \(0\) \\ Synthetic & \(1\) for synthetic rules, else \(0\) \\ Abstract & \(1\) for abstract rules, else \(0\) \\ Handwritten & \(1\) for handwritten rules, else \(0\) \\ Rule given concept & \(\log(\text{number of times rate extracted / number of times concept observed in training data})\) (only for basic rules, \(0\) otherwise) \\ ... without sense & same as above, but with sense tags for concepts removed \\ Synthetic score & model score for the synthetic rule (only for synthetic rules, \(0\) otherwise) \\ Word count & number of words in the rule \\ Stop word count & number of words not in a stop word list \\ Bad stop word & number of words in a list of meaning-changing stop words, such as “all, can, could, only, so, too, until, very” \\ Negation word & number of words in “no, not, n’t“ \\ \hline \end{tabular}
\end{table}
Table 1: Rule features in the transducer. There is also an indicator feature for every handwritten rule.



\({}_{0}\) The \(1\) (boy) \({}_{2}\) wants \({}_{3}\) to \({}_{4}\) (ride \({}_{5}\) the \({}_{6}\) ((red) \({}_{7}\) bicycle))) \({}_{8}\)

(a) Sentence annotated with indexes, and bracketed according to \(b(i)\) and \(e(i)\) from the graph in (b).

\(K\)-best solutions to:

\[\mathop{\arg\max}_{c,k_{1}...k_{m},R_{1},\ldots,R_{m}}\Big{(}\sum^{ c}_{i=1}\mathbf{\psi}^{\top}\mathbf{g}(R_{k_{i}},A_{k_{i}},\mathbf{c},i,c)\] \[+\mathbf{\psi}^{\top}\mathbf{g}(\langle\epsilon,\epsilon\rangle,*, \mathbf{c},c+1,c)\] \[+\sum^{m}_{i=c+1}\mathbf{\psi}^{\top}\mathbf{g}(R_{k_{i}},A_{k_{i}}, \mathbf{c},i+1,c)\Big{)} \tag{10}\]

where the max is over \(c\in 0\dots m\), \(k_{1},\ldots,k_{m}\) is any permutation of \(1,\ldots,m\), and \(R_{i}\in\mathit{left}_{lex}(A_{i})\) for \(i<c\) and \(R_{i}\in\mathit{right}_{lex}(A_{i})\) for \(i>c\). \(*\) is used to denote the concept position. \(\epsilon\) is the empty string.

The best solution to Eq. 10 is found exactly by brute force search over concept position \(c\in[0,m+1]\) and the permutation \(k_{1},\ldots,k_{m}\). With fixed concept position and permutation, each \(R_{i}\) for the \(\operatorname{arg\max}\) is found independently. To obtain the exact \(K\)-best solutions, we use dynamic programming with a \(K\)-best semiring Goodman (1999) to keep track of the \(K\) best sequences for each concept position and permutation, and take the best \(K\) sequences over all values of \(c\) and \(k\)..

The synthetic rule model's parameters are estimated using basic rules extracted from the training data. Basic rules are put into the form of Eq. 9 by

Figure 3: Synthetic rule generation for the rule shown at top. In the rule RHS, the realization for ARG0 is blue, the realization for DEST is red, and the realization for ride-01 is black. For a fixed permutation of the concept and arguments, choosing the argument realizations can be seen as a sequence labeling problem (bottom). The highlighted sequence corresponds to the rule at top.

Figure 2: Example rule extraction from an AMR-annotated sentence.

 

segmenting the RHS into the form

\[{\bf l}_{1}X_{1}{\bf r}_{1}~{}\ldots~{}{\bf c}~{}\ldots~{}{\bf l}_{m}X_{m}{\bf r }_{m} \tag{11}\]

by choosing \({\bf c,l}_{i},{\bf r}_{i}\in W^{*}\) for \(i\in\{1,\dots,m\}\). An example segmentation is the rule RHS in Fig. 3.

Segmenting the RHS of the basic rules into the form of Eq. 11 is done as follows: \({\bf c}\) is the aligned span for \(f\). For the argument realizations, arguments to the left of \({\bf c}\) pick up words to their right, and arguments to the right pick up words to their left. Specifically, for \(i<c\) (\(R_{i}\) to the left of \({\bf c}\) but not next to \({\bf c}\)), \({\bf l}_{i}\) is empty and \({\bf r}_{i}\) contains all words between \(a_{i}\) and \(a_{i+1}\). For \(i=c\) (\(R_{i}\) directly to the left of \({\bf c}\)), \({\bf l}_{i}\) is empty and \({\bf r}_{i}\) contains all words between \(a_{c}\) and \({\bf c}\). For \(i>c+1,{\bf l}_{i}\) contains all words between \(a_{i-1}\) and \(a_{i}\), and for \(i=c+1\), \({\bf l}_{i}\) contains all words between \({\bf c}\) and \(a_{i}\).

The tables for \(\mathit{lex}\), \(\mathit{left}_{\mathit{lex}}\), and \(\mathit{right}_{\mathit{lex}}\) are populated using the segmented basic rules. For each basic rule extracted from the training corpus and segmented according to the previous paragraph, \(f\rightarrow{\bf c}\) is added to \(\mathit{lex}\), and \(A_{k_{i}}\rightarrow\langle{\bf l}_{i},{\bf r}_{i}\rangle\) is added to \(\mathit{left}_{\mathit{lex}}\) for \(i\leq c\) and \(\mathit{right}_{\mathit{lex}}\) for \(i>c\). The permutation \(k_{i}\) is known during extraction in Eq. 8.

The parameters \(\boldsymbol{\psi}\) are trained using AdaGrad [20] with the perceptron loss function [67; 49] for 10 iterations over the basic rules. The features \({\bf g}\) are listed in Table 2.

## 7 Abstract Rules

Like the synthetic rules, the abstract rules \(\mathcal{R}_{A}(G)\) generalize the basic rules. However, abstract rules are much simpler generalizations which use part-of-speech (POS) tags to generalize. Abstract rules make use of a **POS abstract rule table**, which is a table listing every combination of the POS of the concept realization, the child arguments' labels, and rule RHS with the concept realization removed and replaced with \(*\). This table is populated from the basic rules extracted from the training corpus. An example entry in the table is:

\[(\mathrm{VBD},\mathrm{ARG0},\mathrm{DEST})\to X_{1}~{}\langle* \rangle~{}\mathrm{to}~{}\mathrm{the}~{}X_{2}\]

For the LHS \((f,A_{1},\dots A_{m})\), an abstract rule is created for each member of \({\bf c}\in\mathit{lex}(f)\) and the most common POS tag \(p\) for \({\bf c}\) by looking up \(p\), \(A_{1},\dots A_{m}\) in the POS abstract rule table, finding the common RHS, and filling in the concept position with \({\bf c}\). The set of all such rules is returned.

## 8 Handwritten Rules

We have handwritten rules for dates, conjunctions, multiple sentences, and the concept have-org-role-91. We also create pass-through rules for concepts by removing sense tags and quotes (for string literals).

## 9 Experiments

We evaluate on the AMR Annotation Release version 1.0 (LDC2014T12) dataset. We follow the recommended train/dev./test splits, except that we remove MT09 data (204 sentences) from the training data and use it as another test set. Statistics for this dataset and splits are given in Table 3. We use a \(5\)-gram language model trained with KenLM [1] on Gigaword [11], and use \(100\)-best synthetic rules.

We evaluate with the Bleu scoring metric [2] (Table 4). We report single ref

\begin{table}
\begin{tabular}{|l|l|} \hline
**Feature name** & **Value** \\ \hline POS + \(A_{i}\) + “dist” & \(|c-i|\) \\ POS + \(A_{i}\) + side & \(1.0\) \\ POS + \(A_{i}\) + side + “dist” & \(|c-i|\) \\ POS + \(A_{i}\) + \(R_{i}\) + side & \(1.0\) \\ \({\bf c}+A_{i}\) + “dist” & \(|c-i|\) \\ \({\bf c}+A_{i}\) + side & \(1.0\) \\ \({\bf c}+A_{i}\) + side + “dist” & \(|c-i|\) \\ \({\bf c}+\text{POS + $A_{i}$ + side + “dist"}\) & \(|c-i|\) \\ \hline \end{tabular}
\end{table}
Table 2: Synthetic rule model features. POS is the most common part-of-speech tag sequence for \({\bf c}\), “dist” is the string “dist”, and side is “L” if \(i<c\), “R” otherwise. \(+\) denotes string concatenation.

\begin{table}
\begin{tabular}{|l|r|l|} \hline Split & Sentences & Tokens \\ \hline Train & 10,000 & 210,000 \\ Dev. & 1,400 & 29,000 \\ Test & 1,400 & 30,000 \\ MT09 & 204 & 5,000 \\ \hline \end{tabular}
\end{table}
Table 3: Train/dev./test/MT09 split.

 

erence Bleu for the LCD2014T12 test set, and four-reference Bleu for the MT09 set. We report ablation experiments for different sources of rules. When ablating handwritten rules, we do not ablate pass-through rules.

The full system achieves \(22.1\) Bleu on the test set, and \(21.2\) on MT09. Removing the synthetic rules drops the results to \(9.1\) Bleu on test and \(7.8\) on MT09. Removing the basic and abstract rules has little impact on the results. This may be because the synthetic rule model already contains much of the information in the basic and abstract rules. Removing the handwritten rules has a slightly larger effect, demonstrating the value of handwritten rules in this statistical system.

## 10 Related Work

There is a large body of work for statistical and nonstatistical NLG from a variety of input representations. Statistical NLG systems have been built for input representations such as HPSG Nakanishi et al. (2005), LFG Cahill and Van Genabith (2006); Hogan et al. (2007), and CCG White et al. (2007), as well as surface and deep syntax Belz et al. (2011). The deep syntax representations in Bohnet et al. (2010) and Belz et al. (2011) share similarities with AMR: the representations are graphs with re-entrancies, and have an concept inventory from PropBank Palmer et al. (2005).

The Nitrogen and Halogen systems Langkilde and Knight (1998); Langkilde (2000) used an input representation that was a precursor to the modern version of AMR, which was also called AMR, although it was not the same representation as Banarescu et al. (2013).

Techniques from statistical machine translation have been applied to the problem of NLG Wong and Mooney (2006), and many grammar-based approaches can be formulated as weighted tree-to-string transducers Jones et al. (2012) developed technology for generation and translation with synchronous hyperedge replacement (SHRG) grammars applied to the GeoQuery corpus Wong and Mooney (2006), which in principle could be applied to AMR generation.

## 11 Conclusion

We have presented a two-stage method for natural language generation from AMR, setting a baseline for future work. We have also demonstrated the importance of modeling argument realization for good performance. Our feature-based, tree-transducer approach can be easily extended with rules and features from other sources, allowing future improvements.

### Acknowledgments

The authors would like to thank Adam Lopez and Nathan Schneider for valuable feedback, and Sam Thomson and the attendees of the Fred Jelinek Memorial Workshop in 2014 in Prague for helpful discussions. This work is supported by the U.S. Army Research Office under grant number W911NF-10-1-0533. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the view of the U.S. Army Research Office or the United States Government.

## References

* [Banarescu et al.2013] Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _Proc. of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_.
* [Belz et al.2011] Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The first surface realisation shared task: Overview and evaluation results. In _Proc. of the 13th European Workshop on Natural Language Generation_.
* [Bohnet et al.2010] Bernd Bohnet, Leo Wanner, Simon Mille, and Alicia Burga. 2010. Broad coverage multilingual deep sentence generation with a stochastic multi-level realizer. In _Proc. of COLING_.
* [Cheng et al.2010]

\begin{table}
\begin{tabular}{|l|r|r|} \hline Rules & Test & MT09 \\ \hline Full & 22.1 & 21.2 \\ Full – basic & 22.1 & 20.9 \\ Full – synthetic & 9.1 & 7.8 \\ Full – abstract & 22.0 & 21.2 \\ Full — handwritten & 21.9 & 20.5 \\ \hline \end{tabular}
\end{table}
Table 4: Uncased Bleu scores with various types of rules removed from the full system.

 

Aoife Cahill and Josef Van Genabith. 2006. Robust pcfg-based generation using automatically acquired LFG approximations. In _Proc. of COLING-ACL_.
* Collins (2002) Michael Collins. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In _Proc. of EMNLP_.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _JMLR_, 12.
* Dyer et al. (2010) Chris Dyer, Adam Lopez, Juri Ganitkevitch, Johnathan Weese, Ferhan Ture, Phil Blunsom, Hendra Setiawan, Vladimir Eidelman, and Philip Resnik. 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In _Proc. of ACL_.
* Flanigan et al. (2014) Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the abstract meaning representation. In _Proc. of ACL_.
* Galley et al. (2004) Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What's in a translation rule? In _Proc. of HLT-NAACL_.
* Goodman (1999) Joshua Goodman. 1999. Semiring parsing. _CL_, 25(4).
* Graehl and Knight (2004) Jonathan Graehl and Kevin Knight. 2004. Training tree transducers. In _Proc. of HLT-NAACL_.
* Heafield et al. (2013) Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In _Proc. of ACL_.
* Hogan et al. (2007) Deirdre Hogan, Conor Cafferkey, Aoife Cahill, and Josef Van Genabith. 2007. Exploiting multi-word units in history-based probabilistic generation. In _Proc. of ACL_.
* Huang et al. (2006) Liang Huang, Kevin Knight, and Aravind Joshi. 2006. Statistical syntax-directed translation with extended domain of locality. In _Proc. of MMTA_.
* Jones et al. (2012) Bevan Jones, Jacob Andreas, Daniel Bauer, Karl Moritz Hermann, and Kevin Knight. 2012. Semantics-based machine translation with hyperedge replacement grammars. In _Proc. of COLING_.
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In _Proc. of ACL_.
* Langkilde and Knight (1998) Irene Langkilde and Kevin Knight. 1998. Generation that exploits corpus-based statistical knowledge. In _Proc. of COLING-ACL_.
* Langkilde (2000) Irene Langkilde. 2000. Forest-based statistical sentence generation. In _Proc. of NAACL 2000_.
* Nakanishi et al. (2005) Hiroko Nakanishi, Yusuke Miyao, and Jun'ichi Tsujii. 2005. Probabilistic models for disambiguation of an HPSG-based chart generator. In _Proc. of IWPT_.
* Och (2003) Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In _Proc. of ACL_.
* Palmer et al. (2005) Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. _CL_, 31(1).
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proc. of ACL_.
* Rosenblatt (1957) Frank Rosenblatt. 1957. The perceptron--a perceiving and recognizing automaton. Technical Report 85-460-1, Cornell Aeronautical Laboratory.
* White et al. (2007) Michael White, Rajakrishnan Rajkumar, and Scott Martin. 2007. Towards broad coverage surface realization with CCG. In _Proc. of the Workshop on Using Corpora for NLG: Language Generation and Machine Translation_.
* Wong and Mooney (2006) Yuk Wah Wong and Raymond J Mooney. 2006. Learning for semantic parsing with statistical machine translation. In _Proc. of HLT-NAACL_.