SlowGuess/NAACL_2021_OCR huggingface dataset:
We propose neural models to generate text from formal meaning representations based on Discourse Representation Structures (DRSs). DRSs are document-level representations which encode rich semantic detail pertaining to rhetorical relations, presupposition, and co-reference within and across sentences. We formalize the task of neural DRS-to-text generation and provide modeling solutions for the problems of condition ordering and variable naming which render generation from DRSs non-trivial. Our generator relies on a novel sibling treeLSTM model which is able to accurately represent DRS structures and is more generally suited to trees with wide branches. We achieve competitive performance (59.48 BLEU) on the GMB benchmark against several strong baselines.
It is not uncommon for text generation systems to produce natural language output from intermediate semantic representations (Yao et al., 2012; Takase et al., 2016). The literature presents several examples of generating text from logical forms underlying various grammar formalisms (Wang, 1980; Shieber et al., 1990; Carroll and Oepen, 2005; White et al., 2007), typed lambda calculus (Lu and Ng, 2011), Abstract Meaning Representations (AMR; Flanigan et al. 2016; Konstas et al. 2017; Song et al. 2018; Beck et al. 2018; Damonte and Cohen 2019; Ribeiro et al. 2019; Zhu et al. 2019; Cai and Lam 2020; Wang et al. 2020), Discourse Representation Theory (DRT; Basile and Bos 2011; Basile 2015), and Minimal Recursion Semantics (MRS; Horvat et al. 2015; Hajdik et al. 2019).
In this work, we propose neural models to generate high-quality text from semantic representations based on Discourse Representation Structures (DRSs). DRSs are the basic meaning-carrying units in Discourse Representation Theory (DRT; Kamp
The man is going to play the piano. Tom may stop him.
Figure 1: DRS representing two-sentence discourse.
1981; Kamp and Reyle 1993; Asher and Lascarides 2003), a formal semantic theory designed to handle a variety of linguistic phenomena, including anaphora, presuppositions (Van der Sandt, 1992; Venhuizen et al., 2018), and temporal expressions within and across sentences. DRSs are scoped meaning representations, they capture the semantics of negation, modals, and quantification.
Figure 1 displays in box format the meaning representation for a discourse consisting of two sentences. The outermost box is a segmented DRS expressing the rhetorical relation CONTRAST between box  $b_{1}$  representing the first sentence and box  $b_{4}$  representing the second sentence. Boxes  $b_{1}$  and  $b_{2}$  are DRSs, the top layers contain variables (e.g.,  $x_{1}, x_{2}$ ) indicating discourse referents and the bottom layers contain conditions (e.g., Named( $x_{3},$  "tom")) representing information about discourse referents. Variables and conditions have pointers (denoted by  $b$  in the figure) pointing to the boxes where they should be interpreted. Predicates are disambiguated to their Wordnet (Fellbaum, 1998) senses (e.g., male.n.02 and play.v.03).
Although there has been considerable activity recently in developing models which analyze text in the style of DRT (van Noord et al., 2018, 2019; Liu et al., 2019a, 2018; Fancellu et al., 2019), attempts
<sup>1</sup>In Figure 1,  $b_{6}$  is a presuppositional box for the interpretation of the man in the context of the two-sentence discourse.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 397-415 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: DRS from Figure 1 with (a) shuffled conditions and (b) different variable names.
to generate text from DRSs have been few and far between (however see Basile 2015 and Narayan and Gardent 2014 for notable exceptions). This is primarily due to two properties of DRS-based semantic representations which render generation from them challenging. Firstly, DRS conditions are unordered representing a set (rather than a list).<sup>2</sup> A hypothetical generator would have to produce the same output text for any DRSs which convey the same meaning but appear different due to their conditions having a different order (see Figures 1 and 2a which are otherwise identical but the order of conditions in boxes  $b_{1}$  and  $b_{4}$  varies). The second challenge concerns variables and their prominent status in DRSs. Variables identify objects in discourse (such as entities and predicates), and are commonly used to model semantic phenomena including coreference, control constructions, and scope. In Figure 1, variables  $x, e, s, t, p$ , and  $b$  denote entities, events, states, time, propositions and boxes, respectively. Variable names themselves are arbitrary and meaningless posing a challenge for learning. Our generator must verbalize different variable names to the same surface form. The meaning representations in Figures 1 and 2b are identical and both correspond to the same discourse except that the variables have been given different names ( $b_{5}$  in Figure 1 has been named  $b_{1}$  in Figure 2b,  $b_{1}$  is now  $b_{2}, x_{1}$  is  $x_{2}, e_{1}$  is  $e_{9}$ , and so on).
These two problems are further compounded by the way DRSs are displayed, in a box-like format which is intuitive and easy to read but not convenient for modeling purposes. As a result, DRSs are often post-processed in a format that can be handled more easily by modern neural network models. For example, DRS variables and conditions are converted to clauses (van Noord et al., 2018) or DRSs are modified to trees where each box is a subtree and conditions within the box correspond to children of the subtree (Liu et al., 2019a, 2018).
In this paper we propose novel solutions to condition ordering and variable naming. We argue that even though DRS conditions appear unordered, they have a latent order due to biases in the way the training data is created. To give a concrete example, the Groningen Meaning Bank (GMB; Bos et al. 2017) provides the largest collection to date of English texts annotated with DRSs. These annotations were generated with the aid of a CCG parser (Clark and Curran, 2007); atomic DRS conditions were associated with CCG supertags and then semantically combined following the syntactic CCG derivations. Even annotators creating DRSs manually would be prone to follow a canonical order (e.g., listing named entities first, then verbal predicates and their thematic roles, and finally temporal conditions). We propose a graph-based model which learns to recover the latent order of conditions without explicitly enumerating all possible orders which can be prohibitive. We also handle variable names with a method which rewrites arbitrary indices to relative ones which are in turn determined by the order of conditions.
Following previous work, we convert DRSs to a more amenable format. Specifically, we consider Discourse Representation Tree Structures (DRTSs; Liu et al. 2019b) as the semantic representation input to our document generation task, and generate a sequence of words autoregressively. We adopt an encoder-decoder framework with a treeLSTM (Tai et al., 2015) encoder and a standard LSTM (Hochreiter and Schmidhuber, 1997) decoder. Problematically, DRS trees are wide and the number of children for a given node can be as many as 180. It therefore becomes memory-consuming and sparse to assign a forget gate for each child as in the case of conventional ( $N$ -ary) treeLSTM (Tai et al., 2015). We propose a variant which we call Sibling treeLSTM that replaces  $N$  forget gates with a parent gate and a sibling gate. As a result, it reduces
${}^{2}$  An exception are conditions in segmented DRSs whose order can be retrieved deterministically based on the arguments of rhetorical relations. For example, given the relation BECAUSE  $\left( {{b}_{1},{b}_{3}}\right)$  ,we can assume that box  ${b}_{1}$  precedes  ${b}_{3}$  .
memory usage from  $O(N)$  to  $O(2)$ , and is more suitable for modeling wide and flat trees.
Our contributions can be summarized as follows: (1) we formalize the task of neural DRS-to-text generation; (2) we provide solutions for the problems of condition ordering and variable naming, which render generation from DRS-based meaning representations non-trivial; and (3) propose a novel sibling treeLSTM model that can be also generally used to model wide tree structures. We make our code and datasets publicly available.
Let  $S$  denote a DRS-based meaning representation. The aim of DRS-to-text generation is to produce text  $T$  that verbalizes input meaning  $S$ :
$$
T ^ {*} = \underset {T \in \mathbb {T}} {\arg \max} P (T | S, \Theta),
$$
where  $\mathbb{T}$  is the set of all possible texts,  $S$  has an arbitrary order of conditions and indexing of variables, and  $\Theta$  is the set of model parameters.
Our generation model is based on the encoder-decoder framework (Bahdanau et al., 2015) and operates over tree structures. Moreover, prior to training, variable names are rewritten so that their (arbitrary) indices denote relative order of appearance. We propose a novel sibling TreeLSTM for encoding tree structures. The decoder is a sequential LSTM equipped with an attention mechanism generating word sequence  $T = [t_0, t_1, \dots, t_{m-1}]$ , where  $m$  is the length of the text. At test time, DRS conditions are normalized, i.e., they are reordered following a canonical order learned from data, and used as input to our generation model.
We first describe our DRS-to-tree conversion and variable renaming procedures (Sections 2.1 and 2.2). We next present our tree-to-sequence generation model (Section 2.3), and explain how DRS conditions are ordered (Section 2.4).
2.1 DRS-to-Tree Conversion
The algorithm of Liu et al. (2018) renders DRSs in a tree-style format. It constructs trees based on DRS conditions in the bottom box layers, without considering variables in the top layer. This results in oversimplified semantic representations and information loss (e.g., presuppositions cannot be handled). We improve upon their approach by
Figure 3: (a) Box-style DRSs; (b) Tree-style DRSs.
Figure 4: DRSs with relative variables.
merging variables in the top layer with variables in the bottom layer via introducing special conditions.
We collect variables in top layers of DRS boxes to construct a dictionary  $d = \{v : b\}$ , where  $v$  denotes a variable and  $b$  is a presupposition box label (e.g.,  $x_1 : b_1$ ). We then move variables from the top to the bottom layer by expressing them as special conditions  $b : \operatorname{Ref}(v)$  and placing them before conditions on variable  $v$ . For example,  $b_6 : x_1$  in Figure 1 becomes special condition  $b_6 : \operatorname{Ref}(x_1)$  and is placed before condition  $b_6 : \operatorname{Pred}(x_1, \text{male.n.02})$  in Figure 3(a).
Once top variables have been rewritten as special conditions, the resulting DRSs are converted into trees as shown in Figure3(b). Box variables (e.g.,  $b_{1}$ ,  $b_{5}$ ) become parent nodes, while conditions, which are also subtrees, become children.
3https://github.com/LeonCrashCode/Discourse-Representation-Tree-Structure/tree/main/gmb/DRS-to-text
2.2 Relative Variables
We rename variables with regard to their relative position in a given DRS following a predefined traversal order.
We obtain the sequence of box variables by traversing DRSs in an outer-to-inner and left-to-right manner, e.g.,  $[b_5, b_1, b_4, b_3]$  in Figure 1. For SDRSs, we replace variables in discourse relations with  $k_i$ , where  $i$  denotes the  $i$ th box from left to right. For example “CONTRAST( $b_1, b_4$ )” in Figure 1 is rewritten to “CONTRAST( $k_0, k_1$ ). Variables and conditions within presupposition boxes are rewritten to  $B_i$ , where  $i \in \mathbb{Z}$  denotes the distance of the current box to the presupposition box. For example,  $b_1: \operatorname{Agent}(e_1, x_1)$  is rewritten to  $B_0: \operatorname{Agent}(e_1, x_1)$  because it is in the current box  $b_1$ , while  $b_1: \operatorname{Pred}(x_1, “male.n.02”)$  is rewritten to  $B_{-2}: \operatorname{Pred}(x_1, “male.n.02”)$  because it is in box  $b_3$  and two hops away from presupposition box  $b_1$ . We use special label  $O$  for presupposition boxes pertaining to semantic content outwith the current DRS. For example,  $b_6: \operatorname{Ref}(x_1)$  is rewritten to  $O: \operatorname{Ref}(x_1)$  because it introduces a new presupposition box, and  $b_6: \operatorname{Pred}(x_1, male.n.02)$  is rewritten to  $O_0: \operatorname{Pred}(x_1, male.n.02)$  because the condition can only be interpreted in this new presupposition box (now  $O_0$  and previously  $b_6$ ).
We obtain a sequence of general variables by traversing conditions as they appear in the DRS. Variables introduced for the first time are denoted by their type (going from left-to-right), while subsequent mentions of the same variables are rewritten with relative indices denoting their distance from the position where they were first introduced. Take Figure 3(a) as an example. The sequence of general variables is  $[x_1, x_1, e_1, e_1, e_1, x_1, x_2, e_1, x_2, x_2, t_1, t_1, e_1, t_1, x_3, x_3, e_2, e_2, x_3, x_1, e_2, x_1, e_2, e_1]$ , and is rewritten to  $[X, X_0, E, E_0, E_0, X_0, X, E_0, X_0, T, T_0, E_0, T_0, X, X_0, E, E_0, E_0, X_0, X_{-2}, E_0, X_{-2}, E_0, E_{-1}]$ . The DRS from Figure 3(a) is shown in Figure 4 with relative variables.
2.3 Generation Model
Our generation model is based on the encoder-decoder framework, where an encoder is used to encode input DRS trees and a decoder outputs a sequence of words. A limitation of sequential encoders is that they only allow sequential information propagation without considering the structure of the input (Tai et al., 2015; Wang et al., 2019). In
Figure 5: The sibling treeLSTM; grey boxes are hidden representations of nodes, black/red arrows represent sibling/parent information flow.
our case, DRS tree structures are additionally wide (the longer a document, the wider the tree) and relatively flat (see Figure 3(b)). To better model these aspects, we propose a treeLSTM encoder which takes sibling information into account.
As shown in Figure 5, the hidden representations of the sibling TreeLSTM cells are updated from preceding sibling and child nodes. More formally, the hidden representation for node  $j$  is given by:
$$
u _ {j} = \tanh  \left(g ^ {u} \left(\left[ x _ {j}; h _ {j s}; h _ {j p} \right]\right)\right) \tag {1}
$$
$$
i _ {j}, o _ {j} = \sigma \left(g ^ {i o} \left(\left[ x _ {j}; h _ {j s}; h _ {j p} \right]\right)\right) \tag {2}
$$
$$
f _ {j s} = \sigma \left(g _ {s} ^ {f} \left(\left[ x _ {j}; h _ {j s} \right]\right)\right) \tag {3}
$$
$$
f _ {j p} = \sigma \left(g _ {p} ^ {f} ([ x _ {j}; h _ {j p} ])\right) \tag {4}
$$
$$
c _ {j} = i _ {j} \cdot u _ {j} + f _ {j s} \cdot c _ {j s} + f _ {j p} \cdot c _ {j p} \tag {5}
$$
$$
h _ {j} = o _ {j} \cdot \tanh  (c _ {j}), \tag {6}
$$
where  $x_{j}$  is the token input representation,  $h_{js}$  is the hidden representation of the sibling node preceding  $j$ ,  $h_{jp}$  is the hidden representation of the last child of node  $j$  (Equation (1)),  $g^{*}$  are linear functions, and  $\sigma$  is a sigmoid function (Equations (2)-(4)). For each node  $j$ , we obtain its cell input representation  $u_{j}$  (Equation (1)), its input gate  $i_{j}$  and output gate  $o_{j}$  (Equation (2)), and two forget gates  $f_{js}$  (Equation (3)) and  $f_{jp}$  (Equation (4)) for its neighbor cell and the last child cell, respectively. The memory of the current cell  $c_{j}$  (Equation (5)) is updated by the gated sum of its cell input representation and the memories of its neighbor and child cells. The hidden representation of current node  $h_{j}$  is computed with its output gate  $o_{j}$  (Equation (6)).
Finally, a DRS tree is represented by the hidden representations of its nodes  $[h_0, h_1, \dots, h_{n' - 1}]$  as computed by the sibling treeLSTM ( $n'$  denotes the number of nodes). The decoder is a standard LSTM with global attention (Bahdanau et al., 2015).
2.4 Condition Ordering
As discussed previously, DRSs at test time may exhibit an arbitrary order of conditions, which our model should be able to handle. Our solution is to reorder conditions prior to generation by learning a latent canonical order from training data (e.g., to recover boxes  $b_{1}$  and  $b_{3}$  in Figure 1 from boxes  $b_{1}$  and  $b_{3}$  in Figure 2). More formally, given a set of conditions  $R_{set}$ , we obtain an optimal ordering  $R = [r_0, r_1, \dots, r_{n-1}]$  such that:
$$
R ^ {*} = \underset {R \in \pi (R _ {s e t})} {\arg \max } \operatorname {S C O R E} _ {\mathbb {K}} (R | R _ {s e t}), \tag {7}
$$
where  $\pi(R_{set})$  are all permutations of  $R_{set}$ , and  $R^{*}$  is the order with the highest likelihood according to  $\mathrm{SCORE}_{\mathbb{K}}$ . Here,  $\mathbb{K}$  parametrizes  $\mathrm{SCORE}$  as "knowledge" we collect from our training data by observing canonical orders of conditions. Unfortunately, the time complexity of calculating Equation (7) is  $O(n!)$ , we must enumerate all possible permutations for a set of conditions with  $n$  as large as 180. Since this is prohibitive, we resort to graph ordering which allows us to recover the order of the conditions without enumeration.
Graph Construction We construct a graph from the set of DRS conditions which we break down into graph nodes and edges. Conditions in DRSs can be simple or complex according to their type of arguments. A simple condition might have a relation name with two arguments (e.g., Named  $(x_{3},$  "tom") and Agent  $(e_1,x_3))$  , while a complex condition has a scoped name (e.g., possibility  $\diamondsuit$  ) and takes one or more DRSs as arguments. Simple conditions are denoted by a 3-tuple  $(l_s,a_0,a_1)$  where  $l_{s}$  is the condition name (e.g., Named and Agent) and  $a_0$  and  $a_1$  are its first and the second argument, respectively, which could be a variable or constant (e.g.,  $e_1,x_3$  and "piano.n.01"). Complex conditions are a 2-tuple  $(l_c,V_r)$  , where  $l_{c}$  is the scope name, and  $V_{r}$  the set of arguments scoped by the condition. For example, the set of arguments for the possibility scope  $(\diamondsuit)$  in Figure 1 is  $\{e_1,e_2,$ $x_{1},x_{3}$  , "tom", "stop.v.05", "male.n."}.
Condition names become nodes in our graph. Simple conditions are further divided into constant and thematic nodes. Constant nodes are
Conditions	Nodes	Edges
Pred(x1, "male.n.02")	Pred "male.n.02"	a0 = x1
Pred(e1, "play.v.03")	Pred "play.v.03"	a0 = e1
Agent(e1, x1)	Agent	a0 = e1, a1 = x1
Theme(e1, x2)	Theme	a0 = e1, a1 = x2
Pred(x2, "piano.n.01")	Pred "piano.n.01"	a0 = x2
Pred(t1, "now.n.01")	Pred "now.n.01"	a0 = t1
temp_after(e1, t1)	temp_after	a0 = e1, a1 = t1
Figure 6: (a) Conditions and their corresponding graph nodes (b) graph with inverse edges (shown in red).
constructed by concatenating the relation name in the condition with the constant argument (e.g., condition  $\operatorname{Pred}(x_1, \text{"male.n.02"})$  becomes node  $\operatorname{Pred} \text{"male.n.02"}$ ). Thematic nodes correspond to the relation name of the thematic condition (e.g.,  $\operatorname{Agent}(e_1, x_1)$  becomes the node "Agent"). Complex nodes correspond to the name of complex conditions (e.g., possibility  $\diamond$ ).
We insert edges between graph nodes if these share arguments. For example, in Figure 6(b), there is an edge connecting node Pred "male.n.02" with Agent as they share argument  $x_{1}$ . We label this edge with  $a_{1}$  to denote the fact that it is the second argument of Agent. Another edge is drawn between Pred "play.v.03" and Agent (as they share argument  $e_{1}$ ) with label  $a_{0}$  denoting that this is the first argument of Agent. Edges between nodes are bidirectional, with inverse edges bearing the suffix "-of". Edges drawn between constant and complex nodes bear the label "Related", while edges between two constant nodes (with the same variables) bear the label "Equal" (we provide a more formal description in the Appendix).
Ordering Model Given graph  $G = (R_{set},E)$  where  $R_{set} = \{r_0,r_1,\dots,r_{n - 1}\}$  is the set of nodes and  $E$  is the set of edges in  $G$ , our model outputs  $R^{*}$  as the optimal order of  $R_{set}$ .
As shown Figure 6(a), each node is a sequence of words. A BiLSTM is applied to obtain representation  $x_{i}$  of each node  $r_i = [w_0^i, \dots, w_{m-1}^i]$ :
$$
x _ {i} = \operatorname {B i L S T M} ([ w _ {0} ^ {i}, \dots , w _ {m - 1} ^ {i} ]). \tag {8}
$$
We encode the graph with a Graph Convolutional Recurrent network (GCRN; Seo et al. 2018). For
each node  $r_i$ , we collect information from neighbor hidden representations with a gate controlling the information flow from neighbors to current nodes:
$$
h _ {i} ^ {\prime k} = \sum_ {j} g _ {j} ^ {k} \cdot h _ {j} ^ {k - 1}; \tag {9}
$$
$$
g _ {j} ^ {k} = \sigma \left(f \left(\left[ e _ {j i}, h _ {i} ^ {k - 1}, h _ {j} ^ {k - 1} \right]\right)\right), \tag {10}
$$
where  $e_{ji}$  is the embedding of edges from node  $r_j$  to  $r_i$ , and  $k$  is the recurrent step in the GRU. The node hidden representations are updated as:
$$
h _ {i} ^ {k} = \operatorname {G R U C e l l} \left(\left[ x _ {i}; g _ {G} ^ {k - 1} \right], h _ {i} ^ {k - 1}\right) \tag {11}
$$
$$
g _ {G} ^ {k} = \operatorname {G R U C e l l} \left(\frac {1}{n} \sum_ {i} h _ {i} ^ {k}, g _ {G} ^ {k - 1}\right) \tag {12}
$$
where  $g_{G}$  represents the hidden representation of the graph as the average of (hidden) node representations, and GRUCell denotes the gated recurrent cell function. We obtain the hidden representations of nodes in the final recurrent step  $(K)$  as  $H^{K} = \{h_{0}^{K}, h_{1}^{K}, \dots, h_{n - 1}^{K}\}$ .
Our decoder obtains the orders with the highest probability. We avoid enumerating all possible permutations for a set of nodes by generating their order autoregressively with an LSTM-based Pointer Network (PN; Vinyals et al. 2015):
$$
\operatorname {S C O R E} _ {\mathbb {K}} (R \mid R _ {s e t}) = \operatorname {P N} (R \mid R _ {s e t}, H ^ {K}, \theta) \tag {13}
$$
$$
\mathrm {P N} \left(R \mid R _ {\text {s e t}}, H ^ {K}, \theta\right) = \prod_ {i} P \left(r _ {i} \mid r _ {<   i}, H ^ {K}\right) \tag {14}
$$
$$
P \left(r _ {i} \mid r _ {<   i}, H ^ {K}\right) = \operatorname {s o f t m a x} \left(v ^ {\mathrm {T}} \tanh  \left(W \left[ h _ {i} ^ {d}; H ^ {K} \right]\right)\right) \tag {15}
$$
where  $\theta$  are the parameters of the Pointer Network,  $h_i^d$  is the  $i$ th step hidden representation of the Pointer Network, and  $v$ , and  $W$  are parameters. Hidden representation  $h_i^d$  is updated by the input representation of the  $(i - 1)$ th ordered node:  $h_i^d = \mathrm{LSTMCell}(x_{r_{i - 1}}, h_{i - 1}^d)$ . All parameters are optimized with standard back-propagation.
Our experiments were carried out on the Groningen Meaning Bank (GMB; Bos et al. 2017) which provides a large collection of English documents annotated with DRSs. We used the standard training, development, and test splits that come with the distribution of the corpus. All DRSs in the GMB were preprocessed into the tree-based format discussed in Section 2.1. We also extracted from the training data conditions and their order for training our graph ordering model. Dataset statistics are shown in Table 1.
Task	train	dev	test
Generation	7,970	992	1,038
Condition Ordering	133,332	16,493	17,624
Table 1: GMB dataset statistics; number of documents (generation) and number of different sequences of conditions (ordering).
3.1 Condition Ordering
Models and Settings Before evaluating our generator per se, we assess the effectiveness of the proposed condition ordering model (see Section 2.4). Specifically we compare four kinds of graphs: NoEdges, is a graph without edges; FullEdges, is a complete graph where each pair of nodes has edges; SiGraph, is the proposed graph without bidirectional edges; and BiGraph, is the proposed graph with bidirectional edges (see Figure 6). We also consider Counting, a baseline model which greedily orders pairs of conditions according to their frequency of appearance in the training data (see the Appendix for details).