SlowGuess/ACL_2020_OCR huggingface dataset:
Yufei Chen<sup>1</sup> and Weiwei Sun<sup>12</sup>
{yufei.chen,ws}@pku.edu.cn
We propose variable-in-situ logico-semantic graphs to bridge the gap between semantic graph and logical form parsing. The new type of graph-based meaning representation allows us to include analysis for scope-related phenomena, such as quantification, negation and modality, in a way that is consistent with the state-of-the-art underspecification approach. Moreover, the well-formedness of such a graph is clear, since model-theoretic interpretation is available. We demonstrate the effectiveness of this new perspective by developing a new state-of-the-art semantic parser for Minimal Recursion Semantics. At the core of this parser is a novel neural graph rewriting system which combines the strengths of Hyperedge Replacement Grammar, a knowledge-intensive model, and Graph Neural Networks, a data-intensive model. Our parser achieves an accuracy of  $92.39\%$  in terms of ELEMENTARY DEPENDENCY MATCH, which is a 2.88 point improvement over the best data-driven model in the literature. The output of our parser is highly coherent: at least  $91\%$  graphs are valid, in that they allow at least one sound scope-resolved logical form.
Graphs have recently become popular as a strategy for encoding sentence-level semantics, and related data-driven parsing techniques have been making rapid progress. The primary component of popular semantic graphs, e.g. Elementary Dependency Structure (EDS; Oepen and Lønning, 2006) and Abstract Meaning Representation (AMR; Banarescu et al., 2013), is the predicate–argument structure, with the predicate being a concept that takes a number of arguments. Though expressive for many applications, this predicative core does not fully match the need for logical forms that used to stand in the central area of semantic parsing.
Partly due to the lack of model-theoretic semantics, it is rather difficult to add scope information related to quantification, negation and modality to a graph. Partly due to the lack of logical deduction engines, it is rather difficult to directly perform automated reasoning over graphs.
This paper proposes to express logical forms with variable-in-situ graphs for the ongoing advances in graph-centric formalisms, algorithms and neural architectures. This leads us to a novel neural graph rewriting system that combines the strengths of Hyperedge Replacement Grammar (HRG; Drewes et al., 1997) and Graph Neural Networks (Song et al., 2018a). On the one hand, it can be viewed as an improved graph embedding model that explicitly explores recursive structures that are defined by an HRG. On the other hand, it can be viewed as an enhanced graph grammar with which all nodes involved in derivations of graphs are assigned vector-based distributed encodings.
Based on our neural graph rewriting system, we develop a new parser for Minimal Recursion Semantics (MRS; Copestake et al., 2005). By means of the DeepBank (Flickinger et al., 2012) data, our parser achieves an accuracy of  $92.39\%$  in terms of ELEMENTARY DEPENDENCY MATCH, which is a 2.88 point improvement over the best data-driven model in the literature. We also consider the structural validity of logico-semantic graphs following the original design of MRS.
The output of our parser is highly coherent: at least  $91\%$  graphs are coherent, in that they allow at least one sound scope-resolved logical form.
2 Logico-Semantic Graphs
2.1 Logic-Based Meaning Representations
Classic theories of natural language semantics are based on the assumption that the core meaning of a
Source code: https://github.com/draptater/var-parser/
6772
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6772-6782 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(b) Tree-based representation
(c) Variable-reduced graph
Figure 1: Different representations of a logical form.
(d) Variable-in-situ graph
sentence is captured as its truth conditions. Under this assumption, using expressions of some logical languages to encode truth conditions is the de facto approach in formal semantics. Classic logic, e.g. first-order predicate logic, supports precise, consistent and controlled meaning representation via truth-conditional interpretation.
A logical form can be visualized as a pseudo tree, as suggested by Copestake et al. (2005). For example, the formula in Fig. 1a can be encoded as the tree in Fig. 1b. However, the leaves of such a tree are not independent of each other. For instance,  $\mathbf{dog}(\mathbf{x})$  and  $\mathrm{chase}(\mathbf{e}_1,\mathbf{x},\mathbf{y})$  share the same variable  $x$ . Transforming logical forms into trees may enlarge the distance between closely-related nodes and make it difficult for a statistical or neural model to explicitly capture such dependencies. In addition, considering syntactic-semantic similarity, this tree-structured logical form is essentially different from the corresponding syntactic tree, as shown in Fig. 2. Such a tree representation brings difficulties to develop a systematic syntax-semantics interface.
Figure 2: Dependency-based syntactic analysis.
Previous study (Oepen and Lønning, 2006; Copestake, 2009) shows that there are some good engineering reasons for producing a dependency style representation (see Fig. 1c) with links between predicates: It improves readability for consumers of the representation and eases integration with distributional semantics. Exploiting this direction further, we augment such a semantic de
pendency graph with variables (see Fig. 1d). In fact, it is a more straightforward way to encode logical forms using graphs. Comparing the two types of graphs, we can see that the variable-in-situ representation fully specifies what there is in a logical form, while a variable-free graph may lose some information. Take Fig. 1c for example. The following logical form is also compatible with the graph, which is unfortunately a bad reading, since happy, according to its conceptual meaning, is not a scopal predicate.
(1) every(x, dog(x), some(y, cat(y),
$$
h a p y (\mathbf {e} _ {2}, \operatorname {c h a s e} (\mathbf {e} _ {1}, \mathbf {x}, \mathbf {y})))
$$
2.2 Representing Underspecification
Natural language utterances are often ambiguous, i.e., they have more than one reading. Take scope ambiguity, an important type of ambiguity that has been receiving heightened attention by semanticians, for example. Considering the following sentence:
(2) a. Every dog happily chases some cat.
b. some(y, cat(y), every(x, dog(x),
$$
h a p p y \left(e _ {2}, e _ {1}\right) \wedge \operatorname {c h a s e} \left(e _ {1}, x, y\right))
$$
c. every(x, dog(x), some(y, cat(y),
The sentence is ambiguous: it can either mean that for every dog it is the case that it chases some—potentially different—cats; or else it can mean that there is a particular group of cats which are chased by every dog. The two readings are all made up of the same set of predicates and operators, but differ in the relative scopes of certain
every(x, dog(x), some(y, cat(y), chase(e1, x, y) ∧ happy(e2, e1)))
(a) String-based representation
This formula is comparable to the following first-order formula:  $\forall x(\mathsf{dog}(\mathbf{x})\to \exists \mathbf{y}(\mathsf{cat}(\mathbf{y})\land (\mathsf{chase}(\mathbf{e}_1,\mathbf{x},\mathbf{y})\land$  happy(e2,e1))))
6773
scope bearing elements. There are some other natural language constructions that also involve scope ambiguity, e.g. negation and modality.
Underspecification is by now the standard technique to deal with semantic ambiguities in many modern semantic theories, e.g. Underspecified Discourse Representation Theory (Kamp et al., 2011) and Hole Semantics (Bos, 1996). The basic idea behind it is to derive a single compact representation that describes the set of readings for a sentence that exhibits a scope ambiguity. The individual readings can be enumerated from such an underspecified description if it is required (Koller and Thater, 2005), but it is also possible to process underspecified representations directly without enumerating the readings (Koller and Thater, 2010).
In this paper, we make our logico-semantic graph representations expressive to exhibit the complexities of human language semantics to some extent, by adopting a specific formalism for underspecification, i.e. Minimal Recursion Semantics (MRS; Copestake et al., 2005), a widely-used computational semantic framework in NLP. In addition to variables to represent individuals or events, an MRS structure use another kind of element, called handle, to represent out-of-scope relationships between predicates. Each node is assigned with a label handle, and some arguments of a concept are specified as hole handles. Note that a hole argument is different from an event-variable argument, as illustrated by Ex. (1). Handles can be added to current variable-in-situ graph as a new type of node. See Fig. 3 for an example.  $h_1, h_2, h_3, h_4$  and  $h_5$  are labels,  $h_2, h_5, h_7, h_9$  are hole handles. The out-of-scope relationships in logical forms are converted into a set of constraints between holes and labels. For example, if we let  $h_7 = h_4$  and  $h_9 = h_3$ , the MRS will be resolved into reading Ex. (2c); similarly,  $h_4 = h_1$  and  $h_7 = h_3$  for reading Ex. (2b).
To be more precise, a variable-in-situ logicosemantic graph is a graph such that,
- every node must be a predicate, handle or variable;
- every edge must be (1) between a predicate and a variable, encoding predicate–argument relation, (2) between a predicate and a handle, encoding scopal argument or (3) between a predicate and a label, encoding naming convention and tagged by “L.”
Figure 3: Underspecified logico-semantic graph. Handles associated to predicates are labeled with "L," while handles play as arguments are labeled with semantic roles, like "RSTR."
2.3 Structural Validity
Considering any type of logical form-equivalent representation, we need to be careful that our structures are well-formed. MRS provides a principled way to enumerate readings from an underspecified logical form (Niehren and Thater, 2003), showing us a way to validate the output logic structure. We thus define a valid semantic structure as an MRS in which a scope-resolved logical form is allowable. To be more precise, a variable-in-situ logic-semantic graph is valid if and only if there exists at least one fully specified logical form that satisfies all the constraints encoded by the graph.
3 Neural Graph Rewriting
Automatically constructing a semantic representation can be achieved by exploring the compositionality principle: The meaning of a complex expression is a function of the meanings of its parts and of the syntactic rules by which they are combined. In this perspective, both meanings of its parts and the function of syntactic rules can be precisely defined by graph fragments. In this paper, we investigate how to manipulate semantic graph fragments with HRG, a context-free rewriting system for generating graphs. We give a formal description of HRG, and then show how to model syntactic-semantic composition through graph rewriting. Recursive neural networks are also important for handling linguistic data. In this section, we will further augment an HRG with a hypergraph-state LSTM.
3.1 Gluing Graph Fragments with an HRG
An edge-labeled, node-typed hypergraph is a tuple  $H = \langle V, E, l, t, X \rangle$ , where  $V$  is a finite set
6774
①	②	③	④
LHS	N	NP	VP	S
Type	h×	x	xx	∅
RHS(sem.)	h○Body○RSTR○every×ARG0○	○○D○×	○○V○×	○○NP○×VP
Syntax	every	D+N	V+NP	NP+VP
Table 1: Example HRG rules. Throughout this paper, we use filled black nodes to indicate external nodes, arrows to indicate single-node edges and directed arcs to indicate edges connected to two nodes. The edge labeled as  $\mathbf{V}$  in Rule ③ connects more than two nodes whose orders are indicated by tiny numbers around lines. We use single-node edges with underlined terminal labels to represent predicates, e.g. every.
of nodes, and  $E \subseteq V^{+}$  is a finite set of hyperedges. A hyperedge is an extension of a normal edge which can connect to more than two nodes or only one node.  $l: E \to L$  assigns a label from a finite set  $L$  to each hyperedge. Since nodes receive no informative labels, we use single-node edges with terminal labels to represent predicates. This strategy is widely used by HRG-based NLP systems, including Chiang et al. (2013), Peng et al. (2015) and Chen et al. (2018).  $X \in V^{*}$  defines an ordered list of nodes called external nodes, which specify the docking points during graph rewriting.  $t: V \to T$  assigns a type from a finite set  $T$  to each node.
Different from the hypergraphs used by Chiang et al. (2013) and Chen et al. (2018), we highlight the usage of node types which has a significant impact on making parsing results logically coherent. Three node types are utilized: h, x and c, which indicate handle, variable and predicate respectively. During node gluing, we must make sure that the types of nodes are identical. If the type of any node is still unspecified, the type of the other node will be selected. For convenience, we define the type of a non-terminal hyperedge as the tuple of types of all nodes it connects to; we define the type of a graph fragment as the tuple of types of all external nodes in order. For example, the graph fragment of some in Fig. 4 is typed as  $(\mathrm{h},\mathrm{x})$ , which will be denoted as hx for short.
ATyped Hyperedge Replacement Grammar (THRG)  $G = \langle N,T,P\rangle$  is a graph rewriting sys
Figure 4: Semantic composition as graph gluing with the rule in Tab. 1. The top composition is according to rule  $③$ , where the left graph that is labeled as  $\mathbf{V}$  is glued with the right graph that is labeled as NP by combining their separated external nodes that are labeled as "0."
tem, where  $N$  and  $T$  are two disjoint finite sets of non-terminal and terminal symbols respectively.  $P$  is a finite set of production rules of the form  $A \rightarrow R$ , where the left hand side (LHS)  $A \in N$ , and the right hand side (RHS)  $R$  is a hypergraph with edge labels over  $N \cup T$ . The rewriting process replaces a non-terminal hyperedge with the graph fragment specified by a rule's RHS, attaching each external node to the matched node of the corresponding LHS. In the meantime, the co-related nodes in LHS and RHS must be of the same types. Tab. 1 presents four example rules. Rule ③ consists of three nodes and two hyperedges. All three nodes are of type x, indicating that they are variables. One hyperedge has a label NP and connects to one internal node; the other is labelled as VP and connects to one internal node and two external nodes. Fig. 4 presents the composition process for chase some cat, in which Rule ② and ③ are recursively called for semantic construction.
The types of a HRG rule put additional constraints to the combination of subgraphs and in this way the output graph is regularized to some extent. A failed combination is illustrated in Fig. 5.
6775
Figure 5: We deliberately swap the two external nodes of graph fragment cat. This combination can be blocked by type restrictions.
3.2 Recursive Hypergraph-state LSTM
Since we explicitly describe a recursive process, we are able to define a new graph embedding method—encoding graphs along with such a recursive structure. Our strategy is to assign vectors to nodes involved in the composition process in a bottom-up way. Before the application of an HRG rule  $A \rightarrow R$  ( $R = \langle V, E, l, t, X \rangle$ ,  $V = \{n_1, n_2, \ldots\}$ ,  $X = \{e_1, e_2, \ldots\}$ ), the external nodes of all non-terminal edges in  $R$  have been assigned vectors based on preceding composition while other newly introduced nodes are zero-initialized. The vectors assigned to all nodes in  $R$  will be updated according to a Graph Neural Network (GNN), which works by exploiting locality encoded by  $R$ . In this paper, we propose a hypergraph-state LSTM structure to do so. In what follows, we will first introduce our GNN model and then use it to equip an HRG, resulting in a recursive hypergraph-state LSTM model.
Each node  $n_j \in V$  has a node property vector  $x_{n_j}$  to represent its own information, such as the type and the corresponding label of a concept node, and the index of an external node. And another hidden state vector  $h_j$  is employed to hopefully encode the information of its surroundings. The surrounding information of  $n_j$  is collected by multi-step information exchange between  $n_j$  and its neighbouring nodes, denoted as  $\pi(n_j)$ . Two nodes  $n_j$  and  $n_k$  are viewed as neighbours if there is at least one hyperedge that connects them. To keep its own information, we assume that each node has a self loop, i.e.  $n_j \in \pi(n_j)$ . Thus the neighbouring relation is symmetric. An optional label  $l(n_j, n_k)$  can be attached to each neighboring relation.
Each node has an initial state  $h_j^0$ , representing the state when information has not been updated yet. In each step of information exchange, according to  $x_j$  and its previous hidden state  $h_j^{t-1}$ , the new hidden state  $h_j^t$  is calculated from the repre
sensation of itself, its neighbours  $\pi (n_j)$ , and the label of each relation, in a way as generally defined as follows:
$$
\begin{array}{l} \boldsymbol {h} _ {j} ^ {t} = f \left(\left\{\boldsymbol {x} _ {k} \mid k \in \pi \left(n _ {j}\right) \right\}, \left\{\boldsymbol {h} _ {k} ^ {t - 1} \mid k \in \pi \left(n _ {j}\right) \right\}, \right. \\ \left\{l \left(n _ {j}, n _ {k}\right) \mid n _ {k} \in \pi \left(n _ {j}\right) \right\}) \\ \end{array}
$$
Assume that  $\pmb{L}$  is a randomly initialized matrix for encoding neighbouring labels. Summation is utilized to collect information from neighbouring nodes:
$$
\boldsymbol {\Pi} _ {x, j} = \sum_ {k \in \pi (n _ {j})} \left(\boldsymbol {x} _ {k} \oplus \boldsymbol {L} [ l (n _ {j}, n _ {k}) ]\right)
$$
$$
\boldsymbol {\Pi} _ {h, j} ^ {t - 1} = \sum_ {k \in \pi (n _ {j})} \boldsymbol {h} _ {k} ^ {t - 1}
$$
Introducing the LSTM gate mechanism, the state transition can be written as:
$$
\boldsymbol {i} _ {j} ^ {t} = \sigma \left(\boldsymbol {W} _ {i} \boldsymbol {\Pi} _ {x, j} + \boldsymbol {U} _ {i} \boldsymbol {\Pi} _ {h, j} ^ {t - 1} + \boldsymbol {b} _ {i}\right)
$$
$$
\pmb {o} _ {j} ^ {t} = \sigma (\pmb {W} _ {o} \pmb {\Pi} _ {x, j} + \pmb {U} _ {o} \pmb {\Pi} _ {h, j} ^ {t - 1} + \pmb {b} _ {o})
$$
$$
\boldsymbol {f} _ {j} ^ {t} = \sigma (\boldsymbol {W} _ {\boldsymbol {f}} \boldsymbol {\Pi} _ {x, j} + \boldsymbol {U} _ {\boldsymbol {f}} \boldsymbol {\Pi} _ {h, j} ^ {t - 1} + \boldsymbol {b} _ {f})
$$
$$
\boldsymbol {u} _ {j} ^ {t} = \sigma (\boldsymbol {W} _ {\boldsymbol {u}} \boldsymbol {\Pi} _ {x, j} + \boldsymbol {U} _ {\boldsymbol {u}} \boldsymbol {\Pi} _ {h, j} ^ {t - 1} + \boldsymbol {b} _ {u})
$$
$$
\boldsymbol {c} _ {j} ^ {t} = \boldsymbol {f} _ {j} ^ {t} \otimes \boldsymbol {c} _ {j} ^ {t - 1} + \boldsymbol {i} _ {j} ^ {t} \otimes \boldsymbol {u} _ {j} ^ {t}
$$
$$
\boldsymbol {h} _ {j} ^ {t} = \boldsymbol {o} _ {j} ^ {t} \otimes \tanh (\boldsymbol {c} _ {j} ^ {t})
$$
where  $i, o, f$  are the input, output and forget gates of LSTM.  $W$  and  $U$  are the model parameters.
Similar to the tree LSTM (Tai et al., 2015), our recursive hyperedge-state LSTM model composes the states of a graph fragment from input vectors and the representations of its subgraphs. The model alternates between two kinds of steps: (1) graph fragment encoding and (2) state propagation. The process for encoding a non-leaf graph fragment is visualized in Fig. 6. The most important feature of our graph encoding method is that the process is step-wise, making it possible to perform semantic disambiguation and graph encoding iteratively.
In a graph fragment encoding step for  $R$ , we want to get some vectors representing a specific graph fragment for further combination. This can be done by running multilayer hypergraph-state LSTM (denoted as HGS) on  $R$ :
$$
\begin{array}{l} [ \pmb {h} _ {n _ {1}} ^ {T}; \pmb {h} _ {n _ {2}} ^ {T}; \dots ] = \mathrm {H G S} ^ {T} ([ \pmb {h} _ {n _ {1}} ^ {0}; \pmb {h} _ {n _ {2}} ^ {0}; \dots ], \\ [ \boldsymbol {x} _ {n _ {1}}; \boldsymbol {x} _ {n _ {2}}; \dots ], R) \\ \end{array}
$$
6776
Figure 6: A graphical illustration of our recursive hypergraph-state LSTM model. “ $\uparrow$  HGS” represents graph encoding with hypergraph-state LSTM. The final hidden states  $\boldsymbol{h}^T$  of external nodes are used as interface vectors (brown vectors). Solid lines across the boxes denotes state propagation steps, we initialize hidden states according to the corresponding HRG rule, e.g.  $h_{\mathrm{NP},0}^{0} = h_{\mathrm{D},1}^{T} + h_{\mathrm{N},1}^{T}$  and  $h_{\mathrm{NP},\mathrm{i}}^{0} = h_{\mathrm{D},0}^{T} + h_{\mathrm{N},0}^{T}$ .
$T$  represents the number of layers in the hypergraph-state LSTM. For a node  $n_j$  in a lexical graph fragment, we use a zero vector as  $\pmb{h}_{n_j}^0$ . For the non-leaf case,  $\pmb{x}$  and  $\pmb{h}^0$  is acquired from preceding state propagation. Not all final states  $\pmb{h}_{n_1}^T,\pmb{h}_{n_2}^T\dots$  should be kept for further composition. Considering the role played by external nodes in graph gluing, we use the final states of external nodes  $\pmb{h}_{e_1}^T,\pmb{h}_{e_2}^T\dots$  to pass information and call them interface vectors.
State propagation is the preparatory stage of non-leaf graph fragment encoding, in which the interface vectors of its subgraph fragments are combined to calculate  $\pmb{x}$  and  $h^0$  for the next step. Without the loss of generality, we only discuss the case for binary rules in which  $R$  consists of two non-terminal hyperedges. It is worth noting that in non-leaf graph fragment encoding, the hypergraph-state LSTM is operated on a rule rather than the entire graph fragment. The process of encoding a non-leaf graph fragment can be seen as encoding an RHS  $R$  with special initial states originated from interface vectors. The nodes in  $R$  are of three types: unified nodes, passover nodes and newly created nodes. Newly created nodes bring new information to the combined graph frag
ment while the other two kinds of nodes are only used for structural connection. For a newly created node, the node property vector  $\pmb{x}$  is calculated from its own information, and the initial state is a zero vector. A unified node is connected by both non-terminal hyperedges, and therefore receive information from both sides. The initial state  $h^0$  of a unified node is the sum of the two corresponding interface vectors. The property vector  $\pmb{x}$  is redefined as the sum of the two related property vectors. A passover node is a node connected to only one non-terminal hyperedge. And its property vector and initial state are simply copied from the unique corresponding node. For example, the rule VP in Tab. 1 contains one unified node and two passover nodes. Denote the set of corresponding nodes of  $n_j$  as  $\operatorname{cor}(n_j)$ .  $|\operatorname{cor}(n_j)|$  is 0, 1 or 2 for newly created nodes, passover nodes and unified nodes respectively.  $\pmb{x}_{n_j}$  and  $h_{n_j}^0$  for non-leaf graph fragment encoding can be calculated as:
$$
\boldsymbol {h} _ {n _ {j}} ^ {0} = \sum_ {n _ {i} \in \operatorname {c o r} (n _ {j})} \boldsymbol {h} _ {n _ {i}} ^ {T}
$$
$$
\boldsymbol {x} _ {n _ {j}} = \sum_ {n _ {i} \in \operatorname {c o r} (n _ {j})} \boldsymbol {x} _ {n _ {i}} \text {i f} | \operatorname {c o r} (n _ {j}) | \neq 0
$$
4 Parsing to Variable-in-situ Graphs
Following our previous work (Chen et al., 2018), we continue to employ a synchronous grammar to build a practical parser. We integrate a CFG that expresses syntactic composition with an HRG that expresses semantic composition. Semantic construction is divided into two subtasks: syntactic parsing and semantic interpretation. When a phrase structure tree  $T$  is available, a semantic interpreter translates  $T$  to the derivation of graph construction by assigning corresponding HRG rules to the syntactic counterparts. At a single derivation step, there may be more than one HRG rule applicable. In this case, we need a disambiguation model to select a good one.