SlowGuess/ACL_2020_OCR huggingface dataset:
AMR Parsing via Graph Sequence Iterative Inference*
Deng Cai
thisisjcykcd@gmail.com
Wai Lam
wlam@se.cuhk.edu.hk
We propose a new end-to-end model that treats AMR parsing as a series of dual decisions on the input sequence and the incrementally constructed graph. At each time step, our model performs multiple rounds of attention, reasoning, and composition that aim to answer two critical questions: (1) which part of the input sequence to abstract; and (2) where in the output graph to construct the new concept. We show that the answers to these two questions are mutually causalities. We design a model based on iterative inference that helps achieve better answers in both perspectives, leading to greatly improved parsing accuracy. Our experimental results significantly outperform all previously reported SMATCH scores by large margins. Remarkably, without the help of any large-scale pre-trained language model (e.g., BERT), our model already surpasses previous state-of-the-art using BERT. With the help of BERT, we can push the state-of-the-art results to  $80.2\%$  on LDC2017T10 (AMR 2.0) and  $75.4\%$  on LDC2014T12 (AMR 1.0).
Abstract Meaning Representation (AMR) (Banarescu et al., 2013) is a broad-coverage semantic formalism that encodes the meaning of a sentence as a rooted, directed, and labeled graph, where nodes represent concepts and edges represent relations (See an example in Figure 1). AMR parsing is the task of transforming natural language text into AMR. One biggest challenge of AMR parsing is the lack of explicit alignments between nodes (concepts) in the graph and words in the text. This characteristic not only poses great difficulty in concept
prediction but also brings a close tie for concept prediction and relation prediction.
While most previous works rely on a pre-trained aligner to train a parser, some recent attempts include: modeling the alignments as latent variables (Lyu and Titov, 2018), attention-based sequence-to-sequence transduction models (Barzdins and Gosko, 2016; Konstas et al., 2017; van Noord and Bos, 2017), and attention-based sequence-to-graph transduction models (Cai and Lam, 2019; Zhang et al., 2019b). Sequence-to-graph transduction models build a semantic graph incrementally via spanning one node at every step. This property is appealing in terms of both computational efficiency and cognitive modeling since it mimics what human experts usually do, i.e., first grasping the core ideas then digging into more details (Banarescu et al., 2013; Cai and Lam, 2019).
Unfortunately, the parsing accuracy of existing works including recent state-of-the-arts (Zhang et al., 2019a,b) remain unsatisfactory compared to human-level performance, $^{1}$  especially in cases where the sentences are rather long and informative, which indicates substantial room for improvement. One possible reason for the deficiency is the inherent defect of one-pass prediction process; that is, the lack of the modeling capability of the interactions between concept prediction and relation prediction, which is critical to achieving fully-informed and unambiguous decisions.
We introduce a new approach tackling AMR parsing, following the incremental sequence-to-graph transduction paradigm. We explicitly characterize each spanning step as the efforts for finding which part to abstract with respect to the input sequence, and where to construct with respect to the partially constructed output graph. Equivalently,
<sup>1</sup>The average annotator vs. inter-annotator agreement (SMATCH) was 0.83 for newswire and 0.79 for web text according to Banarescu et al. (2013).
*The work described in this paper is substantially supported by grants from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14204418) and the Direct Grant of the Faculty of Engineering, CUHK (Project Code: 4055093).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290-1301 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
we treat AMR parsing as a series of dual decisions on the input sequence and the incrementally constructed graph. Intuitively, the answer of what concept to abstract decides where to construct (i.e., the relations to existing concepts), while the answer of where to construct determines what concept to abstract. Our proposed model, supported by neural networks with explicit structure for attention, reasoning, and composition, integrated with an iterative inference algorithm. It iterates between finding supporting text pieces and reading the partially constructed semantic graph, inferring more accurate and harmonious expansion decisions progressively. Our model is aligner-free and can be effectively trained with limited amount of labeled data. Experiments on two AMR benchmarks demonstrate that our parser outperforms the previous best parsers on both benchmarks. It achieves the best-reported SMATCH scores (F1):  $80.2\%$  on LDC2017T10 and  $75.4\%$  on LDC2014T12, surpassing the previous state-of-the-art models by large margins.
On a coarse-grained level, we can categorize existing AMR parsing approaches into two main classes: Two-stage parsing (Flanigan et al., 2014; Lyu and Titov, 2018; Zhang et al., 2019a) uses a pipeline design for concept identification and relation prediction, where the concept decisions precede all relation decisions; One-stage parsing constructs a parse graph incrementally. For more fine-grained analysis, those one-stage parsing methods can be further categorized into three types: Transition-based parsing (Wang et al., 2016; Damonte et al., 2017; Ballesteros and Al-Onaizan, 2017; Peng et al., 2017; Guo and Lu, 2018; Liu et al., 2018; Wang and Xue, 2017; Naseem et al., 2019) processes a sentence from left-to-right and constructs the graph incrementally by alternately inserting a new node or building a new edge. Seq2seq-based parsing (Barzdins and Gosko, 2016; Konstas et al., 2017; van Noord and Bos, 2017; Peng et al., 2018) views parsing as sequence-to-sequence transduction by some linearization of the AMR graph. The concept and relation prediction are then treated equally with a shared vocabulary. The third class is graph-based parsing (Cai and Lam, 2019; Zhang et al., 2019b), where at each time step, a new node along with its connections to existing nodes are jointly decided, either in order (Cai and Lam, 2019) or in parallel (Zhang et al., 2019b). So far, the recip-
The current partial (solid) and full (solid + dashed) AMR graphs for the sentence "The boy must no go"
Figure 1: AMR graph construction given the partially constructed graph: (a) one possible expansion resulting in the boy concept. (b) another possible expansion resulting in the - (negation) concept.
rocal causation of relation prediction and concept prediction has not been closely-studied and well-utilized.
There are also some exceptions staying beyond the above categorization. Peng et al. (2015) introduce a synchronous hyperedge replacement grammar solution. Pust et al. (2015) regard the task as a machine translation problem, while Artzi et al. (2015) adapt combinatory categorical grammar. Groschwitz et al. (2018); Lindemann et al. (2019) view AMR graphs as the structure AM algebra.
Our approach is inspired by the deliberation process when a human expert is deducing a semantic graph from a sentence. The output graph starts from an empty graph and spans incrementally in a node-by-node manner. At any time step of this process, we are distilling the information for the next expansion. We call it expansion because the new node, as an abstract concept of some specific text fragments in the input sentence, is derived to complete some missing elements in the current semantic graph. Specifically, given the input sentence and the current partially constructed graph, we are answering two critical questions: which part of the input sequence to abstract, and where in the output graph to construct the new concept. For instance, Figure 1(a) and (b) show two possible choices for the next expansion. In Figure 1(a), the word "boy" is abstracted to the concept boy to complement the subject information of the event go-02. On the
Figure 2: Overview of the dual graph-sequence iterative inference for AMR parsing. Given the current graph  $G^i$  and input sequence  $W$ . The inference starts with an initial concept decision  $x_0$  and follows the inference chain  $x_0 \to f(G^i, x_0) \to y_1 \to g(W, y_1) \to x_1 \to f(G^i, x_1) \to y_2 \to g(W, y_2) \to \dots$ . The details of  $f$  and  $g$  are shown in red and blue boxes, where nodes in graph and tokens in sequence are selected via attention mechanisms.
other hand, in Figure 1(b), a polarity attribute of the event go-2 is constructed, which is triggered by the word "not" in the sentence.
We note that the answer to one of the questions can help answer the other. For instance, if we have decided to render the word "not" to the graph, then we will consider adding an edge labeled as polarity, and finally determine its attachment to the existing event go-2 (rather than an edge labeled ARG0 to the same event go-2, though it is also present in the golden graph). On the other hand, if we have decided to find the subject (ARG0 relation) of the action go-02, we are confident to locate the word "boy" instead of function words like "not" or "must", thus unambiguously predict the right concept boy. Another possible circumstance is that we may make a mistake trying to ask something that is not present in the sentence (e.g., the destination of the go-02 action). This attempt will be rejected by a review of the sentence. The rationale is that literally we cannot find the destination information in the sentence. Similarly, if we mistakenly propose to abstract some parts of the sentence that are not ready for construction yet, the proposal will be rejected by another inspection on the graph since that there is nowhere to place such a new concept.
We believe the mutual causalities, as described above, are useful for action disambiguation and harmonious decision making, which eventually result in more accurate parses. We formulate AMR parsing as a series of dual graph-sequence decisions and design an iterative inference approach
to tackle each of them. It is sort of analogous to the cognition procedure of a person, who might first notice part of the important information in one side (graph or sequence), then try to confirm her decision at the other side, which could just refute her former hypothesis and propose a new one, and finally converge to a conclusion after multiple rounds of reasoning.
4 Proposed Model
Formally, the parsing model consists of a series of graph expansion procedures  $\{G^0\to \ldots \to G^i\to \ldots \}$ , starting from an empty graph  $G^0$ . In each turn of expansion, the following iterative inference process is performed:
$$
y _ {t} ^ {i} = g (G ^ {i}, x _ {t} ^ {i}),
$$
$$
x _ {t + 1} ^ {i} = f (W, y _ {t} ^ {i}),
$$
where  $W, G^{i}$  are the input sequence and the current semantic graph respectively.  $g(\cdot), f(\cdot)$  seek where to construct (edge prediction) and what to abstract (node prediction) respectively, and  $x_{t}^{i}, y_{t}^{i}$  are the  $t$ -th graph hypothesis (where to construct) and  $t$ -th sequence hypothesis (what to abstract) for the  $i$ -th expansion step respectively. For clarity, we may drop the superscript  $i$  in the following descriptions.
Figure 2 depicts an overview of the graph sequence iterative inference process. Our model has four main components: (1) Sequence Encoder, which generates a set of text memories (per token)
to provide grounding for concept alignment and abstraction; (2) Graph Encoder, which generates a set of graph memories (per node) to provide grounding for relation reasoning; (3) Concept Solver, where a previous graph hypothesis is used for concept prediction; and (4) Graph Solver, where a previous concept hypothesis is used for relation prediction. The last two components correspond to the reasoning functions  $g(\cdot)$  and  $f(\cdot)$  respectively.
The text memories can be computed by Sentence Encoder at the beginning of the whole parsing while the graph memories are constructed by Graph Encoder incrementally as the parsing progresses. During the iterative inference, a semantic representation of current state is used to attend to both graph and text memories (blue and red arrows) in order to locate the new concept and obtain its relations to the existing graph, both of which subsequently refine each other. Intuitively, after a first glimpse of the input sentence and the current graph, specific sub-areas of both sequence and graph are revisited to obtain a better understanding of the current situation. Later steps typically read the text in detail with specific learning aims, either confirming or overturning a previous hypothesis. Finally, after several iterations of reasoning steps, the refined sequence/graph decisions are used for graph expansion.
4.2 Sequence Encoder
As mentioned above, we employ a sequence encoder to convert the input sentence into vector representations. The sequence encoder follows the multi-layer Transformer architecture described in Vaswani et al. (2017). At the bottom layer, each token is firstly transformed into the concatenation of features learned by a character-level convolutional neural network (charCNN, Kim et al., 2016) and randomly initialized embeddings for its lemma, part-of-speech tag, and named entity tag. Additionally, we also include features learned by pre-trained language model BERT (Devlin et al., 2019).
Formally, for an input sequence  $w_{1},w_{2},\ldots ,w_{n}$  with length  $n$  we insert a special token BOS at the beginning of the sequence. For clarity, we omit the detailed transformations (Vaswani et al., 2017) and denote the final output from our sequence encoder as  $\{h_0,h_1,\dots ,h_n\} \in \mathbb{R}^d$  where  $h_0$  corresponds the special token BOS and serves as an overall rep
resentation while others are considered as contextualized word representations. Note that the sequence encoder only needs to be invoked once, and the produced text memories are used for the whole parsing procedure.
4.3 Graph Encoder
We use a similar idea in Cai and Lam (2019) to encode the incrementally expanding graph. Specifically, a graph is simply treated as a sequence of nodes (concepts) in the chronological order of when they are inserted into the graph. We employ multi-layer Transformer architecture with masked self-attention and source-attention, which only allows each position in the node sequence to attend to all positions up to and including that position, and every position in the node sequence to attend over all positions in the input sequence.<sup>3</sup> While this design allows for significantly more parallelization during training and computation-saving incrementality during testing,<sup>4</sup> it inherently neglects the edge information. We attempted to alleviate this problem by incorporating the idea of Strubell et al. (2018) that applies auxiliary supervision at attention heads to encourage them to attend to each node's parents in the AMR graph. However, we did not see performance improvement. We attribute the failure to the fact that the neural attention mechanisms on their own are already capable of learning to attend to useful graph elements, and the auxiliary supervision is likely to disturb the ultimate parsing goal.
Consequently, for the current graph  $G$  with  $m$  nodes, we take its output concept sequence  $c_{1}, c_{2}, \ldots, c_{m}$  as input. Similar to the sequence encoder, we insert a special token BOG at the beginning of the concept sequence. Each concept is firstly transformed into the concatenation of feature vector learned by a char-CNN and randomly initialized embedding. Then, a multi-layer Transformer encoder with masked self-attention and source-attention is applied, resulting in vector representations  $\{s_{0}, s_{1}, \ldots, s_{m}\} \in \mathbb{R}^{d}$ , where  $s_{0}$  represents the special concept BOG and serves as a dummy node while others are considered as contextualized node representations.
2We obtain word-level representations from pre-trained BERT in the same way as Zhang et al. (2019a,b), where subtoken representations at the last layer are averaged.
<sup>3</sup>It is analogous to a standard Transformer decoder (Vaswani et al., 2017) for sequence-to-sequence learning.
4Trivially employing a graph neural network here can be computationally expensive and intractable since it needs to re-compute all graph representations after every expansion.
4.4 Concept Solver
At each sequence reasoning step  $t$ , the concept solver receives a state vector  $y_{t}$  that carries the latest graph decision and the input sequence memories  $h_1, \ldots, h_n$  from the sequence encoder, and aims to locate the proper parts in the input sequence to abstract and generate a new concept. We employ the scaled dot-product attention proposed in Vaswani et al. (2017) to solve this problem. Concretely, we first calculate an attention distribution over all input tokens:
$$
\alpha_ {t} = \mathrm {s o f t m a x} (\frac {(W ^ {Q} y _ {t}) ^ {\mathrm {T}} W ^ {K} h _ {1 : n}}{\sqrt {d _ {k}}}),
$$
where  $\{W^Q,W^K\} \in \mathbb{R}^{d_k\times d}$  denote learnable linear projections that transform the input vectors into the query and key subspace respectively, and  $d_{k}$  represents the dimensionality of the subspace.
The attention weights  $\alpha_{t} \in \mathbb{R}^{n}$  provide a soft alignment between the new concept and the tokens in the input sequence. We then compute the probability distribution of the new concept label through a hybrid of three channels. First,  $\alpha_{t}$  is fed through an MLP and softmax to obtain a probability distribution over a pre-defined vocabulary:
$$
\operatorname {M L P} \left(\alpha_ {t}\right) = \left(W ^ {V} h _ {1: n}\right) \alpha_ {t} + y _ {t} \tag {1}
$$
$$
P ^ {(\mathrm {v o c a b})} = \operatorname {s o f t m a x} \left(W ^ {(\mathrm {v o c a b})} \mathrm {M L P} \left(\alpha_ {t}\right) + b ^ {(\mathrm {v o c a b})}\right),
$$
where  $W^{V} \in \mathbb{R}^{d \times d}$  denotes the learnable linear projection that transforms the text memories into the value subspace, and the value vectors are averaged according to  $\alpha_{t}$  for concept label prediction. Second, the attention weights  $\alpha_{t}$  directly serve as a copy mechanism (Gu et al., 2016; See et al., 2017), i.e., the probabilities of copying a token lemma from the input text as a node label. Third, to address the attribute values such as person names or numerical strings, we also use  $\alpha_{t}$  for another copy mechanism that directly copies the original strings of input tokens. The above three channels are combined via a soft switch to control the production of the concept label from different sources:
$$
[ p _ {0}, p _ {1}, p _ {2} ] = \operatorname {s o f t m a x} (W ^ {(\text {s w i t c h})} \mathrm {M L P} (\alpha_ {t})),
$$
where MLP is the same as in Eq. 1, and  $p_0, p_1$  and  $p_2$  are the probabilities of three prediction channels respectively. Hence, the final prediction probability
of a concept  $c$  is given by:
$$
\begin{array}{l} P (c) = p _ {0} \cdot P ^ {\left(\text {v o c a b}\right)} (c) \\ + p _ {1} \cdot \left(\sum_ {i \in L (c)} \alpha_ {t} [ i ]\right) + p _ {2} \cdot \left(\sum_ {i \in T (c)} \alpha_ {t} [ i ]\right), \\ \end{array}
$$
where  $[i]$  indexes the  $i$ -th element and  $L(c)$  and  $T(c)$  are index sets of lemmas and tokens respectively that have the surface form as  $c$ .
4.5 Relation Solver
At each graph reasoning step  $t$ , the relation solver receives a state vector  $x_{t}$  that carries the latest concept decision and the output graph memories  $s_0, s_1, \ldots, s_m$  from the graph encoder, and aims to point out the nodes in the current graph that have an immediate relation to the new concept (source nodes) and generate corresponding edges. Similar to Cai and Lam (2019); Zhang et al. (2019b), we factorize the task as two stages: First, a relation identification module points to some preceding nodes as source nodes; Then, the relation classification module predicts the relation type between the new concept and predicted source nodes. We leave the latter to be determined after iterative inference.
AMR is a rooted, directed, and acyclic graph. The reason for AMR being a graph instead of a tree is that it allows reentrancies where a concept participates in multiple semantic relations with different semantic roles. Following Cai and Lam (2019), we use multi-head attention for a more compact parsing procedure where multiple source nodes are simultaneously determined. Formally, our relation identification module employs  $H$  different attention heads, for each head  $h$ , we calculate an attention distribution over all existing node (including the dummy node  $s_0$ ):
$$
\beta_ {t} ^ {h} = \operatorname {s o f t m a x} (\frac {\left(W _ {h} ^ {Q} x _ {t}\right) ^ {\mathrm {T}} W _ {h} ^ {K} s _ {0 : m}}{\sqrt {d _ {k}}}).
$$
Then, we take the maximum over different heads as the final edge probabilities:
$$
\beta_ {t} [ i ] = \max  _ {h = 1} ^ {H} \beta_ {t} ^ {h} [ i ].
$$
Therefore, different heads may points to different nodes at the same time. Intuitively, each head represents a distinct relation detector for a particular
This is different to Zhang et al. (2019b) where an AMR graph is converted into a tree by duplicating nodes that have reentrant relations.
Figure 3: Multi-head attention for relation identification. At left is the attention matrix, where each column corresponds to a unique attention head, and each row corresponds to an existing node.
set of relation types. For each attention head, it will point to a source node if certain relations exist between the new node and the existing graph, otherwise it will point to the dummy node. An example with four attention heads and three existing nodes (excluding the dummy node) is illustrated in Figure 3.
4.6 Iterative Inference
As described above, the concept solver and the relation solver are conceptually two attention mechanisms over the sequence and graph respectively, addressing the concept prediction and relation prediction separately. The key is to pass the decisions between the solvers so that they can examine each other's answer and make harmonious decisions. Specifically, at each spanning step  $i$ , we start the iterative inference by setting  $x_0 = h_0$  and solving  $f(G^i,x_0)$ . After the  $t$ -th graph reasoning, we compute the state vector  $y_{t}$ , which will be handed over to the concept solver as  $g(W,y_{t})$ , as:
$$
y _ {t} = \mathrm {F F N} ^ {(y)} (x _ {t} + (W ^ {V} h _ {1: n}) \alpha_ {t}),
$$
where  $\mathrm{FFN}^{(y)}$  is a feed-forward network and  $W^V$  projects text memories into a value space. Similarly, after the  $t$ -th sequence reasoning, we update the state vector from  $y_{t}$  to  $x_{t + 1}$  as:
$$
x _ {t + 1} = \mathrm {F F N} ^ {(x)} (y _ {t} + \sum_ {h = 1} ^ {H} (W _ {h} ^ {V} s _ {0: n}) \beta_ {t} ^ {h}),
$$
where  $\mathrm{FFN}^{(x)}$  is a feed-forward network and  $W_h^V$  projects graph memories into a value space for each head  $h$ . After  $N$  steps of iterative inference, i.e.,
$$
\begin{array}{l} x _ {0} \rightarrow f (G ^ {i}, x _ {0}) \rightarrow y _ {1} \rightarrow g (W, y _ {1}) \rightarrow x _ {1} \rightarrow \dots \\ \rightarrow f (G ^ {i}, x _ {N - 1}) \rightarrow y _ {N} \rightarrow g (W, y _ {N}) \rightarrow x _ {N}, \\ \end{array}
$$
we finally employ a deep biaffine classifier (Dozat and Manning, 2016) for edge label prediction. The
Algorithm 1 AMR Parsing via Graph  $\leftrightarrows$  Sequence Iterative Inference
Input: the input sentence  $W = (w_{1}, w_{2}, \ldots, w_{n})$
Output: the corresponding AMR graph  $G$  // compute text memories