SlowGuess/NAACL_2021_OCR huggingface dataset:
Jiawei Zhou† Tahira Naseem* Ramón Fernandez Astudillo* Radu Florian* †Harvard University *IBM Research
$\dagger$  jzhou02@g.harvard.edu *{tnaseem, raduf}@us.ibm.com *ramon. astudillo@ibm.com
Abstract Meaning Representation parsing is a sentence-to-graph prediction task where target nodes are not explicitly aligned to sentence tokens. However, since graph nodes are semantically based on one or more sentence tokens, implicit alignments can be derived. Transition-based parsers operate over the sentence from left to right, capturing this inductive bias via alignments at the cost of limited expressiveness. In this work, we propose a transition-based system that combines hard-attention over sentences with a target-side action pointer mechanism to decouple source tokens from node representations and address alignments. We model the transitions as well as the pointer mechanism through straightforward modifications within a single Transformer architecture. Parser state and graph structure information are efficiently encoded using attention heads. We show that our action-pointer approach leads to increased expressiveness and attains large gains (+1.6 points) against the best transition-based AMR parser in very similar conditions. While using no graph re-categorization, our single model yields the second best SMATCH score on AMR 2.0 (81.8), which is further improved to 83.4 with silver data and ensemble decoding.
Abstract Meaning Representation (AMR) (Banarescu et al., 2013) is a sentence level semantic formalism encoding who does what to whom in the form of a rooted directed acyclic graph. Nodes represent concepts such as entities or predicates which are not explicitly aligned to words, and edges represent relations such as subject/object (see Figure 1).
AMR parsing, the task of generating the graph from a sentence, is nowadays tackled with sequence to sequence models parametrized with neural networks. There are two broad categories of methods that are highly effective in recent years. Transition-based approaches predict a sequence of
Figure 1: AMR graph expressing the meaning of the sentence The boy wants to go to New York.
actions given the sentence. These actions generate the graph while processing tokens left-to-right through the sentence and store intermediate representations in memories such as stack and buffer (Wang et al., 2015; Damonte et al., 2016; Ballesteros and Al-Onaizan, 2017; Vilares and Gomez-Rodriguez, 2018; Naseem et al., 2019; Astudillo et al., 2020; Lee et al., 2020). General graph-based approaches, on the other hand, directly predict nodes and edges in sequential order from graph traversals such as breath first search or depth first search (Zhang et al., 2019a,b; Cai and Lam, 2019, 2020). While not modeling the local semantic correspondence between graph nodes and source tokens, the approaches achieve strong results without restrictions of transition-based approaches, but often require graph re-categorization, a form of graph normalization, for optimal performance.
The strong left-to-right constraint of transition-based parsers provides a form of inductive bias that fits AMR characteristics. AMR nodes are very often normalized versions of sentence tokens and locality between words and nodes is frequently preserved. The fact that transition-based systems for AMR have alignments as the core of their explanatory model also guarantees that they produce reliable alignments at decoding time, which are useful for applications utilizing AMR parses. Despite these advantages, transition-based systems still suffer in situations when multiple nodes are best explained as aligned to one sentence token or none. Furthermore, long distance edges in AMR, e.g. re-entrancies, require excessive use of SWAP or
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5585-5598 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: Source tokens, target actions and AMR graph for the sentence  $I$  offer a solution to the problem (partially parsed). The black arrow marks the current token cursor position. The circles contain the action indices (used as ids), black circles indicate node creating actions. Only these actions are available for edge attachments. Notice that the edge actions (at steps 3, 7 and 9) explicitly refer to past nodes using the id of the action that created the node. The other participant of the edge action is implicitly assumed to be the most recently created graph node.
equivalent actions, leading to very long action sequences. This in turn affects both a model's ability to learn and its decoding speed.
In this work, we propose the Action-Pointer Transition (APT) system which combines the advantages of both the transition-based approaches and more general graph-generation approaches. We focus on predicting an action sequence that can build the graph from a source sentence. The core idea is to put the target action sequence to a dual use – as a mechanism for graph generation as well as the representation of the graph itself. Inspired by recent progress in pointer-based parsers (Ma et al., 2018a; Fernández-González and Gómez-Rodríguez, 2020), we replace the stack and buffer by a cursor that moves from left to right and introduce a pointer network (Vinyals et al., 2015) as mechanism for edge creation. Unlike previous works, we use the pointer mechanism on the target side, pointing to past node generation actions to create edges. This eliminates the node generation and attachment restrictions of previous transition-based parsers. It is also more natural for graph generation, essentially resembling the generation process in the graph-based approaches, but keeping the graph and source aligned.
We model both the action generation and the pointer prediction with a single Transformer model (Vaswani et al., 2017). We relate target node and source token representations through masking of cross-attention mechanism, similar to Astudillo et al. (2020) but simply with monotonic action-source alignment driven by cursor positions, rather than stack and buffer contents. Finally we also embed the AMR graph structural information in the target decoder by re-purposing edge-creating steps, and propose a novel step-wise incremental graph message passing method (Gilmer et al., 2017) en
abled by the decoder self-attention mechanism.
Experiments on AMR 1.0, AMR 2.0, and AMR 3.0 benchmark datasets show the effectiveness of our APT system. We outperform the best transition-based systems while using sensibly shorter action sequences, and achieve better performance than all previous approaches with similar size of training parameters.
2 AMR Generation with Action-Pointer
Figure 2 shows a partially parsed example of a source sentence, a transition action sequence and the AMR graph for the proposed transitions. Given a source sentence  $\mathbf{x} = x_{1},x_{2},\ldots ,x_{S}$ , our transition system works by scanning the sentence from left to right using a cursor  $c_{t}\in \{1,2,\dots ,S\}$ . Cursor movement is controlled by three actions:
SHIFT moves cursor one position to the right, such that  $c_{t + 1} = c_t + 1$ .
REDUCE is a special SHIFT indicating that no action was performed at current cursor position.
MERGE merges tokens  $x_{ct}$  and  $x_{ct+1}$  and SHIFTs. Merged tokens act as a single token under the position of the last token merged.
At cursor position  $c_t$  we can generate any subgraph through following actions:
COPY creates a node by copying the word under  $x_{c_t}$ . Since AMR nodes are often lemmas or prop-bank frames, two versions of this action exist to copy the lemma of  $x_{c_t}$  or provide the first sense (frame -01) constructed from the lemma. This covers a large portion of the total AMR nodes. It also helps generalize for predictions of unseen nodes. We use an external lemmatizer<sup>1</sup> for this action.
1https://spacy.io/.
PRED(LABEL) creates a node with name LABEL from the node names seen at train time.
SUBGRAPH(LABEL) produces an entire subgraph indexed by label LABEL. Any future attachments can only be made to the root of the subgraph.
LA(ID,LABEL) creates an arc with LABEL from last generated node to a previous node at position ID. Note that we can only point to past node generating actions in the action history.
RA(ID, LABEL) creates an arc with LABEL to last generated node from a previous node at position ID.
Using the above actions, it is easy to derive an oracle action sequence given gold-graph information and initial word to node alignments. For current cursor position, all the nodes aligned to it are generated using SUBGRAPH(), COPY or PRED() actions. Each node prediction action is followed by edge creation actions. Edges connecting to closer nodes are generated before the farther ones. When multiple connected nodes are aligned to one token, they are traversed in pre-order for node generation. A detailed description of oracle algorithm is given in Appendix B.
The use of a cursor variable  $c_{t}$  decouples node reference from source tokens, allowing to produce multiple nodes and edges (see Figure 3), even the entire AMR graph if necessary, from a single token. This provides more expressiveness and flexibility than previous transition-based AMR parsers, while keeping a strong inductive bias. The only restriction is that all inbound or outbound edges between current node and all previously produced nodes need to be generated before predicting a new node or shifting the cursor. This does not limit the oracle coverage, however, for trained parsers, it leads to a small percentage of disconnected graphs in decoding. Furthermore, nodes within the SUBGRAPH() action can not be reached for edge creation. The use of SUBGRAPH() action, initially introduced in Ballesteros and Al-Onaizan (2017), is reduced in this work to cases where no such edges are expected, which is mainly the case for dates and named-entities.
Compared to previous oracles (Ballesteros and Al-Onaizan, 2017; Naseem et al., 2019; Astudillo et al., 2020), the action-pointer does not use a SWAP action. It can establish an edge between the last predicted node and any previous node, since edges are created by pointing to decoder representations.
Figure 3: Step-by-step actions on the sentence your opinion matters. Creates subgraph from a single word (thing :ARG1-of opine-01) and allows attachment to all its nodes. Cursor is at underlined words (post-action).
This oracle is expected to work with generic AMR aligners. For this work, we use the alignments generation method of Astudillo et al. (2020), which generates many-to-many alignments. It is a combination of Expectation Maximization based alignments of Pourdamghani et al. (2014) and rule base alignments of Flanigan et al. (2014). Any remaining unaligned nodes are aligned based on their graph proximity to unaligned tokens. For more details, we refer the reader to the works of Astudillo et al. (2020) and Naseem et al. (2019).
3 Action-Pointer Transformer
3.1 Basic Architecture
The backbone of our model is the encoder-decoder Transformer (Vaswani et al., 2017), combined with a pointer network (Vinyals et al., 2015). The probability of an action sequence  $\mathbf{y} = y_{1},y_{2},\ldots ,y_{T}$  for input tokens  $\mathbf{x} = x_{1},x_{2},\ldots ,x_{S}$  is given in our model by
$$
\begin{array}{l} \mathbf {P} (\mathbf {y} \mid \mathbf {x}) = \prod_ {t = 1} ^ {T} \mathbf {P} \left(y _ {t} \mid \mathbf {y} _ {<   t}, \mathbf {x}\right) \tag {1} \\ = \prod_ {t = 1} ^ {T} \mathbf {P} \left(a _ {t} \mid \mathbf {a} _ {<   t}, \mathbf {p} _ {<   t}, \mathbf {x}\right) \mathbf {P} \left(p _ {t} \mid \mathbf {a} _ {\leq t}, \mathbf {p} _ {<   t}, \mathbf {x}\right) \\ \end{array}
$$
where at each time step  $t$ , we decompose the target action  $y_{t}$  into the pointer-removed action and the pointer value with  $y_{t} = (a_{t}, p_{t})$ . A dummy pointer  $p_{t} =$  null is fixed for non-edge actions, so that
$$
\mathbf {P} \left(p _ {t} \mid \mathbf {a} _ {\leq t}, \mathbf {p} _ {<   t}, \mathbf {x}\right) = \left[ \mathbf {P} \left(p _ {t} \mid \mathbf {a} _ {<   t}, \mathbf {p} _ {<   t}, \mathbf {x}\right) \right] ^ {\gamma \left(a _ {t}\right)}
$$
where  $\gamma (a_{t})$  is an indicator variable set to 0 if  $a_{t}$  is not an edge action and 1 otherwise.
Given a sequence to sequence Transformer model with  $N$  encoder layers and  $M$  decoder layers, each decoder layer is defined by
$$
\mathbf {d} _ {t} ^ {m} = \mathrm {F F} ^ {m} \left(\mathrm {C A} ^ {m} \left(\mathrm {S A} ^ {m} \left(\mathbf {d} _ {t} ^ {m - 1}, \mathbf {d} _ {\leq t} ^ {m - 1}\right), \mathbf {e} ^ {N}\right)\right)
$$
where  $\mathrm{FF}^m (\cdot)$ $\mathrm{CA}^m ()$  and  $\mathrm{SA}^m ()$  are feedforward, multi-head cross-attention and multi-head self-attention components respectively.  $\mathbf{e}^{N}$  is the output of last encoder layer and  $\mathbf{d}^{m - 1}$  is the output of the previous decoder layer, with  $\mathbf{d}_{\leq t}^{0}$  initialized to be the embeddings of the action history  $\mathbf{y}_{< t}$  concatenated with a special start symbol.
The distribution over actions is given by
$$
\mathbf {P} (a _ {t} \mid \mathbf {a} _ {<   t}, \mathbf {p} _ {<   t}, \mathbf {x}) = \mathrm {s o f t m a x} \left(\mathbf {W} \cdot \mathbf {d} _ {t} ^ {M}\right) _ {a _ {t}}
$$
where  $\mathbf{W}$  are the output vocabulary embeddings, and the edge pointer distribution is given by
$$
\begin{array}{c} \mathbf {P} (p _ {t} \mid \mathbf {a} _ {<   t}, \mathbf {p} _ {<   t}, \mathbf {x}) = \\ \text {s o f t m a x} \big ((\mathbf {K} ^ {M} \cdot \mathbf {d} _ {\leq t} ^ {M - 1}) ^ {T} \cdot \mathbf {Q} ^ {M} \cdot \mathbf {d} _ {t} ^ {M - 1} \big) _ {p _ {t}} \end{array}
$$
where  $\mathbf{K}^M$ ,  $\mathbf{Q}^M$  are key and query matrices of 1 head of the last decoder self-attention layer  $\mathrm{SA}^M(\cdot)$ . The top layer self-attention is a natural choice for the pointer network, since it is likely to have high values for the nodes involved in the edge direction and label prediction. Although the edge action and its pointing value are both output at the same step, the specialized pointer head is also part of the overall self-attention mechanism used to compute the model's hidden representations, thus making actions distribution aware of the pointer distribution.
Our transition system moves the cursor  $c_t$  over the source from left to right during parsing, essentially maintaining a monotonic alignment between target actions and source tokens. We encode the alignment  $c_t$  with hard attentions in cross-attention heads  $\mathrm{CA}^m()$  with  $m = 1 \cdots M$  at every decoder layer. We mask one head of the cross-attention to see only the aligned source token at  $c_t$ , and augment it with another head masked to see only positions  $> c_t$ . This is similar to the hard attention in Peng et al. (2018) and parser state encoding in Astudillo et al. (2020).
As in prior works, we restrict the output space of our model to only allow valid actions given  $\mathbf{x}$ ,  $\mathbf{y}_{<t}$ . The restriction is not only enforced at inference, but
is also internalized with the model during training so that the model can always focus on relevant action subsets when making predictions.
3.2 Incremental Graph Embedding
Incrementally generated graphs are usually modeled via graph neural networks (Li et al., 2018), where a node's representation is updated from the collection of its neighboring nodes' representations by message passing (Gilmer et al., 2017). However, this requires re-computation of all node representations every time the graph is modified, which is expensive, prohibiting its use in previous graph-based AMR parsing works (Cai and Lam, 2020). To better utilize the intermediate topological graph information without losing the efficient parallelization of Transformer, we propose to use the edge creation actions as updated views of each node, that encode this node's neighboring subgraph. This does not change the past computations and can be done by altering the hard masking of the self-attention heads of decoder layers  $\mathrm{SA}^m (\cdot)$ . By interpreting the decoder layers as implementing message passing vertically, we can fully encode graphs up to depth  $M$ .
Given a node generating action  $a_{t} = v$ , it is followed by  $k \geq 0$  edge generating actions  $a_{t+1}, a_{t+2}, \ldots, a_{t+k}$  that connect the current node with previous nodes, pointed by  $p_{t+1}, p_{t+2}, \ldots, p_{t+k}$  positions on the target side. This also defines  $k$  graph modifications, expanding the graph neighborhood on the current node. Figure 4 shows an example for the sentence The boy wants to go, with node prediction actions at positions  $t = 2, 4, 8$ , with  $k$  being 0, 1, 2, respectively. We use the steps from  $t$  to  $t+k$  in the Transformer decoder to encode this expanding neighborhood. In particular, we fix the decoder input as the current node action  $v$  for these steps, as illustrated in the input actions in Figure 4. At each intermediate step  $\tau \in [t, t+k]$ , 2 decoder self-attention heads  $\mathrm{SA}^{m}()$  are restricted to only attend to the direct graph neighbors of the current node, represented by previous nodes at positions  $p_{t}, p_{t+1}, \cdots, p_{\tau}$  as well as the current position  $\tau$ . This essentially builds sub-sequences of node representations with richer graph information step by step, and we use the last reference of the same node for pointing positions when generating new edges. Moreover, when propagating this masking pattern along  $m$  layers, each node encodes its  $m$ -hop neighborhood information.
2Each of these are wrapped around with residual, dropout and layer normalization operations removed for simplicity.
Figure 4: Encoding graph with 2 decoder layers for the sentence The boy wants to go. From top to bottom: target output action sequence, masked decoder self-attention, input action history and partial graph. Edge-creating action steps in the action history are used to hold updated node representations. Action labels and edge direction treatment are removed for clarity.
This defines a message passing procedure as shown in Figure 4, encoding the compositional relations between nodes. Since the edges have directions indicated by LA and RA, we also encode the direction information by separating the two heads with each only considering one direction.
4 Training and Inference
Our model is trained by maximizing the log likelihood of Equation (1). The valid action space, action-source alignment  $c_{t}$ , and the graph embedding mask at each step  $t$  are pre-calculated at training time. For inference, we modify the beam search algorithm to jointly search for actions and edge pointers and combine them to find the action sequence that maximizes Equation (1). We also consider hard constraints in the searching process such as valid output actions and valid target pointing values at different steps to ensure an AMR graph is recoverable. For the structural information that is extracted from the parsing state such as  $c_{t}$  and graph embedding masks, we compute them on the fly at each new step of decoding based on the current results, which are then used by the model for the next step decoding. We detail our search algorithm in Appendix C.
Data and Evaluation We test our approach on two widely used AMR parsing benchmark datasets: AMR 2.0 (LDC2017T10) and AMR 1.0 (LDC2014T12). The AMR graphs are all human annotated. The two datasets have 36521 and 10312 training AMRs, respectively, and share
1368 development AMRs and 1371 testing  $\mathrm{AMRs}^3$ . We also report results on the latest AMR 3.0 (LDC2020T02) dataset, which is larger in size but has not been fully explored, with 55635 training AMRs and 1722 and 1898 AMRs for development and testing set. Wiki links are removed in the preprocessing of data, and we run a wikification approach in post-processing to recover Wikipedia entries in the AMR graphs as in Naseem et al. (2019).
For evaluation, we use the SMATCH (F1) scores $^4$  (Cai and Knight, 2013) and further the fine-grained evaluation metrics (Damonte et al., 2016) to assess the model's AMR parsing performance.
Model Configuration Our base setup has 6 layers and 4 attention heads for both the Transformer encoder and decoder, with model size 256 and feedforward size 512. We also compare with a small model with 3 layers in encoder and decoder but identical otherwise. The pointer network is always tied with one target self-attention head of the top decoder layer. We use the cross-attention of all decoder layers for action-source alignment. For graph embedding, we use 2 heads of the bottom 3 layers for the base model and bottom 2 layers for the small model. We use contextualized embeddings extracted from the pre-trained RoBERTa (Liu et al., 2019) large model for the source sentence, with average of all layer states and BPE tokens mapped to words by averaging as in (Lee et al., 2020). The pre-trained embeddings are fixed. For
3 Although there are annotation revisions from AMR 1.0 to AMR 2.0. Link to data: https://amr.isi.edu/download.html.
There are small variations of SMATCH computation due to the stochastic nature of graph matching algorithm.
Transition system	Avg. #actions	Oracle SMATCH
Naseem et al. (2019)*	73.6	93.3
Astudillo et al. (2020)*	76.2	98.0
Ours	41.6	98.9
target actions we train our own embeddings along with the model.
Implementation Details We use the Adam optimizer with  $\beta_{1}$  of 0.9 and  $\beta_{2}$  of 0.98 for training. Each data batch has 3584 maximum number of tokens, and the learning rate schedule is the same as Vaswani et al. (2017), where we use the maximum learning rate of  $5\mathrm{e} - 4$  with 4000 warm-up steps. We use a dropout rate of 0.3 and label smoothing rate of 0.01. We train all the models for a maximum number of 120 epochs, and average the best 5 epoch checkpoints among the last 40 checkpoints based on the SMATCH scores on the development data with greedy decoding. We use a default beam size of 10 for decoding. We implement our model with the FAIRSEQ toolkit (Ott et al., 2019). All models are trained and tested on a single Nvidia Titan RTX GPU. Training takes about 10 hours on AMR 2.0 and 3.5 hours on AMR 1.0.
Oracle Actions Table 1 compares the oracle data SMATCH and average action sequence length on the AMR 2.0 training set among recent transition systems. Our approach yields much shorter action sequences due to the target-side pointing mechanism. It has also the best coverage on training AMR graphs, due to the flexibility of our transitions that can capture the majority of graph components. We chose not to tackle a number of small corner cases, such as disconnected subgraphs for a token, that account for the missing oracle performance.
Parsing Performance We compare our action-pointer transition/Transformer (APT) model with existing approaches in Table  $2^{6}$ . We indicate the use of pre-trained BERT or RoBERTa embeddings