SlowGuess/NAACL_2022_OCR huggingface dataset:
Andrew Drozdov† Jiawei Zhou‡ Radu Florian* Andrew McCallum† Tahira Naseem* Yoon Kim* Ramon Fernandez Astudillo*
†UMass Amherst CICS
$^{\ddagger}$ Harvard University
$^{\dagger}$ IBM Research
†{adrozdov, mccallum}@cs.umass.edu ‡jzhou02@g.harvard.edu
$\diamond$  yoonkim@mit.edu  $\clubsuit$  {raduf,tnaseem}@us.ibm.com  $\clubsuit$  ramon. astudillo@ibm.com
Transition-based parsers for Abstract Meaning Representation (AMR) rely on node-to-word alignments. These alignments are learned separately from parser training and require a complex pipeline of rule-based components, pre-processing, and post-processing to satisfy domain-specific constraints. Parsing also train on a point-estimate of the alignment pipeline, neglecting the uncertainty due to the inherent ambiguity of alignment. In this work we explore two avenues for overcoming these limitations. First, we propose a neural aligner for AMR that learns node-to-word alignments without relying on complex pipelines. We subsequently explore a tighter integration of aligner and parser training by considering a distribution over oracle action sequences arising from aligner uncertainty. Empirical results show this approach leads to more accurate alignments and generalization better from the AMR2.0 to AMR3.0 corpora. We attain a new state-of-the-art for gold-only trained models, matching silver-trained performance without the need for beam search on AMR3.0.
Abstract Meaning Representation (AMR) was introduced as an effort to unify various semantic tasks (entity-typing, co-reference, relation extraction, and so on; Banarescu et al., 2013). Of existing approaches for AMR parsing, transition-based parsing is particularly notable because it is high performing but still relies on node-to-word alignments as a core pre-processing step (Ballesteros and Al-Onaizan, 2017; Liu et al., 2018; Naseem et al., 2019; Fernandez Astudillo et al., 2020; Zhou et al., 2021a,b, inter alia). These alignments are not in the training data and must be learned separately via a complex pipeline of rule-based systems, pre-processing (e.g., lemmatization), and post-processing to satisfy domain-specific constraints. Such pipelines can fail to generalize well,
propagating errors into training that reduce AMR performance in new domains (e.g., AMR3.0). This work studies how we can probabilistically induce and use alignments for transition-based AMR parsing in a domain-agnostic manner, ultimately replacing the existing heuristics-based pipeline.
To induce alignments, we propose a neural aligner which uses hard attention within a sequence-to-sequence model to learn latent alignments (Wu et al., 2018; Deng et al., 2018; Shankar et al., 2018). While straightforward, this neural parameterization makes it possible to easily incorporate pretrained features such as character-aware word embeddings from ELMo (Peters et al., 2018) and also relax some of the strong independence assumptions in classic count-based aligners such as IBM Model 1 (Brown et al., 1993). We find that the neural aligner meaningfully improves upon various baselines, including the existing domain-specific approach.
To use the neural aligner's posterior distribution over alignments, we explore several methods. Our first approach simply uses the MAP alignment from the neural aligner to obtain a single oracle action sequence, which is used to train the AMR parser. However, this one-best alignment fails to take into account the inherent uncertainty associated with posterior alignments. Our second approach addresses this via posterior regularization to push the AMR parser's (intractable) posterior alignment distribution to be close to the neural aligner's (tractable) posterior distribution. We show that optimizing this posterior regularized objective results in a simple training scheme wherein the AMR parser is trained on oracle actions derived samples from the neural aligner's posterior distribution. Our final approach uses the neural aligner not as a regularizer but as an importance sampling distribution, which can be used to better approximate samples from the AMR parser's posterior alignment distribution, and thus better approximate the otherwise intractable log marginal likelihood.
Human Language Technologies, pages 1086 - 1098
In summary, we make the following empirical and methodological contributions:
- We show that our approach can simplify the existing pipeline and learn state-of-the-art AMR parsers that perform well on both AMR2.0 and AMR3.0. Unlike other approaches, AMR parsers learned this way do not require beam search and hence are more efficient at test time.
- We explore different methods for inducing and using alignments. We show that a neural parameterization of the aligner is crucial for learning good alignments, and that using the neural aligner to regularize the AMR parser's posterior is an effective strategy for transferring strong inductive biases from the (constrained) aligner to the (overly flexible) parser.
2 Background: Transition-based Parsing
2.1 General Approach for AMR
A standard and effective way to train AMR parsers is with sequence-to-sequence learning where the input sequence is the sentence  $w$  and the output sequence is the graph  $g$  decomposed into an action sequence  $a$  via an oracle. The combination of words and actions is provided to a parameter-less state machine  $M$  that produces the graph  $g := M(w, a)$ . The state machine can perform the oracle inverse operation  $O$  when also provided alignments  $l$ , mapping a graph to a deterministic sequence of oracle actions  $a := O(l, w, g)$ . During training the model learns to map  $w \rightarrow a$  (these pairs are given by the oracle  $O$ ), and  $M$  is used to construct graphs  $(a \rightarrow g)$  for evaluation.
2.2 StructBART
In this paper we use the oracle and state machine from StructBART (Zhou et al., 2021b), which is a simplified version of Zhou et al. (2021a). They rely on rules that determine which actions are valid (e.g. the first action can not be to generate an edge). The actions are the output space the parser predicts and when read from left-to-right are used to construct an AMR graph. In this case, the actions incorporate alignments.
Rules The following rules define the valid actions at each time step:
- Maintain a cursor that reads the sentence left-to-right, only progressing for SHIFT action.
- At each cursor position, generate any nodes aligned to the cursor's word. (This is where node-word alignments are needed).
- Immediately after generating a node, also generate any valid incoming or outgoing arcs.
Actions At each time step perform one of the following actions to update the state or graph:
- SHIFT: Increment the cursor position.
-  $\operatorname{NODE}\left( {y}_{v}\right)$  : Generate node with label  ${y}_{v}$  .
- COPY: Generate node by copying word under the cursor as the label.
-  $\mathrm{LA}(y_e, n)$ ,  $\mathrm{RA}(y_e, n)$ : Generate an edge with label  $y_e$  from the most recently generated node to the previously generated node  $n$ . LA and RA (short for left-arc and right-arc) indicate the edge direction as outgoing/incoming. We use  $y_e$  to differentiate edge labels from node labels  $y_v$ .
- END: A special action indicating that the full graph has been generated.
Learning For parsing, StructBART fine-tunes BART (Lewis et al., 2020) with the following modifications: a) it converts one attention head from the BART decoder into a pointer network for predicting  $n$  in the LA/RA actions, b) logits for actions are masked to guarantee graph well-formedness, and c) alignment is used to mask two cross-attention heads of the BART decoder, $^{2}$  thereby integrating structural alignment directly in the model.
StructBART is trained to optimize the maximum likelihood of action sequences given sentence and alignment. More formally, for a single example
$$
\left(w, g, l\right) \sim \mathcal {D}, a := O (l, w, g),
$$
the log-likelihood of the actions (and hence the graph) is given by,
$$
\log p (a \mid w; \theta) = \sum_ {t = 1} ^ {T} \log p \left(a _ {t} \mid a _ {<   t}, w; \theta\right)
$$
1While current state-of-the-art oracles do make use of alignments, some oracles do not make explicit use of alignments to derive action sequences, for example by generating the nodes in the AMR graph in depth-first order from the root and breaking ties according to the order nodes appear in the data file.
2 Alignment is represented in the action sequence through the SHIFT action and cursor position.
for a model with parameters  $\theta$ . Probabilities of actions that create arcs are decomposed into independent label and pointer distributions
$$
\begin{array}{l} p \left(a _ {t} \mid a _ {<   t}, w; \theta\right) = \\ p \left(y _ {e} \mid a _ {<   t}, w; \theta\right) p (n \mid a _ {<   t}, w; \theta) \\ \end{array}
$$
where  $p(y_{e} \mid a_{<t}, w; \theta)$  is computed with the normal output vocabulary distribution of BART and  $p(n \mid a_{<t}, w; \theta)$  with one attention head of the decoder. See Zhou et al. (2021b) for more details.
Alignment (SB-Align) For training, StructBART depends on node-to-word AMR alignments  $l$  to specify the oracle actions. In previous work, the alignments have been computed by a pipeline of components that we call SB-Align.
We introduce our neural approach in the next section, but first we cover the main steps in SB-Align: (1) produce initial alignments using Symmetrized Expectation Maximization (Pourdamghani et al., 2014); (2) attempt to align additional nodes by inheriting child node alignments; (3) continue to refine alignments using JAMR (Flanigan et al., 2014), which involves constraint optimization using a set of linguistically motivated rules.
The StructBART action space requires that all nodes are aligned, yet after running SB-Align some nodes are not. This is solved by first "force aligning" unaligned nodes to unaligned tokens, then propagating alignments from child-to-parent nodes and vice versa until  $100\%$  of nodes are aligned to text spans. Finally, node-to-span alignments are converted into node-to-token alignments for model training (e.g. by deterministically aligning to the first node of an entity). Specifics are described in StructBART and preceding work (Zhou et al., 2021b,a; Fernandez Astudillo et al., 2020).
3 Inducing Alignments
Here, we describe our neural alignment model, which is essentially a variant of sequence-to-sequence models with hard attention (Yu et al., 2016a; Wu et al., 2018; Shankar et al., 2018; Deng et al., 2018). In contrast to SB-Align, our approach requires minimal pre-processing and does not have dependencies on many components or domain-specific rules.
The alignment model is trained separately from the AMR parser and optimizes the conditional likelihood of nodes in the linearized graph given the
sentence.3 The AMR graph is linearized by first converting the graph to a tree,4 and then linearizing the tree via a depth-first search, as in Figure 1. Letting  $v = v_{1},\ldots ,v_{S}$  be the nodes in the linearized AMR graph, the log-likelihood is given by
$$
\log q (v \mid w; \phi) = \sum_ {s = 1} ^ {S} \log q (v _ {s} \mid v _ {<   s}, w; \phi),
$$
where we abuse notation and use  $v_{<s}$  to indicate all the tokens (include brackets and edges) before  $v_{s}$ . That is, we incur losses only on the nodes  $v_{s}$  but still represent the entire history  $v_{<s}$  for the prediction (see Figure 1, left). The probability of each node is given by marginalizing over latent alignments  $l_{s}$ ,
$$
\begin{array}{l} q \left(v _ {s} \mid v _ {<   s}, w; \phi\right) = \sum_ {i = 1} ^ {| w |} q \left(l _ {s} = i \mid v _ {<   s}, w, \phi\right) \times \\ q (v _ {s} \mid l _ {s} = i, v _ {<   s}, w; \phi), \\ \end{array}
$$
where  $l_{s} = i$  indicates that node  $v_{s}$  is aligned to word  $w_{i}$ .
For parameterization, the sentence  $w$  is encoded by a bi-directional LSTM. Each word is represented using a word embedding derived from a pretrained character-encoder from ELMo (Peters et al., 2018), which is frozen during training. On the decoder side, the linearized AMR tree history is represented by a uni-directional LSTM. The decoder shares word embeddings with the text encoder. The prior alignment probability  $q(l_{s} = i|v_{< s},w;\phi)$  is given by bilinear attention (Luong et al., 2015),
$$
\begin{array}{l} q \left(l _ {s} = i \mid v _ {<   s}, w; \phi\right) = \frac {\exp \left(\alpha_ {s , i}\right)}{\sum_ {j = 1} ^ {| w |} \exp \left(\alpha_ {s , j}\right)}, \\ \alpha_ {s, i} = h _ {s} ^ {(v) \top} W h _ {i} ^ {(w)}, \\ \end{array}
$$
where  $W$  is a learned matrix,  $h_i^{(w)}$  is a concatenation of forward and backward LSTM vectors for the  $i$ -th word in the text encoder, and  $h_t^{(v)}$  is the vector immediately before the  $s$ -th node in the graph decoder. The likelihood  $q(v_s \mid l_s = i, v_{<s}, w; \phi)$  is formulated as a softmax layer with the relevant vectors concatenated as input,
$$
\begin{array}{l} q \left(v _ {s} = y \mid l _ {s} = i, v _ {<   s}, w; \phi\right) = \\ \operatorname {s o f t m a x} (U [ h _ {s} ^ {(y)}; h _ {i} ^ {(w)} ] + b) [ y ], \\ \end{array}
$$
3This is because our oracle  $O$  only needs node-word alignments to derive the oracle action sequence.
4To convert the graph to a tree we only include the first incoming edge for each node.
Input sentence
The harder they come, the harder they fall.
AMR Graph
Linearized AMR Tree
Figure 1: (Left) An example of a sentence, its AMR graph, and the corresponding linearized AMR tree. The aligner decoder only incurs a loss for AMR nodes (tokens for nodes are in bold), although it represents the full history. (Right) A visualization of our alignment posterior (blue) and point estimate from the baseline (white box). The uncertainty corresponding to alignment ambiguity is helpful during sampling-based training.
where the softmax is over the node vocabulary and is indexed by the label  $y$  belonging to the node  $v_{s}$ .
Once trained, we can tractably obtain the posterior distribution over each alignment  $l_{s}$
$$
q \left(l _ {s} = i \mid w, v; \phi\right) =
$$
$$
\frac {q (l _ {s} = i \mid v _ {<   s} , w , ; \phi) q (v _ {s} \mid l _ {s} = i , v _ {<   s} , w ; \phi)}{q (v _ {s} \mid v _ {<   s} , w ; \phi)},
$$
and the full posterior distribution over all alignments  $l = l_{1},\ldots ,l_{S}$  is given by
$$
q (l \mid w, g; \phi) = \prod_ {s = 1} ^ {S} q (l _ {s} \mid w, v; \phi).
$$
Discussion Compared to the classic count-based alignment models, the neural parameterization makes it easy to utilize pretrained embeddings and also condition on the alignment and emission distribution on richer context. For example, our emission distribution  $q(v_{s} \mid l_{s}, v_{<s}, w; \phi)$  can condition on the full target history  $v_{<s}$  and the source context  $w$ , unlike count-based models which typically condition on just the aligned word  $w_{l_{s}}$ . In our ablation experiments described in (§6) we find that the flexible modeling capabilities enabled by the use of neural networks are crucial for obtaining good alignment performance.
4 Using Alignments
The neural aligner described above induces a posterior distribution over alignments,  $q(l \mid w, g; \phi)$ .
We explore several approaches using this alignment distribution.
MAP Alignment To use this alignment model in the most straightforward way, we decode the MAP alignment  $\hat{l} = \arg \max_{l} q(l \mid w, g; \phi)$  and train from the actions  $\hat{a} = O(\hat{l}, w, g)$ .
Posterior Regularization (PR) The action sequences derived from MAP alignments do not take into account the uncertainty associated with posterior alignments, which may not be ideal (Figure 1, right). We propose to take this uncertainty into account and regularize the AMR parser's posterior to be close to the neural aligner's posterior at the distributional level.
First, we note that the action oracle  $O(l, w, g)$  is bijective as a function of  $l$  (i.e., keeping  $w$  and  $g$  fixed), so the transition-based parser  $p(a \mid w; \theta)$  induces a joint distribution over alignments and graphs,
$$
p (l, g \mid w; \theta) \stackrel {\text {d e f}} {=} p (a = O (l, w, g) \mid w; \theta).
$$
This joint distribution further induces a marginal distribution over graphs,
$$
p (g \mid w; \theta) = \sum_ {l} p (l, g \mid w; \theta),
$$
as well as a posterior distribution over alignments,
$$
p (l \mid w, g; \theta) = \frac {p (l , g \mid w ; \theta)}{p (g \mid w ; \theta)}.
$$
A simple way to use the neural aligner's distribution, then, is via a posterior regularized likelihood (Ganchev et al., 2010),<sup>5</sup>
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {P R}} (\theta) = \log p (g \mid w; \theta) - \\ \operatorname {K L} [ q (l \mid w, g; \phi) \| p (l \mid w, g; \theta) ]. \\ \end{array}
$$
That is, we want to learn a parser that gives high likelihood to the gold graph  $g$  given the sentence  $w$  but at the same time has a posterior alignment distribution that is close to the neural aligner's posterior. Rearranging some terms, we then have
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {P R}} (\theta) = \mathbb {E} _ {q (l \mid w, g; \phi)} [ \log p (l, g \mid w; \theta) ] + \\ \mathbb {H} [ q (l \mid w, g; \phi) ], \\ \end{array}
$$
and since the second term is a constant with respect to  $\theta$ , the gradient with respect to  $\theta$  is given by,
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {P R}} (\theta) = \mathbb {E} _ {q (l \mid w, g; \phi)} [ \nabla_ {\theta} \log p (l, g \mid w; \theta) ].
$$
Gradient-based optimization with Monte Carlo gradient estimators therefore results in an intuitive scheme where (1) we sample  $K$  alignments  $l^{(1)},\ldots ,l^{(K)}$  from  $q(l|w,g;\phi)$  (2) obtain the corresponding action sequences  $a^{(1)},\dots,a^{(K)}$  from the oracle, and (3) optimize the loss with the Monte Carlo gradient estimator  $\frac{1}{K}\sum_{k = 1}^{K}\nabla_{\theta}\log p(a^{(k)}|w,g;\theta)$ .
It is clear that the above generalizes the MAP alignment case. In particular, setting  $q(l \mid w, g) = \mathbb{1}\{l = \hat{l}\}$  where  $\hat{l}$  is the MAP alignment (or an alignment derived from the existing pipeline) recovers the existing baseline.
Importance Sampling (IS) The posterior regularized likelihood clearly lower bounds the log marginal likelihood  $\mathcal{L}(\theta) = \log p(g\mid w;\theta),^6$  and implicitly assumes that training against the lower bound results in a model that generalizes better than a model trained against the true log marginal likelihood. In this section we instead take a variational perspective and use the neural aligner not as a regularizer, but as a surrogate posterior distribution whose samples can be reweighted to reduce the gap between the  $\mathcal{L}(\theta)$  and  $\mathcal{L}_{\mathrm{PR}}(\theta)$ .
We first take the product of the neural aligner's posterior to create a joint posterior distribution,
$$
q (l ^ {(1)}, \dots l ^ {(K)}; \phi) \stackrel {\mathrm {d e f}} {=} \prod_ {k = 1} ^ {K} q (l ^ {(k)} | w, g; \phi),
$$
where  $K$  is the number of importance samples. Then, Burda et al. (2016) show that the following objective,
$$
\mathbb {E} _ {q (l ^ {(1)}, \dots l ^ {(K)}; \phi)} \left[ \log \frac {1}{K} \sum_ {k = 1} ^ {K} \frac {p (l ^ {(k)} , g \mid w ; \theta)}{q (l ^ {(k)}) \mid w , g ; \phi)} \right],
$$
motonotically converges to the log marginal likelihood  $\log p(g\mid w;\theta)$  as  $K\to \infty$  . A single-sampleMonte Carlo gradient estimator for the above is given by,
$$
\sum_ {k = 1} ^ {K} w ^ {(k)} \nabla_ {\theta} \log p (a ^ {(k)} | w, g; \theta),
$$
$$
w ^ {(k)} = \frac {p (a ^ {(k)} | w ; \theta) / q (l ^ {(k)} | w , g ; \phi)}{\sum_ {j = 1} ^ {K} p (a ^ {(j)} | w ; \theta) / q (l ^ {(j)} | w , g ; \phi)}
$$
are the normalized importance weights (Mnih and Rezende, 2016). Thus, compared to the gradient estimator in the posterior regularized case which equally weights each sample, this importance-weighted objective approximates the true posterior  $p(l \mid w, g; \theta)$  by first sampling from a fixed distribution  $q(l \mid w, g; \phi)$  and then reweighting it accordingly.
Discussion Despite sharing formulation with the variational autoencoder (Kingma and Welling, 2013) and the importance weighted autoencoder (Burda et al., 2016), the approach proposed here differs in fundamental aspects. In contrast to the variational approaches we fix  $q$  to the pretrained aligner posterior and do not optimize it further. Moreover, the lower bound  $\mathcal{L}_{\mathrm{PR}}(\theta)$  represents an inductive bias informed by a pretrained aligner, which can be more suited for early stages of training than even a tangent evidence lower bound (zero gap). This is because, for a tangent lower bound,  $q$  in the Monte Carlo gradient estimate is equal to the true posterior over alignments for current model parameters. Since these parameters are poorly trained, it
5Note that unlike in the original formulation, here we do not optimize over  $q$  and instead keep it fixed. This is equivalent to the original formulation if we define the posterior regularization set  $\mathcal{Q}$  to just consist of the distribution learned by the neural aligner, i.e.,  $\mathcal{Q} = \{q\}$ .
6The log marginal likelihood is intractable to estimate directly due to the lack of any independence assumptions in the AMR parser, since in the AMR parser the alignment variable  $l_{s}$  fully depends on  $l_{< s}$ .
Note that a single sample from  $q(l^{(1)}, \ldots, l^{(K)}; \phi)$  is obtained by sampling from the neural aligner  $K$  times.
is easy for the aligner to provide a better alignment distribution for learning.
Posterior regularization seeks to transfer the neural aligner's strong inductive biases to the AMR parser, which has weaker inductive biases and thus may be potentially too flexible of a model. On the other hand, importance sampling "trusts" the AMR parser's inductive bias more, and uses the neural aligner as a surrogate distribution that is adapted to more closely approximate the AMR parser's intractable posterior. Thus, if the posterior regularized variant outperforms the importance sampling variant, it suggests that the StructBART is indeed too flexible of a model. While not considered in the present work, it may be interesting to explore a hybrid approach which first trains with posterior regularization and then switches to importance sampling.
All code to run our experiments is available online with Apache License, 2.0.
5.1 Data, Preprocessing, and Evaluation
Data We evaluate our models on two datasets for AMR parsing in English. AMR2.0 contains  $\sim 39\mathrm{k}$  sentences from multiple genres (LDC2017T10). AMR3.0 is a superset of AMR2.0 sentences with approx. 20k new sentences (LDC2020T02), improved annotations with new frames, annotation corrections, and expanded annotation guidelines. Using AMR3.0 for evaluation allows us to measure how well our alignment procedure generalizes to new datasets — AMR3.0 includes new sentences but also new genres such as text from LORELEI, $^{9}$  Aesop fables, and Wikipedia.
The primary evaluation of the aligner is extrinsically through AMR parsing, and we additionally evaluate alignments directly against ground truth annotations provided in Blodgett and Schneider (2021)—specifically, we look at the 130 sentences from the AMR2.0 train data (the ones most well suited for SB-Align), which we call the gold test set. Alignment annotations are not used during aligner training and only used for evaluation.
Preprocessing We align text tokens to AMR nodes. As the AMR sentences do not include defacto tokenization, we split strings on space and punctuation using a few regex rules.
For AMR parsing we use the action set described in §2.2. To accommodate the recurrent nature of the aligner, we linearize the AMR graph during aligner training. This conversion requires converting the graph into a tree and removing re-entrant edges, as described in §3.