SlowGuess/NAACL_2022_OCR huggingface dataset:
Evaluation For AMR parsing we use Smatch (Cai and Knight, 2013). For AMR alignment our goal is mainly to compare our new aligner with strong alignment baselines: SB-Align and LEAMR, a state-of-the-art alignment model (Blodgett and Schneider, 2021; Blodgett, 2021). However, our aligner predicts node-to-word alignments, SB-Align predicts node-to-span alignments, and the ground truth alignments are subgraph-to-span. To address this mismatch in granularity, we measure alignment performance using a permissive version of F1 after decomposing subgraph-to-span alignments into node-to-span alignments—a prediction is correct if it overlaps with the gold span. This permissiveness gives advantages the LEAMR and SB-Align baselines (which predict span-based alignments) as there is no precision-related penalty for predicting large spans.
Aligner We use a bi-directional LSTM for the Text Encoder and uni-directional LSTM for the AMR Decoder. The input token embeddings are derived from a pretrained character encoder (Peters et al., 2018) and frozen throughout training; these token embeddings are tied with the output softmax, allowing for alignment to tokens not seen during training. The alignment model is otherwise parameterized as described in §3. We train for 200 epochs. Training is unsupervised, so we simply use the final checkpoint. $^{10}$  Additional training details for the aligner are in the Appendix.
AMR Parser We use the StructBART model from Zhou et al. (2021b) and the same hyperparameters: fine-tuning for 100 epochs (AMR2.0) or 120 (AMR3.0), and using Smatch on the validation
In our early experiments, we used SB-aligner's predictions as validation to find a reasonable range of hyperparameters. Performance does not substantially deteriorate after 50 epochs, so this was not necessary or useful for early stopping. Early stopping based on perplexity performed similarly.
$^{8}$ https://github.com/IBM/transition-amr-parser
9The LORELEI genre (low resource languages for emergent incidents) contains sentences from news articles, blogs, and forums (Strassel and Tracey, 2016). These sentences were specifically used in Bevilacqua et al. (2021) to measure parser out-of-domain generalization.
Method	Beam Size	Silver Data	AMR2.0	AMR3.0
APT (Zhou et al., 2021a)P	10	70K	83.4	-
TAMR (Xia et al., 2021)G	8	1.8M	84.2	-
SPRING (Bevilacqua et al., 2021)	5	200K	84.3	83.0
StructBART-S (Zhou et al., 2021b)	10	90K	-	82.7 ±0.1
StructBART-J (Zhou et al., 2021b)	10	90K	84.7 ±0.1	82.6 ±0.1
StructBART-J+MBSE (Lee et al., 2021)	10	219K	85.7 ±0.0	84.1 ±0.1
BARTAMR (Bai et al., 2022)	5	200K	85.4	84.2
APT (Zhou et al., 2021a)P	10	-	82.8	81.2
SPRING (Bevilacqua et al., 2021)	5	-	83.8	83.0
SPRING (Bevilacqua et al., 2021)G	5	-	84.5	80.2
StructBART-J (Zhou et al., 2021b)	10	-	84.2 ±0.1	82.0 ±0.0
StructBART-S (Zhou et al., 2021b)	10	-	84.0 ±0.1	82.3 ±0.0
StructBART-S (reproduced)	1	-	83.9 ±0.0	81.9 ±0.2
+neural-aligner (MAP)	1	-	84.0 ±0.1	82.5 ±0.1
+neural-aligner (MAP)	10	-	84.1 ±0.0	82.7 ±0.1
+neural-aligner (PR, w/ 5 samples)	1	-	84.3 ±0.0	83.1 ±0.1
+neural-aligner (IS, w/ 5 samples)	1	-	84.2 ±0.1	82.8
Table 1: Results on parsing for AMR2.0 and 3.0 test sets. We report numbers when using single alignments (MAP), posterior regularization (PR), and importance sampling (IS). Also included are number of silver data training sentences used and beam size. PR and IS does not improve with beam search, and hence these numbers are omitted.  $\mathcal{P}$ : Uses partial ensemble for decoder.  $\mathcal{G}$ : Uses graph recategorization.
set for early stopping. We did not tune the hyperparameters of the AMR parser at all as we wanted to see how well the neural aligner performed as a "plug-in" to an existing system. Additional implementation details for parsing are in the Appendix. For posterior regularization (PR) and importance sampling (IS) variants we use  $K = 5$  samples to obtain the gradient estimators.
The full results are shown in Table 1, where we find that our approach can learn state-of-the-art AMR parses for gold-only training and without requiring beam search. We now interpret the main results in more detail.
Pipeline Generalization SB-Align was developed prior to the AMR3.0 release, and because it incorporates a complex pipeline with domain-specific rules, one could argue it is specialized for prior datasets like AMR2.0. In Table 1 our aligner yields relatively stronger StructBART improvements for AMR3.0 than AMR2.0. This result and the relatively little manual configuration our aligner requires (e.g., no rules, lemmatization, etc.) suggest our alignment approach generalizes better to different training corpora and that prior performance of StructBART on AMR3.0 may have been
affected by a lack of generalization.
Graph re-categorization (Zhang et al., 2019a,b) is a commonly used technique in AMR parsing where groups of nodes are collapsed during training and test time, but expanded during evaluation. Results from (Bevilacqua et al., 2021) show re-categorization may be harmful, but our results suggest a different perspective—re-categorization is partially a function of alignment-like heuristics and the lower re-categorization results of SPRING in AMR3.0 reinforce our findings that alignments based on heuristic are difficult to generalize.
Alignment Uncertainty vs. Data Augmentation We improve parsing performance by sampling 5 alignments per sentence in batch (see Table 1). One can argue that our approach simply exposes the model to more data, but we found that training on one-best alignments for longer did not improve results. When looking at our sampling results compared with previous versions of StructBART trained on silver data, we see that our approach even outperforms the benefit of the simpler versions of data augmentation, such as simple self-learning. This suggests that there is possible further improvement by combining both techniques, which we leave for future work.
AMR2.0
LEAMR (Blodgett and Schneider, 2021)	97.4
SB-Align (Zhou et al., 2021b)	89.2
Neural Aligner (ours)	96.5
IBM Model 1	77.2
Neural Aligner w/o pretrained emb.	79.8
Neural Aligner w/ pretrained emb.	96.5
Table 2: (Top) Alignments results using ground truth data from Blodgett and Schneider (2021), where we use the 130 sentences from the gold test set that are from the AMR2.0 train data. (Bottom) Alignment ablation results against the same test set using different alignment model variants. Note that as the ground truth alignment is at the span level, we report a permissive variant of F1 where a prediction is considered correct if it partially overlaps with the ground truth span. This advantages the LEAMR and SB-Align baselines as these can align to spans, whereas our aligners only align to words.
Posterior regularization vs. Importance sampling Training with posterior regularization or importance sampling uses the same number of samples, but in different ways. In posterior regularization, the samples are used to better approximate the posterior regularized objective, which in turn regularizes the AMR parser's posterior more effectively by reducing the gradient estimator variance. In importance sampling, the samples are used to better approximate the AMR parser's intractable log marginal likelihood. We find that importance sampling fails to improve upon posterior regularization for both AMR2.0 and AMR3.0, which indicates that strong inductive biases associated the constrained aligner is a useful training signal for the flexible AMR parser.
Comparing against alignment baselines Our neural alignment method is preferred over SBAlign for two primary reasons: it is relatively easy to use (makes use of word embeddings, depends on less preprocessing, does not require domain-specific rules, etc.) and empirically improves performance (see Table 1). Nonetheless, we conduct an intrinsic evaluation to assess the quality of the predicted alignments—it is desirable that our aligner actually provides accurate alignments.
To verify that improved parsing is due to better alignment, we compare against two strong alignment baselines (LEAMR and SB-Align) on an evaluation set of gold manually annotated alignments. In general, there are only a few hundred
such annotations available, yet we aim to use these alignments on 10s or 100s of thousands sentences for AMR parsing. For this reason all the aligners are trained unsupervised with respect to alignment. The results in Table 2 (top) show our aligner is substantially better than SB-Align and nearly on-par with LEAMR, the current state of the art.
Aligner Parameterization We train the classic count-based IBM Model 1 (Brown et al., 1993) using expectation maximization. We next train our neural aligner without pretrained character-aware embeddings. Our neural aligner is different from the classic IBM model in that (1) it learns the prior alignment distribution, (2) the emission model conditions on the entire sentence  $w$  and the target history  $v_{<s}$ . Finally, adding pretrained embeddings to this model recovers our full model. The results in Table 2 (bottom) indicate that both flexibility and token representation are required to outperform IBM Model 1. Training with word vectors learned from scratch only provides a small benefit, and the best performance is from using pretrained character embeddings, which yields nearly 20 point improvement in our permissive F1 metric.
Oracles for parsing Dynamic oracles in syntactic parsing (Goldberg and Nivre, 2012; Ballesteros et al., 2016, inter-alia) enable oracles to recover from imperfect sequences. Oracles with random exploration in transition-based AMR parsers have been previously explored using imitation-learning (Goodman et al., 2016), reinforcement learning (Naseem et al., 2019) and oracle mining (Lee et al., 2020). In addition to this, Liu et al. (2018) produces multiple alignments via their rule-based system and selects the best based on parser performance. Compared to our proposed posterior regularization training, dynamic oracle works exploit specific linguistic properties of syntactic trees not directly transferable to AMR graphs. Prior work on random exploration and AMR selects oracles based on the Smatch. Our work uses the action space and oracle from StructBART (Zhou et al., 2021b,a), which requires every node to be aligned to a word so that the AMR graph is fully recoverable. We expose the parser to uncertainty by sampling alignment, which does not require computing the Smatch metric. Rather, the aligner is trained separately from the parser using pairs of sentences and their respective AMR graphs.
Neural Alignments The neural aligner we use is closely related to sequence-to-sequence models with hard (i.e., latent variable) attention (Xuet al., 2015; Ba et al., 2015; Wu et al., 2018; Shankar et al., 2018; Deng et al., 2018; Shankar and Sarawagi, 2019, inter alia) and other works on marginalizing over monotonic alignments (Yu et al., 2016b; Raffel et al., 2017; Wu and Cotterell, 2019). In these works, the main goal is to obtain better sequence-to-sequence models, and the actual posterior distribution over alignments is a byproduct (rather than the goal) of learning. However, there are also works that utilize contemporary neural parameterizations and explicitly target alignments as the primary goal (Zenkel et al., 2020; Ho and Yvon, 2020; Chen et al., 2021). Prior work on integrating pretrained aligners with sequence-to-sequence models has generally used the alignments to supervise the intermediate soft attention layers (Liu et al., 2016; Cohn et al., 2016; Yin et al., 2021), in contrast to the present work which formally treats alignments as latent variables.
Alignments in AMR Parsing There is extensive work aligning AMR nodes to sentences. The StructBART model (Zhou et al., 2021b) considered here, previously made use of well established rule-based and statistical AMR aligners (Flanigan et al., 2014; Pourdamghani et al., 2014) trained with expectation maximization and additional heuristics (Naseem et al., 2019; Fernandez Astudillo et al., 2020) that we call SB-Align. Our aligner compares favorably against SB-Align in alignment metrics and downstream parsing performance. The predicted alignments we use come close to Blodgett and Schneider (2021) measured on gold alignment annotations, despite our method leveraging less domain-specific components. Lyu and Titov (2018) and Lyu et al. (2021) incorporate differentiable relaxations of alignments for AMR parsing.
Posterior Regularization for Parsing and Generation Li et al. (2019) apply variational inference and posterior regularization for unsupervised dependency parsing using their transition-based system. Their approach predates large pretrained language models for which the use of structure may play a different role. Li and Rush (2020) use posterior regularization to incorporate weakly supervised alignment constraints for data-to-text generation, also without pretrained neural representations in mind.
In this work we propose a general-purpose neural AMR aligner, which does not use a complex alignment pipeline and generalizes well across domains. The neural parameterization allows the aligner to fully condition on the source and target context and easily incorporates pretrained embeddings, leading to improved performance. Simply using our aligner to produce training data for a state-of-the-art transition-based parser leads to improved results.
We additionally propose a learning procedure using posterior regularization and importance sampling that involves sampling different action sequences during training. These incorporate alignment uncertainty and further improve parser performance. Our results on gold-only AMR training (i.e., without silver data) show that parsers learned this way match the performance of the prior state-of-the-art parsers without requiring beam search at test time.
We are grateful to our colleagues for their help and advice, and to the anonymous reviewers at ACL Rolling Review for their feedback on drafts of this work. We also thank Nathan Schneider and Austin Blodgett for sharing the gold alignment data. AD and AM were supported in part by the Center for Intelligent Information Retrieval and the Center for Data Science; in part by the IBM Research AI through the AI Horizons Network; in part by the Chan Zuckerberg Initiative under the project Scientific Knowledge Base Construction; in part by the National Science Foundation (NSF) grant numbers IIS-1922090, IIS-1955567, and IIS-1763618; in part by the Defense Advanced Research Projects Agency (DARPA) via Contract No. FA8750-17-C0106 under Subaward No. 89341790 from the University of Southern California; and in part by the Office of Naval Research (ONR) via Contract No. N660011924032 under Subaward No. 123875727 from the University of Southern California. YK was supported in part by a MIT-IBM Watson AI grant. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.
We do not foresee specific risks associated with our exploration of alignment for AMR parsing. That being said, there are nuances to our results that future researchers may take into considerations. For instance, our experiments are only for English data in the genres covered by AMR2.0 and AMR3.0. It is not clear how our results translate to other domains (e.g. biomedical text) or other languages. Nonetheless, we are hopeful that are methods would transfer favorably because they are intentionally designed to be easy to use and general purpose.
Jimmy Ba, Ruslan R Salakhutdinov, Roger B Grosse, and Brendan J Frey. 2015. Learning Wake-Sleep Recurrent Attention Models. In Proceedings of NeurIPS.
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Miguel Ballesteros and Yaser Al-Onaizan. 2017. AMR parsing using stack-LSTMs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1269-1275, Copenhagen, Denmark. Association for Computational Linguistics.
Miguel Ballesteros, Yoav Goldberg, Chris Dyer, and Noah A. Smith. 2016. Training with exploration improves a greedy stack LSTM parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2005-2010, Austin, Texas. Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitht, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Symmetric amr semantic parsing and generation without a complex pipeline. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12564-12573.
Austin Blodgett. 2021. Linguistic Interpretability and Composition of Abstract Meaning Representations. Ph.D. thesis, Georgetown University.
Austin Blodgett and Nathan Schneider. 2021. Probabilistic, structure-aware algorithms for improved
variety, accuracy, and coverage of AMR alignments. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3310-3321, Online. Association for Computational Linguistics.
Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263-311.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2016. Importance weighted autoencoders. In ICLR.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
Chi Chen, Maosong Sun, and Yang Liu. 2021. Mask align: Self-supervised neural word alignment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4781-4791, Online. Association for Computational Linguistics.
Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating structural alignment biases into an attentional neural translation model. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 876-885, San Diego, California. Association for Computational Linguistics.
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander M. Rush. 2018. Latent Alignment and Variational Attention. In Proceedings of NeurIPS.
Ramón Fernandez Astudillo, Miguel Ballesteros, Tahira Naseem, Austin Blodgett, and Radu Florian. 2020. Transition-based parsing with stack-transformers. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1001-1007, Online. Association for Computational Linguistics.
Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the Abstract Meaning Representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-1436, Baltimore, Maryland. Association for Computational Linguistics.
Kuzman Ganchev, João Graça, Jennifer Gillenwater, and Ben Taskar. 2010. Posterior Regularization for
Structured Latent Variable Models. Journal of Machine Learning Research, 11:2001-2049.
Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In Proceedings of COLING 2012, pages 959-976, Mumbai, India. The COLING 2012 Organizing Committee.
James Goodman, Andreas Vlachos, and Jason Naradowsky. 2016. Noise reduction and targeted exploration in imitation learning for Abstract Meaning Representation parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1-11, Berlin, Germany. Association for Computational Linguistics.
Anh Khoa Ngo Ho and François Yvon. 2020. Neural Baselines for Word Alignment. In The 16th International Workshop on Spoken Language Translation.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Young-Suk Lee, Ramon Fernandez Astudillo, Thanh Lam Hoang, Tahira Naseem, Radu Florian, and Salim Roukos. 2021. Maximum bayes smatch ensemble distillation for amr parsing. arXiv preprint arXiv:2112.07790.
Young-Suk Lee, Ramón Fernandez Astudillo, Tahira Naseem, Revanth Gangi Reddy, Radu Florian, and Salim Roukos. 2020. Pushing the limits of AMR parsing with self-learning. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3208-3214, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Bowen Li, Jianpeng Cheng, Yang Liu, and Frank Keller. 2019. Dependency grammar induction with a neural variational transition-based parser. In Proceedings of the AAAI Conference on Artificial Intelligence.
Xiang Lisa Li and Alexander Rush. 2020. Posterior control of blackbox generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2731-2743, Online. Association for Computational Linguistics.
Lemao Liu, Masao Utiyama, Andrew Finch, and Eichiro Sumita. 2016. Neural machine translation with supervised attention. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3093-3102, Osaka, Japan. The COLING 2016 Organizing Committee.