SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 1: Average number of actions and oracle SMATCH on AMR 2.0 training data. The average source length is 18.9. * from author correspondence.
Corpus	Model	SMATCH (%)
AMR 1.0	Pust et al. (2015)	67.1
Flanigan et al. (2016)	66.0
Wang and Xue (2017)G	68.1
Guo and Lu (2018)G	68.3 ±0.4
Zhang et al. (2019a)B,G	70.2 ±0.1
Zhang et al. (2019b)B,G	71.3 ±0.1
Cai and Lam (2020)B	74.0
Cai and Lam (2020)B,G	75.4
Astudillo et al. (2020)*R	76.9 ±0.1
Lee et al. (2020)R (85K silver)	78.2 ±0.1
APT smallR	78.2 / 78.2 ±0.0
APT baseR	78.5 / 78.3 ±0.1
APT smallR p.e.	79.7
APT baseR p.e.	79.8
AMR 2.0	Van Noord and Bos (2017)	71.0
Groschwitz et al. (2018)G	71.0
Lyu and Titov (2018)G	74.4 ±0.2
Cai and Lam (2019)	73.2
Lindemann et al. (2019)	75.3 ±0.1
Naseem et al. (2019)B	75.5
Zhang et al. (2019a)B,G	76.3 ±0.1
Zhang et al. (2019b)B,G	77.0 ±0.1
Cai and Lam (2020)B	78.7
Cai and Lam (2020)B,G	80.2
Astudillo et al. (2020)*R	80.2 ±0.0
Bevilacqua et al. (2021)†	83.8
Xu et al. (2020) (4M silver)	80.2
Lee et al. (2020)R (85K silver)	81.3 ±0.0
Bevilacqua et al. (2021)† (200K silver)	84.3
APT smallR	81.7 / 81.5 ±0.2
APT baseR	81.8 / 81.7 ±0.1
APT smallR p.e.	82.5
APT baseR p.e.	82.8
APT baseR (70K Silver)	82.8 / 82.6 ±0.1
APT baseR (70K Silver) p.e.	83.4
AMR 3.0	Lyu et al. (2020)	75.8
Bevilacqua et al. (2021)†	83.0
APT baseR	80.4 / 80.3 ±0.1
APT baseR p.e.	81.2
Table 2: SMATCH scores on AMR 1.0, 2.0, and 3.0 test sets. APT is our model.  $B$  or  $R$  indicates pretrained BERT or RoBERTa embeddings,  $G$  use of graph re-categorization, * improved results reported in Lee et al. (2020). † denotes concurrent work based on finetuning pre-trained BART large models. We report the best/average score ± standard deviation over 3 seeds. p.e. is partial ensemble decoding with 3 seed models.
(from large models) with  $^B$  or  $^R$ , and graph re-categorization with  $^G$ . Graph re-categorization (Lyu and Titov, 2018; Zhang et al., 2019a; Cai and Lam, 2020; Bevilacqua et al., 2021) removes node senses and groups certain nodes together such as named entities in pre-processing. It reverts these back in post-processing with the help of a name entity recognizer. We report results over 3 runs for each model with different random seeds. Given that we use fixed pre-trained embeddings, it becomes computationally cheap to build a partial ensemble
<sup>5</sup>Available under https://github.com/IBM/ transition-amr-parser.  
<sup>6</sup>We exclude Xu et al. (2020) AMR 1.0 numbers since they report 16833 train sentences, not 10312.
Model	Fixed Extra Features	Trained Param.	SMATCH AMR 2.0
Zhang et al. (2019a)B,G	BERT	66.1M*	76.3
Cai and Lam (2020)B	BERT	27.1M	78.7
Cai and Lam (2020)B,G	BERT	26.1M	80.2
Astudillo et al. (2020)R	RoBERTa	21.7M	80.2
Xu et al. (2020) (4M silver)	-	239.1M	80.2
Bevilacqua et al. (2021)†	-	411.8M	83.8
APT smallR	RoBERTa	17.5M	81.7
APT baseR	RoBERTa	21.4M	81.8
APT smallR p.e.	RoBERTa	52.5M	82.5
APT baseR p.e.	RoBERTa	64.3M	82.8
Table 3: Comparison of model parametrization sizes and SMATCH scores on AMR 2.0 test set. Model sizes of previous works are obtained from their officially released pre-trained models. * is an estimate by removing BERT parameters in the released model, where a BERT base model is trained together which is different from the paper description. † denotes concurrent work based on fine-tuning pre-trained BART large models.
that uses the average probability of 3 models from different seeds which we denote as p.e.
With the exception of the recent BART-based model Bevilacqua et al. (2021), we outperform all previously published approaches, both with our small and base models. Our best single-model parsing scores are 81.8 on AMR 2.0 and 78.5 on AMR 1.0, which improves 1.6 points over the previous best model trained only with gold data. Our small model only trails the base model by a small margin and we achieve high performance on small AMR 1.0 dataset, indicating that our approach benefits from having good inductive bias towards the problem so that the learning is efficient. More remarkably, we even surpass the scores reported in Lee et al. (2020) combining various self-learning techniques and utilizing 85K extra sentences for self-annotation (silver data). For the most recent AMR 3.0 dataset, we report our results for future reference.
Additionally, the partial ensemble decoding proves to be simple and effective in boosting the model performance, which consistently brings more than 1 point gain for AMR 1.0 and 2.0. It should be noted that the ensemble decoding is only  $20\%$  slower than a single model.
We thus use this ensemble to annotate the 85K sentence set used in (Lee et al., 2020). After removing parses with detached nodes we obtained 70K model-annotated silver data sentences. Adding these for training regularly, we achieve our best score of 83.4 with ensemble on AMR 2.0.
Model Size In Table 3, we compare parameter sizes of recently published models alongside their parsing performances on AMR 2.0. Similar to our approach, most models use large pre-trained models to extract contextualized embeddings as fixed features, with the exception of Xu et al. (2020), which is a seq-to-seq pre-training approach on large amount of data, and Bevilacqua et al. (2021), which directly fine-tunes a seq-to-seq BART large (Lewis et al., 2019) model. Except the large BART model, our APT small (3 layers) has the least number of trained parameters yet already surpasses all the previous models. This justifies our method is highly efficient in learning for AMR parsing. Moreover, with the small parameter size, the partial ensemble is an appealing way to improve parsing quality with minor decoding overhead. Although more performant, direct fine-tuning of pre-trained seq-to-seq models such as BART would require prohibitively large numbers to perform an ensemble.
Fine-grained Results Table 4 shows the fine-grained AMR 2.0 evaluation (Damonte et al., 2016) of APT and previous models with comparable trainable parameter sizes. Our model achieves the best scores among all sub-tasks except negations and wikification, handled by post-processing on the best performing approach. We obtain large improvement on edge related sub-tasks including SRL (ARG arcs) and Reentrancies, proving the effectiveness of our target-side pointer mechanism.
Ablation of Model Components We evaluate the contribution of different components in our model in Table 5. The top part of the table shows effects of 2 major components that utilize parser state information and the graph structural information in the Transformer decoder. The baseline model is a free Transformer model with pointers (row 1), which is greatly increased by including the monotonic action-source alignment via hard attention (row 2) on both AMR 1.0 and AMR 2.0 corpus, and combining it with the graph embedding (row 3) gives further improvements of 0.3 and 0.2 for AMR 1.0 and AMR 2.0. This highlights that injecting hard encoded structural information in the Transformer decoder greatly helps our problem.
Here we focus on trainable parameters for learning efficiency. For deployment the total number of parameters should be considered, where all the models relying on BERT/RoBERTa features would be on the similar level.
Model	SMATCH	Unlabeled	No WSD	Concepts	Named Ent.	Negations	Wikification	Reentrancies	SRL
Van Noord and Bos (2017)	71.0	74	72	82	79	62	65	52	66
Groschwitz et al. (2018)G	71.0	74	72	84	78	57	71	49	64
Lyu and Titov (2018)G	74.4	77.1	75.5	85.9	86.0	58.4	75.7	52.3	69.8
Cai and Lam (2019)	73.2	77.0	74.2	84.4	82.0	62.9	73.2	55.3	66.7
Naseem et al. (2019)B	75.5	80	76	86	83	67	80	56	72
Zhang et al. (2019a)B,G	76.3	79.0	76.8	84.8	77.9	75.2	85.8	60.0	69.7
Zhang et al. (2019b)B,G	77.0	80	78	86	79	77	86	61	71
Cai and Lam (2020)B,G	80.2	82.8	80.8	88.1	81.1	78.9	86.3	64.6	74.2
Astudillo et al. (2020)*R	80.2	84.2	80.7	88.1	87.5	64.5	78.8	70.3	78.2
APT smallR	81.7	85.4	82.2	88.9	88.9	67.5	78.7	70.6	80.7
APT baseR	81.8	85.5	82.3	88.7	88.5	69.7	78.8	71.1	80.8
Table 4: Fine-grained F1 scores on the AMR 2.0 test set.  ${}^{B/R}$  and  ${}^{G}$  marks uses of pre-trained BERT/RoBERTa embeddings and graph re-categorization processing. * We cite improved results reported in Lee et al. (2020). We report results with our single best model for fair comparison.
Model Configuration	SMATCH (%)
Mono. Alignment	Graph embedding	AMR 1.0	AMR 2.0
72.2 ±0.4	77.5 ±0.2
✓	78.0 ±0.1	81.5 ±0.1
✓	✓	78.3 ±0.1	81.7 ±0.1
No subspace restriction	78.0 ±0.1	80.9 ±0.1
RoBERTa base embeddings	78.0 ±0.1	81.3 ±0.1
BERT large embeddings	77.7 ±0.1	81.4 ±0.1
Table 5: Ablation study of model components. The analysis is with our base model size.
Data oracle variation	SMATCH (%)
Train oracle	Model test
None	98.9	81.7 ±0.1
No subgraph breakdown	97.8	80.6 ±0.1
Create farther edges first	98.9	81.4 ±0.2
Post-order subgraph traversal	98.9	81.8 ±0.1
Table 6: Results of model performance with different data oracles on AMR 2.0 corpus.
The bottom part of Table 5 evaluates the contribution of output space restriction for target and input pre-trained embeddings for source, respectively. Removing the restriction for target output space i.e. the valid actions, hurts the model performance, as the model may not be able to learn the underlying rules that govern the target sequence restrictions. Switching the RoBERTa large embeddings to RoBERTa base or BERT large also hurts the performance (although score drops are only  $0.3 \sim 0.6$ ), indicating that the contextual embeddings from large and better pre-trained models better equip the parser to capture semantic relations in the source sentence.
Effect of Oracle Setup As our model directly learns from the oracle actions, we study how the upstream transition system affects the model performance by varying transition setups in Table 6. We try three variations of the oracle. In the first setup, we measure the impact of breaking down SUBGRAPH action into individual node generation and attachment actions. We do this by using the SUBGRAPH for all cases of multi-node alignments. This degrades the parser performance and oracle SMATCH considerably, dropping by absolute 1.1 points. This is expected, since SUBGRAPH action makes internal nodes of the subgraph unattachable. In the second setup, we vary the order of edge creation actions. We reverse it so that the edges connecting farther nodes are built first. Although this does not affect the oracle score, we observe that the model performance on this oracle drops by 0.3. The reason might be that the easy close-range edge building actions become harder when pushed farther, also making easy decisions first is less prone to error propagation. Finally, we also change the order in which the various nodes connected to a token are created. Instead of generating the nodes from the root downwards, we perform a post-order traversal, where leaves are generated before parents. This also does not affect oracle score, however it gave a minor gain in parser performance.
Effect of Beam Size Figure 5 shows performance for different beam sizes. Ideally, if the model is more certain and accurate in making right predictions at different steps, the decoding performance should be less impacted by beam size. The results show that performance improves with beam size, but the gains saturate at beam size 3. This indicates that a smaller beam size can be considered
Figure 5: Effect of decoding beam size for SMATCH, with our best single models on AMR 2.0 test set.
for application scenarios with time constraints.
With the exception of Astudillo et al. (2020), other works introducing stack and buffer information into sequence-to-sequence attention parsers (Liu and Zhang, 2017; Zhang et al., 2017; Buys and Blunsom, 2017), are based on RNNs and do not attain high performances. Liu and Zhang (2017); Zhang et al. (2017) tackle dependency parsing and propose modified attention mechanisms while Buys and Blunsom (2017) predicts semantic graphs jointly with their alignments and compares stack-based with latent and fixed alignments. Compared to the stack-Transformer (Astudillo et al., 2020), we propose the use of an action pointing mechanism to decouple word and node representation, remove the need for stack and buffer and model graph structure on the decoder side. We show that these improvements yield superior performance while exploiting the same inductive biases with little train data or small models.
Vilares and Gomez-Rodriguez (2018) proposed an AMR-CONVINGTON system for unrestricted nonprojective AMR parsing, comparing the current word with all previous words for arc attachment as we propose. However, their comparison is done with sequential actions whereas we use an efficient pointer mechanism to parallelize the process.
Regarding the use of pointer mechanisms for arc attachment, Ma et al. (2018b) proposed the stack-pointer network to build partial graph representations, and Fernandez-Gonzalez and Gomez-Rodriguez (2020) adopted pointers along with the left-to-right scan of the sentence, greatly improv
ing the efficiency. Compared with these works, we tackle a more general text-to-graph problem, where nodes are only loosely related to words, by utilizing the action-pointer mechanism. Our method is also able to build up to depth  $M$  graph representations with  $M$  decoding layers.
While not explicitly stated, graph-based approaches (Zhang et al., 2019a; Cai and Lam, 2020) generate edges with a pointing mechanism, either with a deep biaffine classifier (Dozat and Manning, 2018) or with attention (Vaswani et al., 2017). They also model inductive biases indirectly through graph re-categorization, detailed in Section 6.1, which requires a name entity recognition system at test time. Re-categorization was proposed in Lyu and Titov (2018), which reformulated alignments as a differentiable permutation problem, interpretable as another form of inductive bias.
Finally, augmenting seq-to-seq models with graph structures has been explored in various NLP areas, including machine translation (Hashimoto and Tsuruoka, 2017; Moussallem et al., 2019), text classification (Lu et al., 2020), AMR to text generation (Zhu et al., 2019), etc. Most of these works model graph structure in the encoder since the complete source sentence and graph are known. We embed a dynamic graph in the Transformer decoder during parsing. This is similar to broad graph generation approaches (Li et al., 2018) relying on graph neural networks (Li et al., 2019), but our approach is much more efficient as we do not require heavy re-computation of node representations.
We present an Action-Pointer mechanism that can naturally handle the generation of arbitrary graph constructs, including re-entries and multiple nodes per token. Our structural modeling with incremental encoding of parser and graph states based on a single Transformer architecture proves to be highly effective, obtaining the best results on all AMR corpora among models with similar learnable parameter sizes. An interesting future exploration is on combining our system with large pre-trained models such as BART, as directly finetuning on the latter shows great potential in boosting the performance (Bevilacqua et al., 2021). Although we focus on AMR graphs in this work, our system can essentially be adopted to any task generating graphs from texts where copy mechanisms or hard-attention plays a central role.
Ramon Fernandez Astudillo, Miguel Ballesteros, Tahira Naseem, Austin Blodgett, and Radu Florian. 2020. Transition-based parsing with stack-transformers. arXiv preprint arXiv:2010.10669.
Miguel Ballesteros and Yaser Al-Onaizan. 2017. Amr parsing using stack-lstms. arXiv preprint arXiv:1707.07755.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178-186.
Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Symmetric amr semantic parsing and generation without a complex pipeline.
Jan Buys and Phil Blunsom. 2017. Robust incremental neural semantic graph parsing. arXiv preprint arXiv:1704.07092.
Deng Cai and Wai Lam. 2019. Core semantic first: A top-down approach for amr parsing. arXiv preprint arXiv:1909.04303.
Deng Cai and Wai Lam. 2020. Amr parsing via graph-sequence iterative inference. arXiv preprint arXiv:2004.05572.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752.
Marco Damonte, Shay B Cohen, and Giorgio Satta. 2016. An incremental parser for abstract meaning representation. arXiv preprint arXiv:1608.06111.
Timothy Dozat and Christopher D Manning. 2018. Simpler but more accurate semantic dependency parsing. arXiv preprint arXiv:1807.01396.
Daniel Fernández-González and Carlos Gómez-Rodríguez. 2020. Transition-based semantic dependency parsing with pointer networks. arXiv preprint arXiv:2005.13344.
Jeffrey Flanigan, Chris Dyer, Noah A Smith, and Jaime G Carbonell. 2016. Cmu at semeval-2016 task 8: Graph-based amr parsing with infinite ramp loss. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1202-1206.
Jeffrey Flanigan, Sam Thomson, Jaime G Carbonell, Chris Dyer, and Noah A Smith. 2014. A discriminative graph-based parser for the abstract meaning representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1426-1436.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212.
Jonas Groschwitz, Matthias Lindemann, Meaghan Fowlie, Mark Johnson, and Alexander Koller. 2018. Amr dependency parsing with a typed semantic algebra. arXiv preprint arXiv:1805.11465.
Zhijiang Guo and Wei Lu. 2018. Better transition-based amr parsing with a refined search space. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1712-1722.
Kazuma Hashimoto and Yoshimasa Tsuruoka. 2017. Neural machine translation with source-side latent graph parsing. arXiv preprint arXiv:1702.02265.
Young-Suk Lee, Ramon Fernandez Astudillo, Tahira Naseem, Revanth Gangi Reddy, Radu Florian, and Salim Roukos. 2020. Pushing the limits of amr parsing with self-learning. arXiv preprint arXiv:2010.10673.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Michael Lingzhi Li, Meng Dong, Jiawei Zhou, and Alexander M Rush. 2019. A hierarchy of graph neural networks based on learnable local features. arXiv preprint arXiv:1911.05256.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. 2018. Learning deep generative models of graphs. arXiv preprint arXiv:1803.03324.
Matthias Lindemann, Jonas Groschwitz, and Alexander Koller. 2019. Compositional semantic parsing across graphbanks. arXiv preprint arXiv:1906.11746.
Jiangming Liu and Yue Zhang. 2017. Encoder-decoder shift-reduce syntactic parsing. In Proceedings of the 15th International Conference on Parsing Technologies, pages 105-114, Pisa, Italy. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Zhibin Lu, Pan Du, and Jian-Yun Nie. 2020. Vgen-bert: Augmenting bert with graph embedding for text classification. In European Conference on Information Retrieval, pages 369-382. Springer.
Chunchuan Lyu, Shay B Cohen, and Ivan Titov. 2020. A differentiable relaxation of graph segmentation and alignment for amr parsing. arXiv preprint arXiv:2010.12676.
Chunchuan Lyu and Ivan Titov. 2018. Amr parsing as graph prediction with latent alignment. arXiv preprint arXiv:1805.05286.
Xuezhe Ma, Zeong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018a. Stackpointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403-1414, Melbourne, Australia. Association for Computational Linguistics.
Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018b. Stackpointer networks for dependency parsing. arXiv preprint arXiv:1805.01087.
Diego Moussallem, Mihael Arcan, Axel-Cyrille Ngonga Ngomo, and Paul Buitelaar. 2019. Augmenting neural machine translation with knowledge graphs. arXiv preprint arXiv:1902.08816.
Tahira Naseem, Abhishek Shah, Hui Wan, Radu Florian, Salim Roukos, and Miguel Ballesteros. 2019. Rewarding smatch: Transition-based amr parsing with reinforcement learning. arXiv preprint arXiv:1905.13370.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038.
Xiaochang Peng, Linfeng Song, Daniel Gildea, and Giorgio Satta. 2018. Sequence-to-sequence models for cache transition systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1842-1852.
Nima Pourdamghani, Yang Gao, Ulf Hermjakob, and Kevin Knight. 2014. Aligning english strings with abstract meaning representation graphs. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 425-429.
Michael Pust, Ulf Hermjakob, Kevin Knight, Daniel Marcu, and Jonathan May. 2015. Parsing english into abstract meaning representation using syntax-based machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1143-1154.
Rik Van Noord and Johan Bos. 2017. Neural semantic parsing by character-based translation: Experiments with abstract meaning representations. arXiv preprint arXiv:1705.09980.