SlowGuess/NAACL_2022_OCR huggingface dataset:
The sentence-level Smatch implementation<sup>4</sup> relies on a pool of candidate node-mappings to efficiently select the next greedy step. The pool includes all the node mappings that can possibly add to the triple F-score. We can restrict this candidate pool further for document-level Smatch if the source and target documents are aligned at the sentence level. More specifically, we can forbid any source to target node alignments where the sentence(s) of the source node is(are) not aligned with the sentence(s) of the target node. Note, that in DocAMR graphs, certain nodes can belong to multiple sentences such as coref-entity or a merged node.
We propose an implementation of Smatch (DOCSMATCH) that assumes alignment between the roots of sentence-level subgraphs of a pair of document-level AMR graphs. Nodes are first categorised by sentences with each node possibly assigned to multiple sentences. Next, the candidate mappings pool is constructed respecting the sentence-level alignments. In particular, a node in one AMR cannot be mapped to a node in the other AMR if none of their assigned sentences are aligned (see appendix A for details). For instance, consider the example in figure 4(c) – the node for the concept now must be aligned to a node in the third sentence in a target graph, whereas the person node, merged from mentions in all 3 original sentences, is not constrained.
DocSMATCH allows us to evaluate the DocAMR development set comprising of 42 documents in roughly 4 minutes with the default four random-restarts. This is a manageable time frame for the purpose of parser comparisons. The original Smatch evaluation for the same setup ran out-of-memory (with up-to 200GB allocation) without a result. Table 3 compares the Smatch scores and the runtimes of our implementation with those of the original Smatch with 1 random restart.
4.3 Coreference Subscore
A side effect of representing and evaluating the document AMR as a single unified graph is that we can not analyze the coreference performance of the parser separately. To mitigate this, we propose and implement a breakdown of Smatch that provides a separate coreference subscore. For the purpose of coreference subscore, all nodes connected to multiple sentences are considered coreferent nodes. Incoming edges for each coreferent node are counted as a part of the coreference subscore, as well as bridging relations and nodes with the labels coref-entity or interloculor-entity. Note, in the case of merged nodes, their incoming edges count towards the coreference scores, however the node themselves (i.e. their instance triples) are not counted towards the coreference score.
Since what is considered a coreference depends on the graph structure, the edges and nodes that are considered coreferent in the gold graph might be different from those in the predicted graph. Therefore, to calculate F1 for the coreference subscore, we consider an edge or node to be a correct match if (1) it has a matching node or edge according to the standard Smatch score, and (2) the node or edge is part of coreference in both the gold and predicted graph. Recall is calculated as a percentage of gold coreference nodes/edges, precision is a percentage of predicted coreference nodes/edges, and F1 is taken as the harmonic average.
We use DOCAMR along with our efficient implementation of Smatch to assess the quality of two document-level AMR parsing systems. First, we develop a pipeline system combining a top-performing AMR parser (Zhou et al., 2021) and a state-of-the-art coreference resolution system
Note also that for the border case where all nodes are connected with all sentences, the constrained version will be same as the original Smatch.
4https://amr.isi.edu/evaluation.html
5Note that there could be a case where the predicted parse of sentence 2 resembles the correct parse of sentence 1 and vice versa without any coreference link between the two then the proposed constraint would prevent finding the optimal alignment. However, this happens quite infrequently. Moreover, we argue that accidental mapping of triples between graphs of entirely unrelated sentences should not be rewarded.
MS-AMR Splits	Double Anno. (Dev1)	Double Anno. (Dev2)	Test
AMR	Coref	Smatch	Coref	Reent	Smatch	Coref	Reent	Smatch	Coref	Reent
Gold	None	87.6	0	72	88.6	0	73	86.4	0	72
Gold	CoreNLP	89.7	34	76	90.6	35	78	90.6	47	80
Gold	AllenNLP	90.5	40	78	91.0	41	79	91.3	53	82
(Anikina et al., 2020)	-	-	-	-	-	-	44.3	17	21
S-BART	None	67.1	0	53	67.8	0	53	67.5	0	55
S-BART	CoreNLP	68.7	28	57	69.3	29	57	71.3	43	63
S-BART	AllenNLP	69.4	33	59	69.8	34	59	72.0	50	65
Table 2: Document-level Smatch, coreference sub-scores (Coref) and reentrancy scores (Reent) on MS-AMR double annotations (Dev1 and Dev2) and test splits – using various combinations of gold and predicted AMR graphs with predicted coreferences from CoreNLP (Clark and Manning, 2016) and AllenNLP (Joshi et al., 2020).
Impl.	Original	DOCSMATCH
Split	R	Time	Smatch	Time	Smatch
Dev1	4	-	-	244	69.4
Dev2	4	-	-	136	69.8
Test	4	-	-	417	72.0
Dev1	1	927	69.5	66	69.3
Dev2	1	945	69.8	41	69.7
Test	1	1314	71.3	104	72.0
Table 3: Comparison of Smatch scores and runtimes (in seconds) between the original Smatch implementation (Original) and our proposed implementation (DOCSMATCH). All results are on our best performing pipeline system. R is the number of random restarts. 'Original' Smatch runs out of memory for  $\mathrm{R} > 1$  .
(Joshi et al., 2020). We also provide the pipeline results with CoreNLP's neural coreference resolution system (Clark and Manning, 2016) v4.3.2 for additional point of comparison. Second, we reevaluate the past best system of Anikina et al. (2020) – this is also a pipeline approach combining AMR parser of (Lindemann et al., 2019) with AllenNLP coreference resolution system. $^{6}$
5.1 Our Pipeline Approach
We use the BART-based structured transformer model of Zhou et al. (2021) to produce sentence-level AMR graphs. In particular, we use the StructBART-S version of their system referred to as S-BART in table 2. This parser produces node-to
token alignments as part of its output – we use these alignments to match coreference systems' outputs with AMR graphs. Text-based coreferences are obtained using the systems of Joshi et al. (2020) and Clark and Manning (2016). Coreference chains are computed from these prediction files.<sup>7</sup>
In order to incorporate this coreference information into the AMR graphs, we first convert node-to-token alignments into node-to-span alignments. The span of a node is defined as the smallest text span containing all the tokens aligned to any of its descendants (to avoid loops, re-entrant edges are removed keeping only the first). With node-to-span alignments, a predicted mention is assigned to the node with the shortest span containing the mention. If there is more than one candidate node, the one with the greatest height, subsuming the other candidates is selected. Instances of coreference within a sentence are ignored assuming that the sentence-level parser has already taken care of them.
Table 2 shows the results on MS-AMR double annotation documents (used here as development set) and its test split. Both gold and predicted AMR graphs are converted to DocAMR before running Smatch evaluation. All numbers are produced using document-level Smatch with 4 random restarts.
Our pipeline approach outperforms the previous best system by a large margin, providing a strong baseline for future research on this task. This is due mainly to difference in quality of the underlying sentence-level parsers.
7Using the package https://github.com/boberle/corefconversion
<sup>6</sup>We obtained the document-level graphs before merge operations from Anikina et al. (2020) for the purpose of re-evaluation in DocAMR format.
Dev1	Smatch	Time (s)
No-Merge	93.3	66
Merge-NE	92.8	66
DocAMR	87.6	90
Merge-All	82.4	153
Table 4: Impact of representation on evaluation scores and runtimes. Comparing double-annotation gold documents (Dev1) with and without coreference links in different document-level representations.
Note that significant improvements in coreference quality result in only small improvements in overall Smatch score - showing that a separate coreference subscore is essential for assessing a system's performance on cross-sentential relations. We also report reentrancy scores (Damonte et al., 2017) for comparison. While the performance gap is more pronounced in reentrancy scores compared to overall Smatch, the gains are best highlighted in coreference subscore. For instance, reentrancy scores improves by up to 2 points from CoreNLP to AllenNLP in all settings - coref subscores, on the other hand, shows up to 7 points improvement giving a finer range for coreference evaluation.
Impact of Representation on Smatch: To highlight how document-level representation can affect Smatch scores and efficiency, we compare the gold double-annotation development set with and without coreference links. In addition to DocAMR we consider three representations: 1) No-Merge: where all coreference nodes are linked via corefentity without any merging 2) Merge-NE: where only Named Entities are merged and 3) Merge-All: where all coreference nodes in a chain are merged (O'Gorman et al., 2018). One of our aims for DocAMR was to ensure that the lack of coreference links is visible in the overall Smatch score. Table 4 shows that DocAMR makes this gap bigger without losing efficiency or semantic information.
MS-AMR Annotations MS-AMR annotations by O'Gorman et al. (2018) include coreference chains, implicit roles and bridging relations. In the context of AMR-based summarization, Lee et al. (2021) present a novel dataset consisting of human-annotated alignments between the nodes of paired documents and summaries to evaluate merge strategies for merging individual AMR graphs into a document graphs. However, they sought out the
merge operations that can serve as cross sentential coreference in the absence of any annotations - they only merge nodes with same surface forms, except for 'person' nodes. Our work, on the other hand, outlines a representation for already available gold annotations, where nodes' surface forms don't match in a large number of cases.
MS-AMR Evaluations and Models O'Gorman et al. (2018) proposes Smatch as primary method for scoring MS-AMRs. They also report CoNLL-F1 relying on Smatch alignments. Adopting the methods from O'Gorman et al. (2018), Anikina et al. (2020) presented a comparative evaluation of various coreference resolution systems over MS-AMR test sets and document-level Smatch evaluations of machine generated sentence-level AMRs augmented with coreference predictions from various systems. The best approach from their study is incorporated as a baseline in §5. Fu et al. (2021) introduce an AMR coreference resolution system that uses graph neural network to model gold sentence-level AMR graphs for coreference predictions. This system assumes gold graphs and is not comparable with document-level parsing systems. Use of gold graphs also alleviates the need for alignments between gold and predicted graphs for the purpose of evaluation. Bai et al. (2021) constructed dialogue-level AMR graphs from multiple utterance level AMRs by incorporating inter-sentence coreference, speaker and identical concept information into sentence-level AMRs.
We have presented DocAMR, a graph representation for document-level AMR graphs based on coreference annotations. Relative to the original sentence-level graphs, DocAMR removes redundancy without information loss. We modified the implementation of Smatch to take advantage of sentence provenance to efficiently search for node alignments when comparing two document-level graphs. Finally, we reported results for a document-level parsing pipeline that can serve as a strong baseline for future work on this task.
Jeffrey Flanigan was supported by the NSF National AI Institute for Student-AI Teaming (iSAT) under grant DRL 2019805. The opinions expressed are those of the authors and do not represent views of the NSF.
Tatiana Anikina, Alexander Koller, and Michael Roth. 2020. Predicting coreference in Abstract Meaning Representations. In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference, pages 33-38, Barcelona, Spain (online). Association for Computational Linguistics.
Xuefeng Bai, Yulong Chen, Linfeng Song, and Yue Zhang. 2021. Semantic representation for dialogue modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4430-4445, Online. Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
Kevin Clark and Christopher D. Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643-653, Berlin, Germany. Association for Computational Linguistics.
Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536-546, Valencia, Spain. Association for Computational Linguistics.
Qiankun Fu, Linfeng Song, Wenyu Du, and Yue Zhang. 2021. End-to-end AMR coreference resolution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4204-4214, Online. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Fei-Tzin Lee, Chris Kedzie, Nakul Verma, and Kathleen McKeown. 2021. An analysis of document graph construction method for AMR summarization. arXiv preprint: arXiv:2111.13993v1.
Matthias Lindemann, Jonas Groschwitz, and Alexander Koller. 2019. Compositional semantic parsing across graphbanks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4576-4585, Florence, Italy. Association for Computational Linguistics.
Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A. Smith. 2015. Toward abstractive summarization using semantic representations. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1077-1086, Denver, Colorado. Association for Computational Linguistics.
Tim O'Gorman, Michael Regan, Kira Griffith, Ulf Hermjakob, Kevin Knight, and Martha Palmer. 2018. AMR beyond the sentence: the multi-sentence AMR corpus. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3693-3702, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Jiawei Zhou, Tahira Naseem, Ramón Fernandez Astudillo, Young-Suk Lee, Radu Florian, and Salim Roukos. 2021. Structure-aware fine-tuning of sequence-to-sequence transformers for transition-based AMR parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6279-6290, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
A Constrained Candidate Node Mappings for DOCSMATCH
Algorithm 1 Constrained candidate node mappings for efficient document-level Smatch computation.