AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite
Authors: Jonas Groschwitz
Shay B Cohen
Lucia Donatelli
Meaghan Fowlie
Venue: EMNLP 2023 Main (2023)
Keywords: evaluation
dataset
corpus
semantic parsing
AMR
sentence-level semantics
semantic graphs
TL;DR: We present an in-depth evaluation suite for AMR parsing that reveals abilities and shortcomings of current parsers

Abstract: We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge set for Abstract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the standard AMR evaluation metric Smatch, close to or even above reported inter-annotator agreement. But that does not mean that AMR parsing is solved; in fact, human evaluation in previous work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sentence meaning. Here, we provide an evaluation suite that tests AMR parsers on a range of phenomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels, to structural generalization, to coreference. GrAPES reveals in depth the abilities and shortcomings of current AMR parsers.

AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite
Jonas Groschwitz University of Amsterdam j. prosshwitz@uuv.nt
Shay B. Cohen University of Edinburgh scohen @ inf.ed.ac.uk
Lucia Donatelli
Meaghan Fowlie Utrecht
Vrij Universiteit Amsterdam L.e.donaeellc @ vu.nl
University
m.fowlie @u.nn
Abstract
ap2
op2
run
We present the Granular AMR Parsing Evalu- ation Suite (GrAPES), challenge set for Ab- stract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the stan- dard AMR evaluation metric Smatch, close to above reported inter-annotate agree- ment. But that does not mean that AMR pars ing is solved; in fact, human evaluation in pre- vious work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sen- tence meaning. Here, we provide an evaluation suite that tests AMR parsers on range phe- nomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels. to structural generalization, to coreference. GrAPES reveals in depth the abili- ties and shortcomings of current AMR parsers.
path
destination
ARG8
ARG1
ARGB
room
centrifuge
nechanic
Figure 1: AMR for The mechanic ran across the room to centrifuge and turned it off.
nical and linguistic interest. We answer the ques- tions of how well parsers can handle pragmatic coreference, cllipsis, PP attachment, rare words, unseen named entities and structural generalization. just to name few. Our Granular AMR Parsing Evaluation Suite (GrAPES) combines selection of existing and novel sentence-AMRR pairs.
A central theme in our development of GrAPES was that our metrics should actually evaluate what they promise to evaluate. To this end, we developed novel evaluation metrics specifically designed for the evaluation categories in our dataset. We also annotated, where necessary by hand, which graph- sentence pairs correct and unambiguous, well which pairs are relevant for each category. We further annotated what part of each graph is relevant, reducing possible distractors.
1 Introduction
Abstract Meaning Representation (AMR; Ba- narescu et al. 2013) parsing, the task of predicting a graph like the one in Fig. for a given sentence, has improved by leaps and bounds in recent years. In fact, parsing performance of recent parsers, as eval- uated by Smatch score (Cai and Knight, 2013), has reached and even surpassed reported human inter- annotator agreement (Bai et al., 2022; Banarescu al., 2013). Has AMR parsing reached human performance, and thus has this form of semantic parsing been solved? Opitz and Frank (2022) per- form human-expert evaluation for AMR parsing and find that, no, AMR parsing is far from solved: only 62% of parser outputs were rated acceptable.
Our work has three goals. First, to give quantita- tive results on a set of different phenomena in AMR parsing, so that we as community know what abil- ities AMR parsers have, and where we can trust them get right. Second, to provide tool that compares AMR parsers in more detail, and makes their differences visible. And third, to allow devel- opers of AMR parsers to see where their systems still struggle - what needs to be improved. In this sense, GrAPES also functions as a challenge to the community. The experimental results we present in this paper confirm that GrAPES can serve these three purposes.
In this work, we present an evaluation suite for English AMR parsing, including new data end met- rics, that measures performance of AMR parsers with unprecedented breadth, detail and accuracy. Instead of computing single score, like Smatch, on single test set, we evaluate parsing perfor- mance on a range of phenomena of practical, tech-
Our main contributions are:
A practically and linguistically informed eval- uation suite with 36 categories grouped into 9

---

sets
guistic phenomena such as reentrancies and named entities. We take this idea further, evaluating on 36 categories, expanding both breadth and depth. Moreover, we put additional emphasis on disentan- gling parsing performance on specific phenomena from overall performance, resulting in more pre- cise and interpretable metrics. Finally, GrAPES in- cludes newly annotated AMR-sentence pairs (both hand-built and grammar-gennrated)) specifically de- signed evaluate certain phenomena.
Fine-grained evaluation metrics by category
Evaluation results and statistical analysis for three recent parsers
Detailed analysis tools for parser developers.
GrAPES is available open source https:// gi wttthu..commgooosiiiiii comm/jgrroschiitt//rrA....
We start by discussing related work in Section 2. We describe our categories and why we selected them in Section 3, and our measures to achieve high quality data and metries in Section 4. We follow this up with an analysis of results for three recent AMR parsers (Section 5) and recommendations for how to use our evaluation suite (Section 6).
Beyond AMR evaluation, a growing literature on structural generalization is relevant to the AMR parsing task in developing evaluation suites for difficult semantic phenomena rooted in linguistic structure. For example, in the COGS dataset (Kim and Linzen, 2020) parsers must predict logical forms for sentences comprised of novel structural combinations, mimicking human ability to general- ize compositionally. This dataset has recently been extended in the SLOG dataset for semantic pars- ing (Li et al., 2023), which targets more difficult structural phenomena using the same logical form. GrAPES takes both AMR and non-AMR bench- marks inspiration for more comprehensive eval- uation for AMR. We include several existing tasks into our dataset: (i) the Winograd Schema Chal- lenge (Levesque et al., 2012), which consists of pairs of sentences with pronominal coreference that require pragmatics to disambiguate; (ii) Putting words into BERT's mouth (Karidi et al., 2021) for word disambiguation simple sentences; and (iii) the Unbounded Dependencies Corpus (Rimell et al., 2009) which contains real life examples of long range dependencies different categories.
2 Related Work
Evaluation for AMR parsing is currently an active field of research. The standard evaluation metric to compare monolingual AMRs (both parsed and hu- man annotated) is Smatch (Cai and Knight, 2013), which uses a graph matching algorithm to compute precision, recall, and F-score for semantic overlap of nodes and edges. Though designed to measure overall semantic adequacy of predicted graph, re- cent advances in AMR parsing performance appear have outgrown Smatch, spawning research on how to develop more mne-grained and interpretable evaluation metrics. Cai and Lam 20.1), in ad- dition to computing traditional Smatch on parser results, present variants of Smatch that emphasize 'core semanties' based on triple relations to the root, predicative core, of the AMR graph. Similarly, Opitz and Frank (2022) show that Smatch often misses small but significant semantic divergences and recommend supplementing the metric with other metrics (including human analysis) to mea- sure more fine-grained semantic adequacy. Opitz (2023) extends this work and separates Smatch into scores for pre-processing, alignment, and scor- ing. The above works reveal inconsistencies in how Smatch measures parser performance and point to the need for more focused evaluation methods.
3 Challenge Categories
We cover broad range of phenomena that are interesting from practical, technical, and linguistic perspectives.
3.1 Four Example Categories
First, let us look at four of the categories in our dataset in some detail; they will illustrate our deci- sions selecting categories below.
Targeted evaluations for AMR parsing in the form of phenomenon-specine benchmarks aim to address this need. Szubert et al. (2020) investigate the role of reentrant edges specifically. Damonte al. (2017) is the closest analogue of GrAPES, presenting an evaluation metric that pinpoints dis- tinct subtasks of AMR parsing. They introduce a set of nine metrics that measure challenging lin-
Frequent predicate senses: AMR builds on OntoNotes (Hovy et al., 2006) to disambiguate senses of predicates. For example in the AMR in Fig. the "-02" suffix in run-02 specifies the sense to be "walk quickly", as opposed to e.g. "op- crate". We say such a predicate sense is frequent if occurs at least 30 times in the AMR 3.0 training

---

set. To ensure that sense disambiquation is actually necessary, we also require that other senses for the same lemma in total occur at least 30 times. One of our categories tests parsing accuracy for such frequent predicate senses.
challenges (rare words, sense disambiguation, etc.) and structural challenges (pragmatic reentrancies, CP recursion, etc.).
Finally, we include a broad range of expected difficulty in our challenge categories. Some tasks are essentially impossible for current parsers, such as the predicate sense disambiguation for unseen senses (we explain why in Section 5). Some we expected to be difficult, such as deep CP recur- sions and pragmatic reentrancies. We also inten- tionally include some categories that we expect current parsers to perform well on, such as the sense disambiguation for frequent predicate senses, to check whether that expectation matches reality. Table 1 shows all categories in GrAPES.
Rare node labels: The node label centrifige in the example in Fig. is rare in the training set (occur- ring up 5 times). One of our categories measures a parser's ability to predict such rare labels.
Pragmatic coreference: AMR directly represents coreference in the graph: The fact that refers to centrifuge in the sentence of Fig. 1 is represented with reentrancy: both the destination destination-edgs of 72 and the ARG1 edge of turn-off-07 point to the same node. We include both pronominal and non-pronominal coreference. Coreference cannot be resolved by syntactic clues alone, and needs semantic and pragmatic information to resolve." We measure parsing performance on this type of reentrancy in its own category, and include more categories for other types of reentrancy.
Dataset and Evaluation Design
With wide range of phenomena selected, our guid- ing principles in creating the actual dataset and the evaluation metries that form GrAPES are:
1. High annotation quality.
Structural generalization for CP recursion: CP (Complementizer Phrase) recursion, as in You knew [that said [that the men leftlcr]cP. can in prin- ciple have unlimited depth. Evaluating parser per- formance on sentences with particularly high CP recursion depth higher than occurred in the train- ing data, up to depth 10- - is one of our structural generalization categories.
Metrics should measure the phenomenon they are supposed to measure, and nothing else.
4.1 Corpus Creation
Our four sources of data are the AMR 3.0 test set, other existing corpora, grammar-generated sen- tences, and hand-crafted sentences. Here we ex- plain how we added them to GIAPES.
3.2 Selecting Categories by Principle
4.1.1 AMRBank 3.0 Test Set
To ensure that our 36 categorics cover a diverse range of phenomena, we looked at our category selection through selection of different lenses.
Most of the phenomena we test already occur least some extent in the test of the AMR- Bank 3.0 (Knight al., 2021). We extract relevant sentences for a range of our categories.
The first is the lens of sparsity. For some de- cisions a parser must make, such as the frequent predicate senses, the parser has plenty of training data. For other phenomena, such as rare node la- bels, the Zipfian distribution of language means that while the node labels themselves are each rare, in total, rare words are common. Finally, some phenomena are truly rare: the deep CP recursions in our structural generalization tasks feature nest- ing of the same grammatical structure to a depth that does not occur the AMRBank at all. These truly rare phenomena are thus more of thcoretical interest (but especially so).
For each such category, a script extracts candi- date corpus entries (sentence-AMP pairs) from the test set. E.g., for rare words, for every node label that occurs 1-5 times in the training set, we pull every entry in the test set with that node label.
We then manually filter the extracted dataset if necessary. This can have multiple reasons. First, some of the extracted examples have annotation er- rors. This is more frequent in some categories - for example, an unseen node label may be unseen sim- ply because it is erroneous, and the corresponding word is not actually unseen, but has been anno- tated differently (correctly!) in the training set. We exclude such errors whenever feasible.
The second lens is that we include both lexical
'n exception are second and first person pronouns mul- tiple mentions of in a sentence refer to the same entity un- ambiguously. We measure these reentrancies in the separate category Unambiguous coreference.
Other sentences are ambiguous, or the AMR guidelines do not fully specify what the correct

---

1. Pragmatic reentrancies Pragmatic coreference
4. Rare and unseen words
6. Entity classification and linking
(T. C)
Rare node labels centrifuge
Types of seen named entities
(T)
Obuma's ; VP said the president forgot his cour.
Types of unseen named entities (
2. Unambiguous reentrancies
Unseen node labels gomn
(T)
LA =
(city :nane (name cop "LA">>
Syntactic (gap) reentrancies
(T)
Seen and/or casy wiki links North Korea = :wiki Korea'
(T)
She wunts and needs enter the room whistling
Rare predicate senses (excl. -01) Loow iee shirts = 10ose-83
(T)
Unambiguous coreference
(T)
Hard unseen wiki links
(T)
ralsed my fists self -iefonnee
Unseen predicate senses (cxcl.-01) (H)
Thiing Chenggong = swski "Koxtinga"
3. Structural generalization
The young reporter filled in for the usual news anchors. fill fill-in-67
7. Lexical disambiguations
Nested control and coordination
Frequent predicate senses He used the tool =
The boy wanted to force the doctor refuse attend and jumped.
Rare edge labels (ARG2+)
(T)
use-b1
We
commertial deselopment (dlevelop-02 :ARG3 we)
Other word ambiguites in Canada be-locatad-at-91
(C. H)
Multiple adjectives
(G)
stronge big antigue square dark container
Unseen edge labels (ARG2+)
(H)
hounce onto the roof (bounca-81
8. Edge attachments
Centre embedding
(G)
:RGG rouf)
PP attachment
(G)
lugged] nuught] left
The astronaut who 1iehe girl who the boy
5. Special entities
Sophie knew the journalistt with the telescope
Long lists
Seen names
Unbounded dependencies
(C)
Please buy a book. gusoline, fish, expensive food.
Unseen names (name :opl "Capitol" :op2 "Hill")
(T)
love and hute paper writing
beer, mup, phone and coul.
Passives
(T)
CP recursion
(G)
Seen dates
(T) TwaY seen
=
(see-81 :ARGI 1)
The lawyer said [thut you knew [that the leftl1]
Unseen dates
(T)
Unaccusatives
mentioned fihar the
(date-entity :nnth 12 :day 22)
Ifell > (fall-e1 ARGG i)
CP recursion coreference
(G)
Other seen entities
(T)
9. Non-trivial word-to-node relations
thougha that doctor heard mentioned that the girls hated her, the ductor
luwyer
Other unseen entities
(T) Ellipsis
(T)
CP recursion relative clause (RC) (G)
cuff him on his cell: 470-5715 =
Mone-nurber-entee vvalue "478-5715"
drive back and forth (two drive modex)
The girls (wwho we claimed ithat you thought Islepeler lee hated the laweer
Multinode word meanings
(T)
baker (person :AR08-of bake-01)
recursion RC + coreference (G)
Imperatives
TT)
The astronaut [whh we soid lliked the lawyer)crlac actuelly hoved her ofter all
Go! + (go-02
:ode
Inperative
:ARGB you)
Table 1: All categories in GrAPES, grouped into sets, Letters in brackets data sources: T= AMR testset, G= = Grammar, H = Handcrafted for GrAPES, C= Other corpora (Levesque et al. (2012) for Pragmatic coreference, Karidi et al. (2021) for Ambiguous words, and Rimell al. (2009) for Unbounded dependencies).
annotation for the sentence should be. We also exclude these sentences, since they can lead false negatives (when the parser predicts one option, but the gold annotation a different one).
with AMRs. We add AMR annotations, or partial AMRs for the relevant subgraphs.
4.1.3 Generation from grammars
In other cases, the heuristics by which we extract the candidates are not precise enough. For instance, our script that extracts reentrancies looks only for undirected cycles in the graph. We hand-annotate these sentences with their category of reentrancy: Pragmatic coreference (Mary thinks Susan likes her). Syntactic gap (She wants to sleep), or Unam- biguous coreference (I like my hair).
For structural generalization categories and for PP attachment, we write synchronous grammars that generate sentences and their graphs, using Alto (Gontrum et al., 2017), and sample from the lan- guage of the grammar. This gives us sentences at every desired recursion depth. For PP attachment, using a grammar allows us to add more lexical variety to sentences, while keeping them pragmat- ically unambiguous, e.g. The professor observed the army with the binoculars; The baker looked at the moon with the spyglass.
For any category, if initial sampling indicates that the rate of erroneous or ambiguous examples, examples that do not fit the category, is above 10,, filter the data by hand, selecting only correct, relevant examples of low ambiguity.
4.1.4 Hand-crafted
Finally, for some rare lexical phenomena, not enough relevant entries occur in the test set. For these we hand-crafted sentences and annotated them with graphs or partial graphs as necessary. We added short, simple sentences such as, for Un- seen predicate senses, The comedian has a dry sense of humor (sense dry-04).
4.1.2 Other existing corpora
We include sentences from the Winograd Schema Challenge (Levesque al., 2012), Putting Words into BERT's Mouth (Karidi et al., 2021) and the Unbounded Dependency Corpus for CCG parsing (Rimell et al., 2009) in our evaluation suite. How- ever, none of these corpora had been annotated
A detailed description on how we obtained the

---

corpus for each category is given in Appendix B.
4.2.2 Prerequisites and Sanity Checks
Even the above phenomenon-specinee metrics can- not always fully isolate the phenomenon, as we will see in the following. To further reduce false negatives, we use prerequisites and sanity checks. Consider a parse of (1) in which ev- erything is right except the node label centrifuge replaced by machine. Using edge recall to mea- sure PP attachment as above, the edge in question is measured as being absent, since there is no edge between nodes labeled run-02 and centrifuge since there is no node labeled centri fuge. For our purposes, though, this should not count as fail- ing at PP attachment, the PP attachment is not the problem: the node label is. For this reason, for these kinds of metrics we also measure pre- requisites: the parts of the graph that need to be present for the evaluation metric to be meaningful, but are not themselves what we look for. In our example, to measure the prerequisites, we check for the presence of nodes labelled centrifuge and run-02, because if these nodes do not exist in the first place, we cannot meaningfully evaluate the ex- istence of an edge between them. For each metrie, we can then use parser's prerequisite score as that parser's ceiling for the phenomenon.
4.2 Evaluation Metrics
To evaluate overall performance on a data set, Smatch is the standard for AMR. Smatch evaluates all phenomena by calculating the precision, recall and Fl for all triples in the graph (e.g. [source, edge label, target]). In our evaluation, however, we usually want to zero in on the phenomenon in question, ignoring other parts of the graph. For this we develop new evaluation tools, some of which we present in the following. A full list of metrics appears in Appendix B.
Prerequisites:
4.2.1
Metries
Recall the AMR in Fig. I for the example (I).
(1)
The mechanic ran across the room to the centrifuge and turned it off.
Node recall: for many lexical phenomena, we check only whether a node exists in the predicted graph with the label in question; e.g. centrifugh fuge in (1) for rare labels.
Edge recall: Often we are interested in the pres- ence of a particular edge. For instance, consider the prepositional phrase (PP) attachment of to the centrifuge - this PP here could, syntactically speak- ing, attach to the verb (ran [... J to the centrifuge) to the noun (the room the centrifuge). To see if the attachment correct, we check if there is any edge in the predicted graph between node labeled run-02 and node labeled centrifuge, with the correct label and direction.
Sanity Checks: In Structural Generalization cate- gories, we consider the whole graph, not just single edges and nodes. We therefore don't have prerequi- sites here. Instead, we use sanity checks. For most categories, these are unnested variants of the phe- nomena. For instance, for CP recursion, we check that a single CP can be embedded, as in She thinks that they left. In some, they are lexical checks, for instance in Long Lists, where we check for each item of a list separately. e.g. if Please buy bread, eggs, and cheese is in the generalization corpus. and the sanity check includes Please buy bread.
A parse that was correct except for drawing this edge wrong would have Smatch score of 92, but our measure would correctly get 0 for this metric on this entry. (That parse would, however, do fine for the Rare Words metric on this same entry.) Exact match: In the Structural Generalization set of GrAPES, we designed the grammars such that there is little in the way of distractors, ambiguity, or lexical challenges. Moreover, by the nature of the task, the graphs are very schematic, with repeated structures. Failing to capture this repetition that is, failure to capture this generalization can be evident from a single misplaced edge, yielding a high Smatch score but poor generalization. For these sentences, therefore, we hold the parser to the high standard of exact match.
In total, GrAPES evaluates 19590 datapoints: 15441 from the AMR testset, 3643 from grammars, 307 from other existing corpora and 199 from hand- crafted examples. In Tables 2 to 5, the rightmost column shows the number of datapoints for each metric for each category.
5 Performance of Current Parsers
5.1 Experimental Setup
To gain insights into the current state of the art in AMR parsing, we evaluate on GrAPES three recent parsers with very different parsing architectures. We evaluate:
In many metrics here we only use recall and not precision. Parsers "cheating" recall hy predicting multiple labels in an attempt to hit the right one is not an issue we observed.

---

Set Category
Metric
AM Parser
C&L
AMRBart
#
Pragmatic coreference (testset)
Edge recall
[02. 18]
08
[03, 22]
39
[25. 55]
Prerequisites
[34, 66]
36
[22, 52]
61
[45, 75]
Pragmatic coreference (Winograd)
Edge recall
[00. 13]
05
[01, 17]
32
[20, 48]
Prerequisites
[62, 88]
[18, 451
65
[50, 781
Syntactic (gap) reentrancies
Edge recall
[14, 391
24
[14, 39]
49
[34, 641
Prerequisites
[39, 68]
59
[43, 721
68
[53, 80]
Unambiguous coreference
Edge recall
[03, 25]
39
[24, 56]
65
[47, 79]
31
Prerequisites
[53, 84]
71
[53, 84]
77
[60, 891
Table 2: Results on reentrancy categories. Gray numbers in square brackets are 95%- confidence intervals.
The AM parser (Groschwitz al., 2018).
5.2 Results
neuro-symbolic compositional parser; we use the version with BERT embeddings, trained
We report our metric results in Tables 2 to 5, and compactly in Table 7. We include 95%-Wilson con- fidence intervals (Wilson, 1927) in square brackets (gray) give the reader an indication of the degree of uncertainty that results from the sample size.
on AMRBank 3.0 (Lindemann et al., 2020).
Cai and Lam (2020) (henceforth, C&L), structured neural parser that iterates between analyzing the string and predicting the graph. AMRBart (Bai et al., 2022), version of BART (Lewis al., 2020) finetuned for AMR parsing with additional graph pre-training. We use the model trained on AMRBank 3.0.
We also include in G:APES tool for visual- ization of gold and predicted graphs. While our quantitative evaluation is already fine-grained, we believe that a qualitative evaluation of examples is crucial interpreting the quantitative results.
Recall that we use the AMRBank 3.0 test set to extract some of our corpus, so in the following, "test set" always refers to that version. C&L was only trained AMRBank 2.0, and use that version here. Since AMRBank 2.0 is subset of AMRBank 3.0, all unseen/rare labels in 3.0 are still unseen/rare for 2.0; however, some of the parsing errors of C&L may be due to the training set, rather than the parsing architecture.
In the following, we present some highlights of our evaluation, both qualitative and quantitative.
The effect of sparsity. Across the board, parsers struggle when little training data is available for the task, and the less training data they have available, the more they struggle. This applies for example to rare and unseen node labels (Table 4): the most recent parser, AMRBart, does not even get half of the unseen node labels right. Interestingly, the unseen node labels are the only category in which the older AM parser outperforms AMRBart, pre- sumably because of the built in copy-mechame mechanis,, which AMRBart lacks. Similar patterns are vis-
TThere are some annotation differences between the 2.0 and 3.0 versions, in the testset of AMRBank to their counterparts and obtained a total Smatch score of (out of 100), indicating nearly identical graphs.
many: we compared the graphs
Set
Category
Metric
AM Parser
C&L
AMRBart
3
Nested control and coordination
Exact match
48 [35, 61]
08
[03, 19]
36
[24, 50)
50
Sanity check
Exact match
100
[77_100]
77
[50, 92]
100
[77.100]
13
Multiple adjectives
Exact match
72
[57, 84]
32
[20. 48]
98
[87,100]
40
Sanity check
Exact match
100 [74,100]
100
[74,100]
100
[74,100]
11
Centre embedding
Exact match
30 [17, 48]
13
[05,
57
[39, 73]
30
Sanity check
Exact match
85 [58, 96]
100
[77,100]
85
[58. 96]
13
CP recursion
Exact match
58 [48, 671
24
[17,
63
[53. 721
100
Sanity check
Exact match
100 [61,100]
100
[61,100]
100
[61,100]
6
CP recursion + coreference
Exact match
01 [00, 04]
09
[05, 14]
46
[39. 53]
182
Sanity check
Exact match
29 [15,
29
88
[69,
24
CP recursion + relative clause (RC)
Exact match
17 [09, 28]
00
17
[09, 28)
60
Sanity check
Exact match
75 [30,
25
75
4
CP recursion RC coreference
Exact match
00 [00, 05]
00
13
70
Sanity check
Exact match
00)
00
80
5
Long lists
Conjunct recall
02 03]
35
93
192,
1872
Conjunct precision
93 [82, 98]
57
197,
45
Unseen :opi recall
00
00
74
[70,
408
Sanity check
Exact match
97 991
81
99
195,100]
111
Table Results on structural generalization.

---

Set
Category
Metric
AM Parser
C&L
AMRBart
#
Rare node labels
Label recall
[39, 66]
56 [53. 60)
69
[66, 73]
Unseen node lahels
Label recall
60
[51, 68)
50
[41, 59]
45
[37, 54]
117
Rare predicate senses (excl. -01)
Label recall
[24. 491
[05. 211
45
[32, 581
56
Prerequisites
[79. 951
73
[60, 831
91
[81, 961
56
Unseen predicate senses (excl -81)
Label recall
05 [01. 171
00
[00, 09]
00
[00, 091
40
Prerequisites
[74. 951
90
[77, 961
85
[71. 931
40
Rare edge labels (ARG2+)
Edge recall
[04, 231
20
[10, 35]
35
[22, 50]
40
Prerequisites
57 [42. 711
55
[40, 691
72
[57, 84]
40
Unseen edge labels (ARG2+)
Fdge recall
08 [03, 22]
14
[06, 29]
33
[20, 50]
36
Prerequisites
[30, 60)
61
[45, 751
53
[37, 68]
36
5 Soen names
Recall
86 [85. 88]
91
[50), 912]
94
193, 955]
1788
Unseen names
Recall
68 [65. 711
60
[56, 63]
76
[73, 791
910
Seen dates
Recall
79 [73. 84]
72
[66, 771
94
190, 961
233
Unseen dates
Recall
47 [40, 54]
57
[50, 641
86
[81, 90]
204
Other seen entities
Recall
80
[75. 851
84
[79. 8S8]
97
194, 991
237
Other unseen entities
Recall
74
[65, 82]
33
[25, 42]
78
[69, 85]
109
6
Types of seen named entities
Recall
83 [81, 851
79
[77. 8.31
92
[90, 931
1628
Prerequisites
85 [83, 86]
91
[89) 92]
94
[93, 95]
1628
Types of unseen named entities
Recall
35 [32, 39]
29
[25. 322
51
[47, 55]
6.59
Prerequisites
59 [55, 63]
54
[50, 58]
70
[66, 73]
6.59
Seen and/or easy wiki links
Recall
70 [68. 72]
82
[80, 84]
87
[85, 88]
2064
Hard unseen wiki links
Recall
00 [00. 01]
18
[14, 231
09
[06, 13]
277
Table Results for sets 4-6: rare and unseen words, and special entities.
ible when comparing the prediction of frequent (Table 5) and rare (Table 4) predicate senses. For example, for C&L, the recall drops from 81 to 11, Named entities (Table 4) show the same picture: unseen entities consistently more difficult to handle than seen ones. While this trend is un- expected, we quantify in new detail for AMR parsing, and provide a consistent method for mea- suring progress.
grad even worse* (Table 2). PP attachment has similar eπrоr rate. Even for one of the best cate- gories here, Frequent predicate senses, among the labels where the lemma correct (i.e. the prereq- uisite satisfied), AMRBart gets about 10% of the senses wrong (Table 5). Since such sense ambigui- ties are so frequent (about one per sentence in the test set), even this small error rate quickly adds up. Structural generalization. Some of our structural generalization categories (Table 3) compare quite directly to the COGS (Kim and Linzen, 2020) and newly extended SLOG (Li al., 2023) datasets. For example, Weißenhorn et al. (2022b) show that finetuning BART on COGS gives 0% exact match for their CP recursion category; here AMRBart ob- tains 63%. This may be because the AMR training set is more diverse than COGS (which is restricted on purpose). While there is still lapp from the realistic language in the AMR training set the generalization examples here, the parser may have more data to make generalization from.
Where the state of the art does well. The most recent parser we test, AMRBart, achieves very high recall (92 and higher) for all categories of seen en- tities and their classification into types (Table 4). Passives and unaccusatives as well as frequent pred- icate senses receive lesser, but still strong scores (Table 5). In structural generalization, AMRBart nearly aces the Multiple adjectives test (Table 3).
Successes and struggles on contextual disam- biguation. While parser's ability make contex- tual decisions tested in many of our categories, it is particularly highlighted in pragmatic corefer- ence (Table 2), as well as word and attachment disambiguations (Table 5). We find that across the board, BMt shows noticeable improvements over the older parsers, and achieves respectable performance. However, there is still much room for improvement. For example, on the pragmatic coref- erences extracted from the test set, among the edges where the prerequisites are satisficd, AMRBart still gets about one third wrong; performance on Wino-
Still, structural generalization is hard. We get less than 50% exact match in most categories, and qualitative analysis shows that the performance drops with depth. For example, all but two cessful parses on CP recursion RC come from samples where there is only one CP. Surprisingly.
"We note that fine-tuned large language models reach a performance of over 95% on the Winograd Schema Challenge (Chowdhery et al., 2022). The lower performance here may be due to less powerful models, or due to additional difficulties in solving the task in the AMR format.

---

Set
Category
Metric
AM Parser
C&L
AMRBart
Frequent predicate senses (incl -01)
Label recall
81
[79, 83]
[79, 831
[84, 88]
1654
Prerequisites
92
190. 93]
[90, 93]
[93,
1654
Word anbiguities (handcrafted)
Recall
77 [63, 861
79
[65, 88]
[80, 971
Word anbiguities (Karidi et al., 2021) attachment
Recall
75 [65, 821
76
[66. 831
[80, 931
Edge recall
53 [48, 59]
43
[38, 491
[61. 711
325
Prerequisites
94 191, 961
[81. 891
193, 971
325
Unbounded dependencies
Edge recall
35 [24. 47]
32
[22, 44]
[34, 57]
66
Prerequisites
65 [53, 761
[47, 701
[53, 76]
Passives
Edge recall
55 [45. 66]
60
[49, 70]
[66, 84]
Prerequisites
75 [64, 83]
73
[63, 82]
80
[70. 87]
83
Unaccusatives
Edge recall
50 [36, 64]
69
[55, 80)
[57, 82]
48
Prerequisites
71 [57, 82]
75
[61, 85]
79
[66, 88]
48
Ellipsis
Recall
03 [01. 151
39
[25, 56]
55
[38, 70]
33
Prerequisites
91 [76. 97]
94
[80, 98]
94
[80. 98]
33
Multinode word meanings
Recall
58 [44, 711
60
[46, 721
84
[71. 92]
50
Imperatives
Recall
34 [25, 45]
43
[33, 55]
66
[55, 75]
76
Prerequisite
82 [71, 891
80
[70, 88]
89
[81, 951
76
Table 5: Results for sets 7-9: lexical ambiguities, edge attachments and non-trivial word-node relations.
the compositional AM parser, which has done very well on COGS (WeiBenhorn et al., 2022a), does not excel here. A possible reason for this may be that we use a version of the AM parser called the fixed-tree decoder, which performs better AMR overall (Lindemann et al., 2020). Weißenhorn et al. (2022a) use the projective decoder, noting that yields better generalization results.
dle pragmatic coreference, long lists, or ellipsis (Groschwitz, 2019). This reflects in the low scores in our corresponding categories (Tables 2, 3, 5). Further difficulties. There serious room for im- provement for all tested parsers in Syntactic (gap) reentrancies (which include reentrancies due e.g. control verbs and coordination), Unambigu- ous coreference (both Table 2), and Ellipsis and Imperatives (Table 5).
One thing note is that the different generaliza- tion categories have similar sentence lengths, but different parser performance. This shows that do not just measure sentenee length effects here.
5.3 Evaluating GrAPES
We have now seen how a range of parsers perform on our evaluation suite. But we also want to exam- ine to what extent we have reached the design goals of GrAPES in terms of granularity and whether our metrics measure exactly what they are supposed to. In particular, we compare GrAPES to the clos- est previous work, Damonte et al. (2017). Table 6 shows the metries of Damonte et al. for the three parsers that we evaluated on GTAPES.
For the generalization categories that also include coreference, showed form of parser bias, where male pro- nouns where more often successfully resolved than female pronouns; details in Appendix A.
qualitative evaluation
"Impossible" tasks. Some tasks not possible do in the classic paradigm of simply training a model on the training data, because external in- formation required. An example of this is the Unseen predicate senses category (Table 4), be- cause the numbers chosen for senses in OntoNotes (like the 02 in run-02 in Fig. 1) are arbitrary with respect to the actual meaning. That is, if the sense not observed in the training data, the only way relate the sense marker the meaning is look it up in OntoNotes, and parser that does not use that external resource cannot perform the task. Con- sequently, all parsers we tested score near here. Similar observations apply to Hard unseen wiki links. C&L and AMRBart use external tools for wiki links (Daiber et al., 2013; Wu et al., 2020), and therefore obtain non-zero accuracy here.
Fine-grained categories matter. First, we can see that using more fine-grained categories actually matters. For example, Damonte et al. use single metric for wiki links ("Wikificatioon), We split this category into seen and (hard) unseen wiki links and show that parser performance on the two is very different (Table 4). Similarly, Damonte et al. have single category for reentrancies. We show that parsers perform noticeably better on unambiguous reentrancies, compared to reentrancies that require pragmatic understanding to resolve (Tables 2, 7).
These more fine-grained evaluation insights are all the more relevant because improving parser per- formance for each of these phenomena may require a different approach. As we noted above, predict-
The AM parser is by principle unable to han-

---

Metric
AM Parser
C&L
AMRBart
ences that the Winograd dataset was built for. But also, for Damonte et al.'s metric, which measures Smatch on specific subgraphs related to reentran- cies, it is difficult to say what "55/100" exactly means. The recall on exactly the reentrant edges relevant to the Winograd schema challenge, which we measure, is more intuitively interpretable.
Unlaheled
78
86
No WSD
76
84
Named Entities
81
74
88
Wikification
71
80
79
Negations
60
70
73
Concepts
86
84
90
Reentrancies
56
62
73
SRL
73
74
83
Prerequisites and sanity checks. Our of pre- requisites and sanity checks further helps in mak- ing our metrics targeted, allowing us pinpoint the actual error types. Compare, for example, the performance of AMRBart on PP attachment and Unaccusatives (both Table 5). The numbers for edge recall are quite close, 66 and 71 respectively. However, on PP attachment, AMR satisfies the prerequisites nearly perfectly 95%, in contrast to the lower prerequisite percentage on Unaccusatives (79%). This allows us to conclude that, correcting for this difference in prerequisites, AMRBart does much better on Unaccusatives than PP attachment. Furthermore, the high parser performance levels on many sanity checks for structural generalization indicate that the difficulty in those categories does not just lie in some of the lexical items used in our grammars, but indeed in the structural gener- alization (Table 3). A qualitative analysis showed that most existing errors on the sanity checks are structural rather than lexical, indicating that even without deep recursion, the structures we test here are not trivial for current parsers.
Table 6: Damonte et al. (2017) metrics (AMR 3.0 test)
AM Purser
C&L
AMR
Bart
Smatch on AMRBank 3.0
75
84
1. Pragmatic reentrancies
04
07
36
2. Unambiguous reentrancies
17
32
57
3. Structural Generalization
32
4. Rare and unseen words
30
25
5. Special entities
73
66
88
6. Entity classification and linking
47
52
60
7. Lexical disambiguation
77
79
8. Edge attachments
48
51
9. Non-trivial word-to-node relations
32
48
68
Table 7: Compact GrAPES results table. Scores
averages over porprerequiiite, non-sanity-check scores. Note that this averages scores that are on the same 100 scale, but not necessarily the same metric.
ing unseen wiki links requires knowledge outside of the standard AMR training data, in contrast re- calling wiki links seen during training. In addition, not all methods are equally suited for pragmatic and syntactic reentrancies, as the limitations of the AM parser on pragmatic reentrancies show.
Successfully targeted metrics. Measuring parser performance for a specific phenomenon, disentan- gled from overall parser performance, is chal- lenge. For example, the "No word sense disam- biguation (WSD)" metric of Damonte et al. com- putes Smatch score, ignoring OntoNotes predicate senses. The difference the original Smatch score should then show the impact of WSD errors. How- ever, for example for AMRBart, both scores are 84, and for the AM Parser, both are 75 the WSD errors disappear during rounding. By contrast, we show that for both frequent (Table 5) and in partic- ular for rare and unseen (Table 4) predicate senses, the parsers make measurable mistakes.
Recommendations
For researchers in AMR parsing who want to show their parser's results on GrAPES, we recommend including the more compact Table 7 in the main paper, as well as highlighting results from specific fine- ainengraneed categories as applicable. complete table, combining Tables 2 5, should be included in the appendix. We encourage users of GrAPES to look at example parser output, to contextualize the metrics. Our evaluation suite includes code for visualization as well as for computing all results (and generating tables) for novel parser output.
Conclusion
We also computed the Reentrancy metric of Da- monte et al. the Pragmatic coreference (Wino- grad) portion of GrAPES, and found that the AM Parser obtains 55/100. This is in stark contrast to the recall of 2% that we measure. part this is due to Damonte et al. measuring all types of reen- trancies, while we focus on the pragmatic corefer-
We have shown that state-of-the-art AMR parsers still struggle with various phenomena, including data sparsity, contextual ambiguity resolution and structural generalization. We provide detailed evaluation suite with custom metrics to measure progress in these areas.

---

Limitations
outputs for Nested Control and Coordination found that the 60% error rate was driven largely by lcxical problems linking the word me to an node (in other parser-category pairings we examined, we did find mostly structural errors). Possible fixes could in- clude finding another metric for some structural generalization categories, perhaps changing the lexical distribution in our Seammar-generatee cor- pora.
Our dataset is designed specifically for AMR. While parsers on other semantic parsing tasks may make similar errors to the ones that we document here, drawing conclusions from our results to the overall state of semantic parsing should be done only carefully. However, we hope that this work serves as inspiration for creating similar evalua- tion suites for other tasks, in particular in syntactic and semantic parsing. While most of our imple- mentation work is not directly usable for other for- malisms (such as our manual AMR annotations or the code to filter the BMRB for specific phe- nomena), some of it could be used with some adap- tation. In particular, our grammars are built with Alto (Gontrum et al., 2017), which is specifically designed for multi-formalism grammars, meaning that our grammars be easily adapted generate different syntactic semantic structures.
Given the scale of our dataset, it includes possi- ble annotation errors and surface-form ambiguities. as is the case with most datasets of that scale. Our inspection of the dataset finds that these are mini- mal, but future work may focus on further cleaning up the dataset or quantifying the level of noise in it. In the case of ambiguity, future work may also create several possible references for each possible reading.
Ethics Statement
Further, our dataset designed for the English language. Some phenomena we test do not appear in all languages; e.g. not all languages can stack adjectives indefinitely. There are many interest- ing phenomena that are more pronounced in some non-English languages, that do not test here, for example how a parser would deal with richer morphology.
We do not see any particular ethical concerns with this work.
A Acknowledgements
Many thanks go to our supporting annotators: Maria Francis, Christoph Otto, and Anna Spasiano. We would also like to thank Alexander Koller, Matthias Lindemann, Ivan Titov, and Juri Opitz. for insightful discussions. Thanks also to Aditya Surikuchi and Sandro Pezelle for feedback on the paper. Last but not least, we would like to thank the reviewers for their helpful comments. This work is funded in part by the Deutsche Forschungsge- meinschaft (DFG, German Research Foundation) 492792184 84. This work is also part of the research programme Learning meaning from structure: neu- ral semantic parsing with minimalist grammars with project number VI.Veni. 194.057, which is funded by the Dutch Research Council (NWO).
One possible application of our dataset is not only in the evaluation of published parsers, but also during the development of new parsers. For exam- ple, the effect of change to the parsing architec- ture, designed to address parsing performance for a specific phenomenon, could be evaluated using GrAPES. However, since many of our examples are drawn from the AMRBank test set, and GrAPES overall has no dev/test split, this can lead to over- fitting to the test set. For now, for development we recommend only using datasets from GrAPES that are not drawn from the AMR test set. When report- ing results GrAPES, if some parts of GrAPES were used during development, that fact should be included in the report with high visibility. We hope to publish a development set for GrAPES in the near future.
References
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Despite our efforts to make our metrics focus precisely the specific tasks, sometimes less rel- evant errors are caught. For instance, the exact match metric for structural generalization can yield a zero if there are lexical errors. The sanity checks are designed to catch such issues, but will not catch all. For example, an analysis of the AM Parser
1: Long Papers), pages 6001 0001-6015. Dublin, Ireland. Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013, Abstract Meaning Representation

---

for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Dis- course, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Jonas Groschwitz. 2019. Methods for taking semantic graphs apart and putting them back together again.
Jonas Groschwitz, Matthias Lindemann. Meaghan Fowlie, Mark Johmson, and Alexander Koller. 2018. AMR dependency parsing with a typed semantic al- gebra. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Papers), pages 1831-1841, Melbourne, Australia. Association for Computational Linguistics.
Austin Blodgett and Nathan Schneider. 2021. Prob- abilistic, structure-awarg algorithms for improved variety, accuracy, and coverage of AMR alignments. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 3310-3321, Online. Association for Computational Linguistics.
Eduard Hovy, Mitch Mareus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of the human lan- guage technology conference of the NAACL, Com- panion Volume: Short Papers, pages 57-60.
Deng Cai and Wai Lam. 2019. Core semantic first: top-down approach for AMR parsing. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 3799-3809, Hong Kong, China. Association for Computational Linguis- tics.
Taelin Karidi, Yichu Zhou, Nathan Schneider, Omri Abend, and Vivek Srikumar. 2021. Putting words in BERT's mouth: Navigating contextualized vector spaces with pseudowords. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10300-10313, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Deng Cai and Wai Lam. 2020. AMR parsing via graph- sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 1290-1301, Online. Asso- ciation for Computational Linguistics.
Najoung Kim and Tal Linzen. 2020. COGS: A compo- sitional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing (EMNLP), pages 9087-9105, Online. As- sociation for Computational Linguistics.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceed- ings of the st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers). pages 748-752, Sofia, Bulgaria. Association for Computational I. Linguistics.
Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardoez, Kira Griffit,, UIf Herm- jakoh, Daniel Marcu, Martha Palmer, Tim O' Gorrman et al. 2021. Abstract meaning representation (amr) annotation release 3.0.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In Pro- ceedings of the 9th international conference on se- mantic systems, pages 121-124.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequeee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 787 7871-7880, Online, Association for Computa- tional Linguistics.
pre-training
Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Pa- pers, pages 536-546, Valencia, Spain. Association for Computational Linguistics.
Bingzhi Li, Lucia Donatelli, Alexander Koller, Tal Linzen, Yuekun Yao, and Najoung Kim. 2023. SLOG: A structural generalization benchmark for semantic parsing. In Proceedings of the 2023 Confer- ence Empirical Methods in Natural Language Pro- cessing (EMNLP), Singapore, Association for Com- putational Linguistics.
Johannes Gontrum, Jonas Groschwitz, Alexander Koller, and Christoph Teichmann. 2017. Alto: Rapid proto- Proceedings of
typing for parsing and translation. the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Com- patational Linguistics, pages 29-32, Valencia, Spain. Association for Computational Linguistics.
Matthias Lindemann, Jonas Groschwitz, and Alexan- der Koller. 2020. Fast semantic parsing with well- typedness guarantees. In Proceedings of the 2020

---

Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages $929-39511 On- line. Association for Computational Linguisties.
Juri Opitz. 2023.
Smatch++: Standardized and ex- tended evaluation of semantic graphs. arXiv preprint arXiv:2305.06993.
polarity
WiGI
dlggrea
how--re--roa--o1
lift-o1
Juri Opitz and Anette Frank. 2022. Better Smatch = bet- ter parser? AMR evaluation is not simple anymore. Proceedings of the 3rd Workshop on Evaluation and Comparison of NLP Systems, pages 32-43, On- line. Association for Computational Linguistics.
ARSZ
ARR2
Laura Rimell, Stephen Clark, and Mark Steedman. 2009. Unbounded dependency recovery for parser evalu- ation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 81 313-221, Singapore. Association for Compu- tational Linguistics.
Figure 2: AMR for The man couldn' lift his son because he was so weak.
Ida Szubert, Marco Damonte, Shay B Cohen, and Mark Steedman. 2020. The role of reentrancies Abstract Meaning Representation parsing. In Findings of the Association for Computational Linguistics: EMNIP 2020, pages 2198-2207.
The kids who said] liked the astronaut actually hated her after all.
where the part in square brackets can be deeper CP recursion. We used other lexical items in place of astronaut, that all were common in the dataset, which all turned out to be stereotypically male: lawyer, doctor and soldier. A manual examination of 30 parses by AMRBart showed that, when the underlined pronoun
Pia Weißenhorn, Lucia Donatelli, and Alexander Koller. 2022a. Compositional generalization with broad- coverage semantic parser. Proceddiggs of the 11th Joint Conference on Lexical and Computational Se- mantics, pages 44 54, Seattle, Washington. Associa- tion for Computational Linguistics.
male (actually hated him after all), the parser correctly resolved the coref- erence 14 of 15 times. However, when the pronoun was female, as in the example above, AM- RBart correctly resolved the coreference only 7 out of 15 times. The difference is statistically signif- icant (two proportion test, - 0.005). This is indicative of gender bias in the model.
Pia Weißenhorn, Yuekun Yao. Lucia Donatelli, and Alexander Koller. 2022b. Compositional generaliza- tion requires compositional parsers, arXiv preprint arXiv:2202.119377
Edwin B Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209-21..
Terry Winograd. 1971. Procedures as represen- tation for data in computer program for under- standing natural language. Technical report, MAS- SACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC.
B Appendix: Corpus and Evaluation Details
In this Appendix we go through the categories one by one, providing for each a description, examples, and explanations of the evaluations used. A few concepts will come up a few times; their definitions are given below.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6397-6407, Online. Association Computa- tional Linguistics.
Definitions
We will use the example in Fig. 2 to illustrate the definitions. It is taken from the Winograd cor- pus (Levesque et al., 2012), and annotated with AMR by us.
Gender bias observations
Our qualitative evaluation revealed form of bias in A Bar for the CP recursion with RC and coref-
erence category. The sentences in this category take the form
Node label recall there is a node somewhere the graph with this label.

---

The reentrant node The node we are interested in that should have two incoming edges. In Fig. 2, this is man, because we are evaluating the ability of the parser to predict that is the man, not the son or someone else outside the sentence.
The city councilmen refused the demonstrators permit because they feared violence.
The city councilmen refused the demonstrators permit because they advocated violence.
Near parent For reentrant node, the "near" parent is near in the sentence, for instance the predicate that selects pronoun. In Fig. 2, this is weak-02.
Evaluation
Metric Edge recall (labelled)
'near' parent to reentrant node 'far' parent to reentrant node
Far parent The "far" parent is the parent that "first" introduces the subgraph, for instance the predicate that selects a full RE. "first" here
Senses? No
Prerequisites reentrant node, near parent, far parent
is either highest co-indexed c-commander in the syntax tree (so, often the predicate that selects the binder) or leftmost in the sentence if the former doesn't apply. In Fig. 2, this is 1ift-01.
B.2 Unambiguous reentrancies
Reentrancies with only one possible interpretation. in contrast to set above. There are two kinds, Syntactic (gap) reentrancies, in which the struc- ture forces the co-reference, and Unambiguous coreference in which the lexical items force the co-reference.
B.1 Pragmatic reentrancies
Description Reentrancies that are not forced ei- ther by the structure or by the use of 1st or 2nd person pronouns (compare Unambiguous reentrancies). Includes third person pronouns, epithets, repetition of the RE, among others
B.2.1 Syntactic (gap) reentrancies
Description Includes control, nominal control, secondary predication, and coordination
Example Obama's VP said the president forgot his briefcase. Here, the president is an epithet for Obama, and his a pronoun referring to Obama, but both of these people could in prin- ciple be someone other than Obama, so these are pragmatically coreferent.
Example She wants and needs to enter the room whistling: want and need are control verbs, they share their ARG0 with enter. wants and needs is coordinated VP, sharing she their ARG0. whistling a subject depictive secondary predicate, also with she as its ARGO.
Dataset sources
Dataset source Test set
Test
extracted automatically; hand- labelled by type of co-reference
extracted automatically; hand-labelled by type of co-reference
Winograd (Levesque et al., 2012) A Winograd Schema is pair of sentences that differ only in one that contain referential ambiguity that resolved in opposite directions in the two sentences. The resolution must re- quire pragmatics, as opposed selectional restrictions such as animacy.
Evaluation
words and
Metric Edge recall (labelled)
'near' parent reentrant node
'far' parent to reentrant node
Senses? No
Prerequisites reentrant node, near parent, far parent
The corpus created
alternative the Turing Test for AI, based on an example from Winograd's dissertation (Winograd, 1971), which comprise the first two sentence of the corpus:
B.2.2 Unambiguous coreference
Description Includes 1st and 2nd person pro- nouns, "self".

---

Example raised my fists in self-defence: 1st per- son pronoun (1, my) yield reentrancy at self adds another.
wanted a subject control verb, which causes boy be the ARGO of force, creating another reentrancy at boy. Force itself an object con- trol verb, so its ARG1, doctor, is also the ARG0 of refuse. Refuse is again a subject control verb, creating another reentrancy at doctor.
Dataset souree Test set
extracted automatically; hand-labelled by type of co-reference
Dataset source Grammar
Evaluation
Care was taken here avoid structural am- biguity, for example choosing verbs that couldn't be interpreted to take nominal ob- ject.
Metric Edge recall (labelled)
"near' parent to reentrant node 'arr' parent to reentrant node
In the example, notice how the agreement keeps it unambiguous. If the main clause verb were in the first person present, the last word would be jump. In this case, it could also be the doctor who jumped.
Senses? No
Prerequisites reentrant node, near parent, far parent
B.3 Structural Generalization
Increasingly deep recursion of various structures (see below). All of these subcorpora are gener- ated by synchronous grammars. A combination of grammar design and sampling constraints prevents ambiguity. For instance, in grammar design, person and number (dis)agreement is leveraged to keep in- tervening phrases from being interpretable as co-referent with other noun phrases. In sampling. some rules cannot be reused, for instance a noun cannot be repeated so that no reentrancy will arise there.
Evaluation
Metric Exact match Edges Label? No
Senses? No
Sanity Check Exact match unnested trol and coordinated non-control verbs, all covering the same vocabulary. e.g. The boy wanted to jump.
B.3.2 Multiple adjectives
Description Noun phrases with stacked adjectives up to deep
Care was taken to keep everything except the phenomenon of interest as simple as possible. This includes single-node nouns (somewhat rare in AMR for humans, hence the repeated uses of girl, boy, astronaut, mechanic, soldier, lawyer, etc.) and ensuring that the words are common enough the training set that our test parsers succeeded on the sanity checks. In this way, we expect that difficulties parsing these sentences reflect difficulty with structural generalisation, not with the building blocks.
Example strange big antique square dark con- tainer
Dataset source Grammar
Ordering restrictions on English adjec- tives is respected: opinion, size, age, shape, colour, material.
Nouns and adjectives carefully cho- sen avoid structural ambiguity, which an adjective could be interpreted as a noun. For instance, we avoid colour names in favour of pale and dark be- cause, e.g., the substring strange red of strange red car could be taken to be strange colour of red, with nominal mod- ification of car.
B.3.1 Nested control and coordination
Description Control structures within control structures and coordination. Both of these phenomena create unambiguous reentrancies, and here they are nested up to 8 clauses or coordinations deep.
Example The boy wanted to force the doctor to refuse to altend and jumped. (depth 4)
Evaluation
VP coordination forces the ARG0 of both wanted and jumped to be boy. Meanwhile,
Metric Exact match Edges Label? No

---

Senses? No
Senses? No
Sanity Check Exact match on single- adjective noun phrases covering the same vocabulary, e.g. A fantastic plate.
Edge labels? Only for Unseen ooi recall Sanity Check Exact match on sentences with the same context but with only one item, covering the whole vocabulary. e.g. Please buy fish; I saw 98 rats
B.3.3 Centre embedding
Description Recursive relatives modifying the subject, up to depth 4
B.3.5 CP recursion
CP recursion is kind of sentential embedding, which Complementizer Phrases (usually headed by that) embed sentence as the object of a verb, as in We said that the astronaut left. Sentential embedding dependencies between verbs and their arguments; for example in The girls who we claimed that you thought slept hated the lawyer, the subject the girls far away from the main verb hated.
Example The astronaut who the girl who the boy hugged taught left
Centre embedding is hard for humans. This example has this structure: The astronaut who [[the girl who the boy hugged) taught] left.
create long- distance
Dataset source Grammar
Evaluation
In addition to distance in the string, we can talk about distance in the constituency tree: the length of the path between the dependent elements. Here the distance between the girls and hated is quite short, the whole relative clause who claimed that you thought slept is just modifier of girls.
Metric Exact match
Edges Label? No
Senses? No
Sanity Check Sentences without relative clauses covering the same vocabulary, e.g. The doctor hugged the mechanic.
GrAPES includes four kinds of CP recursion, corresponding to the full typology of dependency distance in the string and distance in the tree. The first category has no reentrancies or relative clauses, so all dependencies are short in the string and the tree.
B.3.4 Long lists
Description Lists up to length 35, embedded in a simple context. AMR handles coordination with one and node and as many opi edges as needed.
Description CP recursion with reentrancies relative clauses, up depth 10
Example Please buy a book, gasoline, fish, expen- sive food, beer, soap, map, phone and coal.
Example The lawyer said that you knew that the men mentioned that the boys believed that the women left
This will require edges op1 through op9.
Dataset source Grammar
Dataset source Grammar
Evaluations
In sampling from the grammar, peated nouns were allowed, to avoid ac- cidental reentrancies.
Metric Conjunct recall:
Evaluates what per- centage of conjuncts seen in the gold graph are also conjuncts in the predicted graph.
All verbs are either CP-selectors like think or intransitive.
Metric Conjunct precision: Evaluates what percentage of conjuncts seen in the pre- dicted graph are also conjuncts in the gold graph. Since the sample size de- pends on the number of conjuncts in the predicted graph, i.e. differs by parser, we do not report it here.
Evaluation
Metric Exact match
Edges Label? No
Senses? No
Sanity Check Sentences with one embedded
CP. covering all vocabulary
Metric Unseen opi recall:
Notes All dependencies are short in the string and the syntax tree.
for 20, 21,

---

B.3.6 CP recursion with coreference
choice of referent extremely pragmati- cally anomalous.
Description CP recursion with coreference:
Note that there unambiguous third- person reentrancies distance in En- glish - c.g. long-distance reflexives like in Farsi hence the design the 3rd person variants.
1st and 2nd person pronouns at distance of I 10 CPs
3rd person pronouns at a distance of to 10 CPs. Third person pronouns have clarifying phrase to make them unambiguous (see example)
B.3.7 CP recursion with relative clause (RC) Description CP recursion appearing within rela- tive clauses modifying subjects. The CPs push the relativised noun farther from its main pred- icate. We have 1-5 CPs within the relative clause.
Example We heard that you mentioned that I said that the kids liked you.
Example I thought that the doctor heard that we mentioned that the lawyer mentioned that the girls hated him, the doctor
Example The girls [who claimed that you thought slept]ac hated the lawyer
Dataset source Grammar
In sampling from the grammar, no re- peated nouns were allowed except for the target nouns and pronouns, to avoid accidental reentrancies.
While there is no cycle in the graph here, there are two incoming edges to the relativised noun girls: from the main verb in the sentence hated and the deepest embedded verb in the RC, slept.
The 3rd person pronoun and clarifying noun phrase always appear at the end, as objects of the last verb.
The two embedded CPs [that you thought [slept|ce/cp push hated farther from its sub- ject the girls.
First occurrences are not always the first noun phrase in the sentence, and reen- trant 1st and 2nd person pronouns are not always last
They also separate the verb slept from the dependent relativised noun girls and relative pronoun who.
Evaluation
Dataset source Grammar
Metric Exact match
Evaluation
Edges Label? No
Senses? No
Metric Exact match
Sanity Check Sentences with one embedded CP, and coreference, covering all vocab- ulary
Edges Label? No
Senses? No
Sanity Check One CP, c.g. The boys who
For pronouns: one reentrant pronoun pair
thought sneezed hated the doctor.
Notes
Both dependencies are long in the string, but the main clause verb is close in the syntax tree.
For 3rd person: One CP, e.g. The lawyer knew that I hated her, the lawyer.
Note that in the example, slept is in fact CP. It is missing its subject because of the relativisation, and there is no who because of the so-called that-trace effect in English.
Notes
The dependency created by the coref- erence is long in the string and long in the syntax tree.
We use 1st and 2nd person pronouns be- cause they
unambiguous, but they can be made easy by merging all we. and you nodes post-processing Third person pronouns are always ambiguous, but the context here makes any other
B.3.8 CP recursion with RC and coreference Description CP recursion appearing within rel- ative clauses modifying subjects, plus 3rd person pronominal coreference with a strong

---

pragmatic context. (See example). We have 1-7 CPs within the relative clause.
Evaluation
Metric Node label recall
Example The astronaut [who we said [liked the lawyerJcrlRc actually hated her after all
Senses? Yes
Prerequisites None
In addition to the long distance dependency in the RC, the pronoun her creates a reentrancy at lawyer.
B.4.2 Unseen node labels
Description Node labels in the test set that are not present training set
Dataset source Grammar
Example gown
All sentences have the following tem- plate:
Dataset source Test set
The N1 who [relative clause with CP recursion] liked/hated the N2 actually hated/tiked him/her after all
extracted automatically; hand filtered remove annotation errors and words in- consistently annotated in the training set
liked is always paired with hated and vice versa
Evaluation
Metric Node label recall
Evaluation
Senses? Yes
Metric Exact match
Prerequisites None
Edges Label? No
B.4.3 Rare predicate senses (excl. -01)
Senses? No
Description PropBank predicate senses that occur fewer than five times in the training set, but whose predicates also occur with other senses in the training set least once. We exclude the sense since it is universally very com- mon, making to guess blindly.
Sanity Check One CP, e.g. The girls who we claimed liked the doctor actually hated him after all.
Notes While all third person pronouns are tech- nically ambiguous. the contrast between like and hate and the actually..- after all makes it pragmatically very hard interpret the pro- noun be anyone other than the object of the earlier liked/hated
Example Loose shirts, Nursing bras, nursing pads. + 1oose-03
Dataset sources Test set
The reentrancy corresponds to short depen- dency in the string but long one in the syn- tax tree because the antecedent (the full noun phrase) deeply embedded in the subject rel- ative clause, but the pronoun the object of the main verb of the sentence.
extracted automatically; hand filtered remove annotation errors and words in- consistently annotated in the training set
Evaluation
Metric Node label recall
Senses? Yes
B.4 Rare and unseen words
Prerequisites Node label recall without senses
B.4.1 Rare node labels
Description Node labels in the test set that are seen one to five times in the training set
B.4.4 Unseen predicate senses (exel. -01)
Description PropBank predicate senses that do not occur in the training set, but whose predicates occur with another sense in the training set least once. We exclude the sense since it is universally very com- mon, making possible to guess blindly. mnseen_senses_tess tsv?
Example centrifuge
Dataset source Test set
extracted automatically; hand filtered to remove annotation errors and words in- consistently annotated in the training set

---

Example fill-in-07
B.4.6 Unseen edge labels (ARG2+)
The young reporter filled in for the usual
Description Argument edge labels (ARG2-ARG5) that do not occur in the training set with the given predicate.
news anchors.
The young reporter filled in for the usual
news anchors while they were on vaca- tion.
Example The kids bounced the ball five meters onlo the roof.
Dataset sources Hand-writter
Sentences appear in pairs, where the sec- ond extends the first with more pragmatic cues (see example)
(bounce-01
ARG2 distance-quantity
ARG4 roof)
We found that the test set included too few correct examples, so we created our own corpus.
Dataset sources Hrnd-written
As with unseen predicate senses, we found that the test set included too few correct examples.
We chose the predicates by how casy was for us to come up with example sen- tences for them.
Evaluation
Evaluation
Metric Node label recall
Metric Edge recall (labelled)
Senses? Yes Prerequisites Node label recall without senses
from the predicate to the target
Senses? Yes
Prerequisites Both nodes, including Prop- Bank sense
B.4.5 Rare edge labels (ARG2+)
Description Argument edge labels (ARG2-ARG5) that occur one to five times the training set with the given predicate.
Notes
Predicates might not occur in the train- ing set all; the prerequisite test checks for the predicate.
In the example below, this means develop-02 occurs one to five times with an ARG3 edge.
We include sense disambiguation the predicate because the meaning of role varies with the sense of the predicate.
Example we can actually get some commercial de- velopment and have the ability to work closer to where we live. + (develop-02 :ARG3 we)
B.5 Special entities
AMR treats names, dates, and other entities such as URLs, scores, phone numbers, etc specially, with node labelled with the type of entity and details given in attributes. For example, names have a name node with the parts of the name opi at- tributes. We test both and unseen entities.
Dataset sources Test set
extracted automatically; hand filtered to remove annotation errors and words in- consistently annotated in the training set
Evaluation
B.5.1 Seen names
Metric Edge recall (labelled)
Description Named entity names that occur in the training set
Metric Edge recall (unlabelled)
from the predicate to the argument
Example Capitol Hill - (n / name : "Capitol" :op2 "Hill")
Senses? Yes
Prerequisites Both nodes, including Prop-
Bank sense
Dataset sources Test set
Notes We include sense disambiguation on the predicate because the meaning role varies with the sense of the predicate.
extracted automatically
Evaluation

---

Prerequisites None Metric Recall on sequence of opi target node labels. If there no such node, recall is
Example .l.plesse call [him] his cell: 470- 5715...
vvalue "470-5715"
Names are subgraphs rooted in a name node. The components of the name are targets of opi edges. The name is correct if there is name node whose opi edge target labels match the gold sequence of opi edge target labels. In the exam- plc above, this is the sequence Capitol Hill.
Dataset sources Test set
extracted automatically
Evaluation
Metric Recall on the value (here: "470-5715")
Prerequisites None
Senses? N/A
B.5.2 Unseen names
Description Named entity names that don't oceur in the training set
Notes Full list of possible entities: "data-entity", "percentage-entity""" "phone-number-entitty" "mmall-address-entitt""" "url-entity", "correlate-91",
See seen variant, above, for details
B.5.3 Seen dates
"byline-91",
"course-91",
Description Dates where the same graph fragment occurs in the training data
ofane--dgree-o--rrrseemlaaaeee "hyperlink-91",
"fnstead-of-91",
Example December 22, 2002
"publication-91",
"requst-comffimmaiion-""""
(d / date-entity :month 12 :day 22 :year 2002)
"score-entity", "score-on-scale-911, "statisticall-est--91
Dataset sources Test set
"street--address-91", "string-entity", "value-interval", "variable"
extracted automatically
Evaluation
B.5.6 Other unseen entities
Metric Recall sequence of attributes of date-entity. If there is such node,
Description Other special entity types (numerical
values, phone numbers, string literals, urls,
recall is O.
...), where that value has not occurred the training data.
In this example above, this is day 22 month 12 year 2002
Prerequisites None
See seen variant, above, for details
B.5.4 Unseen dates
B.6 Entity classification and linking
Description Dates where the graph fragment does not occur in the training data (some parts of the fragment, such the month the year, may occur in the training data in different date).
Named entity types and their wiki links.
B.6.1 Types of seen named entities
Description Entities that occur in the training set. We test recall on the type (e.g. person, company, canal)
See seen variant, above, for details
B.5.5 Other seen entities
Example County and city officials in Los Angeles tried to determine... -> city
Description Other special entity types (numeri- cal values, phone numbers, string literals, urls,
Dataset sources Test set
...), .), where that value has occurred in the train- ing data.
extracted automatically

---

The type of named entity is the name parent of any name node. In the ex- ample, we get city from (c city :wiki "Los_ "oos_Angeees" :name (n name :op1 "Los" :op2 "Angeles"))
Metric Node label recall
If the graph contains a :wiki edge with the correct target, it counts as correct. Prerequisites None
B.6.4 Hard unseen wiki links
Evaluation
Description Wiki links for entities that do not in the training set, and whose Wiki links are not just the name joined with underscores, e-g. the way Barack Obama's 's wiki link is just Barack_Obama.
Metric Node label recall
Prerequisites There is a name node in the graph with attributes, in order, the same as in the gold graph.
In the example, we look for one with exactly two attributes, Los and Angeles, in that order.
Example Police sources also intimated that more crackdowns by the Hong Kong Po- lice on other Triad gangs will follow. + (oraadoorraniieddcrrin
Notes In AMR, the type is taken from the text if present; otherwise one is taken from the list of standard NE types. See https: //gi thub. com/ amr hlto://atter//ma...mmmmnaeeeeeet mitnamed-entittees
Dataset sources Test set
extracted automatically; filtered to ex- clude annotation errors
The Wiki link of named entity is any wwiki attribute of any node.
B.6.2 Types of unseen named entities
In the example, we get from :wiki criminal-organizatie
Description Entities that do not occur in the train- ing set. We test recall on the type (e.g.
"Triad_ ceaaddoorrniidderriiiiiie (c3
person, company, canal)
name (n2 / name :op1 "Triad")), and it qualifies hard because "Triad_ (oriidddorganize__rriiie just the name, "Triad".
See seen variant, above, for details
B.6.3 Seen and/or easy wiki links Description Wiki links for entities that occur, with their wiki links, in the training set, or whose wiki links are just the name joined with under- scores, e.g, Barack Obama's 's wiki link is just Barack_Obama _bbmaa.
is not
Evaluation
Metric Node label recall
If the graph contains :wiki edge with the correct target, it counts correct.
Example North Korean officials refused to pro- ceed... + "North_Korea"
Prerequisites None
B.7 Lexical disambiguations
Dataset sources Test set
Words in the sentence do not always have straight- forward annotations in the graph. Here we test PropBank predicate senses (e.g. the -01 of use-01) and words with ambiguous meanings such like, which can mean both similar to and enjoy, as Time flies like an arrow vs Fruit flies like a banana.
extracted automatically
The Wiki link of a named entity any :wiki attribute of any node.
In the example, we get "North "orrea" from (c
country :wiki "North_Korea" :name (n / name :0p1 "North" :op2 "Korea"), and it qualifies as easy because "North_Korea" is just the opis in order, "North" and "Korea", joined with
B.7.1 Frequent predicate senses
Description PropBank senses that occur at least times in the training data
Example If he really loves his son, he 'll see him regardless, but this is just a typical control method that people like him use. use-01
Evaluation
use-01

---

Dataset sources Test set
word
Subgraph
as
:tmme
extracted automatically; filtered auto- matically to exclude include-01 and include-91 as they are inconsistently annotated in the training set.
cause-01
like
resemble-01
like-01
really
degree really
real-04
since
since
Evaluation
cause-01
Metric Node label recall, including senses (e.g. use-01)
over
time over
:lccation over
About sentences per word sense
Prerequisites Node label recall, excluding senses (e.g. use)
All of the sentences are either unam- biguous or are pragmatically strongly favoured in the intended way. How- ever, also made point of mak- ing many of the latter technically am- biguous, even it requires some seri- ous mental gy gymnastics. For instance, perhaps fruit does fly like a banana flies.
B.7.2 Other word ambiguities Description Words with multiple meanings.
Example Time flies like an arrow. Fruit flies like banana
Every word sense has one or two en- tries hand-selected from the test set. These are included in the corpus with- out their AMRs since the dataset has restricted access. The test set IDs are listed below.
Dataset sources Two data sets evaluated existing non-AMR corpus and handwrit- ten corpus, which also contains 12 test sentences.
1. Putting Words into BERT's Mouth (Karidi et al., 2021)
Evaluation
Sets of sentences for ambiguous words had, in, on, run, started, with. about, for
Metric Node or edge recall
Some are non-core roles, and we ac- cept both the edge and the reification
Not all senses of the words notated differently in AMR, from the perspective of AMR, the dataset not entirely balanced, but from linguistic perspective, it is.
Edges Label? Yes
Senses? Yes
Prerequisites None
Notes
Test set entries:
e.g. for ambiguous word in, The event in Canada. be-1ocated-at-91
PROXY_NYT_ BBXX__NYT_ENG_200811 11008811180000..55
The event is in June. be-temporally-at-9
DF-199-194215-653. 4
DF-199-194215-653_ 0484.9
Hand-annotatte with full AMRs (provided as supplementary data set in our repository)
00000099900022111110111909909901 32
DF-220-192200--25_7888
DF-220-192400-625.000 24
Handwritten (plus 12 from the test set)
Chosen to complement Karidi et al. (2021)
Contains the following words and meanings:
BWOKY_XXNN_ENG2204411

---

B.8 Edge attachments
Most of these PPs have reifications, both are accepted; c.g. own-01,
A misplaced edge can dramatically change the meaning of a sentence. Here we examine four kinds of edge attachment: PP attachment, in which prepositional phrase can attach to verb, but one is strongly favoured over the other; unbounded dependencies, such (ong-distance wh-cxtraction, relatives clauses, and right node rais- ing: and passives and unaccusatives, in which the syntactic subject of the sentence is the ARG1 of the predicate.
have-03 for :poss
Senses? Only for the reification nodes, if they exist.
Prerequisites Since we are looking for an edge or a reification subgraph connecting two nodes, the prerequisite is the pres- ence of both nodes, modulo senses.
B.8.2 Unbounded dependencies Rimell et al. (2009)
B.8.1 PP attachment
Description The Unbounded Dependencies cor- pus consists of sentences drawn primarily from the Penn Tree Bank, annotated with two dependent words and type of dependency, Some dependencies entail reentrancies, and some only two incoming edges the same node. They are considered unbounded be- cause there is no limit on how far apart, the string and in the constituency tree, the two elements can be. Types are:
Description Specifically designed sentences to test PP attachment ambiguities. If one ignores the lexical content, just structurally speaking, the PP can attach the VP or the NP, but with the lexical content, there is either only one option licensed by selectional restrictions, or strong semantic/pragmatic preference.
Example Sophie knew the journalist with the tele- scope
1. Object extraction from relative clause
Check for edge telescope :poss
2. Object extraction from a reduced relative clause
journalist subgraph (have-03 ARGO
journalist ARG1 telescope (own-01
3. Subject extraction from relative clause
:RGG journalist :RGG telescope
4. Free relative
Dataset source Grammar
5. Object wh-question
6. Right node raising
Grammars are designed so that incorrect attachments are ruled out by selectional restrictions; for instance, knew is incom- patible with a seeing instrument like tele- scope. In other sentences, both readings possible, but one much more likely, For example, in The man sees the moon with the relescope, in principle, the moon could have telescope, but it much more likely be the seeing instrument.
7. Subject extraction from an embedded clause
Example That finished job that Captain Chan- dler and Lieutenant Carroll had begun.
This is an object relative, and we look for an edge between the embedded predicate begun and the relativised noun job: (begin :AGG job)
see Table for templates
Dataset source Unbounded dependencies Rimell et al. (2009)
vocabulary chosen to be present in the training set and be recognised by AMRBart and AM Parser.
This not an AMR corpus
Hand-annotated with relevant nodes and the edges between them.
Evaluation
Evaluation
Metric Edge recall (labelled)
Edge between the root of the mod- ified subgraph and the root of the modifier, e.g. between telescope and journalist.
Metric Edge recall (labelled)
Edge between roots of subgraphs with the meanings the two depen- dent words

---

Verb
Object her ambitions, aspira- tions, career, dreams
P
NP-modifiers
VP-modifiers
gave up, abandoncd
in
mathematics, theatre, crime
anger, a fit of despair, moment of clarity, the 60s, july, 2012, spring
Example:
My sister gave up her ambitions in mathematics My sister gave up her ambitions in a fit of despair onions, mushrooms, for the pasta sauce, the salad, the soup
bought, acquired, purchased, picked up
$5, $10, a few dollars, al- most nothing, an unreason- able amount of money
tomatoes, carrots
Example:
Kim bought mushrooms for the soup Kim bought mushrooms for $5
has kept
this posteard, letter, from necklace, souvenir this information, from knowledge, news,
Minsk, Munich, that adventure, Haiti
has kept
the children, Mark, the po- lice, Jenny
wisdom For thirty years, she has kept this letter from Minsk For thirty years, she has kept this news from Mark
Example:
read, skim, devour
this book, essay, by novel
Barack Obama, J. tomorrow, tonight, Mon- day, Tuesday, candlelight, firelight, lamplight
K. Rowling, Charles Dickens, this young author
Example:
will read this book by Charles Dickens will read this book by tonight
saw, looked at,
the girl, stranger, sol- with dier, journalist
the hat, red T-shirt, weird hair, large eye- brows
peeked at, observed
saw, looked at, pecked at, observed understood, knew, hated, sang to, ad- dressed
the northern lights, with moon, rainfall, army the girl, stranger, sol- with dier, journalist
the telescope, binoculars, spyglass
the telescope, binoc- ulars, spyglass
Example:
The baker the stranger with the weird hair The baker hated the stranger with the telescope The baker the northern lights with telescope
Table 8: PP attachment templates
The dependent words are the words annotated in the corpus to have a de- pendency
(make-01 ARG1 progress-01)
Dataset sources Test set
Senses? No
extracted automatically based on predi-
Prerequisites Existence of both nodes that should be connected with an edge. In the example, begin and job
cates with ARGe
Hand-filtered and categorised to separate unaccusatives (see below), and to remove errors and other constructions such as adjectives and expletives.
B.8.3 Passives
Descríption Passive sentences without an ARGO (so without a by-phrase)
Sentences with by-phrases aren't in- cluded because automatic extraction is much easier if you're 're just looking for predicates with no ARGO.
Example Kimball stated the Tranian government's design is to deflect criticism and pressure and to claim that progress is being made
Evaluation

---

Metric Edge recall (labelled)
Evaluation
on ARG1 edge
Metric Node recall
Senses? Yes
2+ occurrences of node label
Prerequisites Existence of both nodes that should be connected with the edge.
Senses? Yes
Prerequisites One occurrence of the node la-
B.8.4 Passives
bel
Description Unaccusatives: predicates that don't 't have an ARGO in their PropBank frame at all
Notes AMR is abstract, so naturally the alignments are not a part of the corpus.
Example Even though your anxiety levels would increase for duration, they will gradualy decrease so hopefully, overtime your OCD will disappear (:
B.9.2 Multinode word meanings
Description Some words' meanings comprise more than one node the graph
(increase-01 :ARG1 (1 / level)
Example Teacher (person teach-01)
ARG0-of
Dataset sources Test set
Dataset sources Test set
extracted automatically based on predi- cates with no ARG8
Extracted automatically using an align-
Hand-filtered and categorised to separate passives (see above), and to remove er- rors and other constructions such as ad- jectives and expletives.
ment tool, LEAMR (Blodgett and Schneider, 2021)
Entries with a word aligned to two nodes with different labels were extracted
includes double unaccusatives, which have more than one ARGi but ARGO. We only evaluate the ARG1 on these.
Filtered by hand remove alignment and annotation errors.
Evaluation
Evaluation
Metric Subgraph recall
Metric Edge recall (labelled) on ARG1 edge
Edges Label? Yes
Senses? Yes
Senses? Yes
Prerequisites None
Prerequisites Existence of both nodes that should be connected with the edge.
Notes AMR is abstract, so naturally the alignments are not a part of the corpus.
B.9
Nontrivial Word to Node Relations
B.9.3 Imperatives
B.9.1 Ellipsis
Description Imperatives have an imperative mode, and most have (usually latent) you or we (for exhortatives, as in the example). Some also have different overt subject, as in the second example
Description Sometimes word in the sentence is depicted twice in the graph, for example with ellipsis. For instance, in the example below, there are two wri re-0 nodes.
Example Mary wrote a paper and Susan did
Example
Let's go!
Dataset sources Test set
(go-02 02 :mode imperative :ARG we)
extracted automatically using an align- ment tool, LEAMR (Blodgett and Schneider, 2021). Entries with a word aligned to nodes with the same label were extracted, and the sample filtered by hand to remove alignment and anno- tation errors.
Go, Roughriders!
(go-31) :mode imperative :ARG0
Dataset sources Test set
extracted automatically: all entries with an :ode imperative attribute.

---

The few entries without you or we argu- ment to the predicate were hand-filtered to ensure the correct argument is checked (like in Go, Roughriders!)
Evaluation
Metric Edge recall
predicate :ode imperative
predicate to you or we
Edges Label? Yes
Senses? Yes
Prerequisites predicate (in the examples, 52 and go-31)


Reviews:
[{"review_id": "VtXBZAIdqk", "reviewer": "Reviewer_qBgp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. Thank you especially for your question about how our evaluation suite could be generalized to other syntactic or semantic formalisms. As the grammars we wrote are written in Alto (see citation Gontrum et al. 2017 in the paper), a program specifically designed for supporting multiple formalisms, they can be adapted and re-used for formalisms other than AMR. Additionally, the linguistic categories we use as inspiration for our evaluation may serve as templates for evaluating parsers and representational capabilities of other formalisms along the same lines. \n\nIn terms of transferring our actual implementation and annotation work to other formalisms, much of our work is formalism specific: the scripts we used to extract relevant sentence-graph pairs from the AMRBank testset are mostly AMR specific; and the manual filtering and AMR annotation work we did would need to be replicated for any formalism in question. \n\nThat said, we hope our work will serve as inspiration for the creation of similar evaluation suites for other formalisms as we believe the general idea will transfer well. We will use some of the extra space in the final version of the paper to detail the above response if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Most of the parsers are evaluated on standard metric and optimized for that metric. This is paper shows the need to look at different aspects of the parser output rather than single metric. Experiments, results and analysis is very thorough. Results of each category are presented with detailed analysis which gives insights in the drawbacks of the parser and areas for improvement. This is an interesting work and ACL community will benefit from analysis like this. ", "reasons_to_reject": "It would be good to mention how this work can be adapted for other grammatical formalisms like CFGs, CCGs etc.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an evaluation suite for AMR parsing. The authors first describe the categories on which AMR parsers are evaluated on. They create 36 categories grouped into 9 sets motivated from practical, and linguistic perspectives. Then the evaluation metrics designed are explained. The authors then take three state-of-the-art AMT parsers and evaluate them different categories. Results show that though these parsers achieve great average scores, they perform poorly on some of the categories showcasing the uses to improve up on.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ymQmJCV0lk", "reviewer": "Reviewer_Pruy", "summary": "", "questions": "Lines 398-404: There are major updates in AMR3.0 from AMR2.0. Specifically, frames for superlative and comparatives are subsumed under have-degree-91 and have-qaunt-91 in AMR3.0 that do not exist in AMR2.0 treebank annotations. have-degree-of-resemblance-91 covers constructions such as \"(He) is more like (his mother) than (his father).\" and correlate-91 covers constructions such as \"The more I practised, the fewer mistakes I made.\n\nAn easy of way of identifying the difference between AMR3.0 and AMR2.0 test set would be extract the AMR graphs corresponding to identical input texts between AMR2.0 and AMR3.0 and measure Smatch score between the two. \n\nLine 467 & 482-484: To what extent structural generalization can be subsumed under the issue related to long input texts? Which aspect of structural generalization can be decoupled from the input length issue?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments. If accepted, we will make the distinction to previous work more clear. We believe that our work contains major improvements over the previous state of the art of AMR evaluation, and want to take the opportunity to make this argument here.\n\nThe closest predecessor to our work is Damonte et al. (2017). Indeed, Damonte et al. (and its limitations) were a major inspiration for our work. We improve on it in the following ways:\n* As we write in the paper, our work is much more comprehensive (36 vs 9 categories). Getting a more detailed view matters, because often, challenges posed by different phenomena need different solutions, and should thus be evaluated separately.\n   * Example: We split the evaluation of predicted Wikipedia links into one category that looks at links that can be predicted from the text alone, and one that looks at links that need information from actual Wikipedia to be predicted correctly – two tasks that may require very different solutions. We show that the latter is much more difficult, something that can not be seen from Damonte et al.’s evaluation (or any other).\n* In addition to using the existing test set to evaluate parsers on different phenomena, we provide a large amount of new test items, drawn from existing non-AMR corpora, hand-written grammars, and additional sentences of our own. \n* Our metrics specifically evaluate the \"tail\" of the distribution, difficult phenomena where things still go very wrong with current parsers. Hence, our evaluation suite doubles as a parsing challenge, which previous work does not nearly to this extent.\n  * Example: Results of Damonte et al.’s metric for word sense disambiguation (WSD) imply that current parsers are strong at this task. We confirm this for frequent word senses, but demonstrate the parsers’ weak performance on rare (<50% recall) and unseen (~0% recall) word senses.\n* One challenge in evaluating specific phenomena is to isolate the measurement of that phenomenon from overall performance. Damonte et al.’s metrics only partially succeed in this; we greatly improve on this aspect through our more precise metrics and our method of measuring “prerequisites”.\n  * Example: Take our results on the Winograd subset of GrAPES. We measure coreference resolution here, specifically the existence of a reentrant edge E from a node A to a node B. For us to be able to measure the existence of E, the nodes A and B must exist in the first place – this is what we measure in our “prerequisites” metric. On this dataset, the AMParser obtains 2% recall, with 78% correct prerequisites. This shows that the parser cannot do this task well, and the high prerequisites score shows us that indeed the edges are the problem, not the nodes. By contrast, Damonte et al.’s reentrancy metric gives the AM parser a score of 55 (out of 100) on this dataset, erroneously implying a reasonably strong performance on this difficult task. This is because Damonte et al.’s metric measures all edges at a node that has a reentrancy, rather than measuring the one specific reentrant edge (the latter is only possible with our manual annotations).\n* Damonte et al.’s metrics rely on the AMR test set, which has issues relating to ambiguities and annotation errors when it comes to evaluating high accuracy parsers (shown by Opitz and Frank 2022, and confirmed by our experience). We take great care to circumvent these issues in our work.\n\nThe reentrancy metrics in Szubert et al. (2020) are similarly fine-grained as ours, but:\n* They do not measure accuracy directly, but measure how much the overall evaluation score would improve under certain oracle-guided changes to the predicted graph.\n* They divide their metric by oracle-patterns, not by linguistic categories like us.\n\nWe think that this makes our metrics more intuitive and useful, but we acknowledge that for reentrancies specifically, our work could be seen more as complementary to Szubert et al. But note that this makes up only 3 out of our 36 categories.\n\nThank you for bringing the work of Sheth et al. (2021) to our attention. As far as we can see, they only evaluate rare word / OOV properties of intermediate texts, not of the predicted AMRs themselves. To the best of our knowledge, we are the first to evaluate the ability of parsers to predict rare/OOV node and edge labels – and we do it comprehensively, in 11 categories.\n\nConcerning the difference between structural generalization and sentence length: we evaluate several different categories of structural generalization that have similar sentence lengths but vastly different parser results (for example, AMRBART achieves 15% exact match on CP recursion within a relative clause, but 63% exact match on simple CP recursion). This shows that sentence length is not the only factor here. If the paper is accepted, we will clarify the relation between structural generalization and sentence length in the paper.\n\nOverall, we believe that even compared to all existing metrics taken together, our evaluation suite is much more comprehensive, intuitive, precise and challenging, and we hope we convince you as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed test suite and evaluation metrics can be useful tool for AMR parsing error analysis. But I see quite a few overlap between the current proposal and previous work. And there can be simpler diagnostics than elaborate linguistic analysis, e.g. input length vs. structural generation for CP recursion.", "reasons_to_reject": "It is not clear to what degree the proposed fine-grained evaluation metrics overlap with previously proposed metrics. For instance, (Damonte et al., 2017; https://arxiv.org/pdf/1608.06111.pdf) evaluates 9 sub-categories including re-entrancies (mostly capturing coreference), named entity and Wikification. In addition (Szubert et al., 2020; https://aclanthology.org/2020.findings-emnlp.199/) discuss re-entrancy phenomena in depth with tools to identify them (https://github.com/mdtux89/amr-reentrancies), which include ellipsis. Rare word and out-of-vocabulary analyses are part of standard error analysis, as shown in (Sheth et al., 2021; https://aclanthology.org/2021.eacl-main.30.pdf).  Structural generation for CP recursion may be a subcase of the challenge posed by parsing long input texts. \n\nIt will be more informative if the authors compare and contrast the current proposal against previously published tools and metrics and demonstrate to what extent the current contributions are unique and cannot be captured by any of the previous work.", "questions_to_authors": "Lines 398-404: There are major updates in AMR3.0 from AMR2.0. Specifically, frames for superlative and comparatives are subsumed under have-degree-91 and have-qaunt-91 in AMR3.0 that do not exist in AMR2.0 treebank annotations. have-degree-of-resemblance-91 covers constructions such as \"(He) is more like (his mother) than (his father).\" and correlate-91 covers constructions such as \"The more I practised, the fewer mistakes I made.\n\nAn easy of way of identifying the difference between AMR3.0 and AMR2.0 test set would be extract the AMR graphs corresponding to identical input texts between AMR2.0 and AMR3.0 and measure Smatch score between the two. \n\nLine 467 & 482-484: To what extent structural generalization can be subsumed under the issue related to long input texts? Which aspect of structural generalization can be decoupled from the input length issue?", "paper_topic_and_main_contributions": "This paper presents an evaluation suite that tests AMR parsers on a range of phenomena with 36 categories grouped into 9 sets and fine-grained evaluation metrics by category. The suite is designed to answer the questions of how well parsers can handle pragmatic coreference, ellipsis, PP attachment, rare words, unseen named entities and structural generation. The authors apply the proposed evaluation suite to three AMR parsers and discuss evaluation results and provide statistical analysis.\n\nIt is not clear to what degree the proposed fine-grained evaluation metrics overlap with previously proposed metrics. For instance, (Damonte et al., 2017; https://arxiv.org/pdf/1608.06111.pdf) evaluates 9 sub-categories including re-entrancies (mostly capturing coreference), named entity and Wikification. In addition (Szubert et al., 2020; https://aclanthology.org/2020.findings-emnlp.199/) discuss re-entrancy phenomena in depth with tools to identify them (https://github.com/mdtux89/amr-reentrancies), which include ellipsis. Rare word and out-of-vocabulary analyses are part of standard error analysis, as shown in (Sheth et al., 2021; https://aclanthology.org/2021.eacl-main.30.pdf).  Structural generation for CP recursion may be a subcase of the challenge posed by parsing long input texts. \n\nIt will be more informative if the authors compare and contrast the current proposal against previously published tools and metrics and demonstrate to what extent the current contributions are unique and cannot be captured by any of the previous work.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DW9axW39Um", "reviewer": "Reviewer_KxKy", "summary": "", "questions": "Question A: Can you comment on the availability of Grapes for other researchers. How, when and under which kind of licensing will it be published?\nQuestion B: Will the analysis software be published as well?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their kind comments.\n\nWe will of course make the evaluation suite available open source (on GitHub with a GPLv3 license). We simply forgot to mention this in the original draft, sorry! We will publish the dataset, the evaluation code, the software for visualization and manual analysis of parser errors, and the grammars as well as the code we used to generate sentences from the grammars.\n\nThere is one caveat to this: GrAPES relies in part on the AMRBank testset, which is distributed by the LDC (and we thus cannot distribute it ourselves). Thus, to make GrAPES work, the user will need to obtain the AMRBank from the LDC. But since the AMRBank is the standard training and evaluation dataset for AMR parsing anyway, we don’t expect this to be much of a problem for those interested.\n\nThank you also for the point about showing different metrics in one table or even averaging them. Since the metrics are all on the same scale (0-100) and we specify the metric in each row of Tables 2-5, we felt that this was justifiable. We will be more explicit about it in the final paper, thank you for the feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* As the study from Opitz and Frank shows, the usual Smatch evaluation does not necessarily reflect human evaluation. Having having this resource will be very valuable for work on AMR parsing (and beyond), to understand the performance of parsers in detail.\n* The choice of phenomena is generally well founded and explained with a lot of detail in the appendix\n* The creation of the corpus represents a sizeable effort, and the analysis tools represent an additional value\n* Effort of standardizing the evaluation procedures for AMR in future work\n", "reasons_to_reject": "* The value of this work for further research depends on the availability of the test suite, and unless I missed something, there do not seem to be any remarks about the availability in the paper.\n", "questions_to_authors": "Question A: Can you comment on the availability of Grapes for other researchers. How, when and under which kind of licensing will it be published?\nQuestion B: Will the analysis software be published as well?", "paper_topic_and_main_contributions": "This paper introduces the Granual AMR Parsing Evaluation Suite (GrAPES):\n* An evaluation suite (corpus) is contributed for a wide range of phenomena, composed of AMR graphs from AMRBank 3.0 and of newly annotated sentences from a number of other corpora to account for phenomena that are not represented in AMRBank\n* Evaluation metrics and tools for analysis\n* An evaluation of recent parsers like AMRBart\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Every phenomenon is evaluated using an appropriate metric, which is ok. However, one could argue that it is problematic that in tables with the results, scores from different metrics computed over different numbers of datapoints are shown/mixed together, or even averaged over (in Table 6), even though a relative comparison is not possible. I think this point should be mentioned."}}]

Author rebuttal:

Meta-review: This paper introduces a comprehensive new evaluation suite for Abstract Meaning Representation (AMR) parsing, and shows that the AMR parsing remains far from solved.

Pros:
- Thorough evaluation suite covering a range of phenomena, with a practical taxonomy informed by real-world applications.
- Very clearly written, well-structured paper.
- Thorough experiments written up with clear analyses, rich detail.
- Introduces not just a data set, but also standard tooling to compare AMR parsers.
- Data and tools will be publicly available. Clear reproducibility.

Cons:
- Review copy isn't super clear in some parts when it comes to some comparisons with previous work, but if accepted, authors will address this, as per the review thread; as can be seen there, they already have a good understanding of what needs to be added.

This is an exciting contribution to the field; many thanks to the authors.
Decision: Accept-Main