"You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation
Authors: Allyson Ettinger
Jena D. Hwang
Valentina Pyatkin
Chandra Bhagavatula
Yejin Choi
Venue: EMNLP 2023 Findings (2023)
Keywords: semantic structure
AMR
linguistic annotation
LLMs
few-shot
zero-shot

Abstract: Large language models (LLMs) demonstrate an amazing proficiency and fluency in the $\textit{use}$ of language. Does that mean that they have also acquired insightful linguistic knowledge $\textit{about}$ the language, to an extent that they can serve as an "expert linguistic annotator"? In this paper, we examine the successes and limitations of the GPT-3, ChatGPT, and GPT-4 models, focusing on the Abstract Meaning Representation (AMR) parsing formalism (Banarescu et al., 2013), which provides rich graphical representations of sentence meaning structure while abstracting away from surface forms. We compare models' analysis of this semantic structure across two settings: 1) direct production of AMR parses based on zero- and few-shot examples, and 2) indirect partial reconstruction of AMR via metalinguistic natural language queries (e.g., ``Identify the primary event of this sentence, and the predicate corresponding to that event.'').
Across these settings, we find that models can reliably reproduce the basic format of AMR, as well as some core event, argument, and modifier structure$-$however, model outputs are prone to frequent and major errors, and holistic analysis of parse acceptability shows that even with few-shot demonstrations, models have virtually 0% success in producing fully accurate parses. Eliciting responses in natural language produces similar patterns of errors. Overall, our findings indicate that these models out-of-the-box can accurately identify some core aspects of semantic structure, but there remain key limitations in their ability to support fully accurate semantic analyses or parses.

" "You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation
Allyson Ettinger*1, .Jena D. .Jena
Hwang*
Valentina Pyatkin', Chandra Bhagavatula', Yejin Choi
Allen Institute for AI University of Washington jalnys valentinap, chandrab, vejinceealleeaai.......
Abstract
You are an expent Lingultstic onnotator
SENTENCE: "He woke to an angry house darkness the windows."
Large language models (LLMs) show amaz- ing proficiency and fluency the use of lan- guage. Does this mean that they have also ac- quired insightful linguistic knowledge about the language, to an extent that they can serve an "expert linguistic annotator"? In this pa- per, we examine the successes and limitations of the GPT-3, ChatGPT, and GPT-4 models in analysis of sentence meaning structure, focus- ing on the Abstract Meaning Representation (AMR; Banarescu al. 2013) parsing formal- ism, which provides rich graphical representa- tions sentence meaning structure while ab- stracting away from surface forms. We com- pare models' analysis of this semantic struc- ture across two settings: 1) direct production of AMR parses based on zero- and few-shot prompts, and 2) indirect partial reconstruction of AMR via metalinguistic natural language queries (e.g.. "Identify the primary event of this sentence, and the predicate corresponding to that event."). Across these settings, find that models can reliably reproduce the basic format of AMR, and can often capture core event, argu- ment, and modifier outputs are prone to frequent and major er- rors, and holistic analysis of parse acceptability shows that even with few-shot demonstrations, models have virtually 0% success in produc- ing fully accurate parses. Eliciting natural lan- guage responses produces similar patterns of errors. Overall, our findings indicate that these models out-of-the-boo can capture aspects of semantic structure, but there remain key limi- tations in their ability to support fully accurate semantic analyses or parses.
Provide AMR this sentence.
Whong coontination & misplaced modification
and
OP2
GPT-4
wake--01
hot coenpe
feneshot
ARG2
he
house
ARS1
ARGO
window
Incomolats argument
ridd
dariness
AMR Graph Generation
angry
Identify the primary events, its arguments, and modifiers.
Conjunction "and"
Wiong coordnatio
Prim
"acker
Misting argurrent
G shot
Argunents of Event "Ha"
Modifiers Event none
Not event
Primary Event 2: the windows"
Argunents of Event 2: "dorimnss""
Insiid
Meta linguistic
Modiftors of Event 2: nore
NL Response
Figure We compare LLMs' ability generate struc- tured semantic information across two settings: zero- and few-shot generation of AMR parses, and metalin- guistic natural language queries.
Observation of these capabilities has raised impor- tant questions about the extent, robustness, and limitations of the knowledge and analysis abilities of these models specialized domains.
model
In this paper we zero in on the domain of lin- guistic analysis: these models have shown great proficiency with language, but here we ask not just how well the models use language, but how much they know about language. Specifically, we explore to what extent models are able analyze the meaning of a sentence and reproduce the struc- ture of that meaning. Most directly this allows us to conduct a status check on the level of expertise that LLMs have acquired in linguistic analysis, and to assess to what extent linguistic structural annota- tion can be done reliably by LLMs out of the box. At higher level, this investigation has potential implications for the robustness of models' abstract representation of meaning language inputs. We intend for this to serve as a brief status report with respect to model capabilities this domain.
1 Introduction
LLMs in recent years have revolutionized artifi- cial intelligence, showing advanced proficiency and fluency in the use of language, and appearing to possess high levels of expertise and analytical capa- bility across a wide variety of specialized domains.
*Equal contribution

---

GPT4 géneration
Gold Parse
GPT4 (few-shal) goneration
Forn v
Node Preeennn Ces.ee
01
Top Vodo
contrast 01
Level 1
Analysis
Oxenl Accept
ARG1
Noent ReceOCARTTy
Evert Node
ARE1
ARG2
Deerall
Accept
(warrant-01
think-01
warrant-DD
think-01
Extr
ARG react 01
Blegran
Event X
ARGO
ARG1
ARGO
pularity
ARG2
ARET
pularity
ARG1
AgssModd
thing
such
(houghtless-O)
Foact 41
troughdess of
ARGo]
ARCo.al
AROE
LARGa
Level
react-01
mod
Argument
Aequment
ha
Analysis
Sentence: "It didn't warrant such reaction, but he was thoughtless.
Figure 2: We two-tiered semantic evaluation framework gain fine-grained insight into the strengths and limitations of generated parses. In the example, GPT-4 parse generations are compared against the gold parse.
For examining models' capability in analysing linguistic meaning structure, we focus on a case study of the Abstract Meaning Representation for- malism (AMR) (Banarescu et al., 2013). AMR designed to capture the abstract structure of sen- tence meaning, disentangling this structure from surface forms of language. It formalizes semantic structure of sentence into directed graphs that cap- ture "who did what whom" as well detailed abstract information how aspects of the sentence meaning modify and relate to one another.
these models effectively for such purposes.
Related Work
large body work has examined various aspects of syntactic and semantic capabilities in language models (c.f. Mahowald et al., 2023), indicating that LLMs show strong knowledge of syntactic struc- ture, while semantic capabilities are more mixed. Nonetheless, LLMs have also been used for few- shot semantic parsing with some success. In par- ticular, Shin al. (2021) and Shin and Van Durme (2022) find that few-shot learning GPT-3 and Codex produces semantic parses that outperform baselines with comparable training sizes. These semantic parsing datasets, which focus on produc- ing database queries in particular domains, are less complex and domain-generi than AMR, but the results suggest that LLMs should contain aspects of the knowledge needed to analyze semantic struc- ture. As for AMR, pre-trained transformer models have helped to advance the state of the art in AMR parsing, with recent AMR parsers building on the foundation of models like BART (Bevilacqua et al.. 2021; Bai et al., 2022; Lee et al., 2022; Zhou et al., 2021). This indicates that pre-trained models may also pick up representational capabilities rele- vant for supporting AMR.
In our explorations, we examine models' ability to produce the structural meaning information con- tained in AMR parses across three settings: zero- shot generation of AMR parses, fho-shot genera- tion of AMR parses, and zero-shot generation of natural language descriptions. Wc test GPT-3, Chat- GPT, and GPT-4. Our results show that all models are able to reproduce the basic AMR format and structure, and they can in principle produce correct outputs at any level of AMR withh greatest reliabil- ity on core event-arg argument triplets corresponding to subject-verb-objee structures. However, mod- els are prone to frequent and major errors in cap- turing the semantic structure (see Fig. 1, 2), and when assess the parses for overall acceptabil- ity, we see virtually 0% success rate across models. Comparisons between patterns in parse and natural language output settings suggest that these limita- tions are not simply artifacts of the output type, and may reflect more fundamental limitations in mod- els' capacity for semantic analysis. Overall, our findings indicate that although models can execute impressively formatted and partially correct seman- tic parse outputs, the prevalence of errors outside of basic components is such that these models cannot be used reliably out-of-the-box for generating this type structured abstract meaning information, and more involved techniques are needed to adapt
Though these prior works are suggestive that LLMs and pre-trained transformers capture certain aspects of linguistic structure, it is not clear from existing results how detailed or reliable LLMs' ability to analyze meaning structure may be formalisms used for prior few-shot semantic pars- ing simpler and more domain-specific than AMR, and the supervised fine-tuning of BART for AMR parsing obscures the details of what orig- inal knowledge may have been contained in the pre-trained model. To achieve a clearer picture LLMs' ability to analyze rich semantic structure,

---

Level criteria
3.2 Data
Proportion of parse ouputs that... Basic Form meet basic AMR graph format (concept nodes, edge relationships, hierarchical bracketing)
To ensure maximum flexibility and expert-level accuracy in assessment of the above criteria, we carry out our evaluation manually. In choosing to use fine- grained manual evaluation, we necessarily accept tradeoff with respect to scale and general- ization guarantees, as expert manual evaluation is time-consuming. We are not the first to accept this tradeoff: due to cost and increasing complexity of LLM outputs, there is increasing precedent for ana- lyzing model capabilities even on samples of single outputs (e.g.. Bubeck et al., 2023). Here we seek a balance between this kind of single-instance analy- sis and larger-scale coarse-grained evaluation, via fine-graine grained manual analysis on a small exploratory test set sampled across several domains.
Top Node
correctly identify the top parse node
Main Rel
select the correct main event as the highest event relation, disregarding non-eventive relations like and (Fig. 1) or contrast-O1 (Fig. 2)
Overall Accept
constitute valid AMR representation for the sentence, regardless of match to the gold annota- tion (allows credit models they parse the sentence reasonably but differ from gold)
Level 2 criteria
Proportion of parse outuus in which, for each correctly identified top-level event, ... Event
all arguments are present and identified argu- ments of that event
Args
Event
all event modifiers are present and identified as modifiers of that event
Mods
To this end, we compile a sample of 30 AMR gold-pan gold-parsed sentences, randomly selecting 10 sen- tences of varying character lengths from the gold AMR annotated AMR 3.0 (AMR3; Knight et al. 2021) and Little Prince' (LPP) datasets, and also sampling and annotating 10 sentences from web- sites published in 2023 (2023), to test the possibil- ity of memorization of public AMR annotations available in pre-training. This is large enough sample to gain some insight into trends at our dif- ferent levels of analysis, and future studies at larger scale can provide further insight into patterns that emerge in larger samples. See ŠA for more details.
Arg Mods
all modifiers for arguments of that event are present and identified as modifiers of the args
Extra Mods
there are any modifiers added crroncously to events or arguments anywhere in the parse
Table 1: Semantic criteria used for analysis
we directly examine pre-trained models' ability to produce AMR information, and we do so across a number of potentially productive zero- and few- shot settings for maximum insight about model ca- pabilities. We also prioritize fine-grained, manual analysis of models' accuracies at multiple levels of AMR information, in order to provide more de- tailed insights into model capabilities.
4 Zero-shot AMR parsing
Given findings of superior zero-shot performance in a wide variety of domains (Bubeck et al., 2023), we begin by testing models' zero-shot zero-shotcapabilitty for generating AMR graphs directly. Instructions and examples for AMR annotation are widely available online, so it is reasonable to imagine that models may learn to do this task zero-shot well.
3 Evaluation
3.1 Evaluation Framework
Standard metries for evaluating AMR include Smatch and SemBLEU, which provide holistic analysis of node matches between generated and gold AMR parses. While these metrics are well- suited for large-scale quantitative evaluation, they are not adequate for detailed understanding of mod- els' strengths and limitations in capturing AMR information. For more detailed insight, we lay out novel fine- fine-grained evaluation framework. We de- fine two levels: Level 1 criteria to capture basic format, highest-level nodes, and overall semantic accuracy; and Level 2 criteria for assessing accu- racy with arguments and modifiers. Table out- lines the analysis criteria (further details $C).
We input the model the target sentence and the simple instruction "Provide an AMR (Abstract Meaning Representation) parse for this sentence." For ChatGPT and GPT-4, we include the system message " "You are an expert linguistic annotator." Our goal here is to use clear and fair prompts that allow us to assess model capabilities and limita- tions. We do not do elaborate prompt engineering, but take the stance that if prompt is sufficiently clear, then a failure to perform the task is simply a failure- reliance on particular prompt phrasing or structure is an indication of model brittleness.
httpps:/amm iisi edu/ dow.load. html

---

Basic
Main
Top Node
Event Args
Event Mods
Arg Mods
Extra
Model
Accept
Model
Form
Rel
Mods (4)
GPT-3
0.7
0.4
0.3
0.0
AMR
GPT-3
0,4
0.4
0.1
0.4
0-shot ChatGPT
0.7
0.4
0.2
ChatGPT
0.4
0.5
0.2
0,5
GPT-4
1.0
0.4
0.4
0.0
(5-shot)
GPT4
0.5
0.5
0.4
0.4
GPT-3
1.0
0.7
0.5
0.0
Meta- linguistic
GPT-3
0.5
0.3
0.2
0.4
5-shot
ChatGPT
1.0
0.8
0.5
0.0
ChatGPT
0.6
0.4
0.2
0.3
GPT-4
1.0
0.7
0.6
0.1
GPT-
0.6
0.4
0.4
0.3
Table 2: Results of Level 1 analysis of zero- and few- shot parses on the 30 selected sentences
Table 3:
Level analysis of few-shot and NL outputs
ilarity to the test sentence." ?
Since the zero-shot parses fail often even at the basic levels, we limit to Level I analysis for this setting. Results are in the top segment of Table 2. The outputs indicate clearly that these LLMs have been exposed to AMR parse annotations in their pre-training, and have managed learn surface characteristics of AMR structure: we see that for all models a majority of outputs (>70%) show basic AMR format despite the absence of any demonstra- tion in the prompt. Comparison between publicly annotated sentences and newly annotated sentences from 2023 ($B) also shows no noteworthy differ- ence, indicating that output quality is not reliant on presence of test AMR annotations in pre-training- However, beyond the basic form, all models show frequent and substantial errors in the pars- ing. The parses identify the correct top node only 20-40% of the time, reflecting routine failures to in- corporate clausal and discourse relations that AMR often captures in the top rode-and even with the more relaxed criterion of identifying the main event relation, LLMs succeed in only about half of the parses (40%). When we consider the viability of the full structure as an appropriate meaning rep- resentation of the sentence, none of the models produce any fully acceptable AMR parse. These results suggest that out-of-the-box, zero-shot LLM capabilities are limited primarily to mimicking sur- face format of AMR representations, with under- standing of the linguistic functions and phenomena being beyond their zero-shot capabilities.
For few-shot parses, we upply both Level (Ta- ble 2) and Level 2 (Table 3) evaluations. We see that all parses now conform to AMR format, and the main event is now correct majority of the time. Identification of the top node has also improved, with correct outputs in approximately half of cases. However, the percentage of overall parse accept- ability has made virtually no improvement, despite the explicit few-shot demonstrations.
For Level 2 analysis, we see that models have limited reliability in identifying a given event's arguments and modifiers (40-50%) or argument modifiers (10-40%). Additionally, just under half of parses include at least one spuriiussy-identiffed argument or modifier ("Extra Mods"). Qualita- tive analysis indicates that models make diverse errors that can occur at any level of AMR struc- ture, though they show the most reliable accuracy in representing event-argument triplets correspond- ing to subject-verb-o structures. See ŠE for additional examples and discussion on these points.
6 Metalinguistic NL responses
By far the most thoroughly trained format for LLMs is that of natural language, so we next ex- plore models' ability to use natural language to identify and describe the abstract meaning struc- ture relevant for AMR, via prompting for meta- linguistic information about the target sentence. To do this, we formulate natural language prompt instructing the model to identify and break down aspects of the sentence meaning structure corre- sponding to components of AMR, similar to the process that an AMR annotator would use. Our prompt for this setting is shown in $D.
5 Parsing with few-shot demonstrations
Given that zero-shot parsing shows non-trivial lim- itations across all models, we next test how parses improve with few-shot demonstrations of AMR parsing. We use the same instructions (and, in ChatGPT and GPT-4 system message), but we now include the specification "I will first show some ex- amples." " followed by five example sentences with corresponding AMR parses, selected based on sim-
In this setting, the prompt asks for a breakdown of events, arguments, and event/argument modi- fiers, but does not elicit enough information to en- able complete parses. For this reason, focus
"Sentence similarity is computed via Universal Sentence Encoder embeddings (Cer ct al., 2018).

---

on our Level criteria for analyzing these outputs. Results are shown in the bottom of Table 3. We see that the overall patterns of accuracy are strik- ingly similar to those in the few-shot case (with marginal differences that are too small to read into with these small samples). 3 This suggests that lim- itations in the zero- and few-shot parses are not due simply to difficulty in generating parse for- mat, but may reflect more fundamental limitations in models' current capacity analyze semantic structure in language. This conclusion is further supported by the observation that the instructions contained in the metalinguistic prompt do not ap- pear at any level to be fundamentally too difficult for models to interpret: though models make many errors, for every component of the prompt they show in at least some cases the ability to produce correct outputs for that component. See $E for ex- ample outputs in the NL setting and discussion of instruction-folloiingg successes and error patterns,
AMR3
2023
LPP
AMRBart
0.78
0.79
0.75
GPT-4 (No fixes)
0,42
0.26
0.42
Auto fixes
0.48
0.40
0.47
+ Manual fixes
0.51
0.47
0.51
Table 4: AMRBart and GPT-4 few-shot smatch-score
parses and compare against those for the supervised AMRBART parser (Bai et al., 2022) on the same test sentences. Since GPT-4's generated parses are often flawed the point of Smatch not being able to run, we report three methods for obtaining these scores: no fixes, in which all failing parses are simply replaced with single-node placeholder parses; auto fixes, in which some automated format fixes are applied (see $F for details) and remaining errors are replaced with placeholders; and manual fixes, in which supplement automatic fixes with manual fixes correct remaining format errors.
The Smatch results (Table 4) clearly show that GPT- 4 output quality is far below that of the supervised AMR parser, supporting our general observation that the quality of these LLM parses is limited.
Comparing parse vs NL output Side-by-side inspection of parse and NL outputs supports the conclusion that, although errors are not identical, the two output formats may reflect real patterns in models' analytical capabilities. An illustration can be seen in Figure 1, which shows GPT-4 few-shot parse and zero-shot NL outputs for the sentence "He woke to an angry house and darkness in the win- dows". This is simple sentence, but the argument structure of the verb woke lacks the simple sub- ject/object structure that models succeed most and perhaps for this reason in both output formats the model misinterprets the sentence to include two separate events (missing the fact that the angry house and darkness in the windows is a single argument of the waking event) and creating nonsensical argument and modifier structures as result of the mistaken analysis. Additional side- by-side examples are included in Figures 5, 6, and 7 in the Appendix, further illustrating similarities between parse and NL outputs, and supporting the possibility that the observed errors in these outputs reflect real underlying analytical limitations rather than artifacts of instructions or output format.
Conclusion
Our analyses show that LLMs have acquired suf- ficient knowledge of AMR parsing and semantic structure for reliable generation of basic AMR for- mat and partially correct representations of sen- tence meaning. However, see abundant, diverse errors in model outputs, virtually fully accurate parses, and error patterns suggesting real underly- ing limitations in models' capacity to analyze lan- guage meaning. Our findings indicate that models are not currently sufficient out-of-the-box to yield reliable and accurate analyses of abstract meaning structure, and overall that this is domain in which models show only mixed levels of expertise.
We are confident that additional fiddling and clever manipulations can further improve the out- puts of these models, at least on certain dimensions. However, we present these results as a current sta- tus report and reality check to counterbalance fre- quent claims focused on widespread success and intelligence of these models out-of-the-Bel look forward continuing work better under- stand the fundamental strengths and limitations of these models in this domain, and to improve the reliability of semantic analysis capabilities achiev- able through collaboration with these models.
7 Smatch comparison
We
To anchor our results relative to an existing metric, we obtain Smatch scores for our five-shot GPT-4
'Though not included in Table 3, we note that the NL outputs also show comparablc accuracy to the few-shot parses in the Main Rel category for all models.

---

Limitations
Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.1277
In this paper we intend provide an overview and status check on the out-of-the-box capabilii ties of current LLMs for the rich semantic analysis captured by AMR parses. To enable fine-grain teee-graied manual evaluation not possible through standard metrics, we have used a small exploratory test set, and consequently our results do not enable statisti- cal comparisons or claims about how patterns may play out at larger scale. We look forward to future work applying comparable fine-graine -nniiiai analysis on larger samples, to verify what additional pat- terns of success and failure may emerge, and what broader generalizations can be made about model capabilities in this domain.
Damiel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Cuajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint 1rXiv:1803.11175.
Paul Kingsbury and Martha Palmer. 2003. Propbank: the next level of treebank. In Proceedings of Tree- banks and lexical Theories, volume 3. Citeseer.
Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, UIf Herm- jakob, Daniel Marcu, Martha Palmer, Tim O'Gorman, et al. 2021. Abstract meaning representation (amr) annotation release 3.0.
A potentially valuable extension that we do not include here would be a detailed comparison with models' success other (likely simpler) semantic or syntactic parsing formalisms. Given the richness of AMR and our focus on abstract semantic struc- ture per se, we do not include such an analysis in the current work.
Young-Suk Suk Lee, Ramón Fernandez Astudillo, Hoang Thanh Lam, Tahira Naseem, Radu Florian. and Salim Roukos. 2022. Maximum bayes smatch ensemble distillation for amr parsing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3379-5392.
Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fe- dorenko. 2023. Dissociating language and thought in large language models: cognitive perspective. arXiv prepriat arXiv:2301.0663
References
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Popers). pages 6001-6015
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster. Francis Tyers. and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resources Association.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguis- ric annotation workshop and interoperabilityy with discourse, pages 178-186.
Nathan Schneider, Jena D. Hwang, Vivek Srikumar, Jakob Prange, Austin Blodgett, Sarah R. Moeller. Aviram Stern, Adi Bitan, and Omri Abend. 2018. Comprehensive supersense disambiguation of En- glish prepositions and possessives. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics, Melbourne, Australia. Associ- ation for Computational Linguistics.
Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Sym- metric amr semantic parsing and generation without a complex pipeline. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35. pages
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen Jr. Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield fhw-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7699-7715.
Claire Bonial, Julia Bonn, Kathryn Conger, Jena D. Hwang. and Martha Palmer. 2014, PropBank: Se- mantics of new predicate types. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), 14), pages 3013- 3019, Reykjavik, Iceland. European Language Re- sources Association (ELRA).
Richard Shin and Benjamin Van Durme. 2022. Few- shot semantic parsing with language models trained on code. In Proceedings of the 2022 Conference
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar,

---

of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5417-5425.
2023 Sentences. To experiment with sentences verifiably not present in pre-training, we randomly sample sentences from websites published in 2023, To obtain the AMR gold parses (2023), we run the Spring Parser (Bevilacqua et al., 2021) on the sen- tences, and then the output is manually corrected by one of the authors with expertise in AMR anno- tation. These 2023 sentences are used in the test set, and few-shot examples for the 2023 sentences are drawn from the AMR 3.0 training set.
Jiawei Zhou, Tahira Naseem, Ramón Fernandez As- tudillo. Young-Suk Lee, Radu Florian, and Salim Roukos. 2021. atructure-waae fine-tuning of aequence-to-sequenec transformers for transition- based amr parsing. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 2279-6290.
A Dataset Details
A.1 More on Abstract Meaning Representation
A.3 Test Data Selection
AMR3 data was sourced from the AMR 3.0 BOLT and LORELEI instances with publicly available unified anmotation from PropBank (Bonial et al., 2014).* The Little Prince has also been partially an- notated with Universal Dependencies (Nivre et al., 2020) parses (Schneider et al., 2018), and for this work sourced from those The Little Prince in- stances that had both AMR and Universal Depen- dencies annotation. In both cases, we selected sen- tences 40-300 character length to eliminate in- complete phrases as well as overly long sentences, producing 413 AMR 3.0 and 67 The Little Prince instances. We then narrowed the sets to 10 random instances from each of the two data subsets. For 2023 Sentences, from online news sources and blogs with article date stamps of January 2023 or later. We selected 30 sentences, then narrowed the set to 10 instances of varying character lengths.
AMR formalizes semantic structure of sentence into directed graphs that capture the "who did what whom" of the sentence (Banarescu al., 2013). In AMR, events and entities are represented as con- cept nodes, and semantic relationships or semantic roles as edges. AMR abstracts away from syntactic and morphological surface variations in favor of conceptual representation of predicate-argument structure of a sentence in part by adopting English PropBank (Kingsbury and Palmer, 2003) for event representation. In this way, AMR allows meaning generalization across various surface expressions (e.g., "The girl adjusted the machine" and "The girl made adjustments to the machine" would have the same AMR graph). AMR supports other linguistic analysis such as coreference, named entities, and modality (among others).
manually selected sentences
A.2 Data Sources
AMR 3.0. We use the AMR 3.0 dataset (AMR3; LDC2020T02), which includes 59K AMR graphs. The graphs are manually annotated from English natural language sentences from various genres of text including newswire, discussion forums, fiction and web text. We particularly focus on two En- glish subsets: BOLT discussion forum data and LORELEI data. Our few-shot in-context examples and test instances are pulled from the AMR 3.0 training and dev sets, respectively.
B Concerns of Memorization
We took into consideration the possibility that the parses of AMR3 and LLP could be present in the pre-training data of the tested LLMs. This served as the motivation for including the 2023 sentences.
Table 5 shows Level 1 zero-shot parse results broken down by dataset. These results suggest that the quality of the AMR generations is not reliant direct memorization of the annotated parses from pre-training-inn fact, we find the results for 2023 parses to be nearly identical to the LPP results. A closer qualitative look at the parses did not surface any noteworthy differences in parses.
The Little Prince. We use publicly available AMR annotations of the novel The Little Prince by Antoine de Sxint-Exupéry (translation of original French, Le Petit Prince: LPP). The corpus contains 1.5K sentences with their corresponding, manually- created AMR graphs. Our few-shot in-context ex- amples and test instances are non non-overlapping sam- ples drawn from this dataset.
"ht 'https://gli thub. htt.b//wwmrrrooaaaaoorooaaanaae http://ppppaaaleeeaeeeee *We ensured inclusion of diverse lengths via manual veri- fication.

---

Basic
Main
Top
Serience: "If everyone has to spend all their time trying to circumvent the rules, understend the rules or fighting against those following the rules, that is where trouble really begins""
Model
Accept
Form
Rel
Node
GPT-3
0.8
0.5
0.3
(d default-02-e.0
begin sang1 trouble-e.35) stlew really-e.373)
AMR3
ChatGPT
0.7
0.4
0.3
0
GPT-4
1
0.6
0.5
O
<c couse 02-e.1
cang1 (e / everyone-e.4)
GPT-3
0.6
0.3
0.2
0
sang2 (a / and-e. 19
LPP
ChatGPT
0.8
0.2
0.1
0
spend-0l-e.5
:arg1
GPT-4
0.3
0.4
:ang2 (t2 tire-c.6)
:org3 (c2 circament--l1ee.
GPT-3
0.6
0.3
0.3
0
:orgl (r2 / tule-01-e.18))
cop2 (u understond-01-e.1r
2023
ChatGPT
0.9
0.2
0.3
0
GPT-4
0.3
0.4
0
Figure 4: CharGPT output that is not fully up to AMR standards but passes our Basic Form criteria
Table 5: Comparison of the Level analyisis on zero- shot data across the three datasets
Overall acceptability For our overall acceptabil- ity measure, an AMR expert among the authors assessed whether each parse could be a valid rep- resentation of the sentence meaning, based the AMR annotation guidelines, regardless of match to the gold annotation. This was intended to give fairer credit to model outputs- none of the mod- cls' generated parses managed to perfectly match the gold annotation, but it was possible that some parses may still accurately represent the meaning of the sentence, with some annotation differences from the gold parse. So this measure used expert assessment to judge parse validity in this broader sense. These assessments also forgave minor struc- tural/formatting errors, as long as correct seman- tics could be interpreted.
C Analysis Criteria Details
Here we provide further elaboration and illustra- tion with respect to the analysis criteria outlined in Table 1.
Assessment of Basic Form For the Basic Form criterion we simply ask whether or not the pro- duced output looks (generally) consistent with AMR's standard structure. More specifically, a parse should critically retain three basic AMR for- mat components: concept nodes, edge relation- ships (whether ARG# or modifier), and hierarchical bracketing notation. We do not require the parses contain variables (c.g. b in (b boy)) or to use rounded parentheses (e.g. ()), but we find that every generation with the three critical components also includes variables and parentheses. An example of a parse that receives a zero for the Basic Form is given in Figure 3. Figure 4 shows parse that is not fully up AMR standards (e.g. defanlt-OI~e.0 is not a standard format for concept representation) but that we credit for retaining the Basic Form.
Main Rel vs Top Node At times the main event relation in parse will also be the top node, but in AMR non-eventive relations (e.g., discourse mark- ers, conjunctions, modality) can instead take the top node position (e.g- warrant in Figure 2 below the top node contrast-Ol representing and). Main Rel disregards whether models can recognize these non-eventive relations, and focuses instead on mod- els' ability simply to recognize the main event of the sentence.
"Now, with the various "Occupy" movements, (or ought to be) obvious that great many people are less than enchanted with capitalism
Senieenu::
expression in idustrial (sic) democracies, well as in the countries into which capitalists from the industrial democracies go to invest."
(row), [with), [the], [various], ["Oceupy"], [roverents]]
[Lt), [bs], (or) [ought], [to]. (be)
Relaxation on exact match For all of our match- based criteria, we evaluate based on relaxed matches rather than exact match. For concept nodes, we ignore PropBank sense labels and inflec- tional variations, and allow matching based syn- onyms or otherwise differently-realized versions of the target concept. For example, serve-01 is con- sidered a match to serve, served, or serve-02- although the AMR gold parse for "She served as a president for ...") uses have-organizatioon-oleee
[obvious], [Uhak],
Dnam]l [peopte]
[ore], [Des]] [then], [enchanbed]
[with]. [oopitolian), [ond].
E. [its], [expression].
[in], [the], [industrial], [desocrucies] }, [os], [well], [os],
[irü, [tne], [countries],
[ Cinta], [shich], [copitalisess] [free],
[the], [industriol], [derocractes],
[oo]. [to]. [tnvest] 1 3
Figure 3: ChatGPT output receiving Basic Form score of 0

---

for "serve" (the standard AMR method for annotat- ing organizational role, occupation, or profession) we also give credit to generated nodes labeled as any variant of serve.
whether the instructions are too difficult for models to map correct oulputs. However, examination of successes across model generations indicates that no part of the NL setting instructions is fundamen- tally too difficult for the models to interpret and respond to. In the NL response setting shown in Figure 5b, the model is able to identify the primary event and arguments, and sort through and label modifiers for both the arguments and the event. Similar competence can be seen in the parse gener- ation setting (Figure Sa): GPT-4 correctly identifies the main event selection and major arguments and modifiers. For conjunctions between events, we see in Figure 6 that in both NL and parse settings the model is able to handle the central conjunction "and", and break down the two coordinated compo- nents accordingly even even breaking down the second coordinate into its own two component sub-events. On this basis we can have reasonable confidence that the prompts are sufficiently clear and inter- pretable for the models.
For edge match, the only critical distinction is that between arguments (c.g., ARGO, ARGI)" and modifiers (e.g.. :time, :purpose). Models receive credit for identifying argument as ARG even if the number is mismatched, and similarly receive credit for identifying a modifier as a modifier with- out regard to the semantic specificity.
We also relax exact match on AMR's named entity types (e.g., node concept organization in (o / organization :ame (n name :op1 "Morgan" :op2 "Sttanly"))), So long as the node concept is a reasonable match (e.g., company vs. organization), the models receive credit.
This use of relaxed match increases the need for expert manual annotation, but allows us credit general semantic competence beyond match to spe- cific AMR conventions.
D Metalinguistic prompt
Errors are abundant and diverse Though mod- els show the capacity in principle to handle any component of AMR information, examination of generated outputs shows that errors are abundant, diverse, and observable at every level of AMR structure. In addition to the illustration of out- put errors in Figure and Figure 2, we see that even in the largely successful example in Figure 6, the model has misidentified the primary event in the first coordinate- the main point should be that there was issue, not that the speaker boarded the train. Further, in the NL response, it has mis- takenly identified "next to us" luggage, rather than an argument of the pur event.
We use single instruction prompt for the meta- linguistic natural language output setting reported in $6. The prompt used is shown below:
(System: You are an expert linguistic annotator.) Sentence: <replaced with input sentence>
Identify the primary event of this sentence, and the predicate conesponding to that event. If there are multiple equally primary events connected by conjunction like "and", identify the conjunction, and then identify each of the primary events and their corresponding predicates.
For each primary event, identify the arguments of the event predicate, and identify the modifiers those arguments.
modifier of the
Then for each primary event, identify any addi- tional moditiers of that event.
In Figure 7 we see that the model is unable to identify the main event in either the parse gener- ation or the NL. setting. Though the main event relation is most appropriately identified as "imag- ine", in the generated parse the event "amaze" rises as the top event, and in the NL response output, "awaken" is identified the main event. Even if we ignore the non-eventive information captured by cause (arising from the discourse marker marker"thus") and possible-O1 (signalled by the modal "can"") con- cepts, the model fails to show sensitivity to the fact that "imagine my amazement" conveys the central information through which the content in the rest of the sentence is introduced.
E Example outputs and discussion
this section we illustrate with additional exam- ples some of the successes, failures, and overall patterns in the LLM outputs. Figures 5-7 show representative example outputs from GPT-4 in both parse generation (few-shot) and NL response set- tings (and an additional NL output example is in Figure 8). We highlight a number of points.
lnstruction-folloiinng success For the NL out- put setting in particular, the prompt instructions are somewhat complex, so it is worth considering
Finally, Figure 8 shows another more extreme failure in the NL response setting. Here the model
'We simply treat treat ARG#-of's as ARGs.

---

has mistakenly zeroed in on "will look wonderful" the primary event, rather than "it standout piece", and as a result it has defined arguments and modifiers in a variety nonsensical ways.
tence. However, these divergences often stem from the fact that the tasks in these two settings do differr For example, errors like the missing have-degree- 91 in Figure 5a an AMR device meant to struc- ture information extent-is not possible in the NL setting, as this level of structured detail is not re- quested in the prompt. Similarly, "sit down" in the parse in Figure 6a is split into two concept nodes, but this is level of semantic structuring that cannot be gauged in the NL responses.
Most reliable with core event triplets Though errors are diverse and fairly idiosyncratic, one trend that emerges is that models show the most reli- able performance with core event-argument triplets for individual verbs, most often corresponding to subject-ved vereett-vrrb-objett triplets. For example, in Fig- ure 5, in both formats the model clearly identifies the event "churn" and its arguments "the K-pop music sphere" and "newest catchy songs". In Fig- ure 6, in both formats the model correctly structures the event "board" with its arguments "we" and "train", and the event "put" with its arguments "we" and "the luggage". In Figure 7, in both formats the model correctly captures the event "awaken" with its arguments "T" and "an odd little voice". This suggests that the model has solid grasp core verb argument structure-or at least that corre- sponding subject-verb-ojectt triplets-and can reliably map this AMR form. However, beyond this core event structure, model performance be- comes substantially less reliable.
These parallels suggest that patterns successes and errors in our observed outputs are not simple idiosyncrasies of instruction or output format, but may indicate deeper patterns of strength and limita- tion in the models' capacity for semantic analysis.
F Automated format fixes
Our automated format fixes, used for two out of three of our settings for Smatch calculation on GPT- 4 few-shot parses, consist of the three simple rule- based fixes detailed below:
1. Keep only first full AMR, and ignore any sub- sequently generated content (e.g., if model generates multiple separate AMR structures for single sentence).
Parallel patterns between parse and NI. We ob- serve in Section 6 that there are similarities in the basic patterns of success and failure across LLM parse and NL outputs, and we highlight Figure 1 as an example. We see these parallels in Figures 5-7 well: for instance, as we have just discussed, the consistent success verb-argument triplets de- scribed above is seen in both parse and NL outputs for each example. More broadly, Figure we see that in both formats the model is successful on nearly the full AMR structure: it identifies the main verb ("churn"") and its arguments ("sphere" and "song") and modifiers ("vie for"), and it cap- tures semantic modifiers for the arguments in coherent manner (c.g.. "K-pop" is recognized as modifier of "music sphere"). In Figure 6, in both settings the model is successful in identifying "and" as top-level conjunction joining two events, but makes the error of choosing "boarding train" as the main event in the first coordinate of the struc- ture. Similarly, in Figure/ 7 both output settings capture core event structure of "awaken"", but miss out on the central event "imagine".
2. For retained AMR parse, delete any un- matched right parentheses.
3. Identify duplicates among concept variable names (e.g., "s" in sphere)), and replace with non-duplicates.
There are certainly divergences in output errors between different output formats for a given sen-

---

"The K- K-pop music sphere constantly churns out the newest catchy K-pop songs to vie for listeners' attention."
Gold Parse
(21 churn-81
churn-01
:ARGB (22 sphere
constantly
:domain (23 / music
:mod (z4 K-pop)])
:ARG1 (z5 song
sphere
sang
vie-01
:mod 24
:RG11 (z6 / have-degree-2
:ARG2 (27 / new-01 :ARG1 25)
catchy
attend-02
:ARG3 (28 most])
music
:mod (29 catchy))
have-deegree-91
pprrpose (z10 vie-01
person
:ARG8 z2
ARG1 (z11 attend-02
:RGG (212 person
K-pop
new-01
listen-01
:ARG8-of (z13 listen-011 01))))
frequency (z15 constant))
GPT-4 Generation
sphere :RRG1-of (n2 / music-81
churn-01
(Few-shot)
:ARG1 (k / K-pop :wiki "K-pop"
:nome (n4 / name :op1 "K-pop"))
sphere
song
vie-01
:mod (c2 constant))
:RGO-of (c / churn-81
:RRG1 (s2 song
calchy
attention
:mod :RGG--f (n / new-@1
music-01
:mod (c3 / catchy))
nave-degrae- -91
listener
:purpose (v vie-81
:ARGo s2
:RRG (a attention
K-pop
most
poss (1 listener)
:oo (p plural))))))
Legend: Red error;
Green correct Arguments:
Modifiers
Parsing generations Gold parse
Primary event: churns out
Predicate correspanding to the event: churns out Conjunction: None
Arguments of the event predicate:
Agent: The K-pop music sphere
Object: the newest catchy K-pop songs
Modifiers of the arguments:
Agent modifier: None
Object modifier: newest, catchy, K-pop
Additional madifiers of the primary event:
Manner: constantly
Purpose: vie for listeners' attention
(b) Response
Figure 5: A mostly good example

---

"There was no issue as we boarded the train, and we sat down and put the luggage next to us."
Gold Parse
(21 and :op1 (22 issue-0Z
:polarity
itime (23 board-01
Issure 01
and
:RGG (24 we)
:RG1 (25 train)))
:op2 (26 and
board-01
down 01
01
:op1 (27 / sit-doww-02
:RGG 24) 1op2 (28 put-81
luggage
:RGG
:RG1 (29 luggage)
:ARG (218 next-to
train
:ap1 z4))))
GPT-4 Generation (Few-shot)
(a and beard-01
and
:RGG (w / we)
train)
ISGUD
and
:ine (a3 as)
ARGG (i issue :polarity -))
(a2 and
:op1 (s / sit-01
board-01
sit-01
put 01
ARGI (d / dawn))
luggage
:op2 (p put-01
:RGG
next-to
ARGL (1 luggage)
train
US
:op1 (u us))))))
Legend:
Red error; Yellow acceptable; Green correct: Arguments ..... Modifiers
(a) Parsing generations vs. Gold parse
Primary Events: There are two equally primary events connected by the conjunction "and".
2. Primary Event 2: "we sat down" and "put the luggage next to us"
1. Primary Event 1: "we boarded the train"
Predicate corresponding to event (a): "sat down" Arguments of the event predicate (al:
Predicate corresponding to event 1: "boarded"
Arguments of the event predicate:
Subject: "we"
Subject: "we"
Modifiers of the arguments (a]: None
Object: "the train"
Modifiers of the arguments: None
Predicate carrasponding event (b): "put"
Arguments of the event predicate
Additional Modifiers
Subjectt (the "we" down") "
"There issue
Object: "the luggage"
Modifiers arguments (b):
(This modifier indicates that the boarding happened without any problem.)
Modifier Object: "next to us*
Additional Modifiers the event (a) and (b): None
(b) NL Response
Figure 6: A semi-good example

---

"Thus you can imagine my amazement, at sunrise, when I was awakened by odd little voice."
Gold Parse
(c / cause-81
ARG1 (p possible-01
imagine 01
possible--0 01
:RRG (i2 imagine-01
ARGG (y / you)
ARG1 (a / amaze-01
Amaze 01
ARG1 i)
:tmme sunrise
:time-of (w wake-01
ARGO (v / voice
:mod (o / odd)
ke-ee-1
:mod little))
odd
:AGG1 f)))))
GPT-4 Generation (Few-shot)
caUSe
Ca / amaze-01
:moe imaginal
imaginal
poss ole-01
:RGG (i / i)
:RGG (aZ / amake-01
:ARG i
you
amnizee 01
:RGG voice
: (o3 odd)
:oo (1 little))
sunrise
:mnnner (t / thus)
:time (s sunrise)))
Awvake-01
bdd
thus
litll
Legend: Red eroo; Yellow acceptable; Green correct; Arguments:
Modifiers
(a) Parsing generations vs. Gold parse
Primary event: "I was awakened"
Predicate corresponding the event: "was awakened"
Arguments of the event predicate:
Subject: "I"
Object: "by an odd little voice"
Modifiers of the arguments:
Subject: No modifiers
Object: "odd" (modifying "little voice"), "little" (modifying "voice")
Additional modifiers primary event:
Time: "at sunrise"
Reason: "Thus, you imagine my amazement"
(h) NL Response
Figure 7: A bad example

---

"With delicate crochet flower appliques throughout this top, it is a standout piece that will look wonderful either with the matching crochet skirt or paired with simple denim shorts let the top shine." "
Primary event: 'look wonderful*
(21 piece ddmncin (22 / 1L3
Predicate: "look"
:RRG--F (e3 Took-02
:AGGI (24 wonderful. 6E)
Arguments: "it" (referring to the top), "either with the matching crochet skirt paired with simple denim shorts"
pair-e1
:ARG1
(27
crochet)
-ARGI-uF notth--1)))
cop2 (x10)
poir-81 :ARG1 22
Modifiers of arguments: "delicate crochet flower appliques throughout" (modifying "it"/the top)
:ARGZ (21l shorts
:eod (212 dente)
:ARG1-of (213 stmolc-02)
(x14
Additioona modifiers of the event: "standout piece" (madifying the manner which the top will look wonderful), "to the top shine" (modifying purpose)
:ARGI (x15 shine-01
:ARG1 (x16 / top
:eod (217 (hie))3)))
seod Cx18 / standout))
(a) NL Response
(b) Gold Parse
Figure 8: Another bad example from responses


Reviews:
[{"review_id": "7qc9kQWwp6", "reviewer": "Reviewer_ZiDs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and thoughtful comments. Below we address the major points.\n\n**On test set size; concerns about bias.**\n\nWe appreciate the reviewer’s attention to this concern. As we argue in the submission, our aim here was to do finer-grained manual evaluation, to gain insights into model success/failures not available from simple larger-scale quantitative evaluations like smatch or SemBLEU. The trade-off is that we must work with smaller datasets, and there is a bit more subjectivity in the analysis. This choice aligns with the established tradition of qualitative error analysis on small numbers of examples – and we argue that our approach is well in keeping with recent precedent of qualitatively analyzing small numbers of examples to draw insights about model capabilities (e.g., [1]). This trend is influenced in part both by cost and by complexity of outputs, as are we. \n\nSo we absolutely agree with the importance of larger-scale quantitative evaluation, but we believe that there is also an important complementary place for smaller-scale fine-grained evaluation, and here we aim to contribute on that dimension.  \n\nWe will also note that we were careful in our sampling for our test sets, exerting controlled variation in terms of domain, length, and date of publication, to help to minimize potential biases. So while there is still non-zero risk of bias, we hope to minimize that risk while also taking advantage of the insights to be gained from close, fine-grained analysis. \n\n**On simplicity of prompts.**\n\nWe agree that more detailed prompt engineering could influence the results. We believe, however, that there is a strong argument as follows: if a prompt is sufficiently clear, then a failure to perform the task is simply a failure – reliance on particular prompt phrasing or structure is an indication of brittleness in model capabilities. So rather than trying to carefully engineer our prompts to accommodate model brittleness, our goal here is simply to design clear and fair prompts, and then assess model abilities on the task that those prompts describe. \n\n[1] Bubeck, Sébastien, et al. \"Sparks of artificial general intelligence: Early experiments with GPT-4.\" arXiv preprint arXiv:2303.12712 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths and main benefits of accepting this paper are as follow:\n\t1. The paper introduces a comprehensive evaluation framework for assessing the quality of Abstract Meaning Representation (AMR) parses generated by large language models (LLMs). This framework provides high-level and low-level criteria, enabling researchers to evaluate and compare the performance of various LLMs on the AMR task.\n2. The paper offers insights into the linguistic knowledge and capabilities of LLMs, specifically GPT-3, ChatGPT, and GPT-4, in the context of the AMR task. By analyzing the successes and limitations of these models, the paper sheds light on the strengths and weaknesses of current LLMs in understanding and representing abstract semantics.\n", "reasons_to_reject": "The weakness of this paper are as follow:\n\t1.  The paper utilizes a small test set, consisting of only 10 sentences for each dataset. This limited sample size raises concerns about the potential bias in the results. A larger and more diverse test set may provide a more robust evaluation of the LLMs' performance and help mitigate the impact of any dataset-specific biases.\n\t\n The prompts used in the paper are deemed relatively simple. A more sophisticated and refined prompt design could potentially yield different results. Exploring more nuanced and diverse prompts might provide deeper insights into the capabilities and limitations of LLMs in AMR parsing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores the question of whether LLMs have sufficient linguistic knowledge about the language by analyzing the successes and limitations of GPT-3, ChatGPT, and GPT-4 on the AMR task. The contributions of the paper are as follows:\n\nThe paper presents a comprehensive evaluation framework to assess the quality of AMR parses generated by LLMs. This framework includes high-level and low-level criteria for analyzing the structure, validity, and accuracy of the parses.\n\nThe paper provides analysis of LLMs' parse results with different prompts and conducts instance analysis and discussions, thereby analyzing the capabilities and limitations of LLMs in performing the AMR task. This analysis helps readers understand the linguistic abilities of the LLMs mentioned in the paper.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VIaqfTqRUz", "reviewer": "Reviewer_xh4S", "summary": "", "questions": "update to the below.  \n\nRegarding \"We intend here to investigate the capabilities that have been infused in LLMs via the pre-training process\", isn't it kind of strange to have to ask this question w/o  being able to know if the LLMs were trained on AMR material to begin with?  Suppose GPT4 or ChatGPT or whatever is trained on AMR data in some new iteration.  I could imagine someone then writing a paper saying \"big improvement in ChatGPT - it can now produce AMR parses almost as well as a human annotator\".  But I'm not sure that would be meaningful in any way.  \n\nIn general, thanks for the detailed response. I am upgrading my soundness and reproducibility scores. \n\n-----------------------------------------------------------------------------------------------\nA. The paper asks if LLMs can serve as \"an expert linguistic annotator\"?   But an \"expert linguistic annotator\" is not just someone with a fluent command of the language, but it also requires training in the annotation guidelines, etc.    While the LLMs seem to have knowledge, to some extent of the format of AMR, the don't seem to know a lot about AMR.   So perhaps it's doing as well as a typical person would do w/o knowing anything about AMR?  What would happen if instead of 5-shot, you just feed it all the training data, as if training an AMR parser.  One would hope that it could at least learn as well as a specialized parser.  But maybe not?\n\nB. The AMR events seem to encode some amount of word sense disambiguation by referring to particular propbank entries, such as \"churn-01\".  If my understanding of that is correct, did your criteria require getting the number correct?  Were there any cases where it got the wrong number for a word, and so was incorrect for that reason?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful assessment and detailed questions. Below we address individual points and inquiries.\n\n**On the lack of standard metrics like smatch.**\n\nWe appreciate the reviewer raising this point. As we argue in the submission, our aim here was to do finer-grained manual evaluation, to gain insights into model success/failures not available from simple larger-scale quantitative evaluations like smatch or SemBLEU. That said, we agree that inclusion of a more standard AMR metric like smatch will be a helpful addition to contextualize and support our claims! So we provide those numbers here for the reviewer’s reference, and we will happily add them to the revised version of the paper.\n\nFirst, baseline smatch scores for our test sets from the supervised AMRBART parser.\n\nSmatch – AMRBART:\n\n- AMR3: 0.78 \n- 2023: 0.79 \n- LPP: 0.75 \n\nNext, smatch scores for our reported GPT-4 outputs in the few-shot setting (the best output quality we found). Note that the model’s generated parses are often flawed to the point of smatch not being able to run, so we report three methods for obtaining scores: \n\n- [The scores in square brackets] reflect automated format fixes, then manually fix remaining formatting errors as needed.   \n- For (the scores in parentheses), we apply several automated format fixes, then for parses that still fail, replace them with single-node placeholder parse. \n- For {the scores in curly brackets}, we use no automated format fixes, and replace all failing parses with single-node placeholder parses\n\nSmatch – GPT-4 few-shot:\n\n- AMR3 : [ 0.51 ] ( 0.48 ) { 0.42 }\n- 2023 : [ 0.47 ] ( 0.40 ) { 0.26 }\n- LPP : [ 0.51 ] ( 0.47 ) { 0.41 }\n\nIt is clear that, based on these metrics, GPT-4 output quality is far below that of the supervised AMR parser. This helps to contribute to our overall point, while contextualizing with respect to a standard AMR metric, so we will be happy to add these results to the revised version. We appreciate the reviewer making this suggestion.\n\n**On confusing criterion descriptions.**\n\nThank you for pointing out this confusion. The top node in an AMR parse is not always an event concept. Concepts expressed by conjunction, discourse relations and modality can take the top node position in an AMR parse. For example, in a sentence like “However, they protested …”, “have-concession-91” would take the top node position above the “protest-01” node. The “TopNode” (L180) measure assesses whether the top node is correct (i.e., concept of have-concession-91 is accurate). The “MainRel” (L178) measure assesses whether the main event of the sentence surfaces as the top event relation (disregarding any potential non-eventive relations; e.g., credit is given if the concept “protest” does appear as the top event node in the tree). Therefore, MainRel is a more relaxed measure.  We will clarify these details in the updated version of the paper.\n\n**On model error analysis.**\n\nWe hope that the clarification above also helps partly in clearing up what categories of errors are indicated by penalties in each of our measures. We also very much appreciate the reviewer’s suggestion to further bin the errors into categories – this indeed would be ideal. However, what we have found is that making further generalizations at lower levels of granularity is not very feasible because the models’ errors are quite diverse and sentence-specific. We completely agree, however, that it would be beneficial to add more examples to help in showing what this variation looks like, so we provide a couple of illustrative examples here (please see below), and we will happily add these to the main paper and/or appendix in the revised version.\n\n**On providing more training data to LLMs.**\n\nThis is a good observation, and we agree that providing more training data has the potential to improve LLMs’ generation of AMRs. If the reviewer has in mind providing full training sets within the in-context learning setting, there is the practical limitation that inputting entire training datasets as in-context examples is not feasible due to context-length limits. \n\nIf the reviewer has in mind simply fine-tuning the models on AMR generation, we can assume that this would indeed have a positive effect on the generation quality. However, that type of experiment is somewhat outside the scope of our current submission, as the purpose of this submission is specifically to assess the knowledge and capabilities that these LLMs have for AMR generation *out of the box*, given zero-shot or few-shot in-context learning settings. We intend here to investigate the capabilities that have been infused in LLMs via the pre-training process, so investigating the impacts of fine-tuning constitutes a rather different (albeit also very interesting) research question.\n\n**On sense number.**\n\nNo, we never penalize the models for mismatch in sense number. We also do not penalize for generating the incorrect ARG number (e.g., “ARG0” vs “ARG1”) for argument counts (Table 2).\n\n\n**Model error examples. (continued from above)**\n\n> **Example 1**: *“Thus you can imagine my amazement , at sunrise , when I was awakened by an odd little voice”* (LPP sentence)\n\nDiscussion: We see below that the two generations diverge wildly from the gold parse. Across the board GPT4 picks up that “amaze” is an important relation, but it fails to appropriately capture that the top event here is “imagine” (among other problems). The few-shot parse receives 1 for Basic form, and 0 for TopNode, MainRel, and Accept measures.\n\n- GPT4 Zero-shot.\n\n```\n(s / sunrise\n       :manner (a / amaze-01\n                : experiencer (i2 / imagine-01\n                              :ARG0 (y2 / you)\n                              :ARG1 (i / I)\n                              :time (w / wake_up-01\n                                          :ARG0 (i)\n                                          :ARG1 (v / voice-01\n                                                      :mod (o / odd)\n                                                      :degree (l / little)\n                                                       :time-in (s / sunrise))))))\n```\n\n- GPT4 Few-shot.\n\n```\n(a / amaze-01 :mode imaginal\n      :ARG0 (i / i)\n      :ARG1 (a2 / awake-01\n            :ARG0 i\n            :ARG1 (v / voice\n                  :mod (o3 / odd)\n                  :mod (l / little))\n            :manner (t / thus)\n            :time (s / sunrise)))\n```\n\n- GOLD AMR\n\n```\n(c / cause-01\n      :ARG1 (p / possible-01\n            :ARG1 (i2 / imagine-01\n                  :ARG0 (y / you)\n                  :ARG1 (a / amaze-01\n                        :ARG1 (i / i)\n                        :time (s / sunrise\n                              :time-of (w / wake-01\n                                    :ARG0 (v / voice\n                                          :mod (o / odd)\n                                          :mod (l / little))\n                                    :ARG1 i))))))\n```\n\n> **Example 2**: *“Surgical removal of the entire tumor is almost impossible.”* (2023 sentence)\n\nDiscussion: The zero-shot parse below is almost entirely nonsensical and hallucinates the concept survive-01. Few-shot is somewhat improved in that remove-01 now rises correctly as the top event; but the role that surgery-01 and possible-01 play within remove-01 is unspecified. Additionally, remove-01 appears twice in the parse. These errors make it impossible to interpret the parse. The few-shot parse receives 1 for Basic form, 1 for MainRel, and 0 for TopNode and Accept.\n\n- GPT4 Zero-shot.\n\n```\n(s / survive-01 :polarity -\n      :arg1 (c / cell\n            :mod (c2 / cancer))\n      :arg0 (p2 / person :arg1-of (t / think-01\n                                      :arg1 (g / go-09))))\n```\n\n- GPT4 Few-shot.\n```\n(r / remove-01\n      :ARG1 (t / tumor\n            :mod (e / entire)))\n      (s / surgery-01)\n      (p / possible-01 :polarity -\n            :ARG1 (r2 / remove-01))\n      :degree (a / almost))\n```\n\n- GOLD ANNOTATION\n```\n(z1 / possible-01\n      :polarity -\n      :ARG1 (z2 / remove-01\n            :ARG1 (z3 / tumor\n                  :mod (z4 / entire))\n            :manner (z5 / surgery-01))\n      :mod (z6 / almost))\n```\n\n\n> **Example 3**: *“It didn't warrant such a reaction, but he was thoughtless.”* (AMR3)\n\nDiscussion: Again, below the zero-shot parse is entirely nonsensical but much improved in the few-shot setting. Even so, the few-shot parse still makes semantically nonsensical errors. For example, the gold parse captures the concept of “thoughtless” by using “think-01” in conjunction with negative polarity; GPT-4’s few-shot parse is both redundant (“think” and “thoughtless” both mentioned) and semantically lacking (negative polarity should accompany think-01). The few-shot parse receives 1 for Basic form, 1 for MainRel, 1 for TopNode, and 0 for Accept.\n\n- GPT4 Zero-shot.\n```\n(d / did-02\n  :mode \"-\"\n  :ARG1 (w / warrant-01\n    :ARG1 (r / react-01\n      :ARG0 (h / he))\n    :ARG2 (t / thoughtless)\n    :degree (s / such)) )\n```\n\n- GPT4 Few-shot.\n```\n(c / contrast-01\n      :ARG1 (w / warrant-01 :polarity -\n            :ARG1 (r / react-01\n                  :ARG2 (s / such)) \n            :ARG2 (i / it))\n      :ARG2 (t / think-01\n            :ARG1 (t2 / thoughtless\n                  :ARG1 (h / he))\n```\n\n- GOLD AMR\n\n```\n(c / contrast-01\n      :ARG1 (w / warrant-01 :polarity -\n            :ARG0 (i / it)\n            :ARG1 (t / thing\n                  :ARG2-of (r / react-01)\n                  :mod (s / such)))\n      :ARG2 (t2 / think-01 :polarity -\n            :ARG0 (h / he)))\n```\n\n\n> **Example 4**: *“Now, with the various \"Occupy\" movements, it is (or ought to be) obvious that a great many people are less than enchanted with capitalism and its expression in the idustrial (sic) democracies, as well as in the countries into which capitalists from the industrial democracies go to invest.”* (AMR3) \n\nDiscussion: This example below from ChatGPT zero shot demonstrates an example for badly formatted generation. This parse receives 0 across all measures:\n```\n[ [now], [with], [ [the], [various], [\"Occupy\"], [movements] ],\n [it], [be], [ (or] [ought], [to], [be) ],\n [ [obvious], [ [that], [ [a], [great], [many], [people] ],\n   [ [ [are], [less], [than], [enchanted] ],\n     [with], [capitalism], [and],\n     [ [its], [expression],\n       [ [in], [the], [industrial], [democracies] ],\n       [ [as], [well], [as],\n         [ [in], [the], [countries],\n           [ [into], [which], [capitalists], [from],\n             [the], [industrial], [democracies],\n             [go], [to], [invest] ] ] ] ] ] ] ] ]\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is a contribution to the question of what LLMs are capable of, in this case by querying how well they can simulate the role of a annotator.   \n\nUpdate: Another contribution is the definition of the evaluation framework.", "reasons_to_reject": "Update to the below: The authors do a good job in their response of clarifying the evaluation criteria.  The examples should go into an appendix.  I am uncertain though whether this is really a short paper.  The appendix already has a good amount of material related to the evaluation criteria, which is not a secondary aspect of the paper - it's one of its contributions.   \n\nI am somewhat concerned by the authors' comments \"We also very much appreciate the reviewer’s suggestion to further bin the errors into categories – this indeed would be ideal. However, what we have found is that making further generalizations at lower levels of granularity is not very feasible because the models’ errors are quite diverse and sentence-specific.\"    To me this indicates that it's really just too little data, if even a manual evaluation could not categorize the errors in some more abstract way.\n\nI really appreciated the smatch comparison with AMRBART - quite a contrast! \n\n----------------------------------\n1. The evaluation of the AMR parses seems very ad-hoc. I am not very familiar with AMR work, but just a quick check turns up work on evaluation metrics for AMR (e.g. SEMBLEU, Smatch).  Of course, with only 30 sentences, a quantitative measure using those may not be very revealing.  The authors state that they \"prioritize close, fine-grained analyses of the successes and errors of the models in capturing these meaning structures, in order to provide detailed insights into model capabilities.\"  But it is very hard to actually get an idea of what the LLMs are getting wrong from the criteria they give.  It gets the top node wrong 20-40%, \"reflecting routine failures to incorporate clausal and discourse relations in the top node\" - such as what?  There is one example in the main text, and three more in the appendix, but with only 30 sentences, and the promise of \"fine-grained analyses\", it needs some sort of better accounting of what it's getting wrong - e.g. # of sentences due to coordination mistake, # to due to subordinate/main clause confusion, etc.   So the paper is suggestive, but not quite there yet.\n\n2. Connected to that, the description of the criteria is confusing.  The paragraph at lines 172-183 is supposed to correspond, I think, to the columns in Table 1.   Does (178-9) \"select the correct event as the main event concept\" mean just \"select the main event concept correctly\"?   This is a \"more relaxed criterion\" (202-203)  but it's not obvious why. The appendix describes this as \"We evaluate whether the main event of the concept is surfaced as the top event concept in the tree\", which I was also confused by.  I think a couple of full examples would really help - e.g. here is an example of a correct main event relation but incorrect top node.", "questions_to_authors": "update to the below.  \n\nRegarding \"We intend here to investigate the capabilities that have been infused in LLMs via the pre-training process\", isn't it kind of strange to have to ask this question w/o  being able to know if the LLMs were trained on AMR material to begin with?  Suppose GPT4 or ChatGPT or whatever is trained on AMR data in some new iteration.  I could imagine someone then writing a paper saying \"big improvement in ChatGPT - it can now produce AMR parses almost as well as a human annotator\".  But I'm not sure that would be meaningful in any way.  \n\nIn general, thanks for the detailed response. I am upgrading my soundness and reproducibility scores. \n\n-----------------------------------------------------------------------------------------------\nA. The paper asks if LLMs can serve as \"an expert linguistic annotator\"?   But an \"expert linguistic annotator\" is not just someone with a fluent command of the language, but it also requires training in the annotation guidelines, etc.    While the LLMs seem to have knowledge, to some extent of the format of AMR, the don't seem to know a lot about AMR.   So perhaps it's doing as well as a typical person would do w/o knowing anything about AMR?  What would happen if instead of 5-shot, you just feed it all the training data, as if training an AMR parser.  One would hope that it could at least learn as well as a specialized parser.  But maybe not?\n\nB. The AMR events seem to encode some amount of word sense disambiguation by referring to particular propbank entries, such as \"churn-01\".  If my understanding of that is correct, did your criteria require getting the number correct?  Were there any cases where it got the wrong number for a word, and so was incorrect for that reason?", "paper_topic_and_main_contributions": "This paper investigates the capability of GPT-3, ChatGPT, and GPT-4 to produce AMR parses for 30 test sentences, using both zero-shot and five-shot scenarios.  Four evaluation criteria are used in all cases, with an additional four for the five-shot evaluations.  The main contribution is showing that while the LLMs can generate some aspects of a gold AMR parse, it also frequently makes errors of various types.   An interesting aspect of the paper (although in the appendix) is that they also included recent text, for which they created the gold AMR parses, to check whether the LLM's had been learning from earlier officially released material.  They found no significant (informally speaking) difference with such cases.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "References to AMR evaluation, as noted above", "Typos_Grammar_Style_and_Presentation_Improvements": "generally good aside from the comments above.  Problem with the sentence at 544-547 - \"Because *an* AMR parse ...valid parse*s*\""}, "extra_text": {}}, {"review_id": "meeemx5h0P", "reviewer": "Reviewer_4W3E", "summary": "", "questions": "1) How was the acceptability of the generated AMR graph evaluated?\n\n2) Although the generated ARM graphs have errors, is not it easier to edit a partial AMR graph than annotate one from scratch?\n\n3) Regarding the errors produced by the language models, is there a pattern of errors produced by them? \n\n4) Is it possible to apply post-processing to solve or mitigate them?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful assessment and useful questions. Below we address the questions in turn.\n\n**On acceptability metric details.**\n\nFor our overall acceptability measure, an AMR expert among the authors assessed whether each parse could be a valid representation of the sentence meaning, based on the AMR annotation guidelines, regardless of match to the gold annotation. This was intended to give fairer credit to model outputs – none of the models’ generated parses managed to perfectly match the gold annotation, but it was possible that some parses may still accurately represent the meaning of the sentence, with some annotation differences from the gold parse. So this measure used expert assessment to judge parse validity in this broader sense. These assessments also generally forgave minor structural/formatting errors, as long as a correct semantics could be interpreted. \n\nWe thank the reviewer for flagging this confusion, and we will make this description clearer in the revised version.\n\n**On making comparisons using an AMR metric.**\n\nThank you for raising this important suggestion. As we argue in the submission, our aim here was to do finer-grained manual evaluation, to gain insights into model success/failures not available from simple larger-scale quantitative evaluations like smatch or SemBLEU. However, we agree that inclusion of a more standard AMR metric like smatch will be helpful to contextualize and support our claims! We provide those results here, and we will happily add them to the revised version of the paper.\n\nFirst, baseline smatch scores for our test sets from the supervised AMRBART parser.\n\nSmatch – AMRBART:\n\n- AMR3: 0.78 \n- 2023: 0.79 \n- LPP: 0.75 \n\nNext, smatch scores for our reported GPT-4 outputs in the few-shot setting (the best output quality we found). Note that the model’s generated parses are often flawed to the point of smatch not being able to run, so we report three methods for obtaining scores: \n\n- [The scores in square brackets] reflect automated format fixes, then manually fix remaining formatting errors as needed.   \n- For (the scores in parentheses), we apply several automated format fixes, then for parses that still fail, replace them with single-node placeholder parse. \n- For {the scores in curly brackets}, we use no automated format fixes, and replace all failing parses with single-node placeholder parses\n\nSmatch – GPT-4 few-shot:\n\n- AMR3 : [ 0.51 ] ( 0.48 ) { 0.42 }\n- 2023 : [ 0.47 ] ( 0.40 ) { 0.26 }\n- LPP : [ 0.51 ] ( 0.47 ) { 0.41 }\n\nIt is clear that, based on these metrics, GPT-4 output quality is far below that of the supervised AMR parser. We will be happy to add these results to the revised version, to further support our points while contextualizing with respect to a standard AMR metric. We appreciate the reviewer making this suggestion.\n\n**On being easier to edit partial AMRs rather than write from scratch.**\n\nCertainly! This is a good observation. However, this is somewhat orthogonal to the point of our paper – we aren’t arguing that these noisy/flawed AMR parses could not have any use. If desired, existing supervised AMR parsers can already enable that type of parse-then-revise workflow (more effectively, based on smatch above). Our point here is more focused on analyzing and reporting LLMs’ capabilities in this domain.  \n\n**On error patterns and post-processing to mitigate errors.**\n\nThough we can make some generalizations about the errors that we observe in model generations, the fact is that the errors are quite diverse and numerous – so while we certainly expect post-processing to be able to improve the parses, a key challenge is that the necessary fixes tend to be fairly sentence- or parse-specific. To help in illustrating this, and in response to other helpful comments indicating that having more examples would be beneficial, we are including a number of examples with discussion – we have included these in the response to Reviewer xh4S above. Please see that response for some examples to illustrate the kind of variation and idiosyncrasy we observe in these generations and their errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper well-written, strucured, and easy to follow\n\n- Paper presents an interesting qualitative analysis of the ability of language models to produce AMR structures", "reasons_to_reject": "- Lack of quantitative analysis to check the accuracy of language models\n\n- Lack of detail in the qualitative analysis to conclude the acceptability of language models", "questions_to_authors": "1) How was the acceptability of the generated AMR graph evaluated?\n\n2) Although the generated ARM graphs have errors, is not it easier to edit a partial AMR graph than annotate one from scratch?\n\n3) Regarding the errors produced by the language models, is there a pattern of errors produced by them? \n\n4) Is it possible to apply post-processing to solve or mitigate them?", "paper_topic_and_main_contributions": "The paper presents an analysis of the GPT-3, GPT-4, and ChatGPT focusing on Abstract Meaning Representation formalism. The analysis investigates if these tools can serve as an expert linguistic annotator for AMR. This investigation analyzed two settings: the direct production of AMR structures based on examples of zero- and few-shot and the indirect partial reconstruction of AMR via metalinguistic natural language queries.\n\nThe paper is well-writtten, structured, and easy to follow. The main contribution is a qualitative analysis of the ability of language models to generate AMR structures from zero- and few-shot settings. This investigation is interesting, as language models may be used to produce AMR corpora for low-resource languages. However, the paper concludes the analyzed models have virtually 0% success in producing fully accurate parses. From this conclusion, I was curious, how was the acceptability of the generated AMR graph evaluated?\n\nAlthough the generated ARM graphs have errors, is not it easier to edit a partial AMR graph than annotate one from scratch?\n\nRegarding the errors produced by the language models, is there a pattern of errors produced by them? Is it possible to apply post-processing to solve or mitigate them?\n\nFinally, in addition to the performed analysis, I expected to see a comparison between the generated graph by the language models and the gold graphs using an AMR metric.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the LLMs’ linguistic knowledge in serving as an expert linguistic annotator. In particular, the paper analyzes the capabilities of the GPT-3, ChatGPT, and GPT-4 models in generating 
the Abstract Meaning Representation (AMR) of sentences. 

Pros: 
* All reviewers have found that the paper is very well-written and easy to follow.
* The paper presents a thorough analysis of the quality of generated AMRs from  GPT-3, ChatGPT, and GPT-4, assessing the strengths and weaknesses of current LLMs. 
* The finding that “these models out-of-the-box can accurately identify some core aspects of semantic structure, but there remain key limitations in their ability to support fully accurate semantic analyses or parses” will be valuable to the practitioner interested in semantic parsing and analysis of texts.

Cons:
* The reviewers have rightly raised concerns that the analysis has been done with a fairly small dataset (10 sentences per dataset), and this could lead to bias in the results. There is a lack of large-scale quantitative analysis in the paper. 
* * I find the authors’ rebuttal to be convincing that we should allow space for smaller-scale yet comprehensive fine-grained evaluations at ACL venues. As a short paper, the analysis presented in the paper accounts for a solid contribution and will be valuable to the community. 
* Other concerns, such as simplicity of prompts etc, were addressed adequately in the authors response.
Decision: Accept-Findings