On the Role of Semantic Proto-roles in Semantic Analysis: What do LLMs know about agency?
Authors: Elizabeth Spaulding, Shafiuddin Rehan Ahmed, James Martin
Accepted at: acl25

On the Role of Semantic Proto-roles in Semantic Analysis:
What do LLMs know about agency?

Elizabeth Spaulding1 and Shafiuddin Rehan Ahmed2 and James H. Martin1
1University of Colorado Boulder, 2Center for Advanced AI, Accenture
elizabeth.spaulding@colorado.edu, shafiuddin.r.ahmed@accenture.com, james.martin@colorado.edu

Abstract
Large language models (LLMs) are increasingly used in decision-making contexts, yet their ability to reason over event structure—an important component in the situational awareness needed to make complex decisions—is not well understood. By operationalizing proto-role theory, which characterizes agents via properties such as instigation and volition and patients via properties such as change of state, we examine the ability of LLMs to answer questions that require complex, multi-step event reasoning. Specifically, we investigate the extent to which LLMs capture semantic roles such as “agent” and “patient” through zero-shot prompts, and whether incorporating semantic proto-role labeling (SPRL) context improves semantic role labeling (SRL) performance in a zero-shot setting. We find that, while SPRL context sometimes degrades SRL accuracy in high-performing models (e.g., GPT-4o), it also uncovers an internal consistency between SPRL and SRL predictions that mirrors linguistic theory, and provides evidence that LLMs implicitly encode consistent multi-dimensional event role knowledge. Furthermore, our experiments support prior work showing that LLMs underperform human annotators in complex semantic analysis.

1 Introduction
Philosophies of ethics as far back as Aristotle (Freeland, 1985; Shaver, 1985; Weiner, 1995) define moral agents as entities which can be held responsible for their acts, and moral patients as entities which experience the effects of those acts, and are thus objects of concern. In linguistics, psychology, and neuroscience, agency and patiency are studied to understand how individuals perceive, represent, and communicate about actions and their consequences. With LLMs increasingly used for decision-making, further investigation into their capacities for moral reasoning and judgment has been called for (Jiang et al., 2025).

Semantic proto-role labeling
verb-invariant, graded
[The girlSENTIENT] ate fruit.
[The girlINSTIGATION] ate fruit.
[The girlVOLITION] ate fruit.
infer from properties

Semantic role labeling
verb-specific, categorical
<table>
  <tr>
    <th>Roleset ID</th>
    <th>eat.01</th>
  </tr>
  <tr>
    <td>ARGO</td>
    <td>eater</td>
  </tr>
  <tr>
    <td>ARG1</td>
    <td>meal</td>
  </tr>
</table>

[The girlAGENT] ate fruit. [The girlEATER] ate fruit.

Figure 1: Semantic proto-role labeling captures broad properties across verbs, while semantic role labeling captures verb-specific role information. Evidence from neuroscience suggests that both systems are employed complementarily in human language processing.

In response, we examine LLMs through the relationship between semantic proto-role labeling (SPRL) and semantic role labeling (SRL), two sentence-level semantic tasks. Traditional SRL systems rely on fixed sets of categorical roles such as “agent”, “patient,” and “instrument.” Proto-role theory (Dowty, 1991) challenges this view by proposing that the participants of an event are best characterized by a set of graded properties. Dowty’s prototypical agent (“proto-agent”) exhibits properties such as volition, sentience, causality, and movement, whereas proto-patients are more likely to undergo a change of state or be affected by the action.

Additionally, while SRL requires the entities in a sentence to be labeled with verb-specific semantic roles (agent of hitting/hitter, patient of hitting/hittee), SPRL properties are invariant across verbs. Neuroscience research has shown evidence that both verb-specific and verb-invariant event roles are encoded in the brain, and could play complementary roles in processing (Frankland and Greene, 2020).

SPRL and SRL, then, can provide a benchmark for an LLM’s semantic reasoning capabilities as compared to a human’s, a proxy for an LLM’s situational awareness, and scaffolding for fine-grained interpretability of moral judgments. We ground our
experiments in the following research questions: (1) does providing LLMs with semantic proto-role labels as context for semantic role labeling result in accuracy gains compared to no SPRL context?, (2) do errors in semantic role labeling correlate with errors in semantic proto-role labeling?, and (3) how does an LLM perform as the SRL and SPRL tasks increase in complexity? Our results enable us to make the following contributions:

• SPRL context often degrades SRL performance, especially in the highest-performing language model we prompted (GPT-4o), but sometimes provides a non-negligible boost to performance in smaller models.
• SRL errors do not overwhelmingly co-occur with SPRL errors, but SRL errors do overwhelmingly co-occur with SPRL properties that were deemed not applicable to the argument at annotation time.
• GPT-4o performs badly on the prompt variant with the most steps per test instance: an end-to-end SRL and SPRL pipeline. This suggests that the findings that GPT cannot perform complex and detailed semantic analysis of event roles (Ettinger et al., 2023; Bonn et al., 2024) generalize to larger datasets. Additionally, smaller LMs almost always performed better on less complex tasks (e.g. prompts that required only a single token for a response) versus more complex tasks (e.g. prompts that required multiple responses be produced in JSON format). However, GPT-4o performed better in a prompt variant that elicited roles for all arguments for a single predicate, versus only one argument and predicate, suggesting that some added complexity helps to further contextualize a task and boosts performance.

Additionally, while our findings did not produce evidence for positive prompt-level interactions between SRL and SPRL, we found robust evidence that GPT-4o classifies PropBank-style semantic roles consistently with Dowty’s proto-role theory, suggesting that it encodes a hierarchy of event role structure similar to what has been proposed by linguistic theory. There is also faint evidence of such structures emerging in a much smaller LM, the 3b-parameter version of Llama 3.2.

2 Background and Related Work

PropBank and SRL PropBank (Kingsbury and Palmer, 2002; Gildea and Palmer, 2002; Palmer et al., 2005) is a lexical resource that separates verbs into coarse senses, each with a set of semantic roles (the “roleset,” e.g. “eat.01” in Fig. 1). While the 6 core roles are verb-specific, ARG0 usually corresponds to a prototypical agent (the argument which makes the action happen) and ARG1 corresponds to patient (the argument which receives the action). Thus, these arguments can be generalized to an extent across verbs and verb senses. Semantic role labeling (SRL) is a classification task that can use PropBank as its vocabulary, so that a system learns to label sentences for their predicates, arguments, and semantic role labels. Importantly, a proficient PropBank-style SRL system can consistently label the role of verb participants across different syntactic alternations of the same verb (e.g., “I broke [the computer]_{ARG1}” and “[The computer]_{ARG1} broke”). Various neural methods, including graph-based (Zhou et al., 2022; Liu et al., 2023) and syntax-aware (Fei et al., 2021; Zhang et al., 2022) approaches, have achieved impressive performance, and SRL has been leveraged for its promise in multimodal understanding (Sadhu et al., 2021; Bhattacharyya et al., 2023) and situation and narrative modeling (Ash et al., 2024; Balashankar et al., 2023).

SPRL Proto-role theory (Dowty, 1991) offers an alternative to categorical role inventories by focusing on the finer-grained properties of a prototypical agent (e.g., volition, sentience) and properties of a prototypical patient (e.g., change of state, change of possession). For example, in the sentence “The boy threw a rock,” categorical role inventories assign argument “boy” the role Agent, and argument “rock” the role Patient. Work on decompositional semantics1 has formulated the task of semantic proto-role labeling (SPRL) as the assignment of 14 different binary properties to arguments (Reisinger et al., 2015). Semantic proto-role labeling assigns volition, sentience, and instigation to “boy” and change of state, change of location, and was used to “rock.” (All SPR labels, as well as their definitions, can be found in Table 7.) Previous work in SPRL has explored fine-tuned language models, attention-based ensembling, and other neural approaches for label classification (Teichert et al., 2017; Rudinger et al., 2018; Opitz and Frank, 2019; Tenney et al., 2019; Stengel-Eskin et al., 2020, 2021; Spaulding et al., 2023). Sadeddine et al. (2024) offer a comprehensive survey of datasets, parsers, and applications of

1http://decomp.io/
<table>
  <tr>
    <th rowspan="2">Predictions</th>
    <th colspan="3">instigated</th>
    <th colspan="3">volition</th>
    <th colspan="3">aware</th>
    <th colspan="3">sentence</th>
    <th colspan="3">chg-loc</th>
  </tr>
  <tr>
    <th>T</th><th>F</th><th>N/A</th>
    <th>T</th><th>F</th><th>N/A</th>
    <th>T</th><th>F</th><th>N/A</th>
    <th>T</th><th>F</th><th>N/A</th>
    <th>T</th><th>F</th><th>N/A</th>
  </tr>
  <tr>
    <td>ARG0</td><td>90.5</td><td>21.2</td><td>4.9</td>
    <td>77.5</td><td>10.7</td><td>3.3</td>
    <td>64.8</td><td>9.1</td><td>4.1</td>
    <td>62.7</td><td>10.8</td><td>3.5</td>
    <td>29.6</td><td>32.1</td><td>26.5</td>
  </tr>
  <tr>
    <td>ARG1</td><td>6.0</td><td>58.1</td><td>50.7</td>
    <td>15.2</td><td>66.3</td><td>51.7</td>
    <td>23.5</td><td>69.2</td><td>52.9</td>
    <td>25.6</td><td>65.7</td><td>49.8</td>
    <td>53.2</td><td>47.8</td><td>42.8</td>
  </tr>
  <tr>
    <td>ARG2</td><td>2.0</td><td>14.6</td><td>29.6</td>
    <td>4.6</td><td>16.5</td><td>29.8</td>
    <td>7.8</td><td>15.5</td><td>28.8</td>
    <td>7.8</td><td>16.6</td><td>31.1</td>
    <td>10.5</td><td>14.0</td><td>20.5</td>
  </tr>
  <tr>
    <td>ARG3-5</td><td>0.1</td><td>1.1</td><td>2.0</td>
    <td>0.2</td><td>1.3</td><td>2.1</td>
    <td>0.3</td><td>1.4</td><td>2.0</td>
    <td>0.3</td><td>1.4</td><td>2.3</td>
    <td>0.9</td><td>1.0</td><td>1.5</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Predictions</th>
    <th colspan="3">chg-poss</th>
    <th colspan="3">chg-state</th>
    <th colspan="3">created</th>
    <th colspan="3">destroyed</th>
  </tr>
  <tr>
    <th>T</th><th>F</th><th>N/A</th>
    <th>T</th><th>F</th><th>N/A</th>
    <th>T</th><th>F</th><th>N/A</th>
    <th>T</th><th>F</th><th>N/A</th>
  </tr>
  <tr>
    <td>ARG0</td><td>7.1</td><td>33.3</td><td>27.7</td>
    <td>8.3</td><td>36.0</td><td>32.6</td>
    <td>0.9</td><td>35.0</td><td>11.1</td>
    <td>5.7</td><td>34.7</td><td>11.1</td>
  </tr>
  <tr>
    <td>ARG1</td><td>79.2</td><td>47.2</td><td>42.6</td>
    <td>70.8</td><td>43.8</td><td>37.8</td>
    <td>57.6</td><td>47.0</td><td>42.2</td>
    <td>78.7</td><td>46.3</td><td>42.2</td>
  </tr>
  <tr>
    <td>ARG2</td><td>10.9</td><td>13.7</td><td>19.6</td>
    <td>16.2</td><td>13.7</td><td>19.3</td>
    <td>33.6</td><td>11.8</td><td>33.3</td>
    <td>10.1</td><td>12.7</td><td>33.3</td>
  </tr>
  <tr>
    <td>ARG3-5</td><td>2.1</td><td>0.9</td><td>1.4</td>
    <td>0.8</td><td>1.1</td><td>1.4</td>
    <td>0.7</td><td>0.9</td><td>2.2</td>
    <td>1.9</td><td>0.9</td><td>2.2</td>
  </tr>
</table>

Table 1: % of predicted properties that co-occur with GPT-4o’s predicted ARGn on the Ontonotes dataset, for the All-args prompt variant with “n/a” SPRL responses allowed. Denominator for percentages is the total number of arguments that property and value were assigned by the model. Columns do not add up to 100 because the model occasionally outputs an SRL prediction that is not any of the numbered ARGs.

tasks such as SRL and SPRL.

LLMs The largest and most successful LLMs have shown mastery in what Mahowald et al. (2024) call formal linguistic competence—the knowledge of rules and statistical regularities of language—but often fail in functional linguistic competence, or the ability to successfully apply language to real-world situations. Relevantly, functional linguistic competence includes robust situation modeling: the ability to keep track of entities, the relations between them, and their participation in various events across time. In the domain of event roles and relations, Bonn et al. (2024) assess the capabilities of GPT-3 and GPT-4 to do PropBank annotation and find that both perform far below reported human IAA (Bonial et al., 2017). Ettinger et al. (2023) find that GPT-3 and GPT-4 cannot produce AMR (Abstract Meaning Representation; Banarescu et al. 2013) annotations more complex than a core event structure in subject-verb-object form. Both studies, because of the time-intensive manual analysis required, reported results on very small sample sizes. Our study addresses similar questions at a larger sample size and provides evidence that the findings of Ettinger et al. (2023) and Bonn et al. (2024) generalize to larger data.

Previous work has found that pre-trained language models like BERT implicitly learn semantic properties, including SPRL (Tenney et al., 2019; Kuznetsov and Gurevych, 2020). Stengel-Eskin and Van Durme (2022) investigate the effect of providing SPRL context in prompts eliciting interpretations of sentences involving subject control clauses. They find that large language models contain SPR property knowledge but do not directly apply it to all situations. To date, however, there has been no fine-grained study of the capabilities of the most recent LLMs on both SPRL and SRL, and the relation between them.

3 Data

We evaluate SRL on two datasets using the standard test splits: the first one (SPR1 (Reisinger et al., 2015), \( n = 1054 \) predicate-argument pairs) is small, but contains SPRL annotations, which allows a fine-grained analysis of the effects of each proto-role property. The second dataset (an SRL test split from Ontonotes\(^2\), \( n = 44609 \) predicate-argument pairs) is larger but lacks gold-standard SPRL annotations. We use Ontonotes to ascertain whether our analysis from SPR1 might generalize to a larger dataset with a more diverse set of topics and semantic and syntactic constructions.

SPRL results are typically reported on two English-language datasets: SPR1 and SPR2 (White et al., 2016). SPR1 contains 4,912 Wall Street Journal sentences from PropBank annotated by a single annotator based on a set of 16 proto-role properties. 9,738 arguments were annotated for the likelihood (on a Likert scale from 1 to 5) that a property holds for that argument. SPR2 contains 2,758 English Web Treebank (Bies et al., 2012) sentences annotated for a smaller set of 14 properties using a revised, streamlined protocol with two-way

\(^2\)https://github.com/propbank/propbank-release
redundancy in annotation. The sets of properties in SPR1 and SPR2 are slightly different from one another, and neither maps one-to-one to Dowty’s original 10 proto-role entailments. Table 7 shows the properties that were annotated for SPRL. In SPR2, reported inter-annotator agreement is generally acceptable for each property, with White et al. (2016) reporting a Spearman’s rank correlation coefficient of 0.617 and Spaulding et al. (2023) reporting Cohen’s \( \kappa \geq 0.64 \) for all properties when computed over the binarized labels, with an average \( \kappa = 0.75 \).

Previous work (Opitz and Frank, 2019; Rudinger et al., 2018; Teichert et al., 2017; Tenney et al., 2019), formulates SPRL as a 16 (SPR1) or 14 (SPR2) way multi-label binary classification problem and map Likert labels {1, 2, 3} to 0, and {4, 5} to 1. Previous work additionally maps judgments labeled “inapplicable” to 0. In our work, we use standard train/dev/test splits provided in the data where applicable: wherever we report results, those are based off of test sentences.

4 Experimental Setup

We experiment on a variety of open-weight models ranging from 3 billion to 8 billion parameters: Llama 2 and 3 (Touvron et al., 2023; Grattafiori et al., 2024), Qwen2.5 (Qwen et al., 2025), and Tulu 3 (post-trained using Llama 3.1 as a foundation; Lambert et al. 2025). We also experiment on open-weight, open-data models OLMo 1 and 2 (Groeneveld et al., 2024; OLMo et al., 2025) and Pythia (Biderman et al., 2023). Because the OLMo and Pythia authors release their data to the public, we could confirm that those models were not trained on data contaminated with the data we test on (Elazar et al., 2024). Finally, we perform a broad range of experiments that require a longer context window and a more complex array of tasks on GPT-4o (OpenAI, 2024).

We evaluate on a variety of prompt templates with and without certain context. Our main focus is the model’s ability to reason over the semantics of a sentence in a zero-shot setting, and we use PropBank and proto-role theory to validate whether its reasoning is consistent. We utilize three different zero-shot prompt templates for SRL, and one prompt for SPRL alone:

• Pipeline: The model is sequentially prompted through six SRL components: predicate span identification, predicate sense disambiguation, argument span identification, (optionally) semantic proto-role labeling, and semantic role labeling. Previous output is provided as context in later prompts.
• All-args-per-prompt: The model receives oracle-provided predicate spans, sense, argument spans, and roleset details, and must output a JSON with both SPRL and SRL labels for all constituents of a single predicate.
• One-arg-per-prompt: The task is simplified by requiring the model to output only a single semantic role label for one argument using optional gold SPRL context.
• SPRL-annotate: The model is given a true/false prompt for each argument-predicate-property triplet, similar to the human annotation protocol used in SPR1/2 (Table 7).

We evaluate all prompt variants exhaustively on GPT-4o as it is a large-scale, powerful language model capable of advanced reasoning. We also evaluate smaller LMs on SPR1 using the One-arg prompt variant.3 We additionally vary the context and instructions provided to the model in the following ways:
• SPRL context: Depending on the prompt, the model can receive SPRL context either through its previous predictions (Pipeline), gold labels (One-arg), or explicit instructions to generate SPRL output alongside SRL output (All-args). We additionally vary the specific SPRL properties included in each prompt: instigation and change-of-location, volition and change-of-state, all SPRL properties in both SPR1 and SPR2 (i.e., the intersection of the two property sets; SPR1\( \cap \)SPR2), and all SPRL properties in SPR1.
• Oracle predicate spans, predicate senses, and argument spans: All prompt variants (except Pipeline) provide the model with oracle predicate spans, senses, and argument spans. In Pipeline, the model must use its own predictions.
• Allowing “N/A” as a SPRL response: In some All-args experiments, the model is explicitly instructed to label inapplicable properties as “n/a” rather than restricting the response to True/False.

3For All-arg and Pipeline prompts, due to length and complexity, early experiments on smaller LMs were computationally expensive and yielded poor results. Thus, GPT-4o is the main focus of those prompt experiments. Additionally, the token limit for the One-arg and SPRL-annotate prompts was very short for the smaller LMs for computational efficiency.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">SPR1</th>
    <th colspan="2">SPR2</th>
  </tr>
  <tr>
    <th>Micro</th>
    <th>Macro</th>
    <th>Micro</th>
    <th>Macro</th>
  </tr>
  <tr>
    <td>Qwen2.5 (7b) (I)</td>
    <td>4.9</td>
    <td>4.6</td>
    <td>4.9</td>
    <td>4.7</td>
  </tr>
  <tr>
    <td>Pythia (6.9b)</td>
    <td>5.6</td>
    <td>5.5</td>
    <td>16.0</td>
    <td>15.2</td>
  </tr>
  <tr>
    <td>Llama 2 (7b)</td>
    <td>37.7</td>
    <td>32.2</td>
    <td>43.8</td>
    <td>37.8</td>
  </tr>
  <tr>
    <td>OLMo 2 (7b) (I)</td>
    <td>49.1</td>
    <td>40.1</td>
    <td>46.1</td>
    <td>39.6</td>
  </tr>
  <tr>
    <td>Llama 3.1 (8b) (I)</td>
    <td>47.2</td>
    <td>41.8</td>
    <td>54.0</td>
    <td>48.2</td>
  </tr>
  <tr>
    <td>Tulu 3 (8b)</td>
    <td>47.4</td>
    <td>39.8</td>
    <td>54.9</td>
    <td>47.6</td>
  </tr>
  <tr>
    <td>Llama 3.2 (3b) (I)</td>
    <td>48.2</td>
    <td>42.1</td>
    <td>55.6</td>
    <td>49.3</td>
  </tr>
  <tr>
    <td>OLMo 2 (7b)</td>
    <td>50.8</td>
    <td>45.9</td>
    <td>60.7</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>61.3</td>
    <td>51.9</td>
    <td>68.0</td>
    <td>57.2</td>
  </tr>
  <tr>
    <td><i>n/a skipped in eval</i></td>
    <td>9.6</td>
    <td>7.2</td>
    <td>7.3</td>
    <td>7.1</td>
  </tr>
  <tr>
    <td>Qwen2.5 (7b) (I)</td>
    <td>11.1</td>
    <td>10.5</td>
    <td>23.4</td>
    <td>22.2</td>
  </tr>
  <tr>
    <td>Llama 2 (7b)</td>
    <td>51.5</td>
    <td>46.1</td>
    <td>53.8</td>
    <td>46.8</td>
  </tr>
  <tr>
    <td>OLMo 2 (7b) (I)</td>
    <td>70.2</td>
    <td>58.5</td>
    <td>58.9</td>
    <td>51.2</td>
  </tr>
  <tr>
    <td>Llama 3.1 (8b) (I)</td>
    <td>71.6</td>
    <td>65.2</td>
    <td>68.5</td>
    <td>62.1</td>
  </tr>
  <tr>
    <td>Tulu 3 (8b)</td>
    <td>67.1</td>
    <td>59.6</td>
    <td>68.0</td>
    <td>60.4</td>
  </tr>
  <tr>
    <td>Llama 3.2 (3b) (I)</td>
    <td>72.2</td>
    <td>64.9</td>
    <td>70.9</td>
    <td>63.4</td>
  </tr>
  <tr>
    <td>OLMo 2 (7b)</td>
    <td>81.9</td>
    <td>74.8</td>
    <td>79.6</td>
    <td>72.6</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>81.1</td>
    <td>69.9</td>
    <td>79.2</td>
    <td>69.2</td>
  </tr>
</table>

(b) GPT-4o F1 on SPR1. (These prompts also elicited SRL.)

<table>
  <tr>
    <th rowspan="2">Prompt template</th>
    <th colspan="2">Micro-F1</th>
    <th colspan="2">Macro-F1</th>
  </tr>
  <tr>
    <th>SPR1, n/a = False</th>
    <th>SPR1, n/a = False</th>
    <th>SPR1, n/a skipped</th>
    <th>SPR1, n/a skipped</th>
  </tr>
  <tr>
    <td>All-args</td>
    <td>70.68</td>
    <td>60.65</td>
    <td>78.73</td>
    <td>64.21</td>
  </tr>
  <tr>
    <td>All-args, w/ PB GL</td>
    <td>68.8</td>
    <td>57.23</td>
    <td>79.01</td>
    <td>64.43</td>
  </tr>
  <tr>
    <td>All-args, n/a allowed</td>
    <td>75.9</td>
    <td>60.56</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 2: SPRL results evaluated in two modes: first, collapsing “n/a” annotations to *False*, and second, discarding “n/a” annotations from the evaluation set entirely.

• **With PropBank annotation guidelines:** Since a human annotator would typically have access to detailed, PropBank-specific instructions, a variant of the All-args template includes an excerpt from the PropBank annotation guidelines as a system prompt.

Further experiment details can be found in §A.

4.1 Metrics

**SRL:** For *All-args* and *One-arg* variants, we use accuracy and limit our evaluation to core (numbered) roles only (ARGs 0-5). To calculate accuracy, we count a “hit” every time a model produces the correctly numbered role and divide hits by all numbered roles in the evaluation set. For *Pipeline*, we use *exact match* accuracy: predicate span, predicate sense, argument span, and role label must all be exactly correct for a hit.

**SPRL:** We use the micro- and macro-F1 strategies in previous work: In macro-F1, F1 is computed first per property and then averaged, and in micro-F1, F1 is computed and averaged over all properties at once. We provide F1 values by (a) treating “n/a” as *False* and (b) skipping “n/a” completely (and thus, evaluating over a different set of arguments than previous work). In variants in which the model is allowed to produce “n/a” as SPRL output, the “n/a”-annotated arguments are still skipped: that is, we do not give it credit for getting the “n/a” value “correct.” If the model outputs “n/a” on an annotation that is *True* or *False*, we penalize the model as if it had given the opposite value for output.

![Bar chart showing effect of SPRL context on SRL accuracy, GPT-4o on SPR1](page_1012_120_484_246.png)

Figure 2: Effect of SPRL context on SRL accuracy, GPT-4o on SPR1. The y-axis shows the change in accuracy when providing the model with SPRL context (with SRL-alone prompts as the 0% baseline).

**One-arg** prompt variant results on SPR1 are aggregated in Table 3. See Table 4 for GPT-4o results across all prompt variants on Ontonotes. For GPT-4o, the most effective prompt variants are *All-args* and *Pipeline (oracle)*, with the *One-arg* prompting method consistently performing the worst out of all oracle variants. The pure *Pipeline* variant, in which the model is fed its own predictions for each step of the pipeline, achieved less than 25% accuracy on both datasets. The poor *Pipeline* performance is due to errors earlier in the pipeline—see Appendix §B. This seems to further the evidence (Ettinger et al., 2023; Bonn et al., 2024) that LLMs under-perform human annotators in complex, multi-step semantic analysis.

Why did the *All-args* and *Pipeline (oracle)* prompts yield better results than *One-arg*? One possible explanation is that all of the arguments being in the prompt context for the former two variants, as opposed to only one in the latter, forces the
Figure 3: Effect of SPRL context on SRL accuracy, smaller LMs on SPR1.

Figure 4: Effect of SPRL context on SRL accuracy, GPT-4o on Ontonotes.

model to consider all arguments at once and prevent it from assigning, for example, agent-of-action to two different arguments, but more experimentation would be necessary to isolate this effect.

The SPRL results (Table 2) contextualize the SRL results: first, when n/a is not evaluated, GPT-4o performs better on the simpler *SPRL-annotate* prompts than any of the *All-args* prompts, in which it is concurrently predicting SRL. However, in the *n/a = False* evaluation paradigm, the more complex multi-task *All-args* yields much better results.

LMs that perform well on SPRL tend to perform well on SRL, with two notable exceptions: Qwen2.5 performed the worst on SPRL while having the highest SRL accuracy out of all the other small LMs. On the other hand, Llama 3.2 (3b) exhibited competitive SPRL performance while lagging behind in SRL.

Interestingly, models tend to perform better on SPR2, *opposite* the trend in previous work, which has been explained by the SPR2 training set containing (1) less data than SPR1 or (2) more complex predicates and arguments. That zero-shot prompting methods yield better results on SPR2 than SPR1 seem to confirm that (1) was the case. In general, all zero-shot prompts we tested underperformed previous work, suggesting that fine-tuning or few-shot prompting would be a more fruitful method for utilizing LLMs for this task.

5.1 The effects of SPRL context on SRL

We plot the differences in SRL accuracy with 95% confidence intervals in Figures 2 (SPR1) and 4 (Ontonotes) for GPT-4o. These results correspond to the more detailed GPT-4o results in Tables 3 and 4. For the GPT-4o experiments on Ontonotes, a McNemar’s test (\( p < 0.05 \)) indicated a significant difference between the SRL-alone vs. SPRL-context prompt variants for all ARG0, ARG1, and ARG2 responses. The figures show that SPRL context mostly has a negative effect of SRL accuracy for GPT-4o, regardless of whether the SPRL context is in the form of gold labels (in the case of *One-arg*) or explicit instructions to generate SPRL output synchronously with SRL output (*All-args*). We additionally plot the differences in SRL accuracy with 95% confidence intervals for the smaller LMs that surpass the random baseline in Figure 3. We observe that the smaller LMs *can* benefit from SPRL context, but not consistently across models. In fact, among those that surpass the random baseline for SRL accuracy, only the Llama 3.1-based models (Llama 3.1 and Tulu 3) saw an unequivocal benefit from SPRL context. For those two models, we do see that eliciting the fuller sets of properties (SPR1 or SPR1∩SPR2) tends to result in a bigger boost.

**Evaluating only well-formatted responses.** A possible explanation for a drop in performance
<table>
  <tr>
    <th>Model</th>
    <th>SPRL Context</th>
    <th>Acc. (%)</th>
  </tr>
  <tr>
    <th colspan="3">Random Baseline</th>
  </tr>
  <tr>
    <td></td>
    <td>none</td>
    <td>36.91</td>
  </tr>
  <tr>
    <td>Llama 3.2 (3b) (I)</td>
    <td>none</td>
    <td>39.18</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>41.18 (+2.0)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>37.76 (−1.42)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>41.84 (+2.66)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>41.37 (+2.19)</td>
  </tr>
  <tr>
    <td>Pythia (6.9b)</td>
    <td>none</td>
    <td>15.56</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>16.7 (+1.14)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>11.67 (−3.89)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>12.71 (−2.85)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>14.71 (−0.85)</td>
  </tr>
  <tr>
    <td>OLMo (7b) (I)</td>
    <td>none</td>
    <td>45.07</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>42.69 (−2.38)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>45.83 (+0.76)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>51.04 (+5.97)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>47.91 (+2.84)</td>
  </tr>
  <tr>
    <td>Llama 2 (7b)</td>
    <td>none</td>
    <td>25.33</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>33.87 (+8.54)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>36.53 (+11.2)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>39.56 (+14.23)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>37.57 (+12.24)</td>
  </tr>
  <tr>
    <td>OLMo 2 (7b)</td>
    <td>none</td>
    <td>54.08</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>54.55 (+0.47)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>51.23 (−2.85)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>50.57 (−3.51)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>48.67 (−5.41)</td>
  </tr>
  <tr>
    <td>OLMo 2 (7b) (I)</td>
    <td>none</td>
    <td>53.23</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>47.91 (−5.32)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>47.63 (−5.6)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>48.39 (−4.84)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>46.49 (−6.74)</td>
  </tr>
  <tr>
    <td>Qwen2.5 (7b) (I)</td>
    <td>none</td>
    <td>61.01</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>58.06 (−2.95)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>57.5 (−3.51)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>54.93 (−6.08)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>54.46 (−6.55)</td>
  </tr>
  <tr>
    <td>Llama 3.1 (8b) (I)</td>
    <td>none</td>
    <td>51.23</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>53.42 (+2.19)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>57.21 (+5.98)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>57.31 (+6.08)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>55.5 (+4.27)</td>
  </tr>
  <tr>
    <td>Tulu 3 (8b)</td>
    <td>none</td>
    <td>59.3</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>61.57 (+2.27)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition-chg-state</td>
    <td>62.9 (+3.6)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>63.09 (+3.79)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1</td>
    <td>63.66 (+4.36)</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>none</td>
    <td>87.0</td>
  </tr>
  <tr>
    <td></td>
    <td>instigate.chg-loc</td>
    <td>87.67 (+0.67)</td>
  </tr>
  <tr>
    <td></td>
    <td>volition,chg-state</td>
    <td>85.2 (−1.8)</td>
  </tr>
  <tr>
    <td></td>
    <td>SPR1∩ SPR2</td>
    <td>86.05 (−0.95)</td>
  </tr>
</table>

Table 3: Effect of SPRL context on SRL accuracy across several different models on SPR1 (\( n = 1054 \) predicate-argument pairs) on the One-arg prompt variant. (I) indicates instruct-tuned.

when eliciting SPRL is the higher likelihood for the model to output a badly-formatted JSON string, due to an increased number of JSON key-value pairs the model must produce. Automatic evaluation would not capture the knowledge in a badly-formatted string. In qualitative analysis, we saw some instances of the model exhibiting correct judgment, but making a formatting error, and thus, incurring a penalty. We take a subset of responses that are well-formatted and evaluate only on those (Table 11a). However, even when evaluating on only well-formatted responses, GPT-4o mostly performs worse when asked to elicit SPRL. Interestingly, that is mostly true for Llama-3.2 (3b), except for ARGO, in which the SPRL-included prompts always perform better.

**Within-model SRL-SPRL agreement.** Does an LM’s SRL output align with its SPRL output? That is, if an LM outputs a proto-agent SPR label, what is the probability that it concurrently outputs a proto-agent SRL label? Our findings suggest that a model’s semantic role labeling decisions are principled on Dowty’s proto-roles. See Table 1, which shows the likelihood of an SPRL prediction to co-occur with an SRL prediction. For example, the leftmost cell of the top table indicates that 90.5% of all arguments labeled *True* for *instigation* were also labeled ARGO, while only 6% of arguments labeled *True* for *instigation* were labeled as an ARG1. We observe the tendency of a positive response for a proto-agent property to co-occur with an ARGO response, for a negative response for a proto-agent property to co-occur with an ARG1 response, and vice-versa for proto-patient properties (albeit with a weaker effect), suggesting the model encodes an event hierarchy similar to what is theorized in linguistics.\footnote{We also observe, like Reisinger et al. (2015), that Dowty’s *movement* (here, *change of location*) property does not tend to correlate with ARGO.}

We also observe what seems to be a very faint but similar effect even in the smallest LM prompted, Llama 3.2 (3b) (Table 9). Among the arguments it assigned numbered roles, Llama 3.2 (3b) shows a small preference for matching proto-agent properties with ARGO. (The effect all but disappears for proto-patient properties, which are classified at a much lower F1 across models.)

**Explaining SRL errors with SPRL errors?** We hypothesized that eliciting SPRL with SRL would help us to understand *why* a model gets an SRL example wrong. Could a concurrent error in SPRL help us pinpoint the precise dimension of the semantics that led a model to produce an SRL error?
<table>
  <tr>
    <th></th>
    <th colspan="2">ARG0</th>
    <th colspan="2">ARG1</th>
    <th colspan="2">ARG2</th>
    <th colspan="2">ARG3</th>
    <th colspan="2">ARG4</th>
    <th colspan="2">ARG5</th>
    <th>All core roles</th>
  </tr>
  <tr>
    <th>Prompt template</th>
    <th>n = 13754</th>
    <th>n = 22036</th>
    <th>n = 7959</th>
    <th>n = 451</th>
    <th>n = 398</th>
    <th>n = 11</th>
    <th>n = 14609</th>
  </tr>
  <tr>
    <td>One arg per prompt</td>
    <td>78.94</td>
    <td>93.14</td>
    <td>89.33</td>
    <td>74.72</td>
    <td>95.98</td>
    <td>100.0</td>
    <td>87.93</td>
  </tr>
  <tr>
    <td>+initiate,chg-loc</td>
    <td>77.88 (−1.06)</td>
    <td>93.41 (+0.27)</td>
    <td>85.09 (−4.24)</td>
    <td>74.5 (−0.22)</td>
    <td>95.98 (+0.0)</td>
    <td>100.0 (+0.0)</td>
    <td>86.97 (−0.96)</td>
  </tr>
  <tr>
    <td>+volition,chg-state</td>
    <td>76.39 (−2.55)</td>
    <td>93.61 (+0.47)</td>
    <td>81.4 (−7.93)</td>
    <td>74.28 (−0.44)</td>
    <td>95.48 (−0.5)</td>
    <td>90.91 (−9.09)</td>
    <td>85.94 (−1.99)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2</td>
    <td>76.57 (−2.37)</td>
    <td>94.73 (+1.59)</td>
    <td>79.23 (−10.11)</td>
    <td>74.28 (−0.44)</td>
    <td>95.23 (−0.75)</td>
    <td>90.91 (−9.09)</td>
    <td>86.16 (−1.77)</td>
  </tr>
  <tr>
    <td>All args per prompt</td>
    <td>95.04</td>
    <td>94.4</td>
    <td>89.67</td>
    <td>74.28</td>
    <td>94.97</td>
    <td>90.91</td>
    <td>93.55</td>
  </tr>
  <tr>
    <td>+initiate,chg-loc</td>
    <td>95.32 (+0.28)</td>
    <td>91.01 (−3.39)</td>
    <td>82.17 (−7.5)</td>
    <td>77.16 (+2.88)</td>
    <td>94.22 (−0.75)</td>
    <td>90.91 (+0.0)</td>
    <td>90.65 (−2.9)</td>
  </tr>
  <tr>
    <td>+volition,chg-state</td>
    <td>94.69 (−0.35)</td>
    <td>89.48 (−4.92)</td>
    <td>78.18 (−11.49)</td>
    <td>78.49 (+4.21)</td>
    <td>94.22 (−0.75)</td>
    <td>90.91 (+0.0)</td>
    <td>89.0 (−4.55)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2</td>
    <td>94.73 (−0.31)</td>
    <td>92.14 (−2.26)</td>
    <td>84.62 (−5.05)</td>
    <td>78.94 (+4.66)</td>
    <td>94.72 (−0.25)</td>
    <td>81.82 (−9.09)</td>
    <td>91.48 (−2.07)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2, allowing N/A</td>
    <td>95.0 (−0.01)</td>
    <td>90.83 (−3.57)</td>
    <td>81.28 (−8.29)</td>
    <td>75.61 (−1.55)</td>
    <td>93.72 (−1.23)</td>
    <td>90.91 (+0.0)</td>
    <td>90.28 (−3.27)</td>
  </tr>
  <tr>
    <td>Pipeline (oracle)</td>
    <td>94.44</td>
    <td>95.97</td>
    <td>89.57</td>
    <td>77.16</td>
    <td>95.73</td>
    <td>72.73</td>
    <td>94.16</td>
  </tr>
  <tr>
    <td>+initiate,chg-loc</td>
    <td>93.78 (−0.66)</td>
    <td>96.13 (+0.16)</td>
    <td>86.74 (−2.83)</td>
    <td>78.71 (+1.55)</td>
    <td>93.47 (−2.26)</td>
    <td>90.91 (+18.18)</td>
    <td>93.53 (−0.63)</td>
  </tr>
  <tr>
    <td>+volition,chg-state</td>
    <td>93.31 (−1.13)</td>
    <td>96.28 (+0.31)</td>
    <td>86.77 (−2.8)</td>
    <td>76.5 (−0.66)</td>
    <td>95.48 (−0.25)</td>
    <td>72.73 (+0.0)</td>
    <td>93.45 (−0.71)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2</td>
    <td>93.31 (−1.13)</td>
    <td>96.28 (+0.31)</td>
    <td>84.02 (−5.55)</td>
    <td>78.27 (+1.11)</td>
    <td>91.96 (−3.77)</td>
    <td>81.82 (+9.09)</td>
    <td>92.95 (−1.21)</td>
  </tr>
</table>

Table 4: Effect of eliciting SPR1 along with SRL across prompt variants on the Ontonotes dataset. Columns are subsets of the dataset, stratified by the gold argument label. Results are reported as accuracy, i.e., the number of times the model correctly labels an ARG0 as ARG0, divided by the total number of ARG0s in the dataset.

We observe a surprising lack of error agreement: only 56% of all SRL errors co-occur with any SPRL error at all (Table 5a). The majority of SRL errors cannot be explained by an error in a majority of the SPRL properties. Notably, the property with the highest percentage of co-occurring error, manipulated by another, can potentially be explained by idiosyncratic annotation of that property in SPR1.5

While SPRL errors do not overwhelmingly co-occur with SRL errors, SPRL annotations that are marked as not applicable to the sentence do overwhelmingly co-occur: (Table 5b). For args 3-5, this correlation is understandable: 12 out of 12 of those arguments are prepositional phrases (e.g. “The difference in yield ... widened [to more than 5.5 percentage points]ARG4”) for which none of the definitions in Table 7 make sense. Similarly, the ARG2 errors are also often prepositions, adverbs, and numerical values or proportions (“... retail sales grew [0.5%]ARG2”). Among all errors, 37% of the arguments were prepositional or adverbial phrases. Of all the arguments with a gold patient/ARG1 role that GPT-4o missed, 50% were the subject of the sentence (normally where an agent would be).

5.2 Qualitative error analysis

While SPRL context does not overwhelmingly help the model in SRL classification, it allows a finer-grained qualitative error analysis on the model’s semantic reasoning. We identify cases in which, even though the model’s SRL output is in error, its SPRL output is consistent with its SRL decision, and can help explain the SRL decision with greater clarity.

(1) [They]ARG1 would break, the wine would spill out, and the wineskins would be ruined.

(1-GPT) [They]ARG0 would break, ...

Sentence 1 contains an intransitive (unaccusative) alternation of the verb break: in this instance of the verb, there is no agent. In all prompt variants without SPRL context, GPT missed the syntactic cues indicating the lack of an agent, and it mistakenly classified the patient as an agent. Only prompts with SPRL context correctly classified “They” as the patient of the break verb.

(2) At Jefferies’ trading room on Finsbury Circus, a stately circle at the edge of the financial district, [desktop computer screens]ARG2 displayed the London market’s major barometer—the Financial Times-Stock Exchange 100 Share Index.

(2-GPT) ... [desktop computer screens]ARG0 displayed the London market’s major barometer...

In Sentence 2, ARG2 was wrongly classified in every single prompt variant, regardless of the context. Mostly, the model misclassified it as ARG0, the agent of the display action, even in variants where the model responds False for several proto-agent properties (instigation, volition, awareness, sentience) and True for several proto-patient properties (stationary, manipulated by another). One

5See White et al. (2016) for a discussion of the idiosyncrasy of the SPR1 annotator’s manipulated by another responses.
<table>
  <tr>
    <th>SPRL errors</th>
    <th>SRL errors n = 100</th>
    <th colspan="5">GPT-4o Error on:</th>
    <th rowspan="2">Annotation = N/A</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>ARG0 n = 24</th>
    <th>ARG1 n = 40</th>
    <th>ARG2 n = 24</th>
    <th>ARG3-5 n = 12</th>
    <th>All core roles n = 100</th>
  </tr>
  <tr>
    <td>instigation</td>
    <td>7.0% (7)</td>
    <td>25.0% (6)</td>
    <td>67.5% (27)</td>
    <td>75.0% (18)</td>
    <td>100.0% (12)</td>
    <td>63.0% (63)</td>
  </tr>
  <tr>
    <td>volition</td>
    <td>3.0% (3)</td>
    <td>20.83% (5)</td>
    <td>70.0% (28)</td>
    <td>95.83% (23)</td>
    <td>100.0% (12)</td>
    <td>68.0% (68)</td>
  </tr>
  <tr>
    <td>awareness</td>
    <td>5.0% (5)</td>
    <td>20.83% (5)</td>
    <td>62.5% (25)</td>
    <td>95.83% (23)</td>
    <td>100.0% (12)</td>
    <td>65.0% (65)</td>
  </tr>
  <tr>
    <td>sentient</td>
    <td>3.0% (3)</td>
    <td>58.33% (14)</td>
    <td>77.5% (31)</td>
    <td>100.0% (24)</td>
    <td>100.0% (12)</td>
    <td>81.0% (81)</td>
  </tr>
  <tr>
    <td>change of location</td>
    <td>4.0% (4)</td>
    <td>54.17% (13)</td>
    <td>67.5% (27)</td>
    <td>91.67% (22)</td>
    <td>100.0% (12)</td>
    <td>74.0% (74)</td>
  </tr>
  <tr>
    <td>exists as physical</td>
    <td>4.0% (4)</td>
    <td>45.83% (11)</td>
    <td>62.5% (25)</td>
    <td>87.5% (21)</td>
    <td>100.0% (12)</td>
    <td>69.0% (69)</td>
  </tr>
  <tr>
    <td>existed before</td>
    <td>18.0% (18)</td>
    <td>0.0% (0)</td>
    <td>7.5% (3)</td>
    <td>37.5% (9)</td>
    <td>66.67% (8)</td>
    <td>20.0% (20)</td>
  </tr>
  <tr>
    <td>existed during</td>
    <td>2.0% (2)</td>
    <td>0.0% (0)</td>
    <td>7.5% (3)</td>
    <td>33.33% (8)</td>
    <td>66.67% (8)</td>
    <td>19.0% (19)</td>
  </tr>
  <tr>
    <td>existed after</td>
    <td>11.0% (11)</td>
    <td>0.0% (0)</td>
    <td>15.0% (6)</td>
    <td>54.17% (13)</td>
    <td>66.67% (8)</td>
    <td>27.0% (27)</td>
  </tr>
  <tr>
    <td>changes poss</td>
    <td>2.0% (2)</td>
    <td>58.33% (14)</td>
    <td>82.5% (33)</td>
    <td>91.67% (22)</td>
    <td>91.67% (11)</td>
    <td>80.0% (80)</td>
  </tr>
  <tr>
    <td>change of state</td>
    <td>17.0% (17)</td>
    <td>0.0% (0)</td>
    <td>12.5% (5)</td>
    <td>50.0% (12)</td>
    <td>100.0% (12)</td>
    <td>29.0% (29)</td>
  </tr>
  <tr>
    <td>stationary</td>
    <td>13.0% (13)</td>
    <td>54.17% (13)</td>
    <td>67.5% (27)</td>
    <td>95.83% (23)</td>
    <td>100.0% (12)</td>
    <td>75.0% (75)</td>
  </tr>
  <tr>
    <td>location of event</td>
    <td>7.0% (7)</td>
    <td>79.17% (19)</td>
    <td>80.0% (32)</td>
    <td>100.0% (24)</td>
    <td>100.0% (12)</td>
    <td>87.0% (87)</td>
  </tr>
  <tr>
    <td>makes phys contact</td>
    <td>6.0% (6)</td>
    <td>50.0% (12)</td>
    <td>65.0% (26)</td>
    <td>95.83% (23)</td>
    <td>100.0% (12)</td>
    <td>73.0% (73)</td>
  </tr>
  <tr>
    <td>manip by another</td>
    <td>31.0% (31)</td>
    <td>70.83% (17)</td>
    <td>42.5% (17)</td>
    <td>20.83% (5)</td>
    <td>25.0% (3)</td>
    <td>42.0% (42)</td>
  </tr>
  <tr>
    <td>pred changed arg</td>
    <td>13.0% (13)</td>
    <td>8.33% (2)</td>
    <td>27.5% (11)</td>
    <td>75.0% (18)</td>
    <td>91.67% (11)</td>
    <td>42.0% (42)</td>
  </tr>
  <tr>
    <td><i>Any of the above</i></td>
    <td>56.0% (56)</td>
    <td>100.0% (24)</td>
    <td>100.0% (40)</td>
    <td>100.0% (24)</td>
    <td>100.0% (12)</td>
    <td>100.0% (100)</td>
  </tr>
</table>

(a) % of SRL errors that co-occur with SPRL errors.

(b) % of errors that co-occur with a not applicable property.

Table 5: Co-occurrence of GPT-4o’s SRL errors with (5a) SPRL errors elicited in the same prompt, and with (5b) arguments that have been annotated as not applicable for that SPRL property. From the All-args prompt variant on SPR1, with SPRL not evaluated when the annotation is “n/a.”.

possible explanation can still be provided by proto-role properties: several models responded False for the location of event proto-role property, and the PropBank description for ARG2 shown to the model is simply “location.”

(3) [Richard Chamberlain]<sub>ARG0</sub> dresses as a “Mainland haole,” tucking in a Hawaiian shirt and rolling up its long sleeves.

(3-GPT) [Richard Chamberlain]<sub>ARG1</sub> dresses as a “Mainland haole,” ...

Sentence 3 gives us another example in which proto-role properties seem to have helped GPT make an SRL decision: in the prompt variants without SPRL, the model misclassified ARG0 as ARG1 (proto-patient, the one wearing clothes) in the reflexive “dresses”. However, in the variants in which SPRL were elicited, the model correctly classified ARG0 as well as assigning True to strongly proto-agent properties instigation, volition, awareness, and sentience, suggesting that the SPRL context encouraged the model to reason more deeply over the event roles.

6 Conclusion and future work

Our study investigated how LLMs reason about agency and patency by examining the interplay between semantic proto-role labeling (SPRL) and semantic role labeling (SRL). We found that including SPRL context in prompts does not improve SRL accuracy, and in some cases even degrades performance. However, our experiments revealed a consistency between SPRL and SRL LLM output that mirrors Dowty’s proto-role theory. In particular, we tested two models, GPT-4o and Llama 3 (3b), on synchronous SRL and SPRL judgments, and found that they both tend to assign proto-agent properties (e.g., instigation, volition) alongside ARG0 labels, suggesting that the latent event role encodings of these models align with linguistic theory.

Additionally, our experiments prompting GPT-4o through a full SRL-SPRL pipeline provide further evidence that LLMs are not yet capable of complex, multi-step semantic reasoning needed to annotate sentences for rich event structure (Ettlinger et al., 2023; Bonn et al., 2024), suggesting that LLMs deployed in real-world settings may lack the advanced situation-modeling capabilities necessary to make complex decisions.

However, our results show that LLMs do encode something like a latent event role hierarchy, and we suspect that there are better ways to coax the models to employ it. Future research should explore, for example, few-shot learning or fine-tuning strategies to better harness LLMs’ event role knowledge. Such improvements could not only improve semantic role labeling accuracy, but also deepen our understanding of how LLMs model event structure: a necessary component in developing robust AI systems with transparent situation-modeling and decision-making capabilities.
Limitations

Our study was limited in its focus on English. The models and data evaluated are both either all-English, or highly English-centric, and as such, our claims cannot extend across languages. The annotation schemes of the datasets also may not generalize well to other domains, limiting the generalizability of our findings. Fruitful future work could focus its efforts on evaluating multi-lingual LLMs on verb lexica in different languages.

Because small changes in wording or formatting could lead to different performance outcomes, our results are also influenced by the specific prompt formulations we used. While we attempted to capture a broad swath of prompts, much more experimentation needs to be done to understand what prompt-based methods truly can reveal regarding event role knowledge in LLMs.

We also recognize the possibility of data contamination (i.e. that the test sets we evaluated on were included as training data for an LLM), potentially inflating performance due to memorization rather than genuine semantic reasoning. We attempted to mitigate this risk by including open-data models in our analysis, but we cannot entirely rule out the possibility of contamination in models like GPT-4o. Nevertheless, we included GPT-4o and other closed-data models because of their ubiquity in real-world applications. Future work should consider evaluating on entirely novel data to remove this risk.

Ethical Considerations

Finally, we want to emphasize the risk of overestimation of LLM capabilities. If those interested in deploying LLMs interpret our results as evidence that LLMs truly “understand” agency or moral responsibility, they may use these models to support or automate decisions that have significant ethical implications. Such misuse could exacerbate issues like bias in real-world settings. We reiterate that our results (especially in the Pipeline setting) support the finding that language models are not yet capable of complex semantic analysis.

Acknowledgments

We thank the ARR reviewers for their helpful feedback and suggestions. We also gratefully acknowledge the support of the Center for Computational Language and Education Research (CLEAR) at the University of Colorado Boulder.

References

Elliott Ash, Germain Gauthier, and Philine Widmer. 2024. Relatio: Text semantics capture political and economic narratives. Political Analysis, 32(1):115–132.

Ananth Balashankar, Lakshminarayanan Subramanian, and Samuel P. Fraiberger. 2023. Predicting food crises using news streams. Science Advances, 9(9):eabm3449.

Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178–186.

Abhidip Bhattacharyya, Martha Palmer, and Christoffer Heckman. 2023. CRAPES:cross-modal annotation projection for visual semantic role labeling. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 61–70, Toronto, Canada. Association for Computational Linguistics.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Halahan, Mohammad Afah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR.

Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English web treebank.

Claire Bonial, Kathryn Conger, Jena D. Hwang, Aous Mansouri, Yahya Aseri, Julia Bonn, Timothy O’Gorman, and Martha Palmer. 2017. Current directions in english and arabic propbank.

Julia Bonn, Harish Tayyar Madabushi, Jena D. Hwang, and Claire Bonial. 2024. Adjudicating LLMs as Prop-Bank adjudicators. In Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024, pages 112–123, Torino, Italia. ELRA and ICCL.

David Dowty. 1991. Thematic Proto-Roles and Argument Selection. Language, 67(3):547–619. Publisher: Linguistic Society of America.

Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. 2024. What’s in my big data? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Allyson Ettinger, Jena Hwang, Valentina Pyatkin, Chandra Bhagavatula, and Yejin Choi. 2023. “you are an expert linguistic annotator”: Limits of LLMs as analyzers of Abstract Meaning Representation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8250–8263, Singapore. Association for Computational Linguistics.

Hao Fei, Shengqiong Wu, Yafeng Ren, Fei Li, and Donghong Ji. 2021. Better combine them together! integrating syntactic constituency and dependency representations for semantic role labeling. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 549–559, Online. Association for Computational Linguistics.

Steven M Frankland and Joshua D Greene. 2020. Two ways to build a thought: Distinct forms of compositional semantic representation across brain regions. Cerebral Cortex, 30(6):3838–3855.

Cynthia A. Freeland. 1985. Aristotelian actions. Noûs, 19(3):397–414.

Daniel Gildea and Martha Palmer. 2002. The necessity of parsing for predicate argument recognition. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 239–246, USA. Association for Computational Linguistics.

Aaron Grattafiori et al. 2024. The llama 3 herd of models. (arXiv:2407.21783). ArXiv:2407.21783 [cs].

Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Arthur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muenninghoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. 2024. OLMO: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, Bangkok, Thailand. Association for Computational Linguistics.

Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny T. Liang, Sydney Levine, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jack Hessel, Jon Borchardt, Taylor Sorensen, Saadiah Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. 2025. Investigating machine moral judgement through the delphi experiment. Nature Machine Intelligence, 7(1):145–160.

Paul Kingsbury and Martha Palmer. 2002. From Tree-Bank to PropBank. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02), Las Palmas, Canary Islands - Spain. European Language Resources Association (ELRA).

Ilia Kuznetsov and Iryna Gurevych. 2020. A matter of framing: The impact of linguistic formalism on probing results. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 171–182, Online. Association for Computational Linguistics.

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. 2025. Tul3: Pushing frontiers in open language model post-training. (arXiv:2411.15124). ArXiv:2411.15124 [cs].

Wei Liu, Songlin Yang, and Kewei Tu. 2023. Structured mean-field variational inference for higher-order span-based semantic role labeling. In Findings of the Association for Computational Linguistics: ACL 2023, pages 918–931, Toronto, Canada. Association for Computational Linguistics.

Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540.

Team OLMO, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2025. 2 olmo 2 furious. (arXiv:2501.00656). ArXiv:2501.00656 [cs].

OpenAI. 2024. Gpt-4 technical report. (arXiv:2303.08774). ArXiv:2303.08774 [cs].

Juri Opitz and Anette Frank. 2019. An argument-marker model for syntax-agnostic proto-role labeling. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 224–234, Minneapolis, Minnesota. Association for Computational Linguistics.

Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Comput. Linguist., 31(1):71–106.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. (arXiv:2412.15115). ArXiv:2412.15115 [cs].

Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the Association for Computational Linguistics, 3:475–488.

Rachel Rudinger, Adam Teichert, Ryan Culkin, Sheng Zhang, and Benjamin Van Durme. 2018. Neural-Davidsonian semantic proto-role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 944–955, Brussels, Belgium. Association for Computational Linguistics.

Zacchary Sadeddine, Juri Opitz, and Fabian Suchanek. 2024. A survey of meaning representations – from theory to practical utility. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2877–2892, Mexico City, Mexico. Association for Computational Linguistics.

Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Neva-tia, and Aniruddha Kembhavi. 2021. Visual semantic role labeling for video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5589–5600.

Kelly G. Shaver. 1985. The Attribution of Blame. Springer, New York, NY.

Elizabeth Spaulding, Gary Kazantsev, and Mark Dredze. 2023. Joint end-to-end semantic proto-role labeling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 723–736, Toronto, Canada. Association for Computational Linguistics.

Elias Stengel-Eskin, Kenton Murray, Sheng Zhang, Aaron Steven White, and Benjamin Van Durme. 2021. Joint universal syntactic and semantic parsing. Transactions of the Association for Computational Linguistics, 9:756–773.

Elias Stengel-Eskin and Benjamin Van Durme. 2022. The curious case of control. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11065–11076, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Elias Stengel-Eskin, Aaron Steven White, Sheng Zhang, and Benjamin Van Durme. 2020. Universal decompositional semantic parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8427–8439, Online. Association for Computational Linguistics.

Adam Teichert, Adam Poliak, Benjamin Van Durme, and Matthew Gormley. 2017. Semantic proto-role labeling. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1).

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenjin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rou Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molbyog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rash Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. (arXiv:2307.09288). ArXiv:2307.09288 [cs].

Bernard Weiner. 1995. Judgments of responsibility: A foundation for a theory of social conduct. Judgments of responsibility: A foundation for a theory of social conduct. Guilford Press, New York, NY, US.

Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Universal decompositional semantics on Universal Dependencies. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1713–1723, Austin, Texas. Association for Computational Linguistics.

Yu Zhang, Qingrong Xia, Shilin Zhou, Yong Jiang, Guohong Fu, and Min Zhang. 2022. Semantic role labeling as dependency parsing: Exploring latent tree structures inside arguments. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4212–4227, Gyeongju, Republic of
Korea. International Committee on Computational Linguistics.

Shilin Zhou, Qingrong Xia, Zhenghua Li, Yu Zhang, Yu Hong, and Min Zhang. 2022. Fast and accurate end-to-end span-based semantic role labeling as word-based graph parsing. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4160–4171, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

A Experiment details

This section provides precise details for each prompting experiment, exact model names, and specific parameters used when prompts were run. Exact model configurations, prompts, and output provided upon request, and samples can be found in the released code used for the experiments.

A.1 Prompt variants

See Table 6 for examples of each prompt variant.

Pipeline: The model is prompted to provide output for every single component in a six-component SRL pipeline: predicate span identification, predicate sense disambiguation, argument span identification, (optionally) semantic proto-role labeling, and semantic role labeling. Each component is separately processed via a prompt which includes, as context, the prompt and output for the previous component. At the end of the pipeline, the model has access to the full pipeline starting from the raw sentence, including all of the previous predicate span, argument span, and SPRL output, but only has to label one argument per prompt. See further details on this prompt variant in A.2 and an example in Figure 7.

All-args-per-prompt: The model is prompted to provide a JSON with the SPRL and SRL output for all constituents of a single predicate. The predicate spans, predicate sense, argument spans, and roleset information are all provided to the model by an oracle: its only task is to select the correct semantic role label (and/or semantic proto-role label). A sample prompt for this variant can be found in Figure 6. This prompt will allow us to study the effect of eliciting SPRL context from the model itself, without providing gold SPR labels.

One-arg-per-prompt: To evaluate the model on a prompt that requires a simpler output format than a JSON-formatted string, this prompt only elicits the semantic role label for a single argument. SPRL is elicited in a separate prompt from SRL, and, in settings where SPRL is included, it is presented as optional but helpful context. See Figure 5 for an example.

We additionally provide a random baseline for the One-arg variant, in which the semantic role label is randomly chosen from the set of all numbered argument labels in the training set, weighted by the proportion each is found in the training set (so that the random choices are proportional with the labels in the training set).

SPRL-annotate: We evaluate all models on a simple true/false prompt, designed to be similar in format to the questions posed to the SPR1 and SPR2 annotators (Table 7). The models are only prompted on one argument-predicate-property triplet at a time, as in the human annotation protocol. Figure 8 shows an example of this prompt.

<table>
  <tr>
    <th>Prompt name</th>
    <th>Prompt example</th>
  </tr>
  <tr>
    <td>One-arg-per-prompt</td>
    <td>Figure 5</td>
  </tr>
  <tr>
    <td>All-args-per-prompt</td>
    <td>Figure 6</td>
  </tr>
  <tr>
    <td>Pipeline</td>
    <td>Figure 7</td>
  </tr>
  <tr>
    <td>SPRL-annotate</td>
    <td>Figure 8</td>
  </tr>
</table>

Table 6: Prompt index

A.2 Components of SRL pipeline

The pipeline prompts are separated into six separate queries to the model, listed below. Each query’s output is then chained to the beginning of the next prompt as additional context for the model. In the Pipeline (oracle) setting, items 1-4 are not processed through the model. Instead, we process items 1-4 with gold data as an oracle, and chain the prompts as in the non-oracle setting.

1. Predicate span retrieval
   Input: The dog barked.
   Output: The dog <PRED>barked</PRED>.

2. Roleset identification given predicate span from predicate output
   Input: The dog <PRED>barked</PRED>. + a list of rolesets for lemma “bark” and their descriptions
   Output: bark .01

3. Argument span retrieval given predicate span from predicate output
   Input: The dog <PRED>barked</PRED>.
   Output: <ARG>The dog</ARG> <PRED>barked</PRED>.
4. Semantic proto-role labeling given argument and predicate span from argument output
Input: <ARG>The dog</ARG> <PRED>barked</PRED>. + a list of semantic proto-role properties and their descriptions
Output: A JSON string with the listed semantic proto-role properties as keys and booleans as values

5. Semantic role labeling given a roleset, argument span, and predicate span from argument output
Input: <ARG>The dog</ARG> <PRED>barked</PRED>. + a list of roles and role descriptions for bark.01
Output: ARG0

6. Semantic role labeling given SPRL, roleset, argument span, and predicate span from SPRL output
Input: <ARG>The dog</ARG> <PRED>barked</PRED>. + a list of roles and role descriptions for bark.01 + a JSON string with semantic proto-role properties as keys and booleans as values
Output: ARG0

A.3 Prompt context

SPRL context: SPRL context is provided to the model in various ways, depending on the prompt template: in Pipeline, the model has access to its previous predictions; in One-arg, the model is given SPRL context from gold labels; and in All-args, the model is given instructions to produce SPRL output concurrently with SRL output. In every prompt variant, we ablate SPRL context to ascertain the effect of the context on the model’s decision-making.

Oracle predicate spans, predicate senses, and argument spans: All prompt template variants provide the model with oracle predicate spans, predicate senses, and argument spans except Pipeline. And even in Pipeline, the model has access to all of its previous predicate span, predicate sense, and argument span predictions at the end of the pipeline, when the model is finally prompted to make its SRL prediction. None of our experiments prompt the model to make concurrent predicate span, predicate sense, and argument span predictions along with SPRL and SRL predictions.

Allowing “N/A” as a SPRL response: A potential weakness of SPRL modeling is the handling of not applicable annotations. In most previous work, properties annotated not applicable are converted to False in the data, and systems are evaluated on their abilities to classify those instances as False. For some experiments in the All-args prompt variant, we introduce an additional variant in which the model is explicitly instructed to mark properties as “n/a” if that property doesn’t apply to the argument. And, while SPRL performance is not our main concern, when we evaluate, we skip over properties that are marked “n/a” in the annotations.

With PropBank annotation guidelines: An additional concern in evaluating the model on its ability to produce PropBank-style SRL is the lack of detailed, PropBank-specific instruction that a human annotator would have access to, especially because the information found in the PropBank rolesets presented to the model can appear cryptic to a non-expert. While we think it is likely that GPT-4o’s training data included public resources such as the PropBank annotation guidelines, we include in our experiments a variant on the All-args template that includes a relevant excerpt of the PropBank annotation guidelines6 as a system prompt. Specifically, we include Section 1.1, up to and including Table 1.1, and Section 1.3.1, on choosing ARG0 vs. ARG1.

A.4 Parameters and model details

We use gpt-4o-2024-05-13 for all GPT experiments. For the rest of the experiments, we access models through HuggingFace, using the following model codes:

• meta-llama/Llama-2-7b-hf-Instruct
• meta-llama/Llama-2-7b-hf
• meta-llama/Llama-3.1-8B-Instruct
• meta-llama/Llama-3.2-3B-Instruct
• allenai/OLMo-2-1124-7B-Instruct
• allenai/OLMo-2-1124-7B
• allenai/OLMo-7b-Instruct-hf
• EleutherAI/pythia-6.9b
• Qwen/Qwen2.5-7B-Instruct
• allenai/Llama-3.1-Tulu-3-8B

6https://github.com/propbank/propbank-documentation
We instantiate each model according to their default configurations—the only parameter we change is the max_length on generation, which we set to model_max_position_embeddings + 4. Runtimes varied between experiments (number of parameters, prompt length, and number of samples in the datasets all affecting the runtime), but the experiments on the models with the most parameters and the largest datasets took ~20 GPU hours to run on a single NVIDIA TITAN XP or NVIDIA TITAN RTX and the experiments on the models with the fewest parameters took as little as ~20 minutes to run. Exact runtimes per experiment can be provided on request.

B Pipeline performance

The Pipeline prompt variant exhibited perhaps shockingly low accuracy because of the strict nature of the evaluation metrics for the span retrieval components within the SRL pipeline. A semantic role label is only counted correct if the model is able to achieve an exact match on every single component: retrieval of the exact predicate span, classification of the predicate sense, and retrieval of the exact argument span, as well as the correct role label. Below is the gold span annotation (for the argument span retrieval step, before being prompted for the role label) for a test sentence:

(4) To express its determination, [the Chinese securities regulatory department]_{ARG} compares [this stock reform]_{ARG} [to a die that has been cast]_{ARG'}.

Compare the gold annotation to the output from GPT-4o:

(4-GPT) [To express its determination, the Chinese securities regulatory department]_{ARG'} compares [this stock reform]_{ARG} to [a die that has been cast]_{ARG'}.

Out of the three arguments in the gold annotation, GPT only successfully produced the exact span for one of them—a 33% accuracy. Table 8 provides a fine-grained breakdown of the accuracy of the GPT-4o output on the Pipeline prompt variant, contextualizing the very low accuracy shown in Table 4.

Because the model was only able to correctly retrieve 26% of the argument spans in the Ontonotes dataset, it did not even see the other 74% and was automatically penalized for them, which is why the accuracy was so low. Although these experiments show a very low performance overall in the span retrieval tasks, we suspect that GPT-4o is capable of better performance. For instance, trying few-shot approaches, changing the potentially-confusing HTML-style <PRED> and <ARG> tags, or trying a different style of prompt completely could yield better results.

C Full results

We provide results on all model runs, broken down per argument, in Table 10 for GPT-4o and Llama-3.2 (3b).

C.1 Non-compliance with formatting requirements

We perform some evaluations in order to disentangle the effects of non-compliance with the formatting requirements specified by the prompt (i.e., a model failing to correctly format its response). For the All-args prompt template, see Table 11 shows accuracy on only parsable output. For the 3b-parameter Llama-3.2 output, we additionally perform “fuzzy role matching,” which allows “agent” and “patient” as correct responses for ARG0 and ARG1, respectively, because of the large number of responses that used those words instead of the required ARG0 and/or ARG1. As such, we are able to evaluate the actual knowledge within Llama without giving it undue penalty for minor formatting mistakes. We see that the model performs far better under the fuzzy role matching evaluation method.

D Licenses

All data used for prompting and evaluation is under a CC BY-SA 4.0 license. All work was consistent with the intended use of data and models, which have the following licenses and terms: Apache 2.0 for Qwen2.5, OLMo 1 and 2, and Pythia. Llama Community License for Llama 2 and 3 (details here). OpenAI terms of use (here) for GPT-4o. Tulu 3 is subject to the licenses of the models used to train it: Llama Community License, Gemma Terms of Use, and Qwen License Agreement (details here).
<table>
  <tr>
    <th>Prompt</th>
    <td>Identify the Semantic Role for the predicate <PRED>put</PRED> and arg <ARG>We</ARG>.</td>
  </tr>
  <tr>
    <td></td>
    <td>Below are the Semantic Roles for <PRED>put</PRED>:
      ARG0: putter
      ARG1: thing put.
      ARG2: attribute of arg1.
      Respond with only the role label, and nothing else, like so:
      Example: <ARG>The dog</ARG> <PRED>barked</PRED>.
      Response: ARG0
    </td>
  </tr>
  <tr>
    <td></td>
    <td>Text for labeling:
      <ARG>We</ARG> <PRED>put</PRED> some orders together .
    </td>
  </tr>
  <tr>
    <th>Response</th>
    <td>ARG0</td>
  </tr>
</table>

(a) SRL only, <i>one-arg</i> prompt variant.

<table>
  <tr>
    <th>Prompt</th>
    <td>Identify the Semantic Role for the predicate <PRED>put</PRED> and arg <ARG>We</ARG>.</td>
  </tr>
  <tr>
    <td></td>
    <td>Consider the fact that <ARG>We</ARG> has the following properties in making your decision:

    instigation: ARG caused PRED to happen.
    change_of_location: ARG changed location during PRED.

    Below are the Semantic Roles for <PRED>put</PRED>:
    ARG0: putter
    ARG1: thing put.
    ARG2: attribute of arg1.
    Respond with only the role label, and nothing else, like so:
    Example: <ARG>The dog</ARG> <PRED>barked</PRED>.
    Response: ARG0
    Text for labeling:
    <ARG>We</ARG> <PRED>put</PRED> some orders together .
  </td>
  </tr>
  <tr>
    <th>Response</th>
    <td>ARG0</td>
  </tr>
</table>

(b) SRL with SPRL context, <i>one-arg</i> prompt variant.

Figure 5: <i>One-arg-per-prompt</i> template variant. This prompt asks only for the model to pick a semantic role out of a list of semantic roles and provides the SPRL context from the model’s silver predictions, but not in a pipeline context. It only asks for one argument at a time.
Identify the Semantic Roles for the arg(s) <ARG>We</ARG>, <ARG>some orders</ARG>, <ARG>together</ARG> with respect to the predicate <PRED>put</PRED> in the following sentence:

<ARG>We</ARG> <PRED>put</PRED> <ARG>some orders</ARG> <ARG>together</ARG> .

1. Analyze each Argument for Semantic Proto-Role Properties. Below are the properties:
volition: ARG chose to be involved in PRED. true/false
change_of_location: ARG changed location during PRED. true/false

2. Label each Argument for its Semantic Role. Below are the Semantic Roles for <PRED>put</PRED>:
ARG0: putter.
ARG1: thing put.
ARG2: attribute of arg1.

JSON Format:
{
    "text": "We put some orders together .",
    "predicates": {
        "put": {
            "start_char": 3,
            "arguments": {
                "We": {
                    "start_char": 0,
                    "volition": true/false,
                    "change_of_location": true/false,
                    "role": "Identified Semantic Role Label for <ARG>We</ARG>"
                },
                "some orders": {
                    "start_char": 7,
                    "volition": true/false,
                    "change_of_location": true/false,
                    "role": "Identified Semantic Role Label for <ARG>some orders</ARG>"
                },
                "together": {
                    "start_char": 19,
                    "volition": true/false,
                    "change_of_location": true/false,
                    "role": "Identified Semantic Role Label for <ARG>together</ARG>"
                }
            }
        }
    }
}

Original text for labeling:
We put some orders together .

Prompt
Output only the JSON.

Response
{
    "json": {
        "text": "...",
        "predicates": {
            "put": {
                "start_char": 3,
                "arguments": {
                    "We": {
                        "start_char": 0,
                        "role": "ARG0"
                    },
                    "some orders": {
                        "start_char": 7,
                        "role": "ARG1"
                    },
                    "together": {
                        "start_char": 19,
                        "role": "ARG2"
                    }
                }
            }
        }
    }
}

(a) All-args prompt variant, eliciting both SRL and SPRL at the same time.

Identify the Semantic Roles for the arg(s) <ARG>We</ARG>, <ARG>some orders</ARG>, <ARG>together</ARG> with respect to the predicate <PRED>put</PRED> in the following sentence:

<ARG>We</ARG> <PRED>put</PRED> <ARG>some orders</ARG> <ARG>together</ARG> .

Below are the possible Semantic Roles for <PRED>put</PRED>:
ARG0: putter.
ARG1: thing put.
ARG2: attribute of arg1.

JSON Format:
{
    "text": "We put some orders together .",
    "predicates": {
        "put": {
            "start_char": 3,
            "arguments": {
                "We": {
                    "start_char": 0,
                    "role": "Identified Semantic Role Label for <ARG>We</ARG>"
                },
                "some orders": {
                    "start_char": 7,
                    "role": "Identified Semantic Role Label for <ARG>some orders</ARG>"
                },
                "together": {
                    "start_char": 19,
                    "role": "Identified Semantic Role Label for <ARG>together</ARG>"
                }
            }
        }
    }
}

Original text for labeling:
We put some orders together .

Prompt
Output only the JSON.

Response
{
    "json": {
        "text": "We put some orders together .",
        "predicates": {
            "put": {
                "start_char": 3,
                ...
            }
        }
    }
}

(b) All-args prompt variant, eliciting only SRL.

Figure 6: All-args-per-prompt template variants. This prompt asks the model to pick a semantic role concurrently with semantic proto-role labeling properties, and output them all in JSON format.
<table>
  <tr>
    <th>Prompt</th>
    <th>Response</th>
  </tr>
  <tr>
    <td>Identify predicates with &lt;PRED&gt;/&lt;/PRED&gt; tags on the original sentence.<br>Format the response inline with the original sentence, and only respond with the labeled sentence, like so:<br>Example: The dog barked.<br>Response: The dog &lt;PRED&gt;barked&lt;/PRED&gt;.<br><br>Text for labeling:<br>MANY AFFLUENT people place personal success and money above family .<br><br>Response:</td>
    <td>MANY AFFLUENT people &lt;PRED&gt;place&lt;/PRED&gt; personal success and money above family.</td>
  </tr>
</table>

Previous prompt + response + new prompt

<table>
  <tr>
    <th>Prompt</th>
    <th>Response</th>
  </tr>
  <tr>
    <td>Identify predicates with &lt;PRED&gt;/&lt;/PRED&gt; tags on the original sentence.<br>Format the response inline with the original sentence, and only respond with the labeled sentence, like so:<br>Example: The dog barked.<br>Response: The dog &lt;PRED&gt;barked&lt;/PRED&gt;.<br><br>Text for labeling:<br>MANY AFFLUENT people place personal success and money above family .<br><br>Response:<br>MANY AFFLUENT people &lt;PRED&gt;place&lt;/PRED&gt; personal success and money above family.<br>Identify arguments for the predicate &lt;PRED&gt;place&lt;/PRED&gt; with &lt;ARG&gt;&lt;/ARG&gt; tags on the original sentence.<br><br>Format the response inline with the original sentence, and only respond with the labeled sentence, like so:<br>Example: The dog &lt;PRED&gt;barked&lt;/PRED&gt;.<br>Response: &lt;ARG&gt;The dog&lt;/ARG&gt; &lt;PRED&gt;barked&lt;/PRED&gt;.<br><br>Text for labeling:<br>MANY AFFLUENT people &lt;PRED&gt;place&lt;/PRED&gt; personal success and money above family.<br><br>Response:</td>
    <td>&lt;ARG&gt;MANY AFFLUENT people&lt;/ARG&gt; &lt;PRED&gt;place&lt;/PRED&gt; &lt;ARG&gt;personal success and money&lt;/ARG&gt; above family.</td>
  </tr>
</table>

(a) Predicate and argument prompts.

(previous text omitted)...&lt;PRED&gt; personal success and money above family.<br>Response:<br>&lt;ARG&gt;MANY AFFLUENT people&lt;/ARG&gt; &lt;PRED&gt;place&lt;/PRED&gt; &lt;ARG&gt;personal success and money&lt;/ARG&gt; above family.<br>Identify SPRL properties for the predicate &lt;PRED&gt;place&lt;/PRED&gt; and arg &lt;ARG&gt;MANY AFFLUENT people&lt;/ARG&gt;.<br>Give judgements for each SPRL property below.<br>volition: ARG chose to be involved in PRED.<br>change_of_state: state of the world changed during or by the end of PRED.<br>change_of_state: false<br><br>JSON Format:<br>{<br>  "MANY AFFLUENT people": {<br>    "volition": true/false,<br>    "change_of_state": true/false<br>  }<br>}<br><br>Text for labeling:<br>&lt;ARG&gt;MANY AFFLUENT people&lt;/ARG&gt; &lt;PRED&gt;place&lt;/PRED&gt; personal success and money above family.<br>Output only the JSON.<br><br>Response:
```json
{"MANY AFFLUENT people": {
    "volition": true,
    "change_of_state": false
}}
```

<table>
  <tr>
    <th>Prompt</th>
    <th>Response</th>
  </tr>
  <tr>
    <td>(previous text omitted)...Response:<br>```json<br>"MANY AFFLUENT people": {<br>  "volition": true,<br>  "change_of_state": false<br>}<br>```<br>Identify the Semantic Role for the predicate &lt;PRED&gt;place&lt;/PRED&gt; and arg &lt;ARG&gt;MANY AFFLUENT people&lt;/ARG&gt;.<br><br>Below are the Semantic Roles for &lt;PRED&gt;place&lt;/PRED&gt;:<br>ARG0: utterer.<br>ARG1: thing put.<br>ARG2: put to.<br>Respond with only the role label, and nothing else, like so:<br>Example: &lt;ARG&gt;The dog&lt;/ARG&gt; &lt;PRED&gt;barked&lt;/PRED&gt;.<br><br>Text for labeling:<br>&lt;ARG&gt;MANY AFFLUENT people&lt;/ARG&gt; &lt;PRED&gt;place&lt;/PRED&gt; personal success and money above family.<br><br>Response:</td>
    <td>ARG0</td>
  </tr>
</table>

(b) SPRL and SRL prompts.

Figure 7: Pipeline prompts. The above represents a single SRL instance from start (predicate ID) to end (SRL). In the SRL-only setting, the SPRL step is skipped completely so that the prompt gives no SPRL-related context. In the Pipeline (oracle) setting, the predicate, roleset, and argument output is taken from the gold data as an oracle, but the prompt format is the exact same.
<table>
  <tr>
    <th>Prompt</th>
    <td>TRUE or FALSE: "MANY AFFLUENT people" was altered or somehow changed during or by the end of "place" in the following sentence:<br>[MANY AFFLUENT people] [place] personal success and money above family .</td>
  </tr>
  <tr>
    <th>Response</th>
    <td>FALSE</td>
  </tr>
</table>

Figure 8: SPRL-annotate prompt example, designed to be similar to questions posed to annotators in White et al. (2016).

<table>
  <tr>
    <th>Role Property</th>
    <th>Proto-role</th>
    <th>Definition</th>
    <th>Dataset</th>
  </tr>
  <tr>
    <td>instigation</td>
    <td>agent</td>
    <td><i>Arg</i> caused <i>Pred</i> to happen</td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>volition</td>
    <td>agent</td>
    <td><i>Arg</i> chose to be involved in <i>Pred</i></td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>awareness</td>
    <td>agent</td>
    <td><i>Arg</i> was aware of being involved in <i>Pred</i></td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>sentient</td>
    <td>agent</td>
    <td><i>Arg</i> was sentient</td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>change of location</td>
    <td>agent</td>
    <td><i>Arg</i> changed location during <i>Pred</i></td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>exists as physical</td>
    <td>agent</td>
    <td><i>Arg</i> existed as a physical object</td>
    <td>1</td>
  </tr>
  <tr>
    <td>existed before</td>
    <td>[depends]</td>
    <td><i>Arg</i> existed before <i>Pred</i> began</td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>existed during</td>
    <td>[depends]</td>
    <td><i>Arg</i> existed during <i>Pred</i></td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>existed after</td>
    <td>[depends]</td>
    <td><i>Arg</i> existed after <i>Pred</i> stopped</td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>created*</td>
    <td>patient</td>
    <td>Infer from (<span style="color:green">–existed before</span> and <span style="color:green">+existed after</span>)</td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>destroyed*</td>
    <td>patient</td>
    <td>Infer from (<span style="color:green">+existed before</span> and <span style="color:green">–existed after</span>)</td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>change of possession</td>
    <td>patient</td>
    <td><i>Arg</i> changed possession during <i>Pred</i></td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>change of state</td>
    <td>patient</td>
    <td><i>Arg</i> was altered or somehow changed during or by the end of <i>Pred</i></td>
    <td>1&2</td>
  </tr>
  <tr>
    <td>stationary</td>
    <td>patient</td>
    <td><i>Arg</i> was stationary during <i>Pred</i></td>
    <td>1</td>
  </tr>
  <tr>
    <td>location of event</td>
    <td>peripheral</td>
    <td><i>Arg</i> described the location of <i>Pred</i></td>
    <td>1</td>
  </tr>
  <tr>
    <td>makes physical contact</td>
    <td>agent</td>
    <td><i>Arg</i> made physical contact with someone or something else involved in <i>Pred</i></td>
    <td>1</td>
  </tr>
  <tr>
    <td>was used</td>
    <td>patient</td>
    <td><i>Arg</i> was used in carrying out <i>Pred</i></td>
    <td>2</td>
  </tr>
  <tr>
    <td>manipulated by another</td>
    <td>patient</td>
    <td><i>Arg</i> was used in carrying out <i>Pred</i></td>
    <td>1</td>
  </tr>
  <tr>
    <td>predicate changed argument</td>
    <td>patient</td>
    <td><i>Pred</i> caused a change in <i>Arg</i></td>
    <td>1</td>
  </tr>
  <tr>
    <td>was for benefit</td>
    <td>patient</td>
    <td><i>Pred</i> happened for the benefit of <i>Arg</i></td>
    <td>2</td>
  </tr>
  <tr>
    <td>partitive</td>
    <td>patient</td>
    <td>Only a part of portion of <i>Arg</i> was involved in <i>Pred</i></td>
    <td>2</td>
  </tr>
  <tr>
    <td>change of state continuous</td>
    <td>patient</td>
    <td>The change in <i>Arg</i> happened throughout <i>Pred</i></td>
    <td>2</td>
  </tr>
</table>

Table 7: Definitions of the proto-role properties used in the original annotations in both Reisinger et al. (2015) and White et al. (2016). These were included in zero-shot prompts. *Created and destroyed were not labeled directly by annotators; instead, those labels are inferred from different combinations of the existed [before/during/after] labels.
<table>
  <tr>
    <th rowspan="2">Prompt template</th>
    <th colspan="6">GPT-4o Accuracy (% correct)</th>
    <th rowspan="2">All core roles<br> n = 44609</th>
  </tr>
  <tr>
    <th>ARG0<br>n = 13754</th>
    <th>ARG1<br>n = 22036</th>
    <th>ARG2<br>n = 7959</th>
    <th>ARG3<br>n = 451</th>
    <th>ARG4<br>n = 398</th>
    <th>ARG5<br>n = 11</th>
  </tr>
  <tr>
    <td>Pipeline</td>
    <td>31.39</td>
    <td>22.41</td>
    <td>12.92</td>
    <td>8.87</td>
    <td>6.78</td>
    <td>0.0</td>
    <td>23.2</td>
  </tr>
  <tr>
    <td>+instigate,chg-loc</td>
    <td>33.6 (+2.21)</td>
    <td>24.34 (+1.93)</td>
    <td>12.94 (+0.02)</td>
    <td>9.53 (+0.66)</td>
    <td>12.81 (+6.03)</td>
    <td>0.0 (+0.0)</td>
    <td>24.9 (+1.7)</td>
  </tr>
  <tr>
    <td>+volition,chg-state</td>
    <td>33.34 (+1.95)</td>
    <td>24.4 (+1.99)</td>
    <td>13.38 (+0.46)</td>
    <td>10.2 (+1.33)</td>
    <td>12.56 (+5.78)</td>
    <td>0.0 (+0.0)</td>
    <td>24.93 (+1.73)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2</td>
    <td>33.43 (+2.04)</td>
    <td>24.46 (+2.05)</td>
    <td>12.9 (−0.02)</td>
    <td>9.53 (+0.66)</td>
    <td>12.56 (+5.78)</td>
    <td>0.0 (+0.0)</td>
    <td>24.9 (+1.7)</td>
  </tr>
</table>

(a) Effect of SPRL on SRL accuracy for GPT-4o on the full SRL-SPRL pipeline, in which the model must predict everything and is penalized on errors earlier in the pipeline.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th colspan="2"># preds or args evaluated</th>
    <th colspan="2">Accuracy</th>
  </tr>
  <tr>
    <th>N ∈ Onto</th>
    <th>M ∈ GPT-retrieved</th>
    <th># correct</th>
    <th>Full-pipeline (out of N)</th>
    <th>Single task (out of M)</th>
  </tr>
  <tr>
    <td>Pred. span retrieval</td>
    <td>24,167</td>
    <td>n/a</td>
    <td>14,022</td>
    <td>55.72%</td>
    <td>n/a</td>
  </tr>
  <tr>
    <td>Pred. sense disambiguation</td>
    <td>24,167</td>
    <td>14,022</td>
    <td>12,472</td>
    <td>49.56%</td>
    <td>88.95%</td>
  </tr>
  <tr>
    <td>Arg. span retrieval</td>
    <td>44,609</td>
    <td>26,497 (±1.9 args per pred)</td>
    <td>11,809</td>
    <td>26.47%</td>
    <td>44.60%</td>
  </tr>
  <tr>
    <td>SRL</td>
    <td>44,609</td>
    <td>11,809</td>
    <td>10,351</td>
    <td>23.20%</td>
    <td>96.96%</td>
  </tr>
</table>

(b) Fine-grained SRL pipeline results on GPT-4o. The full-pipeline accuracy (what was used in Table 4) shows the score for a task, penalizing for misses in previous tasks. The single-task accuracy shows the score without penalizing the model for previous misses; i.e., the model is only scored on the subset of data that it got correct in the previous component of the pipeline.

Table 8: Performance of GPT-4o on the full SRL-SPRL pipeline.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">(a) Proto-agent properties</th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
  </tr>
  <tr>
    <th>instigated</th>
    <th>volition</th>
    <th>sentience</th>
    <th>chg-loc</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <th></th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
  </tr>
  <tr>
    <td>ARG0</td>
    <td>32.9</td>
    <td>24.4</td>
    <td>39.4</td>
    <td>21.5</td>
    <td>31.9</td>
    <td>23.9</td>
    <td>50.0</td>
    <td>23.7</td>
    <td>46.7</td>
    <td>25.0</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ARG1</td>
    <td>2.6</td>
    <td>14.9</td>
    <td>0.8</td>
    <td>16.9</td>
    <td>6.9</td>
    <td>14.9</td>
    <td>0.0</td>
    <td>14.2</td>
    <td>6.7</td>
    <td>13.4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ARG2</td>
    <td>0.0</td>
    <td>3.3</td>
    <td>0.0</td>
    <td>3.7</td>
    <td>1.7</td>
    <td>3.2</td>
    <td>0.0</td>
    <td>3.1</td>
    <td>0.0</td>
    <td>2.9</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ARG3-5</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>0.0</td>
    <td>1.2</td>
    <td>0.0</td>
    <td>1.1</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>0.0</td>
    <td>0.9</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>other</td>
    <td>61.8</td>
    <td>52.3</td>
    <td>57.5</td>
    <td>52.4</td>
    <td>51.7</td>
    <td>54.1</td>
    <td>47.5</td>
    <td>54.0</td>
    <td>40.0</td>
    <td>53.9</td>
    <td></td>
    <td></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">(b) Proto-patient properties</th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
  </tr>
  <tr>
    <th>chg-poss</th>
    <th>chg-state</th>
    <th>created</th>
    <th>destroyed</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <th></th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
    <th>T</th>
    <th>F</th>
  </tr>
  <tr>
    <td>ARG0</td>
    <td>0.0</td>
    <td>25.6</td>
    <td>34.1</td>
    <td>23.9</td>
    <td>33.3</td>
    <td>24.5</td>
    <td>7.7</td>
    <td>26.0</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ARG1</td>
    <td>0.0</td>
    <td>13.1</td>
    <td>10.2</td>
    <td>13.8</td>
    <td>7.6</td>
    <td>14.0</td>
    <td>23.1</td>
    <td>13.0</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ARG2</td>
    <td>0.0</td>
    <td>2.9</td>
    <td>1.1</td>
    <td>3.2</td>
    <td>3.0</td>
    <td>2.8</td>
    <td>0.0</td>
    <td>2.9</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ARG3-5</td>
    <td>0.0</td>
    <td>0.9</td>
    <td>0.0</td>
    <td>1.1</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>0.0</td>
    <td>0.9</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>other</td>
    <td>0.0</td>
    <td>53.6</td>
    <td>51.1</td>
    <td>54.0</td>
    <td>47.0</td>
    <td>54.5</td>
    <td>53.8</td>
    <td>53.6</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 9: % of predicted properties that co-occur with Llama-3.2-3B’s predicted ARGn on SPR1, for the All-args prompt variant with “n/a” SPRL responses not allowed. Denominator for percentages is the total number of arguments that property and value were assigned by the model. Columns do not add up to 100 because the model often outputs an SRL prediction that is not any of the numbered ARGs.
<table>
  <tr>
    <th rowspan="2">Prompt template</th>
    <th colspan="5">GPT-4o Accuracy (% correct)</th>
  </tr>
  <tr>
    <th>ARG0<br>n = 411</th>
    <th>ARG1<br>n = 476</th>
    <th>ARG2<br>n = 122</th>
    <th>ARG3-5<br>n = 45</th>
    <th>All core roles<br>n = 1054</th>
  </tr>
  <tr>
    <td colspan="6"><i>Model must predict everything and is penalized on errors earlier in the pipeline</i></td>
  </tr>
  <tr>
    <td>Pipeline</td>
    <td>29.44</td>
    <td>23.95</td>
    <td>4.92</td>
    <td>2.22</td>
    <td>22.96</td>
  </tr>
  <tr>
    <td>+instigate.chg-loc</td>
    <td>29.2 (<span style="color:red">-0.24</span>)</td>
    <td>24.16 (<span style="color:green">+0.21</span>)</td>
    <td>4.1 (<span style="color:red">-0.82</span>)</td>
    <td>2.22 (<span style="color:green">+0.0</span>)</td>
    <td>22.87 (<span style="color:red">-0.09</span>)</td>
  </tr>
  <tr>
    <td>+instigate.chg-state</td>
    <td>29.44 (<span style="color:green">+0.0</span>)</td>
    <td>23.95 (<span style="color:green">+0.0</span>)</td>
    <td>4.1 (<span style="color:red">-0.82</span>)</td>
    <td>2.22 (<span style="color:green">+0.0</span>)</td>
    <td>22.87 (<span style="color:red">-0.09</span>)</td>
  </tr>
  <tr>
    <td>+volition.chg-state</td>
    <td>29.44 (<span style="color:green">+0.0</span>)</td>
    <td>24.16 (<span style="color:green">+0.21</span>)</td>
    <td>4.1 (<span style="color:red">-0.82</span>)</td>
    <td>2.22 (<span style="color:green">+0.0</span>)</td>
    <td>22.96 (<span style="color:green">+0.0</span>)</td>
  </tr>
  <tr>
    <td>+volition.chg-loc</td>
    <td>29.2 (<span style="color:red">-0.24</span>)</td>
    <td>24.58 (<span style="color:green">+0.63</span>)</td>
    <td>4.92 (<span style="color:green">+0.0</span>)</td>
    <td>2.22 (<span style="color:green">+0.0</span>)</td>
    <td>23.15 (<span style="color:green">+0.19</span>)</td>
  </tr>
  <tr>
    <td>+inst.chg-loc.chg-state</td>
    <td>29.2 (<span style="color:red">-0.24</span>)</td>
    <td>24.37 (<span style="color:green">+0.42</span>)</td>
    <td>4.1 (<span style="color:red">-0.82</span>)</td>
    <td>2.22 (<span style="color:green">+0.0</span>)</td>
    <td>22.96 (<span style="color:green">+0.0</span>)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2</td>
    <td>29.44 (<span style="color:green">+0.0</span>)</td>
    <td>24.37 (<span style="color:green">+0.42</span>)</td>
    <td>4.92 (<span style="color:green">+0.0</span>)</td>
    <td>2.22 (<span style="color:green">+0.0</span>)</td>
    <td>23.15 (<span style="color:green">+0.19</span>)</td>
  </tr>
  <tr>
    <td colspan="6"><i>Predicate spans, predicate senses, and argument spans provided</i></td>
  </tr>
  <tr>
    <td>One arg per prompt</td>
    <td>79.56</td>
    <td>92.44</td>
    <td>86.89</td>
    <td>97.78</td>
    <td>87.0</td>
  </tr>
  <tr>
    <td>+instigate.chg-loc *</td>
    <td>82.48 (<span style="color:green">+2.92</span>)</td>
    <td>91.18 (<span style="color:red">-1.26</span>)</td>
    <td>87.7 (<span style="color:green">+0.81</span>)</td>
    <td>97.78 (<span style="color:green">+0.0</span>)</td>
    <td>87.67 (<span style="color:green">+0.67</span>)</td>
  </tr>
  <tr>
    <td>+volition.chg-state *</td>
    <td>75.67 (<span style="color:red">-3.89</span>)</td>
    <td>93.28 (<span style="color:green">+0.84</span>)</td>
    <td>81.97 (<span style="color:red">-4.92</span>)</td>
    <td>95.56 (<span style="color:red">-2.22</span>)</td>
    <td>85.2 (<span style="color:red">-1.8</span>)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2 *</td>
    <td>78.1 (<span style="color:red">-1.46</span>)</td>
    <td>93.07 (<span style="color:green">+0.63</span>)</td>
    <td>81.15 (<span style="color:red">-5.74</span>)</td>
    <td>97.78 (<span style="color:green">+0.0</span>)</td>
    <td>86.05 (<span style="color:red">-0.95</span>)</td>
  </tr>
  <tr>
    <td>+instigate.chg-loc *</td>
    <td>84.43 (<span style="color:green">+4.87</span>)</td>
    <td>89.92 (<span style="color:red">-2.52</span>)</td>
    <td>84.43 (<span style="color:red">-2.46</span>)</td>
    <td>95.56 (<span style="color:red">-2.22</span>)</td>
    <td>87.38 (<span style="color:green">+0.38</span>)</td>
  </tr>
  <tr>
    <td>+volition.chg-state *</td>
    <td>73.97 (<span style="color:red">-5.59</span>)</td>
    <td>92.86 (<span style="color:green">+0.42</span>)</td>
    <td>81.97 (<span style="color:red">-4.92</span>)</td>
    <td>97.78 (<span style="color:green">+0.0</span>)</td>
    <td>84.44 (<span style="color:red">-2.56</span>)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2 *</td>
    <td>76.16 (<span style="color:red">-3.4</span>)</td>
    <td>93.28 (<span style="color:green">+0.84</span>)</td>
    <td>82.79 (<span style="color:red">-4.1</span>)</td>
    <td>95.56 (<span style="color:red">-2.22</span>)</td>
    <td>85.48 (<span style="color:red">-1.52</span>)</td>
  </tr>
  <tr>
    <td>All args per prompt</td>
    <td>97.08</td>
    <td>94.75</td>
    <td>87.7</td>
    <td>86.67</td>
    <td>94.5</td>
  </tr>
  <tr>
    <td>+instigate.chg-loc</td>
    <td>97.81 (<span style="color:green">+0.73</span>)</td>
    <td>92.65 (<span style="color:red">-2.1</span>)</td>
    <td>85.25 (<span style="color:red">-2.45</span>)</td>
    <td>82.22 (<span style="color:red">-4.45</span>)</td>
    <td>93.36 (<span style="color:red">-1.14</span>)</td>
  </tr>
  <tr>
    <td>+instigate.chg-state</td>
    <td>96.35 (<span style="color:red">-0.73</span>)</td>
    <td>92.02 (<span style="color:red">-2.73</span>)</td>
    <td>83.61 (<span style="color:red">-4.09</span>)</td>
    <td>86.67 (<span style="color:green">+0.0</span>)</td>
    <td>92.5 (<span style="color:red">-2.0</span>)</td>
  </tr>
  <tr>
    <td>+volition.chg-state</td>
    <td>97.81 (<span style="color:green">+0.73</span>)</td>
    <td>94.12 (<span style="color:red">-0.63</span>)</td>
    <td>85.25 (<span style="color:red">-2.45</span>)</td>
    <td>84.44 (<span style="color:red">-2.23</span>)</td>
    <td>94.12 (<span style="color:red">-0.38</span>)</td>
  </tr>
  <tr>
    <td>+volition.chg-loc</td>
    <td>96.35 (<span style="color:red">-0.73</span>)</td>
    <td>93.07 (<span style="color:red">-1.68</span>)</td>
    <td>89.34 (<span style="color:green">+1.64</span>)</td>
    <td>82.22 (<span style="color:red">-4.45</span>)</td>
    <td>93.45 (<span style="color:red">-0.55</span>)</td>
  </tr>
  <tr>
    <td>+SPR1</td>
    <td>94.16 (<span style="color:red">-2.92</span>)</td>
    <td>91.6 (<span style="color:red">-3.15</span>)</td>
    <td>80.33 (<span style="color:red">-7.37</span>)</td>
    <td>73.33 (<span style="color:red">-13.34</span>)</td>
    <td>90.51 (<span style="color:red">-3.99</span>)</td>
  </tr>
  <tr>
    <td>+SPR1, allowing N/A</td>
    <td>95.62 (<span style="color:red">-1.46</span>)</td>
    <td>92.44 (<span style="color:red">-2.31</span>)</td>
    <td>84.43 (<span style="color:red">-3.27</span>)</td>
    <td>77.78 (<span style="color:red">-8.89</span>)</td>
    <td>92.13 (<span style="color:red">-2.37</span>)</td>
  </tr>
  <tr>
    <td>All args per prompt w/ PB guidelines</td>
    <td>96.84</td>
    <td>93.91</td>
    <td>81.15</td>
    <td>86.67</td>
    <td>93.26</td>
  </tr>
  <tr>
    <td>+SPR1</td>
    <td>95.62 (<span style="color:red">-1.22</span>)</td>
    <td>92.44 (<span style="color:red">-1.47</span>)</td>
    <td>79.51 (<span style="color:red">-1.64</span>)</td>
    <td>73.33 (<span style="color:red">-13.34</span>)</td>
    <td>91.37 (<span style="color:red">-1.89</span>)</td>
  </tr>
</table>

Table 10: Effect of eliciting SPRL along with SRL across prompt variants on SPR1. For the one-arg-per-prompt template, a gold star (*) indicates the use of gold annotated SPR properties, and a silver star (†) indicates the use of the model’s own predicted SPR properties. All other settings elicit the model’s own predictions.
<table>
  <tr>
    <th colspan="6">(a) GPT-4o accuracy, evaluated on well-formatted responses only, SPR1</th>
  </tr>
  <tr>
    <th>Prompt template</th>
    <th>ARG0<br>\( n = 389 \)</th>
    <th>ARG1<br>\( n = 450 \)</th>
    <th>ARG2<br>\( n = 102 \)</th>
    <th>ARG3-5<br>\( n = 35 \)</th>
    <th>All core roles<br>\( n = 976 \)</th>
  </tr>
  <tr>
    <td>All args per prompt</td>
    <td>97.17</td>
    <td>96.0</td>
    <td>90.2</td>
    <td>94.29</td>
    <td>95.8</td>
  </tr>
  <tr>
    <td>+instigate,chg-loc</td>
    <td>97.69 (+0.52)</td>
    <td>93.78 (−2.22)</td>
    <td>87.25 (−2.95)</td>
    <td>91.43 (−2.86)</td>
    <td>94.57 (−1.23)</td>
  </tr>
  <tr>
    <td>+instigate,chg-state</td>
    <td>96.4 (−0.77)</td>
    <td>92.67 (−3.33)</td>
    <td>85.29 (−4.91)</td>
    <td>94.29 (+0.0)</td>
    <td>93.44 (−2.36)</td>
  </tr>
  <tr>
    <td>+volition,chg-state</td>
    <td>98.2 (+1.03)</td>
    <td>94.67 (−1.33)</td>
    <td>86.27 (−3.93)</td>
    <td>91.43 (−2.86)</td>
    <td>95.08 (−0.72)</td>
  </tr>
  <tr>
    <td>+volition,chg-loc</td>
    <td>96.92 (−0.25)</td>
    <td>94.22 (−1.78)</td>
    <td>90.2 (+0.0)</td>
    <td>91.43 (−2.86)</td>
    <td>94.77 (−1.03)</td>
  </tr>
  <tr>
    <td>+SPR1</td>
    <td>97.69 (+0.52)</td>
    <td>95.56 (−0.44)</td>
    <td>87.25 (−2.95)</td>
    <td>94.29 (+0.0)</td>
    <td>95.49 (−0.31)</td>
  </tr>
  <tr>
    <td>+SPR1, allowing N/A</td>
    <td>97.43 (+0.26)</td>
    <td>94.89 (−1.11)</td>
    <td>90.2 (+0.0)</td>
    <td>97.14 (+2.85)</td>
    <td>95.49 (−0.31)</td>
  </tr>
  <tr>
    <td>All args per prompt w/ PB guidelines</td>
    <td>97.69</td>
    <td>95.11</td>
    <td>89.22</td>
    <td>94.29</td>
    <td>95.49</td>
  </tr>
  <tr>
    <td>+SPR1</td>
    <td>97.94 (+0.25)</td>
    <td>95.56 (+0.45)</td>
    <td>88.24 (−0.98)</td>
    <td>88.57 (−5.72)</td>
    <td>95.49 (+0.0)</td>
  </tr>
</table>

(b) Llama-3.2-3B accuracy, evaluated on well-formatted responses only and allowing fuzzy role matching, SPR1

<table>
  <tr>
    <th>Prompt template</th>
    <th>ARG0<br>\( n = 332 \)</th>
    <th>ARG1<br>\( n = 368 \)</th>
    <th>ARG2<br>\( n = 87 \)</th>
    <th>ARG3-5<br>\( n = 36 \)</th>
    <th>All core roles<br>\( n = 823 \)</th>
  </tr>
  <tr>
    <td>All args per prompt</td>
    <td>90.36</td>
    <td>67.12</td>
    <td>51.72</td>
    <td>8.33</td>
    <td>72.3</td>
  </tr>
  <tr>
    <td>+instigate,chg-loc</td>
    <td>92.77 (+2.41)</td>
    <td>57.61 (−9.51)</td>
    <td>24.14 (−27.58)</td>
    <td>8.33 (+0.0)</td>
    <td>66.1 (−6.2)</td>
  </tr>
  <tr>
    <td>+volition,chg-state</td>
    <td>94.58 (+4.22)</td>
    <td>56.79 (−10.33)</td>
    <td>13.79 (−37.93)</td>
    <td>2.78 (−5.55)</td>
    <td>65.13 (−7.17)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2</td>
    <td>93.67 (+3.31)</td>
    <td>57.34 (−9.78)</td>
    <td>17.24 (−34.48)</td>
    <td>13.89 (+5.56)</td>
    <td>65.86 (−6.44)</td>
  </tr>
  <tr>
    <td>+SPR1</td>
    <td>95.18 (+4.82)</td>
    <td>57.61 (−9.51)</td>
    <td>18.39 (−33.33)</td>
    <td>2.78 (−5.55)</td>
    <td>66.22 (−6.08)</td>
  </tr>
</table>

(c) GPT-4o accuracy, evaluated on well-formatted responses only, Ontonotes

<table>
  <tr>
    <th>Prompt template</th>
    <th>ARG0<br>\( n = 13595 \)</th>
    <th>ARG1<br>\( n = 21742 \)</th>
    <th>ARG2<br>\( n = 7654 \)</th>
    <th>ARG3-5<br>\( n = 849 \)</th>
    <th>All core roles<br>\( n = 43840 \)</th>
  </tr>
  <tr>
    <td>All args per prompt</td>
    <td>95.55</td>
    <td>94.98</td>
    <td>91.1</td>
    <td>84.92</td>
    <td>94.28</td>
  </tr>
  <tr>
    <td>+instigate,chg-loc</td>
    <td>95.67 (+0.12)</td>
    <td>91.65 (−3.33)</td>
    <td>83.76 (−7.34)</td>
    <td>85.98 (+1.06)</td>
    <td>91.41 (−2.87)</td>
  </tr>
  <tr>
    <td>+volition,chg-state</td>
    <td>95.22 (−0.33)</td>
    <td>90.41 (−4.57)</td>
    <td>80.61 (−10.49)</td>
    <td>86.45 (+1.53)</td>
    <td>90.11 (−4.17)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2</td>
    <td>95.28 (−0.27)</td>
    <td>92.94 (−2.04)</td>
    <td>86.4 (−4.7)</td>
    <td>86.93 (+2.01)</td>
    <td>92.41 (−1.87)</td>
  </tr>
  <tr>
    <td>+SPR1∩ SPR2, allowing N/A</td>
    <td>95.3 (−0.25)</td>
    <td>91.49 (−3.49)</td>
    <td>82.74 (−8.36)</td>
    <td>84.81 (−0.11)</td>
    <td>91.02 (−3.26)</td>
  </tr>
</table>

Table 11: SRL accuracy on subsets of data in which all responses are well-formatted, ensuring penalties are due to errors in meta-linguistic reasoning, rather than failures to correctly format output. All scores are for the *All-args* prompt variant.

Reviewer paper summaries: The paper investigates the capacity of LLMs to understand event roles through semantic proto-role labeling (SPRL) and semantic role labeling (SRL). By testing LLMs like GPT-4o and smaller models (Llama, OLMo, Qwen2.5, etc.), the study evaluates whether these models encode a latent hierarchy of event roles and how they compare to human semantic understanding. The key questions center around whether SPRL context enhances SRL accuracy and whether LLMs' errors in these tasks are correlated.

The authors evaluate multiple experimental setups to assess how LLMs handle SRL and SPRL, using different prompt formats.

Overall, this paper makes a contribution to computational semantics by critically evaluating LLMs' proto-role reasoning and the metalinguistic competence of LLMs. However, its paradoxical results (SPRL hurting SRL), lack of cross-linguistic testing,  and main focus on GPT-4—limit the strength of its conclusions. Addressing these gaps would enhance the study’s impact and applicability.
Summary of strengths: The study employs a diverse set of models, datasets (SPR1, Ontonotes), and experimental setups (pipeline, one-arg, all-args). This comprehensive evaluation strengthens its findings. 
The findings reinforce prior studies (Ettinger et al., 2023; Bonn et al., 2024) showing that LLMs struggle with complex, multi-step semantic reasoning, despite performing well on isolated tasks.

The paper uncovers an internal consistency in LLMs' proto-role reasoning but also highlights a key issue: SPRL errors don’t strongly correlate with SRL errors. This suggests that LLMs encode event roles but fail to apply them consistently.
Summary of weaknesses: - One of the core research questions investigates whether SPRL improves SRL performance. The results are mixed: while SPRL sometimes boosts performance for smaller models, it degrades performance for high-performing models like GPT-4o. This lack of a clear pattern weakens the argument that SPRL is beneficial for improving SRL and suggests that the integration of SPRL requires further refinement. 
- A significant portion of the analysis centers around GPT-4o, even though other models are evaluated. This focus makes it difficult to generalize conclusions to all LLMs, especially given that open-weight models like Llama 3 perform differently. Further discussion of model-specific behavior would improve the clarity of the findings.
- While the paper acknowledges potential training data contamination, it does not fully address whether GPT-4o’s high performance is due to memorization rather than actual event role understanding.
- The study uses multiple experimental setups (Pipeline, One-arg, All-args), but the rationale for choosing these specific configurations is not deeply explored. There is no clear justification for why some experimental setups degrade SRL performance. Could it be due to prompt complexity, token length limitations, or an inherent flaw in the integration of SPRL into SRL?
- The paper acknowledges that some SPRL properties are difficult to annotate reliably (e.g., manipulated by another has annotation inconsistencies). However, it does not sufficiently address whether these inconsistencies might be skewing results. If SPRL labels themselves are unreliable, this could explain why integrating SPRL sometimes hurts SRL rather than helping.
Comments, suggestions and typos: - Examples of the SRL pipeline (appendix A2) should be included in the main text, otherwise it lacks in clarity.
- Section 3: it is unclear why it focuses on datasets that are not used in the experiment (from line 219). 
- The authors do not specify well which Semantic Proto-Role Labeling (SPRL) properties they use, the details are somewhat scattered across different sections of the paper. They are cited in Table 1 and 3  but not in the main text. A list of these properties appears only in Table 7 of the paper (page 18, at the end of Appendix!).
- Table 7 missing reference
Overall assessment scores: 3.5