Joint End-to-End Semantic Proto-role Labeling
Year: 2023

Abstract:
Semantic proto-role labeling (SPRL) assigns properties to arguments based on a series of binary labels. While multiple studies have evaluated various approaches to SPRL, it has only been studied in-depth as a standalone task using gold predicate/argument pairs. How do SPRL systems perform as part of an information extraction pipeline? We model SPRL jointly with predicate-argument extraction using a deep transformer model. We find that proto-role labeling is surprisingly robust in this setting, with only a small decrease when using predicted arguments. We include a detailed analysis of each component of the joint system, and an error analysis to understand correlations in errors between system stages. Finally, we study the effects of annotation errors on SPRL.

heading: 1 Introduction
text: Semantic analyses of text have been framed (Gildea and Jurafsky, 2000) as extracting structured information in the form of predicates, arguments, and their relations, often called semantic roles. Multiple schemas have been proposed for structuring semantic roles, each with its own benefits and challenges. Semantic proto-roles (Dowty, 1991) offer a way to decompose traditional inventories of thematic roles into simple properties that are both easier to annotate and more generalizable to unseen arguments. These emerge from Dowty’s protorole theory, which assigns properties to arguments based on how agent-like (volition, sentience) or patient-like (change of state, was used) they are. For example, in the sentence “The boy threw a rock,” categorical role inventories assign argument “boy” the role Agent, and argument “rock” the role Patient. Work on decompositional semantics1 has formulated the task of semantic proto-role labeling as the assignment of 14 different binary properties to arguments (Reisinger et al., 2015a).
∗Work done during an internship at Bloomberg. 1http://decomp.io/
Multiple systems have been proposed for automatically assigning proto-roles to predicatearguments pairs in text (Opitz and Frank, 2019; Rudinger et al., 2018; Teichert et al., 2017; Tenney et al., 2019), which have established the feasibility and best practices for semantic proto-role labeling (SPRL). At the same time, this task continues to be either treated in total isolation, assuming gold predicates and arguments, or included in Universal Dependency Semantics (UDS) parsing pipelines (Stengel-Eskin et al., 2020, 2021), which has not included fine-grained analysis on semantic protorole properties themselves. How well does SPRL work when integrated into a semantic extraction pipeline? Are earlier errors compounded by SPRL? Are the same tokens challenging for each stage of the pipeline?
We answer these questions by constructing a joint multi-task model for identifying predicates and arguments, and assigning proto-role properties. Competitive with state-of-the-art for both dependency- and span-based SPRL evaluation, a careful component-wise analysis of our system allows us to make the following contributions. 1) Despite SPRL labeling errorful predicate argument predictions, our results are still competitive with having gold predicates and arguments, and far surpass the only previous work that predicts protoroles jointly with predicates and arguments. Future work should include SPRL scores with predicted arguments. 2) Errors in predicates and arguments do not negatively affect SPRL because the same tokens that are challenging for argument identification are challenging for SPRL. 3) We find that most SPRL errors come from arguments with annotator disagreement, which suggests that these are inherently hard; removing unskilled annotators doesn’t change performance, suggesting that conflict alone is not the source of the problem. We discuss implications for future work on SPRL after our analysis.
723

heading: 2 Semantic Proto-Roles
text: Semantic role labeling (SRL) was first formulated as a natural language understanding task by Gildea and Jurafsky (2000) and quickly proliferated (Surdeanu et al., 2003; Xue and Palmer, 2004; Pradhan et al., 2005) as resources and common evaluation frameworks were introduced (Carreras and Màrquez, 2004, 2005; Pradhan et al., 2007; Surdeanu et al., 2008; Hajič et al., 2009). SRL assigns relationships or roles of an argument and its predicate. Various labels build on different linguistic theories: label inventories that are small and coarse versus large and fine-grained. Common roles include Agent, Patient, Goal, and Location. Neuralbased systems that assign SRL jointly with other related tasks, such as predicate and argument identification, perform just as well or better than models trained on only SRL (Conia et al., 2021; Blloshmi et al., 2021; He et al., 2018; Strubell et al., 2018; Li et al., 2019).
Argument identification can be formulated as dependency-based (find the argument’s syntactic head) or span-based (find the entire argument span). The CoNLL 2004 and 2005 shared tasks (Carreras and Màrquez, 2004, 2005) used spans: an argument is correct only if all argument tokens are correctly identified with the correct argument role. CoNLL 2008 and 2009 (Surdeanu et al., 2008; Hajič et al., 2009) used a dependency-based method, which only requires that the syntactic head of the argument be tagged with the correct argument role. Understandably, span-based is more challenging and scores lag behind dependency-based systems (Li et al., 2019).
SPRL (Dowty, 1991) offers an alternative (Reisinger et al., 2015a) by decomposing traditional semantic roles into properties. The two protoroles are “cluster-concepts” called Proto-Agent and Proto-Patient, which each correspond to an inventory of properties. Certain properties (such as volition or instigation) tend to belong to Proto-Agents, while others (such as change of state and was used) tend to belong to Proto-Patients. This analysis offers increased granularity but without sparsification of the training data.
The state-of-the-art SPRL dependency-based system fine-tunes BERT (Devlin et al., 2019) with a multi-layer Perceptron to assign labels using a linear combination of different BERT layer embeddings (Tenney et al., 2019). For span-based, the leading system uses an attention-based ensemble
and trainable “argument marker embeddings” to indicate which tokens are arguments (Opitz and Frank, 2019). Stengel-Eskin et al. (2020) and Stengel-Eskin et al. (2021) jointly predict UDS graph structures (i.e., the spans of predicates and arguments) with all UDS properties, including semantic proto-roles. Both use a sequence-to-graph transductive model, and Stengel-Eskin et al. (2021) is able to improve the transductive model by integrating transformer architecture. Systems are rarely evaluated in both dependency- and span-based settings, and none have been evaluated on anything but gold predicates and arguments.

heading: 3 Data
text: We report results on two English-language datasets for SPRL: SPR1 (Reisinger et al., 2015b) and SPR2 (White et al., 2016). SPR1 contains 4,912 Wall Street Journal sentences from PropBank (Kingsbury and Palmer, 2002; Palmer et al., 2005; Gildea and Palmer, 2002) annotated by a single annotator based on a set of 18 proto-role properties. 9,738 arguments were annotated for the likelihood (on a Likert scale from 1 to 5) that a property holds for that argument. SPR2 contains 2,758 English Web Treebank (Bies et al., 2012) sentences annotated for a smaller set of 14 properties using a revised, streamlined protocol. In this release, multiple annotators ensured two-way redundancy for each property judgment.
Following previous work (Opitz and Frank, 2019; Rudinger et al., 2018; Teichert et al., 2017; Tenney et al., 2019), we formulate SPRL as a 18 (SPR1) or 14 (SPR2) way multi-label binary classification problem and map Likert labels {1, 2, 3} to 0, and {4, 5} to 1. The task has also been formulated as a regression problem in which SPRL scores are predicted as continuous values (Opitz and Frank, 2019; Rudinger et al., 2018), but we do not include this formulation as a part of our analysis. We additionally map judgments labeled “inapplicable” to 0 to ensure consistency with previous work. We use standard train/dev/tests splits provided in the data. We additionally do analysis on inter-annotator agreement in SPR2 shown in Appendix B.2.

heading: 4 Joint End-to-End SPRL
text: We construct a joint end-to-end SPRL system based on BERT (Devlin et al., 2019) with classification heads for each sub-task (Figure 1). We fine-tune
BERT and the encoder parameters are shared across tasks. We favor BERT as opposed to newer encoders for a direct comparison to Tenney et al. (2019) and Opitz and Frank (2019).2
We construct representations of the input sentences to enable the system to efficiently identify predicates, arguments, and SPRL. For each sentence, we construct an instance with a candidate predicate prepended onto the sentence with a separator. We use linear classification heads3 with sigmoid functions to produce classification probabilities for each token. We place a classification head on the prepended predicate token to determine if it is a predicate. For argument and SPRL, we use separate classification heads on each token in the sentence. Dependency-based models predict argument for the dependency head only, whereas span-based models use an IOE tagging and softmax outputs. The resulting argument-tagged subwords are pooled and concatenated with the predicate representation taken from the sentence into the SPRL binary classification heads. Since we create an input for each possible predicate, we reduce the number of examples originating from incorrect predicates by only using predicate candidates that were verbs. Full model and training details appear in Appendix A.3.
Our model architecture is very similar to Tenney et al. (2019), except we extend it for predicate and argument identification. Additionally, we do not use a linear combination of BERT layers, instead taking BERT’s last layer of BERT.4
2Newer encoders would likely do better, but our goal is an analysis and not obtaining new state-of-the-art scores.
3We experimented with multi-layer Perceptrons (ReLU activation) as heads, but found no consistent improvement.
4In 2019, it was common to use a combination of layer

heading: 5 Experiments
text: We run multiple experiments to isolate the behavior of different components of our system, such as training on only SPRL, as well as the full pipeline. For all experiments, we train both a dependencybased and span-based model.
embeddings based on ELMO, but since then systems use only the final output layer. Results in Table 1 verify that layer combinations are not necessary for this task.
5This score is for BERT-base, which we use in our experiments. The authors gained a small boost of 0.3 F1 points by using BERT-large.
6Scores for the system reported by Stengel-Eskin et al. (2021) were obtained using their released code. These scores represent the best-performing setting: TFMR + EN, tuning the top 8 layers of BERT.
Scoring SPRL is typically reported as micro/macro averaged F1 across the individual SPR binary properties. We report Gold F1 scores that assume the previous stages of the pipeline produced correct predicates and arguments. However, when considering SPRL run on predicted predicates and arguments, we need to adjust the scoring such that we penalize the SPRL score due to mistakes earlier in the pipeline. For other tasks, such as entity linking, we can simply mark a link as “missed” if we fail to recognize an entity with a NER system. However, because SPRL is a binary classification task, scoring is more complex.
We consider two different SPRL scoring methods for false negative predicate or arguments: (1) a lenient score that assumes 0 for all properties, which means that missed arguments do not have any of the properties. (2) A strict score that forces the label incorrect for all properties, which assumes we get all property predictions wrong thereby marking them incorrect. We do not modify the SPRL score for false positive arguments (for which there are no gold labels) since this would change the set of arguments over which each run of the system is evaluated. For example, in the sentence “Bob sat on the chair and I laid on the ground”, if the model predicts that “ground” is an argument for “sat”, then the model would produce property predictions for “ground” even though there are no annotations for this token. Those predictions are ignored entirely, because it is not guaranteed that other runs will also include predictions for this token.
We evaluate each component of our system separately to determine the effects of the pipeline. (1) Train only the SPRL classifiers using gold predicates and arguments. (2) Train arguments and SPRL assuming gold predicates. (3) Train predicates, arguments, and SPRL using inputs first filtered to consider only verbs as predicates. We repli-
cate this training for both span and dependencybased predictions. For each setting, we evaluate under different conditions by decoding assuming gold or predicted labels from earlier in the pipeline.

heading: 6 Results
text: We present an overview of the results for our joint system, but full results appear in Appendix B. Table 1 shows our SPRL system compares favorably to previous work. We show three systems: trained on only SPRL, trained on arguments and SPRL, and trained on predicates, arguments, and SPRL. In all cases, we decode SPRL predictions assuming gold predicates and arguments. Our model matches or surpasses previous span and dependency results on SPR1, but lags slightly behind on span-based SPR2. This confirms previous work that found SPR2 more difficult than SPR1, perhaps because SPR2 has less data and more complex predicates and arguments.
Table 2 shows performance using our strict pipeline scoring, in which we map proto-role property predictions to incorrect for false negative arguments. The drop in F1 from using gold labels is shown in red. While we do worse in a pipeline, with the largest gap being 8 points for dependency-based SPR1, jointly learning predicates slightly improves strict F1 performance on SPRL in the dependencybased models, but degrades performance in the span-based models. Furthermore, SPR1 suffers a larger drop in the strict scoring regime than SPR2, perhaps because SPR2 models were already predicting many of the “harder” arguments incorrectly.
How are SPRL errors related to mistakes earlier in the pipeline? SPRL performance was much lower for arguments that would have been missed earlier in the pipeline. (See Table 3.) Table 2 shows this effect: models with smaller drops from gold were already making errors on incorrect arguments, whereas models with larger drops were likely bet-
ter at handling “difficult” examples. These difficult examples seem to correlate with annotation difficulty. We measure the performance of the system on annotation subsets based on the difference in Likert scores from the annotator. The larger the disagreement in Likert scores betweeen annotators, the worse the model performance (Appendix B.1.2.) To rule out the role of poor annotators, we removed those who had low inter-annotator agreement with others. However, this had almost no effect on F1, suggesting that it is the examples themselves that are challenging, and not the quality of the annotations. Perhaps these arguments are challenging for both tasks, or possibly the BERT encoder learns a poor representation of them. Fortunately, this means that when arguments are correctly discovered, SPRL does a good job on them and that correcting errors may improve both tasks.
Additionally, since we follow previous work by collapsing proto-role annotations marked “inapplicable” into the 0 class, we investigate the effect of excluding “inapplicable” property annotations in Table 7 and find a consistent boost of at least 3 F1 points by excluding inapplicable annotations, suggesting future work may benefit from handling applicability judgements differently, such as in Stengel-Eskin et al. (2020, 2021), who use a hurdle model in which a first classifier determines whether or not a property applies before making the property value judgement. Together, the effects of Likert disagreements and inapplicability of proto-role annotations additionally suggests that normalizing the different annotator responses, as in White et al. (2020), who use a mixed effects model, might lead to better outcomes in SPRL. See Appendix B.1 for a more detailed analysis of all results.

heading: 7 Discussion
text: Our end-to-end SPRL system demonstrates the efficacy of SPRL when combined with a full system. We are competitive with both span-based and dependency-based models and find that joint identification of predicates and arguments still produces a high-performing SPRL system. Future work should evaluate this setting, using both spanand dependency-based models and our proposed scoring method. Furthermore, our work points to the need for focused improvement on challenging arguments, which is harming both argument identification and SPRL. Do these errors show the limits of SPRL since annotators also get them wrong? Do we need better encoder training? Will downstream tasks that consume SPRL labels be robust to these errors? What is the feasibility of a reinforcement learning system that trains on the model’s own output? These questions remain for future work.

heading: Limitations
text: Our analysis of the behavior of SPRL focused on intrinsic task scores. Higher SPRL scores suggest a better system. In practice, we do not yet understand how these scores affect downstream uses of SPRL labels. Furthermore, SPRL datasets are relatively small and are English only. As we are limited to the labels in the existing datasets, we are uncertain about how our results would generalize to larger datasets, new domains, and other languages.

heading: Ethics Statement
text: When deploying a system such as ours on real text, e.g., news, one should carefully consider the implications of labeling real entities with certain protorole properties. For example, answering the question of whether or not an actor instigated some action could have serious ramifications in the real world. Care should be taken so that such cases might be, for example, flagged for human review.

heading: Acknowledgements
text: We thank Elias Stengel-Eskin, Benjamin Van Durme, Igor Malioutov, and Leslie Barrett for their helpful comments and feedback throughout conversations surrounding the project. We additionally acknowledge anonymous Bloomberg employees for assistance in reviewing the paper. Finally, we thank the ACL reviewers for their careful consideration and invaluable feedback.

heading: A Training details
text: We simplify the complex task of joint predicateargument-proto-role learning, as the space of possible predicates, arguments, and proto-role labels is O(|R|n3) for a sentence of n tokens and a set of proto-role properties R. (There are O(n) possible predicates and O(n2) possible argument spans.) First, we made the decision not to train the model on its own output—ie, we use an oracle to identify gold predicate and argument tokens so that nonpredicate sequences and non-argument tokens are ignored in the loss step.

heading: A.1 Pre-processing
text: We shift some of the complexity to the data processing step before any learning occurs by crafting sequences such that only one predicate is considered at a time: for example, the sentence “He stole my toy!” would be split into four separate data points:
He <SEP> He stole my toy! stole <SEP> He stole my toy! my <SEP> He stole my toy!
toy! <SEP> He stole my toy!
The model learns to focus on the first token as the candidate predicate of the sentence. For example, in the sequence stole <SEP> He stole my toy, the model must answer the questions: If stole is the predicate, what are the arguments of the sentence, and what are their proto-role properties? We truncate sequences to a fixed maximum length of 50 and pad shorter sequences to the right.

heading: A.2 Predicate filtering
text: The number of training instances would be quite large if every token of every sentence was used as a predicate candidate, as above. So we undergo a predicate filtering step in which we only select tokens that are labeled as verbs (ie, anything with a POS tag beginning with VB) in the datasets. For every dataset and split, this initial predicate filtering step has a recall of 100. After filtering, the number of training instances in SPR1 is reduced to 8,999 from 83,789, and in SPR2, is reduced to 7,452 from 46,138. Model output for argument identification and SPRL on false positive predicates is ignored in the loss function and evaluations. Table 4 shows full results for the predicate filtering.

heading: A.3 Hyperparameters
text: Using the hyperparameters from previous work as a starting point, we fine-tuned the learning rate and batch size and then kept them fixed based on the highest validation macro-F1 for final experiments. We report scores from a single run from each final experiment. We use a batch size of 8, run for 30 epochs with no early stopping, and choose scores based on the best validation macro-F1. Our learning rate is 0.00001. For each property, we apply loss weights equal to the inverse frequency of that property. Our model, which uses BERT-base, contains 109M trainable parameters, and took roughly 2-6 hours to train on a single GPU depending on the size of the dataset, whether or not we were predicting predicates, and whether or not we were predicting argument spans. We used the Pytorch Lightning7 framework to build and train our model.

heading: B Full results
text: The full proto-role property identification results for all model configurations using a linear classification head can be found arranged in a grid in Table 5. The grid shows the three training methods and the three scoring methods. For training, we have three columns indicating whether or not the model was trained to predict predicates, arguments, and proto-role properties. For training settings in which we do not train the model to predict predicates, note that we do not create sequences with incorrect predicates (ie, the model would never see the sequence He <SEP> He stole my toy!) and the model only sees instances with correct predicates.
7https://www.pytorchlightning.ai, Apache-2.0
For scoring, we show three different scoring methods: (1) gold scores, which assume correct predicates and arguments earlier in the pipeline, for direct comparison to previous work; (2) lenient scores, which assume 0 for all proto-role properties, treating SPRL as a “proto-role retrieval” task; and (2) strict scores, which map proto-role properties to the wrong label if predicates and arguments are falsely predicted as 0 earlier in the pipeline. We do not modify the SPRL score for false positives in predicate and argument identification since this would chnage the set of arguments over which the system is evaluated. The corresponding results for predicate and argument identification can be found in Table 6.

heading: B.1 Evaluating on subsets of data
text: To attempt to tease out the reasons for various errors in the model predictions, we take varying subsets of the data and evaluate separately on each subset. We report sizes of the different subsets we evaluate in Table 8.
B.1.1 Arguments predicted correctly and incorrectly
To further investigate the question of how errors earlier in the pipeline propagate later in the pipeline, we take a subset of arguments which the model predicted correctly and a subset of arguments which the model predicted incorrectly, and calculate the F1 scores for each subset. We report these scores in Table 3. We see large differences in the F1 scores between these subsets, suggesting that arguments that are difficult for the model to identify are also difficult for proto-role property classification.
An example of an argument that all configurations of our SPR2 models struggled with is italicized in the sentence below, with the predicate in bold:
I like I Move CA - Los Angeles Movers, they moved me before, but this time they were awesome :)
None of the models were able to retrieve this argument correctly (neither the head, nor the span). They all made mistakes on at least some proto-role
property predictions: common mistakes among all configurations of the model included false negatives for sentient, false positives for awareness, and false positives for change of location.
Interestingly, only the span-based models predicted that the argument was not sentient, showing that the non-head tokens in the span confused the model. On the other hand, both the dependencybased models understandably predicted that the head of the argument, Movers, changed location, while the span-based models did not make this mistake.
We notice that this sentence might have been difficult for annotators to judge, so we proceed with evaluations of subsets based on annotator judgements and agreement to tease out the association between examples that are difficult for annotators and examples that are difficult for the model.

heading: B.1.2 Differences in Likert ratings
text: For SPR2, which is doubly-annotated, we hypothesized that we could locate examples that are difficult for the model to classify by the difference between the two annotators’ Likert ratings in each property judgement. We construct several different subsets of data, which we refer to as LDi. LDi is the subset of property annotations in which the difference between Likert ratings between two annotators is exactly i. We show F1 scores for different combinations of these subsets in Figure 2, and provide the sizes of each subset in Table 8. We see that the score on the subset containing only property judgements with complete agreement between annotators is far higher than all other scores. As we add property judgements with larger and larger
disagreement between annotators, the scores drop substantially.
B.1.3 Pairwise inter-annotator agreement In section B.2, we show inter-annotator agreement κ scores averaged over each property. We also calculate κ pairwise for each annotator and report these scores in Table 10. We then investigate the extent to which including annotations by annotators with low pairwise agreement affects F1 by creating subsets of data which excludes annotations by annotators with an inter-annotator agreement below some cutoff. We report these scores in Figure 3. Surprisingly, we note that excluding annotators with low pairwise inter-annotator agreement has almost no affect on the F1 score, suggesting that annotator “skill” is less important than the difficulty of each example in SPRL F1.

heading: B.1.4 Applicability judgements
text: Finally, we investigate the extent to which the applicability judgements correlate with difficulty of property prediction. For both SPR1 and SPR2, an “applicable” judgement, indicating whether or not the proto-role property was applicable to the argument in the context of the sentence, was collected for each property in addition to the Likert judgements. As a reminder, annotations marked inapplicable were collapsed into the 0 class regardless of the Likert rating. Thus, in Table 7, we show F1 scores on the subset of annotations marked “applicable” by both annotators (or, in the case of SPR1, the single annotator) versus F1 scores on the entire
dataset. We see a consistent boost of at least 3 F1 points by only evaluating on applicable annotations.
B.2 Inter-annotator agreement
A possible limitation of the currently available SPR data is relatively low average inter-annotator agreement. White et al. (2016) report an agreement of 0.617 using Spearman’s rank correlation coefficient for SPR2. However, this agreement was measured over the Likert scores, which our model will not be predicting. We re-measured both the Likert data and the collapsed binary data using Cohen’s kappa on a per-property basis. We see in Table 9 that when measuring agreement using Cohen’s kappa, collapsing the Likert labels to {0, 1} improves the agreement significantly, resulting in every property having at least κ ≥ 0.64.
We also calculated each annotator’s Cohen’s kappa score pairwise against every other annotator

heading: Dataset Property Likert κ Binary κ
text: (and averaged). We then experimented with scoring our models on a subset of the data in which only judgements by annotators with a certain interannotator agreement were kept. The inter-annotator agreement scores used in these experiments can be found in Table 10.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations (unnumbered, page 5)

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics (unnumbered, page 5)

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1 - Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 and Appendix A

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 (BERT) and Appendix A.3 (Pytorch Lightning)

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix A.3
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 - Data
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 - Data
C 3 Did you run computational experiments? Section 5 - Experiments, Appendix A

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A.3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix A.3 - since we only report from a single run, we do not provide error bars around results

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future work should evaluate this setting, using both spanand dependency-based models and our proposed scoring method. Furthermore, our work points to the need for focused improvement on challenging arguments, which is harming both argument identification and SPRL. Do these errors show the limits of SPRL since annotators also get them wrong? Do we need better encoder training? Will downstream tasks that consume SPRL labels be robust to these errors? What is the feasibility of a reinforcement learning system that trains on the model’s own output? These questions remain for future work