A script is “a standardized sequence of events that describes some stereotypical human activity such as going to a restaurant or visiting a doctor” BIBREF0 . Script events describe an action/activity along with the involved participants. For example, in the script describing a visit to a restaurant, typical events are entering the restaurant, ordering food or eating. Participants in this scenario can include animate objects like the waiter and the customer, as well as inanimate objects such as cutlery or food.Script knowledge has been shown to play an important role in text understanding (cullingford1978script, miikkulainen1995script, mueller2004understanding, Chambers2008, Chambers2009, modi2014inducing, rudinger2015learning). It guides the expectation of the reader, supports coreference resolution as well as common-sense knowledge inference and enables the appropriate embedding of the current sentence into the larger context. Figure 1 shows the first few sentences of a story describing the scenario taking a bath. Once the taking a bath scenario is evoked by the noun phrase (NP) “a bath”, the reader can effortlessly interpret the definite NP “the faucet” as an implicitly present standard participant of the taking a bath script. Although in this story, “entering the bath room”, “turning on the water” and “filling the tub” are explicitly mentioned, a reader could nevertheless have inferred the “turning on the water” event, even if it was not explicitly mentioned in the text. Table 1 gives an example of typical events and participants for the script describing the scenario taking a bath.A systematic study of the influence of script knowledge in texts is far from trivial. Typically, text documents (e.g. narrative texts) describing various scenarios evoke many different scripts, making it difficult to study the effect of a single script. Efforts have been made to collect scenario-specific script knowledge via crowdsourcing, for example the OMICS and SMILE corpora (singh2002open, Regneri:2010, Regneri2013), but these corpora describe script events in a pointwise telegram style rather than in full texts.This paper presents the InScript corpus (Narrative Texts Instantiating Script structure). It is a corpus of simple narrative texts in the form of stories, wherein each story is centered around a specific scenario. The stories have been collected via Amazon Mechanical Turk (M-Turk). In this experiment, turkers were asked to write down a concrete experience about a bus ride, a grocery shopping event etc. We concentrated on 10 scenarios and collected 100 stories per scenario, giving a total of 1,000 stories with about 200,000 words. Relevant verbs and noun phrases in all stories are annotated with event types and participant types respectively. Additionally, the texts have been annotated with coreference information in order to facilitate the study of the interdependence between script structure and coreference.The InScript corpus is a unique resource that provides a basis for studying various aspects of the role of script knowledge in language processing by humans. The acquisition of this corpus is part of a larger research effort that aims at using script knowledge to model the surprisal and information density in written text. Besides InScript, this project also released a corpus of generic descriptions of script activities called DeScript (for Describing Script Structure, Wanzare2016). DeScript contains a range of short and textually simple phrases that describe script events in the style of OMICS or SMILE (singh2002open, Regneri:2010). These generic telegram-style descriptions are called Event Descriptions (EDs); a sequence of such descriptions that cover a complete script is called an Event Sequence Description (ESD). Figure 2 shows an excerpt of a script in the baking a cake scenario. The figure shows event descriptions for 3 different events in the DeScript corpus (left) and fragments of a story in the InScript corpus (right) that instantiate the same event type.We selected 10 scenarios from different available scenario lists (e.g. Regneri:2010 , VanDerMeer2009, and the OMICS corpus BIBREF1 ), including scripts of different complexity (Taking a bath vs. Flying in an airplane) and specificity (Riding a public bus vs. Repairing a flat bicycle tire). For the full scenario list see Table 2 .Texts were collected via the Amazon Mechanical Turk platform, which provides an opportunity to present an online task to humans (a.k.a. turkers). In order to gauge the effect of different M-Turk instructions on our task, we first conducted pilot experiments with different variants of instructions explaining the task. We finalized the instructions for the full data collection, asking the turkers to describe a scenario in form of a story as if explaining it to a child and to use a minimum of 150 words. The selected instruction variant resulted in comparably simple and explicit scenario-related stories. In the future we plan to collect more complex stories using different instructions. In total 190 turkers participated. All turkers were living in the USA and native speakers of English. We paid USD $0.50 per story to each turker. On average, the turkers took 9.37 minutes per story with a maximum duration of 17.38 minutes.Statistics for the corpus are given in Table 2 . On average, each story has a length of 12 sentences and 217 words with 98 word types on average. Stories are coherent and concentrate mainly on the corresponding scenario. Neglecting auxiliaries, modals and copulas, on average each story has 32 verbs, out of which 58% denote events related to the respective scenario. As can be seen in Table 2 , there is some variation in stories across scenarios: The flying in an airplane scenario, for example, is most complex in terms of the number of sentences, tokens and word types that are used. This is probably due to the inherent complexity of the scenario: Taking a flight, for example, is more complicated and takes more steps than taking a bath. The average count of sentences, tokens and types is also very high for the baking a cake scenario. Stories from the scenario often resemble cake recipes, which usually contain very detailed steps, so people tend to give more detailed descriptions in the stories.For both flying in an airplane and baking a cake, the standard deviation is higher in comparison to other scenarios. This indicates that different turkers described the scenario with a varying degree of detail and can also be seen as an indicator for the complexity of both scenarios. In general, different people tend to describe situations subjectively, with a varying degree of detail. In contrast, texts from the taking a bath and planting a tree scenarios contain a relatively smaller number of sentences and fewer word types and tokens. Both planting a tree and taking a bath are simpler activities, which results in generally less complex texts.The average pairwise word type overlap can be seen as a measure of lexical variety among stories: If it is high, the stories resemble each other more. We can see that stories in the flying in an airplane and baking a cake scenarios have the highest values here, indicating that most turkers used a similar vocabulary in their stories.In general, the response quality was good. We had to discard 9% of the stories as these lacked the quality we were expecting. In total, we selected 910 stories for annotation.This section deals with the annotation of the data. We first describe the final annotation schema. Then, we describe the iterative process of corpus annotation and the refinement of the schema. This refinement was necessary due to the complexity of the annotation.For each of the scenarios, we designed a specific annotation template. A script template consists of scenario-specific event and participant labels. An example of a template is shown in Table 1 . All NP heads in the corpus were annotated with a participant label; all verbs were annotated with an event label. For both participants and events, we also offered the label unclear if the annotator could not assign another label. We additionally annotated coreference chains between NPs. Thus, the process resulted in three layers of annotation: event types, participant types and coreference annotation. These are described in detail below.As a first layer, we annotated event types. There are two kinds of event type labels, scenario-specific event type labels and general labels. The general labels are used across every scenario and mark general features, for example whether an event belongs to the scenario at all. For the scenario-specific labels, we designed an unique template for every scenario, with a list of script-relevant event types that were used as labels. Such labels include for example ScrEv_close_drain in taking a bath as in Example UID10 (see Figure 1 for a complete list for the taking a bath scenario)I start by closing $_{\textsc {\scriptsize ScrEv\_close\_drain}}$ the drain at the bottom of the tub.The general labels that were used in addition to the script-specific labels in every scenario are listed below:ScrEv_other. An event that belongs to the scenario, but its event type occurs too infrequently (for details, see below, Section "Modification of the Schema" ). We used the label “other" because event classification would become too finegrained otherwise.Example: After I am dried I put my new clothes on and clean up $_{\textsc {\scriptsize ScrEv\_other}}$ the bathroom.RelNScrEv. Related non-script event. An event that can plausibly happen during the execution of the script and is related to it, but that is not part of the script.Example: After finding on what I wanted to wear, I went into the bathroom and shut $_{\textsc {\scriptsize RelNScrEv}}$ the door.UnrelEv. An event that is unrelated to the script.Example: I sank into the bubbles and took $_{\textsc {\scriptsize UnrelEv}}$ a deep breath.Additionally, the annotators were asked to annotate verbs and phrases that evoke the script without explicitly referring to a script event with the label Evoking, as shown in Example UID10 . Today I took a bath $_{\textsc {\scriptsize Evoking}}$ in my new apartment.As in the case of the event type labels, there are two kinds of participant labels: general labels and scenario-specific labels. The latter are part of the scenario-specific templates, e.g. ScrPart_drain in the taking a bath scenario, as can be seen in Example UID15 .I start by closing the drain $_{\textsc {\scriptsize ScrPart\_drain}}$ at the bottom of the tub.The general labels that are used across all scenarios mark noun phrases with scenario-independent features. There are the following general labels:ScrPart_other. A participant that belongs to the scenario, but its participant type occurs only infrequently.Example: I find my bath mat $_{\textsc {\scriptsize ScrPart\_other}}$ and lay it on the floor to keep the floor dry.NPart. Non-participant. A referential NP that does not belong to the scenario.Example: I washed myself carefully because I did not want to spill water onto the floor $_{\textsc {\scriptsize NPart}}$ .labeledSuppVComp. A support verb complement. For further discussion of this label, see Section "Special Cases" Example: I sank into the bubbles and took a deep breath $_{\textsc {\scriptsize SuppVComp}}$ .Head_of_Partitive. The head of a partitive or a partitive-like construction. For a further discussion of this label cf. Section "Special Cases" Example: I grabbed a bar $_{\textsc {\scriptsize Head\_of\_Partitive}}$ of soap and lathered my body.No_label. A non-referential noun phrase that cannot be labeled with another label. Example: I sat for a moment $_{\textsc {\scriptsize No\_label}}$ , relaxing, allowing the warm water to sooth my skin.All NPs labeled with one of the labels SuppVComp, Head_of_Partitive or No_label are considered to be non-referential. No_label is used mainly in four cases in our data: non-referential time expressions (in a while, a million times better), idioms (no matter what), the non-referential “it” (it felt amazing, it is better) and other abstracta (a lot better, a little bit).In the first annotation phase, annotators were asked to mark verbs and noun phrases that have an event or participant type, that is not listed in the template, as MissScrEv/ MissScrPart (missing script event or participant, resp.). These annotations were used as a basis for extending the templates (see Section "Modification of the Schema" ) and replaced later by newly introduced labels or ScrEv_other and ScrPart_other respectively.All noun phrases were annotated with coreference information indicating which entities denote the same discourse referent. The annotation was done by linking heads of NPs (see Example UID21 , where the links are indicated by coindexing). As a rule, we assume that each element of a coreference chain is marked with the same participant type label.I $ _{\textsc {\scriptsize Coref1}}$ washed my $ _{\textsc {\scriptsize Coref1}}$ entire body $ _{\textsc {\scriptsize Coref2}}$ , starting with my $ _{\textsc {\scriptsize Coref1}}$ face $ _{\textsc {\scriptsize Coref3}} $ and ending with the toes $ _{\textsc {\scriptsize Coref4}} $ . I $ _{\textsc {\scriptsize Coref1}}$ always wash my $ _{\textsc {\scriptsize Coref1}}$ toes $_{\textsc {\scriptsize Coref4}}$ very thoroughly ...The assignment of an entity to a referent is not always trivial, as is shown in Example UID21 . There are some cases in which two discourse referents are grouped in a plural NP. In the example, those things refers to the group made up of shampoo, soap and sponge. In this case, we asked annotators to introduce a new coreference label, the name of which indicates which referents are grouped together (Coref_group_washing_tools). All NPs are then connected to the group phrase, resulting in an additional coreference chain.I $ _{\textsc {\scriptsize Coref1}}$ made sure that I $ _{\textsc {\scriptsize Coref1}}$ have my $ _{\textsc {\scriptsize Coref1}}$ shampoo $ _{\textsc {\scriptsize Coref2 + Coref\_group\_washing\_tools}}$ , soap $_{\textsc {\scriptsize Coref3 + Coref\_group\_washing\_tools}}$ and sponge $ _{\textsc {\scriptsize Coref4 + Coref\_group\_washing\_tools}}$ ready to get in. Once I $ _{\textsc {\scriptsize Coref1}}$ have those things $ _{\textsc {\scriptsize Coref\_group\_washing\_tools}}$ I $ _{\textsc {\scriptsize Coref1}}$ sink into the bath. ... I $ _{\textsc {\scriptsize Coref1}}$ applied some soap $ _{\textsc {\scriptsize Coref1}}$0 on my $ _{\textsc {\scriptsize Coref1}}$1 body and used the sponge $ _{\textsc {\scriptsize Coref1}}$2 to scrub a bit. ... I $ _{\textsc {\scriptsize Coref1}}$3 rinsed the shampoo $ _{\textsc {\scriptsize Coref1}}$4 . Example UID21 thus contains the following coreference chains: Coref1: I $ _{\textsc {\scriptsize Coref1}}$5 I $ _{\textsc {\scriptsize Coref1}}$6 my $ _{\textsc {\scriptsize Coref1}}$7 I $ _{\textsc {\scriptsize Coref1}}$8 I $ _{\textsc {\scriptsize Coref1}}$9 I $ _{\textsc {\scriptsize Coref1}}$0 my $ _{\textsc {\scriptsize Coref1}}$1 ICoref2: shampoo $\rightarrow $ shampooCoref3: soap $\rightarrow $ soapCoref4: sponge $\rightarrow $ spongeCoref_group_washing_ tools: shampoo $\rightarrow $ soap $\rightarrow $ sponge $\rightarrow $ thingsThe templates were carefully designed in an iterated process. For each scenario, one of the authors of this paper provided a preliminary version of the template based on the inspection of some of the stories. For a subset of the scenarios, preliminary templates developed at our department for a psycholinguistic experiment on script knowledge were used as a starting point. Subsequently, the authors manually annotated 5 randomly selected texts for each of the scenarios based on the preliminary template. Necessary extensions and changes in the templates were discussed and agreed upon. Most of the cases of disagreement were related to the granularity of the event and participant types. We agreed on the script-specific functional equivalence as a guiding principle. For example, reading a book, listening to music and having a conversation are subsumed under the same event label in the flight scenario, because they have the common function of in-flight entertainment in the scenario. In contrast, we assumed different labels for the cake tin and other utensils (bowls etc.), since they have different functions in the baking a cake scenario and accordingly occur with different script events.Note that scripts and templates as such are not meant to describe an activity as exhaustively as possible and to mention all steps that are logically necessary. Instead, scripts describe cognitively prominent events in an activity. An example can be found in the flight scenario. While more than a third of the turkers mentioned the event of fastening the seat belts in the plane (buckle_seat_belt), no person wrote about undoing their seat belts again, although in reality both events appear equally often. Consequently, we added an event type label for buckling up, but no label for undoing the seat belts.We used the WebAnno annotation tool BIBREF2 for our project. The stories from each scenario were distributed among four different annotators. In a calibration phase, annotators were presented with some sample texts for test annotations; the results were discussed with the authors. Throughout the whole annotation phase, annotators could discuss any emerging issues with the authors. All annotations were done by undergraduate students of computational linguistics. The annotation was rather time-consuming due to the complexity of the task, and thus we decided for single annotation mode. To assess annotation quality, a small sample of texts was annotated by all four annotators and their inter-annotator agreement was measured (see Section "Inter-Annotator Agreement" ). It was found to be sufficiently high.Annotation of the corpus together with some pre- and post-processing of the data required about 500 hours of work. All stories were annotated with event and participant types (a total of 12,188 and 43,946 instances, respectively). On average there were 7 coreference chains per story with an average length of 6 tokens.After the first annotation round, we extended and changed the templates based on the results. As mentioned before, we used MissScrEv and MissScrPart labels to mark verbs and noun phrases instantiating events and participants for which no appropriate labels were available in the templates. Based on the instances with these labels (a total of 941 and 1717 instances, respectively), we extended the guidelines to cover the sufficiently frequent cases. In order to include new labels for event and participant types, we tried to estimate the number of instances that would fall under a certain label. We added new labels according to the following conditions:For the participant annotations, we added new labels for types that we expected to appear at least 10 times in total in at least 5 different stories (i.e. in approximately 5% of the stories).For the event annotations, we chose those new labels for event types that would appear in at least 5 different stories.In order to avoid too fine a granularity of the templates, all other instances of MissScrEv and MissScrPart were re-labeled with ScrEv_other and ScrPart_other. We also relabeled participants and events from the first annotation phase with ScrEv_other and ScrPart_other, if they did not meet the frequency requirements. The event label air_bathroom (the event of letting fresh air into the room after the bath), for example, was only used once in the stories, so we relabeled that instance to ScrEv_other.Additionally, we looked at the DeScript corpus BIBREF3 , which contains manually clustered event paraphrase sets for the 10 scenarios that are also covered by InScript (see Section "Comparison to the DeScript Corpus" ). Every such set contains event descriptions that describe a certain event type. We extended our templates with additional labels for these events, if they were not yet part of the template.Noun-noun compounds were annotated twice with the same label (whole span plus the head noun), as indicated by Example UID31 . This redundant double annotation is motivated by potential processing requirements.I get my (wash (cloth $ _{\textsc {\scriptsize ScrPart\_washing\_tools}} ))$ , $_{\textsc {\scriptsize ScrPart\_washing\_tools}} $ and put it under the water.A special treatment was given to support verb constructions such as take time, get home or take a seat in Example UID32 . The semantics of the verb itself is highly underspecified in such constructions; the event type is largely dependent on the object NP. As shown in Example UID32 , we annotate the head verb with the event type described by the whole construction and label its object with SuppVComp (support verb complement), indicating that it does not have a proper reference.I step into the tub and take $ _{\textsc {\scriptsize ScrEv\_sink\_water}} $ a seat $ _{\textsc {\scriptsize SuppVComp}} $ .We used the Head_of_Partitive label for the heads in partitive constructions, assuming that the only referential part of the construction is the complement. This is not completely correct, since different partitive heads vary in their degree of concreteness (cf. Examples UID33 and UID33 ), but we did not see a way to make the distinction sufficiently transparent to the annotators. Our seats were at the back $ _{\textsc {\scriptsize Head\_of\_Partitive}} $ of the train $ _{\textsc {\scriptsize ScrPart\_train}} $ . In the library you can always find a couple $ _{\textsc {\scriptsize Head\_of\_Partitive}} $ of interesting books $ _{\textsc {\scriptsize ScrPart\_book}} $ .Group denoting NPs sometimes refer to groups whose members are instances of different participant types. In Example UID34 , the first-person plural pronoun refers to the group consisting of the passenger (I) and a non-participant (my friend). To avoid a proliferation of event type labels, we labeled these cases with Unclear.I $ _{\textsc {\scriptsize {ScrPart\_passenger}}}$ wanted to visit my $_{\textsc {\scriptsize {ScrPart\_passenger}}}$ friend $ _{\textsc {\scriptsize {NPart}}}$ in New York. ... We $_{\textsc {\scriptsize Unclear}}$ met at the train station.We made an exception for the Getting a Haircut scenario, where the mixed participant group consisting of the hairdresser and the customer occurs very often, as in Example UID34 . Here, we introduced the additional ad-hoc participant label Scr_Part_hairdresser_customer.While Susan $_{\textsc {\scriptsize {ScrPart\_hairdresser}}}$ is cutting my $_{\textsc {\scriptsize {ScrPart\_customer}}}$ hair we $_{\textsc {\scriptsize Scr\_Part\_hairdresser\_customer}}$ usually talk a bit.In order to calculate inter-annotator agreement, a total of 30 stories from 6 scenarios were randomly chosen for parallel annotation by all 4 annotators after the first annotation phase. We checked the agreement on these data using Fleiss' Kappa BIBREF4 . The results are shown in Figure 4 and indicate moderate to substantial agreement BIBREF5 . Interestingly, if we calculated the Kappa only on the subset of cases that were annotated with script-specific event and participant labels by all annotators, results were better than those of the evaluation on all labeled instances (including also unrelated and related non-script events). This indicates one of the challenges of the annotation task: In many cases it is difficult to decide whether a particular event should be considered a central script event, or an event loosely related or unrelated to the script.For coreference chain annotation, we calculated the percentage of pairs which were annotated by at least 3 annotators (qualified majority vote) compared to the set of those pairs annotated by at least one person (see Figure 4 ). We take the result of 90.5% between annotators to be a good agreement.Figure 5 gives an overview of the number of event and participant types provided in the templates. Taking a flight and getting a haircut stand out with a large number of both event and participant types, which is due to the inherent complexity of the scenarios. In contrast, planting a tree and going on a train contain the fewest labels. There are 19 event and participant types on average.Figure 6 presents overview statistics about the usage of event labels, participant labels and coreference chain annotations. As can be seen, there are usually many more mentions of participants than events. For coreference chains, there are some chains that are really long (which also results in a large scenario-wise standard deviation). Usually, these chains describe the protagonist.We also found again that the flying in an airplane scenario stands out in terms of participant mentions, event mentions and average number of coreference chains.Figure 7 shows for every participant label in the baking a cake scenario the number of stories which they occurred in. This indicates how relevant a participant is for the script. As can be seen, a small number of participants are highly prominent: cook, ingredients and cake are mentioned in every story. The fact that the protagonist appears most often consistently holds for all other scenarios, where the acting person appears in every story, and is mentioned most frequently.Figure 8 shows the distribution of participant/event type labels over all appearances over all scenarios on average. The groups stand for the most frequently appearing label, the top 2 to 5 labels in terms of frequency and the top 6 to 10. ScrEv_other and ScrPart_other are shown separately. As can be seen, the most frequently used participant label (the protagonist) makes up about 40% of overall participant instances. The four labels that follow the protagonist in terms of frequency together appear in 37% of the cases. More than 2 out of 3 participants in total belong to one of only 5 labels.In contrast, the distribution for events is more balanced. 14% of all event instances have the most prominent event type. ScrEv_other and ScrPart_other both appear as labels in at most 5% of all event and participant instantiations: The specific event and participant type labels in our templates cover by far most of the instances.In Figure 9 , we grouped participants similarly into the first, the top 2-5 and top 6-10 most frequently appearing participant types. The figure shows for each of these groups the average frequency per story, and in the rightmost column the overall average. The results correspond to the findings from the last paragraph.As mentioned previously, the InScript corpus is part of a larger research project, in which also a corpus of a different kind, the DeScript corpus, was created. DeScript covers 40 scenarios, and also contains the 10 scenarios from InScript. This corpus contains texts that describe scripts on an abstract and generic level, while InScript contains instantiations of scripts in narrative texts. Script events in DeScript are described in a very simple, telegram-style language (see Figure 2 ). Since one of the long-term goals of the project is to align the InScript texts with the script structure given from DeScript, it is interesting to compare both resources.The InScript corpus exhibits much more lexical variation than DeScript. Many approaches use the type-token ratio to measure this variance. However, this measure is known to be sensitive to text length (see e.g. Tweedie1998), which would result in very small values for InScript and relatively large ones for DeScript, given the large average difference of text lengths between the corpora. Instead, we decided to use the Measure of Textual Lexical Diversity (MTLD) (McCarthy2010, McCarthy2005), which is familiar in corpus linguistics. This metric measures the average number of tokens in a text that are needed to retain a type-token ratio above a certain threshold. If the MTLD for a text is high, many tokens are needed to lower the type-token ratio under the threshold, so the text is lexically diverse. In contrast, a low MTLD indicates that only a few words are needed to make the type-token ratio drop, so the lexical diversity is smaller. We use the threshold of 0.71, which is proposed by the authors as a well-proven value.Figure 10 compares the lexical diversity of both resources. As can be seen, the InScript corpus with its narrative texts is generally much more diverse than the DeScript corpus with its short event descriptions, across all scenarios. For both resources, the flying in an airplane scenario is most diverse (as was also indicated above by the mean word type overlap). However, the difference in the variation of lexical variance of scenarios is larger for DeScript than for InScript. Thus, the properties of a scenario apparently influence the lexical variance of the event descriptions more than the variance of the narrative texts. We used entropy BIBREF6 over lemmas to measure the variance of lexical realizations for events. We excluded events for which there were less than 10 occurrences in DeScript or InScript. Since there is only an event annotation for 50 ESDs per scenario in DeScript, we randomly sampled 50 texts from InScript for computing the entropy to make the numbers more comparable.Figure 11 shows as an example the entropy values for the event types in the going on a train scenario. As can be seen in the graph, the entropy for InScript is in general higher than for DeScript. In the stories, a wider variety of verbs is used to describe events. There are also large differences between events: While wait has a really low entropy, spend_time_train has an extremely high entropy value. This event type covers many different activities such as reading, sleeping etc.In this paper we described the InScript corpus of 1,000 narrative texts annotated with script structure and coreference information. We described the annotation process, various difficulties encountered during annotation and different remedies that were taken to overcome these. One of the future research goals of our project is also concerned with finding automatic methods for text-to-script mapping, i.e. for the alignment of text segments with script states. We consider InScript and DeScript together as a resource for studying this alignment. The corpus shows rich lexical variation and will serve as a unique resource for the study of the role of script knowledge in natural language processing.This research was funded by the German Research Foundation (DFG) as part of SFB 1102 'Information Density and Linguistic Encoding'.

Q: What are the key points in the role of script knowledge that can be studied?

Q: Did the annotators agreed and how much?
A: For event types and participant types, there was a moderate to substantial level of agreement using the Fleiss' Kappa. For coreference chain annotation, there was average agreement of 90.5%.
Moderate agreement of 0.64-0.68 Fleiss’ Kappa over event type labels, 0.77 Fleiss’ Kappa over participant labels, and good agreement of 90.5% over coreference information.

Q: How many subjects have been used to create the annotations?