The Coreference under Transformation Labeling Dataset: Entity Tracking in Procedural Texts Using Event Models
Year: 2023

Abstract:
We demonstrate that coreference resolution in procedural texts is significantly improved when performing transformation-based entity linking prior to coreference relation identification. When events in the text introduce changes to the state of participating entities, it is often impossible to accurately link entities in anaphoric and coreference relations without an understanding of the transformations those entities undergo. We show how adding event semantics helps to better model entity coreference. We argue that all transformation predicates, not just creation verbs, introduce a new entity into the discourse, as a kind of generalized Result Role, which is typically not textually mentioned. This allows us to model procedural texts as process graphs and to compute the coreference type for any two entities in the recipe. We present our annotation methodology and the corpus generated as well as describe experiments on coreference resolution of entity mentions under a process-oriented model of events.

heading: 1 Introduction
text: Entity coreference resolution is a critical component for understanding most natural language text (Poesio et al., 2023; Sukthanker et al., 2020). However, when events in the text introduce changes to the state of participating entities, it is often impossible to accurately link entities in anaphoric and coreference relations without an understanding of the transformations those entities undergo. For example, events can bring about changes in entities that are not reflected in actual text mentions:
(1) a. Chop the garlic [WHOLE]; b. Put it [CHOPPED] in the pan.
That is, while it is anaphorically bound to the garlic, it is not strictly coreferential, as the garlic has undergone a transformation (Mitkov et al., 2000).
*These authors contributed equally to this work.
Events can also introduce new entities into the discourse or narrative, through the use of creation predicates (Asher, 1993; Badia and Saurí, 2000). This is pervasive in procedural text, where the goal is to describe a sequence of transformations to apply to multiple objects to build up a goal object. This can be seen, for example, in (2a), where the entities are transformed into a hidden result argument, which then licenses the definite NP the mixture in (2b). In addition, procedural text witnesses both argument drop, as in (2d), where the direct object is elided, as well as metonymies, where a container refers to its content, as with bowl in (2d).
(2) a. Mix flour and water in a bowl. b. Set the mixture [FLOUR + WATER] aside. c. Beat the eggs. d. Add ∅ [BEATEN EGGS] to the bowl.
In this paper, we demonstrate how a processoriented event model (POEM), based on Dynamic Event Structure proposed in Pustejovsky and Moszkowicz 2011; Pustejovsky 2013, motivated by and generalized from GL-VerbNet (Brown et al., 2022), can significantly help classify entity coreference in procedural texts. We argue that all transformation predicates, not just creation verbs, output a new entity into the discourse, as a kind of Generalized Result Role, which is typically not textually mentioned (Jezek and Melloni, 2011). This allows us to model procedural texts as input/output (I/O) process graph structures, as shown in fig. 1.
Each edge in the graph represents POEM, an event reduced to an I/O process. The “output” nodes of events are generalization of the result role from the VerbNet frames, as well as placeholders of syntactic drops and shadow arguments. The POEM graph, thus, is one way to serialize the abstraction of complex semantics including event-argument structures, subevent structures, temporal ordering and coreference chains, which we can unfold to re-construct other semantic structures.
12448
For example, from the graph, one can compute the type of (conventional) coreference or what we call a “coreference under transformation” relation for any two entities in the recipe.
To this end, we present CUTL 1, a novel annotation methodology and dataset that integrates both the tracking of entity transformations and coreference chains of entities into a single framework. Our pilot annotation contains 100 double-annotated cooking recipes, showing high agreement on relation F1 scores. Based on our process-oriented semantic model of events, we introduce a distinction between two relations: (i) Coreference under Identity (CuI), where two entities have identical state information; and (ii) Coreference under Transformation (CuT), where some change has occurred distinguishing two entities.
We, then, use various methods from Tu et al. 2023 to paraphrase transformed entities (generalized result role) that do not appear as textual mentions, while being aware of transformations entities have undergone. We use the annotated data to train models to predict various coreference relations between entities and show the value of transformation-aware entity representation in developing a coreference resolution system that works with entities in procedural text. Our experiment also shows an interaction between our semantic model and LLMs to generate reliable and natural paraphrases.
The contributions outlined in this paper include: (1) studying the anaphoric and coreference behavior inherent in procedural texts, focusing on cooking recipes; (2) operationalization of the POEM, where steps in a procedure are annotated with explicit I/O entity nodes, regardless of whether they are mentioned in the text; (3) creation of an annotation guideline and GUI environment based on the event model, identifying events and their semantic class, all ingredient entities, and set of coreference relations between entities, typed according to the kind of transformation; and (4) the creation of a dataset, CUTL, containing these entity coreference links and the events involved.

heading: 2 Related Work
text: Understanding procedural narratives involves many core competencies in language comprehension (Fang et al., 2022). Not only is it crucial to per-
1annotation data, scheme, tool and experiment code is available at https://github.com/brandeis-llc/dp-cutl
form anaphora resolution (Poesio et al., 2016), but equally important is to perform state tracking on the entities as they undergo transformations described in the text (Bosselut et al., 2017).
The task of anaphora resolution covers a range of coreference relations (Poesio et al., 2023), as well as non-identity anaphoric relations, known as bridging phenomena (Clark, 1977; Asher and Lascarides, 1998). Most work on anaphora resolution has focused on declarative narratives or dialogue datasets (Pradhan et al., 2012a; Poesio et al., 2023).
Interestingly, while there are several datasets of procedural texts that have been annotated and studied, these have been mostly in the context of entity state tracking and QA tasks (Mishra et al., 2018;
Yamakata et al., 2020; Tu et al., 2022a), rather than coreference resolution; two notable exceptions include (Mysore et al., 2019) and (Fang et al., 2022).
Examples of how entity state tracking datasets contribute to reasoning and inferencing tasks can be seen with (Bosselut et al., 2017), who presents the Neural Process Networks to model the state changes in procedural texts. The actions and entities are both predefined sets. They use soft attention to select actions and entities from the predefined sets to generate a state embedding for each entity at every step in the recipe. (Dalvi et al., 2019) is an example of how entity state tracking datasets contribute to reasoning and inferencing tasks. The paper extends the ProPara dataset (Mishra et al., 2018; Tandon et al., 2018) which contains texts describing processes. Workers were given a prompt (e.g., “What happens during photosynthesis?”) and then asked to author a series of sentences describing the sequence of events in the procedure. The goal is to predict the state (location, created, destroyed) change of all the participants. Also working with ProPara, (Kazeminejad et al., 2021) approach the task of tracking state change by first parsing every sentence in ProPara with the VerbNet Parser (Gung and Palmer, 2021), and then leveraging the lexical information from VerbNet and PropBank to predict the state change.
The interaction of anaphora resolution with state tracking makes it challenging to classify the relationships that result between entities mentioned in the text, in order to judge whether they are coreferential or somehow related, but not the same. To this end, the above distinction between coreference and bridging (non-identity anaphora) becomes relevant (Hou et al., 2018). This is how Fang et al. 2022 approach the problem of NP reference in procedural text. They first adopt the distinction made in Rösiger et al. 2018 between two types of bridging (referential, where the NP requires an antecedent to be fully understood; and lexical, which may involve any number of lexical semantic relations between two NPs. Their dataset (RecipeRef) of coreference relations includes both coreference and bridging relations. For the latter, they distinguish three types, depending on the state of the entities being associated: (a) no change; (b) physical change; and (c) chemical change.
Another work focusing on anaphora in recipes is Jiang et al. 2020, which introduces RISeC, a dataset for extracting structural information and resolving
zero anaphora from unstructured recipes. Our work is in the same spirit, as they utilize a general lexical resource, PropBank, rather than a limited inventory of pre-defined predicates as in Tasse and Smith 2008 . The corpus provides semantic graph annotations of (i) recipe-related entities, (ii) generic verb relations (from PropBank) connecting these entities, (iii) zero anaphora verbs having implicit arguments, and (iv) textual descriptions of those implicit arguments. The corpus however, does not contain state changes between entities.
Yamakata et al. 2020 introduce a corpus of annotated English recipes. The annotation is a flow graph (i.e., DAG with a single root) including entities and relationships between these entities. The direction of edges also indicates dependencies between actions. The label of edges explicitly specify the state change of entities. While their graph representation is similar to ours in many respects, they do not encode coreference or bridging relations.
There are newly emerging datasets focusing on both anaphora and bridging, many of them released as part of the most recent shared task on anaphora and bridging relation detection (Yu et al., 2022). Unfortunately, procedural datasets were not included in this task.

heading: 3 CUTL Dataset and Annotation Scheme
text: Procedural texts, such as recipes, are interesting to CL researchers for several reasons. One of those is that they are step-driven narratives requiring minimal temporal ordering recognition. As a result, semantic interpretation can focus on the changes that are taking place in the course of a sequence of events in the narrative, while assuming that the events are temporally ordered in a narrative progression. The goal of our CUTL annotation is to create a dataset of cooking recipe texts annotated with the following information:
• Events, typed with their semantic subclass; • Referring expressions of event arguments; • I/O relations between an event and its argu-
ments (Jezek and Pustejovsky, 2019); • Coreference relations between named enti-
ties in the recipe, when they exist.
The relations we adopt reflect the view laid out in Recasens et al. 2011, which distinguishes nearidentity from (true) identity when drawing coreference relations between referring expressions. Thus we identify those relations derived from I/O as near-
identities and other non-I/O coreference relations as true identities.

heading: 3.1 Data Source and Mention Annotation
text: We reviewed publicly available recipe annotation datasets and decided to build our dataset on top of the existing R2VQ corpus (Tu et al., 2022a) from SemEval 2022, as it already contains eventstructural semantic annotation layers.2 Specifically, R2VQ has an SRL layer (SRL columns) that includes verb sense disambiguation, predicateargument structure, and argument role labels. Additionally, it provides domain-specific “cooking entity” labeling (cooking action events, ingredients, tools, habitats) for event and entity spans (ENTITY columns). For this work, our main focus is on cooking actions, food ingredients, and their referring expressions. Thus, to generate lists of ingredients (and referring expression) mentions for the CUTL annotation, we used the union of Patient and Theme arguments from the SRL layer and INGREDIENT and HABITAT labels from the ENTITY column in R2VQ. For event mentions, we used the union of predicate spans from SRL and EVENT from ENTITY. To distinguish simple change of locations from entity state changes (transformations, see §3.2), we hand-labeled the change-of-location verb subclass in order to use it for relation labeling, partly adopting event subclass categories from (Im and Pustejovsky, 2010). Even though the base dataset has argument structures already annotated, because the semantics of the POEM is not directly mappable to semantics “role” names, we only took advantage of argument span annotation. The base dataset also has coreference chain annotation, but it is not compatible to this work because it did not consider near-identity. Thus we discarded the COREF column as well.
To model events as simple I/O transformation processes, our annotation scheme is pivoted on two critical assumptions: (1) textual ordering of events in a recipe reflects the temporal order of cooking actions; and (2) every event predicate has a result, regardless of whether it is mentioned in the text.
Based on the first assumption and considering document length and event number distribution, we sampled 100 recipes from the R2VQ dataset to annotate. This subset does not include any recipe that violates the temporal ordering assumption. Table
2R2VQ data and its specification was obtained from the shared task’s website.
1 shows the statistics of the ingredient entities in the CUTL annotation. Compared to the original R2VQ, CUTL contains much richer hidden entity annotation from the I/O relations. Table 2 shows different types of mentions we used in the CUTL annotation.

heading: 3.2 Coreference Relation Annotation
text: One of the key goals of the annotation task is to identify three types of event-structural information in the text that, together form the fundamental building blocks of the POEM : (1) EVENT PREDICATES, (2) INPUT ENTITIES, (3) RESULT/OUTPUT ENTITIES. For cooking events “input”s are naturally understandable as the ingredients used for an action. Syntactically, we treat all the objects of an event predicate as its inputs (although, often they are hidden from the surface form as we saw in examples in the section 1. Thus, in a sense, an input and the output of an event are coreferential, only considering the transformation that the input underwent during the event. We call this relation Coreference under Transformation (CuT).
The innovative aspect to our model being assumed here is that every event must have one or more result entities, whether they are explicitly
mentioned in the text or not. Compare the recipe steps in example (3) below.
(3) a. [Formevt] the mixture into [pattiesres]. b. [Mixevt] flour and water [∅res]. c. [Removeevt] [skinres1] and [bonesres2] from the halibut. [∅res3].
In (3a), we get a physically re-shaped meat mixture as the result of the action, and [the mixtureent] and [pattiesent] are coreferential under the [formevt] transformation. In (3b), we have two inputs and an aggregated object as the result. Because the result is hidden, there is no token we can directly anchor the mixture to, which we deal with by re-using the event predicate span as the anchor for the result, creating a phantom entity (indicated by RES. prefix below) referring to the output of the transformation. The same applies to the separation process in (3c), which is different from the others in that it results in multiple outputs. Example (3′) shows CuT relations from 3.
(3′) a. [mixtureent] form−−−−−−−−−→ TRANSFORMATION [pattiesent]
b. [flourent] mix−−−−−−−→
AGGREGATION [RES.mixent]
[waterent] mix−−−−−−−→
AGGREGATION [RES.mixent]
c. [halibutent] remove−−−−−−→
SEPARATION [skinent]
[halibutent] remove−−−−−−→
SEPARATION [bonesent]
[halibutent] remove−−−−−−→
SEPARATION [RES3.removeent]
The advantage of using these phantom spans is twofold; (i) we can directly draw a relation between the input and the output or between a new name and a non-mention output (when redescription (Badia and Saurí, 2000) happens in the text); and (ii) when a following event takes a result of the current event as an input, we can pass the newly created phantom node. Example (2′) is the set of coreferences from example (2), showing how phantom spans are used.
(2′) a. [flourent] mix−−−−−−−→ AGGREGATION [RES.mixent]
[waterent] mix−−−−−−−→
AGGREGATION [RES.mixent]
b. [RES.mixent] == REDESCRIPTION [mixtureent]
[mixtureent] set ==
CHANGE-OF-LOCATION [RES.setent]
c. [the eggsent] beat−−−−−−−−−→
TRANSFORMATION [RES.beatent]
d. [RES.beatent] add−−−−−−−→
AGGREGATION [RES.addent]
[RES.setent] add−−−−−−−→
AGGREGATION [RES.addent]
Fang et al. 2022 attempt to work around these issues by treating CuTs as bridging relations to an input entity, but only when the output is “redescribed”
as a text mention. We believe we should avoid using the term “bridging” too liberally for these cases. Furthermore, when the redescription occurs less frequently (only after several transformations), it will identify long-distance bridging relations that require cognitive jumps in the annotators’ mind, which is not necessarily recorded in the annotation data.
We distinguish the CuT relation from Coreference under Identity (CuI), which is the more conventional definition of coreference, and some of bridging relations such as part-whole relation. In addition to one-to-one IDENTITY relations including anaphoric pronouns, we also annotated locational METONYMY and MERONYMIC relations as subtypes of CuI. As discussed earlier, annotators are presented with automatically generated phantom result entities for every event predicate. So the redescription operation is identified as a CuI link in the annotation environment. One note to make here is that when an event predicate falls under the CHANGE-OF-LOCATION semantic subclass (Tu et al., 2022b) and the I/O annotation is single-in and single-out, we call this relation between the input and the output as a CuI even though the relation is mediated by an event, as the only difference the event made is the location of the entity, thus not transformation.
In summary, we used the following typology of coreferences to link two entities. Some are direct links between two entities while the others are mediated by events under the transformation. COREFERENCE UNDER IDENTITY
1. Identity: strict coreference of two entities. 2. Meronymy: relation between two entities
when one end is referring to an inseparate part of the other. 3. Metonymy: links between an ingredient entity and a location entity when the location entity is used as a container for the food. 3 4. Change of location: single-in, single-out under CHANGE-OF-LOCATION transformation.3
COREFERENCE UNDER TRANSFORMATION
3These sub-categories of relations are not annotated by the annotator, but automatically inferred from the structural information or pre-compile R2VQ annotation. Annotators still need to draw a link between two entities, but, for example, when one end is from HABITAT annotation, the relation label is automatically switched to metonym. Or when an annotator draws a link between a single input entity and an event, transformation label is used. However, if the event predicate is pre-annotated as a CHANGE-OF-LOCATION event subclass, the label will be identity instead.
1. Transformation: a one-to-one link between the input node and the output node of a transformation event.3 2. Aggregation: a many-to-one link from input nodes to an output node.3 3. Separation: a one-to-many link from an input node to output nodes
Annotations are encoded as a directed acyclic graph where (1) leaves are primitives (base ingredients), (2) the root is the title of the recipe, corresponding to the final state in the graph, (3) edges represent coreference relations and (4) internal nodes correspond to inputs and outputs of events – these are phantom entities, some of which are linked to redescription nominals via CuI annotation.

heading: 3.3 The CUTLER Annotation Environment
text: We developed a GUI annotation environment, CUTLER. It uses a simple table-based click-only workflow to quickly mark inputs and outputs of an event, types of the event, and coreference groups among entities. Figure 2 shows a screenshot of the CUTLER interface with a quick description of the annotation workflow. We believe the conceptually simple and streamlined interface of the CUTLER annotation environment significantly reduced annotator cognitive load, resulting in improved annotation speed and high inter-annotator agreement. The full guidelines for the CUTL annotation and the CUTLER software are available under opensource licenses in the data and code repository of the work.

heading: 3.4 Inter-Annotator Agreement and Gold Standard Dataset
text: Annotation of the 100 recipes was done in 4 rounds by 7 researchers and graduate students from the linguistics and computer science departments of a US-based university. Each document was dually annotated and Inter-Annotator Agreement (IAA) was computed at the end of each round. Pairs of annotators then met to adjudicate disagreements and create a finalized gold standard annotation. We used pairwise F1 as our primary IAA metric, which was uniformly high across labels, rounds, and annotator pairs with a mean F1 = 86.9. Metonymy and meronymy relations constituted the labels with the highest disagreement. This is partially due to having the fewest instances in the dataset, as well as the inherent ambiguity in each of these labels; during adjudication it was often found that both an-
notations were semantically valid. Encouragingly, CuT-related labels – the primary focus of this work – had consistently high agreement (F1 > 90.0 in the majority of documents).

heading: 4 Coreference resolution with CuT
text: We implemented a coreference resolution system using CUTL dataset. This section will describe the system design and the performance of the system.
Experiment Setup Under the POEM and CuT relations, coreference “chains” now can include phantom entity mentions (with RES. prefix). These phantom mentions serve two purposes; 1) make all event outputs explicit, 2) fill syntactic drop arguments in the following event. However, these mentions textually do not exist in the text, thus cannot be easily modelled by any language modelbased system, that are based on vector embedding of the surface text. To address this problem, we adopted Dense Paraphrasing (DP), a text enrichment technique (Tu et al., 2023) to first recover all drop argument (as empty slots) and “paraphrase” drop argument nodes and RES. nodes, to create natural language representation of the CUTL annotated data. Concretely, we apply both PREFIXP and SUBGRAPH-GPT methods from Tu et al. 2023 to all the drop and phantom entities from the recipe to generate paraphrases. PREFIXP is a heuristic method that paraphrase the entities by prepending the prefixes to reflect changes due to actions. SUBGRAPH-GPT use the GPT-3 model (Brown et al., 2020) to paraphrase the linearized subgraph that is rooted from the drop argument node or the RES. node. Figure 3 shows an example from the different paraphrasing methods.
Once the text is replaced with paraphrase with recovery of of drop arguments and insertion of generalized result nodes, we can use the new text in a coreference resolution task.4 For the coreference resolution task, we adopt the neural coreference model and the configuration from (Fang et al., 2021) and formulate the problem into a joint training of an antecedent assignment model (Lee et al., 2017) and a classification model. The system first detects all possible mentions of coreference. Then, for CuI resolution, the coreference resolution model would learn to assign a set of antecedents to each mention. And for CuT resolution, the bridg-
4We format the task input sentences with additional paraphrases based on simple heuristics: [To get Z], do X, [Y]. Brackets represent the text that is inserted.
ing model will do a multi-class classification on each pair of mentions detected.
Given the hierarchical nature of our CUTL types, we design the coreference resolution in two fashions: (i) Coarse: we only consider if there is transformation due to event, so there will only be CuI and CuT, and we treat all sub-relations under CuI as coreference. (ii) Fine-grained: we consider each relation type as an individual class and only treat Identity as coreference.
Machine learning model details We train a neural coreference resolution model with a configuration similar to (Fang et al., 2022; Lee et al., 2018). Specifically, we use 300-dimensional GloVe embeddings (Pennington et al., 2014) with window
size=2 for head word embeddings. And we train ELMo embeddings (Peters et al., 2018) on both CUTL and RecipeRef corpora. We also trained a CNN model with windows of 3, 4, and 5 to learn the character embeddings, and concatenate all three embeddings as the token representation. For each experiment, we do a 5-fold cross validation and train the model for 20 epochs on 4 NVIDIA Titan Xp GPUs.
Results Since our data contains one-to-many coreferential relation (SEPARATION) and many-toone relation (AGGREGATION), the traditional coreference resolution metrics (Pradhan et al., 2012b) are not suitable for our task, and we evaluate our experiments using F1. Table 3 shows the results of our experiments on 5-fold cross validation in both coarse and fine-grained ways. It is not surprising to see that it is more difficult to do coreference resolution with a more complex set of relation types, given the results of the coarse setting on both inputs are higher than the fine-grained. It also shows that the results from GPT-base paraphrases as inputs are higher than the inputs using DENSE paraphrasing on both overall and most of the fine-grained relations. Table 4 breaks down the results into the fine-grained coreferential relations. On each relation type, we evaluate using MUC, BCUBED and CEAF F1 from (Pradhan et al., 2012b) and their
average values. We also include the F1 scores. The observed outcomes align with the overall performance presented in table 3. It is noteworthy that, with the exception of the Meronym relation, utilizing the GPT-base paragraphs as input yields higher F1 scores compared to the DENSE paragraphs. This finding further supports the notion that the GPTbase paragraphs exhibit improved performance in coreference resolution.

heading: 5 Discussion and future work

heading: 5.1 Measuring agreement of CUTL
text: To compute IAA of the CUTL annotation (§3.4), we tried different metrics including naïve Kappa score and CoNLL coreference score, but decided to go with only F1. We view our research problem differently from traditional coreference tasks in two ways: (1) We have multiple coreferential types. This results in one entity being in more than one coreferential chain. (2) In “separation” relation, an arbitrary number of new hidden arguments can be added which means the set of entities is not fixed. Traditional metrics like Kappa or CoNLL can only measure one aspect of randomness of our data, while F1 can show the agreement in a more all-around fashion. The same logic applies to reporting of our experiment. But it would be an interesting topic to develop new metrics or re-formulate our linking task into a labeling task compatible with Kappa-family metrics.

heading: 5.2 Data selection and limitations
text: The annotation scheme proposed in this work is designed to focus on non-identidy coreference, CuT, and is not able to handle some complex linguistics phenomena. That includes (not limited to) com-
plex temporal ordering, VP or NP ellipsis under conjunction and/or disjunction, event negation. As a result, during data selection process, we had to look for those linguistic features and excluded documents with them from the data set.
Specifically, to limit the scope of the research, we intentionally limited our analysis to data that:
• Is temporally linear
• Has a single terminal state
• Has a high density of object transformations referred to explicitly throughout the text
We chose to work within the cooking recipe domain because it easily satisfies criteria. However, procedural text in general satisfies these three conditions, and our current model is therefore compatible with a broader range of domains than strictly recipes. In future work, we intend to broaden the scope to include more varied domains, such as news data and narratives.
During the manual curation of 100-document subset, we did not encounter any annotation of nominal events, and therefore this work ipso facto involves only events extracted from verbs. Although event recognition is not the primary research focus of this work, being able to additionally identify different types of lexical trigger of events is indeed important when considering broader domains. We plan to integrate our framework with other lexical resources in the future, and event recognition will receive more focus.

heading: 5.3 Event semantics
text: For this study, we directly adopted (Tu et al., 2022b) and used simple three-way subclass categorization for event semantics. In the future, we
will make a finer event type categorization utilizing existing large lexical resources such as GL-VerbNet (Brown et al., 2022). We hypothesize that utilizing finer and semantically loaded event subclasses will help empirical investigations of nominal redescriptions as well as improve automatic paragraph generation.

heading: 6 Conclusion
text: In this paper, we presented a new dataset, CUTL, annotated using a novel integration of integrating event semantics and coreference linking annotation. We applied a process-oriented event model and argument structure as coreference relations between event input(s) and output(s). We showed that using CUTL is a very efficient way of analyzing and annotating entity transformation and coreference chains in procedural text, by conducting pilot annotations on cooking recipe text. The CUTL dataset and annotation material are available under open-source licenses. Additionally, we conducted multi-stage experiments to build the baselines for coreference identifier and classifier that focus on utilizing our human annotations. The results from the coreference resolution systems show that the subgraph representation of our annotation is a good resource for LLMs such as GPT-3 to generate reliable paraphrases in natural language, which can further improve the multi-class coreference resolution task.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
5
7 A2. Did you discuss any potential risks of your work? Our work is creating a new dataset using open-licensed (CC) recipe text describing diverse cuisine and food cultures. We don’t believe that there’s an eminent risk from the linguistic annotation of diverse but neutral text.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The raw data didn’t include any PII

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3,5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3,4
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? annotation work is ongoing and guildeline is being updated iteratively. Tool screenshot and workflow description is provided in the paper (sec3)

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 3
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We used existing dataset to start annotation, hence not directly collected raw data.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 3

heading: Future Work
text: In the future, we
will make a finer event type categorization utilizing existing large lexical resources such as GL-VerbNet (Brown et al. , 2022). We hypothesize that utilizing finer and semantically loaded event subclasses will help empirical investigations of nominal redescriptions as well as improve automatic paragraph generation