RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
The Original OpenPI Dataset
Our work \ours builds upon the OpenPI dataset Tandon et al. ( 2020 ) that tracks entity state changes in procedural texts with an open vocabulary. The procedures are extracted from wikiHow, a web resource containing instructions of everyday tasks. As exemplified in Figure 2 , the input is a procedure which includes a goal (e.g., “remove fog using a potato”) and a sequence of ordered steps (e.g., “rub the cut side of potato on the window”). For each step, the output is an array of 4-tuples describing an entity state change. Each 4-tuple contains an entity, an attribute, a state before the step, and a state after the step (e.g., the window ’s texture was smooth before and sticky after). The task is thus equivalent to predicting an entity state matrix given a procedure, where the axes are step, entity, and attribute, while the value is the before and after states. The data is annotated via crowdsourcing and manually validated.

However, OpenPI lacks canonicalization of entities and the differentiation of salient entities. In our work of \ours , we will address both issues using state-of-the-art models.

Role Content User I am trying to make coffee. First, I put some coffee beans and tap water into the corresponding compartment of the espresso machine. Then, I select the desired type of coffee to make produced. Then I put a mug under the espresso machine and press start. Do you get it? Assistant Yes. User We have the following objects: water , coffee maker , coffee machine , mug , espresso machine . Group those that refer to the same thing. You must include all the provided entities. Do not add any entity that is not provided in the list. Assistant <start of generation> The grouped objects are: - [’water’] - [’coffee maker’, ’coffee machine’, ’espresso machine’] - [’mug’]
Canonicalization
In the original OpenPI dataset, the entities and attributes that undergo change were written by crowd workers. Consequently, the dataset contains different ways of expressing the same entity (e.g., coffee maker , coffee machine , espresso machine in a coffee-making procedure) or attribute (e.g., texture , smoothness , sheen of a paint). Canonicalization by clustering the entities and procedures is thus important for two reasons: 1) it facilitates evaluation, especially in a generation setting, where a model might be wrongly penalized for predicting the paraphrase of some correct entity or attribute; 2) it facilitates further annotation of features such as salience (§ 4 ) of the entities and attributes. Here, we describe efforts to canonicalize the entities and attributes in the evaluation set of OpenPI .
Clustering Entities and Attributes
While canonicalization seems straightforward, it is non-trivial in \ours because clustering is highly context-dependent. For example, the entity torso and paper chunk usually have nothing to do with each other, but in fact refer to the same thing in a procedure of “making a paper bird.”

Clustering Due to the contextual nature of the task, we prompt one of the state-of-the-art LMs gpt-3.5-turbo (a.k.a. ChatGPT) 2 2 2 platform.openai.com/docs/models/gpt-3-5 as shown in Table 1 . We use 3-shot prompting, meaning that the complete prompt includes three handwritten examples and the prompt header of the example to be inferred, only containing the “User” role. The temperature is as default (0.7) and so are other hyperparameters. We aggregate output from five runs of ChatGPT as the final entity cluster and three runs for attribute cluster, as doing so is found to be empirically superior than a single-pass generation. 3 3 3 With results from multiple runs, entity clusters are greedily selected based on their number of occurrences. For instance, if (pan, cookware, container) occurred four times whereas (pan, pot) just once, then the former will be added to the final cluster.

To see if our model can cluster entities and attributes effectively, we evaluate the results using cluster-level precision, recall, and F1 scores with exact match against a set of manually-labeled clusters from 20 procedures in the development set.

We see that ChatGPT scores better in clustering attributes compared to entities. Error analysis shows that two factors contribute to this performance discrepancy. First, most attributes describe the physical properties of an entity. Therefore, attribute clusters are less context-dependent compared to entity clusters. Second, many attributes are shared amongst entities. For instance, out of 1,145 attribute annotations in the development set, 204 of them are "location" .

Cluster expansion Though the existing entities and attributes are now clustered in \ours , there may still be other paraphrases that a model might rightfully predict and wrongfully penalized for. Thus, we again prompt ChatGPT to expand the clusters by generating paraphrases given a cluster of entities or attributes (prompt omitted).

To evaluate the quality of entities and attributes generated from the expansion, we manually rate 20 procedures and find that 83.3% of the generated, paraphrased entities and 59.4% attributes are correct.
This is largely because entity names are oftentimes self-explanatory and less context-dependent whereas the attribute names and their meanings are highly dependant on the context.

Correct No change Nonsense Missing 003 585 (82.3%) 106 (15.0%) 14 (2.0%) 383 (20.3%) 3.5 303 (59.4%) 173 (33.9%) 34 (6.7%) 218 (42.7%)
Discussion and Error Analysis
We observe that the predicting attributes of entities that undergo state changes is a highly challenging task even for state-of-the-art LMs. Although evidently, expansion of clusters improves performance (fairly, as we have shown that the generated paraphrases are mostly correct), false-negatives that result in underestimation of models cannot be eliminated entirely. One interesting observation is that text-davinci-003 greatly outperforms the supposedly more superior gpt-3.5-turbo . To gain even more insights into models’ behavior, we analyze the model output for the schemata prediction sub-task. For each step, we annotate each entity-attribute prediction based on three labels:

Correct, where the entity-attribute indeed go through some changes;

Incorrect, because the entity-attribute actually does not go through any changes;

Incorrect, because the entity-attribute is non-sensical.

In addition, we add any entity-attribute pairs that should have been predicted as going through some change, to measure models’ recall. We randomly sample 20 procedures to perform this error analysis and the results are shown in Table 5 .

Regarding precision, we find that while the majority of the predicted entities are correct, many of the predicted associated attributes are generic ones that do not undergo any change either locally or globally. For example, for the step “Purchase a blackboard eraser”, the attributes predicted by text-davinci-003 for the entity eraser are location (correct), cleanness (no change locally), shape , and size (no change globally). The issue is much more pronounced with gpt-3.5-turbo , with predictions such as location of seller , name of brand , etc, despite that the prompt clearly explains the desired output with an example. We attribute such performance discrepancy to gpt-3.5-turbo ’s decreased ability to follow examples and its inability to understand nuanced instructions. Regarding recall, both models fail to predict many attributes that the human annotator deems changing. Upon qualitative inspection, most of these missing attributes are no less salient than the predicted ones.

We leave to future work the resolution of these issues, which can be mitigated by re-prompting the models by validating if the predicted attributes indeed undergo changes, or simply have them predict the state changes altogether in the first place.
Salience
The original OpenPI is annotated with many parallel entities in each procedure. Often, they vary greatly by importance in accomplishing the task. For example, in a procedure of “cooking a steak”, entities fish , oven , gloves , and spice rack might all be involved, while some are more indispensable than the rest. Intuitively, the knowledge of entity salience helps models focus on what matters in downstream tasks (§ 4.3 ). In \ours , we define two types of entity salience: the global salience refers to the importance of an entity in accomplishing the goal of the procedure, whereas the local salience refers to that in a step.
Annotations
Human Labeling To first procure ground-truth salience labels, two experts (referred to as A1 and A2) annotated entity salience in the first 20 procedures in the development set as the gold standard of entity salience. We devise and follow these annotation criteria in a Likert scale:

without or without mentioning this entity, the procedure or step cannot be done at all (e.g., lemon in “Wash faucet with lemon”)

without this entity, another entity of the same type can be used as a replacement, perhaps with worse outcome or more efforts (e.g., pan in “Sear a salmon” - can also use grill )

without this entity, the procedure or step can be done in principal, though with slightly worse outcome or more efforts (e.g., glove in “Cut off tough branches of a bonsai plant”)

without this entity, the procedure or step can be done, though with negligibly worse outcome or more efforts (e.g., vacuum cleaner in “Drill holes in the wall”)

the entity appears in the procedure or step rather gratuitously, and the lack thereof makes no difference

the entity is irrelevant to the procedure or step

Subjectivity is inevitable even though we strive to minimize subjectivity using this fine-grained scale to capture nuanced situations (e.g., an entity that frequently appears that can be easily replaced versus one that appears only once but is irreplaceable). In later sections, we will see how this scale leads to reasonable inter-annotator agreement and favorable performance on downstream tasks.

LM Prediction We prompt gpt-3.5-turbo , as before, to automatically predict salience. Table 6 shows an example prompt for predicting global salience. As before, we use the default hyperparameters with a temperature of 0.7. We parse the result by extracting the first digit from the generation as the score, and default to 1 whenever impossible.
Utility: Downstream Applications
We argue that entity salience is an effective way to compress information expressed by procedural texts. In other words, states of the most salient entities are sufficient for downstream tasks where entity tracking can be applied to. We provide both qualitative and quantitative evidence on two datasets both in the domain of procedural texts.
Question Answering
CREPE Zhang et al. ( 2023a ) is a dataset for procedural question answering and causal reasoning. Given a procedure (e.g., steps of “baking a cake”), a model predicts the change of likelihood of some event occurrence (e.g., “there is a pleasant smell coming from the oven”) after the execution of each step (e.g., “set the tray in the oven for 45 minutes”).

First, we show a qualitative example in Table 8 in the format of CREPE. Clearly, the model’s third response is wrong, and the user’s first utterance implies that the torch would have been extinguished by the cold wind or at least flickering, making it harder to see what’s in the room. In contrast, the model falsely asserts that there would be natural light , which is impossible given the procedure occurs at night. However, if we use the same prompt as § 3.2 and Table 6 to first get access to the most locally salient entities, have the model predict their state changes, and use the such information as a chain-of-thought, the model is able to answer correctly. Specifically, the model now recognizes that the window is open , causing the wind to enter the room, in turn causing the flames to flicker . With this information equipped, the final predicted answer is now correct as the likelihood to see clearly in the room has decreased. Note that the step of opening the window also involves many other less salient entities, such as person , hands , windowsill , smell , etc., which are filtered out by predicted salience.

Following the qualitative study, we then quantitatively evaluate a gpt-3.5-turbo model on CREPE in a zero-shot manner. As the labels in CREPE are ternary (an event is more, less, or equally likely to happen after a step), we directly prompt the model to provide one of these labels instead of giving a likelihood score as in Table 8 , while keeping the rest of the designs the same.
We compare three settings, in which for each step given an event, the model:

makes a prediction directly;

is provided with all entity state changes (based on § 3.2 ), before making a prediction;

is provided with state changes of entities with a predicted salience score (based on § 4.1 ) no less than a threshold, before making a prediction.

Table 9 clearly shows that provided entity state changes as chain-of-though in the prompt improves the performance. Moreover, excluding information of less salient entities does not decrease, and even in some cases improve performance. By only keeping the entities that have a predicted salience score of 5, which constitute about half of all entities, the prompt becomes shorter and thus reducing cost by about 14% for the entire experiment, while achieving slightly higher performance.
Formal Planning based on Texts
A strand of work has focused on using LMs to plan in a structured and symbolic manner Liu et al. ( 2023 ); Xie et al. ( 2023 ) . Specifically, the task is to convert a a textual description of a procedure to a planning domain definition (PDDL), which can be deterministically solved to find a plan. An instance of PDDL consists of a domain file, which models the pre-conditions and effects of permitted actions, and a problem file, which models the entity states initially and eventually. Here, we show a qualitative example of how identification of salient entities can improve the quality of a generated domain file that models actions.

User Provide a PDDL domain file representing a procedure "Survive in rainy woods": 1. Find natural shelter using the environment 2. Set up containers in the open 3. Stay dry during a rain 4. Stay hydrated After that, also provide a problem file of "collect water to drink". Assistant Here’s a domain file: (define (domain survive-in-rainy-woods) requirements, types, predicates are omitted (:action find-shelter :parameters (?shelter - shelter ?loc - location) :precondition (at ?shelter ?loc) :effect (and (sheltered ?shelter))) (:action set-up-container :parameters (?container - container ?loc - location) :precondition (and (at ?container ?loc) (sheltered ?container)) :effect (and (contains ?container ?water ))) (:action stay-dry-during-rain :parameters (?obj - object ?loc - location) :precondition (and (at ?obj ?loc) (raining ?loc)) :effect (and (not (raining ?loc)) (dry ?obj))) (:action stay-hydrated :parameters (?obj - object ?container - container) :precondition (and (at ?obj ?container) (contains ?container ?water )) :effect (and (not (contains ?container ?water)) ( hydrated ?obj )))) Here’s a problem file: (define (problem collect-water) objects are omitted (:init (at water-bottle woods) (at shelter woods) (at player woods) (raining woods) (collecting-water woods)) (:goal ( hydrated player )))

Table 10 shows an example where an LM provides a domain file based on the procedure “Survive in the woods” and a problem file based on the task “collect water to drink”. Clearly, the goal configuration (hydrated player) can only be realized via the action stay-hydrated , whose pre-condition (contains ?container ?water) can only be satisfied in the second step. However, the LM fails to recognize that the third step “Stay dry during a rain” has the byproduct effect of filling up the containers set up in the second step. In this case, we might simply first provide the LM the salient entities for each step, and the LM successfully fixes the third action stay-dry-during-rain so that it has the effect of containing containing water. Therefore, the problem file can now be solved reasonably with a sequence of all four actions. We leave to future work a larger-scale experiment of the application of salient entity states to planning.
Resulting Dataset: \ours
By adding canonicalization of entities and attributes as well as salience of entities to the evaluation set of the OpenPI dataset, we now fully present \ours . As the procedures and entity state annotations have not changed, \ours still has 55 procedures with 5.0 steps on average. These procedures are collected from wikiHow and their topics are everyday activities. \ours also inherits the original entity-attribute-state changes annotated by crowd workers. After canonicalization, there are 356 canon entities each with 7.6 unique mentions and 5.5 expanded mentions on average, 3240 canon attributes, each with 3.0 unique mentions and 3.3 expanded mentions on average, and 1193 before-after states in the development set. The quality of clustering and expansion and be evidenced in § 3.1 . Regarding salience labels (on a scale of 1 to 5), the global salience of entities has a mean of 3.5 and standard deviation of 1.4; the local salience of entities has a mean of 3.4 and standard deviation of 1.5.
Conclusion
We propose \ours , an improved dataset on open-domain entity tracking in procedural texts. \ours features canonicalization of entities and attributes, based on which we perform a comprehensive benchmarking evaluation of state-of-the-art LMs. \ours also provides human annotation, model prediction, and analyses of entity salience, using which we show qualitative examples on its effective on various downstream tasks.
Limitations
, just like its predecessor OpenPI , includes procedures from wikiHow which may result in homogeneous domains, writing styles, and potentially though unlikely biased, erroneous or unsafe information. Regarding canonicalization, due to the limitation of models and the imperfect human annotations in OpenPI , there still exists false negatives while evaluating with metrics based on exact-match. Regarding entity salience, the definition of “how indispensable an entity is in executing the procedure” is motivated empirically downstream tasks and may benefit from refinement or theoretical support. The evaluation could be more trustworthy given more annotators and more procedures to be annotated. The chosen downstream tasks in this work might not be representative of all use cases of entity tracking.
Acknowledgements
We thank Yash Kumar Lal for providing insights into model performance, and Peter Clark for help with presentation of this paper. This work is supported in part by the DARPA KAIROS Program (contract FA8750-19-2-1004), AFRL (contract FA8750-23-C-0507), the Office of the Director of National Intelligence (ODNI) via the IARPA HIATUS Program (contract 2022-22072200005), the NSF (Award 1928631), and gifts from Roblox and Salesforce. Approved for Public Release, Distribution Unlimited. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ODNI, IARPA, NSF, AFRL, the U.S. Government, or of Roblox or Salesforce. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Figure 1: For each step in a procedure, OpenPI annotates the state change of attributes of entities. Our \ours additionally (shown in red boxes and texts) canonicalizes the entities and attributes and includes their salience scores.; For each step in a procedure, OpenPI annotates the state change of attributes of entities. Our \ours additionally (shown in red boxes and texts) canonicalizes the entities and attributes and includes their salience scores.
Figure 2: An example from the original OpenPI dataset Tandon et al. ( 2020 ) .; An example from the original OpenPI dataset Tandon et al. ( 2020 ) .
Figure 3: Per-procedure correlation of global entity salience between each set of annotations and the ground-truth human annotations.; Per-procedure correlation of global entity salience between each set of annotations and the ground-truth human annotations.
Evaluation of entity and attribute clustering.
Exact match F1 or accuracy and BERTScore on the schemata and states prediction sub-tasks, with and without cluster expansion. The schemata sub-task is evaluated both globally (per-procedure) and locally (per-step).
Exact match F1 and BERTScore of complete sentences including an entity, an attribute, a pre-state, and a post-state, following the original OpenPI paper. Canonicalization and expansion lead to little help for exact match as it is only done on entity and attribute clusters, while the state names can still be expressed in many ways, causing false negatives.
Our chosen prompt for predicting global or procedure-wide entity salience. For local salience, the wording is similar with only one step provided.
Pearson’ r between model prediction and human annotations (A1) of entity salience.
Model performance on CREPE, with and without including state changes of salient entities.
Our prompt for text-davinci-003 for the schemata prediction sub-task, followed by 1-shot demonstration.
Our prompt for gpt-3.5-turbo for the schemata prediction sub-task, followed by 1-shot demonstration.
Our prompt for text-davinci-003 for both the states prediction sub-task and the complete-sentence evaluation format, followed by 1-shot demonstration.
Our prompt for gpt-3.5-turbo for both the states prediction sub-task and the complete-sentence evaluation format, followed by 1-shot demonstration.