Utility: Evaluation of Entity Tracking
Just as the original evaluation set of OpenPI , \ours is meant to benchmark models on entity tracking – given a step in a procedure, predicting the state changes that entities and their attributes undergo. With the entities and attributes in \ours now fully canonicalized, evaluation can be done more fairly. To start with, we follow Tandon et al. ( 2020 ) and have models predict one complete sentence: “ attribute of entity is pre-state before and post-state afterwards”, which is then compared to such sentences in the ground-truth data (Table 4 ). We further make the evaluation more fine-grained by formulating two sub-tasks: i. predicting schemata , namely the entities and their corresponding attributes given a step (e.g., given “turn on the oven”, the temperature of the rack undergo state changes), and ii. predicting the change of states given a step, an entity and an attribute (e.g., given the previous information, the state change is from cool to hot ). This evaluation of first predicting a skeleton tensor of entities and attributes is highly practical, with a notable advantage over previous work (§ 6 ) in closed-domain entity tracking, where states are predicted using given entities and attributes.

On the development set, we run three state-of-the-art LMs: gpt-3.5-turbo , text-davinci-003 4 4 4 platform.openai.com/docs/models/gpt-3-5 Brown et al. ( 2020 ) , and the open-source LLaMA 65B Touvron et al. ( 2023 ) . For each model, we start by separately tackling each of the two sub-tasks 5 5 5 To avoid error propagation, for states prediction, the ground-truth entities and attributes are provided. ; namely, a model first predicts attributes of entities (schemata) given a step, and then predicts a pre-state and a post state (states) given the gold entity-attribute pair. All experiments are via 1-shot prompting. See details on prompt formulation in Appendix B .

For all settings, we consider both exact match (F1 for schemata and complete sentence prediction and accuracy for states prediction) and BERTScore Zhang et al. ( 2020d ) based on deberta-xlarge-mnli He et al. ( 2021 ) .

For the schemata prediction sub-task (Table 3 ), the atomic unit to be evaluated is an entity-attribute pair. We consider both a global evaluation, where predictions are made per-procedure (e.g., what attributes of entities undergo state changes in the procedure), and a local evaluation, where predictions are made per-step. This categorization will reappear in § 4.2 . Schemata prediction is naturally influenced by our entity and attribute clusters. Hence, for exact match we report F1 scores based on exact matches where any entity-attribute prediction that falls under an cluster, obtained by taking a Cartesian product of an entity cluster and an attribute cluster, is considered a true positive. For BERTScore, we calculate the maximum score of a prediction against all entity-attribute strings within all ground-truth clusters. Then, we report the mean score among all predictions as a macro average.

The states prediction sub-task (Table 3 ) is much more straightforward as the entity-attribute pairs are provided and a model only needs to predict a pre-state and a post-state for each. Thus, we simply report the exact match accuracy and BERTScore for each state.