Introduction
Tracking entity states in procedural texts Weston et al. ( 2015 ); Bosselut et al. ( 2018 ); Dalvi et al. ( 2018 ) is closely related to many NLP reasoning tasks. To name a few, question answering about events (e.g., should one use gloves when retrieving the tray from the oven ) often require knowledge of entity states (e.g., the tray becomes very hot while in the oven; gloves insulate heat ) Tandon et al. ( 2019 ); Spiliopoulou et al. ( 2022 ); Zhang et al. ( 2023a ) ; planning Wang et al. ( 2022 ); Brohan et al. ( 2023 ) largely involves actions upon entities resulting in state changes. While most recent work has relied on end-to-end language models (LMs) Huang et al. ( 2022 ) , recent work has shown that explicit modeling entity states benefits LMs in such tasks Zhang et al. ( 2023a ) . Procedural entity tracking is challenging in itself, requiring much understanding of an implicit environment as well as external knowledge of events and entities.

We propose the \ours dataset which builds on OpenPI (Open Procedural Inference) Tandon et al. ( 2020 ) , a large-scale dataset for tracking entity states in procedural texts. OpenPI contains annotations of entities, attributes, and state changes for each step (e.g., after the step “set the pan in a heated oven”, the pan ’s temperature was cool before and hot afterwards). \ours features two critical improvements (see Figure 1 for a demonstration of key features of OpenPI and \ours ):

Canonicalization . Originally, different mentions of the same entity or attribute render evaluation difficult. Here, we prompt LMs to effectively cluster the entities and attributes.

Entity Salience . Originally, a large amount of entities that undergo changes are listed in parallel. Here, we provide both human and model-predicted annotations of their salience.

Regarding canonicalization, clustering different mentions (e.g., coffee maker, espresso machine) of the same entity allows for fairer evaluation. Moreover, as our task of predicting entities, attributes, and states is a generation task with imperfect and incomplete ground-truth references, we show that expanding each entity or attribute cluster with possible paraphrases (thus providing more references) is effective for reducing the false-negative rate. We then comprehensively report various state-of-the-art LMs’ performance of entity tracking on \ours .

Regarding entity salience, we provide both manually annotated and automatically predicted labels. We evaluate them based on correlation with ground-truth data, and show that LMs can reliably predict entity salience with a close-to-human performance. We argue that salient entities act as a means of compression of the most critical information in procedural texts, similar to saliency maps in computer vision Simonyan et al. ( 2013 ) . We proceed to qualitatively and quantitatively show that salient entities, as chain-of-though of LM prompting, benefit downstream tasks such as question answering and classical planning, while reducing cost by excluding less important entities in the prompt.

have following advantages:

The canonicalization of entities and attributes (§ 3 ) that facilitates evaluation (§ 3.2 );

The salience of entities (§ 4 ) that improves performance on downstream tasks (§ 4.3 ).