Procedural Reading Comprehension with Attribute-Aware Context Flow

1 . Introduction . Procedural text describes how entities ( e.g. , fuel , engine ) and their attributes ( e.g. , locations ) change throughout a process ( e.g. , a scientific process or cooking recipe ) . Procedural reading comprehension is the task of answering questions about the underlying process described in the text ( Figure 1 ) . This task , in turn , requires inferring attributed of the entities in the process , and their transitions , which might only be implicitly mentioned . For instance , in Figure 1 , the creation of the mechanical energy in the alternator can be inferred from the second and third sentences . Full understanding of a procedural text requires capturing the interplay between all the components of the process : the affected entities , their attributes and their transitions . Recent works in understanding procedural text develop domain-specific models for tracking entities in scientific processes [ Mishra et al. , 2018 ] or cooking recipes [ Bosselut et al. , 2018 ] . More recently , Gupta and Durrett [ 2019b ] leverage pre-trained language models to obtain general entity-aware representations of procedural text , and predict entity transitions from a set of pre-defined classes independent of entity attributes . Pre-defining the set of entity states limits the general applicability of the model , however , as entity attribute values might be arbitrary spans of text . Moreover , entity attributes can be used for tracking entity state transitions . For example , in Figure 1 , the location of fuel can be effectively inferred from text as engine without the explicit mention of the movement transition in the first sentence . In addition , the phrase converted in the third sentence gives rise to predicting two transition actions of destruction of one type of energy and creating the other type . In this work , we introduce a general formalism to represent procedural text , and develop an end-to-end neural procedural reading comprehension model . Our formalism represents entities , their attributes , and their transitions across time , and the model jointly identifies these attributes and transitions using a dynamic contextual encoding of the text . Our model computes an attribute-aware representation of the procedural text at a certain timestep by leveraging distributions over predicted attribute values , either as a span of text or from a pre-defined set of classes . Then , it pairs this attribute-aware encoding with an entity-aware representation to predict state transitions for entities , thereby capturing the dynamic nature of the entities in the contextual encoding of the process . Our experiments show that our method achieves state of the art results across multiple tasks evaluated in the PROPARA dataset [ Mishra et al. , 2018 ] to track entity attributes and their transitions in scientific processes . Additionally , a simple variant of our model achieves state of the art results in the NPN-COOKING [ Bosselut et al. , 2018 ] dataset . Our contributions are three-fold : ( i ) we present a general formalism to model procedural text , which can be adapted to different domains , ( ii ) we develop DYNAPRO , an end to end neural model that jointly and consistently predicts entity attributes and their state transitions , leveraging pretrained language models , ( iii ) we show that our model can be adapted to several procedural reading comprehension tasks using the entity-aware and attribute-aware representations , achieving state of art results on several diverse tasks . 2 . Related Work . Most previous work in reading comprehension [ Rajpurkar et al. , 2016 ] focuses on identifying a span of text that answers a given question about a static paragraph . In contrast , this paper focuses on procedural reading comprehension , which inquires about how the states of entities change over time . There are several previous works that focus on understanding procedural text in multiple domains . Mishra et al . [ 2018 ] introduced the PROPARA dataset , a collection of procedural texts that describe how entities change during scientific processes ( e.g. , photosynthesis ) , which is paired with questions about several aspects of these processes , such as the entity attributes or state transitions . Bosselut et al . [ 2018 ] focus on cooking recipes , which describe instructions on how to change ingredients . Math word problems [ Kushman et al. , 2014 , Hosseini et al. , 2017 , Amini et al. , 2019 , Koncel-Kedziorski et al. , 2016 ] describe how the states of entities change throughout mathematical procedures . Narrative question answering [ Weston et al. , 2015 , Kočiskỳ et al. , 2018 , Lin et al. , 2019 ] inquires about the state of a story over time . These resources have influenced many models ( e.g. , EntNet [ Henaff et al. , 2017 ] , QRN [ Seo et al. , 2017 ] , MemNet [ Weston et al. , 2014 ] ) that track entities in narratives . The closest of these works to ours , however , is the line of work focusing on the PROPARA and NPN-COOKING datasets . Bosselut et al . [ 2018 ] use an attention-based neural network , the neural process network ( NPN ) , to identify ingredient state transitions in cooking recipes . To track state in scientific processes , models such as Pro-local and Pro-Global [ Mishra et al. , 2018 ] first identify locations of entities using an entity recognition approach and use manual rules or global structure of the procedural text to consistently track entities . Tandon et al . [ 2018 ] leverage manually defined and KB-driven commonsense constraints to avoid nonsensical transitions when predicting entity states ( e.g. , a tree doesn ’ t move to a new location ) . KG-MRC [ Das et al. , 2019 ] maintains a knowledge graph of entities over time and identifies entity states by predicting the location spans with respect to each entity using a reading comprehension model . NCET ( Gupta and Durrett [ 2019a ] ) introduces a neural conditional random field model to maintain the consistency of state predictions . Most recently , ETBERT [ Gupta and Durrett , 2019b ] uses transformers to construct representations of each entity and predict the state transitions from a set of pre-defined classes . In this paper , we integrate these prior observations , and develop a model that jointly identifies entities , attributes , and transitions over time . Unlike previous work that is designed to address either attributes or transitions , our model benefits from the clues that are implicitly and explicitly mentioned for both entity attributes and transitions . Leveraging both aspects of procedural reading comprehension leads us to a general and adaptive framework for the task , and an accompanying model that achieves state of art on several datasets across different domains . 3 . Procedural Text Representation . Procedural text consists of a sequence of sentences describing how entities and their attributes change throughout a process . We introduce a general formalism to represent a procedural text : p = ( E , A , T ) , ( 1 ) where E is the list of entities participating in the process , A is the list of entity attributes , and T is the list of transitions . We formulate the model ’ s input to be a combination of a query about an entity attribute and a partial context of the procedural text . Entities are the participants in a process . For example , in the scientific processes in PROPARA , entities might include elements such as energy , fuel , etc . In the cooking recipe domain , the entities could be ingredients such as milk , flour , etc . Entities could be given a priori based on the task ( e.g. , PROPARA ) or they could be inferred from the context ( e.g. , math word problems ) . Attributes are entity properties that can change over time . We model attributes as functions Attribute ( e ) = val that assign a value val to an attribute of the entity e. The entity state at each time is derived by combining all the attribute values of that entity . Attribute values can be either spans of text or can be derived from a pre-defined set of classes . For example , in PROPARA , an important attribute of an entity is its location , which can be a span of text . The NPN-COOKING dataset introduces several attributes ( such as shape and cookedness ) for each ingredient . Example attributes addressing the entities in PROPARA are modeled as follows : exists ( e ) = { nowhere , unknown , span of text } at loc ( e ) = l→ Assigns the location l to entity e Transitions capture changes in the entity states . More specifically , transitions indicate how entity attributes change over time . We model each transition with an action name and a list of arguments that include the entity and some attribute values . For example , PROPARA consists of four transition types : Create ( e , loc ) , Destroy ( e ) , None ( e ) and Move ( e , loc ) . 4 . Model . We introduce DYNAPRO , depicted in Figure 2 , an end-to-end neural architecture that jointly predicts entity attributes and their transitions . DYNAPRO first obtains the representation of the procedural text corresponding to an entity at each time step ( Section 4.2 ) . It then identifies entity attributes for current and previous time steps ( Section 4.3 ) and uses them to develop an attribute-aware representation of the procedural context ( Section 4.4 ) . Finally , DYNAPRO uses the entity-aware and attribute-aware representations to predict transitions that happen at that time step ( Section 4.5 ) . 4.1 Task Reformulation . Given a procedural text 〈S0 . . . Sk . . . ST 〉 and an entity e , DYNAPRO encodes procedural context Xk at each time step k and obtains the entity-aware representation vector Rk ( e ) . The procedural context is formed by concatenating the query containing the entity name , and a fragment of the procedural text . The entity name and the query are included in the procedural context to construct an entity-aware representation of the context . Since entity attributes are changing throughout the process , we form the context at each step k by truncating the procedural text up to the kth sentence . More formally , the procedural context is defined as : Xk ( e ) = [ cls ] Qe [ sep ] [ Ci ] S0 . . . Sk [ sep ] , ( 2 ) where 〈S0 . . . Sk〉 is the fragment of the procedural text up to the kth sentence , Qe is the entityaware query ( e.g. , “ Where is e ? ” ) , [ Ci ] includes tokens that are reserved for attribute value classes ( e.g. , nowhere , unknown ) , and [ cls ] and [ sep ] are special tokens to capture sentence representations and separators . Note that the input samples for any paragraph are constructed per sentence and entity . So the complexity of the input data is |S|∗|E| per procedural text where |S| and |E| are the average number of sentences and entities in each procedural text respectively . 4.2 Entity-aware Representation . DYNAPRO then uses a pre-trained language model to encode the procedural context Xk ( e ) from Equation 2 and returns the entity-aware representationRk ( e ) = BERT ( Xk ( e ) ) , whereBERT ( wi ) is the embedding of the i-th token from the transformer layer . Hereinafter , we will remove the argument e from equations for ease of notation .

Summary: This paper addresses the task of understanding procedural text, using a reading comprehension based approach. This work proposes jointly modeling entity attributes and transitions in order to build robust entity and attribute representations. The paper shows that using a modern pretrained language model, this system is able to beat previous approaches on ProPara and a cooking recipe based dataset (NPN-Cooking).