SlowGuess/ACL_2020_OCR huggingface dataset:
Nazneen Fatema Rajani $^{1*}$  Rui Zhang $^{2*}$  Yi Chern Tan $^{2}$  Stephan Zheng $^{1}$  Jeremy Weiss $^{2}$  Aadit Vyas $^{2}$  Abhijit Gupta $^{2}$  Caiming Xiong $^{1}$  Richard Socher $^{1}$  Dragomir Radev $^{1,2}$
$^{1}$  Salesforce Research  $^{2}$  Yale University
{Nazneen.rajani, stephan.zheng, dradev}@salesforce.com {r.zhang, yichern.tan, dragomir.radev}@yale.edu
Neural networks lack the ability to reason about qualitative physics and so cannot generalize to scenarios and tasks unseen during training. We propose ESPRIT, a framework for commonsense reasoning about qualitative physics in natural language that generates interpretable descriptions of physical events. We use a two-step approach of first identifying the pivotal physical events in an environment and then generating natural language descriptions of those events using a data-to-text approach. Our framework learns to generate explanations of how the physical simulation will causally evolve so that an agent or a human can easily reason about a solution using those interpretable descriptions. Human evaluations indicate that ESPRIT produces crucial fine-grained details and has high coverage of physical concepts compared to even human annotations. Dataset, code and documentation are available at https://github.com/salesforce/esprit.
Humans learn to understand and reason about physical laws just by living in this world and doing everyday things. AI models, on the other hand, lack this ability and so are unable to generalize to new scenarios that require reasoning about abstract physical concepts like gravity, mass, inertia, friction, and collisions (Bakhtin et al., 2019). We propose Explaining Solutions to Physical Reasoning Tasks (ESPRIT), a framework for explaining qualitative physics reasoning using natural language. Neural networks with knowledge of qualitative physics would have commonsense reasoning abilities about the way the world works (Forbus, 1988). In turn, this could, for example, improve performance on tasks that involve interacting with humans and make human-robot interactions more efficient and trustworthy.
Goal: make the green ball touch the blue ball
Initial scene
Figure 1: An example from the PHYRE dataset (Bakhtin et al., 2019) consisting of a goal, an initial scene, a solution – the action of adding a red ball, and the resulting simulation rollout. Each object color corresponds to an object type. Red: user-added dynamic object; Green and Blue: dynamic goal object; Gray: dynamic scene object; Black: static scene object.
Solution action
Action simulation
Ideally, AI systems would reason about and generate natural language commonsense explanations of physical concepts that are relevant to their behavior and prediction. A key intuition is that natural language can provide an efficient low-dimensional representation of complicated physical concepts. To equip AI systems with this ability, we collected a set of open-ended natural language human explanations of qualitative physics simulations. The explanations include descriptions of the initial scene, i.e., before any physics is at play, and a sequence of identified pivotal events in a physics simulation. Three physical concepts are crucial for our simulation to reach a specified goal state: gravity, collision, and friction.
Our work attempts to build an interpretable framework for qualitative physics reasoning with strong generalization abilities mirroring those of humans. ESPRIT is the first-ever framework that unifies commonsense physical reasoning and interpretability using natural language explanations. Our framework consists of two phases: (1) identifying the pivotal physical events in tasks, and (2) generating natural language descriptions for the initial scene and the pivotal events. In the first phase,
7906
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7906-7917 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: The end-to-end ESPRIT framework for identifying pivotal physical events, extracting the features from pivotal events in a table, and explaining solutions using a table-to-text model for natural language generation. The purple bar is a static goal object.
our model learns to classify key physical events that are crucial to achieving a specified goal whereas in the second phase, our model generates natural language descriptions of physical laws for the events selected in the first phase. We demonstrate ESPRIT on the PHYSICAL REaasoning (PHYRE) benchmark (Bakhtin et al., 2019). PHYRE provides a set of physics simulation puzzles where each puzzle has an initial state and a goal state. The task is to predict the action of placing one or two bodies (specifically, red balls of variable diameters) in the simulator to achieve a given goal. Figure 1 shows an example of a task with a specified goal.
The input to ESPRIT is a sequence of frames from a physics simulation and the output is a natural language narrative that reflects the locations of the objects in the initial scene and a description of the sequence of physical events that would lead to the desired goal state, as shown in Figure 2. The first phase of the framework uses a neural network classifier to identify salient frames from the simulation. For the second phase we experimented with table-to-text models (Puduppully et al., 2019a,b) as well as pre-trained language models (Radford et al., 2018). We evaluated our framework for natural language generated reasoning using several automated and human evaluations with a focus on the understanding of qualitative physics and the ordering of a natural sequence of physical events. We found that our model achieves very high performance for phase one (identifying frames with salient physical
events) and that, for phase two, the table-to-text models outperform pre-trained language models on qualitative physics reasoning.
2.1 PHYRE Benchmark
We build our dataset by extending PHYRE (Bakhtin et al., 2019), a recent benchmark dataset for PHYSICAL REaasoning. PHYRE consists of a set of physics puzzles in a simulated 2D environment. This environment follows simple deterministic Newtonian physics with a constant downward gravitational force and a small amount of friction.
All objects (balls, bars, standing sticks, and jars) are non-deformable, and each object color corresponds to an object type: red is the user-added dynamic object; green and blue are used for dynamic objects that are part of the goal state; purple is for static goal objects; gray is for dynamic scene objects; black is for static scene objects. Each task starts with an initial scene and has a goal state, described in natural language. The task can be solved by placing one or two red balls in the simulation environment and choosing their sizes in a way that when the simulation runs according to the laws of physics the goal state is achieved. No further action can be taken after the simulation starts.
In this paper, we focus on the 25 task templates in the PHYRE dataset that involve the placement
Goal: add one ball to make sure the green ball is touching the purple bar
'https://phyre.ai/
7907
Templates	25
Tasks	2441
Train/Val/Test	1950/245/246
Objects / Task	14
Frames / Task	658
Events / Task	54
Salient Events / Task	7
Tokens / Initial State Description	36
Tokens / Simulation Description	45
Vocabulary Size	2172
Table 1: Statistics for the ESPRIT Dataset.
of a single ball to reach the goal state. Each template defines a set of 100 similar tasks generated by using different parameters for a template such as positions and sizes of objects. All tasks within the same template have the same goal (e.g., "make the blue ball touch the green ball") but somewhat different initial configurations.
2.2 Representing Frames as Structured Tables
We represent the simulation frames as structured tables by extracting information using the simulator module in the PHYRE API. The simulations consist of 60 frames per second. For each object, we collect its id, type (boundary, bar, jar, circle), color (red, green, blue, purple, gray, black), state (dynamic, static), and  $(x,y)$  coordinates. Jars also have an angle of rotation, width, base length, and side length (referred to as just length). Bars have length, width, and angle of rotation while circles have a radius. For each collision between two objects, we collect the  $(x,y)$  coordinates, velocity as a  $(v_x,v_y)$  vector, and the angle of rotation in radians for each object involved in the collision.
Extracting data from the PHYRE simulator. To track the motion of objects through a simulation, we intercepted PHYRE's built-in simulator. First, we created a dictionary of objects and their attributes in the simulation's initial scene (including the predicted action that was performed). It is important to note that the dictionary contains properties of both static and dynamic objects. But because static objects such as the simulation boundary are not affected by the physics in the simulation and their properties never change. So, unless a static object is involved in a collision, we did not
collect any other data about that object during the simulation.
Once this initial pass was made, we extracted the images of frames generated for the 2500 single-ball simulations. Each simulation was run for a maximum of 1000 time steps or approximately 16 seconds. After the initial action is taken, a simulation is considered successful if it reaches the goal state and remains in that state for at least 180 consecutive time steps, the equivalent of three seconds. If a simulation does not satisfy this goal condition, it is considered unsuccessful. In this way, we found solution simulations for 2441 out of 2500 tasks. The remaining 59 task simulations seem more complex and would possibly require a prohibitive number of trials ( $>10000$ ) to reach the goal successfully and so we excluded those from our dataset.
Finally, we mapped the dictionary of objects and attributes in the initial state to the frames derived from the simulator so that we could track how the object's properties change from one frame to another.
Generating tables. The three physical concepts at play in the simulations - friction, collision, and gravity are either a cause or an effect of some collision. Therefore, collisions were the most common physical event in the simulations (average  $= 54$  per task) and so we decided to only record collisions. For every collision extracted, we applied a window of size 3 to fetch frames before and after the collisions to remove any noise and get the more precise timestamp of the collision. Because pivotal events in a solution simulation only occur when two objects collide or separate, like a ball falling onto another or a ball rolling off of an elevated bar, we treat both cases identically.
2.3 Two-stage Annotation Procedure
Based on the simulation screenshots of the initial state and the collision, we employed a two-stage annotation procedure using Amazon MTurk. In the first stage, we showed the goal, the initial state, and all collisions during the simulation. We asked annotators to pick pivotal or salient events by selecting all and only the collisions that are causally related to the placement of the red ball and are necessary for the completion of the goal. In the second stage, we collected human annotations of natural language descriptions for the initial scene and explanations for the sequence of salient colli
2https://phyre.ai/docs/simulator.html
7908
sions annotated during the first stage. We showed the annotators the goal, the initial state with the red ball added, an animated GIF of the simulation, and the frames of salient collisions. We asked them to include descriptions of the shape, color, and position of the objects involved. The annotations for the initial scene and salient collisions are collected in separate text boxes.
Our data statistics are summarized in Table 1. We generated solutions for 2441 tasks, covering 25 different templates. These tasks have an average of 14 objects, 658 total frames, and 54 collision events. We split the tasks randomly into 1950 train, 245 validation, and 246 test. On average, each task has 7 events marked as salient by the annotators. Also, on average the description of the initial state and simulation each have about 40 tokens, with a vocabulary size of 2172.
3 Tasks and Methods
ESPRIT includes the following components:
Pivotal event detection. Given all the collision events in the simulation, select collisions that are crucial to achieving the goal state. Pivotal or salient collisions are collisions that fulfill the following two criteria: (i) causally related to the placement of the red ball, and (ii) necessary for the completion of the given goal.
To train a classifier to detect salient events, we use the following features from the table representation: collision time step, each object's shape, position  $(x,y)$ , velocity  $(v_x,v_y)$ , and angle of rotation. This totals 13 input features. The first object is often static, such as the boundary, while the second is often dynamic, such as the user-placed red circle. We experimented with a decision tree and a neural network MLP classifier to compare with a baseline that classifies every frame as salient. The MLP has three layers with 128, 128, and 32 nodes. There is a  $15\%$  dropout to avoid overfitting and batch normalization between each layer. Finally, a sigmoid node converts the output into a probability from 0 to 1 (anything above  $50\%$  is classified as salient). The models are trained on 59179 collisions (52066 negative, 7113 positive) and tested on 6893 collisions (6000 negative, 893 positive).
Natural language description of initial states. Given a list of objects and their attributes (color, po
sition, type) in the initial frames, generate a corresponding natural language description of the initial scene. The generated text should faithfully describe all the objects in the corresponding input frame.
Natural language explanations for sequences of pivotal events. Given a sequence of pivotal events for a simulation and the goal, generate a natural language description to explain the solution simulation. The generated text should faithfully summarize the simulation by explaining the causal sequence of salient events in it.
The goal of natural language generation for our task is to explain the pivotal physical events in the simulation so that an end user can solve the task more efficiently and reliably. Hence, we experimented with treating the physical event description generation as (1) Table-to-Text Generation and as (2) Language Modeling. The salient event detection component of our system serves as the content selection component of the natural language generation pipeline. We describe the two approaches in the following sections.
3.1 Table-to-Text Generation
For the initial state description, the input is the structured table representation of the initial state, and the model generates a textual description conditioned on the input table. Similarly, for the salient events explanation, the model produces the description given the structured table representation of all the salient events as the input. Effective table-to-text generation can be leveraged to teach AI agents to solve tasks in natural language and output explanation for the steps in the task solution.
For both generation tasks, we use the model from Puduppully et al. (2019b) which is a neural model for table-to-text generation by explicitly modeling entities. Since our desired generations are "entity coherent", in that their coherence depends on the introduction and discussion of entities in discourse (Karamenis et al., 2004), the entity-based table-to-text generation model is a proper method for our task. Unlike previous neural models treating entities as ordinary tokens, following Puduppully et al. (2019b), we explicitly create entity representations for our objects in the physical environment and update their representation as the text is generated.
The model input is a list of table records
<sup>3</sup>We also tried to use Puduppully et al. (2019a), but it requires a domain-specific relation extraction model to generate a specialized input, so we could not use it.
7909
as  $\{r_{j,l}\}_{l = 1,j = 1,\ldots ,|r|}^{L}$  where  $|r|$  is the number of records for this example, and  $L$  is the number of features for each record. For example,  $r_{j,1}$  are values and  $r_{j,2}$  are entities. The output  $y$  is description with words  $y = [y_{1},\dots ,y_{|y|}]$  where  $|y|$  is the length of the description.
Encoder. We first create embeddings  $\mathbf{r}_{j,l}$  of the features  $r_{j,l}$ , and then use a feed-forward layer to obtain the record embeddings  $\mathbf{r}_j$ .
$$
\mathbf {r} _ {j} = \mathrm {R e L U} (\mathbf {W} _ {r} [ \mathbf {r} _ {j, 1}, \dots , \mathbf {r} _ {j, L} ] + \mathbf {b} _ {r}),
$$
where  $\mathbf{W}_r$  and  $\mathbf{b}_r$  are model parameters. From the record embeddings, we then use two methods to create the encoder outputs  $\{\mathbf{e}_j\}_{j=1}^{|r|}$ :
- AVG. We use  $\mathbf{e}_j = \mathbf{r}_j$ , and the first hidden state of the decoder is the average of the record representations:  $\mathrm{avg}(\{\mathbf{e}_j\}_{j=1}^{|r|})$ .
- BiLSTM. To account for the chronological order in the physical simulation, we use a BiLSTM over  $\left[\mathbf{r}_1,\dots ,\mathbf{r}_{|r|}\right]$ , whose hidden states are extracted as  $\{\mathbf{e}_j\}_{j = 1}^{|r|}$ . The first hidden state of the decoder is initialized with the concatenation of the final step hidden states of the BiLSTM.
Entity memory. For each unique entity  $k$  (i.e., one of  $r_{j,2}$  values), we compute  $\mathbf{x}_k$  as the average embeddings of all records which satisfy  $r_{j,2} = k$ . During each decoding step  $t$ , we maintain an entity memory representation  $\mathbf{u}_{t,k}$ , and initialize it at  $t = -1$  as:
$$
\mathbf {u} _ {t = - 1, k} = \mathbf {W} _ {i} \mathbf {x} _ {k},
$$
where  $\mathbf{W}_i$  is a model parameter.
Denote the hidden state of the decoder at  $t$  as  $\mathbf{d}_t$ . We update the entity representation  $\mathbf{u}_{k,t}$  at each  $t$  with a gating mechanism as follows:
$$
\begin{array}{l} \boldsymbol {\gamma} _ {t} = \sigma (\mathbf {W} _ {d} \mathbf {d} _ {t} + \mathbf {b} _ {d}), \\ \pmb {\delta} _ {t, k} = \pmb {\gamma} _ {t} \odot \sigma (\mathbf {W} _ {e} \mathbf {d} _ {t} + \mathbf {b} _ {e} + \mathbf {W} _ {f} \mathbf {u} _ {t - 1, k} + \mathbf {b} _ {f}), \\ \tilde {\mathbf {u}} _ {t, k} = \mathbf {W} _ {g} \mathbf {d} _ {t}, \\ \mathbf {u} _ {t, k} = (1 - \delta_ {t, k}) \odot \mathbf {u} _ {t - 1, k} + \delta_ {t, k} \odot \tilde {\mathbf {u}} _ {t, k}, \\ \end{array}
$$
where  $\mathbf{W}_{d,e,f,g}$  and  $\mathbf{b}_{d,e,f}$  are model parameters, and  $\odot$  is element-wise product.  $\gamma_{t}$  indicates if there should be an update at  $t$ , and  $\delta_{t,k}$  controls the update by interpolating between the previous  $\mathbf{u}_{t-1,k}$  and candidate entity memory  $\tilde{\mathbf{u}}_{t,k}$ .
Hierarchical attention. We then use a hierarchical attention mechanism such that the decoder can first focus on entities and then the records for these entities. We can rearrange the encoder output  $\mathbf{e}_j$  in two-dimensional  $\mathbf{g}_{k,z}$ , where  $k$  is index for entities and  $z$  is the index for records of corresponding entities. For each entity, we can compute the attention over its records along  $z$ , and compute the entity context vector  $\mathbf{s}_{t,k}$ :
$$
\alpha_ {t, k, z} \propto \exp (\mathbf {d} _ {t} ^ {\intercal} \mathbf {W} _ {a} \mathbf {g} _ {k, z}), \quad \sum_ {z} \alpha_ {t, k, z} = 1,
$$
$$
\mathbf {s} _ {t, k} = \sum_ {z} \alpha_ {t, k, z} \mathbf {g} _ {k, z}.
$$
Then we compute the higher level attention over entities along  $k$ , and compute the encoder context vector  $\mathbf{q}_t$ :
$$
\phi_ {t, k} \propto \exp (\mathbf {d} _ {t} ^ {\mathsf {T}} \mathbf {W} _ {h} \mathbf {u} _ {t, k}), \quad \sum_ {k} \phi_ {t, k} = 1,
$$
$$
\mathbf {q} _ {t} = \sum_ {k} \phi_ {t, k} \mathbf {s} _ {t, k}.
$$
Decoder. The encoder context vector  $\mathbf{q}_t$  is then used in the decoder to compute a probability for each output token  $y_t$ :
$$
\mathbf {d} _ {t} ^ {\mathrm {a t t}} = \tanh (\mathbf {W} _ {c} [ \mathbf {d} _ {t}; \mathbf {q} _ {t} ]),
$$
$$
p _ {\text {g e n}} \left(y _ {t} \mid y _ {<   t}, r\right) = \underset {y _ {t}} {\operatorname {s o f t m a x}} \left(\mathbf {W} _ {y} \mathbf {d} _ {t} ^ {\text {a t t}} + \mathbf {b} _ {y}\right).
$$
In both generation tasks, we fine-tune the entity model provided by Puduppully et al. (2019b) for 125 epochs. We use the same training hyperparameters and select the best model using token-match accuracy following Puduppully et al. (2019b).
3.2 Language Modeling
We fine-tune a language model (LM) to generate descriptions of the initial state and explanations for sequences of pivotal physical events using the training split of our dataset. We use the pre-trained GPT-large (Radford et al., 2018) LM, which is a multi-layer transformer-based (Vaswani et al., 2017) model.
For the generation of initial state descriptions, the LM is fine-tuned conditioned on the objects (such as ball, jar, etc.) and their attributes (such as dynamic, static, color, size, etc.) extracted from the simulator described in Section 2.2 and the human written descriptions. So, the input context during training is defined as follows:
$C_{init} = o_1, o_2, \ldots, o_n$ , "In the physical simulation"
7910
where  $o_1, o_2, \ldots, o_n$  is the list of extracted objects with their attributes, e.g., "small red dynamic ball". The model is trained to generate the initial scene description  $s$  according to a conditional language modeling objective. The objective is to maximize:
$$
\sum_ {i} \log P (s _ {i} | s _ {i - k}, \dots , s _ {i - 1}, C _ {i n i t}; \Theta),
$$
where  $k$  is the size of the context window (in our case  $k$  is always greater than the length of  $s$  so that the entire explanation is within the context). The conditional probability  $P$  is modeled by a neural network with parameters  $\Theta$  conditioned on  $C_{init}$  and previous tokens.
For explanations of the salient physical events in the simulation, the LM is fine-tuned conditioned on the initial state descriptions and the human generated reasoning. So, the input context during training is defined as follows:
$C_{sim} =$  "initscene. The red ball is placed and"
The model is trained to generate the physical reasoning  $r$  by maximizing the following objective:
$$
\sum_ {i} \log P (r _ {i} | r _ {i - k}, \dots , r _ {i - 1}, C _ {s i m}; \Theta).
$$
We generate sequences of maximum length 40, use a batch size of 12, train for a maximum of 50 epochs, selecting the best model based on validation BLEU and perplexity scores. The learning rate was set to  $10^{-6}$ , warmed up linearly with proportion 0.01 and weight decay 0.01. We experimented both with temperature 1.0 and lower temperatures (0.1, 0.2) to restrict generation to the physics domain and avoid diversity. For word sampling, we tried top  $k$  as 3 and 5 as well as greedy ( $k = 1$ ). We found that the temperature of 0.1 with  $k = 3$  worked best.
We note that it is not fair to compare the generated text by the table-to-text model and the LM because the input to the table-to-text model is structured with fine-grained details while the input to the LM is an unstructured prompt. A promising approach would be one that uses a table encoder with a pre-trained language model that is more robust and generalizable.
We evaluate our models using both automatic metrics and human evaluations.
4.1 Automatic Metrics
We use precision, recall, and F1 for the pivotal event classification task which can be formulated as a binary classification problem.
For the natural language description of initial frames and solution simulations, we use automatic metrics including BLEU-1, BLEU-2, ROUGE_L, and METEOR using the implementation from Sharma et al. (2017).
4.2 Human Evaluations