SlowGuess/ACL_2020_OCR huggingface dataset:
The automated metrics for generation evaluation are very crude and do not measure the correctness and coverage of actual physical concepts or even the natural ordering in which physical events occur in a given simulation. For example, an object first falls and then it hits the ground or an object first falls on some other object which then causes the second object to be set in motion. So, we deployed human evaluations to measure the quality of the physical concepts captured by our language generation models in terms of validity and coverage.
To measure the validity of initial scene descriptions, we showed humans the generated description for a task, the initial frames from that task, and three random distractor initial scenes from other tasks which may or may not be from the same template. Then, we asked them to select the frame that belongs to the task being described. This evaluates how faithful and accurate the generated description is to the input initial state. If the generated text does not include a detailed description of the objects, their attributes, and their positions, it would be difficult for humans to map them to the correct initial scene.
For evaluating the validity of pivotal events descriptions, we showed humans the generated text for a task, the initial state of that task, and three distractor initial states generated from the same task but with positions of the red ball that do not solve the task. Then, we asked them to select the correct initial state with the red ball that would eventually reach the task goal. A good simulation description should give higher accuracy for humans to choose the correct solution. Note that we also evaluated the human generated initial state description and pivotal events description by asking annotators to match the human natural language descriptions that we collected and found the average accuracy to only be  $70.2\%$  for the initial scene description and  $44.7\%$  for the pivotal events description (Ta
Precision	Recall	F1
Positive	0.01	0.11	0.02
Decision Tree	0.87	0.86	0.87
MLP	0.90	0.91	0.90
Table 2: Results on pivotal events classification.
ble 4). This is because of reporting bias, i.e., humans rarely state events that are obvious (Forbes and Choi, 2017). For example, a falling ball would bounce multiple times or an object pushed off an elevated bar by another object would have a projectile motion. Lack of such fine-grained explanations is what makes the human evaluation of human generated descriptions especially for the sequence of pivotal events have poor accuracy.
The PHYRE tasks incorporate three physical concepts in every simulation — gravity, collision, friction. So, to measure coverage, we show humans just the natural language description of the simulation and ask them to select words that would imply any of the three concepts. For example, "rolling" or "slipping" would imply friction, "falling" would imply gravity, "hit" would imply collision, etc. We note that many physical concepts are very abstract and even difficult to be noticed visually, let alone describe in natural language. For example, moving objects slow down due to friction, but this physical concept is so innate that humans would not generally use words that imply friction to describe what they see. This metric gives us an overview of what degree of coverage the text generation models have for each of the three physical concepts.
For all our human evaluations we used MTurk and collected 3 annotations per instance and report the majority. We paid Turkers 50 cents per instance for the validity evaluation and 50 cents per instance for the coverage evaluation.
5 Experimental Results and Discussion
Table 2 summarizes the performance of the pivotal events classifiers. The decision tree and MLP classifiers get very high performance with 0.87 and 0.9 F1 scores respectively. The baseline classifies every event as pivotal and thus performs very poorly.
From the decision tree, we extract feature importance values for each of the 13 input variables described in Section 3. The most important variable is the time step of the collision, with a weight of 0.178. The most important features for clas
sification were an object's collision position, its velocity, and then its angle of rotation. Given such strong results for identifying pivotal events, we were able to predict the salient events of previously unseen simulations and that helped in the next step of generation descriptions of salient events.
Table 3 shows the performance of the three text generation models using automatic metrics. The table-to-text models perform better than the language model on most of the metrics. The AVG model performs slightly better than the BiLSTM on both generation tasks. However, these metrics are a very crude measure of physical reasoning performance and are not intuitive. The human evaluations, on the other hand, are more informative and insightful.
Human evaluation - validity. While the GPT model can achieve scores comparable to the data-to-text models using automated metrics, its performance using human evaluation is as good as chance, as shown in Table 4. We found that the GPT LM generation was very high-level and is not useful for humans to decide which tasks (among the correct and distractor choices) the generated solution explanation of the initial state and pivotal events match. By contrast, AVG and BiLSTM have significantly higher accuracy, mainly because their output is more fine-grained and so gives a more thorough explanation of the solution. Surprisingly, the human annotations of the descriptions that we collected as ground truth are not perfect either, indicating that humans tend to produce sentences that are not sufficiently discriminate and even sometimes skip obvious details such as whether the ball rolls to the left vs. right.
Human evaluation - coverage. Table 5 shows the results for coverage of physical concepts. The outputs of the GPT model are repetitive and not grammatical, containing little explanation of physical concepts. AVG and BiLSTM, on the other hand, can generate text that contains fine-grained descriptions of physical concepts even sometimes better than those generated by humans. This is because humans don't describe everyday commonsense concepts using fine-grained language, while the AVG and BiLSTM models tend to generate long detailed descriptions containing various words for gravity (e.g., falls, drop, slope, land), friction (e.g., roll, slide, trap, travel, stuck, remain), and collision (e.g., hit, collide, impact, land, pin, bounce).
7912
Initial state description	Pivotal events description
BLEU-1	BLEU-2	ROUGE.L	METEOR	BLEU-1	BLEU-2	ROUGE.L	METEOR
GPT (Radford et al., 2018)	15.37	2.25	20.08	9.93	24.32	3.89	26.82	12.14
AVG (Puduppully et al., 2019b)	15.37	11.38	22.53	24.09	20.53	15.89	29.11	27.38
BiLSTM (Puduppully et al., 2019b)	14.74	10.59	21.35	23.00	20.36	15.48	27.93	26.91
Table 3: Automatic evaluation of initial state and pivotal events descriptions on the test set.
Initial state	Pivotal events
Random classifier	25.0	25.0
GPT (Radford et al., 2018)	23.8	26.8
AVG (Puduppully et al., 2019b)	50.8	36.6
BiLSTM (Puduppully et al., 2019b)	58.1	40.2
Human annotation	70.2	44.7
Table 4: Human evaluation for validity accuracy of initial state and simulation descriptions on test set.
Gravity	Friction	Collision
GPT (Radford et al., 2018)	89.3	2.0	16.0
AVG (Puduppully et al., 2019b)	100.4	61.6	71.8
BiLSTM (Puduppully et al., 2019b)	99.2	70.7	71.1
Human annotation	96.7	43.0	57.0
Table 5: Human evaluation for coverage accuracy of physical concepts in simulation descriptions on test set.
Input records	...green|green_circle_0|OBJ_COLOR|INITIAL_STATEcircle|green_circle_0|OBJ_TYPE|INITIAL_STATEdynamic|green_circle_0|OBJ_STATE|INITIAL_STATE 76|green_circle_0|X|INITIAL_STATE 162|green_circle_0|Y|INITIAL_STATE...
Gold annotation	The red and green balls fall. The red ball lands on the ground and the green ball lands on the red ball and rolls to the right over the black vertical bar.
Generation (AVG)	The red ball lands in the cubby and the green ball lands on top and a little to the right, sending the green ball right. It rolls over the short black wall of the cage and onto the floor, where it keeps rolling right towards the purple goal...
Generation (BiLSTM)	The red ball falls and knocks the green ball off of its curved black platform and to the left. It rolls leftwards and continues falling until it lands on the purple floor...
Table 6: Example input records, gold annotation, and generated simulation description from the AVG and BiLSTM models, taken from example 00014:394. We show only a short segment of the actual input records.
Qualitative analysis. An example of the model inputs and outputs is in Table 6 and taken from simulation id 00014:394. Here we make two observations. First, the generated descriptions are not as succinct as the gold annotations, because our model is obtained from fine-tuning an entity-based model pre-trained on generating long Rotowire game sum
maries (Wiseman et al., 2017). Second, the output generated by the BiLSTM model predicts the incorrect direction of motion for the green ball, an error that is occasionally seen across generation descriptions of both models. This indicates that a table-to-text paradigm for generating such solution explanations is not adequate for learning the direction of motion for the physical reasoning required for these explanations.
Qualitative physics and Visual reasoning. Qualitative physics aims to represent and reason about the behavior of physical systems (Forbus, 1988). McCloskey and Kohl (1983); McCloskey et al. (1983) suggests that people use simplified intuitive theories to understand the physical world in day-to-day life. Earlier works explored using probabilistic simulations to train physical inference through physical simulations (Battaglia et al., 2013; Zhang et al., 2016). Recent papers use neural networks over visual inputs to predict future pixels (Finn et al., 2016; Lerer et al., 2016; Mirza et al., 2016; Du and Narasimhan, 2019) or make qualitative predictions (Groth et al., 2018; Li et al., 2016, 2017; Janner et al., 2019; Wu et al., 2015; Mao et al., 2019). Furthermore, several frameworks and benchmarks have been introduced to test visual reasoning such as PHYRE (Bakhtin et al., 2019), Mujoco (Todorov et al., 2012), and Intphys (Riochet et al., 2018), some of which are combined with natural language for question answering such as NLVR (Suhr et al., 2017), CLEVR (Johnson et al., 2017), and VQA (Antol et al., 2015). In a parallel work, Yi et al. (2020) introduced the CLEVRR dataset for reasoning about collision events from videos with different types of questions. In contrast, we develop the ability to reason and explain the behavior of dynamic physical systems by generating natural language.
Natural language explanations and Common-sense reasoning. Several recent papers propose
7913
to use natural language for explanation and commonsense reasoning (Lei et al., 2016; Camburu et al., 2018; Forbes and Choi, 2017; Chai et al., 2018; Forbes et al., 2019; Rajani et al., 2019; DeYoung et al., 2020). Lei et al. (2016), for example, generate textual rationales for sentiment analysis by highlighting phrases in the input. Forbes and Choi (2017) learn the physical knowledge of actions and objects from natural language. Camburu et al. (2018) propose e-SNLI by generating explanations for the natural language inference problem at a cost of performance. Rajani et al. (2019) propose to use LMs to generate explanations that can be used during training and inference in a classifier and significantly improve CommonsenseQA performance. Bisk et al. (2020) propose to use a question answering task to test the model's physical commonsense and reasoning ability. In contrast to the previous work, we focus on identifying pivotal physical events and then generating natural language explanations for them. We find that this two-step approach works more effectively.
Table-to-text generation. Table-to-text generation aims to produce natural language output from structured input. Applications include generating sports commentaries from game records (Tanaka-Ishii et al., 1998; Chen and Mooney, 2008; Taniguchi et al., 2019), weather forecasts (Liang et al., 2009; Konstas and Lapata, 2012; Mei et al., 2016), biographical texts from Wikipedia infoboxes (Lebret et al., 2016; Sha et al., 2018; Liu et al., 2018; Perez-Beltrachini and Lapata, 2018), descriptions of knowledge bases (ODonnell et al., 2000; Trisedya et al., 2018; Zhu et al., 2019; Yu et al., 2019) and source code (Iyer et al., 2016), and dialog response generation from slot-value pairs (Wen et al., 2015).
Recently, neural encoder-decoder models (Sutskever et al., 2014; Cho et al., 2014) based on attention (Bahdanau et al., 2015; Luong et al., 2015) and copy mechanisms (Gu et al., 2016; Gulcehre et al., 2016) have shown promising results on table-to-text tasks (Wiseman et al., 2017; Gehrmann et al., 2018; Puduppully et al., 2019a,b; Iso et al., 2019; Castro Ferreira et al., 2019; Zhao et al., 2020; Chen et al., 2020a). While traditional methods use different modules for each generation stage in a pipeline (Reiter and Dale, 2000), neural table-to-text models are trained on large-scale datasets, relying on representation learning for generating coherent and grammatical
texts. Puduppully et al. (2019a) propose a neural network approach that first selects data records to be mentioned and then generates a summary from the selected data, in an end-to-end fashion. Chen et al. (2020b) use pre-trained language models to generate descriptions for tabular data in a few shot setting.
ESPRIT uses a two-step approach for qualitative physical reasoning. To train models that can describe physical tasks, we collected open-ended natural language text descriptions of initial states and pivotal physical events in a 2D simulation from human annotators. We then trained a model to identify these pivotal events and then fine-tuned on pre-trained table-to-text generation and language models without using the image representations of the actual simulation frames.
Our results indicate that table-to-text models perform better than language models on generating valid explanations of physical events but there is a lot more room for improvement compared to human annotations. We hope that the dataset we collected will facilitate research in using natural language for physical reasoning.
Reinforcement Learning (RL) agents may be able to solve physical tasks much more efficiently by leveraging natural language reasoning as opposed to model-free approaches that are often highly sample-inefficient. An RL agent that leverages natural language descriptions of physical events to reason about the solution for a given goal (similar to Zhong et al. (2020)) or for reward shaping (similar to Goyal et al. (2019)) could be a compelling line of future research.
More importantly, having a model that can meaningfully reason about commonsense qualitative physics could be interpretable and more robust, as they might focus on the parts of physical dynamics that are relevant for generalization to new scenarios. Such systems are widely applicable to self-driving cars or tasks that involve human-AI interactions, such as robots performing everyday human tasks like making coffee or even collaboratively helping with rescue operations.
We would like to thank Abhinand Sivaprasad for his helpful discussions and annotations. We also thank the anonymous reviewers for their feedback.
7914
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In ICCV.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019. PHYRE: A new benchmark for physical reasoning. In NeurIPS.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. 2013. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):18327-18332.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. In AAAI.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: natural language inference with natural language explanations. In NeurIPS.
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. In EMNLP-IJCNLP.
Joyce Y Chai, Qiaozi Gao, Lanbo She, Shaohua Yang, Sari Saba-Sadiya, and Guangyue Xu. 2018. Language to action: Towards interactive task learning with physical agents. In AAMAS.
David L Chen and Raymond J Mooney. 2008. Learning to sportscast: a test of grounded language acquisition. In ICML.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In ACL.
Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, and William Yang Wang. 2020b. Few-shot NLG with pre-trained language model. In ACL.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. ERASER: A benchmark to evaluate rationalized nlp models. In ACL.
Yilun Du and Karthik Narasimhan. 2019. Task-agnostic dynamics priors for deep reinforcement learning. In ICML.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. 2016. Unsupervised learning for physical interaction through video prediction. In NeurIPS.
Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In ACL.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? In CogSci.
Kenneth D Forbus. 1988. Qualitative physics: Past, present, and future. In Exploring artificial intelligence, pages 239-296. Elsevier.
Sebastian Gehrmann, Falcon Dai, Henry Elder, and Alexander Rush. 2018. End-to-end content and plan selection for data-to-text generation. In INLG.
Prasoon Goyal, Scott Niekum, and Raymond J. Mooney. 2019. Using natural language for reward shaping in reinforcement learning. In *IJCAI*.
Oliver Groth, Fabian B Fuchs, Ingmar Posner, and Andrea Vedaldi. 2018. ShapeStacks: Learning vision-based physical intuition for generalised object stacking. In ECCV.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In ACL.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In ACL.
Hayate Iso, Yui Uehara, Tatsuya Ishigaki, Hiroshi Noji, Eiji Aramaki, Ichiro Kobayashi, Yusuke Miyao, Naoaki Okazaki, and Hiroya Takamura. 2019. Learning to select, track, and generate for data-to-text. In ACL.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In ACL.
Michael Janner, Sergey Levine, William T Freeman, Joshua B Tenenbaum, Chelsea Finn, and Jiajun Wu. 2019. Reasoning about physical interactions with object-oriented prediction and planning. In ICLR.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR.
Nikiforos Karamanis, Massimo Poesio, Chris Mellish, and Jon Oberlander. 2004. Evaluating centering-based metrics of coherence. In ACL.
Ioannis Konstas and Mirella Lapata. 2012. Unsupervised concept-to-text generation with hypergraphs. In NAACL.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In EMNLP.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In EMNLP.
Adam Lerer, Sam Gross, and Rob Fergus. 2016. Learning physical intuition of block towers by example. In ICML.
Wenbin Li, Seyedmajid Azimi, Ales Leonardis, and Mario Fritz. 2016. To fall or not to fall: A visual approach to physical stability prediction. arXiv preprint arXiv:1604.00066.
Wenbin Li, Ales Leonardis, and Mario Fritz. 2017. Visual stability prediction and its application to manipulation. In 2017 AAAI Spring Symposium Series.
Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In ACL-IJCNLP.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In AAAI.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. 2019. The neurosymbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR.
Michael McCloskey and Deborah Kohl. 1983. Naive physics: The curvilinear impetus principle and its role in interactions with moving objects. Journal of Experimental Psychology: Learning, Memory, and Cognition, 9(1):146.
Michael McCloskey, Allyson Washburn, and Linda Felch. 1983. Intuitive physics: the straight-down belief and its origin. Journal of Experimental Psychology: Learning, Memory, and Cognition, 9(4):636.
Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? selective generation using LSTMs with coarse-to-fine alignment. In NAACL.
Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2016. Generalizable features from unsupervised learning. arXiv preprint arXiv:1612.03809.
Michael ODonnell, Alistair Knott, Jon Oberlander, and Chris Mellish. 2000. Optimising text quality in generation from relational databases. In INLG.
Laura Perez-Beltrachini and Mirella Lapata. 2018. Bootstrapping generators from noisy data. In NAACL.