Mind's Eye: Grounded Language Model Reasoning through Simulation
Authors: Ruibo Liu
Jason Wei
Shixiang Shane Gu
Te-Yen Wu
Soroush Vosoughi
Claire Cui
Denny Zhou
Andrew M. Dai
Venue: ICLR 2023 poster (2023)
Keywords: reasoning
alignment
simulation
physics
grounding

Abstract: Successful and effective communication between humans and AI relies on a shared experience of the world. By training solely on written text, current language models (LMs) miss the grounded experience of humans in the real-world---their failure to relate language to the physical world causes knowledge to be misrepresented and obvious mistakes in their reasoning. We present Mind's Eye, a paradigm to ground language model reasoning in the physical world. Given a physical reasoning question, we use a computational physics engine (DeepMind's MuJoCo) to simulate the possible outcomes, and then use the simulation results as part of the input, which enables language models to perform reasoning. Experiments on 39 tasks in a physics alignment benchmark demonstrate that Mind's Eye can improve reasoning ability by a large margin (27.9% zero-shot, and 46.0% few-shot absolute accuracy improvement on average). Smaller language models armed with Mind's Eye can obtain similar performance to models that are 100x larger. Finally, we confirm the robustness of Mind's Eye through ablation studies.

Published as a conference paper at ICLR 2023
MIND'S s EYE: GROUNDED LANGUAGE MODEL REA- SONING THROUGH SIMULATION
Ruibo Liu 2.2 Jason Wei", Shixiang Shane Gu', Te- Yen Wu", Soroush Vosoughi" Claire Cui', Denny Zhou', Andrew M. Dai
"Google Research, Brain Team, DDartmouth College
ABSTRACT
Successful and effective communication between humans and Al relies on a shared experience of the world. By training solely on written text, current language models (LMs) miss the grounded experience of humans in the real-world -hirdd-hhhherr failure to relate language to the physical world causes knowledge to be misrepresented and obvious mistakes in their reasoning. We present Mind's Eye, paradigm to ground language model reasoning in the physical world. Given a physical reasoning question,
computational physies engine (DeepMind's MuloCo) to simulate the possible outcomes, and then use the simulation results as part of the input, which enables language models to perform reasoning. Experiments on 39 tasks in a physics alignment benchmark demonstrate that Mind's Eye can improve reasoning ability by a large margin (27.9% zero-shot, and 46.0% few-shot absolute accuracy improvement on average). Smaller language models armed with Mind's Eye can obtain similar performance to models that are 100x larger. Finally, we confirm the robustness of Mind's Eye through ablation studies.
INTRODUCTION
"In questions of science, the authority of thousand is not worth the humble reasoning of a single individual."
Galileo Galilei, 1632
"Do objects fall proportionately to their weight?" This famous question was once controversial until Galilco's Leaning Tower of Pisa experiment Galilco dropped two balls of different masses from the same height (i.e., experiment) and concluded that their time of descent was independent their mass (i.e., inductive reasoning). Such an experiment-reasoning paradigm has been used by humans for centuries to ground reasoning on complicated problems (Newell, 1980) and transfer learned knowledge unfamiliar domains (Novak & Gowin, 1984).
Current language models (LMs) follow a different path- by training on natural language, they attempt to reverse engineer the physical world, so that they are able to reason about it. Large- scale pre-trained LMs have achieved revolutionary performance on many tasks, such as solving math word problems (Roy & Roth, 2015; Ling al., 2017; Cobbe et al., 2021) and commonsense reasoning (Talmor et al., 2022; Geva et al., 2021). However, these models do not experience firsthand the situations that described by the language (McClelland al., 2020), and lack the ability to find the correct answers by performing experiments like humans. As a consequence, when asked the same free fall question, one of the most widely-used LMs, GPT-3² (Brown et al., (120)-though achieving superhuman performance in many reasoning tasks will generate the wrong answer: "The heavier object will fall faster." (as shown in Figure I). Due to the lack of grounded reasoning, current LMS also have issues in truthfulness (Lin et al., 2021) and factuality (Petroni et al., 2020).
In Physics, Aristotle (384-322 BC) claims that the speed at which two identically shaped objects fall is directly proportional their weights, which was later challenged by Aristotelian commentator John Philoponus. Speciiiicaly, we use texi-davinci-002, which is the "most capable GPT-3 model" at the time of writing from Ope https: : /betaa openai ccm/ docs/mode lovervii

---

Mind's is Eye: Grounded Language Model Reasoning Through Simulation
Vanilla
Question:
Answer from Vanilla LM:
Two baseballs X and are releesed from rest et the same height.
Answer:
Zero-shot Ressoning
heavier basebball
Xis heavier then Y. Which
ground faster because it has more mass end therefore more gravity.
ground
Chain-of-Thought
Chain-of-Taee Thought on Similar Questions
Answer from CoT + LMM
Question:
Answer:
baseballs
released height.
Fow-shot Reasoning
Since accaloration object Rimm,
X heavier than Y.
will more graviny. so mg/m g.
Which berebel will fall to the ground faster?
ground
Mind''s Eye
Mind's Eye Augmented LM
Simulator Augmented Cero/Few--sho Reasoning
Text-to-Code LM
Physica
Question:
Answer:
Randering
Sinusation Rosuls
are released
same height.
and have the acceleration.
X haavier than
So the answer they will at same
Which basebal will fall to the ground faster?
ground
Sinutation
Simuletion based Prompts Injection
Figure 1: Current language models are still challenged by simple questions that require good understanding of the physical world. The answer elicited by Chain-of-Thoupe can still be wrong the required knowledge is missing or misrepresented in LMs. Mind's Eye, instead, enables grounded LM reasoning by directly simulating the scene in the given question. Then the LM can reason over the injected ground-truth rationale generate the correct answers.
To tackle these problems, existing remedies include using improved prompting techniques, such as inserting hand-written decomposed reasoning steps in few-shot demonstrations (Wei et al., 2022: Zhou al., 2022). These methods are inherently limited as their reasoning ability completely relies on the knowledge perpetuated in the LM- their performance could suffer if the knowledge learnt by the LM is incorrect (Petroni al., 2019) or outdated (Dhingra al., 2022). To incorporate external knowledge, retreeeal-augmented LMs such REALM (Guu et al., 2020). RAG (Lewis et al., 2020) and RETRO (Borgeaud al., 2022), retrieve relevant documents as additional evidence for given triplets. However, the
questions, and may also fine-tune the LM on the question-docummnt--an knowledge presented in written language is known to have reporting bias (Bisk et al., 2020), whereby some everyday unspoken facts or rarely seen (but practically possible) compositions are commonly missing text (Paik et al., 2021).
Correct and complete understanding of properties and interactions in the physical world is not only essential to achieve human-level reasoning (Lake et al., 2017), but also fundamental to build a general- porepal embodied intelligence (Huang al., 2022). In this work, we investigate what extent current LMs understand the basic rules and principles of the physical world, and describe how ground their reasoning with the aid of simulation. Our contributions are three-fold:
We propose a new multi-task physics alignment dataset, UTOPIA, whose aim is to benchmark how well current LMs can understand and reason over some basic laws of physics ($2).
The dataset contains 39 sub-tasks covering six common scenes that involve understanding basic principles of physics (e.g., conservation of momentum in elastic collisions), and all the ground-truth -tuth answers are automatically generated by physics engine. We find that current large-scale LMs are still quite limited on many basic physics-related questions (24% accuracy of GPT-3 175B in zero-shot, and 38.2% in few-shot).
We explore a paradigm that adds physics simulation to the LM reasoning pipeline ($3) to make the reasoning grounded within the physical world. Specifically, transform the given text-form question into rendering code, and then run the corresponding simulation physics engine (i.e., MuloCo (Todorov al., 2012)). Finally we append the
first use model to
2

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
simulation results the input prompts of LMs during inference. Our method can plug-am and-play framework that works with any LM and requires neither handcrafted prompts nor costly fine-tuning,
We systematically evaluate the performance of popular LMs in different sizes on UTOPIA before and after augmentation by Mind's Eye, and compare the augmented performance with many existing approaches ($4.2). We find Mind's Eye outperforms other methods by a large margin in both zero-shot and few-shot settings. More importantly, Mind's Eye is also effective for small LMs, and the performance with small LMs be par or even outperform that of 100x larger vanilla LMs.
2
UTOPIA BENCHMARKING
Humans are able to understand their physical environment and intuit rules about the world from embodied experience. The rules and principles behind the real world have been discovered as scientific laws-we humans have ingrained them as knowledge or intuition (Kaiser et al., 1986) to make reliable predictions how observed events will unfold in day-to-day life (Kubricht al., 2017). For example, when driving, we can anticipate when to brake when approaching a stop sign, using intuition or knowledge from Newton's second law of motion. We also know it would be disaster to collide with heavy truck, not only in terms of our knowledge on the conservation of momentum (i.e., the lighter object will have a greater velocity after collision), but also from our embodied experience of collision in everyday life.
We are thus inspired design a physics alignment dataset that covers this knowledge, aiming to benchmark to what extent current LMs understand basic physical concepts and rules. As shown in Table 1, we choose six representative scenes, mainly from textbooks (e.g., high-school Physics). The sub-tasks are defined based on the composition of observed and queried concepts. For example, one task in a motion scene could be: given the observed acceleration the two objects with the same mass, please answer what is the relationship of forces applied on them. In total we have 39 sub-tasks across different scenes, and each sub-task contains various hand-designed questions whose language style is similar that of textbooks as well.
Table 1: We propose UTOPIA, a multi-task physies alignment dataset, investigating the grounded reasoning ability of LMs on 39 sub-tasks. Unlike many other datasets, UTOPIA deliberately describes the questions in relative relations (e.g., greater than) instead of absolute numbers (e.g., 3.5 m/s), to approximate human's 's perceptional sensing ability in real world. The ground-truth answers to the questions are generated by the physics engine, which makes easy to scale UTOPIA to larger sizes.
Scenes
(Simplified) Sample Questions
Concepts
# Tasks
Amy pulls two sleds X and Y with the same force. X hus a greater mass than Y. Friction can be ignored.
mass
Motion
force
6
Which one has a greater acceleration after the same period of time?
velocity
Two boxes X and Y move at the same velocity. We only consider kinetic frictions, and X undergoes smaller friction than Y.
mass
Friction
velocity
Which one has greater velocity after the same period of time (before stop)?
friction
Two balls dropped from the same height. Y has greater mass than X. We ignore the air resistance.
Free fall
height
Which one will hit the ground carlier?
energy
Jason throws baschalls X and Y the same height horizontally. They have the same mass, but X has greater initial horizontal velocity. Which will hit the ground earlier?
velocity
Projection
energy
Two marbles X of the same mass move towards cach other. and have same magnitude of velocity, and the collision is elastic. Which one will have greater velocity after collision?
velocity
Collision
mass
momentum
Two blocks of metal X and Y are released from certain height on a slick slope. has a greater mass than X, and the friction can he ignored.
mass
Incline
height
Which one will have a greater velocity after the same period time?
friction
Table exemplifies some samples in UTOPIA. We deliberately choose to use relative comparison (e.g., "greater than", "smaller than". "the same as", text in purple) rather than actual numbers to
3

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
describe the physical properties, since we thus able to disentangle the effects from numeracy (i.e.. the gain on reasoning is not attributed to better memorization on numbers, which has been reported as "shortcuts" used by LMs (Patel et al., 2021)). This setting is also different from those in mathematical reasoning tasks (e.g.. GSM8k (Cobbe et al., 2021)), where the decomposed reasoning path is typically the procedure of plugging different values into equations- the LM might be able to solve these problems by symbolic manipulation (Razeghi et al., 2022) rather than actual reasoning.
Most existing physics alignment datasets use vision as the primary modality, such images (Zellers et al., 2019), animations (Wu et al., 2017), or videos (Piloto et al., 2022), which loses the flexibility to run on LMs which only takes text imput. PIQA (Bisk al., 2020) and MMLU-Physiess (Hendrycks et al., 2021) are popular physics reasoning datasets used for LM benchmarking; however, their sizes are naturally limited because of required human annotations (e.g., only 206 samples are on physics in MMLU, with college and high school level questions combined). UTOPIA differs from all these datasets as leverages a physics engine to generate data in theory we can obtain unlimited samples-ate and each sample has reliable ground supported by actual simulation. Although in the present work we only take the text-form data for LM benchmarking. the corresponding simulation videos during data generation have been recorded data for future multi-modality research.
3 MIND'S S EYE
As shown in Figure 1, Mind's Eye comprises three main components, text-to-code LM as the front- end, a physics simulation engine (i.e., MuJoCo) as the back-end, and a foundation model (Bommasani et al., 2021) for general reasoning. We detail the implementation of Mind's Eye as below:
Text-to-Code Converter. The objects and dynamics of the simulation is manifested by the rendering code fed into MuJoCo. The rendering code is written in a type of XML file named MCJF', where the physics properties can be easily controlled by changing some key-value pairs. For example, to change the mass of an object to 10, the line of rendering code needed is geom, set (" mamss' '10'). ). We use actual values to express the relative relationships in UTOPIA (e.g.. "greater" will be translated 10 and 1 for the values of the properties be set). We create rendering templates for each sub-task of UTOPTA, and use programs to generate a dataset with 200,000 text-code pairs. In each pair, the question in text appended to the top of the XML code as comments. We then train decoder-only LMs from scratch to learn how to generate the rendering code given the question in comments regr-segrys..vely. We leverage the BPE vocabulary set from GPT-2 (Radford al., 2019) and extend it by several special tokens to represent repeating tabs or spaces. Besides fine-tuning on the dataset with text-code pairs, we also pre-train the model on the C4 dataset (Raffel et al., 2019a) to enhance the model's 's understanding on natural language. All the training is on TPU-v3 Pods and the resulting models have 0.3B and 1.5B parameters (used as default). See $4.1 for training details.
Simulation Augmented Prompting. Once receiving the rendering code, the physics engine will run the corresponding simulation to get the ground-truth outcome. The program that triggers the simulation will also parse the outcome into text-form prompt injections (e.g.. "Hints: Two baseballs take the same time to hit the ground.", as shown in Figure 1). The injection combined with the question will be fed to the foundation model, with which LMs can ground their reasoning with the physical world rendered by the physics engine. We present more details of this procedure in $A.1.
The intuition behind Mind's Eye is imitate the experiment-reasonine paradigm; however, leverage quick and cheap physics simulation as an alternative to actual experiments in physical world. The cognitive analog for Mind's 's Eye might be the mental visualization process, also known as "the mind's eye" (Battaglia et al., 2013; Hegarty, 2004), which often relates to motor processes (Wexler al., 1998) during embodied reasoning (Nathan et al., 2021).
Discussion: Why does Mind's Eye work? Table 2 shows the comparison between Mind's Eye and two other methods in the formulation of the grounding process during LM inference. Assuming knowledge of the physical world aligns with the distribution Pworts, the Zero-shot Reasoner (Kojima et al., 2022) which uses "Let's think step by step." in prompts can be extended to any number of new tasks. However, its reasoning ability will be compromised if the knowledge in LMs is incorrect or outdated. Similarly, incorporating handcrafted reasoning steps rather than a generic phrase, Chain-
DDes for MICF; https: http:p:///mcco......eeooooooo http:///cooss..oeeeeeeeeeteeeeee -tml
4

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Table 2: Comparison in formulation of how the LM inference process is grounded (given question generating answer y). With the aid of simulator (i.c., MuJoCo), Mind's Eye is not only scalable (not requiring human annotation) but also well-grounded with the physical world.
Method Name
Formulation
Scalable?
Grounded?
Zero-shot Reasoner
arg max; [M ar, "Let's think step by step")
X
Chain-of Chain-of-Thought
arg m LM(g] Chain-of- ofouuhhhttthhtth PHummn
x
Ours: Mind's Eye y arg m L. s Simulator(z, -PWodda
of-Thought (Wei et al., 2022) is able elicit LM reasoning in a more explicit way; however, its performance is reported to be sensitive to the quality of human annotated reasoning steps (Lampinen et al., 2022; Dasgupta et al., 2022) (i.e., PHummn is not similar to Pworia). The dependence on human annotation also limits its scalability.
Mind's Eye overcomes these issues by including a simulator into the reasoning pipeline. Given the question, the simulator (i.e., the physics engine) returns the most likely outcome based on its encoded world knowledge. Since the simulator is accurate enough to approximate the physical world, the prompt injection of Mind's Eye basically serves as scoring machine, which puts probability mass on the answer that is best aligned with the rules of physics-the LM reasoning over the injected rationales is thus grounded. Mind's Eye is also scalable since the whole pipeline is automated.
Besides scalability and grounded reasoning, Mind's Eye is also efficient, since it delegates domain- specific knowledge to external expert models (i.e., the MuJoCo engine for expert physics knowl- edge) (Du et al., 2022: Shazeer al., 2017), which decouples general reasoning from domain specific knowledge. The size of the I.M can thus be significantly shrunk since it removes the burden of memorizing all the domain-specific knowledge. Experiments find that 100x smaller LMs augmented with Mind's 's Eye can achieve similar reasoning capabilities as vanilla large models, and its prompting- based nature avoids the instability issues of training mixture-of-exper models (Zoph et al., 2022). The compatibility with small LMs not only enables faster LM inference, but also saves time during model saving, storing, and sharing.
4 EXPERIMENTS
4.1 EXPERIMENTS SETTINGS
Data and Model. For the convenience of benchmarking on huge LMs, we prepare 100 samples for each sub-task, resulting in dataset with about 3,900 samples. We this version of UTOPIA for evaluation across the paper. The MuloCo simulations can achieve 171 fps on one A6000 GPU, and generating 100 simulations of 2 seconds collision scene takes 0.67s. For LMs, besides GPT-3, we have also tested Pathway Language Model (PaLM) (Chowdhery et al., 2022) on 8B, 62B, and 540B checkpoints. All experiments for PaLM are run on TPU-v4 Pods.
Training Details. Training of the JAX -Axeb text-to-code LMs runs on TPU-v3 Pods. The learning rates we use for training 0.3B and 1.5B LMs on C4 are [3.0c-4. 1.8e-4). which are switched to [L8e-4, 0.5c-4) when fine-tuning on the text-code pairs. We use cosine annealing to control learning rate over time with fixed warm-up steps (3k). As mentioned in $A.2, we have also fine-tuned 62B PaLM to study task generalization, which takes about 25 minutes on 64 TPU-v4 chips for each task.
4.2 BENCHMARK RESULTS ON UTOPIA
As shown in Figure 4.2, we run experiments on UTOPIA with GPT-3 and PaLM of different sizes, ranging from 340M (GPT-3 Ada) to 540B (PaLM), Although larger LMS perform consistently better than smaller LMs in both zero-shot and few-shot settings (n = 5). reasoning ability seems to plateau after a certain size, especially in the few-shot setting. In other words, the scaling curve of vanilla few-shot is nearly flat. One interpretation could be that few-shot demonstrations have managed to trigger effective in-context learning (e.g., for learning the answer format), but the lack of grounded reasoning becomes the bottleneck for further improvement.
5

---

Mind's s Eye: Grounded Language Model Reasoning Through Simulation
Vanilla
92.5
Mind's EEe Augmented Zero-shot
Vanilla Few-shol
Mind's Eye Augmented Few-shot
78.1
Accuracy's
S7.7
957
619
47.7
$7.3
26.0
88.
12,7
12,3
3PT: Ada 353M log(350) 2 2.5M
UPT Bobbage 1.38 kgg()300) -3,10 3.1M
FaLM 8S log(8000) 3.9M
PaLM 628 kggb22000) + 8M
GPT3 1738 (gg(0))00)) + 2M
PALM 5428 leg1540000) 5,7M
Figure 2: Scaling law of reasoning UTOPIA when benchmarking LMs in different sizes (log scale). Smaller LMs augmented with Mind's Eye can achieve on par or even outperform larger vanilla LMs in both zero-shot and few-shot settings (e.g.. 29.8% for GPT3 1.3B Mind's s Eye vs. 29% for vanilla GPT-3 175B zero-shot). In few-shot settings (n 5), the scaling potential is unlocked when LMs are augmented with Mind's Eye (red line), the scaling curve is nearly flat in vanilla few-shot mode (orange line), which demonstrates the power of incorporating knowledge from simulations. On average across all sizes, the gain from Mind's Eye is greater in large LMs than in small ones, and greater in few-shot (34.2%, absolute) than in zero-shot settings (18.4%, absolute).
Mind's Eye, however, unlocks the ability to scale (red line in Figure 4.2) by adding knowledge from physics engine simulations. Since the correctness of the simulation results is guaranteed by the physics engine, the reasoning of the large foundation model is thus well grounded. Compared with solely using knowledge perpetuated in LMs (results with vanilla LMs), Mind's Eye is able boost the reasoning ability of LMs by 18.4% in zero-shot and 34.2% in few-shot settings. Interestingly, smaller LMS augmented with Mind's Eye can achieve similar or even better performance than vanilla larger LMs. For example, the accuracy of GPT-3 Babbage 1.3B with Mind's Eye in zero-shot is 29. 29.8%. while that of vanilla GPT-3 Davinci 175B in zero-shot is 29%. This finding demonstrates the effectiveness of the Mind's Eye paradigm decoupling experimenting from aesoningg-heee the domain specific tool responsible for providing ground-truth results, and the LM can mainly focus on general reasoning, whose size can thus be largely decreased.
Comparison with other reasoning enhanced techniques. In Table 3, we compare Mind's Eye with other methods that improve the reasoning abilities of LMs (i.e., GPT-3 175B). In addition to a) Chain-of-Thoughe (Wei et al., 2022), we consider other prompt-based methods such as b) Zero-shot Reasoner (Kojima al., 2022), which uses "Ler's think step by step." in prompts to induce the decomposed reasoning steps of LMs; e) Self- Cons-ssonstenc Decoding (Wang et al., 2022), which an ensemble technique that decodes multiple reasoning paths concurrently to improve the performance of Chain-of- ThTouhhh the recent study d) DiVerSe (Li et al., 2022b) achieves new SotAs on many reasoning tasks, by using pre-trained verifiers to weight good answers more than bad ones in each step decoding. Besides RAG (Lewis et al., 2020) that retrieves knowledge from the memory of 75GB documents, we also consider other fine-tuned LMs which attempt to optimize reasoning ability from different perspectives, such as e) task-scaling TO (version pp) (Sanh et al., 2021), which tunes T5 (Raffel al., 2020) on thousands of prompts to make LMs better understand input prompts, especially in zero-shot settings, and f) model-scaling Minerva (Lewkowycz et al., 2022), which fine-tunes PaI.M (Chowdhery al., 2022) 540B on a newly collected dataset that contains scientific and mathematical documents (c.g., arxiv papers) to improve quantitative reasoning. Though most experiments are running on GPT-3 175B, we also explore the scaling effect by using the 1.3B Babbage model. The 'grounding gain' is the absolute accuracy difference for the 175B model augmented with and without Mind's s Eye. Unless otherwise stated, we use the default parameter settings recommended by competitor methods.
Results shows that Mind's 's Eye outperforms other methods in both zero-shot and few-shot settings significantly, even if using relatively smaller LM (i.e., GPT-3 Babbage 1.3B). Comparing results

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Table 3: Comparison of Mind's Eye and other methods on UTOPIA benchmarking. Zero-shot Reasoner (Kojima et al., 2022), Chain-of- Thhoughtt (Wei et al., 2022) are two prompt-bascd methods that can elicit reasoning large-scale LMs. Self-consistency (Wang et al., 2022) and DiVerSe (Li et al., 2022b) are decoding- time optimization techniques for LMs reasoning. RAG (Lewis et al., 2020) a retrieval augmented LM, while TO (Sanh al., 2021) and Minerva (Lewkowycz al., 2022) are fine-tuned LMs to improve reasoning ability by task-scaling and model-scaling. We present results of Mind's Eye on GPT-3 175B/1. 1.B,, and find that a 100x smaller LM can outperform a vanilla 175B model (ref.) when armed with Mind's Eye. Interestingly, fine-tuning on prompts to better follow human instructions, Instruct-GPT (Ouyang et al., 2022) can achieve nearly perfect physics alignment in fho-shot. We also annotate the grounding gain of Mind's Eye against vanilla GPT-3 175B.
Motion
Friction
Free Fall
Projection
Collision
Incline
Average
Existing Methods
7ero
Few
7ero
Few
7ero
Few
7ero
Few
7ero
Few
7ero
Few
7ero
Few
GPT-3 175B (ref.)
21.5
34.3
24.3
40.0
22.2
44.5
27.0
31.8
26.5
44.8
22.6
33.9
24.0
38.2
GPT-3 1.3B
15.8
34.3
16.7
33.2
6.7
51.8
18.5
33.0
13.5
33.5
19.6
32.4
15.1
36.4
Instruct-GPT 175B
76.0
85.3
60.2
83.2
39.6
47.6
39.8
50.0
49.0
76.8
37.1
68.6
50.3
68.6
Better Prompting
Zero-shot Reasoner
25.0
35.0
31.6
21.6
32.0
28.5
29.0
Chain-of-Thought
41.5
42.2
53.0
40.3
38.0
41.6
42.8
Optimized Decoding
Self-Consistency DiverSe
38.9
44.5
57.2
48.2
48.8
43.9
46.9
20.2
33.1
17.3
29.6
24.3
37.6
23.1
35.6
25.4
39.3
24.5
37.8
22.4
35.5
Augmented LMs RAG (0.4B + 75GB)
7.4
8.3
10.2
5.2
11.4
8.5
8.5
TO-pp (1IB) Minerva (540B)
12.3
29.4
14.3
34.1
11.4
25.5
13.7
27.1
9.5
16.8
17.2
30,9
13.1
27.3
24.5
45.3
29.8
41.0
6.2
15.5
36.3
51.0
23.3
46.3
24.3
43.4
24.1
40.4
Ours: Mind's Eye w/. GPT-3 175B (*)
52.0
82.5
57.0
82.1
30.3
89.8
65.7
83.8
54.0
79.0
52.3
87.8
51.9
84.2
w/. GPT-3 1.3B
33.7
41.7
35.0
42.3
12.8
70.0
34.8
40.0
30.5
45.0
31.8
41.3
29.8
46.7
w/. Instruct- GPT 175B
100.0
100.0
97.8
100.0
$4.9
61.8
99.8
99.4
100.0
84.3
99.1
Grounding Gain (%)
30.5
48.2
32.7
42.1
8.1
45.3
38.7
52.0
27.5
34.2
29.7
53.9
27.9
46.0
GPT-3 175B and 1.3B, we conclude that 1) larger LMs can better leverage Mind's Eye, especially few-shot settings (46.0% vs. 10.3% average grounding gain), probably because larger LMs are more capable of general reasoning, and 2) solely scaling-up up models is not adequate for reliable grounded reasoning performance. For example, when using a 100x larger LM (1.3B + 175B), the few-shot accuracy is merely boosted by 1.8% (absolute; 36.4% - 38.2%) if using vanilla LMs, but that can be boosted by 37.5% (absolute; 46.7% - 84.2%) if the LMs are augmented by Mind's Eye.
Note that Cnstruct-GPT(Ouyyaan et al., 2022) augmented with Mind's Eye is able achieve nearly perfect performance in few-shot settings (68.6% - 99.1%) 1%). This result is promising because it demonstrates the ideal alignment achievable if the LM is given proper reasoning rationale and has good understanding of the questions (as Instruct-GPTT is optimized for instruction following). The improvement from simply better prompting or decoding methods is limited, since their reasoning completely relies on the internal knowledge perpetuated in the LMs, while the knowledge induced from the LMs could be factually wrong. Among augmented LMs, 540B Minerva the best per- forming one but still falls behind Mind's Eye 175B GPT-3, mainly due to the lack of grounded experience for reasoning. We also find the retrieved evidence of RAG sometimes cannot answer the question, even though it includes entities mentioned in the question, and RAG cannot function well in few-shot settings since the retriever module (i.e., DPR (Karpukhin et al., 2020)) was not pre-trained to handle context that contains multiple question- pairs. We have also discussed the domain generalization ability of Mind's Eye in SA.2, and error analysis in $A.3.
4.3 ABLATION STUDY
Do we really need simulation? In Table 4, we show the performance 1) we randomly alter the simulation results in the prompts to mismatch the physics property asked (e.g., asking velocity but including acceleration results in prompts), and 2) we delete the trigger words at the beginning of the prompts (i.e., "Hints:") "). We also test what would happen if we ground the reasoning incorrect
7

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Vanilla
Mind's Eye
Semi-Mind's Eye
Chain-of-Thhuppe (Faw-shot)
*
Zero-shot Reasoner (Zero-shot)
Motion
Friction
Free
Agcuracy'
Accurcy
Accuracy
8
2
of Shots
of Shots
of Shots
Projection
Collision
Incline
86
72
Accuracc
Accuracy
60
Figure 3: The dynamics of reasoning ability on UTOPIA when increase the number of shots from zero to five, to examine whether we achieve similar performance with more in-context demonstrations instead of using external simulators (as Mind's Eye).
simulation results (c.g., Simulation indicates greater but we use smaller in prompts). We find reasoning mismatched simulation results causes substantial performance drops, while including wrong simulation results will further deteriorate the performance, probably because this case the reasoning misguided even if the vanilla LM can answer the question correctly. We take these results as evidence that the correct simulation results are crucial for grounded LM reasoning. Missing trigger words have marginal influence on the performance, which confirms again that reasoning performance mainly depends on the existence and correctness of the simulation results.
Table Ablation study on the simulation results and the trigger words of Mind's Eye to understand their effeets. We also study whether the correctness of simu- lation will affect the reasoning performance.
Table The effect of using different sizes of text-to-code models (T2C) with GPT-3 175B/1.31 as the foundation model (FM) in the zero-shot and few-shot settings.
Ablation Settings
Zero-shot
Few-shot
LM Size (T2C + FM) Zero-shot Few-shot
Mind's Eye (default)
51.9 1.73
84.2 0.79
0.3B 1.3B
25.1 1.77
43.3 1.12
0.3B + 175B
48.3
82.
Mismatched simulation results
30.4 1.65
54.3 1.28
1.5B + 1.3B
29.8 1.65
46.7 0.82
Missing trigger words
51.0 1.12
83.0 p.73
Incorrect simulation
24.6 1.73
39.6 2.99
1.5B 175B (default) 51.9 1.63
84.2 0.79
Can few-shot replace simulation? Given enough in-context demonstrations, can the LM learn how to make its reasoning grounded internally without relying on external simulations? In Figure 3, we present the dynamics of reasoning performance by gradually including more and more in-context demonstrations in few-shot settings on both vanilla LMs and Mind's 's Eye augmented LMs. We also design aug--aegmented baseline. Semi- Semi-Mind's 's Eye, which has the Mind's Eye style in-context demonstration, but the final shot omits the simulation result- in other words, the LM has to perform reasoning on the generated grounding rationale by itself in the final step. This baseline differs from vanilla few-shot it incorporates some simulation results from the physics engine, and it differs from Chain-of -fTouught (CoT) since the reasoning steps CoT are written by humans.
The result demonstrates neither vanilla few-shot (red line) nor semi-augmented few-shot (yellow line) can provide as good an improvement as Mind's 'ss Eye (blue line). We also find the few-shot reasoning performance of CoT (green line) and Gemi-Mindds Eye (yellow line) has some instabilities which depends on whether the few-shot demonstrations happen to have similar reasoning steps as the given question (final step). The effectiveness of Mind's Eye seems echo the findings in Zhao et al. (2021),
8

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
which confirms that the LM tends to ground its reasoning on "recent context", which is coincidentally the simulation results of the given question.
Using smaller text-to-code LM? Text-to-code LMs convert text-form questions into rendering code. To study its scaling effect, we explore several combinations of using text-to-code LMs and pretrained LMs in different sizes. As shown in Table 5, we find using smaller text-to-code LM (0.3B) has slightly smaller impact for larger pretrained LMs, as the accuracy decreases by 4.7% for 1.3B pretrained LM but 3.6% for 175B in zero-shot. We also find the conversion error is somehow mitigated by few-shot demonstrations (e.g. the accuracy decreases by 3.6% in zero-shot but by 2.1% % in sho-shot when the 175B model uses a smaller T2C LM), indicating the benefits on robustness when using a larger pretrained LM.
RELATED WORK
Grounded Reasoning, Early attempts to relate language understanding (Winograd, 1972) or learn- ing (Roy & Pentland, 2002) the physical world mostly rely manually created linguistic and physical rules (Hermann et al., 2017: Berant et al., 2013). Recent studies have claimed that pre-trained large-scale LMs have already memorized enough world knowledge (Roberts al., 2020; Brown et al., 2020). and enhanced reasoning ability can be achieved by proper prompting (Nye et al., 2021; Wei al., 2021; Sanh et al., 2021). Besides adding decomposed reasoning steps (Wei et al., 2022: Zhou et al., 2022). previous work has tried to add hand-written task descriptions (Raffel et al., 2019b) and targeting formats (Marasović et al., 2021) to the prompts, but such human annotation often costly and expensive at scale (Li & Liang, 2021; Shin et al., 2020). Mind's s Eye, instead, presents new paradigm to ground LM reasoning, via automated simulation rather than human crafted rationales.
Augmented Language Models. Inspired by the evidence that humans require sensory perception for grounded reasoning (Sachs et al., 1981), previous work has tried to augment text inputs to LMs with audio signals (Tsai et al., 2019) or visual perception (Bakhtin et al., 2019; Yi et al., 2018), for improved game playing (Lee et al., 2022), faster language learning (Hill et al., 2020), or better decision making in general (Reed et al., 2022). Our approaches seemingly echoes these findings as we leverage the simulation results as the extra sensory input. To endow LMs with updated knowledge, TALM (Parisi et al., 2022) fine-tunes LMs interactively with augmented inputs from API calls, which is rather costly compared to prompt-based Mind's Eye. PaLM-SayCan (Ahn et al., 2022) uses the pre-trained PaLM 540B to help robots better understand complex instructions that require DoanninggMidd' Eye can be viewed as the reverse: it infuses knowledge from external tools (i.c., the MuJoCo physies engine) into LMs, to further unlock the the reasoning capabilities of LMs.
Modeling the World. Learning by trial and error, humans seem able to learn enormous amounts of common sense about how the physical world works (Lerer et al., 2016)- such observation has inspired the idea of developing a neural model to model the world (LeCun, 2022; Craik, 1967). World models should be capable of planning (Li et al., 2022a), predicting (Amos et al., 2018), and reasoning (Ullman et al., 2017) through interactions with the world. Similarly, Mind's Eye proposes paradigm that reasons over the experimental results predicted by simulation, and the experiments are planned beforehand by the text-to-code LM. Using simulated environments to help learning has been widely adopted by research in robotics (MineRL; Guss et al. (2019)) or computer vision (Kubric; Greff et al. (2022)), while our focus is grounded reasoning in the form of natural language.
6 CONCLUSION AND FUTURE WORK
We have proposed Mind's Eye, novel paradigm that enables LMs to ground their reasoning with simulations of the world. We conclude that Mind's Eye is not only effective and scalable but also efficient, as it is able to boost the reasoning performance of small-scale LMs significantly, requiring neither handcrafted prompts nor costly fine-tuning.
We believe the idea of including simulation into the reasoning pipeline can be easily extended to other domains or applications, especially where simulation engines already exist. For example, instead of the physical world, one can a simulation of economic changes or thermodynamics, aiding policy making or engineering. The dynamic nature of Mind's 's Eye where we generate grounding evidence unlocks the scaling potential of these models.

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
ETHICS AND REPRODUCIBILITY STATEMENT
The goal of Mind's Eye is to present general-purpose purpose paradigm that incorporates knowledge from simulation by professional tools into the LM reasoning pipeline. Though we have already seen significant improvement in reasoning with Mind's s Eye grounding, the performance can be affected by how accurate the simulation is. Unqualified simulators may result in wrong simulation results, grounding with which will lead to even worse results than reasoning without grounding. The artifacts of the LM itself can also affect the final performance since it is responsible for the general reasoning (Bai al., 2022; Liu et al., 2021; 2022). Furthermore, done in English, and therefore we do not claim that our reasoning paradigm will generalize across all languages, although our framework has the potential be extended to other languages by using multilingual LMs. For reproducibility, we run experiments mainly with publicly available LMs (e.g.. GPT-3) and choose baseline methods that have open-sourced sourced implementation.
experiments and analysis
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. ArXiv preprint, abs/2204.01691, 2022. URL https://arxiie www.ariww....oooeoeooooooooooooe alss 2204 01691.
Brandon Amos, Laurent Dinh, Serkan Cabi, Thomas Rothörl, Sergio Gomez Colmenarejo, Alistair Muldal, Tom Erez, Yuval Tassa, Nando de Freitas, and Misha Denil. Learning awareness models. In 6th International Conference on Learning Represenaatioss, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: // // /poenrevv ew net/ forum? 1
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint 2rXiv:22/2.080777 2022.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross B. Girshick. PHYRE: A new benchmark for physical reasoning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett Advances in Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5083-5094, 2019. URL https: httpp//cceeddeeecoco ngs wwwrips.cc/papeeeeeeee pccppappr/211111 9/hash/ 41 91 6c1 57 7776766869ccc2 928 81
Advances
Neural Information Processing Systems 32: Annual Conference
Abst html.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):1332--1333 2013.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pp. 1533-15355 1544, Seattle, Washington, USA, 2013. Association for Computational Linguistics. URL https: /aaaaaaaaanlooyyoa.aaaaaaaaaaaat org /trg//D133 D11-D11611011
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439, AAAI Press, 2020. URL 6239.
https: pittps:/aaaet /aaaaiaaaaaa//a///aa/eaaaaaaaaaa org/ ojs/ iidex.n.... php/ AAAI
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the op- portunities and risks of foundation models. ArXiv preprint, abs/2108.0725a. 2021. URL https://arxtt www.///asss........ /210 07258.
10

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George Bm Van Den Driessche, can--Bptistte Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Pagannni, Ge- offrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kama- lika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sívan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2206-22444 PMLR, 2022. URL https: //ppoceeedings... mll press / v1 22l.22/bboo h22/borgeaud222. htmml.lmmmaa..m.....m..mm
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL http:::tt/occm htttpss:/prooceedingg newripe wwtpppprpp/ppppp/ppppcc papppper/22000 2020/ 2020/hash/ hash// 1457c0d6bfcb49 61010 Abstract ratttaaaaaaa.aataat....c...a html.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311, 2022. utLhttps:
//arxiv.org/ abs/ 2204. 02311.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv preprint, abs/2110. 10. 14168, 2021. URL htttp:://axiiee wwwp:/iwwwr/aa////o/oooooo 2210. 14168.
Kenneth James Williams Craik. The nature of explanation, volume 445. CUP Archive, 1967.
Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models show human-like content effects on reasoning. ArXiv preprint, abs/2207.07085 2022. URL https: http///axvvvv../// org/ ha/p/////// /2bb/22/727 2207. 07051.
Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. Tware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273, 2022.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pp. 5547-5569. PMLR, 2022.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot. Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguisties, 9:346-361, 2021. doi: 10.1 (0.1162/ a 003300 URL
ogy org/ www.orrg//00// facllaallllll....o....m...ae.lla
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hermann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3749-3761, 2022.
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A scage dataset of minecraft demonstrations. In Sarit Kraus (ed.), Proceedings of the Eventy-Eighth International Joint Conference Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 2442-2448 ijcai.org. 2019. doi: 10. ha.4996/j/aai2200999//. URL ht //o.///oii// . org /3 jjaii.ccai........ 201 939.
11

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval aug- mented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3929-3938. PMIR, 2020. URL http: /tte://prree proceede ings. mlr w.rr.prassvvlll 9/ guu20a html.
Mary Hegarty. Mechanical reasoning by mental simulation. Trends in cognitive sciences, 8(6):
280-285, 2004.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference Learning Representattons, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
net/ forum? dtd.BBdddBBBBiiae. 3GmQ.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, al. Grounded language learning in simulated 3d world. ArXiv preprint, abs/I 0aos/1706.06551, 2017. URL https:
//arxiv. org/ abs 706 O 06551.
Felix Hill, Andrew K. Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: /ttt:::://ppneeee opwnrevien.eeeeeeeeeeeee net/forum? i d= 1 GryBtwr.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ArXiv preprint, abs/2201.002077, 2022. URL https: arxia. org/ abs / 22002221...2 07207.
Mary Kister Kaiser, John Jonides, and Joanne Alexander. Intuitive reasoning about abstract and familiar physics problems. Memory & Cognition, 14(4):308-3) 1986.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 emnlp-main m000-mainn500 URL https: / aa aclanthology org/2020 emnlp- momnp-main... moo-main.5555
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Neural Information Processing Systems (NeurIPS), 2022.
James R Kubricht, Keith J Holyoak, and Hongjing Lu. Intuitive physics: Current research and controversies. Trends in cognitive sciences, 21(10)749-755, 20177
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James I. McClelland, Jane X Wang, and Felix Hill. Can language models learn from explanations in context? ArXiv preprint, abs/2204.02329, 2022. URL https:
//arxiv. org/ abs/ 2204.220442 02329.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. 2022.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, lan Fischer, Eric Jang. Henryk Michalewski, et al. Multi galti-gammeee decision transformers. ArXiv preprint, abs/2205. 2022. URL https: httpiaxxvvvvv/ org/ abs 2200/ 15241.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference Machine Learning. ICML 2016, New York City, NY. USA. June 19-24. 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 430-438. JMLR.org. 2016. URL http:/ / proceedi ngs. mlr. p.r.ssess/ v lererl 16, html.
12

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis. Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-angmmented generation for snowledge-intensing NLP tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020, URL http:ttst///ooeeediiiiiii 6b49
nwwrips....... cc/ pwtpppppppp/////apppppppppppp hashh/ -oe html.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, al. Solving quanti- tative reasoning problems with language models. ArXiv preprint, abs/2206. 148558 2022. URL
bitts::/arzes org/ abs/ 2206.222222222222 14858.
Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive seciisonnmakinn ArXiv preprini, abs/2202.01771, 2022a. URL wwtpt:tp////c /tte::///aaiiie /ttp//axvvvvvoo.oo abs////////// 2202. 01771.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pp. 089924497, Online, 2021. Association for Computational Linguistics. doi: 10.1 10.18653/v1/2021. acl-long.353. URL https::p//.... aa/ aclanthology, org/ 20lg/2221..22 acl long. 353.
Yifei Li. Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. ArXiv, abs/2206.00336, 2022b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. ArXiv preprint, abs/2109.077 11910099.958 2021. URL ttttps://aratt org/ //bs/////// 21009. 07958.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 158-167, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10.18653/w1//11--0011
URL wwtph:tt//// http://ccaaaathoooooo
Ruibo Liu, Lili Wang, Chenyan Jia, and Soroush Vosoughi. Political depolarization of news articles using attribute-aware word embeddings. Proceedings of the International AAAI Conference on Web and Social Media, 15()):385-396 2021. URL https: /ojs arg..org/ porg/index wwweinidxxx///// php/ ICWSM/ article / /et///vei////// /10069.
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. Quantifying and alleviating political bias language models. Artificial Intelligence, 304:103654, 2022.
Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E Peters. Few-shot self-rationalization with natural language prompts. ArXiv preprint, abs/2111.08284 2021. URL https: //arxivv.// org/ brs//bs// 2111.111111111111 08284.
James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models. Proceedings of the National Academy of Sciences, 1 117(42)25996-25974, 2020.
Mitchell J Nathan, Kelsey E Schenck, Rebecca Vinsonhaler, Joseph E Michaelis, Michael I Swart, and Candace Walkington. Embodied geometric reasoning: Dynamic gestures during intuition, insight, and proof. Journal of Educational Psychology, 113(5):929, 2021.
Allen Newell. Physical symbol systems. Cognitive science, 4(2):135-183, 1980.
Joseph D Novak and D Bob Gowin Learning how to learn. cambridge University press, 1984.
13

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. ArXiv preprint, abs/2112.00114
2021. URL https::///........ / rx arxiv. org/ abw/abas1/. 2112. 00114.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. ArXiv preprint, abs/2203.001555. 2022. URL https:// arxiv. org/ abs 2203. 02155.
Cory Paik, Stéphane Aroca-Ouellette, Alessandro Roncone, and Katharina Kann. The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp- 823-835, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics, doi: 10. 1853/v1/2021 emnlp-main.66. 2021. emn 1 m- i 63.
URL https ac antho org/
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. ArXiv preprint, abs/2205. 12255, 2022. URL https: //as:://aae //wxxvaxvvv//////////////a///aa/ www..ioo..o....oooooooooooo .org /abs/ 2205. 12255.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online, 2021. Association for Computational Linguistics. doi: 10. 00.1653/v1//02/ wnanca maal-mminn. 168. URL https:tp// / aataaaaaatthhhooo org/ 2021 nww.nnaacaaaaa maiancl-miin... 168.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joins Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2473, Hong Kong, China, 2019. Association for Computational Linguistics. doi: (0.18653/11/D19-122 URL https: /acl//clcc// antho l org/D10 9250.
Fabio Petroni, Patrick Lewis, Aleksandra Pikuus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions. In Automated Knowledge Base Construction, 2020. URL https:://ppereey// htt.nee//eeee. forrm??
id=025X0 PPn.
Luis S Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learning in a deep-learning model inspired by developmental psychology. Nature human behaviour, pp. 1-11, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8): 2019. URL https:://ccmtt openai cowp:/eteeeeeee--gggg-------- models booddaaeeeeeuusrrrrrrrreetttttte
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019a.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv preprint, abs//910/ 10683, 2019b. URL htthst::: //arxivv...v.///// org/ org/abs/////eeeoo.ooe/..rec.aatr 1910. 10683.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21 21(140):1-677 2020.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on fhot reasoning. ArXiv preprint, abs/2202.07206, 2022. URL https: //arxivv //tpxiv.rgg/ab///...../ h2tpab////222222 07206.
14

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez. Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. ArXiv preprint, abs/2205.00115,, 2022. URL 2205. 061 75.
org/ abs/
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1418-5426, Online, 2020. Associa- tion for Computational Linguistics. doi: 1111166//1122222.mmmmmmmm //aclantholoo ogyy org/ 2020.0020222 emnlp-main 437.
URL https:
Deb K Roy and Alex Pentland. Learning words from sights and sounds: computational model. 2002.
Cognitive science,
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1743-1752, Lishon, Portugal, 2015. Association for Computational Linguistics. doi: 10. 10.18633//11DD15120000 URL
https: / /twaaaaaannnooyo..oooo org D15 1202.
Jacqueline Sachs, Barbara Bard, and Marie L. Johnson. Language learning with restricted input: Case studies of two hearing children of deaf parents, Applied Psycholinguistics, 2(1):33-54, 1981.
doi: 10.1017/S01427 14400000643.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. ArXiv preprint, abs/2110.08207, 2021. URL https:
// farxiiv.org/ abs 2210 08207.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The aparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttps://openreet net/ Eorum? 11did1111@i.....
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222-4235, Online, 2020. Association for Computational Linguistics. doi: 10. 10.18633/v1/2220 moalp-mmin..33 URL ht tps : aat/aacaannnttt www.attooonnco....ocoooo org/ / 2020 emnlp mww..annn...........mm..... 346.
Alon Talmor, Oni Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsenseqa 2.0: Exposing the limits of ai through gamification. ArXiv preprint, abs/2201 abs/2201.05320, 2022. URL ht htt.tttss///aaattt
org/ abs / 2200 05320.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi:
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6558-6569, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1656. URL https: : http///aaaanntttoyo.ooooooo org/P19 1656.
Tomer D Ullman, Elizabeth Spelke, Peter Battaglia, and Joshua B Tenenbaum. Mind games: Game engines as an architecture for intuitive physics. Trends in cognitive sciences, 21(9):649 6659996655566 2017.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V. Le. Finetuned language models are zero-shot leamners. ICL.R, 2021.
15

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. Conference on Neural Information Processing Systems (NeurIPS), abs/2201.1 1903, 2022.
Mark Wexler, Stephen M Kosslyn, and Alain Berthoz. Motor processes in mental rotation. Cognition, 68(1):77-94, 1998.
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben- gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 153-164, 2017. URL htttps://proceette wew.eeediiinnnnniiii..eeeeeeeeee pctppppe//211//77 //aah// Abstract Abs html.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic VQA: disentangling reasoning from vision and language understanding. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 1039-1050, 2018. URL https: pttpp//pcccennggg.ceee neurips httpcc//aaaa htcppappr//0// /hash/ 5e388 03.0033333.3 3ew.eeeeeedaaeeeeeeeeeeeeeeeeeee d7 6a 9ccd - Abstract. html.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6720-6731. Computer Vision Foundation IEEE, 2019. doi: 10.1109/CVPR.2011 201900088.. 00688. thecvf. com wwt.:ottttttttceeeoo Stw.suaaallcCooseeeeeeeeeeee hta.srseeeeeeonnnn__eeee_e 201
URL http: /opp://cep.ccss.../ / cttaa//ooteet_ceeteeeto CVPR 201 /tttmm/eeeeelaaaaaeeeeeeea _ers__From becognition_eee
2tt___221___pp_________________e paper html.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, I, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 2021. URL http://p/// / /ttpppeeocceg.see//ee/eepe mlr. press/eeeeeeeee. vv39//19/.... zhaaalc.cc...
http///11////1cl..eeee html.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. ArXiv preprint, abs/2205. 2022. URL htlthttss://aaat /tttpss//arxxv.org abs/2200 10625.
Barret Zoph, Irwan Bello, Samcer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. ArXiv preprint, abs/2202.08906,, 2022. URL https: /nttts://arxx: //axivvvvvvv/////////a/..aa///wl www.aiivv.oooooeeeeeooo.oeeooore abs/ 2202. 08906.
16

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
A APPENDIX
A.I DETAILS ABOUT MIND'S EYE
In Figure A.I we have presented the whole procedure on how a text question is converted into rendering code for simulation, and how the simulation results are parsed by the manager program to produce simulation based prompt injection.
Sindation Manager Program
NJCF fil used MuseCo (impltec)
Pelgerrng Sinutddie
Quastion: Bissure higg
heover
Sapenplare" sine"2.3 apt
cumenu" 1
Pacordin Sensary
-Jatrnp
quaa jelarian (ASU)T)))
cametry" -.4 MD
same accelestion
Accole and
Text for LM
Enargec Xwil have greater lénetic energy same period
Figure A1: We demonstrate the whole pipeline of how Mind's Eye generates the simulation based prompts injection for a given physics question. We highlight those attributes in the rendering code that reflect the relative relationship in UTOPIA question. A simulation manager program is responsible for triggering simulation and parse the results. Here we only show two of physics properties that the manager records for demo purpose.
Trained on many text-code pairs, the text-to-code LM is able to generate the corresponding rendering code for the given physics related question. Note that in the generated rendering code, besides the MJCF file itself, in the last line there are special signs #%##% to record the scene name and the physics attributes of tiierestthess, synthesis stage prepared for the text-to-code LM. After fine-tuning, the text-to-code LM should be able to generate such information.
intt-mmiforationn added during fine-tuning data (text-code pairs)
Once the simulation manager program receives the rendering code, it will execute the simulation based it. Sensory data such as velocity, acceleration, energy, etc. will be recorded (via MuJoCo APIs). Finally, the manager program will parse the data in terms of the physical properties of interests (coming with the rendering code, marked by #%.#%.. For example, in Figure A. the sensory data is recorded along with the free fall simulation, and since the asked property is "acceleration" the manager program will extract the sensory ground-truth data only on acceleration, and draws the conclusion "X and Y will have the same acceleration." by filling in pre-defined "conclusion templates" (e.g.. "The (physics property) of X will be /greater/smaller than that of Y.") with observations. Some trigger words will be concatenated before (e.g., "Hints::" ") the generated conclusion. The connection words (e.g., "So the answer is: ") are also added to elicit the final answer. The whole concatenation will be appended after "Answer:" grounded rationale for LM reasoning (as shown in Figure 1 in the main body of the paper).
A.2 SCENE GENERALIZATION WITH MIND'S EYE
By injecting simulation based prompts during LM inference, Mind's Eye does not need fine-tune the LM every time it encounters new contexts. This feature is appealing as we expect Mind's Eye can serve as a powerful tool for building autonomous intelligence that can ground its reasoning or decision making even in unfamiliar contexts. To clearly show the advantages of Mind's Eye on scene generalization, we compare the performance of fine-tuned PaLM 62B with that of vanilla PaLM 62B armed with Mind's Eye.
17

---

Mind's Eye: Grounded Language Model Reasoning Through Simulation
As shown in Figure A.2, find the fine-tuned model can achieve good performance in-domain data, but its performance is quite limited in scenes it has not been fine-tuned on. For example, in Figure A.2 (a), fine-tuned on the Motion scene of UTOPIA accuracy, but the accuracy decreases to 24% if we test this model on the Projection scene (> 50% performance drop). However, shown in Figure A.2 (b), Mind's Eye enables the L.M to perform consistently well, even if the LM has never seen the scene before.
obtain 70% in domain alignment
Motton
Frided
Progetio
Colliia
Froe
0.47
0.21
0.8
0.48
0.79
0.42
0.88
0.40
Projectin-
0.24
0.15
0.32
0.68
0.33
0.10
0.42
0.47
0.40
0.29
0.70
0.38
0.35
0.23
0.82
0.24
0.47
0.87
0.65
0.79
0.76
0.28
0.76
0.78
(b)
Figure A2: We compare the performance of (a) fine-tuned 62B PaLM on six scenes UTTPPIa data respectively with (b) vanilla 62B PaI.M armed with Mind's Eye in zero-shot. Though fine-tuning can achieve good performance on in-domain data, can hardly generalize to OOD cases (i.e., scenes on which the LM has not been fine-tuned). Prompting based Mind's ' Eye can enable large LMs generalize to unseen new scene with consistent alignment performance even in zero-shot settings. In (a), the performance of fine-tuned PaLM 62B in-domain test sets is in diagonal, while that of of-domain is the rest. The zero-shot performance of PaLM 62B + Mind's ''s Eye in six scenes is listed in (b).
A.3 MIND'S 's EYE ERROR ANALYSIS
In Figure A.3 we show three error cases when using Mind's Eye. Ignorance error refers to the reasoning apparently ignores the given grounded rationale. Recency bias means the LM tends to extract the final answer from nearby words (e.g., the object Y). "I doní know" error are the cases where the LM simply generates non-sense when unable to reason about the question, All these cases are much less common when we run experiments on larger LMs.
ignonence
Recency Bias
don''t know
Question:
Question:
Questions
There are two pieces of wood and
dlding
Assuming
redistannn,
moving other horloomslly
horizontaly by at centain vaboltiss They have
objectivve and'y drop certain heights, and Y Thean the deme mase The height of Xin gowater than theer ct'V fwelow wither them stops, ond ghound
the oone maos, Wes only condider kinetic.
that offy.
filesione problem. and't have
which teater their alantic: collidon??
same volocity initiall The
Anewer:
Hinta X hus greater megritudo velccity after the collide don.
and Move acceleration
Hette her emadeer than-X the some pariod of
X. meeon undder
arawver y
the theews Xolil poffer.
Figure A3: We exemplify several error cases when using Mind's Hye. In general, we find these emrors mostly happen in small LMs, because their reasoning ability over given evidence is limited.
18


Reviews:
[{"review_id": "wvO__tiEcWC", "reviewer": "Reviewer_pY5b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to create a pipeline that combines inputs of a language model with a computation physics engine to simulate possible outcomes, attempting to provide external world information into a text-only language model to assess whether this allows them to reason better.  They create a range of such tasks (as a benchmark dataset) and evaluate the improvement in performance across a range of model sizes. They see that this approach of giving in extra information from a physics engine as part of a language models context allows better performance, and additionally allows smaller language models to achieve performance at-par with the larger text-only ones.\n", "summary_of_the_review": "Interesting evaluation paper and release of dataset.\n", "strength_and_weaknesses": "Strengths\n\n1. This paper is well-motivated and attempts a simple and succinct approach towards adding in additional world information into a language model for physical reasoning tasks. \n2. They create and release a dataset that focuses on physical reasoning tasks and use this to evaluate the models to see how they perform with augmented physical information\n3. They illustrate and discuss how this is an attempt at a method that is both scalable and “grounded”---although it is worth noting that the inputs/outputs are still only ever in text format and the base LM has still never experienced input apart from text; however the additional context given by the physics simulator can serve as a proxy for non-textual information.\n\n\nQuestions / Weaknesses\n\n1. Are there any insights on which reasoning abilities might be easier to gain information from from text (e.g., looking at the categories of motion, friction, free-fall etc.). This could be verified via co-occurrence statistics e.g., looking at which related words occur more in the training datasets, as well as which related answers might occur more in correspondence with the question in the prompt.\n2. It might be good to have a memorisation control to test whether or not the answers generated by the model might have simply existed somewhere in it's training data (which means it's not actually using the outputs from the physics simulator to reason/generate the correct response).\n3. Can the authors report error bars and variance in all the accuracy results?\n4. Some missing citations in grounded reasoning for text-only LMs: Li et al., 2021; Patel and Pavlick, 2020.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mLOQzt7_nZV", "reviewer": "Reviewer_46te", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at understanding and improving the ability of current LMs on reasoning over some basic laws of physics. A new text-based automatically generated physics QA dataset, UTOPIA, is proposed, and the authors showed that current LMs struggle on it. The authors then propose Mind's Eye to improve the physics reasoning ability by incorporating physics simulation results in the prompting. Specifically, a separate LM is trained to generate a simulation config from the original question, and the simulation result is appended to the prompt as a hint. With this method, LM zero-shot and few-shot performances are improved significantly across all scales reported and the improvement scales as the model size. ", "summary_of_the_review": "Overall, this paper presents a proof-of-concept way to combine external physics knowledge and the general reasoning ability of LM in a very simple setting. The demonstrated performance gain across scales is impressive and reveals important insights. However, the generalizability of the specific proposed method to more complex settings is unclear.", "strength_and_weaknesses": "Strength:\n- The proposed method is simple and effective at improving the zero-shot and few-shot physics reasoning performances of LMs.\n- Thorough experiments at various scales are performed.\n- The proposed dataset can be generated automatically from simulation and potentially extensible to more complicated scenarios.\n\nWeaknesses:\n- The current form of Mind's Eye seems very specifically designed for the proposed dataset UTOPIA, which only contains simple physics questions that compare 2 objects and are generated by physics simulations to begin with. The generalization to other datasets probably will be bottlenecked by training a good text-to-code converter and the complexity/ambiguity of the physics scene.\n  -  In UTOPIA it is easy to obtain text code pairs for training the text-to-code converter since the dataset itself is generated by simulation. However, for other datasets (like PiQA or MMLU Physics) how can sufficient training data be obtained?\n  - There are also complex physics scenes (e.g. liquid and deformable bottles etc in PiQA) that could be hard or lengthy to fully specify to a simulator. And the resulting physics metric to read out could also be more specific than only a binary comparison.\n- The proposed method of injecting a physics simulator is conceptually very similar to the injection of calculator used in GSM8K dataset (Training Verifiers to Solve Math Word Problems). The specific idea of using physics simulation to help solve textual physics QA is also already explored by previous works (e.g. Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning; the physics here is simulated via a trained neural network to answer the more complex counterfactual questions with video input). These related previous works should also be cited and acknowledged. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ii7my9JhxI", "reviewer": "Reviewer_TaNC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new paradigm – adding a computational physics engine to aid language modeling on physical reasoning tasks.  \n", "summary_of_the_review": "This article greatly improves language models’ reasoning abilities on physical related tasks. This idea is pretty novel and this method consists of human physical reasoning steps: carry about simulation then write sentences about it. ", "strength_and_weaknesses": "*Strength:*\n\nThe idea proposed by this paper is pretty novel and effective; \n\nThis paper is well written and easy to understand;\n\nThis paper provides a new benchmark to test physic related reasoning tasks;\n\nThis paper has completed comparisons with current reasoning enhanced techniques, such as Chain of Thought, “Let’s think step by step” with zero-shot priming, Self-consistency decoding. \n\n*Weaknesses:*\n\nI hope this method could be open sourced, which will promote the development of the whole community.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster