Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper attempts to create a pipeline that combines inputs of a language model with a computation physics engine to simulate possible outcomes, attempting to provide external world information into a text-only language model to assess whether this allows them to reason better.  They create a range of such tasks (as a benchmark dataset) and evaluate the improvement in performance across a range of model sizes. They see that this approach of giving in extra information from a physics engine as part of a language models context allows better performance, and additionally allows smaller language models to achieve performance at-par with the larger text-only ones.

strength_and_weaknesses: Strengths

1. This paper is well-motivated and attempts a simple and succinct approach towards adding in additional world information into a language model for physical reasoning tasks. 
2. They create and release a dataset that focuses on physical reasoning tasks and use this to evaluate the models to see how they perform with augmented physical information
3. They illustrate and discuss how this is an attempt at a method that is both scalable and “grounded”---although it is worth noting that the inputs/outputs are still only ever in text format and the base LM has still never experienced input apart from text; however the additional context given by the physics simulator can serve as a proxy for non-textual information.


Questions / Weaknesses

1. Are there any insights on which reasoning abilities might be easier to gain information from from text (e.g., looking at the categories of motion, friction, free-fall etc.). This could be verified via co-occurrence statistics e.g., looking at which related words occur more in the training datasets, as well as which related answers might occur more in correspondence with the question in the prompt.
2. It might be good to have a memorisation control to test whether or not the answers generated by the model might have simply existed somewhere in it's training data (which means it's not actually using the outputs from the physics simulator to reason/generate the correct response).
3. Can the authors report error bars and variance in all the accuracy results?
4. Some missing citations in grounded reasoning for text-only LMs: Li et al., 2021; Patel and Pavlick, 2020.


clarity,_quality,_novelty_and_reproducibility: This paper is clearly written and seems fairly straightforward to reproduce assuming the authors release the dataset.

summary_of_the_review: Interesting evaluation paper and release of dataset.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new paradigm – adding a computational physics engine to aid language modeling on physical reasoning tasks.  

strength_and_weaknesses: *Strength:*

The idea proposed by this paper is pretty novel and effective; 

This paper is well written and easy to understand;

This paper provides a new benchmark to test physic related reasoning tasks;

This paper has completed comparisons with current reasoning enhanced techniques, such as Chain of Thought, “Let’s think step by step” with zero-shot priming, Self-consistency decoding. 

*Weaknesses:*

I hope this method could be open sourced, which will promote the development of the whole community.

clarity,_quality,_novelty_and_reproducibility: *Quality:*

This paper has high quality. See above strengths. 

*Novelty:*

This paper is pretty novel

*Reproducibility:*

This paper doesn’t provide source code / interaction website.

summary_of_the_review: This article greatly improves language models’ reasoning abilities on physical related tasks. This idea is pretty novel and this method consists of human physical reasoning steps: carry about simulation then write sentences about it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims at understanding and improving the ability of current LMs on reasoning over some basic laws of physics. A new text-based automatically generated physics QA dataset, UTOPIA, is proposed, and the authors showed that current LMs struggle on it. The authors then propose Mind's Eye to improve the physics reasoning ability by incorporating physics simulation results in the prompting. Specifically, a separate LM is trained to generate a simulation config from the original question, and the simulation result is appended to the prompt as a hint. With this method, LM zero-shot and few-shot performances are improved significantly across all scales reported and the improvement scales as the model size. 
strength_and_weaknesses: Strength:
- The proposed method is simple and effective at improving the zero-shot and few-shot physics reasoning performances of LMs.
- Thorough experiments at various scales are performed.
- The proposed dataset can be generated automatically from simulation and potentially extensible to more complicated scenarios.

Weaknesses:
- The current form of Mind's Eye seems very specifically designed for the proposed dataset UTOPIA, which only contains simple physics questions that compare 2 objects and are generated by physics simulations to begin with. The generalization to other datasets probably will be bottlenecked by training a good text-to-code converter and the complexity/ambiguity of the physics scene.
  -  In UTOPIA it is easy to obtain text code pairs for training the text-to-code converter since the dataset itself is generated by simulation. However, for other datasets (like PiQA or MMLU Physics) how can sufficient training data be obtained?
  - There are also complex physics scenes (e.g. liquid and deformable bottles etc in PiQA) that could be hard or lengthy to fully specify to a simulator. And the resulting physics metric to read out could also be more specific than only a binary comparison.
- The proposed method of injecting a physics simulator is conceptually very similar to the injection of calculator used in GSM8K dataset (Training Verifiers to Solve Math Word Problems). The specific idea of using physics simulation to help solve textual physics QA is also already explored by previous works (e.g. Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning; the physics here is simulated via a trained neural network to answer the more complex counterfactual questions with video input). These related previous works should also be cited and acknowledged. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and of high quality. As mentioned in the previous section, the proposed method is conceptually a combination of two previous lines of work. Nonetheless, this combination is novel and reveals unique insights about using LM in a domain constrained by a lack of knowledge.

summary_of_the_review: Overall, this paper presents a proof-of-concept way to combine external physics knowledge and the general reasoning ability of LM in a very simple setting. The demonstrated performance gain across scales is impressive and reveals important insights. However, the generalizability of the specific proposed method to more complex settings is unclear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: There are overall positive reviews without many major issues. The author responses have properly addressed many weaknesses raised in the reviews.

Reviewer 46te pointed two limitations of this work which are valid concerns. However, the paper is still good as the first (successful) attempt in this direction.
justification_for_why_not_higher_score: Reviewer 46te pointed two limitations of this work which are valid concerns.
justification_for_why_not_lower_score: There are overall positive reviews without many major issues. The author responses have properly addressed many weaknesses raised in the reviews.
Decision: Accept: poster