NEWTON: Are Large Language Models Capable of Physical Reasoning?
Authors: Yi Ru Wang
Jiafei Duan
Dieter Fox
Siddhartha Srinivasa
Venue: EMNLP 2023 Findings (2023)
Keywords: physical reasoning
robotics
object-centric
evaluation
benchmark
reasoning
TL;DR: A dataset and benchmark for evaluating the physical reasoning capabilities of language models.

Abstract: Large Language Models (LLMs), through their contextualized representations, have been empirically proven to encapsulate syntactic, semantic, word sense, and common-sense knowledge. However, there has been limited exploration of their physical reasoning abilities, specifically concerning the crucial attributes for comprehending everyday objects. To address this gap, we introduce NEWTON, a repository and benchmark for evaluating the physics reasoning skills of LLMs. Further, to enable domain-specific adaptation of this benchmark, we present a pipeline to enable researchers to generate a variant of this benchmark that has been customized to the objects and attributes relevant for their application. The NEWTON repository comprises a collection of 2800 object-attribute pairs, providing the foundation for generating infinite-scale assessment templates. The NEWTON benchmark consists of 160K QA questions, curated using the NEWTON repository to investigate the physical reasoning capabilities of several mainstream language models across foundational, explicit, and implicit reasoning tasks. Through extensive empirical analysis, our results highlight the capabilities of LLMs for physical reasoning. We find that LLMs like GPT-4 demonstrate strong reasoning capabilities in scenario-based tasks but exhibit less consistency in object-attribute reasoning compared to humans (50\% vs. 84\%). Furthermore, the NEWTON platform demonstrates its potential for evaluating and enhancing language models, paving the way for their integration into physically grounded settings, such as robotic manipulation. Project site: https://newtonreasoning.github.io

NEWTON: Are Large Language Models Capable of Physical Reasoning?
Yi Ru Wang!
Jiafei Duan'
Dieter Fox
Siddhartha Srinivasa
University of Washington NNVDIA
(yiruwang, duanj1, fox, siddh]@cs. washington.cm newtonreasoning.gihhlle github.io
Abstract
noed prepare a work table play area for kids. Which of Solssors, pizza, cake, banana should remove?
Large Language Models (LLMs), through their contextualized representations, have been em- pirically proven encapsulate syntactic, se- mantic, word sense, and common-sense knowl- edge. However, there has been limited ex- ploration of their physical reasoming ahilities, specifically concerning the crucial attributes for comprehending everyday objects. To ad- dress this gap, we introduce NEWTON, a repository and benchmark for evaluating the physics reasoning skills of LLMs. Fur- ther, to enable domain-specific adaptation of this benchmark, we present pipeline en- able researchers generate variant of this benchmark that has been customized to the ob- jects and attributes relevant for their applica- tion. The NEWTON repository comprises collection of 2800 object-atrribute pairs, pro- viding the foundation for generating infinite- scale assessment templates. The NEWTON henchmark consists of 160K QA questions, cu- rated using the NEWTON repository to in- vestigate the physical reasoning capabilities several mainstream language models across foundational, explicit, and implicit reasoning tasks. Through extensive empirical analysis, results highlight the capabilities of LLMSs for physical reasoning. We find that LLMSs like GPT-4 demonstrate strong reasoning ca- pabilities in ecenario-based tasks but exhibit less consistency in object-attribute reasoning compared humans (50% vs. 84%). Further- more, the NEWTON platform demonstrates its potential for evaluating and enhancing lan- guage models, paving the way for their inte- gration into physically grounded settings, such as robotic manipulation. Project site: https: / //newtonreasoning ng. gi thub.io
Gaket
Sassonsi
Pizza
Zoo of LLMs
GPTs
BERTS
Figure 1. As works begin leveraging LLMs in physically grounded contexts, is crucial to understand whether such models possess the ability to reason about everyday scenarios. NEWTON, repository, pipeline, and benchmark, facilitates evaluation of various LLMs in a physical reasoning context.
(Zhao et al., 2023). As we start using LLMs in phys- ically embodied pipelines (Driess et al., 2023; Ahn et al., 2022; Wu et al., 2023), it is crucial to com- prehensively understand the extent that LLMs can perform physical reasoning. Some studies have pro- posed generalized frameworks to assess language model performance (Ribeiro et al., 2020; Kiela et al., 2021), while others have designed question answering and reading comprehension datasets probe LLMs (Zellers al., 2018; Chen et al., 2019; Rogers et al., 2023). However, few have explored the physical reasoning ability of LLMs.
Physical reasoning involves the cognitive pro- cess of comprehending and predicting the dynam- ics of physical systems based on observable phe- nomena and fundamental principles (McCloskey et al., 1983; Carey, 2000). It encompasses the capacity to make sense of the world by applying knowledge of attributes, such as brittleness, mal- leability, etc. By considering brittleness, for in- stance, we recognize the need to handle an object with caution, while malleability suggests that an ob- ject can be casily reshaped under pressure without fracturing. These abstract concepts enables reason-
1 Introduction
Natural Language Processing (NLP) has made re- markable progress using contextualized represen- tations trained on extensive unprocessed text data

---

ing about the response of objects interactions or changes in the environment, shown in Figure Nevertheless, creating an evaluation framework for physical reasoning is difficult, primarily due to the lack of paired object-attribute data. Humans possess wealth of knowledge regarding the inter- nal structure of objects and their interactions with the physical world. However, this knowledge is of- ten implicitly acquired, making it difficult to explic- itly represent such information. Previous studies in the field of physical reasoning, such as Bisk et al. (2020); Aroca-Ouellette et al. (2021), have focused on common-senne -eenss reasoning or small-scale valida- tion. As a result, the need for comprehensive and systematic assessment of physical reasoning remains an open problem.
(Deitke et al., 2023), YCB (Calli et al., 2015),
Scanned Objects by Google (Downs al., 2022).
and Amazon Berkeley Objects (Collins et al., 2022).
To establish a common representation of categories for the combined objects, we match the object ti- tle of each 3D object and approximate it to the nearest WordNet Synset (Miller, 1995). Object cat- egories are then filtered for redundancy (repeated categories), ambiguity (uncommon, abstract cate- gories), and irrelevance (non- monmmiaiilleblle objects), resulting in a curated collection of 700 common houschold manipulable objects. Using a subset of the objects (5%), iterations of pilot studies were run to fine-tune the format of humann-facin ques- tions, and identify the most comprehensive yet non- redundant set of physics attributes. Ultimately, we identified categories of malleability, elasticity, stiff- ness, softness, sharpness, surface smoothness, sur- face hardness, and brittleness.
To address this gap, we propose NEWTON, a repository, pipeline, and benchmark designed to evaluate the physical reasoning capability of LLMs. The NEWTON repository consists of labeled object-centric data, crowd-sourced for 700+ ob- jects across 8 physical attributes. The NEWTON pipeline introduces method for systematically generating infinite evaluation questions tailored to specific use cases. The NEWTON benchmark consists of 160K pre-generated questions of pro- gressive difficulty, spanning tasks of foundational attribute comprehension, explicit upplication, and implicit scenario- based analysis. Extensive empiri- cal findings demonstrate the unique contributions of NEWTON, revealing its usefulness for evalu- ating LLMs' understanding of underlying physics principles that dictate the behavior and properties of objects in everyday scenarios. Moreover, NEW- TON effectively compliments the existing reper- toire of reasoning benchmarks and datasets, further enhancing the potential to assess and refine the physical reasoning capabilities of LLMs.
2.2 Crowdsourcing Framework
In order to acquire accurate ground truth annota- tions for object-attribun pairs, we have devised a likert-seale annotation setup where annotators are tasked with selecting the most appropriate option from a 3-point scale to depict the given pair. The utilization of 3-point likert scale serves two pri- mary purpose:: simplicity and representation of extremes. The objective of the interface is to col- lect responses that facilitate the categorization of data into distinetly opposing groups based on at- tribute conditions. Thus, the 3-point scale offers a straightforward Decisiin-makinn process for users, minimizing complexity, and allowing for the selec- tion of extreme options when there is a high level of certainty, while uncertain responses prompt the selection of the middle option. To gather these annotations, we have employed Label Studio En- terprise 1, create tasks where a minimum of four annotators contribute overlapping annotations for each object-attribute pair.
2 NEWTON Repository
At the core of constructing an evaluation frame- work for physical reasoning lies the need for a representation that captures the essential attributes of objects. In this section, we highlight the NEW W- TON repository, including the identification and shortlist of objects and attributes, and obtaining a set of consistent abject-atrribute annotations.
2.3 Annotation Process
We provide an example of the annotation interface in Figure 2. Each annotation task consists of three main components: a question which specifies the object category and attribute, a set of 10 randomly- sampled images illustrating common objects within the object category, and three answer choices. The question is designed to incorporate some descrip-
2.1 Objects and Attributes
To preserve grounding to physical objects, we lever- age the mainstream 3D object datasets: Objaverse
http:///umaanniguullcoo/aaaaaaap

---

Annotation
Filtering
Template Generation
LLM Evaluation
Category Filtering
Annotation Interface
Retrieving Object
WordNet Sunset
NEWTON Benchmark
Categories
Matching
What the spolarity cattributes
Foundational Object
Table
Irelovant
objeect out aobject bo, cabject 2>, sobject 3o, and cobject 4o?
Attrbule
Chinawaro
Comprehension
Bettin
Overlapping
Walking
Explici Applicaion
WMaling
Backpack
Backpask
Syneet
Specification
Condition
Satlstying Condition
Impliott Scenario- Analysia
Object Datasets
Bluaberiie
Onject
Ambigunus
Inter annotator Agreement
Attribute
Polarity: Posltive
Objects Not
Britle
Condition
Brittoness
Malleability
NEWTON Repository of Altrict- Pairs
Sample Shuffle
Smeothmees Surface Hardiwass
Elastiiiyy
Object-Attriban Template Formulation
Final Template
Assess!
What is most brittle object out cobject 1>, cobject 2b, <object So. sobject do-?
Sharpness
High Soffmenss
High
High Malleablityy
Britfeness
Low Bithesss
Pigure 2. NEWTON Pipeline. In addition facilitating the curation of the NEWTON repository and benchmark, the NEWTON pipeline enables convenient extensibility evaluation measures to suit any scenario. The pipeline consists of four main components, including annotation, filtering. template generation, and LLM evaluation. The annotation component starts retrieving object categories from 3D object datasets, the categories which will be filtered for irrelevancy, redundancy, and ambiguity, and matched with WordNet Synset to remove overlapping categories. We then oblain the object-attribus templates after combination with the physical attributes, and conduct the crowdsourcing process. Each object-attribute sample has a minimum of four overlapping amnotations, the agreement between which used to filler the annotations and form the NEWTON repository of object-attribute pairs. The template generation step begins with a generic template, which will filled through condition specification and object sampling. With the generated questions, we form a benchmark 160K questions. The pipeline also enables formulation of infinite personalized evaluation prompts to suit any intended scenario.
tion of the attribute of focus, and prompts the an- notator to select an option which best describes the object, given a particular attribute topic. The an- notators are asked to focus on the textual category. however, a visualization of 10 randomly-sampled thumbnails are included for reference. We provide the suite of prompts used for different attributes and potential options in Table 1. For each task, mini- mum of four annotations are collected, resulting in a total of 20 000+ annotations.
3.2 Filtering
The selection of confident object-attribute pairs is contingent upon the level of agreement between annotators. To ensure reliability, a minimum of four annotations that overlap mandated for each aatritt pair, enabling the calculation of inter-annotator agreement. Inter-annotator agree- ment calculated as a percentage of annotators who agreed on the majority response of each object- attribute task. A stringent filtering threshold of 0.75 is applicd, necessitating at least three out of the four annotations to exhibit agreement. Additionally, an- notations with extreme likert scores (either 1 or 3) preserved, thereby climinating ambiguous responses and maintaining clarity in the dataset.
3 NEWTON Pipeline
Leveraging the object-attribute repository, we in- troduce a pipeline which enables systematic gen- eration of diverse questions based pre-defined personalized templates, as shown in Figure 2. The process starts with crowd-soureing human anno- tations on object attributes, which is then filtered based on inter-annotator agreement. The filtered data is then used to populate template-based ques- tions in a systematic manner, a process which is extensible to a diverse range of personalized sce- narios, beyond those established in the paper.
3.3 Template Generation
Leveraging the NEWTON repository of object-attribute pairs, pre-defined templates can be filled. Each template is defined with associated object slots and attribute condi- tions. The condition statements are defined as (attribule, poltributepplaarity.. defines the physics attribute used to group, polarity defines whether to employ the highest or lowest extremity, and padding specifies how many objects of the opposite polarity to use. Each condition statement is designed obtain a group of n objects, consisting of one object of the specified attribute and polarity, and n - 1 objects of the negative polarity. Using the condition
where attribute
3.1 Annotation
Detailed analysis of the annotation interface and workflow is described in Section 2.2 and 2.3. A minimum of four annotators contributed overlap- ping annotations for each pair. Over 20 000 anno- tations were collected, featuring 700+ objects and 8 unique physical reasoning attributes.

---

Treplaa:
dational Attribute Comprehension, Explicit Ap- plication, and Implicit Scenario-Based Analysis. These tracks are selected to align with facets within Bloom's 's cognitive taxonomy, including compre- hension, application, and analysis (Adams, 2015).
I1 eneil beise cid
clyed more dicegle live
Whate coeeituu of cMASRO
while
sndior cul cacp scablid
restroil conerreeeien compp
Fhec appled
vhjon deo on vctiched cally
erger leve day rand w
The underlying task involves multiple choice ques- tion answering, where given a query q and up to four possible candidate choices, c1..44 the language model must select the correct option, Ceorrect of which there is exactly one for any given query. Foundational Attribute Comprehension. The first step to understanding concepts in an object- centric manner is make the connection between objects and their attributes. This one-dimensional reasoning is the core assessment strategy within the Foundational Attribute Comprehension track, where cach question involves understanding a sin-
Tookk
vhio caato ciily with tapaciiess
cigat las
11,k uijo do docopoole giooe.
obja ca cocl viasedrar nai Cas 10 vople
-h saloud brarl
bece das oo
desi las cerieess Orlar oolestio otiee by otbor to colccce
cnlyee to ank dhand
Hape
3
gle object-attribute pair. This track serves as a means to gauge the disparities in the distribution of comprehension between humans and language models regarding object attributes. Questions in this track closely mirror those presented to hu- man annotators, as shown in Table 1, with mini- mal adjustments made to accommodate the diverse prompting formats required for different models. Explicit Application. To be able to apply knowl- edge of understanding attributes in reasoning-type tasks is crucial for language-model integration in downstream tasks. Hence, the explicit application task aims to evaluate the language model's capacity to effectively apply their understanding of object at- tributes in explicit reasoning tasks. Through a com- bination of Boolean and multiple choice questions, language models must reason about the correctness of statements concerning individual objects pairs of objects, shown in Table 1.
ontion and movemattit
culpect colypa colk call specc
bald A
los voapling fruit ait you peaso thon true impec.
S dudl dight
silivo sight
Ocrdid Wiick of coppon drieen catia iy- co stalll dioecc
caps
Oreaeo frved porper
Conate thare uow whick was deick bocing dicaso Eis.
1r Jenbert foltems Higs Tleeiiy
Table 1. Question Templates. Styles of questions across the three tracks in the NEWTON benchmark. There are three tracks: foundational attribute comprehension, explicit applica- tion, implicit ecemario-bused analysis. Track I: Foundational attribute comprehension consist of identical questions to those used in the human-annotation process. Track Explicit appli- cation consist of questions where the object attribute men- tioned explicitly in the query, which formatted Boolcan or Multiple Choice style. Track 3: Implicit Scenario-based Analysis consists of implicit questions, where the attribute(s) of focus is not explicitly mentioned.
statements as a filter, we can obtain groups of object-atrribute pairs which satisfy the given conditions and fill in the object slots, the process of which is shown in Figure 2.
Implicit Scenario-based Analysis. This track as- sesses the language model's aptitude for reasoning in scenario-based tasks where the attribute to be inferred is not explicitly mentioned. Each prompt presented within this track has two components: context and question. The context serves as de- scription of the scenario, and implicitly highlights the attributes(s) of focus. The question presents candidate objects, one of which is the correct an- swer. We show example templates in Table 1.
4 NEWTON Benchmark
We introduce the NEWTON benchmark, a tool to assess the cognitive ability of language models to understand and reason about physical attributes of everyday objects. The NEWTON benchmark comprises of three progressively challenging tracks, and have a combined 160k questions covering 700+ objects and 8 unique attribute categories.
4.2 Statistics
4.1 Tasks
In total, NEWTON benchmark consists of 160K questions distributed over the three tasks of Foun- dational Autibute Understanding, Explicit Appli-
NEWTON benchmark has 160k questions dis- tributed over three reasoning tracks, namely Foun-

---

cation, and Implicit Scenario-Based Analysis. We provide a visualization of the distribution and anal- ysis of the data in the Appendix. In comparison
Cenkod cnd AlHibad)
Specitattion
COSTEST sesue in Mou
Dinettee tutern
Torto Ger
CONTECT to bel putal saltoe
TOMA Boci SH12 SOIC SUICE DIOLO
to other datasets and benchmarks which have ex- amined the topic of Physical Reasoning, NEW- TON differs in its object-centric focus, providing unmatched diversity and scale as shown in Table 2.
Acte-ppputs Templena
une pariien sere sere setente crte ace te aeneteer eTer, MIAA SEREMIIT, MOTHEM
parling pard akgere inte on depereen stora, shick INOIE escmaal
BENORLL 20 SEREEEaCE stold
toatigges sotee por the
Factors
PIQA
PROST
NEWTON (ours)
Object-Atrribtee Attribute
soa paccaa aostiand strecta cade aic st DNpACtANE JESEN, KSUCK MUDICL ENEEONE, EAAL.
Physies Attributes
N/A
8
Objects
N/A
20
600+
Evaluate!
Question Style
2 answers
answers
2-4 answers
Questions
16k
19k
160k
Figure 3. Synthesis of additional challenge sets. Using NEWTON's broad object-atrribute combinations, users can design custom challenge sets for their needs. The process starts with Context and Attribute Specification. Objects are then autonatically filtered based on chosen attributes into positive and negative sets. With these, users can quickly create tailored challenge templates. This helps evaluate language models in user-defined contexts, optimizing prompt strategies and model selection.
Multi-lovel Evaluation
Table 2. Dataset Comparison. Comparison of NEWTON with two other henchmarks aimed at Physics Understand- ing and Reasoning. PIQA (Bisk et al., 2020) is a dataset aimed at physics common-sense reasoning, without a focus object-centrin attribute understanding. PROST (Aroca- Ouellette et al., 2021) tackles physical and affordance reason- ing from an object-centric approach, using small subset of objects, Our dataset examines the understanding of language models from a physical reasoning perspective, with rich and diverse set of objects, attributes, and questions.
Model
Parumeters
Foundation
Flaz Meee-Alpaa-GPPP.....
3
T5
Flan, GPT4-Alpaca
Delly-v2
6.9 B
Pytiia
datbbeicks dolly-15k
Flan-55-Smll
50 M
TS
Collection
4.3 Synthesis of Additional Challenge Sets
Flas T3 Base
250 M
TS
Flan Collection
Ten-TT-lammp
750 M
T5
Flam Collection
The NEWTON Repository of Object-Attribute pairs can be used to synthesize additional chal- lenge sets, as seen in Figure 3. The process begins with Context and Attribute Specification. where users identify a context, and relevant attributes. Next, object filtering involves using the identified attribute(s) to automatically filter objects into a positive set and a negative set. Using the grouped objects, one can specify query templates, and au- tomatically populate the templates to synthesize a diverse and customized challenge set. Using this, language models can be evaluated for accuracy in the specific user-identified context to find the op- timal prompting strategy and model. Examples of the generated dataset are shown in the Appendix.
T--x-T-XLL
3
Collection
Alpaca
7:30 B
LLaMA
Alpace Date
BERT-SWAG
110.1 336.2
BERT
SWAG Datascl
GPT-Tubo
GPT33
GPT-4
Table 3. Model Details. We provide the details of models evaluated using NEWTON, including the name of the model, number of parameters, the underlying foundation model, and the instructed or QA datasets used to fine-tune the foundation model. Note that dashes represent undisclosed details.
tions of different models. Aside from the particular formatting requirements necessary for inference by different models, we make no changes to the for- mat of the prompt. Prompt structures for different families of models are illustrated in the Appendix. Models. We provide an outline of the various mod- els sench-marker with NEWTON in Table 3. We consider several families of large-scale pre-trained models,
5 Results
on different instructed or ques- tion answering datasets.
In this section, we evaluate the performance of state-of-the-ar models on NEWTON. Specifically, we quantitatively analyze the performance of the models on the tasks of foundational attribute com- prehension, explicit application, and implicit anal- ysis. We also qualitatively examine the patterns in errors made by the models.
Metrics. Two metrics are used in the evaluation of LLMs' performance across the three benchmark tracks: Agreement (%) and Accuracy (%). Agree- ment (%) is used to evaluate Track 1, while accu- racy is used to evaluate Track 2 and 3.
Agreement
Accuracy
Reul
(1)
5.1 Experimental Design
Where RLM denotes the response from the lan- guage model, Rнм denotes the majority human response, and Ry denotes the human response. The
Query Templates. We make minimal changes to the prompts of language models between evalua-

---

agreement metric regards the human agreement per- centage as an upper limit and adjusts the calculated accuracy based on this percentage. This adjustment results in a higher emphasis on questions with sub- stantial human agreement, while assigning compar- atively less significance to questions where human annotator responses exhibit greater diversity. This metric aids in evaluating the extent to which lan- guage model responses align with those of humans. On the other hand, the accuracy metric considers the human-majjan monmaty response as the definitive label and thus has a maximum attainable value of 100%. This metric gauges the proportion of responses that align with the majority-voted human response.
its dominance in attributes such as surface hard- ness, softness, malleability, and sharpness. Alpaca- LoRA-7B (Hu et al.. 2021) also excels in boolean questions, particularly in the areas of stiffness, surface smoothness, and brittleness. Addition- ally, other models like Flan-Alpace Alpuca-GPT4-XL Flan-T3-XL (Chung et al., 2022), and GPT-3.5- Turbo demonstrate competitive results in various attributes.
Some language models are capable of decision- making in implicit scenario-basee tasks. In Table 6, we present quantitative evaluation of language models across nine scenario templates. The results show that GPT-4 (OpenAI, 2023) consistently out- performs other models by a significant margin in the majority of scenarios. It achieved an impres- sive overall average of 87.7% across the defined scenarios, compared to the average of 44.5% for all the evaluated models. However, it is important to note that GPT-4 struggles to provide consistently accurate responses in certain scenarios, particularly in Scenario 8. This indicates the need for evalu- ation schemes that carefully consider the specific deployment situations in which language models will be utilized.
5.2
NEWTON Benchmark as Diagnostic for Knowledge on Physical Attributes
The threc-track setup of NEWTON benchmark enables analysis of Language Models' ability to comprehend. apply, and analyze physics attributes of everyday objects. Through quantitative analy- sis, we draw several insights.
Language models have non-uniform compre- hension of physics concepts of objects. The foun- dational attribute understanding task requires clas- sification of objects based on physical attributes. Through the task, as shown in Table 4, we find that there an inconsistent performance of dif- ferent models across the physical attributes en- compassed by NEWTON. While GPT-4 (OpenAI, 2023) dominates in terms of overall performance, different models excel at different attributes. For instance, Flan-T5-Bass (Chung et al., 2022) had strong performance across elasticity, stiffness, sur- face smoothness, and malleability, while Flan-T5- XL. (Chung et al., 2022) excelled at softness. Qual- itatively analyzing the common responses of lan- guage models, we find that typical errors occur due to one of three reasons: hallucination, con- servatism, and misunderstanding. We illustrate examples of model outputs in the Appendix.
5.3 Ablative Studies
In this section, we provide an analysis of NEW- TON, focusing on potential ways of leveraging NEWTON to enhance model performance in a physical reasoming context, and examining the con- sistency of LLMs with regard to model size, ques- tion polarity, and answer positioning.
Fine-tuning using NEWTON. Aside from using NEWTON to create evaluation tools. it serves as a resource for fine-tuning pre-trained LMs, improv- ing their grasp of physical concepts. We experi- ment with fine-tuning on Track 2 questions that explicitly address object attributes, then evaluate on Track involving implicit attribute reasoning. Track 2 and Track questions, detailed in Table I, are distinct. Fine-tuning focuses on multiple- choice tasks using the base BERT model (Devlin et al., 2019), initially trained on SWAG (Zellers et al., 2018). We fine-tune with subsets of NEW- TON's Track 2 5k, 10k, and 40k samples. Mod- els are assessed on Track 3's implicit reasoning using NEWTON. Figure 4A reveals significant en- hancement in language models (e.g., BERT) when NEWTON is part of pre- pre-training training, with increas- ing performance as fine-tuning samples rise. This
Language models are capable of applying their knowledge of objects to comparison-basee based tasks. By utilizing the templates highlighted in Table I, we observe that language models exhibit pro- ficiency in boolean and multiplc-choice reasoning tasks for physical reasoning of object attributes, similar to the observations highlighted in Kada- vath al. (2022). Notably, GPT-4 (OpenAI, 2023) consistently performs well across both boolean and multiple-choice style questions, showcasing

---

Language Model
Agrvement (%)
Elasucity
Stiffeess
Surface Smouthaess
Surface Hardness
Sofiness
Britleness
Malleability
Sharpness
Overall
Dolly-V2-7B
23.7
11.7
4.1
3.5
1.3
8.6
2.5
7.6
Flan- Man--Apaca-GPT--N
9.3
1.1
59.7
19.9
25.1
6.1
9.5
7.4
15.2
TS-smll
79.8
4599
61.9
28.1
25.4
6.1
77.0
4.7
41.7
Flan- TarLaage
Flan-TS-XI.
9.3
4.4
15.6
42.4
59.9
6.1
99
54.6
27.5
SolfeddAA-V--s--aam
61.9
95
Alpuca-LoRa-T7H
9.3
0.0
44.3
0.6
14.4
3,9
9.5
1,
10.0
GPT-Turbu
22.5
0.0
0.7
13.5
6.6
6.2
53.8
1.1
16.4
GPT-4
51.3
8.0
14.1
47.1
59.5
76.5
44.1
58.4
49.7
Average
20.3
7.2
32.4
21.6
22.8
11.8
23.7
13.8
19.6
78.7
Table 4. Track 1: Foundational attribute comprehension results for various language models. We report the agreement percentage, computed percentage of responses which agree with the majority voted human response, weighted by the inter-annotator agreement. We also provide the overall averaged agreement across language models, and also across attributes. In addition, report the inter-annotate agreement average for the listed attributes for reference.
underscores NEWTON's potential for improving LLMs' physical reasoning through fine-tuning.
questions according the position of the ground truth answer, as shown in Figure 4C. Among the models, UnificdQA demonstrates the lowest differ- ence in the accuracy between highest and lowest scoring accuracy positions, as quantified by t- statistic of 15.0 and p < 0.0001. On the other hand, Flan-Alpaca-GPTe GPT4-XL exhibits the largest difference between the highest and lowest position accuracy, quantified by a I-statistic of 170.1 and p 0.0001. The observation that models exhibit inconsistencies in accuracy by position is also high- lighted in (Aroca-Ouellette et al., 2021).
Language Model Size and Impact on Perfor- mance. To assess the influence of model size on performance, we focus on the Flan-T5 series and analyze the performance of small, base, large, and XL model sizes, as shown in Table We observe that, in general, larger model sizes lead to improved performance for Track 2 and 3. However, for Track I, the Flan-T5-Base model demonstrates the best performance. This inconsistency is likely attributed to the nature of the questions, as those in Track 1 are more descriptive in nature, while questions in Tracks 2 and 3 are more concise. The contrasting question styles likely account for the varying out- comes observed across different model sizes, as certain models may excel handling longer, more detailed queries while others excel in providing responses to shorter, more focused questions.
Effect of Prompt Engineering, We analyze the in- fluence of prompt engineering on both GPT-Turbo and GPT-4. Table 8 illustrates our exploration of five distinct approaches to formulating prompts. These strategics are denoted as VI, V2, and V3, representing three different versions of prompts distinguished by their word choices. The Ensem- ble method amalgamates the outcomes from VI, V2, and V3, utilizing majority voting system to determine the final answer. Meanwhile, the Instruc- tion strategy supplements each prompt with of instructions, comprising eight example questions and corresponding answers that cover diverse at- tributes. Notably, these examples do not overlap with any of the tested questions. It's noteworthy that the Ensemble and Instruction-based prompt techniques enhance performance for Boolean-style questions. However, the performance remains com- parable or, in some cases, even less favorable for Multiple Choice-style questions. These findings underscore the significance identifying an ap- propriate prompt and crafting variations that are attuned to diverse question styles.
Polarity and Position Bias in Language Mod- els. We explore the impact of question polarity on model performance using NEWTON benchmark Track 2. Prompting questions categorized into Positive and Negative polarity, where Positive po- larity questions include phrases like "is," "is more," and "is the most." while Negative polarity ques- tions involve phrases like "is not," "is less." and "is the least" (as outlined in Table 1). Conducting a t-test reveals that GPT-4 and Dolly-V2-7B have no significant difference between the means of the dif- ferent polarity groups, while for other models, there is statistically significant difference (p < 0.05) between the means of the different polarities, in- dicating the presence of a polarity bias, as shown in Figure 4B. Additionally, we investigate the pres- ence of bias based on answer position by grouping

---

Language Model
Elasticity
Stifiess
Surface Smoothness
Surface Hardness Booleun (Accuracy %)
Softness
Brittlnness
Mallesbslitt
Sharpness
Overall
Dolly-V2-7B
3.1
2.6
3.1
4.4
2.8
2.5
3.1
2.4
3.0
Flan- Man--Apaca-GPT--N
$4.8
61.5
49.8
99.5
58,
49.2
48.88
68 1
56.2
T5-mall
50.3
Flan-13 15-Base
36.8
35.3
34.8
34.6
30.8
40.8
37.1
26.3
34,5
Flan- TarLaage
Flan-TS-XL
61.4
6014
50.5
65.3
74.0
51.4
59.2
78.8
62.9
CaafiddQ-VV--S--..a
51.7
49.8
Alpuca-LoRa-
61.0
6822
67.9
63.9
62.3
69.5
62.7
66.8
65.2
GPT-Turbu
49.5
58.5
51.6
54.4
67.5
51.7
61.5
81.5
59.8
GPT-4
66.6
61.7
60.8
66.8
78.0
66.8
71.5
82.3
69.6
Average
47.6
49.0
46.1
47.6
33.6
46.1
48.5
36.8
49.5
Multiple Choice (Accuracy
Dolly-V2-7B
14.4
9.8
11.7
13.2
12.6
11.7
11.4
10.6
11.9
PannAAppca-GPTA-N.
78.3
Plan-T--Sma
23.6
34.8
25.1
23.9
23.7
24.0
23.9
24.4
24.2
15-Base
39.9
369
23,2
42.0
32.1
57.1
Flan-TS-Lam TS-Large
61.0
56.9
23.0
72.4
64.6
32,8
38.3
90.4
54,9
Flan- T3-XL.
70.5
77.9
42.5
82.5
77.6
30,0
54.7
92.9
66.1
UnnfaedOA-V2-TS-L.a
61.7
59.8
28.8
71.3
63.7
27.4
50.3
75.1
54.8
Alpace-LolRa-78
26.2
27.8
24.1
26.6
26.3
24.3
24.6
31.2
26.4
GPT-Turbo
GPT-4
75.4
73.8
64.6
84,8
91.7
67.8
72.8
98.5
78,7
Average
Combined (Accuracy %)
Dolly-V2-7B
8.3
6.1
7.3
3.8
7.3
6.9
6.9
6.2
7.2
64.9
70.4
43.7
69.9
67.4
36.5
54.2
79.1
60.8
Flan- TS-Small
39.6
37.8
37.1
36.7
38.6
36.7
39.2
41.1
38.4
TS-Bae
29.1
35.1
Flan- LS-Lage
$1.0
48.4
33.6
46.4
56.8
31.2
38.6
69.8
47.1
TS-XL
SonfiddQA-V2-T---aa
53.6
55.6
39.6
64.3
62.3
39.5
50.0
64.1
53.77
Alpaca-LoRR--R
46.5
45.7
GPT-Turbo
49.3
63.7
47.3
67.7
70.3
48.4
53.0
86.7
60.9
GPT-4
70.7
67.6
62,7
75.7
84.3
67.2
72.1
89.8
73.9
48.6
50.3
39.4
52.7
54.4
38.4
45.1
61.3
48.8
Table S. Track 2: Explicit application evaluation results on various LLMs. We separate the questions into two streams, Boolean, which consists of True/False style questions, and Multiple Choice, which consist of QA style questions with four answer choices. We report the model accuracy across each stream, as well as the combined accuracy. For each stream, we report an averaged accuracy percentage across all models for each physical reasoning attribute. We also report averaged accuracy percentage across all attributes, gauge the overall understanding of language models across all physical reasoning attributes.
Semariin
Parameters
Track
Track
Track
Language Modd
3
S
Usail
Flon-TT-Small 80 M
0.0
38.4
29.8
GPT4-XL
Flan-TS-Base
250 M
E5-mmall
27.8
18.9
33.
312
268
31.1
2935
LlngrS-LLage
780
13.7
47.1
46.2
Uba- Lans
Flen T5 Bac
06.3
969
37.3
47.
189
350
2611
223
335
16.
761
367
218
189
315
441
214
302
34.2
482
Han-TT-XL.
64.4
56.9
21.
738
311
56.
06.
33.
852
35A
Table 7. Model Impact on Performance. provide comparison of different sizes the Flan-T5 models evaluated using the different tracks the NEWTON benchmark.
GPE-Tudo
GPS4
751
942
837
Anerage
318
36.
354
255
224
343
310
2510
391
443
Table Implicit scenario-based analysis. We present racies of language models the nine scenario-based tasks from the NEWTON benchmark. Scenario range of scenario based questions, ranging from arrangement, tool-use, safety. provide average across each nario measurement the overall scenario complexity, as well as an average encompassing the overall performance model across nine given scenarios.
guage, works like Hong al. (2021); Duan et al. (2021) have investigated physical reasoning in vi- sual contexts, and datasets like (Clark et al., 2018; Kembhavi et al., 2017: Zellers et al., 2018 mainly assessed physical reasoning through physics ques- tions for benchmarking pretrained models. Some studies have assessed model proficiency in the con- text of physical reasoning: PIQA (Bisk et al., 2020) models physical commonsense, while PROST (Aroca-Ouellette al., 2021) offers ques- tions on physical reasoning concepts. contrast, our work (NEWTON) introduces a framework for evaluating and improving large language models'
represents
6 Related Works
Evaluation of Reasoning Abilities. In the past year, significant progress has been made in explor- ing Natural Language Processing (NLP) develop- ments attributable the advent large language models. Prior studies like Bakhtin et al. (2019) focused on basic physical reasoning without lan-

---

Avarage Perforaance on Firetuning with NEWTON
Accunsiy by Polerty
Accunscy by Paaton
Averaga Paate (13
80
73.4%
73.45
68-
Methed
Figure 4. Ablations. From left to right: A) BERT tunint results using NEWTON, note the increase in accuracy on the unseen implicit questions after finetuning on NEWTON, using of sample of 5000, 10000, and 40000, respectively. B) Accuracy by question polarity, where positive polarity represents questions phrased with is, is more, and is the most, while negative polarity represents questions phrased with is not, is less, and is the least. C) Accuracy by position, where the position value indicates the placement of the correct answer within the sequence of possible options in the question template.
11M Poorpiling Method
55
SH
So
B
M
SI
Ovrill
Most of these were in embodied contexts, where agents answered questions by exploring environ- ments (Das et al., 2017; Gordon et al., 2018), per- formed tasks with sub-steps (Shridhar et al., 2020), or rearranged scenes given instructions (Szot et al., 2022). Unlike these tasks which focus on visual and semantic attributes. NEWTON challenges lan- guage models with questions about explici//mmlliit physical object properties, some beyond visual in- ference. More recently, LLMs are being explored for grounding robotic manipulation, often gener- ating code or instructions for actions/skills (Liang et al., 2022; Ahn et al., 2022). Yet, there's 's gap in using LLMs for object-centric physical reason- ing in manipulation, which stems from limited ev- idence of LLMs' grounding abilities. Our work focuses on this by exploring and evaluating such abilities in physically grounded settings.
Beelean (Areurary 5)
CPI-Tu
GPI-Turto Instrcttoon
63.0
08.
56.5
57.5
45.
642
GPI-4+31
68.5
67.5
66.5
65.1
63.5
63.5
7h0
16.
71.
GP
65.0
80.0
35,0
72.9
71.
83.5
1L5
09.
21.5
GPT4+
15.5
71.5
61.0
88.5
7310
38.0
1200
K3.5
76.6
GPI4 Drnemblle
61.5
83.5
75.
Intraction
15.5
900
68.5
7300
908
Mittiple
GPI-Tirbo
46.5
77.5
45.5
$2.5
71
41.5
4::
90.5
678
GPI-Tucb V2
25.5
56.0
35.5
52.5
35.0
66.5
39.5
61.5
46.2
GPI-Ticto V3
54.5
55.0
31.5
89.5
53.5
57.0
3500
19.5
52.9
GPI-Turtu Exoemble
31
67.0
36.5
08.0
340
45.9
GPI-Turler Intraction
3033
53.5
340
63.5
49.3
52.0
25:0
44.7
GPT4+ VI
76.5
500
87.5
91.5
05.0
70.0
792
GPI-4 V2
06.1
17.4
85.1
73.5
12.5
06.4
15.5
08.0
40.5
86.3
75.0
78.5
10.5
76.3
21.0
GPT4 Brocablie
65.5
08.5
47.0
84.5
81.
77.0
1L.
87.5
72.9
GPT44 Indraction
63.5
57.5
5401
71.9
6511
81.
31.01
75.0
562
Currhtaed (Acearacy
4))
GPI-Teto VI
14.5
47.5
45.0
61.6
GPI-Twto V2
63.7
50.3
50.3
53.3
56.5
53.1
GPi-Taclo V3
542
54.1
45.2
65.0
35.2
59.0
460
18.2
33.4
GPi-T Enccable
458
61.5
45.
63.3
51.2
53.6
45.5
67.0
55.4
GPI-Tirbo hoomattinn
462
011
49.2
66.6
11.5
45.5
45.5
75.0
544
G114-
68:2
613
720
25.4
73.3
785
738
Ersestble
39.
Table 8. Prompt Engincering. We tested five prompting strategies on GPT-Turbo and GPT-4: three variations (V1, V2, V3) by word choice, an Ensemble majority vote from V1-V3, and an Instruction method with eight sample Q&As. Questions were split into Boolean (True/False) and Multiple Choice types. We present accuracy for each stream and overall, as well as average accuracy per physical reasoning attribute. E. St. SS. SH. So, B. M. Sh, represents Elasticity, Stiffness, Surface Smoothness, Surface Hardness, Softness, Brittleness, Malleability, and Sharpness, respectively.
Conclusion
In this work, we present NEWTON, a repository, pipeline, and benchmark to support the assess- ment and refinement of LLMSs in the context of physical attribute understanding of everyday ob- jects. The NEWTON repository offers large collection of object-attribute pairs, enabling the generation of infinite test scenarios for evaluating language models, using the NEWTON pipeline. To demonstrate its usefulness and potential, we in- troduce the NEWTON benchmark, which consists of 160K questions involving 700+ objects and 8 at- tributes. These questions cover three distinct tracks: fundamental object understanding, explicit applica- tion, and implicit analysis. NEWTON is built enable comprehensive and automatic evaluation of language models across an array of scenarios, and support the reliable incorporation of LLMs into physically grounded contexts and applications.
performance in physical reasoning. With 2800 object-attribute pairs and 160K QA prompts, it pro- vides unmatched scale and extensibility, potentially setting a new standard for evaluation.
Applications of Language Models in Physically Grounded Contexts. With the rising popularity of pre-trained LLMs, it's natural to consider their role in physically grounded scenarios like robotics. Before the L.M surge, efforts were made to link vision and language (Das et al., 2017; Gordon et al., 2018; Shridhar et al., 2020; Duan et al., 2020).

---

Limitations
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439.
While this paper sheds light on language models' physics reasoning abilitiiss it's crucial to note its limitations. Firstly, data collection through crowd- sourcing introduces potential human errors, despite extensive filtering efforts by the authors. Secondly, the dataset's categories don't encompass the full range of real-world objects, limiting evaluation scenarios to those described. Thirdly, the paper doesn't it explore different language models' suitabil- ity for various prompting strategies. Nonetheless, NEWTON could potentially be adapted for such evaluations in the realm of physical reasoning. De- spite these limitations, we hope NEWTON encour- ages further research in benchmarking physical reasoning and enriching language models' under- standing in a physically grounded context.
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. 201 S. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 in- termational conference on advanced robotics (ICAR). pages 510-517. IEEE.
Susan Carey, 2000. The origin of concepts. Journal of Cognition and Development, 1(1):37-41.
Michael Chen, Mike D' Arey, Alisa Liu, Jared Fernan- dez, and Doug Downey. 2019. Codah: An adversar- ially authored question-answer dataset for common sense, arXiv preprint arXiv: 1904,04365.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi. Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-fintture language mod- els.
Acknowledgements
The authors would like to thank members of the Personal Robotics Lab (PRL), Robotics and State Estimation Lab (RSELab), and UW NLP Group for fruitful discussions and insightful feed- back on the manuscript. Yi Ru Wang is sup- ported by the Natural Sciences and Engineer- ing Research Council of Canada (NSERC). This work was (partially) funded by the National Science Foundation NRI (#2132848) and CHS (#2007011), 1), DARPA RACER WHHR0011-21-C 0171), the Office of Naval Research (#N00014-17- 1-2617-P00004, #2022-016-01 UW, and #1 140209- 405780), and Amazon.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.00 180..0557..
Jasmine Collins, Shubham Goel, Kenan Deng, Achlesh- war Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang. Tomas F Yago Vicente, Thomas Dideriksen, Himan- shu Arora, et al. 2022. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Patiern Recognition, pages 21 21126-211 1336.
References
Nancy E Adams. 2015. Bloom's taxonomy of cognitive learning objectives. Journal of the Medical Library Association: JMLA, 103(3):152.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2017. Embodied question answering.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Pinn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. 2022. Do as can, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt. Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi, 2023. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 13 13142-13153.
Stéphane Aroca-Ouellette Cory Paik, Alessandro Ron- cone, and Katharina Kann. 2021. Prost: Physical reasoning of objects through space and time.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.

---

Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas McHugh, and Vincent Vanhoucke. 2022. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553- 2560. IRER.
Douwe Kiela, Max Bartolo, Yixin Nie. Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vid- gen, Grusha Prasad, Amanpreet Singh, Pratik Ring- shia, Zhiyi Ma. Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynahench:
Rethinking benchmarking in nlp.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lchter. Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu. Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2022. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:220.007553.
Jiafei Duan, Samson Yu, Soujanya Poria, Bihan Wen, and Cheston Tan. 2021. Pip: Physical interaction prediction via mental imagery with span selection. arXiv preprint arxiv 2 009.04683, 1.
Michael McCloskey, Allyson Washburn, and Linda Felch. 1983. Intuitive physics: the straight-down be- lief and its origin. Journal of Experimental Psychol- ogy: Learning. Memory, and Cognition, 94))636.
Jiafei Duan, Samson Yu. Hui Li Tan, and Cheston Tan. 2020, Actionet: An interactive end-to-end platform for task-based data collection and augmentation in 3d environment. In 2020 IEEE International Confer- ence on Image Processing (ICIP), pages 1566-1570.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-4))
OpenAL 2023. Gpt-4 technical report.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behav- ioral testing of nip models with checklist.
IEEE.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari. Joseph Redmon, Dieter Fox, and Ali Farhadi. 2018. Iqa: Visual question answering in interactive environments. In Proceedings the IEEE conference on computer vision and pattern recogni- tion, pages 4089-4098.
Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading com- prehension. ACM Computing Surveys, 55(10):1-45
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A bench- mark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEFF/CVV con- ference on computer vision and pattern recognition. pages 10740-10749.
Yining Hong, Li Yi. Josh Tenenbaum, Antonio Torralba, and Chuang Gan. 2021. Ptr: A benchmark for part- based conceptual, relational, and physical reasoning. Advances in Neural Information Processing Systems, 34: 14447427-
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Andrew Szot, Karmesh Yadav, Alex Clegg, Vincent- Pierre Berges, Aaron Gokaslan, Angel Chang, Mano- lis Savva, Zsolt Kira, and Dhruv Batra. 2022. Habitat rearrangement challenge 2022. https: // aihabi wtwpbbittttrrggg/eeooo wwttr////caaallleeerrrrrrnnneeee 2022.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer. Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston. Sheer El-Showk, Andy Jones, Nelson Elhage. Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph. Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know.
Jimmy Wo, Rika Antonova, Adam Kan, Marion Lep- ert, Andy Zeng, Shuran Song, Jeannette Bohg. Szy- mon Rusinkiewicz, and Thomas Funkhouser. 2023. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference, arXiv preprint 1rXiv:1808.005
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong. Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li. Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie. and Ji-Rong Wen. 2023. A survey of large language models.
Aniruddha Kembhavi, Minjoon Sco, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannanch Ha- jishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattera recognition, pages 4999-5007,

---

A Appendix
inaccurate understanding, In Figure 9, we examine the common failure modes of language models for a set of 10 questions from Track 2: Explicit At- tribute Understanding. We find that for most mod- els, the failure mode occurs due to the tendency to hallucinate. For instance, in Figure 9, for question 6, 7. and 8, Dolly fails to provide answer which reflects the given options.
A.1 NEWTON Benchmark Statistics
NEWTON has three main tracks: fundamental attribute understanding, explicit application, and implicit scenario-based analysis. The three tracks comprise of total of 160K questions, covering 700+ objects and attributes. Each object anno- tated with 1-8 tags of object attributes. We present visualizations of the data-statistics for Track 1, 2. and 3 in Figure 5, 6, and 7, respectively. Figure 5 illustrates the data distribution of questions and attributes, total number of tokens per attribute, and number of attributes remaining after filtering for each object category. Figure 6 illustrates the data distribution for the explicit application track, and includes an analysis on the distribution of the top 100 highest occurring categories, percentage dis- tribution of different attribute occurrences within the track, counts of question polarity and type, and total tokens for the questions in each attribute. Fig- ure illustrates the data distribution for the implicit scenario-basee based analysis track, and includes a per- centage fraction of different scenarios, and the total number of tokens for questions in each scenario. Taking a closer look, we can see the initial set of attributes have a bias towards the clasticity. sharp- ness, softness, and malleability attributes, due to the process of filtering inconsistencies To ensure balanced dataset, template formation of Track 2 questions involved additional re-sampling step ensure the questions cover the attributes in a uni- form way, hence why Track 2 questions are more uniform in nature. Track scenarios are manually designed for each scenario, and hence data balance was not a primary focus.
A.4 Query Templates for Language Models
Different models require different variations of prompt imput, and the unsuitable prompt could lead to an inaccurate assessment of the capability of language models. Hence, we adapt the question template for different families of models to adhere to the most effective prompting strategy, as shown in Figure 10.
A.5
Synthesizing Additional Challenge Sets with NEWTON
We show a snapshot of the generated dataset using the procedure for synthesizing additional challenge sets in Figure 11. Note that all objects also have corresponding 3D models and 2D thumbnails, as shown in Figure 12 should the user wish to extend the dataset to a vision-language setting.
A.6 Comparing Agreement and Accuracy Scores
In Table 10, we show the performance of various models when evaluated using the accuracy mea- sure. In comparison to Table 4, which provides the quantitative results using the agreement measure, there is an overall increase in the absolute percent- ages, since the upper-threshold for the maximum attainable value has increased from human agree- ment percentage to 100%. However, we note that the conclusions mentioned in Section 5.2 remain the same.
A.2 Detailed Results for NEWTON Benchmark Track 2
Track involves mixture of question types (Boolean, multiple choice), and questions which differ by polarity (positive, negative). We provide a detailed breakdown of results by polarity for mod- els evaluated on the NEWTON Benchmark, Track 2, in Table 9.
A.7 Dataset Examples
In this section, we provide a snapshot of the dataset, as shown in Figure 12. Each generated question is tagged with corresponding choices, ground truth response, question polarity, and relevant 2D RGB thumbnails and 3D object models. While NEW- TON is designed for prompting and assessing lan- guage models, potential extensions could also lever- age the paired RGB and 3D models to assess visu- ally grounded models.
A.3 Qualitative Results
We present example for model outputs for the Foundational Attribute Understanding (Track 1) in Figure 8. Note the common mistakes which language models make include (1) a tendency to choose the middle option, (2) hallucination, and (3)

---

Conriction of Drfinent AbrbzAe
Top & Bottom 10 Categories
faucet
corkscrew
helmell_helmet
bunk_bes bed
spectacles
spear
Object Categories
yolk
strawberry
newspaper
pencil box
shoulder_bag
notebook
bamboo
organizer
parasol
parchment
pea
adhesive tape
Count
Figure NEWTON Benchmark Track data statistics. We highlight the data distribution of questions and attributes, total number tokens per altribute, and number of attributes remaining after filtering for each object category.
the quedions la each attibuute
Derkety
Weleebiity
000000
40000
2004
29000
Surfice tartees
Atriuzes
Figure 6. NEWTON Benchmark Track data statisties. We examine the data distribution for the explicit application track, and illustrate the percentage distribution of different attribute occurrences within the track, counts of question polarity and type, and total tokens for the questions in each attribute.

---

Counts of Different Scenarios
tokens the questions scenarios
plastic jaaakicpackging
baltcase_packing
100000
Total number TDoeeen
11.9%
80000
fragile_gffts
Bockkackkpackkngg
11.9%
11.9%
60000
sandpaper_plactee _pcacinga
11.9%
20000
kids_pia hids_paay_area para
11.9%
hammering
backpack jaacking
Ragile Dets
lanmerrin
Insolation
platiic packagong
rebot tunting
candpaper placing
sulicase, Backing
play
robsi_tossing
insulation
Scenarios
Figure? 7. NEWTON Benchmark Track 3 data statistics. Graphs show the percentage fraction of different scenarios, and the total number of tokens for questions in each scenario.
Dell-V2-7B
11/33/26
26/29/12
2.1366/26
44/S2/36
18/29/26
25/25/24
3.1/32/29
24/28/2.0
3./33//26
Fhn-Alpps-GPP4A0
SL8/57.7/510
615773A444 4836
038/5167480
555761//5L
SL./5433/50
492/43..554 54.4
485/459/515 51.5
68.1/230/9922
56.25588/55 53,7
Flan-T5-Smm Small
5../55.//5.9
485/520/45.
43:55,21469
51.4/8844/44 54.4
413/436/530
52.4748.1 044748611560 56,7
55.4:523/586
$1.2/501//53 52.3
Flan-T5 Base
565/43007 306
343/376/1 6335
345:3001 21.3
01//32.77794
438/312/-144
37.17 34.17 39.8
26.33002222
54.5/4477
34.3
P5an-T51 arr
1231/221/454
801 /30.578000
478:-39/453
106/5631/349
502/51-17470
28/222/33
183-7 59 36-7
53.15733/00
(117-488700 79.1
Flan-TS-XL
614/065/55 568
604/5411 1858
30.55399/301
653/51A/7922
76.0/78.111 199
S/A/5L7/8L2
20.167.5:500 50.
71.3/792/784
62.9688./6
UalleOAA-1221 Large
4651472/455
51.7152.755 190.
438/518//75
51A1558/550 58.0
61.1/622::00
5366 53614877/25
217144421553 5552
54.166L8143
52.7/53.2/5
Alpera-Loke-e
T..//1.5/510
65.2/95.1100 1098
07.1 781178111 1551
63.7059 102.0
(2.3/19::55
035/069/733
02.71 737/5131
66.8/422149
05.21
GPE//obe
055/413/553
515/625/514
5147555/552
075/6533667
5171159:554
015116775663
595/
665/6550
617/201/514
6806 6/10
063/744//511
760/3632413
08/05/000
71.53 3023/323
8335781 4/512
Muttple towralllustthssbeggtee
Dell-V2-78
164:1537/553
03/104
11:2/120/144
00/3/300//0 158
125/122/334
117/125/108
134/94/154
006/96/111
11:9
Flan Pin-Aljes-GPT4XM
166/552//000
372/275/463
8051 00511921887 80.7
78.3/455/310
226/308/144
00.41225558 58,0
918/9866/83 /85.0
05.01
68.26666668 63.5
Flan-T5 Small
25./2133/300 200
213/295/300
138/363/211 21.
257/348/226
258/226/254
23.0 2457 13.0
24.4/466/12 1422
24.2/
31.6
Plan-TS- Hass
50//6.7/114 116
775
212/226/239
1300455 35.
014/5T4/314
148/176/171
30.17 30 75.0
571108777777
55.5
Plan-T5-1 arys
210 161/162/20
1144562//251
565//8977000
1181488/167
T011 12.7 41.5
454/9460/90
56.91
Plan-75-A1
715/81.2/500 253
425/39//60 2543
775/
212/212/228
567/574/5188
16.1
ChlledOAA Large
01.77 (8777887483 43.5
505/552//633
238/250/31.
3133/633/233
63.71 (37//066155
214/26/252
503/455/600
75.1 105
56.11561
Alpara- La
5.2/27.07
274/266/220
366 75667566110
263/269/25
263/252/229
366/318/3744
31333855/299
36.3
000//5144477
81122112/011
T55/368/993
467/000074
453/323//660
02.0
754/66.3/304
715/363/213
VL7/918/214
078/2511/014
728/088/768
48./960/991
Dell-V2-7B
83/81/88
61/65/56
7:3/921699
98/79192
15/24/933
69:1.3764
69/61/088
62/5875A
122/31/73
Pha-Aippa-GPP4NL
6.9:60578003
104/35/6653
43.7/398/473
659/340//63 65.9
674/58445565
363/33.7/333
561/558/568
7611/3007LL
618:6833/583
Flan-T5 Snaall
36/4113/76
171/385/361
167/38/313
3733313111
41.1/441/30
30 373
Plan-T5-Han Hase
T&T 141/41177
NN13112/30
1631:22//74 17A
164/279/540
1437 16.7 32.5
408/596//455
6617416/364
Flan-T5-Larp ary
510/97.374.5
316/316/76.
4641426 911
1500
312/311/235
3800 41.14111
55:5/246/011
Flan-T5-XL.
K587 15.87
66/7557756 56.2
500/4555/228
1861432/483
139/9:3/754 76.
412/448/357
52.1/52/5122
85.3/800//20
CalledgA-12 L2-Tg--
$3.81 65.2/45.0
556/540/575
20
64375220 65.6
62.1/8.1132 155.6
30.33308/355
5LD7 425/555
/41/7226/55
33.7155997
Alpara- 78
0:5/51 12.0
4147454744
15.7/007 241.
1587 4957
90.3/01//
463/
GPP
613/6801/93
63.77883/00 (0.55
$6.7780111
616/7311/22
627/005/668
35.0/810/699
863/626/960
672//68/661
12.1/805/662
$9.8/806/99
Table 9. Track 2: Explicit application evaluation results on various language models. We separate the questions into two streams. Boolean, which consists of True/False style questions, and Multiple Choice, which consist of QA style questions with four answer choices. We report the model accuracy across each stream, as well as the combined accuracy. separated by the polarity of the question. For each stream, report averaged accuracy percentage across all models for each physical reasoning attribute. We also report averaged accuracy percentage across all attributes, to gauge the overall understanding of language models across all physical reasoning attributes.
Language
Agresment (%)
Elasscio
Snodaas
Stzaee Hudness
Maleokllie
Stapaos
VV-7B
26.0
5.5
4.7
GPT-AN.
75.5
25.8
0.0
363
LolledQA-
786
39.1
Alpars-LRRa-7B
11.2
36,4
08
G.T-Turbo
0.9
16.4
GPI4
57.8
133
35.5
Table 10. Accuracy Performance of Models on Foundational Attribute Comprehension. We report the accuracy percentage, computed as percentage of responses which agree with the majority voted human response. The upper threshold is 100%. which represents perfect overlap with the ground truth (majority voted human response).

---

Deky
agii Wnreereee
Ho
Modelle uale parkcc olgee
Figure& 8. Example of responses from NEWTON Track 1. Note that the query is marked in the yellow text hox, and the correct answer to choose is marked with green highlight. Responses from different LLMs are shown in blue text-boxes, and correct responses are marked with green check-mark.
Template 1:
sxplasation.
andy.?
"ccpk ien 15
Dunniions "cquuest Lin hexedf
"ctppt on "coptioon
(b) "clptiion
*cdptioon
*copsion
Model:: UniledAA, Flan-TS, Flan-Appca- V2
Answers
Template 3:
Inatrsation
deaceiben task, paiced with seput that
reaposen apecoeeaatlly complaten
aeR Inatrsations
Equeatian gons
<cype
3
[c] *coppios
0244
5010 semee c 2n sec to
[6] *cape los $5
Inpubs
24 2 Nac servee
Lofic
000000115 3M
Figure 10. Prompting Templates. We show examples of the queries used evaluating language models.
Behe
Leaer arser
cocroge dee c1d Bend ptacpeii
Me Salia
12 Pahee
CETOORE
803000 201
Figure 9. Success Failure Examples. Typical failure modes of various models a sample of NEWTON questions. Models typically fail due hallucination and inaccurate un- derstanding.

---

Contest
Qusstion
Gt
Chdes,1
Cholou_
Choioe
parthaedd into bocota
flwee should bottore
parthased sejecssiion
puslins zu sarchaeed odeeco bocala Desaetrent
Whitrel breed bauthul buil be
bolaart
breal
sanc
wiher pasking parthased oijects litu bocala depattrent stans
Which Iaskartaal cannewal, bouque, cardered thould al bottore
badaetall
Eastetbel
Sougat
anhoon pucking ththeeed iejecti intoe becolt cesutinent stote
ohich breebors whigeed orean, buth yut, humberger should put bottore
bith rut
Hodelly
web0od ore b4t nut
Pambergg
when packing parchased dejects into bocada repartivnnt ston..
Which of omea commeel bisebull Las bei shout be put bottors
bet
cealet
bisebel
bed
seheen pooking parthaeed slgects iatoe bocotta rejuttient stwe.
Whahh Manberry black, subble gane bur should the bettoer
Mark
1kabery
10ch
parchaved derects intor bocala ceseetrwent
Which dlics, siclls
buttoms
parthased lano becada
santissrr blnd,
parthased atjects
grnen
putking burchesee sarcheed oblects intou bocala deputiment
rutorssher carsution,
linesttde thoidd bottont
when parking parthased otjects into bocata department ston.
Which ballin, calinet, bouter, Jut, gaithe thouldbbe patae botton!
ciblnt
artern puthing par havdd slject sator bexutt deauttteettttnnnn cesuttient store
Whith Miberrs baller siit, conmmedl
Sloudd be
bettaer
Cat
Bockeery
calt
wheee pusking purchaved oepect into bocals cepaetmennt stor.
Which paaadds cresonnt, yoil banderna, chachebbe d should be pat bottons
Chederbsard
lampohede
Chedaa Dourc
seheen puoking gi parthased ongeen lito becatta resuetrvnt ctwe.
Whiihn gurlic chis, chocdlane, micttie bul bkarbeity thoudd out 2e bettur
Iihern chaaed indor bocals cepartrwent
Whitih bookey
baszuat shoudd be
bottent
parthared
theuldhee bottom!
parthased otjects
curhing sarthaue inbects
disaatrant
torate, crwehone,caaber shepld
itheen pasking sarchased sbjects lndor bocala depetrvent stow.
Wiithh derrentine commeest Blaciberry shlvshoddd aut betior?
demedine
athon ge petthoed sngerts iito bexota desatront stor.
whiih tie Jack, poon, lxotien, Satrate cloudd boput betturr
conbiation
contiaation
b4ath
letten pusking surchaeed derecia into bocala cesettrvnt stor..
WWith pouth crnuten sevforse lantiv should put botoon!
peach
secforee
lenatre
when porking parthased snjeen lnto bocata reputtien cow.
Whith of pasttyy scaft geeen sniee plant be pu bonteel
patty
plaee
ouches chaved desects intor bocula cesattrent
of bean line stburts sarfloer
stf botient
parchased lnto bocat
Which
bettur?
posking pur bhood
pecinnuu oacelo chen
Figure 11. Snapshot of generated challenge-set example. In this example, we demonstrate synthesis of a multiple choice challenge set. We see that each synthesized sample has context, question, and ground truth annotations.
polatty
shike
diice
shicke
NZ
Sulee
roos band co the
pos
buttell
berbel
daer
ressa brilke?
Iumebal cap
dieasse
achium
MC
Sehreo:
Witch lait
zbaa Jolfs
Expoak
Figure 12, Dataset Sample. We provide a snapshot of the NEWTON Benchmark. Questions are populated automatically with meaningful object types. Each question is labelled with the attribute, object category, ground-truth object, question polarity. question type, and several candidate object options. Each candidate object also has several corresponding 3D object models, should the user wish to use NEWTON with perception in the loop.


Reviews:
[{"review_id": "sZ53MzQ0LK", "reviewer": "Reviewer_kDhf", "summary": "", "questions": "A. While I appreciate the NEWTON data repository and benchmark QA task, it’s not immediately clear to me the value of the “NEWTON Pipeline” for synthesizing more bespoke challenge sets. From the paper, there are no examples of what this process would look like — could you speak more to how you would hope this dataset get used?\n\nB. Can you confirm that the crowd-sourcing component described in Section 2.2 will be released along with the rest of the framework? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments and helpful feedback. Here are our responses addressing the questions and comments that were raised:\n\n> *\"A large part of physical reasoning is a notion of grounding to some externalized environment state; while the questions in NEWTON are targeting text-only physical reasoning, I could imagine different conditions objects could be in (e.g., a torn paper bag, a backpack in disrepair) that affect how one might reason over physical properties — a feature the current NEWTON framework seems to lack. Is there a way to extend the benchmark to adding reasoning conditioned on richer object descriptions? Is this already supported?\"*\n\nWe thank the reviewer for pointing this out, and concur that different conditions that objects could be in affects reasoning over physical properties. In order to encompass these diverse object conditions, it becomes necessary to incorporate perception into the process. This entails representing the object’s state through mediums like images or point clouds. NEWTON currently contains thumbnails and 3D models paired with each category. These visual references were at the disposal of annotators during the crowdsourcing phase, thereby grounding their responses to a certain extent in the depictions of objects. To support more diverse conditions outside of the distribution of objects provided, the authors are actively working on an extension which enables part-level annotations that are grounded on perception, where, for instance, annotators can identify the location of a tear on a paper bag, etc.\n\n> *\"While I appreciate the NEWTON data repository and benchmark QA task, it’s not immediately clear to me the value of the “NEWTON Pipeline” for synthesizing more bespoke challenge sets. From the paper, there are no examples of what this process would look like — could you speak more to how you would hope this dataset gets used?\"*\n\nWe thank the reviewer for pointing this out. We have taken the feedback into account and added an additional section with an example of synthesis of additional challenge sets in the Appendix (the updated version will be made available upon notification of results, per EMNLP policy). \n\nThe synthesis process begins with Context and Attribute Specification, where users identify a context, and relevant attributes. Next, object filtering involves using the identified attribute(s) to *automatically* filter objects into a positive set and a negative set. Using the grouped objects, one can specify query templates, and automatically populate the templates to synthesize a diverse and customized challenge set. Using this, language models can be evaluated for accuracy in the specific user-identified context to find the optimal prompting strategy and model. Given the task of packing purchased items into a box at a department store, our objective is to evaluate the language models’ ability to factor in the attribute of stiffness. To achieve this, we create a novel challenge set that aligns with the specific requirements and focus of the task. Since we have access to a large repository of object-attribute pairs, we can effortlessly generate nearly infinite combinations to populate the question template. This empowers users to harness the NEWTON repository of object-attribute pairs to generate questions which suit their own application. By doing so, they can assess the language models’ competence in addressing the task and identify the most suitable language model for their needs.\n\n> *\"Can you confirm that the crowd-sourcing component described in Section 2.2 will be released along with the rest of the framework?\"*\n\nYes, we aim to open-source the interface for the crowd-sourcing component.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is the gold standard for a resource paper on a timely and important topic! As folks in the robotics and vision communities are turning more and more to using LLMs for things like task planning, questions of physical reasoning over fine-grained attributes (beyond existing benchmarks like PIQA) has become increasingly necessary, and this work does an incredible job filling that void, and doing so in a way that can be easily extended by the community.\n\nFurthermore, the existing evaluation on the initial NEWTON QA benchmark is already illuminating, and the findings in just running the baselines will have immediate impact to the community.", "reasons_to_reject": "I think this is a fresh and useful contribution to the field, so I’m definitely not advocating for rejection in any way; the following is a small weakness/observation that I’m hoping the authors can address to help ensure the longevity of this resource!\n\nA large part of physical reasoning is a notion of grounding to some externalized environment state; while the questions in NEWTON are targeting text-only physical reasoning, I could imagine different conditions objects could be in (e.g., a torn paper bag, a backpack in disrepair) that affect how one might reason over physical properties — a feature the current NEWTON framework seems to lack. Is there a way to extend the benchmark to adding reasoning conditioned on richer object descriptions? Is this already supported?\n", "questions_to_authors": "A. While I appreciate the NEWTON data repository and benchmark QA task, it’s not immediately clear to me the value of the “NEWTON Pipeline” for synthesizing more bespoke challenge sets. From the paper, there are no examples of what this process would look like — could you speak more to how you would hope this dataset get used?\n\nB. Can you confirm that the crowd-sourcing component described in Section 2.2 will be released along with the rest of the framework? \n", "paper_topic_and_main_contributions": "This paper introduces NEWTON, a new benchmark of 160K templated question/answers for investigating fine-grained physical reasoning abilities (ability to recognize properties such as texture, brittleness, along with deeper relative reasoning — “I am packing a grocery bag; which object should I put at the bottom?”). Notably, NEWTON also releases a data repository consisting of a curated collection of 700 objects and associated (cleaned) physical attributes, along with a crowdsourcing framework for sourcing more data. Finally NEWTON introduces a custom pipeline that enables new users to synthesize personalized template questions, to quickly generate custom Q/A sets for bespoke dataset/object/attribute splits.\n\nIn addition to the resource contribution, the paper evaluates many existing large-scale LLMs on the NEWTON QA benchmark, with careful analysis and thorough findings on the types of physical reasoning current LLMs are good at, and more importantly, where they have room to improve.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "Tzp7119vEx", "reviewer": "Reviewer_EC6n", "summary": "", "questions": "A: Is there a particular reason why the Agreement metric is used to evaluate Track 1 while the Accuracy metric is used to evaluate Track 2 and 3? \n\nB: Are the polarity and position bias problem addressed in the benchmark evaluation results (i.e. scores in table 4 & 5), or only be taken care in the ablation study? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful suggestions and feedback, and address all the concerns below:\n\n> *\"The question can be ambiguous without giving a specific instance of the object, as the object can be made of different materials that leads to different physical properties. For example, a metal bowl and a glass bowl may have different degree of brittleness, while a flat and a inflated basketball may have different degree of sftiffness. As a result, asking the physical property for a general object type does not make sense.\"*\n\nWe thank the reviewer for bringing up this point, and agree that for some objects, different objects made of different materials will exhibit different physical properties, and hence may raise ambiguities. We note that this was also a main consideration when we implemented the pipeline, and hence motivated some of the design choices for the crowdsourcing and data filtering stages. During the annotation crowdsourcing stage, we designed the interface such that annotators are provided with a set of images which represent relevant objects for the specific object type (as seen in Figure 2). This enabled the annotations to be grounded on common objects within the category. To further disambiguate the responses, we enforce a minimum of 0.75 inter-annotator agreement for all annotated object-attribute pairs. For particular attributes/objects which exhibit high variance in responses across annotators, we exclude them from the subsequent phases of the benchmark QA dataset synthesis. Understanding that the nature of physical reasoning holds inherent subjectivity, our intention is to mitigate this subjectivity through our methodology. By grounding human annotations on objects (which inherently captures possible state and material variations), and by enforcing high inter-annotator agreement, we hope to reduce the presence of subjectivity within the collected annotations. Furthermore, in our pursuit of minimizing subjectivity in annotating physical attributes of objects, a prospective avenue involves breaking down objects into constituent parts and annotating at a part-level granularity. This is an ongoing next step.\n\n> *\"The metrics (as described in Appendix A.1) are hard to understand. As a result, it is not clear what the numbers mean in the result tables. For example, if the accuracy for binary boolean questions in Table 5 is measured by the number of correct answers divided by the total number of questions, simply reversing the prediction of Dolly-V2-7B would yield an accuracy of 97% which is clearly wrong.\"*\n\nThanks for raising this question! We have modified the paper with a more in-depth explanation of the metrics in the Appendix A.1, this updated version will be made available upon notification of results, per EMNLP Policy. The agreement metric regards the human agreement percentage as an upper limit and adjusts the calculated accuracy based on this percentage. This adjustment results in a higher emphasis on questions with substantial human agreement, while assigning comparatively less significance to questions where human annotator responses exhibit greater diversity. This metric aids in evaluating the extent to which language model responses align with those of humans. On the other hand, the accuracy metric considers the human-majority response as the definitive label and thus has a maximum attainable value of 100%. This metric gauges the proportion of responses that align with the majority-voted human response.\n\nTo answer the question regarding Dolly, Dolly-V2-7B is an instruction-following language model. Similar to GPT-3.5-Turbo and GPT-4, the model’s output is generated in a free-form manner. This means that it may be possible to generate responses that are hallucinations, etc. Hence, the accuracy metric filters for answers which closely resemble the correct answer. Incorrect responses as an example could be: more than one answer given, invalid answer (answering c when only a and b are possible), or no response (i.e. N/A). Hence, we cannot reverse the predictions of the models, since it is often the case that the model does not output responses that can be reversed. We provide some examples of incorrect responses below:\n\n`A raincoat is less sharp than a drill.\nA) True\nB) False\nCorrect answer: True`\n\n* **dolly-v2-7b**: It's a false assumption. Drill and raincoat are two different objects. Therefore, their hardness cannot be compared.\n\n`Ginger is less soft than a birthday cake.\nA) True\nB) False\nCorrect answer: True`\n\n* **dolly-v2-7b**: In adulthood, you become more aware of the sharp, sudden bursts of craziness that are part of the human experience. As a counterweight to the craziness, there’s a need for order. And one of the ways we provide that order is with the definition of soft and hard. The underlying, soft skeletal structure of an adult human’s body provides a very logical definition of a “soft” human: one that’s blended, one that’s orderly, and one that doesn’t go over the top. A soft human tends to be more stable, controllable, mature, and grounded. On the other hand, the overly chewy and crunchy of a “birthday_cake” human often reminds us of the irrational, impulsive, immature, and emotional that we once was. We fight that impulsiveness, that recklessness, that part of ourselves that would like to feel like we’re sixteen again and don’t have a care in the world. As an interesting aside, for many, the exact opposite is true: they’re considered “too soft” to be called adult. But as a whole, adults are defined by their ability to hold it together when crisis hits.\n\n`A waffle is more hard on the surface than baseball bat.\nA) True\nB) False\nCorrect answer: False`\n\n* **dolly-v2-7b:** This answer is tricky. The way waffle involves ridges and valleys on top of a smooth surface is very different from baseball_bat, which is ridged and has a smooth surface. On the one hand, waffle's ridges and valleys can cause mechanical friction, which could make waffle more damaging than baseball_bat. On the other hand, baseball_bat is more destructive since it has ridges and valleys that can cause mechanical friction. This leads to a chicken-and-egg type of question where we don't know which factor is the most important.\n\nAs can be seen, Dolly occasionally outputs responses which are hallucinated and does not match any of the given boolean options. In these scenarios, it is not possible to simply reverse the prediction. We have provided additional examples of correct vs. incorrect responses in the updated Appendix A.4 and Figure 11, this will be made available in the paper upon notification of results. \n\n\n> *\"Is there a particular reason why the Agreement metric is used to evaluate Track 1 while the Accuracy metric is used to evaluate Track 2 and 3?\"*\n\nThanks for bringing up this question. The reason that the Agreement metric is used to evaluate Track 1, while the Accuracy metric is used to evaluate Track 2 and 3 is due to the nature of the tracks. Track 1 is designed to assess whether the reasoning capabilities of language models resembles that of humans. Hence, for this track, we treat the human agreement as an upper bound, and scale the accuracy measures by the human agreement measure for each assessed question. Doing so, we are effectively putting more weight on the questions for which there was high human agreement, and less weight on questions where there were more variance across human annotators. Since the reviewer brought up this question, we introduce the following table, which measures the performance of various models on the Track 1 questions with the accuracy measure.\n\n| **Language Model** | **Agreement (%)** | | | | | | | | | | |\n| ------------------ | ----------------- | - | - | - | - | - | - | - | - | - | - |\n|                    | **Elasticity**    | **Stiffness** | **Surface Smoothness** | | **Surface Hardness** | **Softness** | **Brittleness** | | **Malleability** | **Sharpness** | **Overall** |\n| **Dolly-V2-7B**                    | 26.0  | 15.3  | 5.5  | | 4.7  | 1.4  | 1.4  | | 10.0  | 2.6  | 8.6  |\n| **Flan-Alpaca-GPT4-XL**             | 11.2  | 1.5   | 75.5 | | 25.8 | 28.4 | 7.5  | | 12.0  | 8.2  | 18.3 |\n| **Flan-T5-small**                  | 0.0   | 0.0   | 0.0  | | 0.0  | 0.0  | 0.0  | | 0.0   | 0.0  | 0.0  |\n| **Flan-T5-Base**                   | 88.8  | 58.6  | 78.6 | | 36.7 | 28.8 | 7.5  | | 87.8  | 5.2  | 48.5 |\n| **Flan-T5-Large**                  | 0.0   | 0.5   | 78.6 | | 39.1 | 28.8 | 7.5  | | 12.2  | 4.6  | 16.6 |\n| **Flan-T5-XL**                     | 11.2  | 5.4   | 19.1 | | 53.9 | 66.3 | 7.5  | | 12.4  | 59.6 | 31.3 |\n| **UnifiedQA-V2-T5-Large**          | 0.0   | 0.0   | 78.6 | | 39.1 | 11.4 | 7.5  | | 12.0  | 26.0 | 17.2 |\n| **Alpaca-LoRa-7B**                 | 11.2  | 0.0   | 56.4 | | 0.8  | 16.4 | 4.6  | | 12.0  | 2.0  | 12.1 |\n| **GPT-Turbo**                      | 26.0  | 0.0   | 0.9  | | 16.4 | 7.5  | 7.0  | | 67.6  | 1.2  | 18.9 |\n| **GPT-4**                          | 57.8  | 10.3  | 17.7 | | 58.6 | 66.3 | 92.5 | | 50.6  | 64.8 | 57.1 |\n\nWe find that since the upper bound of performance is now 100% (as opposed to the human agreement percentage with the agreement metric), there is an overall increase with all metrics across all models. However, the general observed trend remains the same as Table 4, which uses the agreement metric, and the conclusions drawn in Section 5.2 remain. \n\n> *\"Are the polarity and position bias problem addressed in the benchmark evaluation results (i.e. scores in table 4 & 5), or only be taken care in the ablation study?\"*\n\nWe thank the reviewer for raising this point. The polarity and position bias problem is specifically examined in the ablation study, where we recognized that the problem is common across the majority of examined language models. To delve into the influence of polarity in a more comprehensive manner, we provide Table 8 in the Appendix, which scrutinizes the models’ performance in scenarios where the polarity is positive vs. negative. In the NEWTON pipeline, automatically generated questions are tagged with the polarity of the question, and the corresponding position of the correct answer. This design facet enables users of NEWTON to systematically examine the effects of polarity and position bias for any benchmark generated using NEWTON. Recognizing that mitigation of such biases is an open area of research within NLP, we hope that NEWTON can contribute to the evaluation of strategies aimed at addressing these biases. \n\n> *\"The captions are too small. Text in Figure 3 is hard to see. The data statistics in figure 4 and 6 are hard to see.\"*\n\nThanks for these valuable suggestions! We have updated our paper to reflect the changes, which will be made available upon final notification of results, per EMNLP policy.\n\n> *Note on Reproducibility*\n\nWe noticed that the reviewer indicated a 2 for reproducibility. We have tried our best to make our submission as reproducible as possible by including the source code, dataset, and a detailed outline of the model parameters, validation performance, and evaluation metrics in the main paper. We welcome any additional suggestions to improving the reproducibility of our work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of setting up evaluation benchmarks for physical reasoning capability of LLMs is well-motivated. The proposed resource is a magnitude larger in terms of the number of questions than existing benchmarks, and also covers more objects and physical attributes. ", "reasons_to_reject": "- The question can be ambiguous without giving a specific instance of the object, as the object can be made of different materials that leads to different physical properties. For example,  a metal bowl and a glass bowl may have different degree of brittleness, while a flat and a inflated basketball may have different degree of sftiffness. As a result, asking the physical property for a general object type does not make sense. \n- The metrics (as described in Appendix A.1) are hard to understand. As a result, it is not clear what the numbers mean in the result tables. For example, if the accuracy for binary boolean questions in Table 5 is measured by the number of correct answers divided by the total number of questions, simply reversing the prediction of Dolly-V2-7B would yield an accuracy of 97\\% which is clearly wrong. ", "questions_to_authors": "A: Is there a particular reason why the Agreement metric is used to evaluate Track 1 while the Accuracy metric is used to evaluate Track 2 and 3? \n\nB: Are the polarity and position bias problem addressed in the benchmark evaluation results (i.e. scores in table 4 & 5), or only be taken care in the ablation study? ", "paper_topic_and_main_contributions": "The paper proposes Newton, a new benchmark to evaluate model's physical reasoning capabilities. The data collection is performed through a crowdsourcing process using the Label Studio Enterprise platform. Each annotation task consists of a question, a set of images, and three answer choices. A minimum of four annotators contribute overlapping annotations for each object-attribute pair, and a stringent filtering threshold of 0.75 is applied to ensure reliability. Over 20,000 annotations were collected, featuring 700+ objects and 8 unique physical reasoning attributes. After filtering, there are 160K questions in the benchmark. The authors evaluate 10 recent language models on the benchmark. They find that LLMs overall do not have a consistent comprehension of foundamental physical attribute, while some model (e.g. GPT4) can perform quite well on implicit scenario-based analysis where the reasoning of physical attribute is used to make decision under certain situation.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The captions are too small. Text in Figure 3 is hard to see. The data statistics in figure 4 and 6 are hard to see. "}, "extra_text": {}}, {"review_id": "FUpXgdOOvn", "reviewer": "Reviewer_G7Xo", "summary": "", "questions": "1. Could you please try some experiments to see if the prompting approach can improve the performance?\n\n2. I want to learn more about what is the difference between this paper and existing EQA tasks.\n\n3. The paper makes a claim that the dataset also studies how observations affected decision marking. Does this dataset contain the visual image shown in Figure 1? Or it is purely the text?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for their insightful comments and questions, and provide our responses below:**\n\n> *\"Lack of awareness of work called \"Embodied Question Answering (EQA)\"*\n\nThanks to the reviewer for pointing this out! We have modified the related works, where we discuss the relevance and differences between embodied question answering and our work. In particular, embodied question answering is centered around questions with answers visually grounded to a physical environment. While certain physical attributes can be differentiated explicitly, there are implicit attributes which cannot be determined purely through visual cues (i.e. elasticity, stiffness, etc.), and requires some degree of common-sense reasoning. Our work is focused specifically on physical attributes on an object level, by curating a large repository of object-attribute pairs. This can potentially complement existing EQA benchmarks, by augmenting questions with an additional layer of physical reasoning. \n\n> *\"Need more experience to verify the argument \"current LLMs do not have physical reasoning capability\". For example, I am curious to see if we provide a prompt, i.e., a few examples, will the model have better performance? I think for the current capability of GPT4, the number will be higher than the results reported in the paper.\"* and *\"Could you please try some experiments to see if the prompting approach can improve the performance?\"*\n\nWe thank the reviewer for the suggestion of experimenting with prompt engineering! We have conducted additional experiments and highlighted the results in the following tables (Boolean, Multiple Choice, Overall) \n\n**Boolean:**\n\n| **LLM + Prompting Method** | Elasticity | Stiffness | Surface Smoothness | Surface Hardness | Softness | Brittleness | Malleability | Sharpness | Overall |\n| ------------------------ | ---------- | --------- | ----------------- | --------------- | -------- | ----------- | ----------- | --------- | ------- |\n| **GPT-Turbo + V1**        | 51.5       | 60.5      | 53.0             | 56.0            | 71.0     | 53.5        | 57.0        | 80.5      | 60.4    |\n| **GPT-Turbo + V2**        | 56.5       | 71.5      | 64.0             | 52.0            | 49.5     | 68.5        | 56.5        | 61.0      | 59.9    |\n| **GPT-Turbo + V3**        | 54.0       | 53.5      | 59.0             | 60.5            | 59.0     | 61.0        | 56.0        | 60.0      | 57.9    |\n| **GPT-Turbo + Ensemble**  | 55.0       | 68.5      | 64.5             | 60.5            | 58.0     | 62.5        | 57.0        | 69.0      | 61.9    |\n| **GPT-Turbo + Instruction**| 62.0       | 68.0      | 56.5             | 68.0            | 57.5     | 55.0        | 66.0        | 80.5      | 64.2    |\n| **GPT-4 + V1**            | 68.5       | 67.0      | 66.5             | 65.5            | 82.5     | 63.5        | 73.0        | 86.0      | 71.6    |\n| **GPT-4 + V2**            | 66.0       | 80.0      | 58.0             | 72.0            | 71.5     | 83.5        | 71.5        | 69.5      | 71.5    |\n| **GPT-4 + V3**            | 75.5       | 71.5      | 61.0             | 88.5            | 73.0     | 88.0        | 72.0        | 83.5      | 76.6    |\n| **GPT-4 + Ensemble**      | 68.5       | 76.5      | 61.5             | 79.0            | 77.0     | 85.5        | 75.0        | 83.0      | 75.8    |\n| **GPT-4 + Instruction**   | 77.5       | 85.5      | 75.5             | 90.5            | 80.0     | 68.5        | 72.0        | 96.5      | 80.8    |\n\n**Multiple choice:**\n\n| **LLM + Prompting Method** | Elasticity | Stiffness | Surface Smoothness | Surface Hardness | Softness | Brittleness | Malleability | Sharpness | Overall |\n| ------------------------ | ---------- | --------- | ----------------- | --------------- | -------- | ----------- | ----------- | --------- | ------- |\n| **GPT-Turbo + V1**        | 46.5       | 72.5      | 48.5             | 82.5            | 78.0     | 41.5        | 41.0        | 92.0      | 62.8    |\n| **GPT-Turbo + V2**        | 25.5       | 56.0      | 36.5             | 52.5            | 55.0     | 44.5        | 38.5        | 61.5      | 46.2    |\n| **GPT-Turbo + V3**        | 54.5       | 55.0      | 31.5             | 69.5            | 53.5     | 57.0        | 36.0        | 66.5      | 52.9    |\n| **GPT-Turbo + Ensemble**  | 36.5       | 54.5      | 33.0             | 67.0            | 56.5     | 45.0        | 34.0        | 65.0      | 48.9    |\n| **GPT-Turbo + Instruction**| 30.5       | 53.5      | 34.0             | 63.5            | 49.5     | 32.0        | 25.0        | 69.5      | 44.7    |\n| **GPT-4 + V1**            | 74.0       | 76.5      | 70.0             | 87.5            | 91.5     | 65.0        | 71.0        | 98.0      | 79.2    |\n| **GPT-4 + V2**            | 49.5       | 66.5      | 37.5             | 83.5            | 70.0     | 73.5        | 69.0        | 82.0      | 66.4    |\n| **GPT-4 + V3**            | 75.5       | 68.0      | 40.5             | 84.0            | 75.0     | 78.5        | 70.5        | 76.0      | 71.0    |\n| **GPT-4 + Ensemble**      | 65.5       | 68.5      | 47.0             | 84.5            | 81.5     | 77.0        | 71.5        | 87.5      | 72.9    |\n| **GPT-4 + Instruction**   | 62.5       | 57.5      | 54.0             | 71.5            | 65.0     | 50.5        | 31.0        | 74.0      | 58.2    |\n\n\n**Overall:**\n| **LLM + Prompting Method** | Elasticity | Stiffness | Surface Smoothness | Surface Hardness | Softness | Brittleness | Malleability | Sharpness | Overall |\n| ------------------------ | ---------- | --------- | ----------------- | --------------- | -------- | ----------- | ----------- | --------- | ------- |\n| **GPT-Turbo + V1**        | 49.0       | 66.5      | 50.7             | 69.2            | 74.5     | 47.5        | 49.0        | 86.2      | 61.6    |\n| **GPT-Turbo + V2**        | 41.0       | 63.7      | 50.2             | 52.2            | 52.2     | 56.5        | 47.5        | 61.3      | 53.1    |\n| **GPT-Turbo + V3**        | 54.2       | 54.2      | 45.2             | 65.0            | 56.2     | 59.0        | 46.0        | 63.2      | 55.4    |\n| **GPT-Turbo + Ensemble**  | 45.8       | 61.5      | 48.8             | 63.7            | 57.2     | 53.8        | 45.5        | 67.0      | 55.4    |\n| **GPT-Turbo + Instruction**| 46.2      | 60.8      | 45.2             | 65.8            | 53.5     | 43.5        | 45.5        | 75.0      | 54.4    |\n| **GPT-4 + V1**            | 71.2       | 71.8      | 68.2             | 76.5            | 87.0     | 64.2        | 72.0        | 92.0      | 75.4    |\n| **GPT-4 + V2**            | 57.8       | 73.2      | 47.8             | 77.8            | 70.8     | 78.5        | 70.2        | 75.8      | 69.0    |\n| **GPT-4 + V3**            | 75.5       | 69.8      | 50.7             | 86.2            | 74.0     | 83.2        | 71.2        | 79.8      | 73.8    |\n| **GPT-4 + Ensemble**      | 67.0       | 72.5      | 54.2             | 81.8            | 79.2     | 81.2        | 73.2        | 85.2      | 74.3    |\n| **GPT-4 + Instruction**   | 70.0       | 71.5      | 64.8             | 81.0            | 72.5     | 59.5        | 51.5        | 85.2      | 69.5    |\n\nIn total, we experiment with 5 methods of prompting, including three paraphrased variations of the initial prompt, prompt ensembling using the three variations (finding and using the most frequent occurring response), and example-based prompting (providing 8 unique examples in the prompt). We find that for boolean style questions, ensemble and example-based prompting leads to improved performance for GPT-Turbo and GPT-4 for most physical attributes. However, for multiple choice questions, such methods yield similar or worse performance as compared to directly prompting. This outcome demonstrates the importance of prompt engineering, and its varying impact on different types of tasks. An interesting future direction is to learn and find the most suitable type of prompt given different question styles and attribute topics to maximize the model’s accuracy.\n\n> *\"The scope of physical reasoning. I could imagine that there might be some physical property does not catch by this dataset. Could the instructions provided in the paper consider all the physical properties?\"*\n\nWe thank the reviewer for bringing up this important point! While we tried to be as comprehensive as possible in selecting the physics attributes considered, there indeed exists many other variations of attributes which are of importance to different applications. It is essential to recognize that the versatility embedded within NEWTON’s design facilitates seamless integration of further attributes. For instance, new attributes could be incorporated into the dataset through the same procedure described in Figure 2, involving annotation and filtering. Given our intention to make the annotation interface openly accessible, the acquisition of additional annotations can be done in a streamlined manner. This approach not only facilitates augmentation of the dataset with new attributes, but also promotes a convenient and collaborative means of enhancing NEWTON’s comprehensiveness.\n\n> *\"I want to learn more about what is the difference between this paper and existing EQA tasks.\"*\n\nWe thank the reviewer for this question. Existing embodied question answering tasks primarily take a perception-oriented approach in understanding objects and scenes. Tasks typically include searching for items near another item, determining the visual characteristics of a particular object, among others. Our benchmark focuses on understanding the physical attributes of objects, some of which are implicit (i.e. elasticity, brittleness, etc.) and cannot be determined through typical perception means (RGB, RGB-D, etc.). NEWTON can potentially complement existing EQA tasks by adding an additional layer of QA tasks involving physics reasoning in scenes, and can help with informed incorporation of language models in physically grounded settings (i.e. robot manipulation tasks).\n\n> *\"The paper makes a claim that the dataset also studies how observations affected decision marking. Does this dataset contain the visual image shown in Figure 1? Or it is purely the text?\"*\n\nWe thank the reviewer for bringing up this point! The proposed dataset not only has language-based annotations, but also has annotations coupled with the object category, thumbnails (2D image), and 3D models. While incorporating and benchmarking with the provided visual information is beyond the scope of the current paper, our dataset is capable of supporting evaluation of Vision Language models, in addition to purely language models. We have updated the paper with examples of the questions, associated object categories, and paired images in Appendix A.9 and Figure 12, all of which will be made available upon notification of results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written. It has a clear motivation and the experiment section\n\n2. The experiments make sense to me, and ablation studies make the argument stronger. For example, I appreciate that the author fine-tune their models on this dataset.\n\n3. This paper provides a systematic way to study the problem of physical reasoning.", "reasons_to_reject": "1. Lack of awareness of work called \"Embodied Question Answering (EQA)\". The major issue of this paper is that it does not discuss the relationship between the concept of physical reasoning and EQA tasks (https://openaccess.thecvf.com/content_cvpr_2018/papers/Das_Embodied_Question_Answering_CVPR_2018_paper.pdf). In EQA, the robotic agent navigates rich 3D environments to answer questions such as \"what is the color of the ball?\" or \"how many apples in the kitchen\", which needs language understanding, visual reasoning, and goal-driven navigation to succeed. For Track 3: Implicit Scenario-based Analysis questions such as I am packing a backpack. Question: Which of <object 1>, <object 2>, <object 3>, <object 4> should I put at the bottom?, this task feels very similar to the nav-pick-place task in EQA (see this paper: https://aihabitat.org/challenge/rearrange_2022/). I think the author should add more discussions about this.\n\n2. Need more experience to verify the argument \"current LLMs do not have physical reasoning capability\". For example, I am curious to see if we provide a prompt, i.e., a few examples, will the model have better performance? I think for the current capability of GPT4, the number will be higher than the results reported in the paper.\n\n3. The scope of physical reasoning. I could imagine that there might be some physical property does not catch by this dataset. Could the instructions provided in the paper consider all the physical properties?", "questions_to_authors": "1. Could you please try some experiments to see if the prompting approach can improve the performance?\n\n2. I want to learn more about what is the difference between this paper and existing EQA tasks.\n\n3. The paper makes a claim that the dataset also studies how observations affected decision marking. Does this dataset contain the visual image shown in Figure 1? Or it is purely the text?", "paper_topic_and_main_contributions": "The paper proposes a new benchmark, called Newton, that systematically studies the problem of physical reasoning, which involves the cognitive process of comprehending and predicting the actions and dynamics of physical systems based on observation and common sense. The existing benchmark does not study the problem of physical reasoning. The dataset consists of 700 objects across 8 physical attributes and 160K pre-generated questions. In addition, the dataset creation process consists of 4 main components: (1) annotation that let annotators to best describe the property of the object; (2) filtering that eliminates ambiguous data points; (3) template generation that generates the question, and (4) evaluation that benchmarks existing LLMs.  In addition, the questions of interest consist of (1) Foundational Attribute Understanding that concerns the physical property of the objects, (2) Explicit Application that groups the object class, and (3) Implicit Scenario-based Analysis that chooses the object based on the condition.\n\nFinally, several conclusions are made. First, LLMs do not have a good comprehension of the physical concept of objects and are capable of decision-making in implicit scenario-based tasks. And few ablation studies are also provided such as fine-tuning the model can improve the performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "EQA paper as mentioned above.", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents "NEWTON", a benchmark designed to evaluate fine-grained physical reasoning abilities in Large Language Models (LLMs). NEWTON comprises 160K templated question/answers that focus on properties such as texture and brittleness, as well as deeper relative reasoning scenarios. The benchmark is built using a curated collection of 700 objects with associated physical attributes. A unique feature of NEWTON is its custom pipeline that allows users to synthesize personalized template questions for custom dataset/object/attribute splits. The paper also evaluates several LLMs on the NEWTON QA benchmark, providing insights into their strengths and weaknesses in physical reasoning. The benchmark aims to fill the gap in evaluating LLMs' ability to reason about physical properties and scenarios.

NEWTON is a timely and valuable contribution to the field, addressing the increasing need for benchmarks that evaluate LLMs' physical reasoning abilities, especially as robotics and vision communities increasingly use LLMs. The resource is extensive, covering a wide range of objects and physical attributes, making it significantly larger than existing benchmarks. The paper is well-structured, with clear motivations and comprehensive experiments. Ablation studies further strengthen the paper's arguments, and the fine-tuning of models on the dataset is a commendable effort. Overall, the paper provides a systematic approach to studying physical reasoning, making it a significant addition to the community.

While NEWTON is a valuable contribution, it has some limitations. The benchmark might lack the ability to reason based on richer object descriptions, such as the condition of objects, which can affect physical reasoning. Questions can be ambiguous due to the general nature of objects, as objects made of different materials can have varied physical properties. The authors, during the rebuttal, provided the reviewers with effective answers and new relevant experiments.
Decision: Accept-Findings