Physical Reasoning and Object Planning for Household Embodied Agents
Authors: Ayush Agrawal
Raghav Prabhakar
Anirudh Goyal
Dianbo Liu
Venue: Accepted by TMLR ()

Abstract: In this study, we explore the sophisticated domain of task planning for robust household embodied agents, with a particular emphasis on the intricate task of selecting substitute objects. We introduce the \textbf{C}ommonSense \textbf{O}bject \textbf{A}ffordance \textbf{T}ask \textbf{(COAT)}, a novel framework designed to analyze reasoning capabilities in commonsense scenarios. This approach is centered on understanding how these agents can effectively identify and utilize alternative objects when executing household tasks, thereby offering insights into the complexities of practical decision-making in real-world environments. Drawing inspiration from factors affecting human decision-making, we explore how large language models tackle this challenge through four meticulously crafted commonsense question-and-answer datasets featuring refined rules and human annotations. Our evaluation of state-of-the-art language models on these datasets sheds light on three pivotal considerations: 1) aligning an object's inherent utility with the task at hand, 2) navigating contextual dependencies (societal norms, safety, appropriateness, and efficiency), and 3) accounting for the current physical state of the object. To maintain accessibility, we introduce five abstract variables reflecting an object's physical condition, modulated by human insights, to simulate diverse household scenarios. Our contributions include insightful human preference mappings for all three factors and four extensive QA datasets (2K, 15k, 60k, 70K questions) probing the intricacies of utility dependencies, contextual dependencies and object physical states. The datasets, along with our findings, are accessible at: \url{https://github.com/com-phy-affordance/COAT}. This research not only advances our understanding of physical commonsense reasoning in language models but also paves the way for future improvements in household agent intelligence.

Published in Transactions Machine Learning Research
Physical Reasoning and Object Planning for Household Embodied Agents
Ayush Agrawal National University of Singapore
ay 2y.agroual8122@mmii.co com
Raghav Prabhakar IIT--Hydrabb,, India
Anirudh Goyal
amirudhsoyoll11198mma...om
Deepmind, London
Dianbo Liu National University of Singapore
dianbo Breus, edu..g
Reviewed on OpenReview: httpss//ooooooee.ee...eee//ee/es
Abstract
In this study, we explore the sophisticated domain of task planning for robust household embodied agents, with a particular emphasis on the intricate task of selecting substitute objects. We introduce the CommonSense Object Affordance Task (COAT), novel framework designed to analyze rcasoning capabilities in com- monsense scenarios. This approach is centered understanding how these agents can effectively identify and utilize alternative objects when executing household tasks, thereby offering insights into the complexities of practical decision-making in real-world environments. Drawing inspiration from factors affecting human decision- making, we explore how large language models tackle this challenge through four meticulously crafted commonsense question-and--a rules and human annotations. Our evaluation of stale-of-the-ar language models these datasets sheds light three pivotal considerations: 1) aligning object's in- herent utility with the task at hand, 2) navigating contextual dependencies (socictal norms, safety, appropriatennss, and efficiency), and 3) accounting for the current physical state of the object. To maintain accessibility, we introduce five abstract variables reflecting an object's physical condition, modulated by human insights, to simulate diverse houschold scenarios. Our contributions include insightful human preference mappings for all three factors and four extensive QA datasets (2K, 15k, 60k, 70K questions) probing the intricacies of utility dependencies, contextual dependencies and object physical states. The datasets, along with our findings, are accessible at: hhttps://githiii cetiittub.comocoeo www.cccmmoooooeecco wttpe////oooooooaaoooddaaaaa phy-affordancee/COTTT This research not only advances our understanding of physical commonsense reasoning lan- guage models but also paves the way for future improvements in household agent intelligence.
datasets featuring refined

---

Published in Transactions Machine Learning Research (May /2024)
1 Introduction
Humans, as beings innately attuned their surroundings, traverse a world where conversations, decisions, behaviors, and understanding are deeply embedded in the underlying fabric of sit- uation. Their engagement with the world entails commonsense (background) knowledge about entities-pae propeeties......s..... spatial relations, events, causes and effects, and other social norms (MMcCarth 1959 [Winograd] 1972); (Davis & Marcus 2015)). The importance of situational awareness is starkly evident in our daily tasks, where choosing objects for specific activities showcases our adaptability to different settings. Consider the straightforward task of cutting cake how do we determine which object is suitable for this task? When person needs to select an object to accomplish this task, an array of factors will affect the choice. For example, we must choose something capable of cutting (Utility suitable for cutting cake (conteztual appropriateness), and likely in appropriate physical condition to be used (physical state). These considerations would be to ensure the appropriateness, case, and safety of those cutting the cake as well as who will eat the cake. Although these ionss might seem trivial and inluitive to us humans, they are still an important aspect to consider when developing embodied household agents. Such reasoning capabilitics can be potentially leveraged by embodiod agents to generate action plans for human requirements represented in natural language. In this work, we propose CommonSense Object Affordance Task: textual physical commonsense task to evaluate most appropriate object selection capabilities in the presence of various alternative objects.
Object Level Selection
Physical State Level Selection
Task: Cutting Cake Utility: Cutting
Task: Cutzing Cale Utility: Cutting
Uuility: Cutting
Unility Level
Context Lavel
Plysical
All Objjcts
Utility Objects
Context Objects
Confgurrtioos
1. Brrolcon Knife
1.
Khife
Kuife
Knife
2. Clean Sciesors
2. Susssrs
3. Clean Pillow
4. Busy Stool
5. Clean Knife
Figure 1: We divide the whole mecision-making process into 2 broad phases. Pruning out options firstly based on Object Level then Physical State. Within the Object level, we further divide into 2 sub-steps: Utility and Conteztual Appropriateness. We highlight this method's adeptness in comparing appropriateness across array of factors and coming up with substitute object even in the absence of the ideal object [Cake Knife]. Our work provides QA datasets about this type of commonsense reasoning
2

---

Published in Transactions Machine Learning Research [May/2024]
Recent advancements in large language models (LLMs) (Zuu et al.) 2023 Peng et al. 2023 Zhang
ct al. 2023 Brown et al., 2020 Chowdhery et al. 2022 Touvron et al. 2023 OpenAl
2023 have
significantly enhanced our ability to extract rich commonsense knowledge from extensive web data. To analyze this task and evaluate the current capabilities of Language Models across such human commonsenss--riintte reasoning, we develop this task as a decision-makinn process spanning 3 major aspects: Utility: The concept of Utility, a focal point in previous research Speer et al.) 2017); elucidates our understanding of an object's functionality in a variety of situations. Although ConceptNet (Spcer ct al., 2017) has been a crucial tool for identifying object-utility relationships, its nature a human-compiled knowledge graph has led to the pursuit. of more dynamic sources. We curate Object-Utilitty mappings pertaining to this aspect and a 2K QA dataset to evaluate the object selection capabilities of language models based on required utility. Context: Our decision-making extends beyond mere utility. To account for the various situational factors such as safety, social norms adherence, effort optimization, efficiency, and situational appropriateness, we introduce the second aspect as: contertual appropriateness This adeptness in judgment arises from our ingrained commonsense, sculpted by experience and intuitive physical understanding. To evaluate the reasoning capabilities of various language models across this aspect, we generale Coiect--Uiiity-aaaaae mappings and curate a 15K MCQ QA dataset. Physical State: Previous work Li et al. (022) has shown how object choice depends on various physical variables. To make this mamnonssnse-orrenteen we add a layer of abstraction and introduce 5 abstract variables to depict the current physical state of the object. To observe how object usability evolves with various abstract physical state variations, we generate human preference mappings and curate 2 QA datasets (specifically focused on analyzing object usability with varying physical states). These two physical state datasets summed up to 130K MCQ question-answer pairs combined. Thus overall, we curated 4 QA datasets.
aspect more human
In Figure D we illustrate example of using these 3 aspects to select the best feasible object. For the task of cutting a cake where the following objects are available: a Broken Knife, Clean Scissors, a Clean Pillow, and a Clean Knife. Beginning with pruning objects based on their utility(cutting) brings our locus primarily on Knife and Scissors. Further knowledge about the task (cutting a cake) leads to the dismissal of Scissors as a suitable tool for cake cutting. Finally, upon considering the physical state of Knives, the Clean Knife emerges as the obvious choice. This explains the three key factors we explore for evaluating task-specific object selection capabilities the utility of the object, its contextual appropriateness, and its current physical state.
Such commonsense reasoning capabilities not only allow us to judge the appropriateness of an object in the context of the given task but also help us in successfully coming up with an appropriate substitute object in the absence of the most ideal object (Here: Cake Knife). Such skills, if equipped with embodied agents, will enhance their rcasoning capabilitics and make them adept in planning tasks in scenarios where the ideal object is not available.
Main Contributions: In this study, we made the following contributions:
Creation and provision of human- prefeeceee mappings across all 3 aspects of the CommonSense Object Affordance Task(COAT)
'This shouldn't be confused with the overall objective of choosing object that maximizes the utility. This could be comprehended as "function' "aspect" in the focus of the given task
3

---

Published in Transactions Machine Learning Research (May /2024)
Introduction of 4 major novel CommonSense-bel based QA Datasets, facilitating an in-depth analysis of how object usability evolves under different utility requirements, contextual scenarios, and physical states
Evaluation of Large Language Model baselines on these dalasets, accompanied by a detailed analysis of their performance in multi-step abstract reasoning scenarios.
2 Dataset Creation
To systematically investigate the capacity of LLM to conduct human-style physical commonsense reasoning and preferences across three crucial factors, we have devised an experimental framework centered around 75 household tasks, carefully curated to span 22 distinct utilities, The experiment involves a diverse inventory of 100 objects sourcod from the AI2Thor Simulator (Spcer et al. 2017 ensuring relevance and diversity within household context.
1. Tasks: are high-level household activities that could be accomplished by a human or embodied agent. Example: Cutting a Cake. See Task List)
2. Utilities: are different aspects of a high-level task. A task can comprise or more utilities.
For the example of Cutting Cake, the utility could be Cutting. While for the task of Making an Omelette, utilities could be Mixing, Heating etc. See Table 2a
3. Objects: are subset of objects available in AI2Thor (Kolve et al. Table 26
2022 Simulator. See
Utilities
Objects
Carrying
Comfort
Heating(vesse
Bowl
Bed
Pan
Cleaning
Washing
Mixing(tool)
DishSponge
SinkBasin
Spoon
Disposing
Cutting
Mixing (vessel)
GarbageCan
Knife
Cup
Storage
Entertainment
Heating(source)
Fridge
Laptop
Microwave
Reading
Breaking
Increasing Height
Newspaper
BaseballBat
Chair
Eating
Writing
Physical Activity
Apple
Pen
Dumbell
Decoration
Light Source
Surface Support
Statue
Floor Lamp
Counter CounterTop
(a) representational subset of utilized Utilities
(b) A representational subset of utilized Objects
The following section gives an overview of the annotation tasks and the process of creating Com- monSense Reasoning Datasets.
2.1
Human Preference Collection Utility
2.1.1
Incorporating GPT3.5-turbo (Brown et al. 2020) along with human commonsense annotations, we meticulously established mapping between utilities and objects. These are called Utility Objects. Notably, each object may be associated with multiple utilitics, and converscly, single utility can
4

---

Published in Transactions Machine Learning Research [May/2024]
be linked to various objects. Table provides an overview of the utilities along with their associated objects utilized in our experiments. More Information about the annotation process can be found in Appendix D
2.1.2 Contextual Appropriateness
In evaluating object utility, it is crucial to recognize that suitability for specific tasks vary significantly. Take, for example, the multifaceted use of candle. While it. possesses the inherent ability to generate heat, employing a candle for heating soup introduces a range of practical limitations. This observation underscores the complexity of human preference and decision-making in the context of object utility. Key factors influencing these choices include efficiency (as illustrated by the impracticality of using a candle for heating soup), safety considerations (such as the risks associated with standing on an armchair), social norms and constructs (exemplified by the unconventional choice of serving wine in bowl), and the overall appropriateness of an action (e.g., the disposal of eggshells in sink basin). To systematically explore these dynamics, engaged human annotators in study designed to assess the selection of appropriate objects for specified tasks and utilities
2.1.3 Physical State
The selection of objects for specific tasks is influenced not only by intangible factors such as safety and social constructs but also by the object's 's current physical state. Prior research, including the works Li et al. (2023) and Gao et al. (2023), has employed various physical parameters to examine Large Language Models' (LLMs) comprehension of an object's physical attributes. In our study, we shift the focus to task planning under non-ideal conditions, necessitating reasoning about potential substitute objects. To this end, we have developed five distinct variables, each represented by abstract symbolic terms. These variables have been derived directly from the A12Thor Simulator, facilitating their broader applicability and potential integration into the burgeoning field of Embodied AI. Table ! delineates these variables and their corresponding abstract values. Here, Already In Use variable is used to represent the availability of an object for use. Some examples of an object in reversible-using state are the object getting recharged, a wet object, or an object in a reversible state (meaning it will need time get back to the idcal state or is temporarily being used by someone else). Whereas in an irreversible-using state, the object could be broken, depleted, out of stock, and thus is in irreversible state of use. Further details about the chosen physical variables are elaborated in Appendix A.1
Variables
Abstract Values
material
Metal, Wood, Plastic, Glass, Ceramic, Stone, Wax, Fabric, Rubber, Food, Paper Sponge, Organic, Soap
mass
Light, Medium, Heavy, Super-Heavy
temperature
Cold, Room RoomTemp, Temp, Hot
already in use condition
Free, Reversible-Usingg Irreversible-Using
Dirty, Clean, Broken
Table 1: Abstract Values for Various Variables
5

---

Published in Transactions Machine Learning Research [May/2024]
Gathering Common Object Configurations In the context of this study, a Conf configuration denotes the physical state of object characterized by five variables. While conceivable in the realm of Madame Tussands, it remains highly improbable in everyday household scenarios. Thus, to ensure the relevance of configurations to common household scenes, human annotators were tasked with selecting plausible and frequently occurring variable values for each object. (See Appendix D
chair might be
Ranking Object Configurations In our study, we not only provided configurations that occur commonly but also tasked the annotators with categorizing the configurations of object into three distinct classes: Ideal, Moderate, and Bad. This classification the anticipated time required for an agent commence the task with given object configuration. Utilizing these categorizations, we constructed two comprehensive datascts comprising 130,000 questions specifically designed to assess the physical commonsense reasoning capabilities of Large Language Models. Further details on this process are elaborated Appendix D
predicated on their assessment of
2.2 CommonSense QnA Datasets
Based on Utility Appropriateness, Contextual Appropriateness, and Physical State. We ereated 4 CommonSense QA datasets.
1. Task-u This experiment was based on pruning objects based their compatibility with the utility specified in the question. We curated Objectwiiltty Level QA Dataset and utilized Object- Utility Mappings (obtained from human annotators) for setting the ground truth.
2. Task-0: This experiment based pruning objects based only contextual factors that affect an object's task-specific appropriateness. We curated another Objectcontext Level QA Dataset and used Context Mappings for setting the ground truth.
3. Task-1 & Task-2: These experiments were based on pruming out objects based on only physical state variations (described by 5 symbolic variables). We curated 2 Variable Level Datasets and utilized human annotations for setting the ground truth.
2.2.1 Objectutility Level Dataset
To evaluate the object-utility aligument in LLMs, we curated an Object Level QA dataset (with ground truth obtained from human annotations) and made LLMs choose the most appropriate object for given utility. Here we specified information about the context and physical state of objects. This was done to evaluate solely billit-bbssed selection capabilities.
Example Question of Task-u
Question: Which of the following objects would be best suited for the purpose of "heating(source) "?
Options:
(A) Spatula
(B) StoveBurner
Correct Answer:
B
"These different from the tasks(activities) used datasets
6

---

Published in Transactions Machine Learning Research (May /2024)
2.2.2 Oojectcontext Level Dataset
To evaluate the reasoning capabilities of LLM when choosing objects over contextual factors, we curate an Object Level QA datasct. Here, previously recorded Context Mappings were kept as ground truth. (See Aunotation Task 2.1.2). Here, we specified no information about the physical state, thus assuming every object to be an ideal configuration. This was done create QnA datasets focused solely on object selection capabilities based on contextual factors.
Question Every question be assigned <Task, Utility> combination and framed in the way shown below:
Question
What object would you be choosing for <utilliy>> when you are tasked to <task>?
Based on the sampling strategy and the number of options in the prompt, we created 4 variations of objectccntext level dataset. An example of such variation is shown below.
Options
1. Variation-1 For each question, we randomly sampled 1 context object and 1 utility object both belonging to the same utility
Example Task 0 Variation
Question ID: 1,
Utility: heating(source),
Question: Which of the following objects would be best suited for the purpose of "heatingssource) when tasked to "reheating coffee"?
Options:
(A) Toaster
(B) StoveBurner
Correct Answer: B
2.2.3 Physical Configuration Level Dataset
Based on Common Configurations generated in the annotation task 2.1.3 we create 2 Variable Level QA datascts to analyze the reasoning capabilities of Language Models on pruming out options based on their current. physical state. The datasets differ in the level of difficulty and the level of reasoning required to answer the questions correctly. We describe the creation process this section. The question in both datasets remains the same that of Object Level Dataset. However, unlike the first dataset where the options were objects, this time give various Configurations of Context Objects options. Here we ensured all options were appropriate according to the question's utility and context. This was done to evaluate the object selection capabilities solely based on the physical state, thus precluding the possibility of wrong answers due to a wrong object being selected (due to the other 2 factors).
Details about other variations Task 0,1,2 found here Appendix B
7

---

Published in Transactions Machine Learning Research (May /2024)
Configuration
object name: Microwave, mass: super-heavy, temperature: RoomTomp, material: Metal already in use: free condition: clean
For Objectutility level Dataset, we sampled <Utility Objects>> based on the question's ''s util- ity and for Objectsconext level Dataset, we sampled <Context Objects> based on question's 's <Task,Utility> combination. However, use different approach for generating physical con- figuration datasets. Here, we classified the configurations of Context Objects into three broad categories: 'Idcal," "Moderate," and 'Bad." Each category is defined by specific variable values that delineate their characteristics. The 'Ideal' category represents configurations in their optimal states, facilitating the specified task without additional time/matte simemmaterial penalties. In contrast, the "Moderate" category includes configurations that deviate from these ideal states, resulting both time and material costs for their utilization. The models assess these options basod on their estimated costs. Lastly, the "Bad" calegory comprises configurations that make the Context. Objects unusable (even after considering potential penalties). Both "Moderate" and 'Bad' configurations Sub-Optimal Configurations, offering a muanced understanding of the varying degroes of object usability.
grouped under
By sampling options from these sets of configurations 2.1.3 we divide our efforts into creating 2 physical configuration datasets:
A. Ideal Configuration Dataset In alignment with its name, the "Ideal Configuration' dataset involves questions with the correct answer as Ideal Configuration of Context Object of the question's associated <taskt,tility>> combination. To systematically analyze the behavior of models, we introduce 12 distinct variations of this dataset. The creation of these variations designed to progressively augment. the complexity of the datasets, facilitating a comprehensive analysis of model behaviors. Each of the 12 variations comprises approximately 5,000 question- answer pairs, with differing counts of aattios-rranging from 5 options to 2 options per question. Along with the varying number of options, different sampling techniques help us study their behavior concerning different object distributions, the deliberate variation in the number of options enables us to evaluate the success rate variations of Large Language Models (LLMs) with increasing levels of required reasoning.
also ablated on various sampling techniques. While
Process: To create these variation datasets, we sampled a Task for n number of times, where n is proportional to the total count of all Commonly Occurring Configurations of its Utility Objects. [Amotation Task 2.1.1 For given Question's <Task, Utility> Combination, randomly sample a Context Object Trom the pool of Context objects. (obtained from 2.1.2 An example of sampling the remaining options is explained below:
For 5 option datasets:
1. Variation-1 :
randomly selected Context Object's Ideal Configuration + 4 randomly sampled sub-optimal configurations of the same Context Object.
2. Variation-2 randomly selected Context Object's Ideal Configuration + 2 randomly sampled sub-optimal configurations of the same Context Object + 2 randomly sampled sub-

---

Published in Transactions Machine Learning Research [May/2024]
optimal configurations of different Context Object belonging the same <Task,Utility> combination
Example for Task 1 Variation 1
Question ID: 1,
Utility: heauunn
Question: Which of the following objects would be best suited for the purpose of (heating(source)) when tasked to "reheating coffee"?
Options:
(A) object name:
Microwave, mass: super-heavy, temperature: RoomTemp, material: Metal, already in use: free, condition: clean
(B) object StoveBurner,
super-heavy, temperature: RoomTemp, material: Metal irrewersible-usinng condition: dirty
already
(C) object name:
CoffeeMachine, mass: heavy, temperature: RoonTemp, material: Metal already in use: irreversible-using, condition: dirty
(D) object name: Microwave, mass: super-heavy, temperature RoomTemp, material: Metal already in reversible-using, condition: clean
(E) object name: Microwave, mass: super-heavy, temperature: RoomTemp, material: Metal, already use: irreversible-using, condition: broken
Correct Answer:
B. Sub-Optimal Configuration Dataset Although the process of selecting an ideal configuration is challenging for language models, typically it does not require intricate mulli-step reasoning involving rigorously excluded all idcal
consideration of wide range of factors. To evaluate their reasoning abilities configurations (particularly when faced with only sub-optimal options) from our sampling methodology. This deliberate exclusion necessitates that the models engage in more sophisticated reasoning by considering various physical state variables, thereby testing their capacity for abstract reasoning. By focusing exclusively on sub-optimal configurations, this methodological shift enables a more thorough investigation into the language models' ability to navigate and reason through complex scenarios in the absence of clear-cut ideal solutions.
Proccss: To comprehensively assess language modcls' abstract reasoning capabilities when con- fronted with sub-optimal configurations, create another Variable Level QA dataset and introduce 14 variations of this dataset. Like the previous dataset, each dataset is constructed using distinct sampling strategies and has a varying number of options. Across all 14 datasets, we maintain a consistent structure of nearly 5,000 questions.
Each question this dataset variation is associated with a Task and Utility combination. While the set of questions remains consistent with previous datasets, the sampling of each task is now proportional to the count of 'Moderate Configurations + Bad Configurations' (i.e., the count of Sub-Optimal Configuratoons for that question's associated <Task, Utility> combination). An example of 2 sampling techniques used for generating variation datascts is explained below:
4The remaining variations in sampling techniques and option be found in Appendix

---

Published in Transactions Machine Learning Research (May /2024)
For 5 option dataset
1. Variation-1: We sample all 5 options from Moderate Configurations of the Context object of the question's associated <Task, combination.
2. Variation-2: We sample 4 options from Moderate Configurations and option from the Bad Configurations of the Context object of the question's associated <Task,,tility> combination.
Example Task 2 Variation
Question ID: 1,
Utility: heating(source),
Question: Which of the following objects would be best suited for the purpose of "heating(source) when tasked to "reheating coffee"?
Options:
(A) object name:
CoffeeMachine, mass: heavy, temperature: RoonTemp, material: Metal, already in use: reversible-using, condition: dirty
(B) object StoveBurner, mass: heavy, temperature: RoonTemp, material: Metal,
already
reversible-using. condition: clean
(C) object name: StoveBurner, mass: heavy, temperature: RoonTemp, material: Metal, already use: free, condition: dirty
(D) object name:
CoffeeMachine, mass: heavy, temperature: RoomTemp, material: Metal, already in use: reversible-using, condition: clean
object Microwave, mass: heavy, temperature: RoomTemp, material: Metal,
already use: reversible-using, condition: clean
Correct Answer: C
3 Experimental Setup & Results
Using these 4 datasets, we evaluate various Large Language Models benchmark their performance on major themes:
1. R1: Objoct-level Commonsense Reasoning (performance on utility and various contextual factors including social constructs, feasibility aspects, etc.)
2. R2: Physical State level Commonsense Reasoning (performance commonsense under- standing of various physical variables and how they affect decision making)
We evaluate and compare the performances of various large Language Models using the following metrics:
1. Accuracy:
The fraction of questions answered correctly by the Language Model.
2. Bad Rate: The fraction of questions in which the chosen answer belonged to the "Bad" configuration pool.
"The remaining variations in sampling techniques and option be found in Appendix
10

---

Published in Transactions Machine Learning Research (May /2024)
3.1
Dataset Summary
Task
# #Variation
#Q
Av
Options
GT
4
2K
500
Objects
Utility Objects
4
15.5K
3.8K
Utility Objects
Context Objects [Ò]
12
58,7K
1.9K
O's Configurations
O's Ideal Configurations
14
68.9K
1.K
O's SubaptConfigurations
' BestSubopt Configurations
Table 2: Summary of Datasets Used for Experiments
3.2
Glossary
Term
Definition
objects
a set of 100 household objects set of 22 abstract utilities
ufilities
tasks
set of 75 household activities <Task,Utility> combination indicating the specific task aspect to emphasize symbolic variable used to explain an object's physical state complete description of object using symbolic variables mapping between utility and object; facilitates Utility-Objects mapping between task, utility and object; facilitates Context-Objects A state where object is ready perform task without requiring additional time or effort
question
variable
configuration
ulility-mappiin
maperngnappig
ideal-com configuration
marked by the simultancous occurrence of all ideal variable values in its description A state requiring additional effort to reach an ideal condition for task performance, marked by the presence of moderate variable values.
moodemate-coffiuuratto
bud-confeguration
An inoperalive slate of an object with irreparable issues, marked by the presence of 'Bad Variable" values.
sub-optimal configuration
group of 'Moderate' and 'Bad' configurations
3.3 Results
Task
Analysis:
Table 3: Here, as move from left to right, the number of options increases in the dataset (from 2 options to 5 options). The findings suggest a near-perfect alignment of objects and their associated utilities. Thus, from here onwards we focus our analysis and discussions object selection capabilities across contex- tual factors and physical stale variations
Accuracy
Model
2-opt 3-opt 4-opt 5-opt
PaLM
98.60 96.80 96.20 92.70
GPT3.5-Turbo
97.30 96.20 94.90 92.10
#Q=question count; Av=average question count
11

---

Published in Transactions Machine Learning Research (May /202))
Task 0 Analysis: We observe from Table that the performance of GPP3.5-Tuube and PaLM outperform other models with much smaller mumber of parameters. This may be attributed to their size as well as the amount of internet data they've been trained on. They both showcased similar performance, suggesting similar object-level reasoning capabilities. Even though the performance of every model was observed to be impressive, Mistral-7B outshone all other models of similar size as well as both 13B models. Upon analyzing the trend average accuracy across various datasets for TasueO 3 we note an important trend implying a drop accuracy as we increase the number of options. This suggests degradation in reasoning capabilities as the number of comparisons increases. This trend was observed in Task and Task 2 wel [ Thus through Table 1 and Figure 3 we get a fair evaluation of the reasoning capabilities of language models over object-level reasoning tasks. [R1]
Task-0 Variations
Model
2-opt
3-opt
4-opt
5-opt
2-apt BI B.3-apt
#4-apt
5-opt
v1
v2
v3
v4
100
PaLM
90.0
74.2
68.3
65.0
G-T3.--Tuubb
88.8
71.9
68.3
66.9
Accuracy (9))
vieunal3B
71.0
54.5
49.3
46.2
LLama2-13B
76.5
58.2
50.9
46.9
Vicuna7b
51.0
34.5
28.5
26.5
Mistral-7B
76.2
57.9
50.2
47.2
Matra-TB
ChIGLMD-E
ChatGLM-6B
62.0
42.0
34.2
27.6
Model
ChatGLM2-6B
62.9
44.6
34.3
35.4
Figure 3: Average Accuracy of various models on Task 0 as we increase option count
Table 4: Model accuracy when evaluated on Task-0
Task 1 Analysis: Table 5 summarizes the performance accuracy of different models on Task-1 datasets where models were lasked to reason based on physical configuration of objects (using Ideal Configuration Datasets). This task was aimed at judging if language models have an understanding of the difference between Ideal Configuration and Sub-Optimal Configurations. Here also we witness the superior reasoning capabilities of GPT3.5 5-rbo and PaLM, with the latter outperforming the former on each dataset by average of 8.8%. Amongst the smaller models, we see Mistral7B dominating all other 7B and 6B models. Here Vicuna7B and ChatGLM-6B performed very close to random performances and thus were excluded from further analyscs. For 13B models, LLama2-13B showcased its superior reasoning capabilities and was on average 7.6% more accurate than Vicuma33B, Here, apart from the falling average accuracy with increasing options, we also notice some interesting behaviors as we increase the object diversity (i.e an increase the number of sub-optimal configurations of different context object (of same <Task, Utility> com- bination) barring the object who's Ideal Configuration is already in the options the correct answer).
#See Figure 9a
trends
Task:1 and Task: Average Accuracies for each dataset type
12

---

Published in Transactions Machine Learning Research (May/2024]
Figure
illustrates the decreasing performance of all small models as we increase the ob-
Model
Accuracy Task Variationsf
5-option
4-option
3-option
2-option
v1
v2
v3
v4
v5
v6
v7
v8
v9
v10
v11
v12
PaLM (Chowdhery et al. 2022
85.9
74.6
81.4
89.4
80.0
80.5
84.8
92.0
84.6
88.5
95.0
91.7
GPT3.5 Turbo [Drown ct 2020
74.0
64.5
70.7
78.9
69.5
69.3
75.2
89.2
78.0
82.6
91.3
80.4
Vicuma 13B Chiang et al 2023
14.5
44.0
36.4
53.2
50.9
53.3
42.9
00.6
59.9
51.1
67.8
66.6
LLama2-10 13 Touvton ct 81. 2023
49.1
48.2
46.9
54.2
54.0
56.5
54.3
70.6
67.3
70.2
78.96
75.5
VicumaBB Chiang et al 2029
24.0
24.6
24.5
31.7
32.8
32
32.2
40.5
39.2
41.0
56.8
56.8
Mistral- liang el a..
2025
37.2
34.6
30.7
44.4
42.4
44.2
39
53.6
51.9
47.0
74.2
69.7
ChatGLM 6B es al. 2022h
25.8
25.6
21.7
31.2
30.4
31.6
28.3
38.8
39.5
36.0
53.4
52.0
ChatGLM2-000 Du et al.
20222
31.8
31.4
30.0
39.4
40.9
40.6
40.5
54.0
53.4
51.2
68.0
66.0
Table 5: Performance accuracy for various models when evaluated on Task 1 (Ideal Configuration Dataset)
5-Options
4-Optans
3-0ytoons
2-Opiions
Accuracy Trends Fowe Each Medel
2
3
2
Diversity Objects
PaLM
Vicuna
LLmm2-113
Vicuna7B
Mistral-7B
ChatGLM-6B
Mha6
Figure 4: Comparative plot showcasing the variations in Task:1 performances we keep incrcasing the object diversity in options from left to right.
ject diversity in the options. This sheds light on the existing bias towards using a commonly used object rather than choosing an object decided upon reasoning over every object's complete physical state. However, for big models like PaLM and GPrr.5-Turbo, we notice an improvement in accuracy extreme object diversities. Thus conclude that even though there was drop in accuracy in PaLM and GPT3.5-Turbo for more diverse options unlike the small models, they were not answering excessively based on their bias towards the commonly used object.
Task 2 Analysis: Table summarizes the performance of various models on Task-2, where the models were asked reason over the best choice of object configurations from the Sub-Optimal Configuration Datasets. This task could be interpreted as finding the option that would be the least time-consuming and most appropriate amongst variety of Sub-optimal Configurations of
13

---

Published in Transactions Machine Learning Research (May /2024)
Model
Accuracy for Task Variations
5-option
4-option
3-option
2-option
v1
v2
v3
v4
v5
v6
v7
v8
v9
v10
v11
v12
v13
v14
PaLM
32.4
38.0
46.3
55.3
64.1
40.8
47.6
57.7
63.2
52.4
64.8
69.0
70.2
80.7
5-rro5-Turbo
28.3
30.6
37.5
46.4
61.6
34.6
40.1
50.72
61.2
46.1
56.7
71.3
61.1
80.2
V3cuna13B
22.5
23.9
28.0
32.0
32.8
27.7
31.0
35.3
41.2
37.3
42.9
50.0
51.8
08.1
LLama2-13B
23.0
24.4
44.9
43.4
53.9
39.9
50.5
66.2
57.4
75.7
Mistral-7IB
20.7
22.4
27.8
25.8
27.8
25.8
29.0
32.3
37.6
35.0
40.6
17.7
52.6
63.7
ChatGIMM2-6B
22.2
30.6
36.3
50.7
Vicuna-7B
20.3
21.6
21.6
21.7
22.7
26.4
25.9
27.6
28.2
33.3
35.6
38.3
48.5
ChauGLM-GB 6B
21.5
22.4
22.6
23.9
23.5
25.0
27.3
29.2
29.2
33.1
34.8
36.3
48.2
53.6
Table 6: Accuracy for various models when evaluated on Task 2 (Suboptimal Configuration Dataset) Context Objects of the question's <Task,Utility> combination. Here, we sampled some moderate configurations (neither Ideal nor Bad) and some Bad Configurations. The best amongst the moderate ones was kept as the Ground Truth. [Refer Appendix D Our observations reveal consistent superiority of - and PaLM across all models. Notably, GPT-3.5-Turbo consistently lags behind PaLM by average margin of 3.7%. Despite their commendable comparative performance, both models exhibit limitations in comparing various physical variables of moderate configurations, resulting in a significant performance downturn. Even this time, we observed Vicuna7B and ChatGLM-6B exhibiting erratic behaviors reflected in their consistent random outputs. While I.Lama2- 13B performed superior to all other small-scale models, the general observed order was Chat PaLM. In addition to the drop in average accuracy with increasing options, Figure shows the
Mistral-7B < Vicunal3B < LLama2-131 GPT3.--Turbo
5-Options
4-Options
3-Options
2-Options
Accuracy Trends For Each Model
3
No. of Bad Configurations
PaLM2
GPT3.5-Turbo
vicunal3B
LLama2-13B
Mistral-7B
ChaiGLLM26B
Figure 5: Comparative plot showeasing the variations in Task:2 performances as we keep increasing the Count of Bad Configurations in Options from loft to right.
trend enhanced performance as we increase the count of Bad Configurations within a type of dalaset. This could be attributed to the ability of models differentiate Bad Configurations from Moderate Configurations. To delve deeper and analyze what fraction of the responses were correct and what fraction was from the Bad Configurations, we make use of another metric: Bad Rate.
14

---

Published in Transactions Machine Learning Research [May/2024]
Table 7 shows the percentage of questions where a Bad Configuration was predicted as the correct answer. In our evaluations, this would mcan the model went wrong with the reasoning in these questions. To probe LLM reasoning when presented with varied amount of "bad configurations" we went on to increase the fraction of bad configurations present in each question's 's options as we moved from left to right([v1-vv5,, [v6-v99], [v10-v12]). With an increased fraction of options belonging to 'bad configurations' we expected an increase in the bad rate. good model would have a smaller magnitude of the bad rate as well as larger gap between the fraction of the bad options line (dotted line) and their bad rate value. Figure 6 further showeases the trend of observed bad rates and the Irend of the increase of the fraction of bad options we inserted in the prompt. While PaLM showed the least rise in bad rates, we observed -Lama2-13B outperforming all
and GPT2.5- other models and consistently trying to achieve PaLM and PPP3.5--urbo performance. Based on these figures and analyses, we can safely conclude that most models had a sense of what bad configuration is but showed limited reasoning capabilities to evaluate moderate configurations based on abstract physical variables. Thus, through Task and Task 2, we were able to evaluate and analyze commonsense reasoning capabilities of language models over physical state variables [R2]]
Model
Bad Rate For Task_ 2 Variations
5-option
4-option
3-option
2-option
v1
v2
v3
v4
v5
vß
v7
v8
v9
v10
v11
v12
v13
v14
PaLM
4.2
10.3
21.2
35.9
6.0
15.3
36.8
8.8
31
19.3
GPT3.5-Turbo
5.5
12.5
24.4
38.4
7.7
17.5
38.8
9.7
28.7
19.8
vicuna13B
11.2
23.2
39.0
67.2
15.0
33.3
55.8
18.7
50
31.6
LLama2-13B
7.2
18.8
29.3
55.1
9.6
24.6
46.1
13.0
33.8
24.3
Mistral-7B
15.8
28.7
48.2
72.2
17.0
35.7
62.4
19.5
52.3
36.3
ChatGLM2-6B
15.0
31.5
48.6
71.0
17.0
34.6
63.7
20.4
49.3
38.6
Table 7: Bad Rate for various models when evaluated on Task 2 (Suboptimal Configuration Dataset). It signifies the fraction of chosen options that were bad choices. (-) is used for datasets where there were no bad options
4 Conclusion & Future Work
Accurately reasoning over an object's current physical state an important aspect of developing robust embodied agents that can accomplish tasks even if the ideal objects are not available. We created 3 step framework to break the decision-making process that humans go through mentally while choosing an object for task completion. We created 4 major datasets to evaluate object-level and physical state-level reasoning capabilities in LLMs. Regarding Object Level Reasoning CapabilitiesRRI)): Large Language Models were found to have near-perfect object- utility aligument [lask-u]. We further found that small models performed decently when evaluated on tasks judging their object-level reasoning capabilities based on contextual factors [task-O]. However, their performance dropped as we increased the number of options in the question. For large LLMs like PaLM and GPT3.5 Turbo, observed very impressive object-level selection capabilities. However, here as well we saw drop in accuracy as we increased the number of options in each question.
For additional experiments and fine-tuning results see Appendix F.
15

---

Published in Transactions Machine Learning Research (May /2024)
PaLM GPr35-Turbe
LLmma2-338 vicuna138
Mistral-78
ChatGLLM--65
723
Bad Bad rates(% (%)
35.3 38-4
3037
21.2
Model
Figure Bad rates for various option datasets we increase the count of bad options from left right. Dotted Lincs reprosent the fraction of options that bad in each datasct, whereas the difference between the dolied lines and bars tells us about the model's ability to not gel confused as we increase the count of bad options from left to right on the x-axis
*PAM SOPELLTTAR
Madel
Made
Figure 7: Bad rates for various 4lleft) and (righht option datasets as increase the count of bad options. While the dotted line represents the fractions of options that were bad in each dataset, the difference between the dotted lines and bars depicts the model's ability to not get confused as we increase the count of bad options from left right on the x-axis
Regarding Physical State-Level Reasoning (R) While evaluating commonsense reasoning over an object's physical state, we noticed that all language models displayed impressive abilities to identify Ideal Configurations on task-1. For small models, we noticed a decreasing accuracy within each fixed count option dataset dataset(task-1) as we increased the object diversity. This brought forth the internal bias of these small models to stick to object commonly used for task, condition to be readily used. However, for larger models,
even if it is not in Ideal State we observed a lesser degradation in such accuracy. There was a stark difference in how large and small LLMs behaved when tasked to choose the most appropriate moderate configuration. While all language modcls displayed an increase in accuracy as we increased the fraction of Bad Configuration ions each question (lask-2), not all were able to avoid confusion between Moderate Configurations and Bad Configurations. This ability identify and choose Moderate Configurations amongst mixture of Bad and Moderate Configurations decreased decrcased the size of LLMs. (as
16

---

Published in Transactions Machine Learning Research (May (202))
shown by bad rate plots). Similar to task-u, task-0, and even in task-1 and task-2, we observed a decrease in accuracy rates kept increasing the number of options in cach question.
Given these observations, we can safely conclude that language models like GPT3.5- Turbo, PaIM, and Llama2-13B can prune out appropriate objects based on utility[[aakk: and contextual Sectors[Task, 0] and extreme physical configurations (Ideal and Bad Configurations) to an impressive extent. However, they face a certain level difficulty in comparing amongst Moderate Configurations (as this requires certain amount of abstract reasoning equipped with a commonsense understanding of the world around them) [R2]. Smaller language models showcase sub-optimal behavior over [R.] and very poor behavior over [R2]
Our work opens up an avenue for improving the language model's abstract multi-step rea- soning for estimating the physical affordance of everyday objects used in household activities. Future efforts would be directed towards integrating these datasets to train Embodied Language agents and proving their competence of our 3-step architecture in successful task completion when situations aren't ideal. Judging the variable values in the real world could be tricky affair; thus, although the current work focused handerafted variables, calculating these variables and learning new latent variables from multi-modal inputs for effective analysis and reasoning about an object's applicability seems foresccable domain to explore.
Limitations
This work focuses dealing with contextual connotations associated with object when deciding whether to use it as a substitute for task execution. We further considered abstract physical variable level analysis to highlight the evolution of usability with various physical abstractions. While determining the values of these variables may appear straightforward the A12Thor Simulator, achieving the same in real-life scenarios requires a resilient model. Even if we can calculate the variables, there is a limitation which an object's state could be represented using abstract physical variables. When comparing objects, sometimes need to understand their exact situation to decide their usability. To develop robust embodied agents capable of dealing with such explicit reasoning along with abstract commonsense reasoning capabilities, further work needs to be directed along with integrating multi-modal reasoning capabilities in addition to commonsense reasoning. In addition, in this study, we assumed that all the objects were allowed to be used by the agent. In some cases, it might be possible that the human companion of the agent might have kept an object a certain way and didn't 't want disturbed. Thus, the agent might need re-calculate the object preference as per this newly imposed human preference. Further works along this line would enable us to move an inch closer toward Embodied agents capable of such constrained planning capabilities in addition to multi-modal commonsense reasoning.
5 Related Works
A lot of work has been done in the domains related to the scope of this paper. In this section, we summarize some of them:
17

---

Published in Transactions Machine Learning Research (May /2024)
Probing Language Models
Understanding what LMs know after large-scale pre-training is an active rescarch area (Rogers al. 2020). Various probing methods have been developed (Tenney
et al.
2019b);
Petronii et al. 2019 and investigations show that LMs capture linguistic (Tenney
et al.
201 Jay;
(Liu et al. 2019) factual (Petroni et al. 2019; [Rbertt et al.) 2020); (Dai et al.
2022
commonsense knowledge (Wang et al. 2019;; [orbes et al. 2019). and even acquire grounded
concepts
Patel & Paylick 2021).
CommonSense QA Datasets Evaluating to what level commonsense world understanding LMs possess has been explored by many. |Gu et al.] 2023) analyses mental models of LLMs and aligns them with improved models about everyday things; (iisk et al. 2019) consisted of questions requiring physical commonsense reasoning. Recently, there has been lot of work in NLP to utilize commonsense for QA, NLI, etc. (Sap et al. 2019); (Talmor et al. 2019). Many of these approaches seek to effectively utilize ConceptNet by reducing the noise retrieved from it (Lin et al. 2019) Kapanipathi et al.) 2020) There have been several other QA Datascts to benchmark CommonSense
Reasoning abilities Language Models. Some them include: (Geva al. 2021 YYang et al.
20181; Miihaylov et al. 2018
Reasoning in LLMs Reasoning is a crucial aspect of intelligence, influencing decision-making, problem-solving, and other cognitive abilities. (Huang & Chang 2023 presents the current state of research on LLMs' reasoning abilities, exploring approaches to improve and evaluate their reasoning skills. DDziri et al. 2023) investigates problems associated with multistep reasoning with LLMs Some of the works dealing with lackling reasoning in small models are: (Magister et. al.
2023
Fu
let al. 2023) Shridhar et al. 2023
18

---

Published in Transactions Machine Learning Research (May /2024)
References
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Morcira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Yuqian Dai, Mare de Kamps, and Serge Sharoff. BERTology for machine translation: What BERT knows about linguistic difficulties for translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 6674 6690, Marseille, France, June 2022. European Language Resources Association. URL
1.719
Ernest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM, 5899)992-1 aug 2015. ISSN 0001-0782. doi: 10.1145/2701413. URL https:://oi.orgg// 111////11115//7111111
Zhengxiao Du, Yujic Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022a.
Zhengxiao Du, Yujic Qian, Xiao Liu, Ming Ding, Jiczhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 320-335, 2022b.
19

---

Published in Transactions Machine Learning Research [May/2024]
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya. Sanyal, Scan Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality, 2023.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. Do neural language representations learn physical commonsense?, 2019.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning, 2023.
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In arXw preprint arXiv:2309.088 02561, 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies, 2021.
Yuling Gu, Bhavana Dalvi Mishra, and Peter Clark. Do language models have coherent mental models of everyday things? In Proceedings of the 61sl Annual Meeling of the Association for Computational Linguistics (Volume Long Papers), pp. 1892-1913, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 20.1863//1122223xcloorg httpss///allanthoooo.ooo//////// acl aol-ooo..... 106
URL
Jie Huang and Kevin Chen-Chuan Chang. Towards rcasoning in large language models: A survey, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Pavan Kapanipathi, Veronika Thost, Siva Sankalp Patel, Spencer Whitchead, Tbrahim Abdelaziz, Avinash Balakrishnan, Maria Chang, Kshitij Fadnis, Chulaka Gunasekara, Bassem Makni, Nicholas Mattei, Kartik Talamadupula, and Achille Fokoue. Infusing knowledge into the textual entailment task using graph convolutional networks. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8074-8081, Apr. 2020. doi: 10.1609/aaai w3..0/0.........0311... 6.188. URL https://ojs aaai httpa//wwpppppaaaaaarrrcee/ee//a
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Anirdddaa Kembhavi, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai, 2022.
Klaus Krippendorff. Computing krippendorff's alpha-reliability. Departmental Papers (ASC).
University of Pennsylvania, 2011. URL https::/reposiiorry. www.:uppwnnnneeeeeeeeeee
Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Qi Liu, Lingpeng Kong, and Xu Sun. Can language models understand physical concepts?, 2023.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning, 2019.
20

---

Published in Transactions Machine Learning Research (May/2024)
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transfcrability of contextual reprosentations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 1073-1094, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18655v1/N199-12. URL https 1al.aanthooogy.or//N1-111
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason, 2023.
John McCarthy. Programs with common sense. In Proceedings of the Teddington Conference on the Mechamization of Thought Processes, pp. 75 91, London, 1959. Her Majesty's Stationary Office. html
URL
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering, 2018.
OpenAL. Gpt-4 technical report, 2023.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2021.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases?, 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? Tn Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 5418 5426, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ /ttp222000.0mmmmaamaaamm org/2000 wwn.p-main..nmm 437.
URL https://aclaantologg
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866 2226 2020. doi: 10.1162/tan a 00349. URL htttps://cclanthoopgg org.orgg/////// tacl-1.54 55
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. Atomic: An atlas of machine commonsense for if-then reasoning. Proceedings of the AAAT Conference on Artificial Intelligence, (3(01) 30027 3035, Jul. 2019. doi: 10. v3301...30011333111300 URL https::/ojs. aaai. org/ /eerrg//ndxx...e php/ AAT/articleevvew//A600
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilities into smaller language models, 2023.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnel 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI17, pp. 4414-4451. AAAI Press, 2017.
21

---

Published in Transactions Machine Learning Research [May/2024]
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega: A question answering challenge targeting commonsense knowlodge, 2019.
Ian Tenney, Dipanjan Das, and Ellic Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Avnual Meeting of the Associalion for Compulational Linguistics, pp. 4593 4601, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10. 10.18653/ 11/119-1452. URL https////cccaannoooooooog///////
Tan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations, 2019b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vodanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Tsabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Fric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Augela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. Does it make sense? and why? a pilot study for sense making and explanation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4020-4022 4020- Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1 /P19//13331 URL https: www.laathooooo.ogg///1-----
Terry Winograd. Understanding natural language. Cognitive Psychology, 3(1):1-19) 1972. ISSN 0010-
0285. doi: http:://ddooooo//////o////////op cumm//sfeeeee/rrtceee//////00112
URL https: //ws:://www.. sciencedi
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinor, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answcring, 2018.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient finc-tuning of language models with zero-init atlention, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- languaee understanding with advanced large language models, 2023.
22

---

Published in Transactions Machine Learning Research (May/2024)
A Appendix
A.1 Dataset Specifics
2010
(a) Plot showing number of objects(x) for each util4b) Plot showing number of lasks(x) for each utility(y)
ity(y), obtained after Utility based pruning 2.1.1
A.2 Variables
Here we describe the variables used to describe an object's physical state. We kept it at an abstract level to judge basic commonsense reasoning capabilities.
1. mass: Based on an estimate of the weight of an object: (i) light(0,1 Kg] (ii) medium Kg] (iii) heavy[5,10 Kg] (iv) super-heavy[> 10 Kg]
2. material: what material is used to make that object
3. temperature: the surface temperature of the object: Coftoot/RoomTaae
4. already in use: tells us about the availability the object: reversillesssinniireeeeeelle using/free
5. condition: tells us about the condition of the object: broken/clean/dirty
A.3 Human Annotations: Object-Utilte Mappings
Table 2.1.1 summarizes the collected and refined Object-Utilltyy pairings. Throughout this work, we have referred to these as Utility Mappings.
B Dataset Creation
In addition to the variations explained in 2.2.3 we further create 3 more types of datasets for cach of the lasks. These would be each consisting of 4, 3, and 2 options. Our sampling method for these options enables us to analyze and ablate over reasoning capabilities of LLMs a zero-shot manner. The datascts arc:
23

---

Published in Transactions Machine Learning Research (May/200 /2024)
B.1 Task 0
1. Variation-2: For each question, we sampled 1 context object and 2 utility objects belonging to the same utility.
2. Variation-3: For each question, we sampled 1 context object and 3 utility objects belonging to the same utility.
3. Variation-4: For each question, we sampled 1 context object and utility objects belonging to the same utility.
B.2 Task
5 option datasets
1. Variation-3: Random context object's Ideal Configuration + 4 randomly sampled sub- optimal configurations of same Task and Utility's different context object
4 option datascts
1. Variation-4: Random context object's ' Ideal Configuration + 3 randomly sampled sub- optimal configurations of the same context object
2. Variation-5: Random context Object's is Ideal Configuration + 2 randomly sampled sub- optimal configurations of the same context object + randomly sampled sub-optimal configurations of different context object belonging to the same <Task ty> combination
3. Variation-6: Random context. Object's Ideal Configuration + randomly sampled sub- optimal configurations of the same context object randomly sampled sub-optimal configurations of diffcrent context object belonging to the same <Task, combination
4. Variation-7: Random context object's Ideal Configuration + randomly sampled sub- optimal configurations of the different context object belonging to the same <Task,Utility>> combination
3 option datasets
1. Variation-8: Random context object's Ideal Configuration + 2 randomly sampled sub- optimal configurations ions of the same context object
2. Variation-9: Random context object's Ideal Configuration + 1 randomly sampled sub- optimal configuration of the same context object + 1 randomly sampled sub-optimal configuration of different context object belonging to the same <Task,Utility> combination
3. Variation-10: 10:
Random context object's 's Ideal Configuration + 2 randomly sampled sub- optimal configurations of the different context object belonging the same <Task, combination
24

---

Published in Transactions Machine Learning Research (May /2024)
2 option dataset
1. Variation-11: Random context object's Ideal Configuration + 1 randomly sampled sub- optimal configurations of the same context object.
2. Variation-12: Random context object's Ideal Configuration + randomly sampled sub- optimal configurations of the different context object belonging to the same <TaskkUtility: combination
B.3 Task 2
5 option dataset
1. Variation-3: We sample options from the Moderate Configurations and 2 options from the Bad Configurations of the same <Task, Utility> combination's context objects
2. Variation-4: We sample 2 options from the Moderate Configurations and 3 options from
the Bad Configurations of the same <Task, Utility> combination's context objects
3. Variation-5: We sample option from the Moderate Configurations of the context objects of that particular <Task, Utility> combination, allow sampling oquivalent options long either of them is not the correct answer. We also sample 4 options from the Bad Configurations of context objects of that particular <Task, Utility> combination
4 option dataset
1. Variation-6: We sample 4 options from the Moderate Configurations of the context objects of that particular <Task, Utility> combination. Here we allow sampling equivalent options as long as either is not the correct answer.
2. Variation-7: We sample 3 options from the Moderate Configurations of the context object of that particular <Task,Utility> combination. We allow sampling equivalent options as long as either is not the correct answer. We also sample option from the Bad Configurations of the random context objects of that particular <Task, Utility> combination
3. Variation-8: We sample 2 options from the Moderate Configurations of the context object of that particular Utisity combination. We allow sampling equivalent options as long as either is not the correct answer. We also sample 2 options from the Bad Configurations of the random context objects that particular <Task, Utility> combination
4. Variation-9: We sample option from the Moderate Configurations of the context objects of that particular <Task, Utility> combination. Here, we allow sampling equivalent options as long as either is not the correct answer. We also sample 3 options from the Bad Configurations of context objects of that particular <Task, Utility> combination
3 option dataset
1. Variation-10: We sample options from the Moderate Configuration ions of the context objects of that particular <Task, Utility> combination. Here, we allow sampling equivalent options as long as either is not the correct answer.
25

---

Published in Transactions Machine Learning Research (May /202))
2. Variation-11: We sample 2 options from the Moderate Configurations of the context objects of that particular <Task, Utility> combination. Here, allow sampling equivalent options long either is not the correct answer. We also sample 1 option from the Bad Configurations of context objects of that particular <Task, combination
3. Variation-12: We sample option from the Moderate Configurations of the context object of that particular <Task,Utility> combination. We also sample 2 options from the Bad Configurations of the random context objects of that particular <Task, Utility> combination
2 option dataset
1. Variation-13: We sample 2 options from the Moderate Configurations of the context objects of that particular <Task, Utility> combination, here we allow sampling equivalent options as long as either them is not the correct answer.
2. Variation-14: We sample option from the Moderate Configurations of the context object of that particular <Task,ttility>> combination. We also sample option from the Bad Configurations of the random context objects that particular <Task, Utility> combination
C
Results
2-Options
3-Options
4-pptions
Options
90
22-pttiess
#3-Opdenss
Opdinns
5-Options
60
Arcuraty
50
-0
Vicaati
PAM
OPTAA
Sland
Mied
Charzin 43
Model
(a) Average Accuracy of Various models Task 1 as(b) Average Accuracy of Various models Task 2 we increase option count
we increase option count
D Annotation Process
The entire annotation process was text-based and was executed by circulating a text-based ques- tionnaire. Participant demographic spanned various university-level academic departments and consisted students and researchers who volunteered for such annotations. Figure 10 summarizes the entire annotation process for generating Ground Truths for all datasets.
D.1 Human Annotations:
Ubility-Object Mappings
Creation of ubility-object mappings that were further used as the backbone for all the tasks and datascts involved the use of GPT3.5-Turbo and Human Annotation. This was done by using
26

---

Published in Transactions Machine Learning Research (May 22024)
SAFEE
LEGEND
Callek Cemerce Cuidg
Talk 12
war Minei Poliersso
Colerten Taakk Opfore
Figure 10: Figure summarizing our annotation process
GPT3.5-Turbe to output Utilities for the 100 selected AI2Thor objects. From this, we then selected a random subset (alter cross-checking it. and used it to create options while generaling QnA to gather human annotation for Utility-Object Mappings. The annotators were asked to label 100 objects with utilities from a list of 22 utilities. The inter-annotator agreement was calculated by wwanoottooo-mulit-aaaa
formulating this as a variable number of labels per object. The annotator agreement was 89.2%, suggesting a high degree of agreement within the annotators. The consolidated utility-object mappings are found Link
scenario where each annotator could annotate a
D.2 Human Annotations: Task-Object Mappings
To curate ground truth task-object mappings, also called Context Mappings; ask the annotators to choose objects appropriate for <Task, Utility> combination amongst the utility objects. As one question can have more than 1 possible correct object, we calculated inter-annotator agreementt by modeling this as a process similar to the previous annotation task. The annotator agreement was observed be: 81.0%, suggesting a high degree of agreement amongst the annotators. The question posed the annotators was similar the ones used curate Task (Object Level Dataset), and the obtained responses used Ground Truth for Task 0 Dataset. The processed GT can be found here; Link
D.3 Human Annotations: Common
Mappings
To get the common variable values for all the objects, we further ask the annotators to provide all commonly occurring variable values of cach object. Using these, we created all possible configurations. Upon calculating the inter-annolator agreement as earlier annolation tasks, we observed an inter- annotator agreement of 89.9 when averaged across all 5 variables. The processed output can be found here: Link
27

---

Published in Transactions Machine Learning Research (May /2024)
D.4 Human Annotations: Ideal Object Configurations
Further, we ask the annotators to categorize variable values into 3 categories: Ideal, Moderate, and Bad. 'Ideal" refers to an ideal state of the object; 'moderate' means you have to spend some time getting the object in an ideal state before it can be used, whereas 'bad' means the object is unusable. Some variable values are obvious, such as "free", which would be ideal, whereas; "reversible-using" would be moderate, and prreversible-using" would be bad. So we only ask them give preference for variables like Material. The observed Krippendorff's reliability alpha (Krippendorl] 2011] among the raters for classifying material variable values into categorical variables: 'Ideal', "Moderate' and 'Bad" was 0.87, suggesting high degree of agreement amongst the annotators. The Ideal Configurations can be found Link
D.5 Human Annotations: Moderate Configurations
After classifying the variable values into these categories, we asked them to arrange the values in increasing order of their appropriateness for given <Task,Utility> combination. For ranks as ordinal variables, we observed Krippendorff's alpha value to be 0.89, showing high agreement amongst the annotators. Further, we sct a penalty for each moderate variable value and consequently generale penalty scores for each configuration: material penalty and time penalty. It is using these penalties we further arrange the configurations based on a time penalty and then a material penalty. This helps us create relative ranking within the moderate configurations and enables us to sample 'moderate' options when curating Task Dataset.
D.6 Human Annotations: Bad Configurations
For the Bad Configurations, we set abnormally high values for material and time penalties. These configurations help us sample 'bad' options when curating Task Dataset. The sub-optimal configurations, including 'moderate' and 'bad" configurations, can be found here: Link
E
Prompts used
Prompts used for various models can be found at this link. Link
28

---

Published in Transactions Machine Learning Research (May/2024)
F
Example Responses
Task 0
Question ID: 1,
Utility: heating(source).
Question: Which the following objects would be best suited for the purpose of "heating(source) when tasked "reheating coffee"?
Options:
A) object name: Toaster,
(B) object name: StoveBurner
Correct Answer:
PaLM
Answer: B
Reason: Stove burners safe and feasible, require minimal effort, and reheat coffee minimal time.
GPT3.5-Turbo
Answer: B
Reason: Option StoveBurner, would be best suited for the purpuse of reheating coffee in terms of safety, fcasibility, minimuma effort, and minimum time. Using stove buruer allows for precisc control of heat, enabling gradual and controlled reheating of the coffce. It's safer than using toaster, which designed reheating liquids and could pose safety risk result spillage damage the
mothod for rehcating liquids and
LLama2-13B
Answer: A
Reason: Toaster is the hest option reheating coffee as designed specifically for heating bread and other toastable items, and it can heat up quickly and evenly. Using stove burner would require more effort and time to heat the coffee, and may not heat it evenly. Additionally, using a toaster is generally safer than using an open flame on a stove burner, as it has built-in safety features such as a cool-touch exterior and automatic shut-off.
Vicuna7B
Answer: B
Option B, stove burner, the best choice for reheating coffee is efficient heating source that requires minimal effort and time. It allows precise temperature control and can easily used to reheat coffce without causing any damage to the coffee or creating any safcty hazards.
Reason:
29

---

Published in Transactions Machine Learning Research (May /2024)
Vicuna-135
Answer: B
A stove burner would be the best option for reheating coffee it provides precise temperature control, allowing for adjustment to the desired temperature. This ensures that the coffee is heated evenly and efficiently, without the risk of burning overheating. Additionally, stove burner convenient and casily accessible option, requiring minimal effort and time to On the other hand, toaster may be the best option is not designed for heating liquids and may not provide consistent accurate temperature control.
Reason:
Mistral7B
Answer: B
While toaster is convenient, option, stove burner would he the best suited for heating coffee it provides control the temperature and the of hest being applied. It also allows for larger surface be heated, which be beneficial reheating coffee. Additionally, using burner is safer and energy efficient than using toaster
Reason:
ChatGLM-6B
Answer: A
Reason: Toaster convenient efficient option for reheating coffee compared burner.
ChatGLM2-6B
Answer:
Toaster device that heats up food items such coffee. It convenicut and safer option compared burner. The toaster has simple operation, maintenance and can be used up single serving of coffee time. The time takes heat up the coffee minimal and there no need for additional effort operate it.
Reason:
Task 1
Question ID:
Utility: heouing)
Question: Which the following objects would best suited for the purpose of "heating(source)" when tasked to "reheating coffee"?,
Options:
(A) object name:
Microwave, mass: super-heavy, temperature: RoonTemp, material: Motal,
ready in use: free, condition: clean
(B) object name: StoveBurner, mass: super-heavy, temperature: RoonTemp, material: Metal al-
ready in use: irreversible-using, condition: dirty
(C) object
heavy, temperature: RoonTemp, material: Metal al- ready in irreversible-using condition: dirty
CoffeeMachine,
(D) object name: Microwave, mass: super-heavy, temperature: RoomTemp, material: Metal al-
ready in use: reversible-using, condition: clean
F object name: Microwave, mass: super-heavy, temperature: RoonTemp, material: Metal, al-
ready in use: irreversible-using condition: broken
Correct Answer: A
30

---

Published in Transactions Machine Learning Research (May/2024) 20024)
PaLM
Answer: A
Reason: Microwaves are safe and easy to use, and they can heat coffee quickly and evenly. They are also relatively lighbweight and easy to move, so they can be used in a variely of settings.
GPT3.5-Turbo
Answer:
Reason: Microwave (option D) is the best choice as it is available (reversible-using), in clean condition, and suitable for reheating coffee. It is not irreversibly occupied and can be used for the task without a time penalty.
LLama2-13B
Answer:
Reason: Microwaves are designed specifically for heating and cooking, making them safer choice compared to stovetop burners coffee machines. Additionally, microwaves generally made of durable materials and have more delicate design, which makes them less fragile than stovetop burners coffee machines. Furthermore, microwaves are quick and efficient when it comes to heating liquids, taking less time than stovetop burners or collee machines. Finally, option A has already been used (already in use: 'free") and its condition clean, which suggests that ready for immediate use.
Vicuna7B
Answer: D
Reason: This is because microwave is a common and convenient appliance for reheating coffee quickly and efficiently, and it can be used multiple times without any issues. Addition- ally, a microwave is generally safe and easy to use, and it does not require any time penalty or additional setup get started with the task.
Vicuna 13B
Answer:
Reason: A microwave is the best suited object for the purpose of reheating coffee as it is fast and efficient way to heat liquids. It is also reversible use object as it can be used again after the task completed. Additionally, a microwave is a safe and convenient option as eliminates the need for hot surfaces or open flames, which can be hazardous. The mass and material properties microwave also make it a safe and practical choice for this task.
31

---

Published in Transactions Machine Learning Research (May/2024) 20024)
Mistral-7B
Answer: A
Reason: Based on the given physical variables, object A (Microwave) would be the best suited for the purpose of heating(source) when tasked to reheating coffee. The microwave is readily available and clean, which means it can be used immediately without any additional time required for cleaning or preparation. Additionally, microwaves are generally safe for use heating lood and are less likely to break down compared to other materials. The microwave temperature also safe level for heating food. While other options such as stove burners and coffee machines may also be suitable for heating coffee, they may require additional time and effort to clean and prepare for use, and their availability may be limited due to current use or maintenance requirements. Therefore, the microwave is the best choice for reheating coffee in household setting.
ChatGLM-6B
Answer: D
Reason: The best option for reheating coffee would be the Microwave object with id 5, which has already been mentioned to be reversible and in use, but also has a lower mass compared to other options, making easier handle and transport. Additionally, metal microwave, which can provide better heating efficioncy compared other options made of different materials.
ChatGLM2-6B
Answer:
stove burner suitable for heating coffee as it has high temperature and is made metal, which is conductive
Reason:
can efficiently transfer heat. Additionally, in a state reversible using, which means temporarily unavailable but can used after some time. The condition dirty, which is not. relevant to the task of rehcating coffee.
32

---

Published in Transactions Machine Learning Research (May/2024) 20024)
Task 2
Question ID: 1,
Utility: heating (souuree,
Question: Which of the following objects would be best suited for the purpose of "heating(source) when tasked to "reheating coffee"?,
Options:
(A) object
CoffeeMachine, heavy, temperature: RoonTemp, material: Metal, already in reversible-using, condition: dirty
(B) object name: CoffeeMachine, mass: heavy, temperature: RoonTemp, material: Metal, already use: irreversible-using, condition: clean
(C) object name:
CoffeeMachine, mass: heavy, temperature: RoonTemp, material: Metal, already in use: free, condition: dirty
(D) object namesMicoowve,, mass: super-heavy, temperature: RoonTemp, material: Metal, already use: reversible-using condition: dirty
(E) object name:StoveBurner, mass: super-heavy, temperature: RoonTemp, material: Metal, already in use: irreversible-using condition: clean
Correct Answer: C
PaLM
Answer: A
Reason: The coffee machine is the most efficient way to reheat coffee, and it is also the safest option.
GPT3.5-Turbe
Answer: E
Reason: StoveBurner is available, clean, and be used without time penalty (reversible-using). It is suitable for heating and reheating tasks and is generally quick heat up.
LLama2-133
Answer: C
Reason: The Coffee Machine is the best configuration for rcheating coffce bocause it is the least time-consuming, and most reliahle option. Although it is currently dirty, it can he cleaned after use, making it more reliable choice than the Microwave or StoveBurnnr, which may have permanent damage if used with dirty or wet objects. Additionally, the Coffee Machine is already available and does not require any time to 'ee started' like the Microwave StoveBurner, which need time to hest up. Therefore, the Coffee Machine is the best option for reheating coffee based commonsense reasoning related to the provided physical variables.
Vicuna-7B
Answer: C
Reason: Coffee Machine is the best option the purpose "heating(source) when tasked 'reheating coffee" because it is already available not use, and it also in dirty condition. Therefore, it is the most suitable option to use for this task.
33

---

Published in Transactions Machine Learning Research (May/2024) 22024)
Vicuna-153 13B
Answer: A
The best option for reheating coffee is the 'Coffee Machine' because it is specifically designed for making and heating coffee. It has 'reversible-using' availability status, meaning it can be used after some time. It is also made of metal, which is safe and durable material for heating purposes. Additionally, it alrcady dirty, which means it be cleaned after without any extra time penalty. Overall, the 'Coffee Machine' is the most suitable option for reheating coffee.
Reason:
Mistral-7B
Answer: A
Reason: The Coffee Machine with already in use reversihle using and condition dirty is the best option for reheating coffee. Tt. is readily available and can he easily cleaned before use, making safe and efficient.
ChatGLM-6B
Answer: A
'he Coffee Machine is the best suited for the purpose of heating(source) when tasked reheating coffee, as it lightweight, easy to handle and can be used multiple times without any loss of heat. Additionally, it is metal and can be easily heated to high temperature, making suitable for heating coffee.
Reason:
ChatGLM2-6B
Answer:
The Coffee Machine has a free state which means can readily used without time penalty. Additionally, it has a dirty condition which temporary and can overcome. also made metal which may not safe for food contact. It also a heavy object which may be difficult move.
Reason:
34

---

Published in Transactions Machine Learning Research (May /2024)
Utility
Objects
Microwave, Stove Burner, Toaster, Oven, Coffee Ma- chine, Candle
Heating (source)
Wine Bottle, Watering Can, Vase, Bottle, Cup, Mug,
Carrying
Pot, Kettle, Bowl, Plate, Spray Bottle
Dish Sponge, Scrub Brush, Phunger, Paper Towel
Roll, Soap Bar, Soap Bottle, Towel, Newspaper, Toi- let Paper, Cloth, Hand Towel, Tissue Box, Vacuum
Cleaning
Cleaner
Washing
Bathtub Basin, Sink Basin, Toilet
Cutting
Butter Knife, Knife, Fork, Spatula, Spoon
Disposing
Garbage Can, Bathtub Basin, Sink Basin, Toilet
Mixing (tool)
Ladle, Fork, Spoon, Spatula, Butter Knife, Knife
Mixing (vessel)
Pan, Bowl, Kettle, Cup, Plate, Pot, Mug
Heating (vessel)
Pan, Bowl, Kettle, Cup, Plate, Pot, Mug
Drawer, Cabinct, Dresser, Fridge, Laundry Hamper,
Storage
Safe, Shelf, Shelving Unit, Desk, Box
Television, CD, Remote Control, Cell Phone, Laptop,
Entertainment
Desktop
Comfort
Bed, Armchair, Chair, Dog Bed, Sofa, Ottoman
Reading
Book, Newspaper, Desktop, Laptop, Cell Phone
Desk, Armchair, Chair, Coffee Table, Footstool, Din- ing Table, Countertop, Stool, Ottoman, Sofa, Side
Increasing Height
Table, Dresser, Bed
Time
Watch, Alarm Clock, Cell Phone, Desktop, Laptop Apple, Bread, Potato, Lettuce, Egg, Coffee, Tomato, Salt
Eating
Physical Exercise
Basketball, Baseball Bat, Dumbell, Tennis Racket
Writing
Pen, Pencil, Cell Phone, Laptop, Desktop
Coffee Table, Countertop, Desk, Dining Table, Dog
Surface Support
Bed, Dresser, Floor, Footstool, Ottoman, Side Table,
Sofa, Stool, Chair, Armchair, Bed
Light Switch, Window, Desk Lamp, Blinds, Curtains,
Light Source
Candle, Floor Lamp
Statue, House Plant, Room Decor, Teddy Bear, Table-
Decoration
top Decor, Poster, Painting
Butter Knife, Knife, Fork, Spatula, Plate, Ladle, Bas-
Breaking
ketball, Tennis Racket, Dumbbell, Remote Control,
Baseball Bat, Spoon
Table 8: Utilities and Objects
35

---

Published in Transactions Machine Learning Research (May /2024)
F.1
Fine-tuning Results
Upon finc-tuning a language model with subset of various datasets that we curated, we expected see an increase in the accuracy of the model. Below, we present the results obtained after we fine-tuned PaLM model on Vertex Al.
F.1.1 Task-0 Fine-tuning: Model for Object Level Selection
Due to the limitation of computational resources, we selected a slice of 400 examples of 5-option variation dataset and fine-tuned the PaLM language model for 40 training steps. In Table g present the comparison of the results before fine-tuning and after such minimal fine-tuning. Owing to the increase in accuracy across all variations after fine-tuning just 450 examples of the 5-option datasets for 40 training steps, we can safely expect substantial increase when fine-tuned on a larger split of datasets for larger number of training steps.
we
F.1.2 1,2 Fine-tuning: Model for Physical State Level Selection
Due to the limitation of computational resources, selected slice of 1200 examples which included 450 examples from all 3 variations of task-1's 5-option variation dataset Table and 750 examples from all variations of task-2's 5-option variation bataset|[see Table 6 We further fine-tuned a single PaLM language model for 40 training steps. Table I0 presents the result comparison before fine-tuning and after such minimal fino-tuning.
We note common observations observed after fine-tuning both these models:
1. Even with fine-tuning them on small subset of 5 variation datasots, we got an increase in accuracy in all datasets.
2. We got impressive results with minimal fine-tuning for 40 training steps. We can safely expect a substantial increase when fine-tuned on larger split of datasets for a larger number of training steps.
F.2 Full Pipeline Evaluations
Comments on Previous Experiments
Previously, we designed (task-u, task-O) and (task-1, task-2) to evaluate object level and physical state level choosing capabilities, respectively. Here we ensured that all other factons were kept ideal while evalaatng a specific factor. For example, while evaluating object-level selection capabilities, we didu't specify the physical state and instructed the model to assume them in perfectly usable conditions. Whereas, while evaluating physical state level-hased selection capabilities, we only provided the context (best suitable) objects corresponding to the <utility, task combination, thus eliminating emrrrs arising from selecting the wrong objects. These were designed to evaluate object-level reasoning and physical state-level reasoning abilities of LLMs **indiiduullly*.
To evaluate the performance of language models when tasked to employ both these reasoning abilities (object level and physical state level), we designed new datasets consisting of options where either the object could be inappropriate, the physical state could be inappropriate, both.
36

---

Published in Transactions Machine Learning Research (May /2024)
Physical Level Accuracy
Model
Obj. Level Accuracy
2-opt
3-opt
4-opt
5-opt
Model
2-opt
3-opt
4-opt
5-opt
t1
t2
t1
t2
t1
t2
t1
t2
PaLM
90.0
74.2
68.3
65.0
PaLM
92.5 86.9 88.7 75.2 80.4 66.9 71.3 60.4
PaLM FT
91.5
75.6
70.9
68.2
PaLM FT
99.2 88.8 94.9 79.3 92.5 72.6 89.3 66.0
Table 10: 1 compare the average
Table 9: [Task-O]Her) we compare the aver- age accuracy of the PaLM language model be- fore and after fine-tuning when evaluated on various types of fixed option-court datasets for task-O (as in Table 4 We can observe a substantial increase in accuracy for task-O performance even by fine-tuning on such a small subset of our data (400 examples for just 40 training _steiiggs_sess
curacy of the PaLM language model before and after fine-tuning when evaluated on 500 questions of each variation of task-1 and task-2 dataset. (from Table fffrom various types fixed-count datasets. For each value, we averaged the accuracy of all variations that were a part of that fixed count dataset. We observed substantial increase in accuracy for task-1 and task-2 small subset
performances even after line-tuning of data (1200 examples with task- bask-1:task-2 :task-2 ratio was 3:5) for just 40 training steps.
F.2.1 Fullideal Dataset
In this dataset, the correct answer would be the ideal configuration of the context object. Meanwhile, the other present options could include sub-optimal configurations of context objects, any configurations (ideal, sub-optimal) of utility, and any unrelated random object. We created around 30 variations of 15K QnA Pains with varying option counts and ratios of different object tupistut context, random).
Example Question of Fullideal Dataset
Question ID: 1,
Utility: hsauiens
Question: which of the following objects would be best suited for purpose of "heating Options:
(A) object name: Toaster, mass: medium, temperature: RoomTemp, material: Metal,
already use: reversible-using, condition: dirty
(B) object name:
CoffeeMachine, mass: heavy, temperature: RoonTemp, material: Metal
already in use: free, condition: clean
(C) object CoffeeMachine, already
heavy, temperature: RoomTemp, material: Metal irreversible-using condition: clean
(D) object name:
ButterKnife, mass: light, temperature: RoonTemp, material: Metal, already in use: free, condition: clean
Correct Answer: B
F.2.2
Fullmoderate Dataset
This datasel consisted of variations 15K QuA pairs with varying option counts and various ratios of object types (utility, context, random). Here, the correct answer would be the most appropriate moderate configuration of the context object. The other options could include
37

---

Published in Transactions Machine Learning Research (May /2024)
context abjects(worss moderate and bad configurations); any configuration of objects compatible with the question's utility, and unrelated random objocts (idcal, modcrate, bad).
Example Question of noderate Dataset
noderate
Question ID: 1,
Utility: isouren:
Question: Which of the following objects would be best suited for the purpose of "heating(source) "?
Options:
(A) object name:
Toaster, mass: mediun, temperature: RoomTemp, material: Metal,
already in use: reversible-using, condition: dirty
(B) object name:
CoffeeMachine, mass: heavy, temperature: RoonTemp, material: Metal already use: reversible-using, condition: clean
(C) object name:
CoffeeMachine, mass: heavy, temperature: RoonTemp, material: Metal already in use: irreversible-using, condition: clean
object ButterKnife, light, temperature: RoonTemp, material: Metal, already free, condition: clean
Correct Answer: R
F.2.3
Observations
PaLM Av. Accuracy
Dataset
2 2-opt
3-opt
4-opt
5-opt
Ideal
95.67
91.7
86.83
83.04
Moderate
77.83
62.08
55.03
48.68
Table 11: Average accuracy for single prompt evaluations in PaLM across variations of Full QA Dataset, i.e., accuracy averaged across various dataset variations for each fixed count dataset. The impressive performance of PaLM on Fideal dataset is anomaly; also saw its prowess in figuring out Ideal configurations of appropriate context objects even in Task-1 5 Further, we also observed in Task how all language models (including PaLM) suffered when they were tasked to figure out suitable sub-optimal configurations. The same poor performance witnessed once through PaLM's accuracy on F moderate datasct.
38

---

Published in Transactions Machine Learning Research (May /2024)
Figure 11: Variation Level anal- ysis of PaLM model's accuracy across variations of Full QA Dataset. For each dataset, con- text object was set as the correct answer. Here (Context, Utility, Random) denote the count of each such object amongst the sampled options for each dataset variation.
Ideal Moderate
80
Accuraty (h)
60
50
30
Rusdion
2-0pt
3-Opt
4-Opt
5-Opt
Figure 11 plots the trend in accuracy we vary the count of <utility+random> objects in the dataset options while increasing the context objects, thus increasing the level of difficulty. As expected (owing to the impressive utility level pruning capabilities), the peak accuracy for each fixed option count dataset occurs at maximum random objects. However, the worst accuracy was obtained when all the objects in the options were set as context options. We also observe improvement in accuracy whenever we increase the random object count concept object count, supporting previous conclusion of commendable object-utility mappings in language models like PaLM. In addition, PaLM's performance on the Ideal dataset is superior to its performance on the Moderate Dataset (just like we saw in task-1 and task-2 previously). Also, the observation of poor performance on noderrat dataset when all context objects are present aligned with our previous observations of our Task-2 ablations. (finding a suitable sub-optimal configuration amongst various sub-optimal configurations of context objects 6 Here, we could notice that each fixed count dataset is marked by constant trend of accuracy drop whenever move towards incrcasing the context objects from left to right.
F.3 Modular Setup
Owing to the below par performance of the PaLM language model Fodderate dataset, we experi- mented with modular approach of breaking the question down into 2 levels as introduced in this work; Object Level and Physical State Level. This method consists of 2 parts:
1. Object Selector: We slice out the object names of the options and pass them as separate question to the LLM. From here, we expect a list of (reeetsrr we could have multiple options consisting of configurations of context objects?) appropriate for the given <Utility, Task> combination.
2. Selecting Physical States of Selected Objects: On the basis of the object names received from stage-1, we slice out the options whose name belongs to that list and again call an LLM and ask it to analyze which option amongst those has a configuration that would be most suitable for the given <Utility, Task combination.
"To evaluate the merits of such technique, we improvement, shown in Figure I and Table
Froderate dataset (as it had wide margin for
39

---

Published in Transactions Machine Learning Research (May /2024)
PaLM Av. Accuracy F noderate
Prompt
2-opt
3-opt
4-opt
5-opt
Single
77.83
62.08
55.03
48.68
Modular
81.96
70.11
61.52
54.37
Table 12: Average accuracy for single prompt and modular prompt evaluations in PaLM across variations of moderate Dataset. Here, a single prompt means providing all option configurations a single prompt the language model for evaluation. We notice the increase performance when we switch a modular prompt regime across all fixed object count variations.
-i-single pronpt
5-Opi
bo-iiiiiprooapp -0-moodude
4-0pt
375
e50
1LA
Figure 12: Comparative performance of single prompt method and modular prompt method implemented using PaLM and evaluated on 5- option variation of F moderate Dataset
Figure 13: Comparative performance of single prompt method and modular prompt method implemented using PaLM and evaluated 4- option variation of F moderate Datasct
single prompt
modular
2-Opt
pronot
-t-modskar
97.6
Context
Uiility
100
68.0
70.2
80.3
Figure 14: Comparative performance of single prompt method and modular prompt method implemented using PaLM and evaluated 3- option variation of F mederate Dataset
Figure 15: Comparative performance of single prompt method and modular prompt method implemented using PaLM and evaluated on 2- option variation of F Fnoderate Dataset.

---

Published in Transactions Machine Learning Research (May /2024)
F.4 Observations
and 15 nightlight the improvement in performance across all variations except few. The increase in the average accuracy for each type of fixed count dataset adds substance to our work's i argument of breaking an object selection task into 2 broad phases (object selection and physical state selection). While the performance of PaLM was enhanced across nearly all variations, we also witnessed a few cases where there was a drop in accuracy(or no improvement). This was observed in datasets where only context objects were present in options, and thus, for such variations, modular categorization couldn't 't lead to performance gain. This is not a new observation; we had previously seen similar cases in single prompt technique (where evaluating such variations led to poor performance [See Orange plot in Figure 11 and task-2 evaluations. However,
Figure 12 13 14
interesting thing observed when analyzing object selector I.MM's responses was the confusion and random behavior it sometimes exhibited when tasked to output list of correct objects in the presence of more than 1 correct (context objects). For example, in the case of Fideal and maderata experiments shown in Figure II or like even in task-1 or task-2 dataset questions all these tasks could have options which contain multiple context objects corresponding
that <Utility, Task> combination. Thus, to tackle such cases, we must always prompt our object-level selector LLM to output all the objects it considers appropriate for the given <Utility, Task> combination. Due to the random behavior of language models, observed cases where one more appropriate objects were discarded in the object-level stage. This often led to cases where the most appropriate sub-optimal configuration was discarded by rejecting the object name. (and thus its physical configuration couldn't be compared)
F.5
Future Work
The next steps in this research direction would be fine-tuning the physical state selector LLM with task-1 and task-2 datasets to enhance its capabilities in judging object affordance given the physical state variables. In addition, fine-tuning the object level selector LLM with task-0 dataset and a multiple correct MCQ QuA (with outputs as lists of correct object names) would allow the object level responses to be more human behavior aligned. This would reduce the number of cases in the modular approach where the pipeline failed due to the failure of the object-level selector LLM. Future works would be aimed at comparing the modular approach consisting of such finetuned LLMs with single prompting method and modular approach employing off-the-shelf LLMs.
41


Reviews:
[{"review_id": "2wnslmaXb1", "reviewer": "Reviewer_RSFx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* This paper is well motivated. I believe the question they  study is a practical and important task for embodied agents.\n* The proposed decision-making framework  seems logical and resembles human's decision process.\n* The writing is clear and easy to follow. \n\nWealnesses\n* I don't quite understand Figure 5. What's on the y axis? Similar to Figure 6.\n* More analysis could be given in the result section. What other observations do the authors have besides the accuracy? Is there a pattern they see when different language models fail? Is there a reason that why PaLM performs much better than GPT3.5 with Task 1 and Task 2?\n* If they give more in-depth analysis of the results, we might draw more useful conclusions or insights about how future work could improve the household tasks addressed in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new framework that aims to analyze common LLM's reasoning ability when comes to selecting objects for household tasks.The authors decide to format this decision process into three phases, each pruning out options based on utility, context and physical state. To quantitatively investigate LLM's performance on these household tasks, they come up with a Utility-object mapping by using GPT3.5 and human annotations. Then they create a dataset with QnA questions with progressive difficulty aiming to stress test LLM's reasoning ability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall this paper is well-motivated, and propose useful framework and datasets for future work to study the same question. It would be great if the authors could give more insightful analysis in the result section.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3Ptrx0aiuc", "reviewer": "Reviewer_i1W5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n- Problem choice is important. Multi-step reasoning and physical reasoning are both things LLMs struggle with. The class of approach, a dataset and task, also makes sense - it has been clear for a long time, but seems to become increasingly clear, that the dataset is one of the most important factors in building large models. \n- The dataset is well-designed. Given the motivating principles, it executes them very well in a way that's simple, extensible, and applicable to other domains. If training on this dataset led to benefits for LLMs, it would be a good point for expansion into more objects and settings, as well as a good template for building more datasets in difficult domains. \n- The Task/Utility+Object-Utility-Configuration framework, plus the Ideal-Moderate-Bad configuration split, leads to rich and complex problems with opportunity to perform important ablations, from a compact set of factors. \n- Physical state breakdown is intuitive\n- Task construction is strong and well-explained. Like I said above, there are clearly lots of operations for testing different aspects of physical reasoning in a methodical way with this setup. The sub-optimal configuration dataset is especially interesting.\n- Paper is overall very well-written clear. Paper is easy to understand (in a good way!) Only exception is the conclusion - it's one big paragraph of text right now, would be better to break it up. The writing itself is good. \n- Figures could use more annotation but overall are well-presented\n- Experimental section backs all of the claims made in the conclusion, and is good to see. It's helpful to know where LLMs are right now, and this experiment set is thorough. It's also clear that there is plenty of margin for improvement on such a task. \n\n#### Weaknesses\n**Motivation**\n- The intro is predicated on the idea that humans reason in the exact way the paper lays out: first, they look for object utility (as defined and populated by this paper), then they look at contextual appropriateness (a wide, vague consideration, and also as defined by this paper), then they look at physical state. The boundaries between each of these three are not clear either. Overall, I find this breakdown very unintuitive and unnatural: for example, the clean knife case. What if there was a dirty knife? Where would that be eliminated - in the very broad \"contextual appropriateness\" phase? And the scissor indeed has a \"cutting\" utility, but is a scissor actually meaningfully capable of the type of cutting involved in cutting a cake, any more than a stick is? I realize that the real world is physically and semantically far more complex than any realistic set of systematically collected annotations can hope to capture, but 1) that doesn't mean gross simplifications work and 2) the entire intro is predicated on this being a human-like method of reasoning. \n- The paragraph \"to illustrate... current physical state\" is extremely presumptuous about how humans reason. I genuinely cannot relate to this type of reasoning (as demonstrated in my scissor analysis above), and while I'm one person, there doesn't seem to be cognitive science literature backing any of these specifics - the only psych/cogsci references are related only at a high level and do not discuss such a reasoning framework in meaningful detail. This then makes it difficult to take the motivation seriously. \n- I recognize that the Li et al. 2023 result talks about reasoning over physical variables being unhumanlike, and the paper asserts that the abstraction of the physical variables to its configuration framework is meant to make these considerations more humanlike, but we don't have proof that it *is* humanlike.\n\n**Dataset creation**\n- The contextual appropriateness consideration makes sense at a high level but is difficult to understand concretely. The paper says it \"engaged human annotators in a study designed to assess the selection of appropriate objects for specified tasks and utilities\" - this needs far more explanation. Examples, frameworks, what the humans were asked, etc. This is a very vague consideration that overlaps (in my mind, at least) with both utility and physical state, and it would help to understand it better.\n\n**Experiments**\n- Figures, though nice looking and readable, would benefit from being annotated with numbers (at least axes). The tables are there, but the axes labels should be there as well - that would be a better way to deliver the message.\n- Ultimately, this paper does not show us that training on this dataset improves LLMs. We need that result for it to be meaningful, since there is no theoretical backing. The human motivation wouldn't really help even if it were warranted, because we are trying to improve LLMs' QA performance and not their efficacy as models of human psychology. That experiment needs to happen. Of course this is a big ask, but it is what it is. Right now, the paper is an idea - a fleshed out one with all the tools ready to go, but not a proven one (and not one that is well-motivated another way).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents the CommonSense Object Affordance Task. This is a multiple-choice QA dataset that presents a task that needs an object with a certain utility to be completed, then asks what object or (object, physical state configuration) should be selected out of two to five options. The paper presents three considerations for such reasoning: the object's inherent utility, the contextual dependencies/appropriateness, and the current physical state, and designs its tasks accordingly. It also argues that humans use these considerations to make such judgments themselves. This is the primary motivation for the paper. The contributions are human annotations on COAT, the datasets themselves, and LLM baselines.\n\nThe paper goes through the creation of datasets, specifically how questions and answer options are generated. There are three tasks: one for eliminating objects based on contextual appropriateness, two for eliminating objects based on physical state configurations. Each dataset has thousands of questions. The Object-level dataset presents objects without the required utility, objects with the required utility but contextual inappropriateness, and objects with the required utility and contextual appropriateness. There are two physical configuration level datasets: one that has ideally-configured objects among the options, and one that has only moderately well-configured objects that would work but not ideally, and badly-configured objects that would not work. These present various challenges. \n\nThe experimental setup shows performance of 6-8 pretrained LLMs on answering MCQs from each of these datasets. The findings are that large models perform best, having more choices is generally harder, having a larger diversity of objects (not just physical configurations) is harder but bigger models are more robust to it, and having a higher percentage of bad configurations among the options leads to more frequent selection of bad configurations even when moderate configurations are always present and will be better.\n\nThe paper concludes with summarizing the results and suggesting that this data may be an avenue for improving the language model's abstract multi-step reasoning for estimating the physical affordance of everyday objects used in household activities.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Motivation: either find cognitive science literature that actually shows that humans use this type of reasoning (and ideally that it benefits them, but that's optional), or source motivation from somewhere else. In my opinion, it's okay to simply say this is a design that makes sense *if that design works* (which we don't know). Ultimately the paper needs results that show that training on COAT improves LLM performance, but even if there were something that could motivate this paper in the absence of these results, a claimed but unwarranted connection to human psychology doesn't.\n- Tone down some of the other claims in the paper. For example, \"extremely low-level utilities that could be assigned to a high level task\" - the example given is \"task: cutting cake\", and \"utility: cutting\". I disagree that the task is much higher level than the utility in this case, and probably in a lot of cases (though there are certainly some where it is, such as helping a person with the shower task vs. cleaning utillity) \n- Give more examples of the considerations, especially the contextual awareness one.\n- Show examples of questions that LLMs actually succeeded/failed on, not just examples that illustrate the concept. It's not surprising that Bad and Ideal are separable, but it's interesting that Moderate is separable from them (and the Bad Rate result shows that it's not totally separable from Bad, so it would be nice to see examples to build our intuition of what, exactly, LLMs succeed and fail on) \n- Add experiments training or fine-tuning LLMs on these tasks.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LyJWijuNpu", "reviewer": "Reviewer_WAN4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* To my knowledge, this is a new quantitative evaluation of LLMs that seems broadly relevant to deploying LLMs in household scenarios. Bluntly, if an LLM performed poorly on these tasks (as the current results show), it would likely lack a robust understanding of (human-centric) object affordances, and likely be unfit for any automated decision-making related to these tasks.\n* The evaluation is multiple-choice question answering in every-day household settings, so it is reasonable to expect LLMs that claim to have broad general-purpose “knowledge” to exhibit reasonable performance on these tasks without fine-tuning (which the paper does not perform).\n* I think the community would find the current middling performance of LLMs on these tasks interesting and motivating for future work.\n\n\n# Weaknesses\n* The evaluation appears to combine the “utility” and “context” phases. The paper needs justification either for why the phases are separated in Fig. 1, or why only the “context” phase is evaluated. E.g., “example task 0 variation 1” includes the “context” object “coffee”, whereas, based on my understanding of the “utility” phase, I’d expect an evaluation that does not include the context object (“coffee”) to be present somewhere, but it appears that it is not. If no context-free phase is evaluated, then I’d suggest changing the 3-phase modularization to 2-phase”.\n* The evaluation appears to only test the modules for “context” and “physical state level”, instead of the entire 3-phase pipeline end-to-end (i.e., by using the outputs of each phase module as inputs to the next phase). This is a serious shortcoming that undermines the presence of the 3-phrase decomposition / modularization. Why does the paper introduce this modularization if it is not evaluated?\n* Page 3: Unclear at this point in the paper what a “concept” is in “Object-Concept-Task mappings” \n* Ideally, Fig. 1 should make the inputs to the ‘utility’ and ‘context’ stages clearer, partly to help justify separating these stages. I’d expect the inputs to the ‘context’ stage include some additional ‘context’ that is not available at the ‘utility’ stage, e.g., the first phase does not observe that the object to be cut is a “cake” but the second phase does observe this.\n* The name “utility” is ambiguous because it’s also associated with the concept of “maximizing utility, since the intended output of the entire 3 phase decision-making process is the utility-maximizing object. Using the word “function” instead of “utility” would resolve this, as the first phase filters objects by their function, e.g., to objects that perform “cutting”. At the least, the text and figures that accompany the introduction of the name “utility” should clarify to readers this potential confusion.\n* The utility annotation process is unclear (Appendix D and S 2.1.1). It’s not clear how  GPT3.5-turbo is used during annotation. D.1 states “Creation of utility-object mappings … involved the use of GPT3.5-Turbo”, yet only describes how the “annotators” (presumably all human) were involved. The paper needs to be more direct and specific about how GPT3.5-Turbo is used during annotation.\n* S2.1.3, the acronym “LLM” is typically used to abbreviate “Large Language Model”, yet here it’s used to abbreviate “Language Learning Model”. This also conflicts with its usage on Page 2 (“large language models (LLMs)”)\n* Fig. 5, 6 need y-axis labels.\n* Fig. 7 has “Bad Accuracy” as the y-axis label, yet this was not introduced in the text. Is this the same as “Bad Rate”? The paper should clarify. \n* The Fig. 7 caption should clarify that the dotted lines indicate the ratios of bad configurations to all configurations in each of the evaluations.\n* The text refers to a Fig. 8, but there exists no Figure with a Figure 8 caption. I think the Fig. preceding S.4 needs a Figure number.\n* The paper states “we generate human preference mappings and curate 2 QA consisting of 130K MCQ question-answer pairs” and the main contributions state “Introduction of three major novel CommonSense-based QA Datasets”. These statements seem inconsistent (2 != 3). The paper should clarify.\n* The Dataset summary number of questions sums to 143.1K (= 68.9K + 58.7K + 15.5K), which is inconsistent with the statement  “we generate human preference mappings and curate 2 QA consisting of 130K MCQ question-answer pairs” (143.1K != 130K).\n* The dataset is missing some description of the nature of the annotators, including how they were recruited and how they were incentivised to label.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an evaluation of language models according to how well they can perform question answering tasks of household object affordances. The paper introduces several datasets of Question Answering collected from human annotators. The datasets measure performance on three different tasks: “object-level pruning”, “variable level pruning task 1” (with ideal configuration present in the options), and “variable level pruning task 2” (with no ideal object configuration present in the options). Several LLMs are evaluated on variants of each task, which results in a set of performance comparisons between each. Broad trends of relative performance are visible in the results, as well as absolute performance trends that illustrate that current LLMs do not perform very well on these tasks, especially task 1 and 2.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses for a list of proposed adjustments. Of those, the critical weaknesses are the absence of the evaluation of the full proposed pipeline, and the missing eval of one of the proposed phases.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There were mixed opinions among the reviewers, but having taken a closer look, the AE is satisfied that the work holds value, even if the manuscript lacks some polish, and leaves open some obvious room for improvement, such as training on these datasets to improve LLM reasoning abilities (even though within this work, the datasets are mainly curated for evaluating pre-trained LLMs rather than for improving them). 

The AE has suggested minor revisions above: namely, improve the introduction to tone down claims about human decision making that are not established in the appropriate literature, and incorporate a brief description of the findings from the newly added appendix to motivate the experiment design. Besides this, a thorough grammar and typo check is also recommended.
Decision: Accept (poster)