Venue: NAACL 2024

# _MacGyver:_ Are Large Language Models Creative Problem Solvers?

 Yufei Tian\({}^{1}\)Abhilasha Ravichander\({}^{2}\)Lianhui Qin\({}^{24}\)Ronan Le Bras\({}^{2}\)Raja Marjieh\({}^{3}\)Nanyun Peng\({}^{1}\)Yejin Choi\({}^{25}\)Thomas L. Griffiths\({}^{3}\)Faeze Brahman\({}^{25}\)

Work was done during Yufei's internship at AI2. Code and data available at: [https://github.com/allenai/MacGyver](https://github.com/allenai/MacGyver)

###### Abstract

We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. To this end, we create MacGyver, an automatically generated dataset consisting of over 1,600 real-world problems deliberately designed to trigger _innovative usage of objects_ and necessitate _out-of-the-box thinking_. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. MacGyver is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions. Finally, we provide a detailed error analysis of LLMs, and demonstrate the potential of enhancing their problem-solving ability with novel prompting techniques such as iterative step-wise reflection and divergent-convergent thinking.

This work **(1)** introduces a fresh arena for intelligent agents focusing on intricate aspects of physical reasoning, planning, and unconventional thinking, which supplements the existing spectrum of machine intelligence; and **(2)** provides insight into the constrained problem-solving capabilities of both humans and AI.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5303–5324

June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Creativity has long been considered the driving force behind modern civilization, and one of the hallmarks of human intelligence Guillord (1967); Hennessey (1995). As large language models (LLMs) have become increasingly powerful, researchers have begun to investigate their reasoning ability in problem-solving tasks Yao et al. (2022); Brahman et al. (2023) and their capacity for creativity as demonstrated by expressing humor and generating artistic content Mittal et al. (2022); Hessel et al. (2023); Ramesh et al. (2022); Chakrabarty et al. (2022); Tian et al. (2023). However, everyday activities that involve creative thinking have not been studied to the same extent. In this work, we contribute a benchmark for creative problem solving, hoping to critically assess modern LLMs when it comes to 'thinking out-of-the-box'.

To bridge this gap, we curate MacGyver, _a novel unconventional problem-solving dataset_ consisting of 1,683 sets of verbal problems that require human-like creativity in the realm of physical reasoning. Drawing inspiration from the cognitive science literature Duncker and Lees (1945), we collect problem scenarios that deliberately push

Figure 1: Examples of the problems in our MacGyver dataset with the GPT-4 and human answers (continued in Figure 11). Pictures, drawn by DALL-E 3, are solely for illustration purposes and may not accurately reflect the text. In our experiment, all inputs to human and LLMs are natural language texts.



against _functional fixedness_--a cognitive bias that limits an agent from employing familiar tools in innovative ways. Notably, leveraging the _generative_ strength of LLMs and the _verification_ strength of humans, we design a novel and labor-efficient pipeline to collect progressively more challenging scenarios (SS2). These scenarios are verified by humans as requiring unconventional usage of objects to find a solution. For example, solving problem (a) in Figure 1 requires using the wine bottle as a makeshift rolling pin.1 Each problem in our dataset is paired with at least one human-provided or verified solution. To the best of our knowledge, MacGyver is the first dataset of unconventional everyday problems requiring two key elements of creativity Guilford (1967a): _divergent_ thinking (to come up with creative or unconventional usage of objects) and _convergent_ thinking (to accomplish a goal efficiently).

Footnote 1: If the problem is unsolvable given the presented tools and constraints (problem b in Figure 1), we expect the agent to identify such infeasibility and provide a short justification.

Next, we use the resulting dataset as a _benchmark_ to evaluate the creative problem-solving abilities of both human participants and recent LLMs, including GPT-3.5, GPT-4, PaLM2, Claude2, and Llama2 OpenAI (2022, 2023; Anil et al., 2023; Touvron et al., 2023; Anthropic, 2023). Our results in SS4 reveal a substantial gap between most LMs and human. While the best performing LM, GPT-4, complements the capability of an arbitrary human under certain domain-specific settings (_e.g., fixing a hole on the wall_), humans' collective wisdom is so far still invincible. Additionally, LLMs struggle to identify unsolvable problems and either exhibit misleading helpfulness or are ultraconservative in inappropriate cases. In SS5, we present detailed comparison between human and machine.

Finally, a qualitative analysis of LLM responses reveals two common _failure modes_: (1) models propose physically infeasible, unnecessary, or wrong solution steps that deviate from the intended goal, and 2) models hallucinate unavailable tools or do not adhere to constraints specified. We propose two _prompting strategies_ to mitigate these common error types: (1) a self-reflection based strategy to iteratively verify the feasibility of each generated step and then modify as necessary, and 2) a cognitive-science-inspired strategy of first diverently exploring the potential use of presented tools and then converging on the problem solution. Experimental results show the efficacy of both strategies in boosting models performance (SS6). We hope MacGyver will serve as a useful resource for

* Evaluating LLMs and autonomous agents in new challenges involving real-world scenarios, innovative object usage, and physically feasible actions;
* Enhancing LLMs' creativity and physical-related reasoning skills; and
* Providing useful insight and resources to researchers in other fields such as computational cognition and psychology

## 2MacGyver Dataset

LLMs have demonstrated utility for idea generation Girotra et al. (2023). Therefore, instead of asking humans to come up with thousands of constrained scenarios from scratch, we design a progressive refinement pipeline to explore LLMs' potential to generate problem settings quickly and at scale (SS2.1). Human annotators then verify that each problem is concrete and requires creativity (SS2.2). Each instance in our dataset includes a constrained problem setting paired with at least one human-provided or verified solution (SS2.2, SSC.2).

### Progressive Problem Refinement for Dataset Creation

Figure 2 provides an illustration of our problem collection pipeline, showing how we combine human and machine inputs. Specifically, we propose a progressive problem refinement approach that gradually increases problem complexity by 1) adding specific object properties (_e.g.,_ material, size, etc.) as constraints to eliminate a previous solution and 2) adding distracting objects that are not involved in the solution. From a cognitive perspective on problem-solving Knoblock (1991), the first refinement step removes the most straightforward solution path, while the second step further complicates the problem by adding branches to the search space.

We implement this pipeline through a dialogue interaction with GPT-4. Human assessment results (detailed in appendix C.3) confirm that both steps within the progressive refinement approach pose additional challenges to LLMs, and after the two iterations, the original problem requires more creativity and becomes more challenging 



### Human Verification Process

After the refinement process, we involve human verifiers to judge if the final versions of the problems **1)** are solvable, unsolvable, or need more clarification (_e.g.,_ the setup is vague, which will be discarded), and **2)** for those solvable, whether solving them efficiently requires creative thinking (_i.e.,_ using objects to achieve goals they were _not_ originally designed for --unconventional usage). Each problem is annotated by three human verifiers, with average inter-annotator agreement (IAA, measured by Cohen's Kappa) of 0.67 and 0.77 for tasks **1)** and **2)**, respectively. Finally, we pair each problem with a gold answer. For the solvable subset, it is a step-by-step feasible solution. For the unsolvable subset, it is an explanation why the stated goal cannot be achieved (detailed in SSC.2).

In total, we created 1,683 problems, with a detailed breakdown in Table 1. Of those, 78% are solvable and 22% are unsolvable. Another 7% of all problems were discarded after being annotated by at least one annotator to be ambiguous or contradictory. For solvable problems, 82% require using tools in an innovative or unconventional manner.

### Diversity Control and Check

Intuitively, we want to avoid generating multiple problems with familiar goals and constraints. In this section, we summarize our measures to ensure the collected problems are _diverse, comprehensive, and free of repetitive patterns_.

Diversity ControlWe hand-craft more than 50 tags of locations and activities, aiming to ensure that our data collection pipeline delves into a variety of topics. These predefined tags are integrated into the prompt that we used to query GPT-4 for problem curation at Iteration 1. The detailed list of all tags can be found in Table 6.

Diversity CheckAfter the final iteration, we parse the objects presented as tools among all generated problems. Intuitively, we consider two similar objects with different properties (_e.g., plastic knife_ and _metal knife_; eyeglasses_ and _magnifying glass_) to be different. In total, 3,800 unique tools were identified. We compute their frequency and use GPT-4 to analyze their affordances (Appendix Table 8; Figure 3). We found that _holding items_ and _covering_ are the top two types, followed by _tying or connecting_ and _cleaning_. The long tails in both illustrations signify a desirable level of diversity.2

Footnote 2: Refer to Appendix C.4 for more details such as the detailed list of all tags, the most frequent tools and their affordances, and the prompt used to analyze tool affordance.

## 3 Assessing the Task Difficulty

To gauge the challenge of our task posed to the most recent LLMs, we evaluate the zero-shot performance of GPT-4 [9]. Nevertheless, existing automatic evaluations fall short to assess

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Problem (All)** & **Solvable** & **Unsolvable** & **Total** \\ \hline Count & 1,306 & 377 & 1,683 \\ Percentage & 77.6\% & 22.4\% & 100\% \\ \hline \hline
**Problem** & & & \\
**(Solvable Subset)** & **Unconv.** & **Conv.** & **Total** \\ \hline Count & 1,073 & 233 & 1,306 \\ Percentage & 82.2\% & 17.8\% & 100.0\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the entire MacGyver dataset (top). Number of solvable problems that require unconventional use of tools (bottom).

Figure 2: Progressive problem refinement with GPT-4. Starting from a vanilla version (_i.e.,_ Iteration 1), we carefully design refinement steps that gradually increase the problem’s complexity by adding specific object properties as constraints to veto a previous solution (_i.e.,_ Iteration 2), and adding distracting objects that are (likely) not involved in the solution the problem (_i.e.,_ Iteration 3). After that, human verifiers judge the quality of refined problems.



the efficacy of a presented solution. Therefore, we recruit human annotators to evaluate the quality of the GPT-4's answers on the _entire_MacGyver.

Assessment Setup. For a solvable problem, human annotators are asked to judge if the presented solution is **1.1**_feasible and efficient3_, **1.2**_feasible yet inefficient_, or **1.3**_infeasible_. The machine-generated answer may also wrongly assume the problem is unsolvable and gives a wrong justification (**1.4**). For an unsolvable problem, they need to judge if the presented answer **2.1**_correctly identifies the problem as unsolvable_, and **2.2**_gives the right justification_. Similarly, the answer may also wrongly assume the problem is solvable and give a wrong solution (**2.3**).

Footnote 3: A solution is considered efficient if it has no redundant or unnecessary steps, and it is unlikely that the problem can be solved with less labor or using fewer steps.

GPT-4 Performance. We report the performance on the solvable and unsolvable subset in Figure 4. Our preliminary findings indicate that, **firstly**, LLMs as strong as GPT-4 still exhibit limitations in solving unconventional problems, with only 18.9% likelihood of providing an efficient solution, while 37.5% likelihood of providing an infeasible solution. Analysis in the later section (SS6) shows that one common mistake is it failing to realize the consequences of actions and tool affordances in the given context (_e.g.,_ proposing to use chopsticks to lift up the egg yolk). **Secondly**, GPT-4 displays overconfidence, often suggesting solutions to problems that are inherently unsolvable. This could be partially due to GPT-4 being trained with RLHF Ouyang et al. (2022), maximizing its helpfulness. Moreover, the model struggles to discern whether a problem description is sufficiently concrete for resolution or too ambiguous, necessitating additional context Liu et al. (2023).

## 4 Benchmarking Humans and LLMs

A natural follow-up question is how well modern LLMs perform on this task, as compared to humans. We thus evaluate the performance of several recent LLMs (_i.e.,_ PaLM2, Claude2, Llama2, GPT-3.5 and GPT-4) on a representative sample of the entire MacGyver dataset which contains 323 problems. In addition, we gauge the capability of average humans on the same set of tasks.

### Collecting Independent Human Responses

We assessed human capability by recruiting participants who are new to this task. To this end, independent solutions were collected from a pool of \(N=252\) UK participants on Prolific. We intentionally used a different platform and target population from those of the human evaluators (_i.e.,_ MTurk and US) to minimize any chances of overlap. For a given problem, participants indicated whether they believed the problem is solvable, unsolvable, or required further clarification. If solvable, they provided a step-by-step solution, and otherwise explained why the problem was unsolvable. Overall, we elicited an average of six responses per problem and each participant contribute to up to five different problems.

Figure 4: Left: Human-evaluated GPT-4 performance on all 1,306 problems from the MacGyver that humans think are **solvable**. Right: GPT-4 performance on all 377 problems that humans think are **unsolvable**. _Correct for the right reason_ means that the LLM correctly identifies the problem is unsolvable, and gives the right justification. _Correct for the wrong reason_ means that it correctly identifies the problem is unsolvable, but gives an incorrect justification.

Figure 3: Affordances of the presented tools in our MacGyver dataset and their frequency (and count). Note that one object may have multiple affordances (_e.g.,_ paddle boards can be used for boating, reaching high areas, and exercise).

 



### Collecting Machine Responses

We collected solutions from seven different LLMs using Nucleus sampling (Holtzman et al., 2020) and return the top one sequence (\(T\)=\(0.7\) and \(p\)=\(0.95\)). In the prompt, we instruct an LLM to either provide a feasible and efficient solution to a problem when it believes the problem is solvable, or otherwise a justification explaining why the given problem is unsolvable. To explore whether different sizes of the same model plays a role in its problem solving ability, we include three variations of Llama2 (_i.e._, -7b, -13b, -7bb), as well as two variants of GPT model family (_i.e._, gpt-3.5-turbo, gpt-4-0613).

Additional GPT-4 ResponsesFor a fair comparison with humans, we emulate the same setup in SS4.1 by obtaining multiple solutions per problem from a single LLM. Since exhaustive human evaluation is costly, we opted to elicit multiple solutions exclusively from the most capable LLM, GPT-4. Multiple manually-designed instructions are used to prompt GPT-4 in order to reduce repetition among separate sessions of API calls. More details can be found in Appendix D.1.

### Human Evaluation

Human annotators were asked to evaluate if a presented answer is correct by selecting one out of six fine-grained categories: **A (or B)** correctly giving a feasible and efficient (or less efficient) solution to a solvable problem; **C** correctly identifying an unsolvable problem and giving the right justification; **D** giving a partially incorrect answer; **E** giving a mostly or entirely wrong answer; and **F** failing to identify the correct solvability status.4

Footnote 4: Screenshots of the human evaluation interface can be found in Appendix Figure 23 and 24.

### Benchmark Results

We report the benchmark results in Figure 5. Category **A**, **B**, and **C** are the three aspects of correct responses, while the remaining **D**, **E**, and **F** are aspects of the wrong ones. At a glance, despite varying in their characteristics, all of the benchmarked LLMs lag behind the performance of humans.

#### 4.4.1 Performance with Single Effort

As is mentioned in SS4.2, only the top one response is collected for a LLM per problem. Hence, we first list the LLMs' performances with their _single best answers_ on left of Figure 5.For human participants, there is no single person who approached all problems. Therefore, to simulate _an arbitrary person's individual_ performance, we take a random response from each problem.

We see that most recent LLMs achieve a mere 35% to 42% chance of success. Although GPT-4 and Claude2 stand out among the tested LLMs, their best attempts still under-perform an arbitrary average person with total correct rate of 65.1% (sum of category **A**, **B** and **C**).

We observe that different families of LLMs exhibit dissimilar behaviors. For example, PaLM2 and GPT-4 are overly verbose and often suggest solutions to problems that are inherently unsolvable

Figure 5: **Left**: Benchmark results of seven LLMs and human with a single effort. For human participants, since there is no single participant who worked on all problems, we take a random response from each problem. We color-code the three categories indicating fine-grained aspects of correctness or falseness. **Right**: Comparison between GPT-4 and human where we evaluated multiple solutions per problem. The best performance, which can be viewed as an upper bound, is computed by taking the individual best answer (out of 6) for each problem. The actual numbers are reported in Table 5 in appendix A.2.



(as seen by their remarkably low performance in category **C**: correctly identify an unsolvable problem). In contrast, Llama2-7b, Claude2, and GPT-3.5 are more conservative and fail to realize a constrained problem can still be solvable (reflected in their high numbers in category **F**). Comparing the three variants of Llama2, we find that the larger models (13b, 70b) excel in correctly identifying solvability (category **F**). The smaller model (7b) is more subject to falsely recognizing a constrained problem as unsolvable. Beyond this, however, it appears that scale alone does not significantly unleash any creative problem-solving capabilities.

#### 4.4.2 Performance with Multiple Efforts

Recall that we collect multiple solutions per problem for GPT-4 and humans. With these, we compute the _average_ and _best_ performance. The best performance, which can be viewed as an upper bound, is computed by taking the individual best answer for each problem. The results are shown on the right of the same figure. In addition, we compute the _majority_ performance by considering a binary annotation (_i.e.,_ correct or wrong) of each problem. We find that the majority of humans are \(79.3\%\) correct, surpassing that of GPT-4 (\(73.3\%\)).

We see that on average human participants are slightly worse than GPT-4 in coming up with a correct solution (especially inefficient ones, category **B**), which is potentially owing to functional fixedness. In general, humans still out-perform GPT-4 due to the fact that GPT-4 seldom correctly identifies an unsolvable problem. Moreover, the best of the four human answers, which can be considered as a form of _collective wisdom_, clearly leads to a near perfect performance.

Finally, humans seem to struggle with certain problems (category **F**). We hypothesize that an individual person, who likely does not have domain-specific knowledge in all aspects of life, may not outperform a single LLM such as GPT-4, which is trained on massive amount of data and a wide variety of tasks. However, when considered collectively as a group, with each person contributing their unique expertise and wisdom, human intelligence exceeds that of LLMs. To verify our hypothesis and gain deeper insights into the relationship between the intelligence of humans and LLMs, we conduct further analyses in the next section.

## 5 Comparing GPT-4 with Humans

### Humans have higher variance than LLMs.

We plot the kernel density estimate (KDE) of individual human and GPT-4 responses in Figure 6. We can see that humans either approach a problem perfectly or fail totally. Namely, once humans understand the task and acquire the relevant knowledge, they can always propose a feasible and often the most efficient solution. On the contrary, GPT-4 responses fall more into the middle (mostly/partially wrong, or inefficient), owning to its ability to aggregate information from a wide range of sources it has been trained on. However, GPT-4 is sometimes ignorant of tool affordances or consequences of its proposed actions, lacking the depth of understanding that humans possess (see more detailed error analysis in SS6.1).

### Humans possess better general everyday knowledge, but less domain-specifically.

Next, we visualize the capability of humans and GPT-4 on individual problems in a 2D plot (Figure 7). Accordingly, we convert categorical labels into numerical scores ranging from 0 (Fail) to 1 (Perfect), and take the average score across solutions. We also plot the diagonal line: the farther away a point is from this, the larger the gap between human and GPT-4 performance.

We find that humans are better at solving tasks in categories likely to be familiar to them, such as _household_ and _personal life_. For those requiring domain-specific knowledge such as _gardening/farming/fishing_, GPT-4 performs better. The same holds when we manually inspect the outliers: those few problems that belongs to everyday categories yet humans are poor at. Unsurprisingly, they are problems such as demonstrating the concept of refraction without a prism (category: school), and

Figure 6: The kernel density estimate of individual human and GPT-4 answers.

 

making a sundial (category: beach), which an average person might have little experience with. Refer to SSA.1 for examples and other comparisons.

Overall, the different creative strengths of humans and AI systems suggests that the most effective solutions to tasks requiring thinking "out-of-the-box" might arise from a _collaborative approach_ leveraging the strengths of both parties.

## 6 Enhancing LLMs' Problem Solving

Here, we investigate whether different prompting strategies can enhance the problem-solving abilities of existing LLMs. In SS6.1, we conduct a detailed error analysis on GPT-4, showing it is weakest at identifying the correct tool affordance and physical feasibility. In SS6.2, we propose two new prompting strategies that effectively reduce its mistakes.

### Error Analysis for GPT-4

To better understand the limitations of LLMs and provide insight for potential improvement, we manually analyze 200 solutions generated by GPT-4 marked as infeasible by human annotators. We identified five common failure modes in Table 2.

We find that GPT-4 is highly prone to proposing _physically infeasible, unwanted, or wrong actions_. In Table 2, error type _(1) wrong tool usage_ accounts for \(\sim\)half of all the errors made (42.4%), followed by _(2) not achieving the goal_ (17.7%). It is crucial to highlight that _LLMs act in a fictional setting_, failing to realize the consequences of their proposed actions and the affordances of tools in the given unconventional context. While one can argue that LLMs lack direct interaction with the physical world, the human solvers similarly contemplate the same task purely in their minds, without any visual or physical cues. We also observe _two types of hallucination_: _(3) using unavailable tools_ and _(5) unfaithful to constraints_, which account for 16.9% + 9.5% = 26.4% of all the errors made.

### Improving LLMs via Prompting

The common error types in Table 2 motivates us to explore techniques to enhance LLMs' problem solving abilities. Specifically, we explore two prompting strategies as illustrated in Figure 8:

* **Iterative Step-Wise Reflection** : A self-reflection-based strategy. After the LLM generates an initial solution, we prompt it to _verify_ if each step is physically feasible and afforded. Subsequently, it modifies the original solution iteratively until no more modifications are needed.
* **Divergent-Convergent Thinking**: A cognitive-science-inspired strategy. The LLM is prompted to first enumerate the affordance of each object (_i.e.,_ divergent thinking) and conclude whether they are useful, followed by generating the steps

Figure 7: 2D visualization of human (x-axis) and GPT-4 (y-axis) performance on individual problems. Each dot represents a problem, with its color representing seven different categories. Humans are better at solving problems that they are familiar with (_e.g.,_ household), than those requiring domain-specific knowledge (_e.g.,_ gardening/farming/fishing).

 

towards the goal (_i.e.,_ convergent thinking).

We implement both prompting strategies with GPT-4, Claude2, and Llama2-13b on 180 randomly-sampled solvable problems that do not overlap with those used in SS6.1. The performance of the standard prompting and two proposed improvements for GPT-4 (and the remaining two LLMs) are shown in Figure 9 (and Appendix A.3).

For GPT-4, both proposed prompting methods contribute to a reduction in infeasible solutions. Intuitively, Self-Reflect, which is designed to verify the feasibility of steps, has a larger improvement in reducing infeasible solutions (9.7% vs 4.3% drop); while Div-Conv Thinking, which is designed for better preparation before generating the solution, is more helpful in generating efficient solutions (6.5% vs 2.2% gain). Comparing all three LLMs, Div-Conv Thinking is shown to be beneficial for all, both in terms of efficiency and feasibility, but Claude2 and Llama2's performances do not improve with Self-Reflect. Such a finding implies that, smaller models so far still lack the inherent ability to self-reflect and reason about physical consequences which GPT-4 is capable of.

## 7 Related Work

Creativity TheoryGulford (1967) defines a meaningful creative process as an interplay between spontaneous (divergent, to come up with novel ideas) and controlled (convergent, to satisfy the demand of the task) modes of thinking. Kaufman and Beghetto (2009) categorize human creative activities into four dimensions (Table 3), ranging from everyday innovation that ordinary people have knowledge of (_e.g.,_ removing wrinkles on a shirt without possession of an iron) to highly eminent innovation that few people engage with.

In the AI-related creativity community, everyday innovation which better reflects the activities that most people may engage in, is under-explored possibly due to the lack of a sizable dataset. For example, recent work Koivisto and Grassini (2023) study problems with four objects: rope, box, pencil, and candle. We bridge this gap by contributing a dataset with 1,600 everyday problems. collins2022structured In the AI-related creativity community, everyday innovation which better reflects the activities that most people may engage in, is under-explored possibly due to the lack of a sizable dataset. For example, recent work Koivisto and Grassini (2023) study problems with four objects: rope, box, pencil, and candle. We

Figure 8: Proposed prompting methods: iterative step-wise reflection (left), divergent-convergent thinking (right).

\begin{table}
\begin{tabular}{l l p{142.3pt}} Model & Description & Example \\ Mini-C & Developmental achievement in the learning process. & A pupil applying a strategy learned in a math class into her science project. \\ Little-C & Everyday innovation that ordinary people engage with, & Removing wrinkles on a shift without possession of an iron. \\ Pro-C & Professional expertise & Writing poems or stories that receive professional recognition. \\ Big-C & Legendary innovation that redirect a field. & Albert Einstein arriving at general relativity. \\ \hline \end{tabular}
\end{table}
Table 3: The Four-C model of creativity.

Figure 9: Results of different prompting strategies with GPT-4 in a zero-shot fashion: 1) vanilla prompting, 2) iterative step-wise reflection (self-reflect), and 3) divergent-convergent thinking (div-conv).

 

bridge this gap by contributing a dataset with 1,600 everyday problems.

Cognitive Bias_Functional fixedness_ is a cognitive bias limiting our ability to use familiar objects in novel ways. For example, struggling to see a chair as anything other than a seat exemplifies this. These biases subtly impact our daily decisions, often unconsciousiously. Over 82% of the solvable problems in MacGyver require using tools unconventionally to bypass such a bias. A similar work to ours Collins et al. (2022) explored LLMs' problem-solving ability in out-of-distribution reasoning tasks.

Machine Physical ReasoningPrevious research such as Hong et al. (2021) and Bakhtin et al. (2019) investigated physical reasoning in visual contexts. In the realm of language-based physical reasoning, prior studies primarily focused on understanding physical concepts and attributes of various objects, such as PROST Arroca-Ouellette et al. (2021), and NEWTON Wang et al. (2023). Relatedly, SWAG Zellers et al. (2018) introduced the task of grounded commonsense inference about physical situations. PIQA Bisk et al. (2020), which tests machines' physical commonsense reasoning ability is most similar. While proficiency in addressing problems in MacGyver involves all the above abilities, our emphasis extends beyond. We focus on unconventional tool usage, reasoning over the affordance of tools and ruling out unnecessary ones, and how individual objects can be used in combination to achieve a complex goal.

## 8 Discussion and Conclusion

Significance of WorkWe propose a new playground and the accompanying MacGyver dataset for creative problem solving, which covers a broad range of topics for _everyday innovation_, such as household, training, and outdoor sports, which is _orthogonal_ to the existing areas of reasoning and creativity, and adds to the spectrum of machine intelligence.

The area of daily innovation, or "little-c" according to the creativity theory (Table 3), is a _standalone type_ of creativity and better reflects the creative activities that normal people engage with, but is much less studied than math, logical reasoning, or writing problems. These so-called daily activities can be complex too, by involving multiple-step planning for efficiency, ruling out possibilities in a large search space, using multiple tools in an unconventional manner that even humans find difficult. Namely, solving these "daily activities" requires different kinds of creativity from scientific discovery, art, _etc._, and have a high potential for AI making people's daily life more enjoyable.

ConclusionWe present MacGyver, a novel benchmark focusing on everyday innovation that is carefully collected with quality and diversity control. We evaluate and compare both LLM and human performances, and highlight failure modes of LLMs in proposing physically feasible actions towards a goal. Nonetheless, we find LLM capabilities to be complementary to human capabilities under certain domain-specific settings. We propose two new prompting methods that effectively improve this reasoning ability in LLMs.

## 9 Future Opportunities

We hope MacGyver dataset opens the door to multiple future directions that will contribute to the broader goal of creating _AI systems that can intelligently and flexibly interact with their surroundings_. For example in this paper, we provide a preliminary attempt to improve the capability of LLMs via two prompting strategies. We encourage future investigation into planning and reasoning strategies to enhance LLMs with physical knowledge and spatial understanding, and to reduce hallucination. To further ameliorate the mistakes made by LLMs in a fictional setting, future work are encouraged to build embodied agents that can interact with physical or simulated worlds and receive feedback from the environment.

Finally, we encourage automatic evaluation methods for this complex reasoning task. For example, using LLMs to extract claims from the candidate solutions, and examine the physical feasibility (or predict the consequences) of proposed actions based on some physical world knowledge.

### Acknowledgements

This work was funded in part by DARPA MCS program through NIWC Pacific (N66001-19-2- 4031), the NOMIS Foundation, and the Allen Institute for AI. We thank Jena D. Hwang, Ilia Sucholutsky, Mosaic team members, and the anonymous reviewers for the helpful discussions 



### Limitations

Measuring how well a model can solve creative problems is hard due to the lack of standardized automated metrics. For example, assuming the availability of multiple references, popular automatic NLG metrics exhibit a weak correlation with human judgment, with Pearson correlation coefficients of 0.07 for BLEU-2/BLEU-3 Papineni et al. (2002) and 0.12 for BertScore Zhang et al. (2019). Our experiments thus rely on human evaluation process, which is relatively slow and costly. Therefore, new proposals for efficient and automatic evaluation framework for creative and sequential planning could be a compelling future direction. In addition to the MacGyver Dataset, we release human annotations for all the solutions tested in benchmarking. We hope these additional 4,100 answer-annotation pairs, containing a full gradient of correctness (completely wrong, partially correct, correct but less efficient, and perfect), will facilitate future works in automatic evaluation.

Another limitation of our study lies in the nature of our problems being generated by an LLM, GPT-4. Despite its strengths in exploring a unique and novel angle of problem-solving, it might also exhibit inherent biases and tendencies of the underlying model. Given GPT-4's predominant training on English-speaking data, we may inadvertently reflect the cultural nuances of North American and European contexts.

## References

* Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. _arXiv preprint arXiv:2305.10403_.
* Anthropic (2023) Anthropic. 2023. Model card and evaluations for claude models.
* Aroca-Ouellette et al. (2021) Stephane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann. 2021. Prost: Physical reasoning about objects through space and time. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 4597-4608.
* Bakhtin et al. (2019) Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019. Phyre: A new benchmark for physical reasoning. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In _Proceedings of the AAAI conference on artificial intelligence_, volume 34, pages 7432-7439.
* Brahman et al. (2023) Faeze Brahman, Chandra Bhagavatula, Valentina Pyattah, Jena D Hwang, Xiang Lorraine Li, Hirona J Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2023. Plasma: Making small language models better procedural knowledge models for (counterfactual) planning. _arXiv preprint arXiv:2305.19472_.
* Chakrabarty et al. (2022) Tuhin Chakrabarts, Vishakh Padmakumar, and He He. 2022. Help me write a poem: Instruction tuning as a vehicle for collaborative poetry writing. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 6848-6863, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Collins et al. (2022) Katherine M Collins, Catherine Wong, Jiahai Feng, Megan Wei, and Joshua B Tenenbaum. 2022. Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks. _arXiv preprint arXiv:2205.05718_.
* Dumcker and Lees (1945) Karl Dumcker and Lynne S Lees. 1945. On problem-solving. _Psychological monographs_, 58(5):i.
* Girotra et al. (2023) Karan Girotra, Lennart Meincke, Christian Terwiesch, and Karl T Ulrich. 2023. Ideas are dimes a dozen: Large language models for idea generation in innovation. _Available at SSRN 4526071_.
* Guilford (1967a) Joy P Guilford. 1967a. Creativity: Yesterday, today and tomorrow. _The Journal of Creative Behavior_, 1(1):3-14.
* Guilford (1967b) Joy Paul Guilford. 1967b. The nature of human intelligence.



Beth A Hennessey. 1995. Social, environmental, and developmental issues and creativity. Educational Psychology Review7, pp. 163-183. Cited by: SS2.
* J. Hessel, A. Marasovic, J. D. Hwang, L. Lee, J. Da, R. Zellers, R. Mankoff, and Y. Choi (2023)Do androids laugh at electric sheep? humor understanding benchmarks from the new yorker caption contest. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 688-714. External Links: Document, Link Cited by: SS2.
* A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi (2020)The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link Cited by: SS2.
* Y. Hong, L. Yi, J. B. Tenenbaum, A. Torralba, and C. Gan (2021)PTR: a benchmark for part-based conceptual, relational, and physical reasoning. CoRRabs/2112.05136. External Links: Link, 2112.05136 Cited by: SS2.
* J. C Kaufran and Ronald A Beghetto (2009)Beyond big and little: the four c model of creativity. Review of general psychology13 (1), pp. 1-12. External Links: ISSN 0003-691X, Link, Document Cited by: SS2.
* C. A Knoblock (1991)Search reduction in hierarchical problem solving. In AAAI, Vol. 91, pp. 686-691. External Links: Link Cited by: SS2.
* M. Koivisto and S. Grassini (2023)Best humans still outperform artificial intelligence in a creative divergent thinking task. Scientific Reports13 (1), pp. 13601. External Links: Link, Document Cited by: SS2.
* A. Liu, Z. Wu, J. Michael, A. Suhr, P. West, A. Koller, S. Swabzamdipta, N. Smith, and Y. Choi (2023)We're afraid language models aren't modeling ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 790-807. External Links: Link, Document Cited by: SS2.
* A. Mittal, Y. Tian, and N. Peng (2022)AmbiPun: generating humorous puns with ambiguous context.. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 1053-1062. External Links: Link, Document Cited by: SS2.
* O. Al. (2022)HotDucing chatgpt. External Links: Link Cited by: SS2.
* L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2022)Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pp. 27730-27744. External Links: Link, Document Cited by: SS2.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311-318. External Links: ISBN 978-1-4503-3213-1, Link, Document Cited by: SS2.
* A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen (2022)Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (1), pp. 3. External Links: Link, 2004.06125 Cited by: SS2.
* N. Reimers and I. Gurevych (2020)Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, External Links: Link, Document Cited by: SS2.
* Yufei Tian, Anjali Narayan-Chen, S. Oraby, A. Cervone, G. Sigurdsson, C. Tao, W. Zhao, T. Chung, J. Huang, and N. Peng (2023)Unsupervised melody-to-lyrics generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 9235-9254. External Links: Link, Document Cited by: SS2.
* H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashkykov, S. Batra, P. Bhagava, S. Bhosale, et al. (2023)LIMA 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: SS2.
* Y. Ru Wang, J. Duan, D. Fox, and S. Srinivasa (2023)Newton: are large language models capable of physical reasoning?. arXiv preprint arXiv:2310.07018. Cited by: SS2.
* S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2022)React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: SS2.
* R. Zellers, Y. Bisk, R. Schwartz, and Y. Choi (2018)SWAG: a large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 93-104. External Links: Link, Document Cited by: SS2.
* T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi (2019)BERTScore: evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

 



## Appendix A Additional Results

### Comparing GPT-4 with Humans

What percentage of individual humans outperform AI?Figure 10 compares human and machine by showing the number of problems (out of 323) that at least 20%, 40%, 60%, 80%, and 100% human participants (or GPT-4) answer correctly. Given the unique strengths and knowledge scores of different individuals, it is less likely that all human participants can answer the same problem correctly. However, there is a higher chance where at least 60% human participants know the answer.

Complementary capabilities of human and AI.Figure 11 presents two examples showing the complementary capabilities of human and AI in creative problem solving. In problem (c), human participants find a more efficient solution to heat the pizza than GPT-4. In problem (d) requiring domain knowledge gardening, humans fail to solve this highly-specialized task, whereas the LLM has equipped itself with such domain knowledge during massive pre-training.

What tools are human more proficient at?Recall that in SS5.2 we convert the categorical labels into numerical scores ranging from 0 (Fail) to 1 (Perfect) to conduct problem-wise analysis. Similarly, we conduct object-wise analysis by first parsing the tools presented in each problem, and then calculating the same numerical scores for each tool. Note that we opt to parse all the tools presented in the problem setting instead of those actually used in a proposed solution, because being able to reason about the potential usage of presented tools and conclude to _not_ use a possible tool is also an keystone towards intelligence. We identify several tools that humans and GPT-4 attempt most differently and report them in Table 4. For example, humans are more proficient at attempting magnifying glass, rocks, calculators, knifes, etc., whereas AIs are better attempting mirrors, gloves, and scarves. In general, there are more tools humans are proficient at.

### Benchmark Results

We report the benchmark results in Table 5. Category **A**, **B**, and **C** are the three aspects of correct responses, while the remaining **D**, **E**, and **F** are aspects of the wrong ones. At a glance, despite varying in their characteristics, all of the benchmarked LLMs lag behind the performance of humans.

### Enhancing LLMs' Problem Solving

Results with Claude2 and Llama2We report the performance of the standard, zero-shot prompting and two proposed improvements for Claude2 and Llama2-13b in Figure 12 and Figure 13.

Figure 11: Detailed examples showing the complementary capabilities of human and GPT-4. In problem (c), human participants find a more efficient solution to heat the pizza than AI. In problem (d), humans fail to solve this highly-specialized task to repel aphids, whereas the LLM has equipped itself with domain knowledge on gardening during massive pre-training.

Figure 10: Number of problems (out of 323) that at least 20%, 40%, 60%, 80%, 100% human participants (or GPT-4) answer correctly.



Different from GPT-4 (shown in Figure 9), the self-reflection strategy does not help any of these two models to reduce infeasible answers. When prompted to reflect on its previous answer, Llama2 always claims that its original answer is mistaken and attempts to correct itself blindly. We hypothesize that these two LLMs are weaker than GPT-4 and lack the inherent ability to faithfully conduct complicated physical reasoning. On the other hand, we see that Divergent-Convergent Thinking is beneficial for all LLMs across all dimensions.

## Appendix B The Four-C Creativity Model

Kaufman and Beghetto (2009) propose the Four-C model (Table 3), categorizing human creative activities into Mini-C: developmental creativity in the learning process, Little-C: everyday innovation that ordinary people have knowledge of and engage with (such as removing wrinkles on a shirt without possession of an iron) , Pro-C: professional expertise such as writing poems or painting artwork, and Big-C: highly eminent innovation that few people engage with.

## Appendix C More Information on the MacGyver Dataset

### Human Verification Process

After generating the challenging scenarios, we involve human verifiers to judge if the final versions of the problems **1)** are solvable (_i.e.,_ it is possible to find a reasonable solution using the presented tools), unsolvable, or need more clarification (_i.e.,_ the setup is vague or contradictory, which will be discarded), and **2)** for those solvable, whether solving them efficiently requires creative thinking (_i.e.,_ using objects to achieve goals they were _not_ originally designed for --unconventional usage). Each problem is annotated by three human verifiers from Amazon Mechanical Turk. The detailed verification interface can be found in Appendix D.4. The average inter-annotator agreement (IAA), measured by Cohen's Kappa, are 0.67 and 0.77 for tasks **1)** and **2)**, respectively.

### Collecting Gold Solutions

We provide more details on the final step of our data collection --to pair each problem with a gold answer. For the solvable subset, the answer is a feasible solution written step by step. For the unsolvable subset, the answer is a correct explanation for why the stated goal cannot be achieved.

To save human effort, we start by leveraging the generative strengths of a powerful LLM, _i.e.,_ GPT-4. Specifically, we first prompt GPT-4 to generate a solution for each problem in the MacGyver dataset. Then, human verifiers assess whether the

\begin{table}
\begin{tabular}{l l c} \hline \hline  & Object & \begin{tabular}{c} Human-AI \\ Difference \\ \end{tabular} \\ \hline A. human\textgreater{}AI & \begin{tabular}{c} magnifying glass \\ rock \\ calculator \\ kitchen knife \\ hair tie \\ paper cup \\ zip ties \\ pen \\ kettle \\ old t-shirt \\ sunscreen \\ \end{tabular} & \begin{tabular}{c} 0.602 \\ 0.447 \\ 0.386 \\ nadir \\ paper cup \\ zip ties \\ pen \\ delt+shirt \\ unscreen \\ \end{tabular} & \begin{tabular}{c} 0.4602 \\ 0.445 \\ 0.386 \\ nadir \\ paper cup \\ o.292 \\ o.283 \\ kettle \\ old t-shirt \\ sunscreen \\ \end{tabular} & \begin{tabular}{c} 0.447 \\ 0.405 \\ 0.386 \\ 0.359 \\ 0.292 \\ 0.281 \\ 0.252 \\ 0.255 \\ \end{tabular} \\ \hline B. human\textless{}AI & \begin{tabular}{c} mirror \\ gardening gloves \\ scarf \\ tablecth \\ clothespins \\ \end{tabular} & 
\begin{tabular}{c} -0.314 \\ -0.307 \\ -0.289 \\ -0.253 \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table}
Table 4: Tools that human are more proficient at leveraging or deciding to not leverage than AI (GPT-4 in our case), and vice versa.

Figure 12: Results of different prompting strategies with Claude2. We compare 1) vanilla prompting, 2) iterative step-wise reflection (reflect), and 3) divergent-convergent thinking (div-conv).

Figure 13: Results of different prompting strategies with Llama2-13b. We compare 1) vanilla prompting, 2) iterative step-wise reflection (reflect), and 3) divergent-convergent thinking (div-conv).

 

generated solutions are valid. Only if **all three verifiers** agree that a solution is valid, it becomes part of our dataset. Otherwise, we ask human workers to write down a solution (for solvable subset) or a justification (for unsolvable subset).

### Does the data collection pipeline result in progressively challenging problems?

To test whether our data creation pipeline (in Figure 2) is indeed iteratively posing challenge to a previous iteration, we collect GPT-4 answers to iteration 1, 2, and 3 of 200 problems, and run the same human evaluation process described in SS4.3.

GPT-4's performance on all three iterations of the same set of problems can be found in Table 7. As the problems get iteratively refined, the ratio of feasible and efficient solutions decrease, and the ratio of infeasible answers increase. This reflects that most potent LLM, GPT-4, indeed finds the problems increasingly challenging.

### Diversity Control

Tags used for Diversity ControlBefore the first iteration, we hand craft more than 50 tags of locations and activities, aiming to ensure that our data collection pipeline delves into a variety of topics. The tags cover diverse range of human activities, from indoor ones such as _home arrangement_ and _working in the office_, to outdoor ones such as _hiking_, _gardening_, and _playing with water_. These predefined tags are integrated into the prompt that we used to query GPT-4 for problem curation at Iteration 1. We list all the tags (_i.e.,_ locations and activities) used to curate the dataset in Table 6. They are introduced to prompt the LLM for diversity control, and can be broadly divided into Indoors/Household, Neutral, and Outdoors.

Generation in BatchAll problems are generated and refined in batches of 15 rather than one by one, as we find out the former results in significantly higher diversity. We then leverage a widely-used sentence transformer Reimers and Gurevych (2020) to filter out any newly generated problem that is semantically similar to the existing ones in our database.

Analyzing Tool AffordanceWe leverage GPT-4 to analyze the affordance of presented tools in the MacGyver dataset. Specifically, we start with a small set of hand-crafted affordance as seed. Despite being required to choose only from this fixed list of affordances, GPT-4 does not strictly follow our instruction, and sometimes returns new types that are not included in the seed list. We then gradually expand the list of affordances with newly generated ones.

For eliciting tool affordances, we use the prompt shown in Figure 14.

Commonly-presented tools and their frequenciesIn total, more than 3,800 different tools appear in our MacGyver dataset. We list in Table 8 16 commonly-presented tools, their featured affordances, and frequency. The number of unique tools and the long tails in distribution signify a desirable level of diversity.

\begin{table}
\begin{tabular}{l|c c c|c|c c c|c} \hline  & \multicolumn{4}{c|}{**Correct (\%)**} \\ \cline{2-9}  & **A. Eff.** & **B. Less** & **C. Unsalorable** & **Correct in Total (\(\uparrow\))** & **D. Partial** & **E. Mostly** & **F. Fail to Identify** & **Wrong in Total (\(\downarrow\))** \\ \hline \multirow{3}{*}{**Llama2-7b**} & 8.9 & 18.1 & 8.5 & 35.5 & 6.9 & 27 & 30.6 & 64.5 \\  & 11.7 & 28 & 2.3 & 42.0 & 12.1 & 32.3 & 13.6 & 58.0 \\  & 11.6 & 24 & 5.6 & 41.2 & 14.0 & 27.2 & 17.6 & 58.8 \\  & 14.7 & 25.9 & 0.0 & 40.6 & 10.8 & 35.5 & 13.1 & 59.4 \\
**Claudez 1** & 14.0 & 22.2 & 16.5 & 52.7 & 8.2 & 12.3 & 26.7 & 47.2 \\
**GPT-3.5** & 13.8 & 15.4 & 11.4 & 40.6 & 10.2 & 11.4 & 37.8 & 59.4 \\
**GPT-4 (Random)** & 24.8 & 35.5 & 2.1 & 62.4 & 11.9 & 14.9 & 10.8 & 37.6 \\
**Human (Random)** & 27.6 & 27.6 & 9.9 & **65.1** & 5.6 & 10.8 & 18.6 & **35.0** \\ \hline \multicolumn{9}{c}{Multiple efforts} \\ \hline Average GPT-4 & 24.8 & 33.2 & 5.0 & 63.0 & 12.5 & 15.7 & 8.7 & 36.9 \\ Average Human & 26.2 & 28.7 & 12.9 & 67.8 & 5.1 & 10.2 & 16.9 & 32.2 \\ \hline
**Best GPT-4** & 62.5 & 21.1 & 8.7 & 92.3 & 2.2 & 4.3 & 1.2 & 7.7 \\
**Best Human** & 72.8 & **15.2** & **10.8** & **98.8** & **0.6** & **0.6** & **0.0** & **1.2** \\ \hline \end{tabular}
\end{table}
Table 5: **Top**: Benchmark results of seven LLMs and human with a single effort. For human participants, there is no single participant who worked on all problems. So we take a random response from each problem. **Bottom**: Comparison between GPT-4 and human where we evaluated multiple solutions per problem. The best performance, which can be viewed as an upper bound, is computed by taking the individual best answer (out of 4) for each problem. We use boldface to denote the best performance and underline to denote the second best.

 



## Appendix D Experimental Details

### Benchmark Setup

Recruiting MTurk EvaluatorsWe used qualification tasks to recruit 160 qualified annotators on Mechanical Turk. They are paid over 18 USD per hour for all the evaluation and verification tasks.

Collecting Human Solutions on ProlificAll participants of human study provide informed consent in accordance with an approved Princeton University institutional review board (IRB) protocol [(10859)]. For a given problem, participants indicated whether they believed the problem is solvable, unsolvable, or required further clarification. If solvable, they provided a step-by-step solution, and otherwise they explained why the problem was unsolvable. A screenshot of the elicitation interface is shown in Figure 22.

Collecting Multiple GPT-4 Responses in BenchmarkRecall that in SS4.2, we elicit multiple solutions exclusively from the most potent LLM, GPT-4, to emulate the same setup of human study. To align with the varying number of human responses for different problems, we adjusted the quantity of collected GPT-4 answers to match that of human answers. On average, we elicited four GPT-4 solutions per problem through separate API call. To this end, four manually-designed instructions are used to prompt GPT-4 to reduce repetition among separate sessions. For each API call, we still adopt Nucleus sampling and return the top one sequence.

### Analyzing Results

Each machine-generated or human-written answer is annotated by three Mturk workers, with an average IAA of 0.71 as measured by Cohen's Kappa, indicating a substantially strong agreement. Interestingly, we notice that human workers disagree more often when deciding whether a solution is efficient or inefficient. Upon further investigation, we realize this is partially due to the limitation of individual annotator's capability - a person who is unaware of the most efficient solution might label a sub-optimal one as highly efficient. Therefore, for those generated solutions linked to solvable problems, instead of taking the _majority_ vote, we take the _worse_ labels as the golden label (_e.g.,_ taking _'ineff:'_ from _{'eff:'_, _'ineff:'_, _'eff:'_}). For all other cases, we still take the majority votes as gold labels. We find such modification leads to a more accurate set of labels.

\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Indoors/Household** & **Neutral** & **Outdoors** \\ \hline bedroom & at a party & at the beach \\ closet or storage organization & classroom and university lecture hall & backyard gardening \\ cooking a complex dish & dog training & beach cleanups, or planning a beach event \\ dining room & garage & boat trip \\ fitness workouts & going out for a meal & campsite setting \\ gym and sports facilities & plants, flowers and garden & city streets and sidewalks \\ hair styling and care & public speaking & construction work \\ home improvement & recycling and waste management & desert survival \\ in a hotel room & school and student activity & exploring a cave \\ indorors arrangement & school science fair & farm duties \\ kitchen & science laboratory & forest and jungle \\ library & swimming & hiking, camping, and traveling \\ living room & university campus & in the parks \\ office and work & vehicle maintenance & in the rain \\ packing things up & weather preparation and response & in the winter \\ personal grooming and beauty routine & & in the zoo \\ shopping & & on the playground \\  & & playing with snow \\  & & playing with water \\  & & rooftop terrace \\ \hline \hline \end{tabular}
\end{table}
Table 6: The tags (_i.e.,_ locations and activities) used to curate the dataset for diversity control. They can be broadly divided into Indoors/Household, Neutral, and Outdoors.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Solutions** & **Feasible** & **Feasible** & **Infeasible** & **LLM says** \\  & **\& eff.** (\(\uparrow\)) & **\& inf.** (\(\downarrow\)) & **unsolv.** (\(\downarrow\)) \\ \hline Iteration 1 & 39.1\% & 36.8\% & 24.0\% & 0.1\% \\ Iteration 2 & 34.7\% & 32.2\% & 31.7\% & 1.4\% \\ Iteration 3 & 25.4\% & 37.9\% & 35.7\% & 1.0\% \\ \hline \hline \end{tabular}
\end{table}
Table 7: GPT-4 performance on iteration 1, 2, and 3 of 200 problems. Numbers in each row add up too 100%.



* --_Instruction. -->
2You need to write the most common affordances of an item. Please choose one or more options from the following:
3<-_Seed list to expand with___. -->
4Container/holding items, covering, heating, measuring, drawing/writing, cleaning, sitting /stepping, tying or connecting, illumination, stretching, starting fire, sealing, cutting, separation, reaching high areas, powering devices, digging, making noise, flatten, cutting, gripping things, reflecting , eaten as food.
5
6<-_Examples. -->
7Here are some examples:
8rice : eaten as food
9case : container/holding items, protection , covering ruler : measuring, straightening
10box :container/holding items
11pencil : drawing/writing ,
12
13
14<-_Actual Task. -->
15Please write the common types of affordances of the following tools .
16
17
18
19N

Figure 14: The prompt used to analyze tool affordance. We start with a list of affordances as seed. We gradually expand our list thanks to the fact that GPT-4 does cannot strictly follow our instruction and occasionally generates other affordances not belonging the predefined set.

### The Prompts for Improving LLM's Ability

Figure 15 and Figure 16 list the actual prompts for Self-Reflection and Divergent-Convergent Thinking.

 

* <--_Round1:___-->
* User: {Problem Statement}
* If the problem is solvable , provide a concise solution . Use step1, step2, etc , and mention the tools to achieve each step . Use as few steps as possible and the answer should ideally be less than 100 words.
* If you cannot find a feasible solution , just say that it is not possible and give a very short justification .
* Assistant : {Answer}
* <--_Round2:___->
* User: Now, please verify if each step is physically feasible and afforded . After that , modify the solution if needed.
* Use the following format :
* Step 1: ...
* Step 2: ...
* <--_Requalminomodificationisneeded._==>
* Conclusion 1: Whether the problem is indeed solvable given all the constraints
* Conclusion 2: (If still solvable ) No modification needed/Modification needed.
* Modified solution :
* Assistant : {Response and Updated solution }
* <--_Repeatuntilnomodificationisneeded._==>

Figure 16: Prompt used for the divergent-convergent thinking strategy.

Figure 15: Prompt used for the step-by-stepverifystrategy.

 

Thanks for participating in this HIT Please read the instructions carefully.

In this task, you will be presented with a problem and a solution. Your task is to 1) judge by yourself if the problems are solvable (i.e., you can find a **reasonable and safe** solution to the problem using the presented tools), and 2) **judge if solving the problem requires using tools in an unconventional way**, and 3) verify if a proposed solution is **feasible, efficient** or not.

* For those you select as [noitemsep]you will be asked to provide an optional short explanation to justify your choice. Next, you will read an answer verify its correctness.
* For those you select as [noitemsep]you will be asked to decide if solving the problem efficiently requires using tools in an unconventional way. After that, you will read a potential solution and judge if the presented solution is feasible, efficient, and uses tools unconventionaly.

Here are the definitions and examples of new concepts:

_Solvable and Unsolvable Problems:_

* A problem is [noitemsep]if and only if you can achieve the stated goal 1) using only the presented tools, and 2) in a safe and reasonable way.
* For example, the problem below is [noitemsep]because Carosine is the only available tool that provides source of heat but it is not safe to use car engines to cook food.
* At a food truck park, your takeout food is getting cold and you want to reheat it. Tools available: car engine, foil paper that is very thin and tears easily, a rock, a can of soda, a cloth bag. How do you reheet the food taking into account the condition of available tools?
* For example, the below problem is [noitemsep]because It is not clear why the keys on a keyboard stopped working, thus unclear how to solve the problem.

_A few keys on your keyboard have stopped working, and you don't have a replacement. You have a pencil sharpener, a box of push pins, a booklet of envelopes, a rubber band, a pagerclip, a sticky note pod, and a glue stick, but the sticky notes have lost their stickiness and the paperclip too stiff. How can you temporarily fix the problem?_

_Feasible/Infeasible/Efficient/inefficient Solutions:_

* A solution is feasible if it 1) proposes a valid approach to solve the stated problem, and 2) each step is achievable using the presented tools.
* An evaluation is infeasible if 1)the proposed approach doesn't solve the stated problem, or 2)at least one step is NOT achievable.
* An evaluation is efficient if 1)it has no redundant or unnecessary steps, and 2)it is unlikely that the problem can be solved using a fewer number of steps.

_Conventional and Unconventional Usage of Tool:_

* Conventional Usage of Tests refers to the traditional and established ways to use a tool or item. Items are specifically designed to fulfill such goals effectively.

For example, using a broom to sweep the floor is a conventional usage.
* Unconventional Usage of Tests: refers to the unusual ways to utilize a tool or item. Items are NOT originally designed to fulfill such goals.

For example, using the same broom's broomstick to reach a high place is an unconventional usage.

Figure 17: Human Annotation Interface for Data Collection and Difficulty Assessment, Page 1.

 



**Example Problem:** On a desk placed against the wall, you are given a candle, a sturdy paper box without a top (roughly the size of your hand), plenty of thumbtacks, and a box of matches. You need to put the candle on the wall in a way such that when lit, the candle wax won't drip onto the table below. How would you process with only these items?

Figure 19: Human Annotation Interface for Data Collection and Difficulty Assessment, Page 3.

Figure 18: Human Annotation Interface for Data Collection and Difficulty Assessment, Page 2.





## Task

Please carefully read the problem below, and then judge by yourself if the problem is solvable.

Figure 21: Human Annotation Interface for Data Collection and Difficulty Assessment, Page 5.

Figure 20: Human Annotation Interface for Data Collection and Difficulty Assessment, Page 6.



Please carefully read the following problem, and write down a solution on your own:

You are outside building a snowman, but you/ve lost your gloves and your hands are getting too cold. You only have a list of just boiled hot water, a snow scoon, a hairbrush, a small shower, a plastic container for a part of thick wood socks, and a thermos. The wool socks are already darp from sweet and snow, so they would make your hands colder instead. How can you continue making your snowman without getting frostble? Please provide your step-by-step solution in the textbox below. Note: use the format "Step I. Use them AI (and A2) to achieve action/goal A.."

Figure 23: Human Evaluation Interface for Benchmarking, Page 1.

 



## Task

Please carefully read the problem-solution pair below, and then judge if the solution is correct. $(problem)

**Answer:**

\(\$\)(solution)

**Question: Is this answer correct? Choose one of the following:**

\begin{tabular}{l l}  & **Question: Is this answer correct? Choose one of the following:** \\  & **[label=]** \\  & **[label