Venue: NAACL 2025

# Thread: Thinking Deeper with Recursive Spawning

Philip Schroeder, Nathaniel Morgan, Hongyin Luo, James Glass

###### Abstract

Large language models (LLMs) have shown impressive capabilities across diverse settings, but still struggle as the length and complexity of the context increases. To address this challenge, we propose Thinking Recursively and Dynamically (ThReaD). Thread frames model generation as a thread of execution that, based on the context, can run to completion or dynamically spawn new threads. By spawning, threads can offload work (e.g., thinking, retrieving information) to child threads, which only return tokens needed for the parent thread to do its work. We apply Thread in the settings of LLM task solving and question answering, where the dynamic threading allows the model to recursively decompose the given task or question into progressively simpler subproblems that can be solved by separate child threads. We test Thread, implemented using a few-shot learning approach, on diverse benchmarks for agent tasks and data-grounded question answering. Thread achieves state-of-the-art performance with GPT-4 and GPT-3.5 on these benchmarks, including ALFWorld, TextCraft, and WebShop, along with two new benchmarks, DataCommons QA and MIMIC-III ICU QA. In addition, Thread outperforms existing frameworks by 10% to 50% absolute points with smaller models, including Llama-3-8b and CodeLlama-7b.

MIT Computer Science and Artificial Intelligence Lab, Cambridge MA, USA

## 1 Introduction

Large Language Models (LLMs) have shown success in diverse settings Wei et al. (2022); Huang and Chang (2023), but their performance degrades as context length and complexity grows Dziri et al. (2023); Liu et al. (2024); Qin et al. (2023). This constraint limits their efficacy in settings that require more work (thinking, retrieving information, analyzing, etc.) than can fit into a concise line of generation. To address this limitation, we propose Thinking Recursively and Dynamically (ThReaD). Thread is a general framework where model generation is treated as a thread of execution that, based on the context, can independently run to completion or dynamically spawn new threads in a recursive fashion. When a thread spawns a child, the child generates conditioning on context that derives from the parent's token sequence. Spawning child threads allows work, such as internal thinking or interacting with an external environment, to be completed on behalf of the parent, without directly adding to the parent's context. Child threads return only the information (tokens) needed for the parent to complete its work. In effect, spawning enables the model to dynamically adapt the amount of work or intermediate computational steps used to produce different parts of its token sequence.

The synchronization and spawning mechanisms of Thread can vary based on the setting. Figure 1 shows an example of applying Thread in a synchronous setting, where a parent waits for a child in a form analogous to Thread.join() in multithreaded programming. In this example, a parent thread pauses generation until the child execution completes and the child returns output tokens that are appended directly to the token sequence of the parent before the parent proceeds generation. Thread improves the flexibility of model generation, allowing the model to adapt, through recursive spawning, the amount of work it does based on the given problem, without overextending its context. Importantly, this unfolds with limited need for explicit rules or hard coded logic. Instead, Thread relies on the model's ability to infer the appropriate continuation given the context of each thread at the time of execution. Finally, Thread is agnostic to the type of sequence. Depending on the underlying model, Thread can be applied in varied settings, including multi-modal applications, and fulfill varied purposes (e.g., carrying out calcu

lations, generating thoughts, retrieving information, robot manipulation).

In this paper, we consider Thread in the settings of agent tasks (Figure 1(a)) and question answering (Figure 1(b)). In these settings, Thread enables the LLM to recursively decompose a task, or question, into progressively simpler sub-problems that can be solved by child threads in a compartmentalized manner. A child thread infers the specifications of its sub-problem from its parent's token sequence. If needed, child threads can troubleshoot their sub-problem, and spawn their own threads, all without distracting their parent. We test Thread, using a few-shot learning approach, on benchmarks consisting of question-answering and agent tasks. For these problems, we apply Thread with join synchronization, allowing parent threads to see feedback from child threads before defining their next step. For example, in the agent setting, if the model is given the task of cleaning a bowl and putting it on the countertop, the main thread may spawn a child with the sub-task of finding a bowl. This child can then spawn further threads for handling the complexities of navigating to and checking different locations for the bowl, with each thread returning only the information needed for its parent to proceed with its sub-task. Based on the feedback with regard to finding the bowl, the main thread can spawn a new thread to re-attempt the sub-task or spawn threads to execute the remaining sub-tasks (i.e., washing the bowl, putting the bowl on the countertop).

Thread has several advantages over existing frameworks in these settings. To start, Thread addresses limitations of methods that require the model to solve the problem in one line of context [234; 50] by allowing the model to dynamically offload work by spawning child threads. Further, unlike methods for task decomposition [68, 79, 80, 10, 11], Thread enables the model to adapt its decision-making in real time as it receives feedback from each step, as depicted in Figure 2. Existing methods either do not allow for plan adaptation [50, 10] or only adapt plans by calling pre-existing sub-task handlers [68] or by adding more sub-steps within the existing plan when a sub-task fails [80, 10]. Finally, Thread provides a more unified framework compared to these methods, which require separate prompting mechanisms for separate planner and executor modules. Instead, Thread can be implemented with the same few-shot prompt used for every thread at every step of the task completion.

We evaluate Thread on agent tasks and data-grounded question answering. Thread significantly outperforms prior methods, achieving state-of-the-art performance with GPT-4 and GPT-3.5 on ALFWorld, TextCraft, and WebShop, along with two new benchmarks, DataCommons QA and MIMIC-III ICU QA. Thread also shows success with smaller models, outperforming prior methods by 10% to 50% absolute points across the benchmarks with Llama-38b and CodeLlama-7b.

Figure 1: **Thread with join synchronization. Thread frames model generation as an execution thread that can dynamically spawn new threads. In the example with join synchronization, when a thread spawns a child, it pauses generation until feedback is returned. Child threads generate starting from context derived from their parent’s token sequence. When the child completes, it returns output tokens (colored bars), which are added to the context of the parent before it continues generating. \(\phi\) and \(\psi\) are functions that control information flow from parent to child and from child to parent, respectively, by defining the tokens that are propagated based on the thread’s token sequence.**



Thread Framework

Thread frames model generation as an execution thread that, given the context, can run to completion or spawn new threads. A thread consists of generating with a model, \(G\), given a specific starting context, \(c\). For the main thread, \(c\) is the initial seed context, \(c_{0}\) (e.g., context provided by a user). For each child thread, \(c\) derives from the parent's token sequence. A thread continues until it meets some termination criteria, such as a stop token.

```
function Thread(\(c\), \(Y\)) while True do  \(Y=Y+G(c+Y)\) if\(Y\) spawns a child thread then \(Y=Y+\psi(\textsc{Thread}(\phi(Y),[]))\) elseif\(Y\) ends the thread then return\(Y\)
```

**Algorithm 1**Thread with join synchronization

### Threat with Join Synchronization

We present Thread in a setting where, analogous to how the join() method is used in multithreaded programming, a parent thread waits for the execution of the child to complete before proceeding. When the child execution completes, the child's output is returned directly to the parent. In this setting, Thread can be implemented using the recursive function shown in Algorithm 1. The function takes two inputs: the context for the thread, \(c\), and the tokens generated so far by the thread, \(Y\). We treat a token sequence as a list of tokens.

Thread begins with \(c=c_{0}\) and \(Y=[\,]\). The model generates, \(G(c+Y)\), one token at a time conditioning on the given context, \(c\), appended with the growing sequence of generated tokens, \(Y\).

If \(Y\) spawns a child thread, then a new thread is created with \(\textsc{Thread}(\phi(Y),[\,])\) where \(Y\) for the child thread is initialized as an empty list and \(c\) is based on the token sequence of the parent, \(\phi(Y)\). The output tokens of the child thread are appended to the parent's token sequence and the parent continues generating. If \(Y\) ends the thread, then the thread returns \(Y\) (its full token sequence).

The \(\phi\) function defines the context for a child thread based on the full token sequence of the parent thread at the time the child is spawned, including tokens directly generated by the parent or returned as output from previous child threads of that parent. The \(\psi\) function defines the output tokens of a child thread based on its full token sequence at the time the thread ends, including tokens directly generated by the child or returned as output from threads that it spawned.

### Alternative Synchronization and Spawning Mechanisms

Above, we show Thread in a setting where a parent always waits for a child to complete and the child's output is always appended directly to the token sequence of the parent before the parent proceeds. However, depending on the setting, Thread can involve varied mechanisms for synchronization and spawning and, in scenarios without sequential dependencies, can include asynchronous multithreading, allowing parent threads to continue generating without waiting for child threads to complete (improving overall efficiency).

### Defining Parent-Child Information Exchange with \(\phi\) and \(\psi\)

The functions \(\phi\) and \(\psi\) control the propagation of information from parent to child and from child to parent, respectively. Similar to the synchronization and spawning mechanisms mentioned above, these functions can vary based on the setting in which Thread is implemented. For example, \(\phi\) and \(\psi\) can range from simple functions that compress or decompress information in the token sequence to entirely separate models that transform the sequence in more complex ways. In the section below, we describe how we define \(\phi\) and \(\psi\) when implementing Thread in the settings of agent tasks and question answering.

### Dynamically Adapting Intermediate Work in Token Generation

The Thread framework enables a thread of model generation to spawn a child thread to produce output tokens that can serve as future tokens for the parent. In effect, this enables the model to adapt, as needed, the amount of work or computational steps used to produce different tokens. The intermediate work added through recursive spawning can range from deeper thinking and reasoning to interacting with external environments or sources of information. As depicted in Figure 3, the work associated with each part of the parent's token sequence is represented by a thread tree organized based on connections between threads. These trees reflect how threads interact to form output tokens and the computational work associated with each component of the overall model generation 



## 3 Applying Thread for Agent Tasks and Question Answering

In this paper, we test Thread in the settings of question answering and agent tasks with LLMs, where Thread enables the model to dynamically decompose the given problem into simpler sub-problems that are completed by separate threads. We apply Thread using an in-context learning approach, where the same few-shot prompt is used for every thread at every step of the task completion or question answering. Since these problems benefit from the model adapting to feedback as it defines its next steps, we apply Thread with the join synchronization described above.

**Thread spawning and termination based on special stop tokens.** We implement a spawning mechanism using a special stop token, \(\omega_{listen}\), which pauses the thread generation and marks the start of the output the thread expects from the child. The context for the child thread is defined by \(\phi\) based on the token sequence of the parent that occurs before \(\omega_{listen}\). A thread ends when it generates the end token, \(\omega_{end}\). As exemplified in Figure 2, we use => as \(\omega_{listen}\) and END as \(\omega_{end}\).

### Implementing Thread with few-shot learning.

We leverage a few-shot learning approach to implement Thread. The few-shot examples are comprised in a single prompt with examples of successful spawning and problem solving at different thread depths, including how the token \(\omega_{listen}\) should be used. As a result, the approach does not require explicit rules to define when new threads should be spawned and how parent threads should respond to feedback from a child. Instead, these mechanisms are all implied by the examples provided in the few-shot prompt. Changing the spawning and parent-child dynamics simply requires modifying the prompt. All threads are provided the same few-shot prompt for every step of the problem completion. Thus, Thread only requires generating a single prompt for a given setting. This prompt, \(q\), is prepended to the context for each thread, \(G(q+c+Y)\), forming the full context from which the thread will generate.

**Defining \(\phi\) and \(\psi\)**. As described above, the functions \(\phi\) and \(\psi\) control information flow from parent to child and from child to parent, respectively. The function \(\psi\) returns the result from a child thread based on its full token sequence. As depicted in Fig

Figure 2: When given a task (a) or question (b), Thread can be used to help the model, through recursive spawning, decompose the problem into progressively simpler sub-problems that are solved by child threads. In these examples, the context for a child thread is based on the last line of the parent’s token sequence.



ure 2, we implement \(\psi\) as a function that returns the tokens included in the print statement at the end of the child thread's token sequence. In addition, \(\psi\) appends the token <= to the output to mark the end of the child's output within the parent's token sequence. We implement \(\phi\) as a function that extracts the last line of the parent's token sequence to create the context for the child thread. To improve the ability of parent and child threads to efficiently organize, use, update, and propagate shared information, we leverage the coding-based skills of the model to define variables that represent pieces of information that are important for the given problem. For example, if the task is to find an item on an e-commerce site that has certain attributes, the model can define a variable, such as obj_attributes, and spawn a child to search for items with obj_attributes instead of having to list all of the attributes. Then, when \(\phi\) processes the token sequence from the parent, it instantiates the variables, allowing the child to see, for example, the full list of attributes. We also test Thread implemented without using these variables, with results shown in the appendix.

**Threads performing actions in an environment.** Threads can listen for feedback when performing an action in an environment the same way they listen for feedback from a child thread. They simply generate tokens that represent the action and then produce the token \(\omega_{listen}\) to listen for feedback from the environment. The action, \(a\), is then executed in the environment, \(E\), and the output, \(o=E(a)\), is appended to the thread's token sequence the same way it is done for the feedback from a child thread (Algorithm 2). The thread can then continue generating based on the feedback.

## 4 Experiments

We evaluate Thread on 5 benchmarks for agent tasks and question answering using large and small models, including GPT-4, GPT-3.5, Llama-3-8b, Llama-2-7b, and CodeLlama-7b. For each benchmark, we use the same prompt for all models.

We include details regarding models, prompts, and experiments (including ablations) in the appendix. Examples of the prompts are shown in Appendix G Details regarding the QA benchmarks are provided in Appendix F. We release all code and data at [https://github.com/philipmit/thread](https://github.com/philipmit/thread).

### ALFworld

ALFWorld is a suite of text-based environments, implemented in TextWorld, designed to align with the embodied ALFRED benchmark Shridhar et al. (2021). It includes 6 different task types that instruct the agent to accomplish a goal by interacting with a simulated household with actions defined in text. Following Yao et al. (2023), we evaluate an agent on 134 unseen evaluation games, including six task types: Pick, Clean, Heat, Cool, Look, and Pick2. Like previous methods, such as ReAct Yao et al. (2023), we implement Thread with one few-shot prompt per task. We also test Thread using task-general prompting.

Results.Thread significantly outperforms all prior methods (Tables 1 and 2), including those that require the agent to have access to external memory consisting of past experiences with the task. With GPT-4, Thread achieves a combined success rate of 98.5% with 100% success in 4 of the 6 tasks. With GPT-3.5, Thread shows a combined success rate of 95.5%, outperforming all methods by over 9% absolute points. With smaller models (Llama-3-8b and CodeLlama-7b), Thread improves upon prior methods by 30% to 55% points.

Finally, Table 8 shows the results when testing Thread with task-general prompting. To implement the task-general prompt, we split the prompt into one set of examples for the main thread and a second set of examples for all other threads (with the same sets of examples used for all tasks). We see that GPT-3.5 achieves the same combined success rate with the task-general prompting as it does with the task-specific prompting. Further, while the performance of Llama-3-8b and CodeLlama-7b degrades with task-general prompting, it remains a significant improvement over task-specific prompting with prior methods (Table 2).

### TextCraft

TextCraft is a text-based environment inspired by the crafting component of Minecraft Prasad et al. (2023). The tasks involve building Minecraft items with crafting commands using available resources from the environment. Following the work of Prasad et al. (2023), we evaluate Thread on the test set containing 200 samples. We implement Thread using a single few-shot prompt and compare its performance to previous methods used for TextCraft Prasad et al. (2023); Wang et al. (2024); Shinh et al. (2024); Yao et al. (2023) 

Results.Thread outperforms prior methods by at least 20% absolute points with GPT-3.5 and at least 40% points with Llama-3-8b and 30% points CodeLlama-7b (Table 3). Further, Llama-3-8b with Thread outperforms GPT-3.5 with all prior methods and CodeLlama-7b with Thread outperforms GPT-3.5 with all prior methods except TDAG.

### WebShop

WebShop is an online shopping environment with over a million real-world products Yao et al. (2022). The benchmark requires the model to interact with the website to purchase a product based on specifications provided by a user. The evaluation metrics include the success rate, which is the percentage of products that were purchased with full success, and the score, which is the average percentage of desired attributes covered by the purchased items. Following prior work Shinn et al. (2024); Prasad et al. (2023); Zhou et al. (2023), we evaluate Thread on a test set of 100 instructions.

Results.Table 4 shows the success rate and score for each method. We again separate the results based on whether the method requires the model to have access to external memory. With GPT-3.5, Thread outperforms other prompt-only methods by an absolute 4% in success rate and over 10% in score and outperforms RAP Kagaya et al. (2024) by 1% in success rate (with a similar score). Thread achieves an absolute 10%, or greater, improvement in success rate with Llama-3-8b and CodeLlama-7b. Llama-3-8b with Thread achieves a higher success rate than GPT-3.5 with all prior methods, with the exception of RAP.

### DataCommons QA

DataCommons QA is a benchmark consisting of questions that can be answered using data provided by Google DataCommons Guha (2019) [https://datacommons.org](https://datacommons.org). The questions range from comparing statistics in different locations to making predictions regarding future trends. The test

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & **Requires** & \multirow{2}{*}{**Method**} & \multirow{2}{*}{**All**} & **Pick** & **Clean** & **Heat** & **Cool** & **Look** & **Pick2** \\  & **ext. mem.** & & & & & & & & \\ \hline \multirow{6}{*}{GPT-3.5} & & Reflexion Shinn et al. (2024) & 76.1 & 75.0 & 80.6 & 69.6 & 76.2 & 83.3 & 70.6 \\  & & AdaPlanner Sun et al. (2023) & 82.8 & 91.7 & 87.1 & 82.6 & **95.2** & 50.0 & 82.4 \\  & Yes & RAP* Kagaya et al. (2024) & 85.8 & 95.8 & 87.1 & 78.3 & 90.5 & 88.9 & 70.6 \\  & & AutoGuide* Fu et al. (2024) & 79.1 & - & - & - & - & - & - \\ \cline{2-10}  & \multirow{2}{*}{No} & ReAct Yao et al. (2023) & 53.7 & 45.8 & 48.4 & 69.6 & 66.7 & 55.6 & 35.3 \\  & & DecompF Khot et al. (2023) & 84.3 & 91.7 & 87.1 & 82.6 & 90.5 & 83.3 & 64.7 \\  & & ADaP Prasad et al. (2023) & 82.1 & 87.5 & 83.9 & 78.3 & 90.5 & 83.3 & 64.7 \\  & & THREAD & **95.5** & **95.8** & **93.5** & **95.7** & **95.2** & **100** & **94.1** \\ \hline \multirow{6}{*}{GPT-4} & Yes & RAP* Kagaya et al. (2024) & 94.8 & 95.8 & 90.3 & **100** & **95.2** & **100** & 88.2 \\ \cline{2-10}  & \multirow{2}{*}{No} & ReAct Yao et al. (2023) & 87.3 & 83.3 & 77.4 & 95.7 & 85.7 & **100** & 88.2 \\ \cline{1-1}  & & DecomP Khot et al. (2023) & 89.6 & 87.5 & 87.1 & 91.3 & 90.5 & 94.4 & 88.2 \\ \cline{1-1}  & & ADApT Prasad et al. (2023) & 91.0 & 91.7 & 87.1 & 95.7 & 90.5 & 94.4 & 88.2 \\ \cline{1-1}  & & THREAD & **98.5** & **100** & **100** & **100** & **95.2** & **100** & **94.1** \\ \hline \hline \end{tabular}
\end{table}
Table 1: **ALFWorld task-specific success rates (%). Results are separated based on whether the method requires the agent to access to external memory. *As reported in Kagaya et al. (2024) (RAP) and Fu et al. (2024) (AutoGuide).**

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Requires** & **Method** & **Llama-3-8b** & **Llama-2-7b** & **CodeLlama-7b** \\ \hline \multirow{2}{*}{Yes} & Reflexion Shinn et al. (2024) & 25.4 & 11.2 & 27.6 \\  & AdaPlanner Sun et al. (2023) & 28.4 & 11.9 & 29.9 \\ \hline \multirow{4}{*}{No} & ReAct Yao et al. (2023) & 20.1 & 12.7 & 23.1 \\  & DecomP Khot et al. (2023) & 37.3 & 14.2 & 33.6 \\ \cline{1-1}  & ADaP Prasad et al. (2023) & 30.6 & 15.7 & 35.1 \\ \cline{1-1}  & THREAD & **71.6** & **22.4** & **91.0** \\ \hline \hline \end{tabular}
\end{table}
Table 2: ALFWorld success rates (%) for all tasks combined.

 

set includes a total of 140 questions. We implement Thread with a few-shot prompt and compare its performance to three baselines: Reflexion Shinn et al. (2024), Natural Language Embedded Programs (NLEP) Zhang et al. (2023), and NLEP+ReAct, an extension of NLEP that allows the model to evaluate intermediate outputs of its analysis as it answers the question.

Results.Table 5 shows the accuracy of each method on DataCommons QA. Thread outperforms prior methods by over 10% absolute points with GPT-3.5, Llama-3-8b, and CodeLlama-7b. Llama-3-8b with Thread outperforms GPT-3.5 with all prior methods.

### Mimic-Iii Icu Qa

The MIMIC-III ICU QA benchmark consists of patient-focused questions based on clinical time-series data made available by MIMIC-III Johnson et al. (2016). The benchmark reflects a setting in which a healthcare provider can ask natural language questions about patients in the intensive care unit (ICU) and receive an answer from the language model based on the relevant patient data. The test set includes 160 questions. Similar to above, we implement Thread with a few-shot prompt and compare its performance relative to Reflexion Shinn et al. (2024), NLEP, and NLEP+ReAct.

Results.As shown in Table 6, Thread significantly outperforms prior methods across all models except Llama-2-7b. All methods show the lowest performance with Llama-2-7b, which is consistent across all benchmarks.

## 5 Related Work

The Thread framework relates to prior approaches for dynamically adapting work, or computational steps, used during model generation. Previous approaches for dynamically adapting computation based on the given problem require special training Goyal et al. (2023); Nye et al. (2022) or novel model architectures Graves (2016); Banino et al. (2021); Dehghani et al. (2018). Thread provides a more general and flexible framework that, as we show, can be applied without modifications to the underlying model architecture and without further training. In addition, unlike methods that allow the model to adapt the amount of internal computations Goyal et al. (2023); Graves (2016); Banino et al. (2021); Dehghani et al. (2018), the intermediate work used to supplement model generation with Thread can involve work that ex

\begin{table}
\begin{tabular}{l l c c c c c c c} \hline \hline

- \\ \cline{1-1}  & AutoGuide* Fu et al. (2024) & 46 & 73.4 &
- \\ \hline \multirow{9}{*}{No} & ReAct Yao et al. (2023) & 37 & 59.5 & 31 & 54.1 & 10 & 18.5 & 17 & 49.2 \\  & DecomP Khot et al. (2023) & 43 & 58.7 & 35 & 56.3 & 14 & 29.6 & 21 & 57.1 \\ \cline{1-1}  & ADApT Prasad et al. (2023) & 44 & 60.0 & 35 & 58.2 & 13 & 28.7 & 21 & 56.4 \\ \cline{1-1}  & TDAG Wang et al. (2024) & 45 & 64.5 & 37 & 63.5 & 14 & 29.8 & 23 & 58.5 \\ \cline{1-1}  & THREAD & **49** & **76.3** & **47** & **70.4** & **20** & **48.5** & **40** & **68.9** \\ \hline \hline \end{tabular}
\end{table}
Table 4: WebShop success rate (SR; %) and score (%).

\begin{table}
\begin{tabular}{l l l l l} \hline \hline

- \\ ReAct Yao et al. (2023) & 20.5 & 12.5 & 8.0 & 10.5 \\ DecomP Khot et al. (2023) & 68.5 & 47.0 & 12.0 & 38.5 \\ ADaPT Prasad et al. (2023) & 52.5 & 23.5 & 12.0 & 18.0 \\ TDAG Wang et al. (2024) & 73.5 & 48.5 & 14.0 & 31.0 \\ THREAD & **93.5** & **92.0** & **20.0** & **71.0** \\ \hline \hline \end{tabular}
\end{table}
Table 3: TextCraft success rate (%). *As reported in Prasad et al. (2023).
 

tends beyond internal thinking, such as retrieving information or interacting with an external environment. In addition, Thread allows for greater interpretability, since all of the intermediate work is performed by generating meaningful tokens.

In this paper, we apply Thread in a setting where it improves the ability of LLMs to decompose a given problem into simpler sub-problems. There has been significant prior work involving the decomposition of problems with neural modular modeling architectures (Andreas et al., 2016; Talmor and Berant, 2018; Min et al., 2019; Jiang and Bansal, 2019; Gupta et al., 2019; Perez et al., 2020; Khot et al., 2021). Later work has used fine-tuning or in-context learning with modern LLMs for decomposition with multi-step tasks (Khot et al., 2023; Wang et al., 2023), mathematical reasoning (Gao et al., 2023; Lee and Kim, 2023), and program synthesis (Murali et al., 2018; Nye et al., 2019; Zheng et al., 2023). To handle complex tasks, a more recent approach, called ADaPT, has improved upon these methods by using a planner to further decompose a task when a failure is encountered Prasad et al. (2023), which was further extended to a multi-agent setting with TDAG (Wang et al., 2024). Unlike prior methods, such as Decomposed Prompting (Khot et al., 2023), ADAPT and TDAG enable the model to adapt a plan upon task failure. In contrast to these approaches, Thread has the advantage of allowing the model to adapt its decision-making in real time as it receives feedback from each step. With Thread, each step of the plan is specified only after receiving feedback from the previous step. In addition, Thread enables the model to flexibly specify and, if needed, re-specify sub-tasks when a sub-task fails or when unexpected feedback is returned. Finally, Thread, which uses the same prompting for every thread at every step of the problem, provides a more unified framework than ADaPT and TDAG, which involve separate prompting and few-shot examples for the "planner" and "executor" modules. With these methods, the planner module outlines a full plan and the executor follows all steps of the plan. If the executor fails, the planner is tasked with further decomposing the failed step into more sub-steps. Instead of outlining a full plan and attempting all steps, Thread involves specifying one step, allocating this to a child thread, and receiving feedback from the child thread before proceeding.

## 6 Conclusion

We introduce Thread, a general framework in which model generation is treated as a thread of execution that can dynamically offload work, such as thinking or retrieving information, by spawning new threads. Thread enables a model to adapt, through recursive spawning, the amount of intermediate work it uses to produce different parts of its token sequence. We apply this framework in the settings of agent tasks and question answering. We show that Thread, implemented using a few-shot learning approach, achieves state-of-the-art performance on diverse benchmarks.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Method** & GPT-3.5 & Llama-3-8b & Llama-2-7b & CodeLlama-7b \\ \hline Reflexion (Shinn et al., 2024) & 38.1 & 19.4 & 8.8 & 16.9 \\ DecompP (Khot et al., 2023) & 61.9 & 51.3 & 13.1 & 30.6 \\ NLEP (Zhang et al., 2023) & 35.6 & 17.5 & 8.1 & 15.6 \\ NLEP+ReAct & 43.1 & 23.8 & 12.5 & 21.9 \\ THREAD & **71.3** & **61.9** & **18.8** & **58.1** \\ \hline \hline \end{tabular}
\end{table}
Table 6: MIMIC-III ICU QA accuracy (%).

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Method** & GPT-3.5 & Llama-3-8b & Llama-2-7b & CodeLlama-7b \\ \hline Reflexion (Shinn et al., 2024) & 37.9 & 24.3 & 10.7 & 20.7 \\ DecompP (Khot et al., 2023) & 64.3 & 57.9 & 15.7 & 31.4 \\ NLEP (Zhang et al., 2023) & 28.6 & 21.4 & 11.4 & 19.3 \\ NLEP+ReAct & 41.4 & 27.1 & 14.3 & 24.3 \\ THREAD & **77.1** & **67.9** & **22.1** & **62.1** \\ \hline \hline \end{tabular}
\end{table}
Table 5: DataCommons QA accuracy (%).

 



## Acknowledgements

This work was supported by Quanta Computer, Inc. We also thank the Google DataCommons team for their help with developing the DataCommons QA benchmark.

## Limitations

The implementation of Thread in this paper does not involve any explicit mechanisms for error handling and instead relies on the inherent reasoning ability of LLMs to respond to unexpected feedback from child threads or from the environment. An advantage of Thread is that individual threads can fail without directly affecting other threads. However, the context necessary to self-correct may be lost if error reporting is not handled appropriately. More work is needed to develop robust error detection and recovery mechanisms to ensure that the information necessary for self-correction is preserved and utilized effectively. In addition, our implementation of Thread involves limited communication between the parent and child threads, where the context for the child thread is based on the last line of the parent's token sequence. This can result in the child missing information that could help with its work, leading to less efficient or effective generation. Overall, more work is need to improve the propagation of information, as defined by the functions \(\phi\) and \(\psi\), between parent and child threads.

## Ethical Statement

In this work, we show how Thread can be applied to improve LLM agent task completion and question answering. LLMs are being increasingly deployed to autonomously interact with external environments and humans. By improving this capacity, our work has the potential for amplifying risk associated with automated decision-making or facilitate harmful use of LLMs. Addressing these risks requires careful consideration of ethical guidelines, robustness checks, and transparency in algorithm deployment to ensure that advancements in LLMs contribute positively to societal welfare.

## References

* J. Andreas, M. Rohrbach, T. Darrell, and D. Klein (2016)Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48. Cited by: SS5.
* A. Banino, J. Balaguer, and C. Blundell (2021)PonderNet: learning to ponder. In 8th ICML Workshop on Automated Machine Learning (AutoML), Cited by: SS5.
* M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser (2018)Universal transformers. In International Conference on Learning Representations, External Links: Link Cited by: SS5.
* N. Dziri, X. Lu, M. Solar, X. Lorraine Li, L. Jiang, B. Y. Lin, S. Welleck, P. West, C. Bhagavatula, R. Le Bras, et al. (2023)Faith and fate: limits of transformers on compositionality. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: SS5.
* Y. Fu, D. Kim, J. Kim, S. Sohn, L. Logeswaran, K. Bae, and H. Lee (2024)Autoguide: automated generation and selection of state-aware guidelines for large language model agents. arXiv preprint arXiv:2403.08978. Cited by: SS5.
* L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig (2023)Pal: program-aided language models. In International Conference on Machine Learning, pp. 10764-10799. Cited by: SS5.
* S. Goyal, Z. Ji, A. Singh Rawat, A. Krishna Menon, S. Kumar, and V. Nagarajan (2023)Think before you speak: training language models with pause tokens. In The Twelfth International Conference on Learning Representations, Cited by: SS5.
* A. Graves (2016)Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983. Cited by: SS5.
* R. Guha (2019)Datacommons. In 2019 IEEE International Conference on Big Data (Big Data), Vol. , pp. 1-1. Cited by: SS5.
* N. Gupta, K. Lin, D. Roth, S. Singh, and M. Gardner (2019)Neural module networks for reasoning over text. In International Conference on Learning Representations, Cited by: SS5.
* J. Huang and K. Chen-Chuan Chang (2023)Towards reasoning in large language models: a survey. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1049-1065. Cited by: SS5.
* Y. Jiang and M. Bansal (2019)Self-assembling modular networks for interpretable multi-hop reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4474-4484. Cited by: SS5.
* A. EW Johnson, T. J. Pollard, L. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi,

and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. _Scientific data_, 3(1):1-9.
* Kagaya et al. [2024] Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. 2024. Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents. _arXiv preprint arXiv:2402.03610_.
* Khot et al. [2021] Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2021. Text modular networks: Learning to decompose tasks in the language of existing models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1264-1279.
* Khot et al. [2023] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In _The Eleventh International Conference on Learning Representations_.
* Lee and Kim [2023] Soochan Lee and Gunhee Kim. 2023. Recursion of thought: A divide-and-conquer approach to multi-context reasoning with language models. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 623-658.
* Liu et al. [2024] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Parangape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. _Transactions of the Association for Computational Linguistics_, 12:157-173.
* Min et al. [2019] Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirizi. 2019. Multi-hop reading comprehension through question decomposition and rescoring. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6097-6109.
* Murali et al. [2018] Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, and Chris Jermaine. 2018. Neural sketch learning for conditional program generation. In _International Conference on Learning Representations_.
* Nye et al. [2022] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2022. Show your work: Scratchpads for intermediate computation with language models. In _Deep Learning for Code Workshop_.
* Nye et al. [2019] Maxwell Nye, Luke Hewitt, Joshua Tenenbaum, and Armando Solar-Lezama. 2019. Learning to infer program sketches. In _International Conference on Machine Learning_, pages 4861-4870. PMLR.
* Perez et al. [2020] Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
* Prasad et al. [2023] Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. 2023. Adapt: As-needed decomposition and planning with language models. _arXiv preprint arXiv:2311.05772_.
* Qin et al. [2023] Guanghui Qin, Yukun Feng, and Benjamin Van Durme. 2023. The nlp task effectiveness of long-range transformers. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 3774-3790.
* Shinn et al. [2024] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. _Advances in Neural Information Processing Systems_, 36.
* Shridhar et al. [2021] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. Alfworld: Aligning text and embodied environments for interactive learning. In _International Conference on Learning Representations_.
* Sun et al. [2023a] Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2023a. Adaplanner: Adaptive planning from feedback with language models. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Sun et al. [2023b] Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, and Mohit Ilyyer. 2023b. Pearl: Prompting large language models to plan and execute actions over long documents. _arXiv preprint arXiv:2305.14564_.
* Talmor and Berant [2018] Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 641-651.
* Wang et al. [2023] Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2609-2634.
* Wang et al. [2024] Yaoxiang Wang, Zhiyong Wu, Junfeng Yao, and Jinsong Su. 2024. Today: A multi-agent framework based on dynamic task decomposition and agent generation. _arXiv preprint arXiv:2402.10178_.
* Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Wolf et al. [2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_ 

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. _Advances in Neural Information Processing Systems_, 35:20744-20757.
* Yao et al. (2023) Shunya Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In _International Conference on Learning Representations_.
* Zhang et al. (2023) Tianhua Zhang, Jiaxin Ge, Hongyin Luo, Yung-Sung Chuang, Mingye Gao, Yuan Gong, Xixin Wu, Yoon Kim, Helen Meng, and James Glass. 2023. Natural language embedded programs for hybrid language symbolic reasoning. _arXiv preprint arXiv:2309.10814_.
* Zheng et al. (2023) Wenqing Zheng, SP Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhanyyang Wang. 2023. Outline, then details: Syntactically guided coarse-to-fine code generation. In _International Conference on Machine Learning_, pages 42403-42419. PMLR.
* Zhou et al. (2023) Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023. Language agent tree search unifies reasoning acting and planning in language models. _arXiv preprint arXiv:2310.04406_.

 



## Appendix A Illustration of Intermediate Work used to Produce Tokens

Figure 3 illustrates how Thread enables the model to dynamically adapt the amount of work or computational steps used to produce different parts of its token sequence. The illustration is based on the Thread 0 from Figure 1. The "supplemental work" for a token reflects the amount of additional tokens, generated by spawned threads, used to produce each token within Thread 0. The work associated with each part of the token sequence is represented by a thread tree organized based on connections between parent and child threads. These trees reflect how threads interact to form output tokens and the computational work associated with each component of the overall model generation.

## Appendix B Implementing Thread for Agent Tasks and Question Answering

We release all code and data at [https://github.com/philipmit/thread](https://github.com/philipmit/thread). As described in section 3, we implement Thread with join synchronization for the problems evaluated in this paper. We implement a spawning mechanism using a special stop token, \(\omega_{listen}\). A thread ends when it generates the end token, \(\omega_{end}\). We use \(\Rightarrow\) as \(\omega_{listen}\) and END as \(\omega_{end}\).

We implement Thread in each setting using a fixed few-shot prompt showing examples of successful spawning and problem solving at different thread depths. All threads are provided the same few-shot prompt for every step of the problem completion. This prompt, \(q\), is prepended to the context, \(c=q+c\), for each thread, forming the full context from which the thread generates.

As shown in Algorithm 2 below, threads can get feedback from the environment the same way they get feedback from a child thread. They generate tokens that represent the action and then produce the token \(\omega_{listen}\) to listen for feedback from the environment. As shown in Algorithm 2, the action, \(a\), is executed in the environment, \(E\), and the output, \(o=E(a)\), is appended to the thread's token sequence before it continues generation. The function \(\xi\) parses the action from the token sequence. As with prior methods such as ReAct Yao et al. (2023), actions are specified following the '>' token.

```
functionThread(\(c\), \(Y\)) whileTruedo do \(Y=Y+G(c+Y)\) if\(Y\) performs an actionthen \(a=\xi(Y)\) \(o=E(a)\) \(Y=Y+o\) elseif\(Y\) spawns a child threadthen \(Y=Y+\psi(\texttt{Thread}(\phi(Y),[]))\) elseif\(Y\) ends the threadthen return\(Y\)
```

**Algorithm 2** Thread with join synchronization for tasks involving actions in environment

To ensure consistency across benchmarks and with prior work, we use Meta-Llama-3-8B for Llama-3-8b, Llama-2-7b-hf for Llama-2-7b, and CodeLlama-7b-hf for CodeLlama-7b available on Huggingface Wolf et al. (2019) and use gpt-4-0613 for GPT-4 and gpt-3.5-turbo-instruct for GPT-3.5 from the OpenAI API, with the temperature set to 0 for all experiments.

## Appendix C Additional Experiments with GPT-3.5

Table 7 shows the mean and standard error of the performance of Thread with GPT-3.5 across 5 runs on each benchmark. Overall, the results are consistent across multiple runs. Table 7 also shows the average and max thread depths for each task.





## Appendix D Task-general Prompting for ALFWorld

Table 8 shows the results when testing Thread with task-general prompting in ALFWorld. To implement the task-general prompt, we split the prompt into one set of examples for the main thread and a second set of examples for all other threads. The same sets of examples used for all tasks. GPT-3.5 achieves the same combined success rate with the task-general prompting as it does with the task-specific prompting. Further, while the performance of Llama-3-bb and CodeLlama-7b degrades with task-general prompting, it remains a significant improvement over task-specific prompting with prior methods (Table 2).

## Appendix E Error and Ablation Analysis

### Error Types

To identify what types of errors Thread reduces, we classify the errors that are responsible for each method's failures. Descriptions and examples of each error type are provided in Tables 9 and 10 for the agent tasks and Tables 11 and 12 for the QA tasks. We classified these errors through hand review of each failure case, where the first error type to occur within the failure case was identified as the error type responsible for the failure. Figures 4a, 4b, 5, and 6a show the error counts across different methods and tasks. We focus this analysis on the prompt-only methods to provide a more clear evaluation of how the novel aspects of Thread change performance relative to other methods that leverage few-shot learning.



the model to select from a fixed set of pre-defined sub-task handlers (e.g., Decomposed Prompting) or do not accommodate sub-task feedback when handling failures (e.g, ADAPT and TDAG). Figure 7 shows an example where a child thread, which is tasked with checking a cabinet for a plate, returns feedback saying the cabinet is closed. This is a common problem, where a child thread (or a sub-task handler/executor in Decomposed Promoting or ADaPT) return some unpredictable intermediate result instead of fully completing its task (which, in this case, involves opening the cabinet and checking for a plate). To address this with Thread, the parent thread spawns a new child thread with a modified sub-task specification (which includes opening the cabinet and checking if there is a plate) in order to prevent the new child thread from repeating the mistake made by the previous child. Prior methods do not allow for these dynamic, nuanced adjustments to the plan to accommodate the full range of possible intermediate feedback involved in complex tasks.

**Shortening action sequences reduces failures caused by misinterpreting environment feedback.** We see that errors in interpreting environment feedback are lowest with Decomposed Prompting and Thread. One potential explanation for this is that, by decomposing the task into more sub-components than other methods, Decomposed Prompting and Thread require each line of model generation to execute shorter sequences of consecutive actions. As a result, each line of model generation needs to manage shorter action-observation sequences and, therefore, is able to more accurately interpret and keep track of the information returned from the environment.

Figure 5 shows the number of times GPT-3.5 misinterprets environment feedback when executing action sequences of different lengths when using the different methods. By significantly reducing the number of cases where the model has to perform long action sequences, Decomposed Prompting and Thread reduce the number of interpretation errors.

### QA Tasks

**Task decomposition reduces failures that are caused by performing the wrong analysis.** In Figure 6a, we see that Decomposed Prompting and Thread show fewer failures caused by the model performing the wrong analysis for the given question. This is similar to what we see with the agent tasks where problem decomposition improves the planning. By separating the line of model generation that defines the plan (defining the analysis type) from the line of model generation that is responsible for executing the plan (completing the analysis), the model can more effectively accomplish both tasks.

**Flexible sub-task specification reduces failures that are caused by runtime errors.** We see that Thread shows the fewest failures that are caused by runtime errors. With Thread, when an error occurs, the error is returned to the parent thread, which can adapt its plan by modifying the sub-task specification and spawning a new thread (as discussed above). In addition, based on the ablation analysis (Figure 6b), we see that the variables used in Thread may also reduce failures due to runtime errors, especially with smaller models such as Llama-3-8b.

## Appendix F DataCommons QA and MIMIC-III ICU QA Benchmarks

We built the DataCommons QA benchmark utilizing data provided by Google Data Commons Guha (2019) with a focus on U.S. data. Data Commons contains publicly available data, aggregated from sources such as the Center for Disease Control and Prevention and the World Health Organization.

Generation of the benchmark started with the production of questions which fit the following criteria: 1) answerable by information provided by Google Data Commons, 2) broad enough to encompass information sampled between different locations (e.g., two counties), 3) lacking in subjectivity for the creation of a verifiable answer. The benchmark consists of question templates, which generate the questions of the benchmark, and ground-truth programs, which generate the answers to the questions of the benchmark. This format allows for the creation of a large quantity of diverse questions. Each question template consists of a sentence containing placeholders such as {variable_text} which, upon random sampling of a variable, are altered accordingly, facilitating the automated production of a variety of questions based on one question template. An example of a question template is as follows: 'Was {variable_text} in {entity1} increasing or decreasing from {timel} to {time2}?'. Time series data was sampled at the city, state, county, and country level for the United States. Data retrieval from Google Data Commons 

* [15] M. C. Bostrom, A. K. Goyal, and R. J. Snoek. The impact of the model on the performance of the models. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 237-240, Dec 2016.
* [16] D. P. Kingma and J. Ba. Adam: a method for stochastic optimization. _arXiv preprint arXiv:1912.08010_, 2019.
* [17] T. L. Chuang, Y. Bengio, and Z. Chen. Learning to align neural networks with deep learning. In _Advances in Neural Information Processing Systems (NIPS'09)_, pages 1359-1360, 2009.
* [18] H. Li, X. Zhang, and W. Sun. Deep residual learning for image recognition. In _IEEE International Conference on Acoustics and Speech (ICASSP)_, pages 135-140, May 2009.
* [19] I. Goodfellow, I. Guyon, and V. Vanhoucke. Generative adversarial nets. In _Advances in neural information processing systems_, volume 1, pages 1035-1040, 2016.
* [20] N. Mirza, F. Ozair, and E. Rabinovich. Imagenet classification via deep convolutional neural networks. In _Advances in neural information processing systems_, volume 1, pages 1035-1040, 2016.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l} \hline
**Error type** & **Examples** \\ \hline Inadequate plan & **ALFWorld:** The plan does not include cleaning the plate when the model is tasked with putting a clean plate on the table. The plan does not involve opening the cabinet to check for a plate when an unexpected sub-task result is returned (e.g., “The cabinet is closed”). **TextCraft:** The plan involves crafting an object before acquiring one of the precursors. The plan does not finish checking if a material (e.g., bamboo) can be retrieved from the environment when an unexpected result is returned (e.g., “Could not find valid recipe for bamboo.”). \\ Invalid action & **ALFWorld:** The model incorrectly specifies the action for examining an object with the desklamp. **TextCraft:** The model incorrectly states a crafting recipe. \\ Misinterprets environment feedback & **ALFWorld:** The model continues looking for a plate after the environment indicates “You see a plate 1”. **TextCraft:** The model proceeds as though it has successfully fetched bamboo despite the environment returning “Could not find bamboo”. \\ Other & **ALFWorld:** The model reaches the maximum number of actions allowed in the ALFWorld environment. **TextCraft:** The model runs out of context. \\ \hline \end{tabular}
\end{table}
Table 10: Examples of error types for agent tasks.



* [25] M. C. B. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "The \(\alpha\)-function of the \(\beta\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-406. ACM, 2008.
* [26] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-388. ACM, 2008.
* [27] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-407. ACM, 2008.
* [28] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-407. ACM, 2008.
* [29] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-410. ACM, 2008.
* [30] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-412. ACM, 2008.
* [31] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-413. ACM, 2008.
* [32] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-413. ACM, 2008.
* [33] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [34] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [35] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [36] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [37] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [38] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [39] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [40] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [41] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [42] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [43] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [44] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [45] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [46] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [47] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [48] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [49] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [50] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [51] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [52] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-416. ACM, 2008.
* [53] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-481. ACM, 2008.
* [54] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-481. ACM, 2008.
* [55] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-481. ACM, 2008.
* [56] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-481. ACM, 2008.
* [57] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-491. ACM, 2008.
* [58] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-491. ACM, 2008.
* [59] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-491. ACM, 2008.
* [60] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 10th International Conference on Computational Logic_, pages 379-491. ACM, 2008.
* [61] A. G. K. Wilson, J. R. Walsh, and D. W. Zeppenfeld, "A new \(\alpha\)-function for a given \(\alpha\)-function," in _Proceedings of the 1 

was conducted through the REST API provided by the website.

A "ground-truth program" was created for each question template to generate a verifiably correct answer for each question within the benchmark. These programs varied in length from 100-200 lines of code and analyzed data acquired from the Google Data Commons API in accordance with the constraints tailored by the question it was answering. For example, if a question template included a constraint about the years of data to be sampled, the ground-truth program was constructed to generate a year range for the answer. Code within the program would subsequently be created to filter and analyze the data acquired from Google Data Commons accordingly by applying whichever mathematical concept was required for the answer (e.g., correlation, linear regression, median, mean). Consequently, ground-truth programs formed the backbone of the benchmark through the creation of answers to the diverse questions which comprised the question template set. Figure 8 shows an example with the question "From 2015 to 2021, was the rate of asthma increasing faster in Boston or LA?".

The MIMIC-III ICU QA was created using the same approach as described above. However, questions were instead based on clinical time-series data provided by MIMIC-III Johnson et al. (2016). We outline all steps, along with the code, to reproduce the benchmarks at [https://github.com/philipmit/thread](https://github.com/philipmit/thread). We release the full dataset for DataCommons QA. Due to restrictions with MIMIC-III data access, we cannot directly release the dataset for MIMIC-III ICU QA. However, upon gaining access to MIMIC-III following the instructions outlined at [https://physionet.org/content/mimiciii/1.4/](https://physionet.org/content/mimiciii/1.4/), you can reproduce the full dataset using the code that we release.

```
From2015to2021wastherateofasthmaincreasingfasterinBostonorLA7 Atablewiththeasthmaretfrom2015to2019inBostonandLAisshownbelow:={table_dir}c= YearAtshmateRostonAsthmaretAL201510.98.4201611.38.5201710.99.0
201811.18.8202011.49.220211.139.6
``` Basedonthedataint(table_dir),theslopeofthestashmatreintBostonism>=0.06c= andlnLtalsim<=0.18c=.18c=.18c=.18c=.18c Therefore,weseesthatratofasthmaincreasedfastincalthanBoston. Afigurerepresentingthedatain(table_dir)isshowbelow:=*> ```

Figure 8: Question and example response with the DataCommons QA benchmark.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Error type** & **Examples** \\ \hline Retrieves wrong information & The model retrieves data on smoking prevalence of all people when asked specifically about the female population. \\  & Performs wrong analysis & The model compares values from 2014 to 2021 when asked to compare values from 2018 to 2021. \\ Misinterprets results of & The model indicates a variable is increasing despite its analysis showing that it is decreasing. \\   Runtime error & The model tries to concatenate an integer to a string. \\ Other & The model runs out of context. \\ \hline \hline \end{tabular}
\end{table}
Table 12: Examples of error types for QA tasks.

 

Prompts

Below, we provide examples of the prompt used for each benchmark. The given task or question is highlighted in yellow. The text corresponding to the spawning of child threads is highlighted in blue, pink, orange, and green. The full prompts for each task can be seen at [https://github.com/philipmit/thread](https://github.com/philipmit/thread).

### DataCommons QA

Question: From 2018 to 2021 was the percentage of people with diabetes increasing faster in Willows or Villa Ridge? Here is a table showing the percentage of people with diabetes during 2018 to 2021 for Willows and Villa Ridge!

\([table\_dir]<=\)

Based on the data in {table dir} from 2018 to 2021 the slope of the percentage of people with diabetes in Willows was >>0.3357<= and in Villa Ridge was =>0.1642<=

Therefore, the percentage of people with diabetes was increasing faster in Willows.

Final Answer: Windows
#END#

Here is a table showing the percentage of people with diabetes during 2018 to 2021 for Willows and Villa Ridge: ==

location1 = 'Willows' location2 = 'Villa Ridge' timeframe = '2018 to 2021' variable = 'percentage of people with diabetes' if the variable {timeframe for location number [ Location1 ], i 



### Mimic-III ICU QA

Question: Was the average systolic blood pressure of patient X from hour 10 to 20 of their ICU stay higher or lower than the average among all patients who expired in the hospital? Here is a table showing the systolic blood pressure of patient X from hour 10 to 20 of their ICU stay. (table_dir)!=

Based on the data in [table_dir] the average systolic blood pressure of patient X was \(\approx\)138.18<=

The average systolic blood pressure among all patients who expired in the hospital was \(\Rightarrow\)120.81<=

Therefore, the average systolic blood pressure of patient X from hour 10 to 20 was higher than the average among all patients who expired in the hospital.

Final Answer: higher

#END#

Here is a table showing the systolic blood pressure of patient X from hour 10 to 20 of their ICU stay. (table_dir)!=

patient_ID = 'X'

timeframe = 'hour 10 to 20'

variable = 'systolic blood pressure'

#END#

# The systolic blood pressure during hour 10 to 20 for patient X was \(\neq\)

patient_ID = 'X'

timeframe = 'hour 10 to 20'

time1= '10'

time2= '20'

variable = 'systolic blood pressure'

...

print('You have retrieved {data_patient_in_timeframe}.')

#END#

Based on the data in /path/to/data_table_txt the average systolic blood pressure of patient X was \(\approx\)

table_dir = "path/to/data_table_txt"

variable = 'systolic blood pressure'

table = pd_read_csv(table_dir, sepa^-%)

# > print(table) >>

...

variable_col_name = 'Systolic blood pressure'

y = table[variable_col_name] y_mean=np_mean(y) print(y_mean)



#END#

The average systolic blood pressure among all patients who expired in the hospital was variable = 'systolic blood pressure" table = pd_read_csv(PATIENT_DATA_DIR, sep=~$)
print(table) => ... ... ... filter_col_name = 'MORTALITY_INHOSPTIAL' variable_col_name = 'Systolic blood pressure' target_patient_group = table[table|variable_col_name]==1] y = target_patient_group|variable_col_name] y_mean=np.mean(y) print(y_mean) #END#

### ALFWorld

You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine I, a counterton I, a dmingitable 3, a dmingitable 2, a dmingitable 1, a drawer 1, a fridge I, a garbageecan 1,a microwave 1, a sidetable 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster I. Your task is to clean some apple and put it asidetable! target_location_ID = 'sidetab 1' cleaning_location_ID = 'sinkbasin 1' likely_location_IDs_for_obj = ['fridge 1', 'countertop 1', 'cabinet 4', 'cbinet 3', 'caibinet 2', 'cabinet 1', 'diningtable 3', 'diningtable 2', 'diningtable 1', 'siderable 1', 'sinkbasin 1', 'garbageecan 1', 'drawer 1', 'stoveburner 4', 'stoveburner 3', 'stopburner 2', 'stoveburner 1'] obj = 'apple' I need to perform the following steps: First. Ineed to find and take the {obj}_ISO have found and taken the {obj_ID}.<= Next.I need to clean the {obj_ID} with the {cleaning_location_ID}.<= You have cleaned the {obj_ID} with the {cleaning_location_ID}.<= Finally, I need to put the {obj_ID} in the {target_location_ID}.>You have put the {obj_ID} in the {target_location_ID}.<= #END#

Ineed to find and take the apple.= obj = 'Apple' My sub-task is to find and take the {obj}. I will check the locations in {likely_location_IDs_for_obj} one by one. location_i = likely_location_IDs for_obj[0] Indeed to bejeck if there is the {obj} in the {location_i}.= There is no {obj} in the {location_i}.<= Location_i = likely_location_IDs_for_obj[1] ... ... I need to check if there is the {obj} in the {location_i}.=>There is no {obj} in the {location_i}.<= location_i = likely_location_IDs_for_obj[2] I need to check if there is the {obj} in the {location_i}.=>There is no {obj} in the {location_i}.<= location_i = likely_location IDs for_obj[3] I need to check if there is the {obj} in the {location_i}.=>There is no {obj} in the {location_i}.<= location_i = likely_location IDs for_obj[4] I need to check if there is the {obj} in the {location_i}.=>There is no {obj} in the {location_i}.<= location_i = likely_location IDs for_obj[5] I need to check if there is the {obj} in the {location_i}.=>There is no {obj} in the {location_i}.<= location_i = likely_location IDs for_obj[10] I need to check if there is the {obj} in the {location_i}.=>You have found and taken the {obj_ID} from the {location_i}.<= print('You have found and taken the {obj_ID}.') #END#

Ineed to check if there is the apple in the {ridge I}.<= location_i = 'fridge 1' obj = 'apple' My sub-tasks is to check if there is the {obj} in the {location_i}. \(>\geq\) go to {location_i} =>The {location_i} is closed.<= > open {location_i} =>You open the {location_i}. The {location_i} is open. In it, you see a lettuce 2, a mug 2, a potato 2, and a tomato 1.<=

obj_IDs_found = ['let lettuce 2', 'mug 2', 'potato 2', 'tomato I'] obj_number = "There is no obj_number' obj_ID = "There is no obj_ID' print('There is no {obj} in the {location_i}.')
#END#

Ineed to clean the apple 3 with the sinkbasim I:=5 cleaning_location_ID = 'sinkbasim I' obj_ID = 'apple 3' My sub-task is to clean {obj_ID} with the {cleaning_location_ID}. > go to {cleaning_location_ID} \Rightarrow?On the {cleaning_location_ID}, you see nothing.<= >clean {obj_ID} with {cleaning_location_ID} =>You clean the {obj_ID} using the {cleaning_location_ID}.<= print('You have cleaned the {obj_ID} with the {cleaning_location_ID}.') #END#

**I need to put the apple 3 in the sideltable 1.**

target_location_ID = 'sidetable 1' ob_ID = 'apple 3' My sub-task is to put the {object_ID} in the {target_location_ID}. > go to {target_location_ID} \Rightarrow?On the {target_location_ID}, you see a cup 1, a lettuce 1, a peppershaker 3, a potato 1, and a saltshaker 1.<= > put {obj_ID} in/on {target_location_ID} \Rightarrow?You put the {obj_ID} in/on the {target_location_ID}.<= print('You have put the {obj_ID} in the {target_location_ID}.') #END#**

### WebShop

Instruction:1 would like a 3ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars obj_attributes = ['3 ounce bottle', 'Bright citrus', 'For sensitive skin'] max_price = 50.00 obj = 'deodorant' I need to perform the following steps: First, Ineed to retrieve search results for [obj] that are less than [max_price] dollars with the attributes {obj_attributes}.~you have retrieved {results_under_max_price}.<= Next, Ineed to identify the item in {results_under_max_price} that matches the most attributes {obj_attributes}.~you have identified {item_to_purchase}.<= Finally, Ineed to purchase {item_to_purchase} with {obj_attributes}.~you have purchased {item_to_purchase}.<= #END#

Ineed to retrieve search results for deodorant that are less than 50.00 dollars with the attributes: {3 oune bottle}.; Brighteitrus! Forsensitive skin I:=5 obj = 'deodorant' max_price = 50.00 obj_attribute1 = '3 ounoette' obj_attribute2 = 'Bright citrus' obj_attribute3 = 'For sensitive skin' > search[{obj}; {obj_attribute1}; {obj_attribute2}; {obj_attribute3}] => [Back to Search] Page 1 (Total results: 50) [Next >>] [B08KBVJ4XN] Barrel and Oak - Aluminum-Free Deodorant, Deodorant for Men, Essential Oil-Based Scent, 24-Hour Odor Protection, Cedar & Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 1 oz, 2-Pack) $15.95 [B078GWRC1IJ] Bright Citrus Deodorant by Earth Mama I Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce $10.99 [B078GTKVXY] Ginger Fresh Deodorant by Earth Mama I Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce $60.99 <= | I will now filter for results that are less than {max_price}. results = ['Bo8kBVJ4XN', 'Bo78GWRC1J', 'Bo78GTKVXY'] results_prices = [15.95, 10.99, 60.99] 

results_under_max_price = [result for result, price in zip(results, results_prices) if price < max_price] print('You have retrieved {results_under_max_price}.')
#END#

**I need to identify the item1[B08KBVJ4XN; B078GWRCIJ] that matches the most attributes: [3 outence bottle], Bright citrus". For sensitive skin]:=95item2=B078GWRCIJ~objattributes=['3 outence bottle]=['3 outence bottle', 'BRight citrus', 'For sensitive skin'] My sub-task is to identify the item in [{item1}, {item2}] that matches the most attributes: {obj_attributes}.

**I need to count the number of attributes in {obj_attributes} that {item1} has_existsThis item has 1 attribute.c=1 need to count the number of attributes in {obj_attributes} that {item2} has_existsThis item has 2 attributes.c=items=[{item1}, {item2}] item_attributes=[1, 2] item_to_upreduce= next(iter(items[item_attributes.index(max(item_attributes))])) print('You have identified {item_to_upreduce}.') #END#******

* Bright citrus,
* Tor sensitive Skin} that B08KBVJ4XN has_exists_
 

I can now select [Buy Now].

> click[Buy Now] =>You have clicked Buy Now.<= print('You have purchased {item_to_purchase}.') #END#

### TextCraft

Crafting commands!

craft 3 dark oak sign using 6 dark oak planks, I stick|

craft 4 dark oak planks using I dark oak log|

craft 1 stick using 1 planks|

craft 4 stick using 2 bamboo|

craft 4 oak planks using 1 oak log|

craft 1 dark oak fence using 2 stick, 4 dark oak planks|

craft 1 warped starts using 6 warped plans|

craft 3 oak sign using 6 oak planks, I stick|

Goal-carfaltdarkoakignign.

craft_command_list = ['craft 3 dark oak sign using 6 dark oak planks, 1 stick', 'craft 4 dark oak plans using 1 dark oak log', 'craft 1 stick using 1 planks', 'craft 4 stick using 2 bamboo', 'craft 4 oak planks using 1 oak log', 'craft 1 dark oak fence using 2 stick, 4 dark oak planks', 'craft 1 warped stairs using 6 warped planks', 'craft 3 oak sign using 6 oak planks, I stick')

craft_command_item_list = ['dark oak sign', 'dark oak banks', 'stick', 'stick', 'oak planks', 'dark oak fence', 'warped stairs', 'oak sign']

target_item = 'dark oak sign'

target_item_count_total = 1 idx = craft_command_item_list.index(min{x for x in craft_command_item_list if target_item in x}, key=len))

target_craft_command = craft_command_list[idx]

To result [target item], I need to perform the following action until:have at least [target_item count_total] [target_item];

target_craft_command = You have completed the action.<=

#END#

Ineed to perform the following action until? Have at least I dark oak sign: craft 3 dark oak sign using 6 dark oak plants, I stick|

target_item = 'dark oak sign'

target_item_count_total = 1 target_craft_command_result_count = 3 precursor1 = "dark oak planks"

precursor2 = 'stick'

precursor2_count_command = 1 Since I need at least 1 {target_item} and each action produces 3 {target_item}, The number of times I need to perform the action is ceiling of 1/3, which is 1.

target_craft_command_reps = 1 precursor1_count_total = precursor1_count_command * target_craft_command_reps precursor1_count_total = precursor2_count_command * target_craft_command_reps Next, I need to get or craft [precursor1].

To start, first need to check if I can get [precursor1], count_total [precursor2]. ==You cannot get the material.<=

Since I cannot get [precursor1], I need to craft it.

idx = craft_command_item_list.index(min{x for x in craft_command_item_list if precursor1 in x}, key=len))

precursor1 = craft_command_item_list[idx]

precursor1_craft_command = craft_command_list[idx]

To craft [precursor1], I need to perform the following action until I have at least {precursor1_count_total} {precursor1}:

precursor1_craft_command = yOu have completed the action.<=

Next, I need to get or craft [precursor2].

To start, I first need to check if I can get [precursor2_count_total] {precursor2}. ==You cannot get the material.<=

Since I cannot get [precursor2], I need to craft it.

ids = craft_command_item_list.index(min{x for x in craft_command_item_list if precursor2 in x}, key=len))

precursor2_craft_command_item_list[idx]

precursor2_craft_command = craft_command_list[idx]

To craft [precursor2], I need to perform the following action until I have at least {precursor2_count_total} {precursor2}:

\begin{tabular}{l l} Precursor2_craft_command = yOu have completed the action.<=

Finally, I will perform the action 1 time.

> craft {target_craft_command_result_count} {target_item} using {precursor1_count_command} {precursor1},

precursor2_count_command {precursor2} =Yaafled 3 minecraft:{target_item}.<=

print('You have completed the action.') #END# 

If first need to check if I can get 6 dark oak planks =

get_item ='dark oak planks'

get_item_count_total = 6

p get[get_item_count_total] {get_item} =>Could not find {get_item}.<=

I cannot get the {get_item}.

print('You cannot get the material.')

#END#