StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows
Authors: Yiran Wu
Tianwei Yue
Shaokun Zhang
Chi Wang
Qingyun Wu
Venue: COLM (2024)
Keywords: workflows, state machines, LLM, prompt engineering
TL;DR: We propose StateFlow, a novel LLM-based task-solving paradigm that conceptualizes complex task-solving processes backed by LLMs as state machines.

Abstract: It is a notable trend to use Large Language Models (LLMs) to tackle complex tasks, e.g., tasks that require a sequence of actions and dynamic interaction with tools and external environments.
In this paper, we propose StateFlow, a novel LLM-based task-solving paradigm that conceptualizes complex task-solving processes as state machines.
In StateFlow, we distinguish between "process grounding” (via state and state transitions) and "sub-task solving” (through actions within a state), enhancing control and interpretability of the task-solving procedure. A state represents the status of a running process. The transitions between states are controlled by heuristic rules or decisions made by the LLM, allowing for a dynamic and adaptive progression.
Upon entering a state, a series of actions is executed, involving not only calling LLMs guided by different prompts, but also the utilization of external tools as needed. 
 Our results show that StateFlow significantly enhances LLMs' efficiency. For instance, StateFlow achieves 13\% and 28\% higher success rates compared to ReAct in InterCode SQL and ALFWorld benchmark, with 5$\times$ and 3$\times$ less cost respectively. 
We also show that StateFlow can be combined with iterative refining methods like Reflexion to further improve performance.

Published as a conference paper at COLM 2024
StateFlow: Enhancing LLM Task-Solving through State- Driven Workflows
Yiran Wu
Tianwei Yue MathGPTPro tianwei. Llanee..yue@matiggpppo..co
Pennsylvania State University yiran. www.ww.wpuuuuoooo deu
Shaokun Zhang
Chi Wang
Pennsylvania State University shaokun. whw.g@ssuaan@@gdddeeeco
Microsoft Research Redmond wang. www.ccccosooot.ccom com
Qingyun Wu
Pennsylvania State University qingyun. wuêpsu. deu
Abstract
is notable trend use Large Language Models (LLMs) to tackle complex tasks, e.g,, tasks that require a sequence of actions and dynamic interaction with tools and external environments. In this paper, we propose StateFlow, a novel I.LM-based task-solving paradigm that conceptualizes complex task-solving processes as state machines. In StateFlow, we distinguish between "process grounding" (via state and state transitions) and "sub-task solving" (through actions within a state), enhancing control and inter- pretability of the task-solving procedure. A state represents the status of running process. The transitions between states are controlled by heuristic rules decisions made by the LLM, allowing for dynamic and adaptive progression. Upon entering a state, a series of actions executed, involving not only calling LLMs guided by different prompts, but also the utilization of external tools as needed. Our results show that StateFlow significantly enhances LLMs' efficiency. For instance, StateFlow achieves 13% and 28% higher success rates compared to ReAct in InterCode SQL and ALFWorld benchmark, with 5x and 3x less cost respectively. We also show that StateFlow low can be combined with iterative refining methods like Reflexion further improve performance.
Introduction
LLMs have increasingly been employed to solve complex, multi-step tasks. Specifically, they
have been applied to tasks that require interactions with environments (Yang et al.) 2023a
Yao et al. 2022a Shridhar al. 2020 Zelikman et al., 2022) and those tasks that beneht
from utilizing tools such as web search and code execution (Mialon et al. 2023 Wu et al.
2023b Wang al. 2023c). In approaching these tasks, there is typically desired worktlow,
or plan of actions based on heuristics that could improve the efficiency of task solving (Kim al. 2023 Wu al. 2023a). common practice in the context of LLMs, such as ReAct (Yao et al. 2022b) and the vast customization of GPTSs is to write a single prompt that instructs the models to follow a desired procedure solve the task (Dohan et al. 2022). The L.M is called iteratively with the same instruction, along with previous actions and feedback from tools/ environments. This relies on LLMs' innate capability to determine the current task-solving status and perform subsequent actions autonomously. Despite the impressive abilities of LLMs, it is still unrealistic to expect LLMs to always judge the status of current progress correctly. It is also almost impossible to reliably track these judgments and their decisions of subsequent action trajectory. Given these considerations, we pose the research question: How can exert more precise control and guidance over LLMs?

---

Published as a conference paper at COLM 2024
Init P
Observe M--E
Select
SELECT
Verify
Submit
End
Exit
Bash Task
SELECT
Error
Error M.--E
Prompter
Execution
My LM with instruction T
Init
Solve -E
Verify
Submit
End
M,-E
Exit
SQL Task
The StateFlow models for the SQL and Bash task. Init and End state are basic components of state machines, and states like Observe, Solve, Verify, Error can be adapt- able across various tasks. When reaching a state, a sequence of output functions defined is executed (e.g., Mj E means to first call the model and then call the SQL, execu- tion). Execution outcomes are indicated by red arrows for failures and green for successes. Transition to different states is based on specific rules. For example, success 'Submit' command, the model transits to End state.
Figure 1:
In this paper, we propose StateFlow, a new framework that models LLM workflows as state machines. Finite State Machines (FSMs) (Mealy] 1955 Moore et al.) 1956 used as control systems to monitor practical applications, such as traffic light control (Wagner et al. 2006). A defined state machine is model of behavior that decides what to do based current status. A state represents one situation that the FSM might be in. Drawing from this concept, want use ISMs to model the task-solving process of LLMs. When using LLMs to solve a task with multiple steps, each step of the task-solving process can be mapped to state. For example, to solve the InterCode (Yang et al., 2023a) SQL task, a desired procedure to first gather information about the tables and columns in the database, then construct query to retrieve required information, and finally verify the task is solved and end the process. We can convert this workflow to a set of states (See upper left of Figure Within each state, we define a sequence output functions, which will be called upon entering the state. The output functions would take in the context history and output new context to be appended to the history, which can be a tool, an LLM with a specific instruction, or prompter. Based the current state and context history, the StateFlow model would determine the next state to transit to. The task-solving process progresses by transitioning through different states and calls to corresponding output functions, and ends until final state is reached. Thus, StateFle low enhances control over the task-solving process and seamlessly integrates external tools and environments.
We evaluate StateFlow on the SQL task and Bash task from the InterCode (Ynng et al. 2023a benchmark and the ALFWorld Benchmark (Shridhar al., 2020). The results demonstrate the advantages of StateFlow over existing methods in terms of both effectiveness and efficiency. With GPT-3.5, StateFlow outperforms ReAct by 13% on InterCode SQL task and 28% on ALFWorld, with 5x and 3x less LLM inference cost respectively. StateF low is orthogonal to methods that iteratively improve future attempts using feedback based on previous trials (Shinn et al. 2023 Madaan et al.) 2024 Prasad et al. 2023).. Notably, we show that StateF low can be combined with Reflexion (Shinn et al. 2023), improving the success rate ALFWorld from 84.3% to 94.8% after 6 iterations.
Our main contributions are the following: (1) We introduce StateFlow, a paradigm that models LLM workflows state machines, allowing better control and efficiency in LLM- driven task solving. We provide guidelines on how build with the State low framework and illustrate the building process through case study. (2) We three different tasks to
2

---

Published as a conference paper at COLM 2024
illustrate the effectiveness and efficiency of StateFlow, with improvement in performance and a 3-5x cost reduction. We also perform an ablation study to provide deeper insights into how different states contribute to the performance of StateFlow. (3) We show that StateFlow can be combined with iterative refining methods to further improve performance.
2 Background
Finite-state Machines We first introduce state machines, which we will to formulate our framework. A finite state machine (automaton) is a mathematical model of a machine that accepts a set of words or string over an input alphabet (Hopcroft al. 2001 Carroll
& Long 1989), where read of symbols would lead to state transitions. The automaton would determine whether the input is accepted or rejected. A basic Deterministic Finite- state Machine (DFSM) is a Deterministic Finite-state acceptor, usually used as language recognizer that determines whether an input ends in an accept state. An acceptor not suitable in our modeling of LLM generations, where we want to determine actions to be performed and produce outputs between states. Instead, we base our model on transducer finite-state machine, which is sextuple (I,r,S, 8, (o) (Rich el. 2008), where abd are the input and output alphabet (finite non-empty set of symbols), 5 is a finite
non-empty set of states, the output function, So is the initial state, 8 is the state transition function (6 : S x + S), and F the set of final states.
3 Methodology
In this section, first define the StateFlow model. We then provide general guideline for constructing StateFlor low model and illustrate with a detailed case study.
3.1 StateFlow
In a finite state machine, state carries information about the machine's history, tracking how the state machine has reached the present situation (Wagner et al., 2006). It feasible to conceptualize the task-solving process with LLMs as state machine. Different from traditional FSMs, StateFlow doesn't have the concept of input tape but solely depends on the context history, which is cumulative record of all past interactions. StateFlow employs a set of instructions = T, T2,- to guide the language model generation at different
states. This is equivalent constructing a set of LLM agents p with a specific instruction T; This dynamic prompting approach ensures that the language model receives the most relevant guidance at each state, improving its ability to focus on a specific step. Following the definition of finite state machines, we formulate a StateF low model to be sextuple (S, So, F, (2) and explain each of the components under the LLM scenario:
States S A state encapsulates the current status of running process, essentially an abstrac- tion of the context history. Upon entering state, a predefined set of actions is executed. For example, entering an error state implies the process encounters an issue, triggering the execution predetermined error-handlig actions.
Initial state So The process begins at the initial state when receiving the input task question..
Final States A set of final states when the process is terminated, which is a subset of S.
Output I We define I to be an infinite set of messages (unit of text) consisting of prompts P, language model responses C, and feedback from tool/ /eniironment O: T= (P, O), which represents all possible messages that can be generated within StateFlow. We further define context history to be 1 which is a list of messages that have been generated. $ x 1 could be view static prompts P that will added to the context directly from instructions T that are used upon calling an LMM
snapshot of a running StateFlor fow (Rich et al., 2008). Here we distinguish
State transition& & Based on the current state and context history, the transition function would determine which state to go to (8 S r" - -). This could mean string matching of
3

---

Published as a conference paper at COLM 2024
the last input, for example, checking if 'Error' is in the message from code execution. We can also explicitly employ an LLM to check for conditions and determine what the next state.
Output Functions Ω Here we define Ω to be set of output functions, where func- tion w takes the whole context history and generates an output (w I + 1). The output function can be an LLM, tool call, or a prompter function that returns a static prompt (e.g., P, E and Mj in Figure []. The generated response will then be added to the context history.
Algorithm 1 StateFlow
Require: task Q max transitions M, model
(0, F, 5,,,1, Q), we define s.outputs to be list of functions [wi,...,wjj] wj E Q for
each S S
1: D +
2: Counter f- 0
The process starts at so when task Q is ap- pended to the context history I and ends when reaching one of the final states (See Algorithm[]] []. We also use a counter that de- fines maximum turns of transitions to pre- vent infinite loops. The process returns the exit state and the whole context history in the end.
3: 4 50
4: while do
5:
for w in s.outputs do
6:
do I* <- I (T*)
7:
end for
8:
s s 1-8 r*)
9:
return s, s if Counter 2 M
end while
11: return s,
3.2 Deployment Guideline
In this section, we provide guidelines for de- ploying StateFlov low models, with reference to the case study discussed in Section 3.3 In general, StateFlow is designed for tasks that require a designated process to solve. Essentially, creating state machine involves transforming abstract control flow or human reasoning into formalized logical model, grounded in comprehensive understanding of the task at hand.
Defining States To define the states, we start with identifying an ideal workflow of a given task. A state should represent distinct phase or step in the process, defined with enough granularity to capture key milestones and decision points. With the basic workflow in mind, need to think about possible situations during the process. Specifically, handling failures is an important part of the state machine, where a Error state is commonly used to handle failures. A set of fine-grained states might lead to better control over problem-solvine process. For example, it is possible to identify different types of errors and use different ways to handle them, but this adds complexity in defining the model trade-off.
Defining output functions Within each state, we need to define a set of outputs. In practice, we need to identify the set of tools we will use, and what instructions we should send to the LLM. For example, in the Solve state defined for the bash task, we first send an instruction that asks for bash command, call the model, and then execute the bash command. This is a typical sequence of sending instructions to LLMs and then utilizing tools.
Defining Transitions We identify two possible state transitions: A static string matching with the LLM responses or tool executions. For example, the tool execution might return specific string like "execution failed", which can be used to determine the transition. Also, it is common to instruct LLMs to follow a certain template to generate outputs, so we can extract strings from the response. 2. Use LLM for an explicit checking the current status when previous context is too random to be used for string matching. For example, we can send the context history to a model to ask whether a given problem is solved.
3.3 A Case Study of StateFlow Design on SQL
We use the SQL task from the InterCode (Yang et al. 2023a) benchmark to illustrate how to identify a workflow and convert it to a StateFlor model. The benchmark is evaluated with ReAct (Yao et al.) 2022b), few-shot prompting that requires annotations of example trajectories of actions, thoughts, and observations. These trajectories are human-crafted, and they reflect how one thinks is a good process solve the problem.
4

---

Published as a conference paper at COLM 2024
See Figure 2 for one ReAct trajectory for the SQL task: (1) the process starts with a 'SELECT' query and results in error. (2) At an error, the next thought is to explore the tables, so the SHOW TABLES command is executed to retrieve all tables. (3) After getting the tables, the next step to explore the "highschoolers' table with the 'DESC' command. (4) With knowing what the table contains, the next thought is to use the select query to solve the question. (5) Finally, it confirms the output contains relevant info, and submits. The trajectory demonstrates what to do based on previous history, which is similar to state transitions in StateFlow While the trajectory starts with 'SELECT' query but results in an error, we believe a better workflow would be to first explore the tables and use the 'SELECT' command. Based on this, we construct the states be: Init -> Observe -> Solve -> Verify -> End (See Figure In each state, the model instructed to perform a specific action. For example, we ask the model to submit if the task is verified in state Verify fy and explore tables at an error in state error.
Question: What are names and grades each high schoolar?
Thought 1: should write SQL command that selects the name and grade fields from a table about high schoolers. Action executafSELEC name, grade FROM high schhollss Observation Error: Table "network, 1.high schoolers' doesn't exist
Thought
doesn't exist, check what tables are avaiable use.
sighless
Action execute[SHOw TABLES] Observation 2: ((friand)), (highschooler)), ('Ikasj]
Thought 3: There is table called "highschoolers" : should chack what information this table contains. Action exccuta[DESC highschaoler] Observation [MD,) ...., ('namee. ...,, ['grade,..]]
Thought 4:1 can now write SQL command that answers the question. Action 4: exccute(SELEC) name, grade FROM highschaoler Observation 4: [[ohhni 12), I'llalyy 10),
Thought The cusput contains the and answering the quastion. submit this the Action
Figure 2: A ReAct few-shot example for the SQL task. From the example, can abstract a general workflow to solve the problem.
Experiments
4.1 InterCode Benchmark
We first experiment with 2 tasks from the InterCode benchmark (Yang et al. 2023a): (1) SQL: The InterCode-SQU. adapts the Spider dataset for MySQL, containing 1034 task instances. For each task, a MySQL interpreter is set up with all relevant tables within docker container. dataset has 200 task instances curated from the NL2Bash
(2) Bash: The InterCode-Bate dataset. We use the same hyperparameters Zhang et al. (2023a b 2024b); Zheng et al. (2023) for both two benchmarks. Specifically, we allow a max of 10 rounds of interaction with the environment. We evaluate with OpenAl GPT-3.5-Turbo and GPT-4-Turbo (both with the 1106 version) and the temperature is set to
Baselines. We compare StateFlow with two prompting strategies used in the InterCode benchmark. (1) Plan & Solve (Wang et al. 2023b): two-step prompting strategy to first ask the model to propose plan and then execute it. (2) ReAct (Yao et al. 2022b): shot prompting method that prompts the model to generate thoughts and actions. Additionally, since we observed an ideal workflow for the SQL task in Section 3.3 we edit the ReAct prompt used in the benchmark accordingly to see it already improves performance, named ReAct.Refined, and evaluate it on the SQL task (See Table and 21 for details).
Metrics. We present metrics provided by the benchmark and we also report the LLM usage of each method. Success Rate (SR): task is considered success only if the reward is 1. Error Rate: percentage of commands that fails. Turns: number of interactions with the environment. Cost: the cost of calling an LLM API in US dollars.
StateFlow Setup. For both tasks, we prompt the model to generate thought and action at each turn. Each prompt consists of 3 components: (1) instruction: details of what the LLM should perform at the current state; (2) examples: partial thought or action steps from the ReAct examples as demonstrations; (3) response format: explain the thought-action template. These prompts are put in the system message of each LLM agent and are not visible to other agents. SQL: We construct 6 states for the SQL task (See Figure [ and Section for case study). In Init, we always execute the 'SHOW TABLES' command. In state Observe, Solve, Verify, Error, when the execution output is an error string, we will transit to state Error. In any of states Solve, Verify, Error, successful DDESC' will
5

---

Published as a conference paper at COLM 2024
Question: What is the rank of the tallest poker player?
Last Execution Fails
Error
Init
Error
Prompter:
Thought:
Model: Instruction (Ti): Maase carefully
message
Action: exeCute[SHOW TABLES]
wrong.
Execution: (Ppeople). (poker_ pplayer])]]
Thought: The error message indicates that the Y'Heightl" not...
State transition:
ALWAYS
Observe
Action: execute(DESC people]
SQL: Observation: (1Poople_JD, Int, (Nationality, -.,
Observe
State transition:
c Last action is DESC
Solve
Model: Instruction (h)): the DEScRIBE (taole, name) or DESC (table rame]
Solve
Thought: need to understand the structure the poker_player table to see...
Model:
Instruction (Tit Based on the understanding of the sables and the problem, formulate
SELECT the
Action: Grecute[DESC poker_player]
question
Execution: [/Pover_Player_JD. ...), (People_ID...
Thought: should
SOL command money rank the...
State transition:
Last action is DESC
Action: executa[SELECT p Money_Rank FROM poker_player JOIN paople ON ...]
Observation: ((68.0,)
Solve
Model: Instruction (Ti) Basad on the understanding of the taeles and the with SEL
State transition:
C Last action is SELECT
Verify
problem, formulate
question
Thought:
command
Verify
money rank of the...
Model:
Instruction (Ti): tarefuly theck output answers the question Make sure the outpptt only display heids that the problem ssks for.
Action: execute (SEL ECT Money_Rank FROM ppkyer ORDER BY Earnings DESC LIMIT 1]"
Thought: The output displays the money rank the tallest poker player...
[[58.0,)
submit
State transition:
SELECT Verify
SQL: Observation: ((69.0,)
Verify
State transition:
+ Last action is submit
End
Model: Instruction (Ts): Carefully check the output answers the question. ousput display feeks
for.. Thought: The output does not spectiically address the tallest poker player... need to modify...
End Exit
execute(SELECC Money_Rank FROM poker_playor
Earnings DESC, Haight
DESC LIMIT
Init
Observe
Solve
Verify
End
SQL: Observation: Error executing Unknown colurn
'Height'
Error
Figure 3: An example of the StateFlow execution for the SQL task. In this example, StateFlow runs through all states to reach final answer.
GPT-3.5
GPT-4
SR
Turns!
Error
Cost.
SR1
Turns |
Error,
Cost |
Plan & Solve
47.68
4.31
12.5
2.38
56.19
5.39
1.79
44.7
ReAct ReAct_Refine
50.68
5.58
16.3
17.7
60.16
5.26
3.87
147
57.74
5.47
3.82
18.1
57.93
5.01
2.49
141
StateFlow 63.73
5.67
6.82
3.82
69.34
5.11
1.89
36.0
Table 1: Evaluation of the Intercode SQL dataset with GPT-3.5 and GPT-4. Best metrics of each model is in Bold. Second-best is Underlined.
transit to Solve; successful 'SELECT' will transit to Verify. In Verify, use LLM to self- evaluate, which is proven useful by Weng et al. (2022); Xie et al. (2023). Bash: For the bash task, we define StateFlow model consists of states Init, Solve, Verify, End, Error. The states are similar to SQL, and the transition only depends on whether the execution is successful not (Figure [T.. See details in Appendix A.1

---

Published as a conference paper at COLM 2024
GPT-3.55
GPT-4
SR
Turns
Error
Cost
SR
Turns
Error
Cost
Plan & Solve
23.5
4.98
25.8
0.74
20.5
5.15
21.0
9.59
ReAct
32.5
5.52
13.2
3.28
31.5
3.86
9.90
20.40
StateFlow
36.0
3.90
8.74
0.63
39.0
2.95
7.85
5.02
Table 2: Evaluation of the InterCode Bash dataset with GPT-3.5 and GPT-4. Best metrics of each model is in Bold. Second-best is Underlined.
Result and analysis on SQL On GPT-3.5, our refined ReAct version increases the success rate by 7% over the original ReAct prompt. But StateFlow can further improve over the refined version by 6%, with 5x less cost. We note that the big difference in cost mainly comes from the prompt token use. The ReAct prompt has 2043 tokens with 4 example trajectories, while the longest instruction from StateF low has only 400 tokens. The LLM is called iteratively with the whole prompt, thus the difference in cost accumulates. Compared to Plan & Solve, StateFlor uses 1.6x more cost but improves the SR by 17%. ReAct Refined and our methods follow similar workflow and have low error rates. On GPT-4-Turbo, StateFlow can improve over ReAct by 10% in SR but has 3x less cost. We further look into results from different levels of difficulties and found that the success rate drops significantly for harder tasks (Details in Appendix A.2). StateFlow also shows a greater improvement over ReAct on hard and extra hard tasks, which often require complex joins across multiple tables. We found that state Error in StateFlow is crucial, it is encountered in 20% of extra hard tasks compared to only 9% in easy tasks, highlighting its role in improving performance on challenging queries.
Result and analysis Bash. On the bash task, StateFlow outperforms other methods while efficiently interacting with the environment. Switching to GPT-4-Turbo has little effect on the methods, where the two baselines even suffer from decrement in accuracy. While the success rate is low, the average reward is high as 0.8 (See Table 7 in Appendix). Our investigation shows that 58.5% of the bash tasks have a positive reward greater than 0.5, while only 0.5% of the failed tasks have positive reward greater than 0 in the SQL task. This is because bash task sometimes consists of two requests (retrieve information or configure file), making it harder to completely solve task. Please see more details and results in Appendix A.2 and
Ablation of States. To understand how dif- ferent states contribute the accuracy in StateFlow, we perform additional ablations and analysis with the SQL task (See Table (1) We remove the Observe state. Note that in the original prompt for state Solve, we also instruct the model to call 'DESC' if nec- essary, so the model can still explore tables, but not with explicit state and instruc- tion to perform this action. (2) We remove the error state and rely on the verify state to correct mistakes. (3) We remove the verify state and add sentence in Solve to prompt the model to submit when finished. The table shows that removing any of the states results in a drop in performance. When Verify state is removed, StateFlor low has the lowest cost and error rate, matching the idea that more corrections to the results will be performed with an explicit Verify state. Removal of the Error state leads to drop of 5% in SR, and an increase in turns, error rate, and cost, showing that the Error state plays an important role in the workflow. Removal of the Observe results in the lowest SR and highest cost, showing that 'Observe' is the most important state.
SR
Turns
Error %1
Cost
% f
$
StateFlow
63.73
5,67
6.82
3.82
No. Verify
62.28
5.18
5.96
3.68
No_ Error
58.80
5.72
11.6
4.05
JbsOvve
57.83
6.00
17.0
4.64
Table 3: Ablation of states the InterCode SQL dataset with GPT-3.5-Turbo. Best metrics in Bold. Second-best is Underlined.
7

---

Published as a conference paper at COLM 2024
Error M
Process
'Heat/CoolCClean"
Pick
Wrogg
M-+E
Processed
Init
Plan
Pick
"Pick/Pick
Put M->E
Done
End
M--E
Pick
Exit
Look"
Done
FindLamp
Found
UseLamp E
E: Environment
Mj instruction
Figure 4: The StateFlow model for ALFWorld. For Plan, we call the LLM directly. For other states (except Init and End), we first call LLM with an instruction and then call the environment to get feedback. In state Pick, when the correct object is picked, we transit to different states based on task type. For states Pick, Process, FindLamp, UseLamp, Put, we stay in the current state if the corresponding task is not completed, represented by gray semi-circle arrows.
Pick
Clean
Heat
Cool
Look
Pick 2
All
Cost $ 4.
ReAct
83.3
36.6
53.6
58.7
63
41.2
55.5
6.6
ALFChat (2 agents)
87.5
60.2
44.9
65.1
38.9
43.1
58.2
6.9
ALFChat (3 agents)
84.7
60.2
69.6
77.8
68.5
41.2
67.7
6.1
StateFlow
91.7
83.9
85.5
79.4
92.6
62.7
83.3
2.6
Table 4: Performance and cost of StateF low and other methods on ALFWorld benchmark with GPT-3.5-Turbo. We report average success rate of 3 attempts.
4.2 ALFWorld
ALFWorld (Shridhar et al., 2020) a synthetic text-based game implemented in the TextWorld environments (Côté et al. 2019). It contains 134 tasks across 6 distinct task types: move one or two objects (e.g., put one/two cellphone in sofa), clean / cool/heat an object with
object (e.g., clean/ cotee/col///aaa cool/heat some apple and put it in sidetable) and examine lamp (e.g., look at bowl under the desklamp). The agent is required to navigate around household setting and manipulate objects through text actions (e.g,, go to desk 1, take soapbar from toilet 1), and the environment will give textual feedback after each action. We experiment with GPT-3.5-Turbo (1106). For all experiments, we follow AutoGen (Wu et al., 2023a to use the BLEU metric to map output to the valid action with the highest similarity.
Our implementation is also based on AutoGer!! More details are in Appendix B.1
Baselines. We evaluate with: 1. ReAct (Yao et al., 2022b): We use the two-shot prompt from the ReAct. Note there is specific prompt for each type of task. 2. ALFChat (2 agents) (Wul al. 2023a]: two-agent system setting from AutoGen consisting of an assistant agent and an executor agent. ALFChat is based on ReAct, which modifies the ReAct prompt to follow a conversational manner. ALFChat (3 agents): Based on the 2-agent system, it introduces grounding agent to provide commonsense facts whenever the assistant outputs the same action three times in a row.
StateFlow Setup. The StateFlow model is shown in Figure The process starts at Plan, where a plan to solve the task is generated. We note similar planning is also used in the ReAct prompting. Then, transit to Pick to instruct the model to search for the target object and take it. We stay in Pick until the target object is picked.
We AutoGen v0.2.17.
8

---

Published as a conference paper at COLM 2024
We identify the target object by calling an LLM at the beginning and use it ground truth. a wrong object is picked, we go to the Pick.Error state, where the wrong object is put down. If the correct object is picked, we transit to the next state (Process, Put FindLamp) based on the task type. We follow the ReAct template to prompt the model to generate thought and action each time. Also, use task-specific planning examples and instructions in Plan and Process. See details in Appendix B.1
Results and analysis. Table 4 shows the results for the ALFWorld benchmark. We record the ac- curacy for each type of task and also the cost for LLM inference. We can see that StateFlow achieves the best performance on all 6 tasks, and significantly outperforms all baseline methods on the whole dataset. It improves over ReAct by 28% and ALFChat (3 agents) by 15%. At the same time, StateFlow uses 2.5x less cost. With
1.0
Success Rate
0.8
0.7
0.6
StateFlow, decompose long prompt into shorter but more concise prompts to be used when entering state. Thus, reduce the prompt tokens used while making the model fo- cus on sub-task for better responses.
0.5
Cumulative Cost
ReAct Reflexion
20
To understand the failure reasons of StateFlow, analyzed all 23 failed tasks and classified them based on the ending state. The analysis revealed that 15 out of 21 tasks ended in the Pick state, indicating that finding the correct object is the most challenging part. The failures in the Pick state were due to three main reasons: the LLM hallucinating about the object's location, picking the wrong object, and getting stuck in loops be- tween locations. We refer to Appendix B.2 for more analysis of the results. In Appendix B33 show that further decomposition of the states in StateFlow can increase the success rate to 88.8%, with 15% less cost.
16
10
5
Interations
Figure 5: StateFlow and ReAct inte- grated with Reflexion. StateFlow can further be improved with Reflexion, with much less cost incurred than Re-
Act.
Integration with Reflexion. We incorporate
StateFlow with Reflexion (Shinn et al. 2023), showing that StateFlow can be combined with iterative refining methods improve its performance. Reflexion reflects from pre- vious unsuccessful trials and stores the feedback in memory for subsequent trials. We can either use ReAct or StateFlou low as the basic executor. We run StateFlow fow +Reflexion and ReAct+Reflens Reflexion for 6 iterations, until both ReAct and StateFlow stop performance improvement. In Figure StateFlow +Reflexion further improves from 84% to 94.8%, with the total cost increased from $2.9 to $8.6. While ReAct+Reflexing improves from 55.2% to 74.6%, the total cost for it increased from $7.1 to $27.9.
Related work
Different prompting frameworks have been proposed to enhance LLM reasoning pro- cesses (Zhang et al. 2023d Wu et al.) 2022; Sel et al. 2023 Ning et al. 2023 Zhou et al.) 2022 Zhang al. | 2023c Zelikman al. 2023). Tree of thoughts (ToTs) (Yoo et al. 2023 models the reasoning process as tree and employs DFS or BFS search to explore sequential thoughts. Tree-of-Tieee Thought by (Long 2023) models the thoughts as trees but relies on a rule- based verifier to determine if a thought is valid and performs refining or backtracking based on controller. Graph of Thoughts (Besta et al. 2023 models the process as directed graph and defines 3 types of transformations at node the graph: aggregation, refining, and generation. These frameworks have better control over LLM's intermediate steps, but they are not well-designed to consider LLM workflows with external tools and environments.

---

Published as a conference paper at COLM 2024
StateFlow considers external feedback and also allows the design of complex patterns for more difficult tasks, where any state can be connected with the definition of state transitions. The step-wise search from ToTs (Yao et al. 2023) can easily be applied to StateFlow When we call LLM in a state, we can generate several responses and employ an evaluator to select the best one.
LLMs have been used to interact with environments (Deng et al. 2023 Yao et al. 2022a
Shridhar et al. 2020) and tools (Paranjape et al. 2023; Gou al. 2023 Scciick et al. 2023 Gao
et al. 2023; Yang et al. 2023b; Zhang et al. 2024a|| Zou et al. 2024). ReAct (ao et al. 2022b uses a tew-shot prompting strategy that generates the next achion based on past achios, and observations, which have been proven effective. Follow-up works employ iterative refining to improve from previous trials (Sun et al. 20244 Prasad et al. 2023). Reflexion (Shinn al. 2023) and Self-Refine (Madaan et al. 2023) generate reflections from the past to improve future trials. In parallel to our work, Liu & Shuai (2023) adapts a state machine to record
and learn from past trajectories. Extending from Tolls and RAP (Hao et al., 2023), (Zhou et al. 2023) proposes an LLM-based tree search incorporating reflection and feedback from the environment. These methods incur additional costs from the interactions searching processes, StateFlow is orthogonal to these methods, and some of them can be used on top of StateFlow to further improve performance.
LLMs are becoming a promising foundation for developing autonomous agents (Xi et al. 2023 Wang et al., 2023a Wu et al. 2023a Li et al. 2023 Hong et al. 2023 Sumers et al. 2023).
AutoGen (Wu et al. 2023a) offers an open-source plattorm for developing LLM-based agents.
MetaGIT (Hong et al. 22233) proposes a multi-agent framework for software development
and CAMEL (Lr ( al. 2023) proposes a framework for autonomous agent cooperation. In
StateFlow, specialized agents with different instructions are used in different states.
Conclusion
In this paper, we propose StateFlow, a novel problem-solving framework to use LLMs for complex, multi-step tasks with enhanced efficiency and control. Stateflow grounds the progress of task-solving with states and transitions, ensuring clear tracking and management of LLMs' responses and external feedback. We can define sequence of actions within each state solve a sub-task. StateFlow requires humans have good understanding of given task and build the model and prompts. An intriguing avenue for further research lies in the automation of StateF low model construction and prompting writing, leveraging LLMs to dynamically generate and refine workflows. Further, the idea of employing active learning strategies to iteratively adjust or "train" a StateFlow, adding or removing states automatically based on task performance, presents a promising path toward maximizing efficiency and adaptability in complex task solving.
Acknowledgements
We would like to thank Hanjun Dai and Eric Zelikman for their reviews and helpful feedback. We also thank Yu Tong (Tiffany) Ling from MathGPTPro for her help in creating demonstration figures.
References
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski,, Hubert Niewiadomski, Piotr Nyczyk, al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.0965: 2023.
John Carroll and Darrell Long. Theory of finite automata with an introduction formal languages. 1989.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld:
10

---

Published as a conference paper at COLM 2024
A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pp. pp. 41-75. Springer, 2019.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070, 2023.
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-Dickstein, et al. Language model cascades. arXiv preprint arXiv:2207.1044 2022.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, arXiv preprint arXiv:2312.10957, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujuu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11777 2023.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023.
John E Hopcroft, Rajeev Motwani, and Jeffrey D Ullman. Introduction automata theory, languages, and computation. Acm Sigact News, 32(())60 65, 2001.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491 2023.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large scale language model society, 2023.
Jia Liu and Jie Shuai. Smot: Think in state machine. arXiv preprint ariiv:2222::222174555 2023.
Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.1765.1 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
George H Mealy. A method for synthesizing sequential circuits. The Bell System Technical Journal, 34(5):1045-1079, 1955.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: survey. arXiv preprint arXiv:2302.07842. 2023.
Edward F Moore et al. Gedanken--xperimenns on sequential machines, Automata studies, 34:129-153, 1956.
11

---

Published as a conference paper at COLM 2024
Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, and Yu Wang. Skeleton-of-thought Large language models can do parallel decoding. arXiv preprint arXiv:2307.15337 2023.
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09011 2023.
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models. arXiv preprint arXiv:2311.0577 005772, 2023.
Elaine Rich et al. Automata, computability and complexity: theory and applications. Pear- son Prentice Hall Upper Saddle Saddle River, 2008.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Lu Wang, Ruoxi Jia, and Ming Jin. Algorithm of thoughts: Enhancing exploration of ideas in large language models. arXiv preprint arXiv:2008.1037) 2023.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366 2023.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interac- tive learning. arXiv preprint arXiv:2010.03768, 2020.
Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive architectures for language agents. arXiv preprint arXiv:2309 02420,022722 2023.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang, Adaplanner: Adap- tive planning from feedback with language models. Advances in Neural Information Processing Systems, 36, 2024.
Ferdinand Wagner, Ruedi Schmuki, Thomas Wagner, and Peter Wolstenholme. Modeling software with finite state machines: practical approach. CRC Press, 2006.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432 2023a.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot ththugtt- reasoning by large language models. arXiv preprint arXiv:2305.04091, 2023b.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. arXiv preprint arXiv=2309.10691, 2023c.
Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. Large language models are reasoners with self-verification. arXiv preprint arXiv:2212.09561, 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint ArXiv:2308.08155 2023a.
Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. Promptchainer: Chaining large language model prompts through visual programming. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-10, 2022.
12

---

Published as a conference paper at COLM 2024
Yiran Wu, Feiran Jia, Shaokun Zhang, Qingyun Wu, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, and Chi Wang. An empirical study on challenging math problem solving with gpt-4. arXiv preprint arXiv:2306.01337, 2023b.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.0788
2023.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. Self-evaluation guided beam search for reasoning. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Stan- dardizing and benchmarking interactive coding with execution feedback. arXiv preprint arXiv:2306.18898, 2023a.
Kaiyu Yang, Aidan M Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. Leandojo: Theorem proving with retrieval- augmented language models. arXiv preprint arXivv2306.15626, 2023b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards
scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and
Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.106/ 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310.02004, 2023.
Shaokun Zhang, Feiran Jia, Chi Wang, and Qingyun Wu. Targeted hyperparameter op- timization with lexicographic preferences over multiple objectives. In The Eleventh international conference on learning representations, 2023a.
Shaokun Zhang, Yiran Wu, Zhonghua Zheng, Qingyun Wu, and Chi Wang. Hypertime: Hyperparameteer optimization for combating temporal distribution shifts. arXiv preprint arXiv:2305.18421, 2023b.
Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: Influence-driven selective annotations empower in-context learners in large language models. arXiv preprint arXiv:2310.10875 2023c.
Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. Training language model agents without modifying language models. arXiv preprint arXiv:2402.1111 11359, 2024a.
Shaokun Zhang, Xiawu Zheng, Guilin Li, Chenyi Yang, Yuchao Li, Yan Wang, Fei Chao, Mengdi Wang, Shen Li, and Rongrong Ji. You only compress once: Towards effective and elastic bert compression via exploit-explore stochastic nature gradient. Neurocomputing, pp. 128140, 2024b.
Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models. arXiv preprint arXiv:2308.0437), 2023d.
13

---

Published as a conference paper at COLM 2024
Xiawu Zheng, Chenyi Yang, Shaokun Zhang, Yan Wang, Baochang Zhang, Yongjian Wu, Yunsheng Wu, Ling Shao, and Rongrong Ji. Ddpnas: Efficient neural architecture search via dynamic distribution pruning. International Journal of Computer Vision, 131(5): 1234-1249, 2023.
Andy Zhou, Kai Yan, Michal Shlapentokh-Rothmall Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023,
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10622, 2022.
Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge poison- ing attacks to retrieval-uggmented generation of large language models. arXiv preprint arXiv:2402007867, 2024.
14

---

Published as a conference paper at COLM 2024
A InterCode
A.1 Experiment Details
For ReAct and Plan & Solve, we use the code from InterCode repository The StateFlow models for the ablation study on the InterCode SQL task are in Figure and the full metrics for the ablation study are shown in Table 8 See S bash example with StateFlow. We include also two examples of ReAcE and ReAct Refined Table 20 and 21 for comparison.
For the InterCode benchmark, we recorded different metrics provided by the benchmark and we also recorded the LLM usage of each method. Additional metrics: (1) Reward: For SQL, the reward is calculated by Intersection over Union (loU) of the latest execution output generated against the gold output. For Bash, customized function is used to evaluate the performance against file system modifications and the latest execution output. (2) Token Count: We also recorded the prompt tokens (input), and completion tokens (output) used by each method. See Table and Table for detailed comparisons.
See Table 16 and 17 for the instructions used for SQL StateFlow model and Table 18 and 19 for instructions used for Bash. A uniform prompt that introduces the overall environment is put in the system message, and specific prompt is put in the head of the user message.
Code is available at https: //gs thub. cow http.//ooyyoaao///tttteeeeee
A.2 Additional Analysis
The additional analysis below is based on results with the GPT-3.5-Turbo model.
SQL. The SQL dataset consists of different levels of difficulties (See T) The success rate drops with harder tasks, and the difference SR between easy and extra hard tasks is great as 50% with StateFlow. Harder queries usually pose several constraints the data and require looking across tables and joining information from them (Extra hard example: "Which distinctive models are produced by maker with the full name General Motors or weighing more than 3500?". Easy example: "Give the city and country of the Alton airport.") ") However, StateFlow greatly improves over other baselines extra tasks, leading to 20% improvement compared to ReAct. Since harder tasks require information across tables, they are more likely to result in errors. We collected the states traversed for tasks that are solved successfully, and found that only 9% of the easy tasks go over state Error, while 20% of the extra hard tasks have state Error visited. This indicates that the state Error in StateFlow plays an important role in the performance gain in hard tasks.
hard and
Easy
Medium
Hard
Extra
All
Plan & Solve
80.6
49.1
29.1
13.2
47.7
ReAct
72.2
57.6
35.6
15.7
50.7
ReAct_Refined
77.0
66.1
44.8
19.9
57.7
StateFlow
87.9
62.9
59.8
36.7
63.7
Table 5: Success Rate of different level of difficulties InterCode SQL with GPT-3.5-Turbo.
Bash A Bash task consists of one both of the following two requests: 1. retrieving information that can be acquired through output (e.g, "find files in /workspace directory which are modified 30 days ago") and 2. changing configuration of file/folder (e.g.. "change permissions for all PHP files under the /testbed directory tree to 755") (Yang et al. 2023a). To complete Bash task, all required commands need to be correct, making difficult to achieve a reward of 1. We collect the rewards from the failed bash tasks of StateFlow and find that 58.5% of the bash tasks have a positive reward greater than 0.5. For
SQL, only 0.5% of the failed tasks have positive reward greater than O. This shows that
15

---

Published as a conference paper at COLM 2024
SR % f
Reward
Turns /
Error
Cost $
Average p-token +
Average
% 4
c-token |
Plan & Solve
47.68
0.4893
4.31
12.46
2.38
1998
154
GPT-3.5
ReAct
50.68
0.5257
5.58
16.33
17.73
16456
348
ReAct_ Refined
57.74
0.5928
5.47
3.82
18.05
16782
340
Try Again* *
56.38
0.5762
7.62
34.73
6.61
6098
145
StateFlow
63.73
0.6637
5.67
6.82
3.82
3128
281
SF Chat
60.83
0.6356
5.38
5.01
5.59
4965
220
Plan & Solve
56.19
0.5793
5.39
1.79
44.7
3065
416
ReAct
60.16
0,6277
5.26
3.87
147
12951
419
GPT-4
ReAct Refined
57.93
0.6104
5.01
2.49
141
12377
421
StateFlow 69.34
0.7223
5.11
1.89
36.0
2700
261
SF_Chat
70.41
0.7329
4.84
1.20
49.2
3907
283
Table 6: Results of the Intercode SQL with GPT-3.5 and GPT-4 with all metrics. We also include SF 3F_Chat an alternative of StateFlow, and another baseline Try Again with oracle setting("). Best metrics of each model is in Bold. Second-best is Underlined.
SR %/
Reward
Turns
Error
Cost st
Average p-token
Average c-token |
%
ReAct
32.5
0.7674
5.52
13.23
3.28
15529
442
GPT-3.5
Plan & Solve
23.5
0.7472
4.98
25.78
0.74
3232
225
Try Again*
49.5
0.8453
6.88
19.54
0.83
3833
159
StateFlow
36.0
0.8033
3.90
8.74
0.63
2667
232
SF-Chat
37.0
0.8011
3.04
9.95
0.79
3658
148
ReAct 31.5
0.7724
3.86
9.90
20.40
9027
392
GPT-4
Plan Solve
20.5
0.7280
5.15
21.03
9.59
3460
444
StateFlow 39.0
0.8059
2.95
7.85
5.02
1835
225
SF Chat
37.5
0.8015
2.86
7.27
7.04
3113
135
Table 7: Results of the Intercode Bash with GPT-3.5 and GPT-4 with all metrics. We also include SF _Chat, an alternative StateFlow, and another baseline Try Again with an oracle setting(").
many of the tasks are partly solved. In Section A.3 the results of "Try Again" show that signals from the environment are very useful and help the model understand how much process it has made to solve the task. Currently, follows a simple workflow of solve->verify. However, since the bash task consists of two distinct requests mentioned, we believe it is possible to improve the performance with more complex StateFlow model, with different states constructed for each request.
StateFlow model designed for bash
A.3 Additional Results
SF_Chat We present the results of SF_Chat, an alternative version of StateFlow, in Table and Instead of creating individual LLM agents with specific instructions, we directly append that instruction to context history, imitating user's 's reply in conversation. For this alternative, we construct the context history in a conversational manner. The observations and instructions are appended as user messages, and replies from models are appended as assistant messages. We can see that SF SF_Chat has a similar performance to StateF low. By directly appending the instructions, SF_Chat takes fewer turns to finish the task and has a lower error rate than StateFlow. In trade-off, the cost of SF_Chat is higher. We note that SF SF_Chat might not be suitable for tasks that require many turns of interactions (e.g., ALFWorld), because the cost would be high with the instruction prompts accumulated in the context history.
16

---

Published as a conference paper at COLM 2024
SR %
Reward 1
Turns J
Error %
Cost $
Average p-token +
Average c-token
StateFlow
63.73
0.6637
5.67
6.82
3.82
3128
281
No_Verity -e-irrfyy 62.28
0.6473
5.18
5.96
3.68
3070
244
NrrEr
58.80
0.6091
5.72
11.58
4.05
3280
318
No-Observe
57.83
0.6041
6.00
16.95
4.64
3816
337
Table 8: Ablation of states on the SQL dataset with GPT3.5-Turbo with all metrics. Best metrics in Bold. Second-best is Underlined.
Try Again We also include results of another baseline "Try Again" from InterCode Bench- mark with GPT-3.5-Turbo. Try Again is an iterative feedback setup from InterCode mimic human software development (Yang et al. 2023a). In this setup, the model can receive ground-truth reward from the environment at each execution of the command and stops when the task is solved correctly reaches max turns. Then the max reward from all the executions is retrieved. We note that this is an oracle setting not used in StateFlow and other baselines. In setting, we use the model to determine when to stop and submit the answer, and only the result before submission is evaluated. From Table 6 we can see that Try Again doesn't work well in the SQL task, and that the performance is slightly worse than our refined version of ReAct. It also has high error rate of 34.73%. However, with the bash task, Try Again significantly outperforms other methods. This discrepancy indicates the difference in the nature of the two tasks. In the SQL task, the reward is mostly 0 or 1. The observation commands such as "DESC" and a wrong "SELECT" command receive 0, and there are only few cases where the "SELECT" command is partially correct to receive a partial reward. In this case, the reward signal is not very useful. However, the bash tasks from InterCode are explicitly selected with utilities > 4 (require several commands), and each correct command can receive partial reward. Thus, the reward signal can help the model understand how much progress it has made and provide guidance, leading to significant improvement in performance.
B ALFWorld
B.1
Experiment Details
For ALFWorld, we use ReAct from Reflexior which has the same implementation as the original ReAct repository. For ALFChat, we used the code from AutoGen Evaluatior We allow maximum of 50 rounds of interactions with the environment. For ReAct and StateFlow, we follow a text completion manner to use the chat-based model GPT-3.5-Tun In this experiment, we follow Reflexion put all instructions and interaction history in one user message when querying the model. The ALFChat is essentially a chat version of ReAct, where the examples are converted into a history conversation between the 'user' and 'assistant'.
Details on StateFlow Setup We refer to Table 13 type of input task, as a prior knowledge, is used all methods. ReAct and ALFChat use different few-shot examples for different types of tasks. Similarly, we have different planning examples for different tasks, as shown in Table 13 In Process, we also have three different prompts corresponding to heat, cool, and clean. In Figure 7, we illustrate activated states for different tasks. As discussed in the main paper, we identify the object interest by calling the same LLM at the beginning of the task. In Pick, only when detect string match of "You pick up A" (A is the object needed), we would transit the next state. Similarly, in Fi FindLamp, we transit to UseLamp only if string "desklamp" is matched. In Process, we match the pattern "You heat/cool/clean" to proceed. Note that this feedback is from the
14
15 for the prompts we used. The
17

---

Published as a conference paper at COLM 2024
environment. Finally at state Put and UseLamp, transit to End only if the task succeeds or fails. A task is considered success we receive "Done=True" from environment, and considered fail if the same response is generated by the LLM for three consecutive rounds (following AutoGen). In this experiment, the environment feedback "Done" is available to all methods to terminate the process upon success. Please Figure 10 for an example.
B.2 Additional Analysis
To understand the failure reasons of StateFlow, we manually went through the 23 failed tasks of one attempt and classified them based on the ending state (See Table Ending in different states can show how much progress has been made a task. For example, a task ending in Cool implies that the correct object is being picked, but not cooled correctly. From the table, is to go around the household to find the correct object. For the failed tasks that ended in Pick, summarize three failure reasons: 1. The LLM hallucinates about taking the target object from locations where does not exist. 2. The LLM takes the wrong object. 3. The LLM gets stuck loops between two locations.
see that 15/21 tasks end in Pick. This suggests that the most difficult part
Ending State
Pick/Find
Put
Cool
Heat
FindLamp
Error
All
StateFlow (7-state)
15
2
1
StateFlow (10-state)
3
0
0
13
Table 9: Count of failed ALFWorld tasks that end in different states. We also include the StateFlow model with 10 states for comparison.
B.3 Additional Experiments
Adding more states to StateFlow In the states defined for ALFWorld, we allow different types of actions to be performed. For example, in Pick, the model is instructed to either go around the household with the 'go to frecept)' command, open receptacles with 'open', and take an object with 'take {obj} from (recept)' command (the format annotation adopted from Prasad et al. )2023) We further divide these actions and add 3 more states and test its performance (See Figure 6). In Figure 10 we show the results of StateF low, and StateFlow with more states. The overall performance increases from 83.3% to 88.8%, with 15% less cost. Table
9 indicates that the primary contribution to performance improvement comes
from dividing the Pick action into Find and Take.
Pick
Clean
Heat
Cool
Look
Pick 2
All
Cost $
StateFlow (7-state)
91.7
83.9
85.5
79.4
92.6
62.7
83.3
2.6
StateFlow (10-state)
100
92.5
94.2
87.3
90.7
58.8
88.8
2.2
Table 10: Performance and cost of StateFlow and StateFlow with 3 more states. We report an average success rate of 3 attempts. More states and division of tasks can further improve performance at even a lower cost.
Additional results with GPT-3.5-Instruct. In experiments, (e.g., GPI-3.5-Turbo because they are more powerful and have been studied extensively lately. Original ReAct was tested on completion model text-davicin-002, which has been depreciated. The recommended replacement is GPT-3.5-instruck and StateFlow on N0T-335-turbo-instrree and find that the performance drops for both methods (see Figure 12. Compared to results on GPT-3.5-Turbo, ReAct has a drop of 4% and StateFlow has drop of 9%.
the latest chat models
We also tested ReAct
Addition results with open-source models We further test StateFlow and ALFChat with agents on 3 open-source models: Mistral-7b, Llama-8b and Llama3-70b. We observe that
18

---

Published as a conference paper at COLM 2024
ALFChat (3 agents)
StateFlow
Acc (%)
Cost ($)
Acc (%)
Cost ($)
Mistral-7b
1.5
0.708
6
2.03
Llama3-8b
50.7
0.52
62.7
0.38
Llama3-70b
83.6
2.53
94
1.83
Table 11: Accuracy and cost comparison on ALFWorld with different open-source models.
For this experiment, we compare with the strongest baseline and run the test one time.
Pick
Clean
Heat
Cool
Look
Pick 2
All
Cost S
ReAct
62.5
41.9
78.3
57.1
38.9
23.5
51.5
StateFlow
87.5
66.7
87
57.1
77.8
64.7
73.9
5.0
Table 12: Performance and cost on ALFWorld with GPT-3.5-instruct. Here we report success rate of only one attempt. The performances of both methods drop compared to result with GPT-3.5-Turbo, on which ReAct achieves 55.5% and StateF low achieves 83.3% in overall success rate.
StateFlow outperforms 3-agent ALFChat with agents across the open-source models. It is worth noting that Mistral-7b struggles to complete the task using both methods, suggesting the model itself is not equipped to handle tasks within ALFWorld. However, Llama3-70b with StateFlow can hit 94% accuracy, surpassing the performance of GPT-3.5 Turbo.
Processed
GoTo
Heat Caat/Canl
Act
"Pick/Pick 2'
Plan
Take
Put
"Look
auoQ
FindLamp
Found
Uselamp
Done
End
Figure 6: StateFlow model for ALFWorld with addition states. The additional states are marked blue. In states Pick, Process and Put of the original model, there are actually two actions to be performed: 1. go to some receptacle. 2. take/ prrcesss/s//////eeeeeeeeee ppt an object. Here, we further split it, using one state for one action.
19

---

Published as a conference paper at COLM 2024
Task is Heat, Cool or Clean
Error
Process
Processed
Plan
Pick
Put
Done
End
Task is Pick or Pick
Error
Init
Plan
Pick
Put
Done
End
"Pick 2"
Task is Look
Error
Init
Plan
Pick
End
Done
FindLamp
Found
UseLamp
Figure 7: Active states for different type of tasks in ALFWorld. Only 4-6 states active for a single task. States Plan, Pick, Error and Put are sharable across tasks.
20

---

Published as a conference paper at COLM 2024
No Error State
SELECT
Aways
Observe
DESC
Solve
SELECT
Verity
Submit
End
No Observe State
SELECT
Init
Solve
Verify
Submit
End
DESC
Error
Erter
salecT
No Verify State
Observe
Solve
Subnit
End
BUCCESS
Error
Enror
Figure 8: StateF Low model for the ablation study with InterCode SQL task.
Convert
'/tssbed//
Verify
textfilee.txt" single hexadecimal value.
Model:
Init
Thought:
characters ("hastbed) single successfullly
tectfile
ALWAYS
Solve
Solve
SQL: 5455577420639005520777777788
Model: Inatruction: question, please give command
sobi
action submit
End
executelhead hestbadtaxtfllesstt
Endl
Thought:
'head" command the fil single
Exit
haxadecimal
SQL: Observation: /bin/bahh: line 1t oodt command not found
Init
Solve
Verify
End
Execution Error
Error
Emrr
Model: Instruction Please carefully check your ast command
and outaus understand what went wrong.
Thaught: soems that the 'ood' command not avaiiabee, will use the 'dd command to get
navadocimal dump the first 16 characters of the file
Action: exccutoled -A - +N 18 hesttbochoottlle..ttt awk 31,rint tr-d
SOL:
Observation: 44558972009999803201777777778880
State transition:
Execution Success
Verify
Figure 9: Example of StateFlow on Bash task.
21

---

Published as a conference paper at COLM 2024
Pick
Household setting: You are middle room. Looking quickly around you, you
cabinet cabinet 3, cabinet cabinet cabinet coffeemachine countertop 3, countertop countertop 1, drawer 3, drawer drawer 1, tridge 1, a garbagecan microwave shelf 3, a shelf 2, a shelf a sinkbasin 1, stoveburer stoveburner 3, stovebumer
Model: Instrrtioon: Flease follow the plan chack receptacion houschold one one find the object of
Thought: Thought: found the apple (1) on the counkertop top. need to take it.
Action: take apple from countertop
ALFWorld:
Otsarvation: You pick
from countertop
State transition:
@ Correct objectt picked
Solve
Your task to: heat some and put garbagecan.
Heat
Init
Model: Instruction: new object interest with you. Now, please ge microwaee heat the otject.
State transition:
Always
Plan
Action: microwave
ALFWorld:
Observation: The microwave is closed.
Plan
transition:
@ Objject Not heated
Heat
Model:
Instruction: You are given househeld setting and task Please make plan complete the tadk...
srcomplish
Heat
Plan:
for apple go to girbagecan and put it down. apple is more likely to agpear fridge (1), countertop (1-3), diringtable (1), drawer (1), cabinet (1-6), garbagecan (1). can check one by ome, starting with frigge 1.
search go
Instr inttrnction You now take object inter with you, Now, plaase to the cicwwoee heat the object...
open
ALFWorld: Observation: You open the microwave The microwave is In it, sea nathingg
State transition:
① Always
Pick
State transition:
a Objject Not heated
Heal
Pick
Heat
Model: Instruction: Plaase folaw check recoptadles by object
Model: Instruction: You now take the object of interest with you Now, plane the microwave to heat the ozject.
Action: go to fridge
Action:
apple
ALFWorld:
Observation: The fridge closed.
ALFWorid:
Observation: You heat the apple using the microwave
State transition:
No Pick
Pick
State transition:
e Object hoatad
Pick
Put
Model: instruction Please follow pan to check receptadles in househalld one one find the object interst..
you go the required place down object.
Action: open fridge
Action: to garbagecan 1
ALFWorld: Observation: You open the tridge 1. The fridge open, In you see bowl pan 1, plate 1, and tomato
ALFWorld: Observation: On the garbagacan 1, you see papertoweiroll 1, and a potato
transition:
State transition:
@ No Pick
Pick
Put
Pick
Model: Instructions now take the object inter esthi you. New, please requined place down the object.
Model: Instruction: Please
receptacies in
household
Thought:
didn't find the fridge. should check possible location.
Action: put apple garbagecan
ALFWorid:
countertap
ALPworld:
Observation: countertop 1, you see apple 1, egg 1, mug 2, spapbottle spoon and a vase 2.
transition:
End
End
State transition:
G No Pick
Pick
Exit
Figure 10: Example StateF on ALFWorld.
22

---

Published as a conference paper at COLM 2024
## Examples
Your task is to: put some spraybottle on toilet.
Plan: need to 1. search around for spraybottle and take it. 2. go to toilet and put down. A spraybottle
is more likely to appear in cabinet (1-4), countertop (1), toilet (1), sinkbasin (1-2), garbagecan (1). I can check one by one, starting with cabinet 1.
Your task is to: find some apple and put it in sidetable.
Plan: I need to 1. search around for an apple and take it. 2. go to sidetable and put it down, An apple is more likely to appear in fridges (1), diningtables (1-3), sidetables (1), countertops (1), sinkbasins (1), garbagecan (1). can check one by one, starting with fridge 1.
pick clean then place
You must use the sinkbasin to clean the object.
## Examples
Your task is to: put a clean lettuce in diningtable.
Plan: I need to 1. search around for some lettuce and take it, 2. go to sinkbasin and clean it, 3. go to diningtable and put it down. First I need to find a lettuce. A lettuce is more likely to appear in fridge (1), diningtable (1), sinkbasin (1), stoveburner (1-3), cabinet (1-13). I can check one by one, starting with fridge
Your task is to: clean some apple and put it in sidetable.
Plan: I need to 1. search around for some apple and take it, 2. go to a sinkbasin and clean it, 3. go to sidetable and put it down. First need to find an apple. An apple is more likely to appear in fridges (1), diningtable (1-3), sidetable (1), countertop (1), sinkbasin (1), garbagecan (1). I can check one by one, starting with fridge 1.
.hhee.lacce ##i Examples
Your task is to: heat some egg and put it in diningtable.
Plan: I need to 1. search around for an egg and take it, 2. go to microwave and heat it, 3. go to diningtable and put it down. An egg is more likely to appear in fridge (1), countertop (1-3), diningtable (1), stoveburner (1-4), toaster (1), garbagecan (1), cabinet (1-10). I can check one by one, starting with
fridge 1.
Your task is to: put a hot apple in fridge.
Plan: I need to 1. search around for an apple and take it, 2. go to microwave and heat it, 3. go to fridge and put it down. An apple is more likely to appear in fridge (1), diningtable (1), coffeetable (1), drawer (1), cabinet (1-13), garbagecan (1). can check one by one, starting with fridge 1.
pick cool then place ## Examples
Your task is to: cool some pan and put it in stoveburner.
Plan: need to 1. search around for a pan and take it, 2. go to fridge and cool it, 3. go to stoveburner and put it down. An pan is more likely to appear in stoveburner (1-4), sinkbasin (1), diningtable (1), countertop (1-2), cabinet (1-16), drawer (1-5). I can check one by one, starting with stoveburner 1.
Your task is to: put a cool mug in shelf.
Plan: I need to search around for a mug and take it, 2. go to fridge and cool it, 3. go to shelf and put it down. A mug is more likely to appear in countertop (1-3), coffeemachine (1), cabinet (1-9), shelf (1-3), drawer (1-9). can check one by one, starting with countertop 1.
look at obj #t Examples
Your task is to: look at bowl under the desklamp.
Plan: need to 1. search around for bowl and take it, 2. find desklamp and use it. First need to find bowl. bowl is more likely to appear in drawer (1-3), desk (1), sidetable (1-2), shelf (1-5), garbagecan (1). can check one by one, starting with drawer 1.
Your task is to: examine the pen with the desklamp.
Plan: I need to 1. search around for pen and take it, 2. find desklamp and use it. First I need to find a pen. A pen more likely appear in drawer (1-10), shelf (1-9), bed (1), garbagecan (1). I can check one by one, starting with drawer 1.
pick_two_ol ## Examples
Your task is to: put two creditcard in dresser. Plan: I need to 1. search around for creditcard and take it, 2. go to dresser and put it down. 3. find another creditcard, 4. go to dresser and put it down. First I need to find the first creditcard. A creditcard is more likely to appear in drawer (1-2), coutertop (1), sidetable (1), diningtable (1), armchair (1-2), bed (1). can check one by one, starting with drawer Your task is to: put two cellphone in sofa.
Plan: I need to 1. search around for cellphone and take it, 2. go to sofa and put it down. 3. find another cellphone, 4. go sofa and put it down, First I need to find the first cellphone. A cellphone is more likely to appear in coffeetable (1), diningtable (1), sidetable (1-2), drawer (1-4), sofa (1), dresser (1), garbagecan (1). I can check one by one, starting with coffeetable
Table 13: Few-shot examples for State Plan in ALFWorld, Same as ReAct and other baselines, we use one prompts for each type of task.
23

---

Published as a conference paper at COLM 2024
Head of system message
You are given a description of a household, please interact with the household to solve the task.
This is a simulation of and all the actions are high-level shortcuts. Follow the instructions to give your next reply.
## RESPONSE FORMAT
Reply with the following template (<...>> is the field description):
Thought: <your thought>
Action: <-our action
Action:
<your action?>
In you reply, you can give both thought and an action, or just an action. You can only give one action at à time.
Environment feedback
After each your turn, the environment will give you immediate feedback.
Observation: <observation.>
Pick
## Instructions
Please follow the plan to check receptacles in the household one by find the object of interest. Each time, you can observe all the objects in the receptacle. Determine if the object you are looking for that receptacle.
You need to find the EXACT object that is asked for. For example, if you need to find a "soapbar", only take it when you see "soapbar {i}" in the receptacle, instead of a "soapbottle (i)". Use "open (recept)" command to open receptacle.
## Examples
Use "go to (recept)" command to go to the receptacle:
Action: go to 1
Action: go fridge
Action: go diningtable
Take the object only if the place have the exact object you are looking for: Thought: Now find the soapbar (1). Next, need to take it.
Action: take spraybottle from cabinet
Thought: Now find the apple (1). Next, need to take it.
Action: take apple from diningtable
Error
You just took the wrong object. Please put it down with the "put obj in on place" command, where "place" is the place where you took the object from. Please also give your thought of what is the next place to check based on the plan.
## Examples
Thought: accidentally took the tomato instead of the apple. need to put it back, and then check diningtable 2.
Action: put tomato in/on diningtable 1
Thought: The object want to take a soapbar, but I took a soapbottle. I need to put it back, and then check the sinkbasin 1.
Action: put soapbottle in/on toilet 1
Process (Heat)
## Instructions
You now take the object interest with you. Now, please go to the microwave heat the object. You must first go to the microwave with the "go to (microwave)" command. Then, use the "heat {obj] with (microwave)" command to heat the object. You don't need to open the microwave or put the object in the microwave.
## Examples:
If you just picked the object, go to the microwave first:
Action: go to microwave
Then heat the object:
Action: heat apple with microwave 1
Action: heat bread with microwave
Table 14: Instructions of StateFlow for ALFWorld.
24

---

Published as a conference paper at COLM 2024
Process (Cool)
## Instructions
You now take the object of interest with you. Now, please go to the fridge to cool the object. You must first go to the fridge with the "go to [fridge)" command. Then, use the "cool {obj} with (fridge)" command to cool the object. You don't need open the fridge put the object in the fridge
## Examples:
If you just picked the object, go to the fridge first:
Action: go to fridge 1
Action: go fridge
Then cool the object:
Action: cool pan with fridge 1
Action: cool potato 2 with fridge
Process (Clean)
## Instructions
You now take the object of interest with you. Now, please go the sinkbasin to clean the object. You must first go the sinkbasin with the "go {sinkbasin] command. Then, use the "clean {obi] with (sinkbasin)" command to clean the object. You don't need water or soap to clean the object.
## Examples:
Go the sinkbasin first:
Action: go sinkbasin
Action: go to sinkbasin 2
Then clean the object:
Action: clean lettuce 1 with sinkbasin
Action: clean soapbar with sinkbasin 2
Find Lamp
## Instructions
You have found and taken the object, now please go around to find a desklamp. Plese use "go to [place(" command to go to different places in the houschold to find desklamp. Use "open (place)" command to open closed place if you need to.
## Examples
Action: go to sidetable
Action: go to dresser
Use Lamp
## Instructions
You now find desklamp. Please the "use {desklamp)" command look the object. [ddeklampp" denotes the desklamp you just found. You should not perform any other actions.
## Examples:
1. Observation: On the sidetable you desklamp 3, newspaper 1, and statue 2.
Action: use desklamp
On the sidetable 2, you see a alarmclock 1, a desklamp 1, and a pen Thought: Now find desklamp (1). Next, I need to use it.
Action: use desklamp
Put
## Instructions
You now take the object of interest with you. Now, please go to the required place put down the object. You must first go to the receptacle with "go to {recept)" command.
If the receptacle is closed, use the "open (recept)" command to open it. You can only take one object at time. If you task is to put two objects, please go to the required place to put the first object first. When you are at the place, use the "put obj in/on place" command to put down the object.
## Examples:
Always go to the receptacle first:
Action: go to sidetable
Action: go toilet
Action: go diningtable
Then put down the object:
Action: put apple 3 in/ on sidetable 1
Action: put soapbar 4 in/on toilet 1
Table 15: Instructions of StateFlow for ALFWorld. (Continued)
25

---

Published as a conference paper at COLM 2024
System message
Interact with a self.setting system using self.language queries to answer question.
Observe
## Instructions
Use the DESCRIBE [table_ or DESC iable_name command to understand the structure of the relevant tables. Only give one DESC command in action.
## Examples
Action: Execute|DESC highschooler]
Action: execute(DESC friends]
## RESPONSE FORMAT
For action, put your SQL command the execute[] block.
Reply with the following template (<..>> is the field description, replace it with your own response):
Thought: <your thought which table(s) as/aee relevant in one short sentence>
Action: saocutef<youu command>]
Error
## Instructions
Please carefully read the error message to understand what went wrong. If you don't have enough information to solve the question, you can use the DESC [table name] command to explore another table. You may want to review other tables to see if they have the information you need.
## Examples
Thought: A 'transcripts' table exists, but it doesn't have the 'release_date column I came up with. T should find out what columns are available.
Thought: The 'friend'' table has two ids. I should check if the 'highschooler' table has name associated with an
Thought: The 'contestants' is table, it is not a column in 'people'. need to check the 'contestants' table to see how to get the contestant names. Thought: get single number that is the number of likes that the high schooler Kyle has. This should be the answer.
## RESPONSE FORMAT For action, put your SQL command in the executel] block. You should only give one command to execute per turn. Reply with the following template (<...> the field description, replace it with your own response):
Thought: <your thought on why this query is error and whether you should gather more information fix the error in one sentence"> Action: execute[<your command>]
Verify
## Instructions Carefully check if the output answers the question exactly. Make sure the output only display fields that the problem asks for. If the output contains any extra fields, please revise and modify your query (column alias is fine, no need to round numbers). - If the output doesn't t answer the question, please revise and modify your query. You may use more about the tables. If the output answers the question exactly, please submit the query with this "Action: submit" command.
to learn
## Examples
Thought: The output displays the contestant names and also contestant count. Although the count is used for sorting, should not displayed in output. I should modify my query only select the contestant names.
Thought: The question asks for the total population for North America. However, the output also has the continent id. I should modify my querý to only select the total population.
## RESPONSE FORMAT For action, put your SQL command in the execute[] block. If the problem is solved, your action should be "Action: submit".
Reply with the following template (<...> is the field description, replace it with your own response, "I" is the "or" operation):
Thought: <your thought on whether the output and command answers the problem>>
Action: executel <your new command> submit
Table 16: Instructions of StateFlow for SQL task (part 1).
26

---

Published as a conference paper at COLM 2024
Solve
#ff Instructions Based on the understanding of the tables and the problem, formulate SQL query with SELECT that answers the question EXACTLY. Use specific clauses like WHERE, JOIN, GROUP BY, HAVING, etc necessary. you need more information of another table, use DESC explore the table.
Notes: You should construct your command that the output answers the question exactly. For example, If the question asks for count, your command should output single number. Only select the field the question asks for. Do not include relevant but unnecessary fields such as ids counts, unless the question specifically asks for it. No need to CAST or ROUND numbers unless the question asks for it.
## Examples:
Thought: should write SQL command that selects the names from a table about high schoolers in ascending order of their grades. Grade should not be selected.
Action: execute(SEI.Fc FCT name, grade FROM high_schoolers ORDER BY high sethhsschooossssreee ASC] Thought: can use the SUM and AVG functions to get the total population and average area values for North America.
Action: xecute(oxecut((SEEECT SUM(population) AS total totall population, AVG(area) AS avg area FROM countries WHERE continent 'North America' AND area > 3000]]
Thought: should write SQL query that gets the name field from contestants and exclude the name of 'Jessie Alloway'
Action: execute(SELECT contestant FROM contestants WHERE contestant.name != 'Jessie Alloway']
Follow the RESPONSE FORMAT to give your thought and action.
## RESPONSE FORMAT
For action, put your SQL command in the execute[] block. Reply with the following template (<...> is the field description, replace it with your own response):
Thought: <your thought constructing command to answer the query exactly>
Action: execute|< <yyur command>]
Table 17: Instructions of StateF low for SQL task (part 2).
27

---

Published as a conference paper at COLM 2024
System message
Interact with Bourne Shell system using BASH queries to answer question. Follow the user's instructions to solve the problem.
Solve
## Instructions
Given the question, please give a self language command to solve it.
## Examples
Thought: can try to use 'od' (octal dump) command to get a hexadecimal dump and stitch together the values into one continuous string.
Action: executefod -A n n1 x1 -N 16 /testbed/texteee /t/t/t/xflleeetttt - awk '1 =1;print' - tr -d '1
Thought: should find the paths all java files in the testbed directory, then apply the word count command each path.
Action: execute[find /testbed -name "*jav"" -type f-xec md5sum + sort - uniq -D -w 32 - awk
'print $1']
Thought: should find the paths to all php files in the testbed directory, then apply the word count command to each path.
Action: execute|find /testbed -name "*.php" -type f-exec cat + wc -I]
Thought: The 'du' command is useful for printing out disk usage of a specific directory. I can use the
-h option to print in human readable format and the option to only print the total disk usage.
Action: executefdu -sh sh/ww /workspace]
Follow the RESPONSE FORMAT to give your thought and action. ## RESPONSE FORMAT For action, put your BASH command in the executel] block. Only give command per turn. Reply with the following template (<...> the field description, replace it with your own response):
Thought: <-our thought in one sentence>
Action: (aecute[<your command>]
Table 18: Instructions of StateFlow for Bash task (part 1).
28

---

Published as a conference paper at COLM 2024
Error
## Instruction
Please carefully check your last command and output to understand what went wrong. Revise and modify your command accordingly or try another command.
## Examples
Observation: "bin/baske line I: xxd: command not found Thought: Seems like xxd is not available. can try use 'od' (octal dump) command get
hexadecimal dump.
Action: executelod -A n x1 -N 16 /ttesteed//eeeeee seabbed//exttile7.txtt
Follow the RESPONSE FORMAT to give your thought and action. ## RESPONSE FORMAT
Reply with the following template (<...> is the field description):
Thought: <your thought in one sentence?>
Action: execute[<yoon <-your command>]
Verify
## Instructions
Carefully check if the question is answered.
Please check the desired tasks have been performed. If the question also asks for output, please check your last command and output, and make sure the output is in the desired format, and doesn't contain any extra fields. If the desired tasks have been performed, please submit the query with this "Action: submit" command.
## Examples
Thought: This gives me storage information for every folder under the workspace directory, but only need the storage for just the "orkspacce/" directory. The '-s' option should help with this.
Action: executeldu du -sh / /workspace]
Thought: This shows the output hashes and they have the same values, indicating that these files are duplicates. However, the file names are also shown, which are not needed.
Action: execute[find /testbed -naee "e javaa" -type -exec md5sum + sort uniq -w awk ppiitt $1']
Thought: This shows me too much information, only need the total number of lines. I should add up the lines together and output single number.
Action: find / /tstbed -nae "* "".php" -type f-xecc cat wI] Thought: The hello.te file has been created successfully the testbed/ directory, and contains the Hello World text. I can submit. Action: submit
Please follow this RESPONSE FORMAT give your thought and action. ## RESPONSE FORMAT For action, put your [self.language command in the executel] block. Only give one command per turn. If the question is solved, your action should be "Action: submit". Reply with the following template (<...>> is the field description, replace it with your own response, "I" the "or" operation):
Thought: <your thought on whether the question is answered in one sentence
Action: doocute[<your new command>] 1 submit
Table 19: Instructions of StateFlow for Bash task (part 2).
29

---

Published as a conference paper at COLM 2024
ReAct Prompt Example 1
Question: What are the names and grades for each high schooler?
Thought 1: should write a SQL command that selects the name and grade fields from table about high schoolers.
Action I: execute[SELECT name, grade FROM tigh_sscoolers]
Observation 1: Error executing query: Table 'network_11hm 1.high_schoolers doesn't exist
Thought 2: The high schoolers table doesn't 't exist, need check what tables are available to use. Action 2: execute[SHOW TABLES]
Observation 2: [ ("friend',), ("highschooler') 7 ('likes')l
("friend),(hhgghhhoollll
Thought 3: There is table called 'highschoolers'. should check what information this table contains. Action execute[DESC highschooler]
Observation 3: I(ID'', 'int', 'NO', 'PR', None, 'auto increment"), ('name', 'text', 'YES', ", None, "), ...] Thought 4: The highschooler table contains the name and grade columns. can now write SQL command that answers the question
Action 4: execute[SELECT name, grade FROM highschooler]
Observation I('Jonn', 12), ('Haley', 10), ('Alexis', 11), ('Jordan', 12), ('Austin', 11), (TTifaany,, 99, ('Kris', 10), ("Jessica', 11), ('Jordan', 9), ('Brittany', 10), ('Logan', 12), 1 ('Gabriel', 9), ('Cassandra', 9), ('Andrew', 10), (" ('Gabriel', 11), ('Kyle', 12)]
Thought 5: The output contains the name and grade for each high schooler, answering the question. I can submit this as the answer.
Action 5: submit
ReAct. ReAct.Refined Prompt Example 1
Question: What are the names and grades for each high schooler?
Thought 1: should first find out what tables available in this MySQL database that can help me answer this question.
Action execute[SHOW TABLES]
Observation 1: ('frriend',), ,),
(" ("highschooler') ,), ('iikes'))
Thought 2: There is a table called 'highschoolers'. I should check what information this table contains.
Action 2: eXECute[DESC highschooler]
Observation 2: [(11D, 'int', 'NO', 'RI', None, incremencr,,,, ('name', 'text', 'YES', None, "., Thought 3: The highschooler table contains the name and grade. I can now write a SQL command that answers the question.
Action 3: execute(SEL. F.C. name, grade FROM highschooler] ...mmmee original example)
Table 20: (Example 1) Comparison of example trajectories from ReAct and refined version of ReAct for the SQL task. For the new version, we always start with the "SHOW TABLES" command. In the original ReAct prompt, it always starts with a select query but results in error.
30

---

Published as a conference paper at COLM 2024
ReAct Example 2
Question: What are the names of the contestants whose names are not 'Jessie Alloway'
Thought 1: should write a SQL query that gets the name field from contestants and exclude the name of "Jessie Alloway'
Action I: execute[SELECT name FROM contestants WHERE name != 'Jessie Alloway'l
Observation Error executing query: Unknown column 'name' in 'field list'
Thought 2: It looks like there table called 'contestants', but it doesn't have a name column. I need to find out what columns are available.
Action 2: execute[DESCRIB] contestants]
Observation 2: [[' ('contestantt,ummber 'int', 'NO', 'PRI', None, 'auto_increment",, ("contestant_namee 'varchar(50), (NO',...)
Thought The available columns for the 'contestant' table are 'contestant number' and 'contes- tant name'. can now write a SQL query.
Action 3: execute(SELECT contestant name FROM contestants WHERE contestant name != 'Jessie Al- loway'] Observation 3: ((Edwina Burnam',), )) ("Tabatha Gehling', )) ('Kelly Clauss',), ,), ('Alana Bregman',) ,), ('Jesiie Eichman',), ,), ('Allie Rogalski',) ,), ('Nita Coster', ,), ('Kurt Walser',) ,), ('Ericka Dieter') "), ('Loraine NygrenTanie Mattioli',]]
Thought 4: The SQL query worked, and the output shows all the contestant's names, but does not include 'Jessie Alloway'. I am finished with this query.
Action 4: submit
refined Example
Question: What the names of the contestants whose names are not 'Jsssie Alloway'?
Thought 1: should first find out what tables are available this MySQL database that can help me answer this question
Action 1: execute[SHOW TABLES]
Observation 1: [[' ['coonestantss'
Thought 2: It looks like there a table called 'contestants', but it doesn't have a name column. I need to find out what columns are available.
Action 2: execute[DESCRIBB contestants]
Observation 2: [(' ('contestant.numbee 'int', 'NO', 'RR', None, 'auto intteeincremnt",, ('contestant.namee ...)]
Thought The available columns for the 'contestant' table are Contestant,numblt and 'contes- tant name'. I can now write SQL query.
Action 3: execute[SEL.FCe SCTFCT contestant name FROM contestants WHERE contestant name != 'Jsssie Alloway'l
(samm as original ReAct example)
Table 21: Example 2) Comparison of example trajectories from ReAct and refined version of ReAct for the SQL task.
31


Reviews:
[{"review_id": "YNUrSG3vSI", "reviewer": "Reviewer_af37", "summary": "The paper introduces StateFlow, a framework for enhancing task-solving processes in LLMs by using finite state machines (FSMs) to model complex workflows. The primary idea is the combination of state-driven control and dynamic interaction with external tools, thus improving the efficiency and interpretability of LLMs in multi-step problem-solving tasks. By structuring the LLM's task-solving process into distinct states and transitions, it enables better control and guidance, improving performance metrics over baseline methods like ReAct.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and suggestions!\n\n**> 1st Point**\n\nWe do admit that the concept is present in the domain of LLM agents. However, we kindly argue that StateFlow has a different research focus and both of us make contributions to the community.\nSpecifically, StateFlow focuses on offering a formalized and systematic framework to construct LLM workflows, enhancing clarity and interpretability. While other works introduce specific workflows for agents. For example, LATs  unifies reasoning, acting, and planning with MCTS and ADAPT [4] explicitly plans and decomposes subtasks iteratively.\n\n**> 2nd Point**\n\nWe acknowledge that benchmarks like ALFWorld could be perceived as “controlled” due to its structured tasks. But we would like to highlight that the Bash Task is derived from real-world pages, showcasing StateFlow’s effectiveness. We believe that the three diverse datasets we use adequately show the applicability of StateFlow. \n\n**> 3nd Point**\n\nDue to the limited rebuttal time, we are not able to provide the experiments for other baselines in such a short time, especially given that some methods lack available source code. We really appreciate your suggestion and will definitely include more baselines in the future version after we carefully implement them!\n\n**> 1st question**\n\nStateFlow has two mechanisms to handle highly dynamic environments. Specifically,\n(1) As mentioned in Section 3.2, states can be designed to be less specific (e.g., using a single state to handle all errors), making them more adaptable to dynamic environments. \n\n(2) StateFlow can also fit tasks with evolving goals by creating state transitions that detect changes in task objectives. When the change is detected, the system can transit to a handling state for the anomaly and alert developers to take appropriate action. The modular nature of StateFlow allows for easy modifications: new states can be added without altering the overall behavior, and pre-defined states can be reused by simply adding new connections.\n\n**> 2nd question**\n\nFor complex tasks involving multiple steps and conditions, StateFlow provides a structured and simplified approach to state management. Specifically, (1) each condition is handled individually, making the system easier to debug and improve. (2) Running trajectories (such as those shown in Figure 3) can be easily retrieved for analysis, and failures can be examined (as detailed in Appendix B.2) to refine the StateFlow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The use of FSMs to structure task-solving processes helps in breaking down complex tasks into clearer, more manageable steps, which enhances the interpretability and \"debuggability\" of LLMs in complex scenarios.\n- The paper presents empirical evidence where StateFlow outperforms existing methods like ReAct in specific benchmarks with lower cost, indicating its effectiveness in controlled scenarios.\n- The potential ability to integrate StateFlow with other iterative and refining methods.", "reasons_to_reject": "- For me, the main concern of this paper is that the core idea of applying state machines to manage LLM workflows seems to repackage a well-established concept without providing transformative novelty. In the context of LLMs, while the paper presents this as a novel methodology, lots of agent already consider the similar ideas, albeit without explicitly claiming the FSM label. \n- It seems that the demonstrated benefits of StateFlow are primarily shown in \"controlled\" benchmarks, which might not fully convince of its broad applicability or superiority in diverse real-world scenarios. \n- I can resonate with the authors' choice of baselines, as there are indeed many agent-based methods that are complex and difficult to implement. However, I feel that the choice of baselines might be slightly weak. I would have liked to see comparisons with other methods, many of which are referenced in the related work section. This would help to contextualize StateFlow's performance and innovative value more robustly within the current landscape.", "questions_to_authors": "- It seems that the states required to be pre-defined. Considering that human cognitive processes are not strictly confined to finite states, can you provide more insights on how StateFlow performs in highly dynamic environments where the tasks or the nature of the interactions change over time? Specifically, how does StateFlow adapt when faced with tasks that have evolving goals or when unexpected changes occur in the environment?\n- Any thoughts or ideas on some more complex tasks, e.g., SWE-bench or real-world Web interaction?", "extra_scores": {}}, {"review_id": "4qCdnUKBcL", "reviewer": "Reviewer_voLS", "summary": "StateFlow presents a structured approach by modeling task-solving as state machines, allowing for a systematic progression through tasks, which slightly enhances the efficiency and interpretability of LLMs. The method shows modest improvements in performance on benchmarks such as InterCode SQL and ALFWorld, suggesting potential benefits over existing models like ReAct.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions. We tested the ALFWorld dataset with Mistral-7b, Llama3-8b, Llama3-70b and show the results below.  Additionally, we tested the stronger 3-agent ALFChat for comparison.\n\n## Mistral-7b\n| Method            | Accuracy | Cost  |\n|-------------------|----------|-------|\n| ALFChat (3 agents) | 1.5%     | 0.708 |\n| StateFlow         | 6%       | 2.03  |\n\n\n## Llama3-8b\n| Method            | Accuracy | Cost  |\n|-------------------|----------|-------|\n| ALFChat (3 agents) | 50.7%    | 0.52  |\n| StateFlow         | 62.7%    | 0.38  |\n\n## Llama3-70b\n| Method            | Accuracy | Cost  |\n|-------------------|----------|-------|\n| ALFChat (3 agents) | 83.6%    | 2.53  |\n| StateFlow         | 94%      | 1.83  |\n\nWe notice that Mistral-7b struggles to complete the task using both methods. This suggests that the Mistral-7b model itself is not equipped to handle tasks within ALFWorld. Additionally, we observe that StateFlow outperforms 3-agent ALFChat while also requiring fewer computational resources. This highlights the effectiveness of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method appears to perform well on tasks from both InterCode and ALFWorld. It also seems to reduce costs when compared with ReAct and ALFChat.", "reasons_to_reject": "To validate the effectiveness of the method, it is recommended to conduct more experiments using other models. How does this method perform with open-source LLMs such as LLaMA3 and Mistral (e.g., 7B and 70B versions)?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fxIpKeMQmj", "reviewer": "Reviewer_JAb2", "summary": "The paper addresses the broad topic of LLM prompt efficiency, specifically in the context of task-oriented interactions with external tools. The paper's main contribution is to propose structuring interactions as a task-specific finite state machine, where the task is decomposed into subtasks (states) with a dedicated instruction and criteria for transitioning out of the state. This formalism is evaluated on two LLM model checkpoints (GPT-4-Turbo, GPT-3.5-Turbo) across three broad tasks: SQL and Bash tasks from the InterCode benchmark and ALFWorld, a text-adventure environment. The proposed method is compared against the InterCode benchmark methods, ReAct and Plan & Solve, reporting on success rate, error rate, and LLM usage/cost.\n\nThe paper is generally easy to read and well motivated. The key innovation of the paper is the development of a state machine-inspired decomposition approach to LLM prompting. While motivated by and demonstrated on tool interaction tasks, this concept could be adapted to others where the the broader request can be decomposed into subtasks that have individual success criteria. The primary demonstrated benefit is efficiency -- fewer inference calls with fewer tokens, thus less cost, to achieve competitive or better results.\n\nThe primary weakness of the the paper is in its structure. It spends a lot of time describing finite state machines from a formal basis and attempting to match elements of the StateFlow method to this. However, the paper does not motivate why this is necessary or what value there is in doing so, especially as it has to modify the original FSM formalism to account for elements specific to the LLM interaction setting. There is an interesting result here, but the desire to mathematically align with the FSM definition gets in the way. It should be enough to say that StateFlow is inspired by FSMs, provide a brief schematic, and then move on to the actual details of the experiments.\n\nThe paper acknowledges that StateFlow's success is entirely dependent on the implementer to define an FSM with associated instructions and transition criteria for the specific task. In its current form, this makes it more of a strategy for structuring solutions to problems than a general method. Finally, in the appendix, there are details to enable reproducibility, including providing code in GitHub.\n\n======\n\nEdit: After rebuttal, the decision has changed to accept based on promised changes", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**> 1st rejection point: The paper should be revised first in order to be stronger….**\n\nThank you for your valuable feedback! We included detailed FSM definitions to provide a formal structure and connect our method to these concepts to aid understanding.  We appreciate your suggestion and will revise the paper to condense these sections. We'll also move more experimental details and analysis from the appendix to the main text to strengthen the paper in our future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of decomposing complex requests into a state-machine structure is worth promoting.", "reasons_to_reject": "The paper should be revised first in order to be stronger. Too much of the paper is spent reviewing the formal definition of finite state machines and attempting to adapt that to the LLM setting. This should be removed or relegated to the appendix, and replaced with content already in the appendix around the experimental details and analysis conducted.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iGoRiWKsdw", "reviewer": "Reviewer_t4Kc", "summary": "This paper proposes a task-solving paradigm, StateFlow, which conceptualizes complex task-solving processes as a state machine. This approach separates the control of task decomposition from sub-task solving and achieves better performance with fewer interaction turns compared to baselines on the InterCode SQL and ALFWorld benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**> 1st Reject Point: This workflow seems effective in complex task-solving…**\n\nThanks for your feedback. We admit that our method is limited to simple tasks, especially those that only require one single step. We kindly clarify our method is mainly focused on complex task-solving processes that require several steps or interactions with tools and environments. We will make this point more clear in our future version. \n\n\n**> 2nd Reject Point: This framework is not entirely independent of human efforts…**\n\nThank you for your feedback. We agree that our method involves human effort and task observation.  We kindly argue that it is common across different tasks and environments, as seen in many related works [1, 2]. We are also aware of this concern and we provide clear guidance in Section 3.2.\nAdditionally, the state's design is not a complex procedure. Specifically, it's not essential to account for every possible scenario; states can be designed to be broadly applicable to various situations. For instance, in the bash task, we created only Solve, Verify, and Error states despite the variety of questions and potential errors. While the ablation experiments show performance drops, it is important to note that these results still surpass the baselines with significantly less cost. \n\n[1] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.\n\n[2] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023.\n\n**> 3rd Reject Point: This framework primarily enhances process control…**\n\nThanks for your feedback! We admit that our method focuses on process control by breaking down a complex workflow into state transitions and states. But it also provides the model with more specific and concise instructions for each action to enhance the model's reasoning abilities and reduce errors. We also want to note that our method is agnostic to reasoning works like ToTs and Reflexion to further improve reasoning and reduce tool parameter errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. LLM-based complex task-solving is a significant research area, and this author tackles the problem through a controllable paradigm design, which enhances effectiveness and has the potential to advance development in this field.\n2. The proposed workflow achieves higher performance in two task scenarios while utilizing fewer interaction turns, demonstrating its utility.\n3. This paper is well-written and includes all necessary details.\n4. The experiments convincingly demonstrate performance improvements over various baselines, including ReAct and multi-agent architectures.", "reasons_to_reject": "1. This workflow seems effective in complex task-solving, but for simpler tasks such as translations or reading comprehension, which are foundational abilities of language models, it may not offer significant improvements. This may also explain why the performance gain is greater on the ALFWorld benchmark than on the InterCode SQL benchmark.\n2. This framework is not entirely independent of human efforts. To deploy StateFlow for a specific task, we must carefully define the States, which are central to adapting it for specific tasks, along with output functions and transitions. Designing states is challenging because it initially requires observing task-solving examples. With the basic workflow in mind, we must consider all possible scenarios that might arise during the process. Moreover, as the ablation experiments show, the omission of any essential state can lower performance. This task presents challenges not only for models but also for humans.\n3. This framework primarily enhances process control and does not improve reasoning or subtask solving; therefore, common errors, such as tool parameter errors, still occur.\n\n============================\n\nAfter rebuttal:\n\nInitially, I gave the paper a relatively high score due to its interesting method. If the authors can solve my questions, **I will raise the score to defend this work**. However, the authors just agree with my opinion but **do not make an effort to address my concerns**. Thus, I **decrease the score**.\n\nAfter another rebuttal:\n\nI have changed the score to 7 and did not raise the score due to the limitation of the paper in the method. However, I still think this work is interesting and deserves to be published.", "questions_to_authors": "See Reasons To Reject.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a StateFlow framework to better control LLM to solve highly complex tasks. The method utilizes the concept of FSMs to model the task-solving process in LLMs, and proposes a few common components (observe, select, verify etc.) to help the model transit between different states, reaching a more correct final state for solving the task.

Pros:
- Reviewers agree that the proposed method is relatively novel and well-motivated: StateFlow helps LLMs break complex tasks into clearer, more manageable sub-steps. This could improve the controllability and interpretability of LLMs.
- The empirical gains are fairly significant over multiple tasks, with a reduced cost.

Cons:
- Most reviewers point out that the biggest weakness of this paper is that all the states need to be pre-defined and all the tasks experimented can be well-controlled. It is difficult to know whether such framework can be easily adapted to real-world complex tasks.

Minor:
- Some reviewers mentioned that the structure of this paper can be made more clear, like the description of FSMs can be condensed and the authors should provide more details and analysis for the method and experiments.
Decision: Accept