SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks
Authors: Bill Yuchen Lin
Yicheng Fu
Karina Yang
Faeze Brahman
Shiyu Huang
Chandra Bhagavatula
Prithviraj Ammanabrolu
Yejin Choi
Xiang Ren
Venue: NeurIPS 2023 spotlight (2023)
Keywords: interactive reasoning
text game
agents
action planning
large language models
TL;DR: SwiftSage: An Agent Framework for Complex Interactive Reasoning Tasks, integrating the strengths of behavior cloning of smaller LMs and prompting LLMs to improve task completion performance and efficiency.

Abstract: We introduce SwiftSage, a novel agent framework inspired by the dual-process theory of human cognition, designed to excel in action planning for complex interactive reasoning tasks. SwiftSage integrates the strengths of behavior cloning and prompting large language models (LLMs) to enhance task completion performance. The framework comprises two primary modules: the Swift module, representing fast and intuitive thinking, and the Sage module, emulating deliberate thought processes. The Swift module is a small encoder-decoder LM fine-tuned on the oracle agent's action trajectories, while the Sage module employs LLMs such as GPT-4 for subgoal planning and grounding. We develop a heuristic method to harmoniously integrate the two modules, resulting in a more efficient and robust problem-solving process. In 30 tasks from the ScienceWorld benchmark, SwiftSage significantly outperforms other methods such as SayCan, ReAct, and Reflexion, demonstrating its effectiveness in solving complex interactive tasks.

SWIFTSAGE: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks
Bill Yuchen Lin
Yicheng Fu 4 Karina Yang" 2 Faeze Brahman 13
Shiyu Huang
Chandra Bhagavatula
Prithviraj Ammanabrolu 67
Yejin Choi 31
Xiang Ren 21
Allen Institute for Artificial Intelligence
University of Southern California UUniversity of Washington 6 44Parddigm Inc.
TTinnghua University
University of California, San Diego
MMsaicML
https://swiittsagg.iittth
Abstract
We introduce SWIFTSAGE, a novel agent framework inspired by the dual-process theory of human cognition, designed excel in action planning for complex interactive reasoning tasks. SWIFTSAGE integrates the strengths of behavior cloning and prompting large language models (LLMs) enhance task completion performance. The framework comprises two primary modules: the SWIFT module, representing fast and intuitive thinking. and the SAGE module, emulating deliberate thought processes. The SWIFT module is small mccoder-deoodd LM fine-tuned on the oracle agent's action trajectories, while the SAGE module employs I.LMS such as GPT-4 for subgoal planning and grounding. We develop heuristic method harmoniously integrate the two modules, resulting in a more efficient and robust problem-solving process. In 30 tasks from the ScienceWorld benchmark, SWIFTS significantly outperforms other methods such as SayCan, ReAct, and Reflexion, demonstrating its effectiveness in solving complex interactive tasks.!
Introduction
The advancement of artificial general intelligence is largely dependent on the development of agents that are proficient in complex interactive reasoning tasks. These agents should be capable of exhibiting problem-solving abilities akin to humans within dynamic, open- open-world environments [26. 7]. For example, the Science benchmark [36] features task where an agent must determine the electrical conductivity of an unknown object. In a simulated environment, the agent must navigate to appropriate rooms, locate and acquire essential items, such as batteries and light bulbs, build circuit, perform an experiment, and interpret the results. Tackling such a complex interactive task demands agents to exhibit long-horizon planning, long-term memorization, subgoal decomposition, spatial reasoning, exception handling, and commonsense knowledge capabilities [37].
There three primary approaches developing agents capable of addressing complex interactive reasoning tasks: (1) (deep) reinforcement learning (RL). (2) behavior cloning (BC) [34] through sequence-to-setee to-e-uunceeee (seq2seq) learning [33], and (3) prompting large language models (LLMs) [6]. In addition to conventional RL methods such as DRRN [14], interactive reasoning can be framed as seq2seq task, where the input text serves the current state description and the output text corresponds to the subsequent action [9, 3]. By leveraging numerous gold trajectories generated by oracle agents, becomes feasible to fine-tune Transformer models [35], like T5 [25], to effectively imitate the
CCntactt: yuchen @all wenalenaii.. enai.org
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Ror Rach Pimestep
Por Rach Pimestep
Rop Rnch Bound:
Gracle paths for training data.
Demo: An oracle 2ath
Dema: orace path manually annotated subgoels
The full action history of the previous falled round.
offline imitation learning
task of boiling water
Task Your is to bail tin.
Task: Your task is to boil ön.
Small LM
Astion ge litshen
Action to kitchen
LLM
reflection
Only when needed:
> you moved to kitchan
you moved to kitchen
Demo: w/j manual annotations
Action look araund
Action 2: laok around
3> Por Kach Timestep
Task: Your tas is to boil tin.
> In this kitchen, you can see...
you
Demo: ReAct's version.
Action history <m i-]
Astion 2:4: pick >> metal pot in nwentory now
pet
(::: think now need place the metol pot on heater Ok.
Task:
Your task boil tin.
Promots for object locating, planning & tracking subgonls and online exception handling
reflection of previously falied ars in the ast round.
LLM
GPT-4]
Action metal
metal pot in inventory now
Action history (c= to 1] for this rsandd Redct)
LLM
Next subgoels
best generations for Antion
LLM
LLM
Brounding
LLM
Reranking
Templates
Action Buffer: Action
Aztion put metal pat an stave
Action t. pat pat store
Action metal
Action t+ Action t+2....
SayCan
ReAct
Reflexion
SwiftSagge
Figure I: Comparing methods of prompting LLMs to build agents for interactive tasks.
behavior of these oracle agents. Recent studies have also demonstrated that generative agents based on prompting LLMs, such as GPT-4, can produce reasonable plans and actions [18, 15, 32].
Although the aforementioned methods exhibit remarkable performance in relatively simple tasks, their ability to generalize to more complex and demanding tasks is limited. Both RL-based and seq2seq-ban -based BC approaches effectively acquire knowledge from the environment through large- scale interactions and learn general action patterns from oracle agents. However, they face difficulties in decomposing tasks into subgoals, maintaining long-term memory, generalizing to unseen tasks, and handling exceptions. In contrast, instruction-tuned LLMs [24] demonstrate the ability to generate reasonable high-level plans for complex tasks and adapt their outputs based on human feedback. Yet, grounding their outputs executable actions in the environment remains challenge. These procedures also lack the capability to efficiently handle enviromment-specifee exceptions that prevent agents from adhering to the LLM's plans. Additionally, previous methods such SAYCAN [1], REACT [41] and REFLEXION [30] require a new inference with LLMs for each time step, making them considerably costly and inefficient (see Figure 1).
Inspired by the dual process theory [39, 16), we propose a novel framework that enables agents to closely emulate how humans solve complex, open- world tasks. The (roce-p theory posits that human cognition is composed of two distinct systems: System 1, characterized by rapid, intuitive, and automatic thinking; and System 2, which entails methodical, analytical, and deliberate thought processes. System is reminiscent of seq2seq methods, which learn through imitation of oracle agents and primarily operate utilizing shallow action patterns. Conversely, System bears resemblance to LLMs that excel in applying commonsense knowledge, engaging in step-by-step reasoning, devising subgoal strategics, and exercising self-reflection. Thus, our proposed method, SWIFTSAGE, is designed to enable both fast and slow thinking in complex interactive reasoning tasks. It effectively integrates the strengths of behavior cloning (representing System 1) and prompting LLMS (emulating System 2), resulting in significant enhancements in task completion performance and efficiency.
the SWIFT module and the SAGE module. The SWIFT module is a small encoder-decoder LM, fine-tuned on a T5-large (770m) checkpoint using the searched oracle trajectories of training tasks. It encodes short-term memory components, such as previous actions, observations, visited locations, as well as the current environment state. Then, it decodes the next individual action. This module simulates the fast, intuitive thinking characteristic of System 1. The SAGE module, representing the deliberate thinking of System 2, utilizes LLMs, such as GPT-4, and is structured around two prompting stages: planning and grounding. In the planning stage, we prompt LLMs to locate necessary items, plan and track subgoals, as well as detect and fix potential exceptions and mistakes. In the grounding stage, we focus on utilizing LLMs to transform the output subgoals derived from the planning stage into a sequence of actions by demonstrating potential action templates. Unlike prior methods, where LLMS only generate the next immediate action, our procedures engage in longer-term action planning. To harmoniously integrate the SWIFT and S SAGE modules, we developed a heuristic algorithm that determines when to (de)activate the SAGE module and how to combine the outputs effectively with action buffer mechanism.
Specifically, SWIFTSAGE consists of two primary modules:
2

---

In a comprehensive evaluation on 30 task types from the Science World benchmark, SWIFTSAGE significantly outperforms other methods, achieving a state-of- t-ttherrt average score of 84.7. In comparison, SAYCAN scores 33.8, REACT obtains 36.4, and SWIFTS AGE more cost-effective and efficient, requiring much fewer tokens per action for L.LM in- ference than previous methods. This considerable performance advantage highlights the effectiveness and efficiency of the SWIFTSAGE framework in addressing complex interactive tasks.
reaches 45.3. Moreover,
2 Background and Related Work
2.1 Complex Interactive Reasoning
We define interactive reasoning as the problems where agents are tasked with accomplishing goal within an interactive environment, typically simulated by engines such as AI2Thor [17] and TextWorld [11]. Our focus lies the textual environment of ScienceWorld [36] and the complex interactive tasks it supports. Simple interactive tasks, like those created in ALFWorld [31] and TWC [21], primarily involve searching for and placing objects as well as performing basic actions within a single location. Many of these simple tasks have been almost solved by recent works.
In contrast, tasks in Science World exhibit greater complexity, characterized by more challenging task planning and a significantly larger action space (encompassing 10 locations, 200 types of objects with varying states, and 25 types of actions). Furthermore, agents may encounter random, unforeseen obstacles, such as broken stoves or missing soil, which hinder the execution of planned actions. a result, agents must adapt and re-plan accordingly. for example, by seeking alternative heat sources or using a shovel on the outside ground to get soil. These challenges demand that agents possess skills in oonghooizonn planning, long-term memory, subgoal decomposition, exception handling, and commonsense nowleegge-capabiltteee that are not explicitly required for simple interactive tasks.
2.2 Reinforcement Learning and Imitation Learning Methods
DRRN. Interactive tasks can naturally be framed sartially-observable Markov decision processes (POMDPs), enabling the application of RL-based methods. Deep Reinforced Relevance Network (DRRN) [14] is a standard baseline method to learn agents within text environment. It aims to learn representations of observations and actions separately and train a policy network to select actions from candidates based feedback from the simulated environment. CALM [40] is reranking- based method that combines DRRN with a causal language model (LM) fine-tuned with oracle transcripts. In essence, the causal LM captures task-specific and enviromment-specifee knowledge through imitation learning, and the DRRN learns to rerank the predictions from the LM.
The KG-A2C [2] method uses an OpenIE technique [4] to represent environment states with graph structures and dynamically update these graphs. These graphs guide policy networks by constraining the combinations of action templates and objects. This method has been shown to be effective in other domains such as for multimodal embodied agents [22].
Behavior cloning for offline imitation learning. Behavior cloning is an imitation learning method that trains seq2seq Transformer offline with action transcripts of similar training tasks generated by oracle agents [34, 3]. During training, it uses the previous action, observation at time step / - 1, and the current observation as input and learns output the next action. The Text Decision Transformer (TDT) is textual variant of the Decision Transformer [9], which also employs behavior cloning and uses the same data. The primary innovation of TDT is the introduction of reward-to-go as part of the inputs, enabling the model to learn predicting actions that maximize future expected rewards.
2.3 Prompting LLMs for Action Planning.
Language models (LLMs) such GPT-4 have shown promise for action planning in interactive tasks [18, 15, 32, 38). In this paper, we adapt three prominent methods to complex interactive reasoning tasks in ScilnceWorld: SAYCAN [1], REACT [41], and REFLEXION [30].
SAYCAN [1] is a straightforward agent that integrates an LLM with a value function of underlying policies regarding grounding affordances (i.e., the feasibility of action in the environment). We
3

---

need to provide the history and current environment as textual inputs to LLMs for generating ranked list of action candidates. This action list is then reranked based on a value function.
REACT [41] presents a virtual "think" action, enabling LLMSs to generate subgoals during action planning. This approach requires human annotators supply examples of correct subgoals for each task type, employing few-shot in-context learning to teach LLMs when and how to 'think' in order to plan subsequent subgoals, in addition to providing complete action trajectories.
REFLEXION [30], a recent work building on REACT, proposes a multi-round approach enabling LLMs use the history of previously failed rounds to refine their planning for the next round. This self-reflection mechanism helps LLMs improve after each failed attempt. However, this may not be practical in real-world applications for many tasks, as actions in failed trials can be irrecoverable.
All three methods require a new LLM inference at each time step to predict the next immediate action, resulting in inefficient and costly agents. REACT and REFLEEXOON require human annotations of correct subgoals for each unseen task type. Moreover, is difficult to generalize REFLEXION to real-world situations where trial-and-erro approaches can be infeasible for embodied tasks.
2.4 Dual-Process Theory
The dual-process theory [39, 16] is a cognitive psychological framework proposing the existence of a fast and slow thinking systems in the human brain. This influential theory has found widespread applications across various fields, highlighting the critical role of both systems in shaping human cogmition [5, 8. 12. 20, 23]. By integrating the complementary strengths of both systems, agents can effectively and efficiently handle diverse challenges in real-world scenarios. Inspired by this, we aim construct generative agent that utilizes a small seq2seq LM System for associative reasoning via behavior cloning while developing System 2 for analytical reasoning by prompting LLMs.
3 SWIFTSAGE: A Generative Agent with Fast and Slow Thinking
In this section, we first establish the problem. Then, we present the two core modules, SWIFT and SAGE, individually. Lastly, we demonstrate the integration of these two modules. resuutinn in a harmonious and effective interactive reasoning process.
3.1 Problem Formulation
Environment and tasks. We focus on complex interactive reasoning tasks situated in virtual textual environments such as ScienceWorld [36]. Science provides an optimal setting for developing and evaluating agents in complex tasks, comprising 30 distinct task types covering 10 topics in science experiments. It features 10 locations, including an art studio, workshop, kitchen, living room, bedroom, bathroom, foundry, greenhouse, outdoor area, and connecting hallway. The environment includes 200+ object types with multiple states (e.g., open, activated) and supports 25 action templates, resulting in an intractable search space. The simulator can generate numerous variations of each task type, providing a rich training ground. In each variation, the agent and environment initialization, such as the locations and states of objects, will differ. A plethora of training variations encompassing all task types are available for training agents. Additionally. it provides a handcrafted oracle agent to search for successful transcripts with minimal actions for offline learning.
Evaluation done on set of testing variations with unseen combinations of required objects and situations, thus substantially different from the training variations. For example, a training variation may involve boiling water, while a testing variation could require boiling tin. Therefore, is crucial to ensure the agent's 's compositional generalization ability for effectively handling real-world scenarios.
Interactions. Given a task variation, an agent is provided with the task description D and the initial enviromment state (t = 0). The task description D is a text specifying a high-level goal, e.g., "Your task to test if an unknown substance A is electronically conductive." " At each time step i, the agent generates an action A, based on set of supported action templates (e.g. pick up X, use X Y). Ag is always "1 oook a around" for showing initial environment information. Upon receiving an action from the agent, the environment produces feedback in four dimensions:

---

Task: Your task is to melt ice cream.; Time: 14; Score: 35; Action history; <extra id 0> Action 5 (+5); open fridge
-> You opened fridge. you see ice cream... [..]] <extra Action 14 (+0): move metal pot to stove
> You move the metal pot to the stove, 
 Current environment: This room kitchen. You see: a fridge (closed) a sink an oven (closed, turned off) stove (turned off; on it: a metal pot containing ice cream ...).
[!...]]
Inventory. an orange, ... 
 Visited: workshop, hallway, kitchen 

What should be the action?
Swift:
ITS-large imitation learning)
Next action:
Obs. 15:
The stove appears
X
Time switch
activate stove
broken and can't be activated.
Sage: Prompting LLMs (GPT-4) for Planning and Grounding the Next Subgoals
Task History (t=1-15) Env.
Plans: Q1: Q2:
Q::
Q: & Qz: Locate Needed Objects
Q4: 1. Find working heat source
Q: & Qs: Plan & Track Subgoals
2. Place metal pot with ice cream on heat source
3. Wait until the ice cream melts,
Qs: Detect & Exceptions
Qs: You tried to use the broken stove heat source. To
Action buffer
Ground
this, you should try using the oven the kitchen or
open oven
Env
move metal pat to oven close oven
Demo + Task Env. History + Plan Action Types:
</>
LLM
POUR By: pour object container e.g. POUR/water, pot)
%
activate oven
Convert next subgoals to sequence actions:
An example of how SWIFTSAGE works with fast and slow thinking. The SWIFT module is offline trained via imitation learning with small LM such T5-large (770m). When it is necessary, for example, encountering an exception, we switch to the SAGE module that prompts LLMs (e.g., GPT-4) for planning and grounding the next subgoals, resulting in an action buffer.
Figure 2:
Observation O provides direct feedback on the action A, regarding its effects on the envi- ronment the information queried. For example, A, of "use thermometer substance in metal pot" may result an O like "The temperature is 80F""
the
Environment E, represents the current room in which the agent is situated and provides details about all visible objects. Object visibility based on container states, e.g., objects within a closed fridge are not included in E, until the agent performs an action like "open fri fridge."
Inventory 4 lists objects picked up by the agent, which is particularly useful when agents collect items from different locations to complete the task.
Score S, represents the agent's cumulative score ranging from 0 to 100. When a required interme- diate state is achieved, the score increases with a positive reward.
3.2 SWIFT: The Module for Intuitive and Associative Thinking via Imitation Learning
Imitation learning is used to construct an agent that learns to mimic oracle agents in various training scenarios through seq2seq learning. Previous methods, such as TDT [36], mainly employ one-hop history as input context and learn to output the subsequent action A, [36]. However, these methods exhibit limitations due to their restricted context of action history and harmful biases arising from data imbalance. To address these issues, we introduce our SWIFT module, depicted in Figure 2.
Representation for longer history. We expand the conventional one-hop BC to multi-hop by incorporating a sliding window of observations and rewards for the K = 10 recent actions. Addi- tionally, we include a special field for visited rooms (without duplication). This approach aims to provide agents with a longer context and prevent unnecessary room navigation. The input format as follows: "Task: D: Time:
1; Score: S,-1; Action history: [Ar4 (+R,-1) O,- 1 1* loops from K to 1*/; Current room: F.-1: Inventory: Te-1; Visites rooms: (E) F F2 Here, R. S, S represents the reward at f, and FA is the location name at t..
Balanced imitation learning. To avoid bias caused by data imbalance for seq2seq learning, we down-sane sown-sampled specific types of tasks and actions to achieve more balanced final dataset for training. We used the Ta-lerge with 770 million parameter and instruction-follovin efficient agent that we named SWIFT. Our empirical results show that the SWIFT module performs much better than TDT (11 billion) despite being 15x smaller in size.
ability [10], creating an
5

---

The SWIFT module exhibits greater accuracy during initial time steps, enabling it to attain higher scores in the early stages of a complex task. However, it often fails to generalize unseen situations. The module also has a tendency to repeat meaningless actions when its learned plans yield exceptions from the environment (e.g., the broken stove in Figure 2). This partly due to the nature of imitation learning, which prioritizes emulating the observable actions of oracle agents rather than their intrinsic planning abilities. Besides, since the oracle trajectories contain only the shortest, correct actions, is thus also challenging for the SWIFT to learn how to fix mistaken actions.
3.3 SAGE: The Module for Deliberate and Analytical Thinking via Prompting LLMs
While the SWIFT module acquires surface knowledge about the environment and task types through imitation learning, lacks two key abilities essential for complex interactive reasoning: 1) gener- alizable planning and tracking of subgoals, and 2) robust handling of exceptions. Prior research has shown that LLMs outperform smaller LMs in these abilities. They can perform step-by-step reasoning to devise concrete plans for tasks and self-refine their outcomes. However, the performance of prior methods remains unsatisfactory in complex interactive tasks such as those in WorlnceWorld.
We introduce a novel two-stage approach, named SAGE. This method initially acquires higher-level recommendations from LLMs during the planning stage, followed by their translation into specific action sequences in the grounding stage. By decoupling the planning and grounding processes, SWIFTSAGE effectively generates a series of actions for completing the planned subgoals.
Planning stage. In this stage, leverage LLMs to plan based the current state. Specifically, we prompt LLMs with a single prompt that includes summarized version of the task description and action history, and asks the following five key questions:
Q1 (locate objects): "To complete the task, which objects do need to collect? Please list them and their possible locations one by one."
Q2 (track objects): "Are there any objects that have not been collected yet?"
Q3 (plan subgoals) "To complete the task most efficiently, what are the important sub- goals to achieve? Please list the subgoals one by one."
Q4 (track progress): "Considering these subgoals, what have I already completed? And which subgoal should 1 focus on right now?"
Q5 (handle except i ons): "Have I made any mistakes that might prevent me from effi- ciently completing the next subgoal? If any. how should fix them?"
Before posing the five planning-related questions, we condense the entire action history (Acr and Oct), and the current environment information 1 Q1 and Q2 pertain to objects, as acquiring all necessary objects serves as the foundation for effective task planning. By addressing these questions, we ensure that LLMs develop a comprehensive understanding of the current environment. Q3 prompts LLMs to engage in by-p-p planning by decomposing the task into a series of subgoals. Q4 acts as a follow-up question, allowing the agent to monitor progress based on the action history and determine completed subgoals. subsequently focusing on the remaining tasks. Lastly, Q5 is employed to identify and address potential exceptions. These questions can be further tailored with additional environment-sppecffe hints, thereby enhancing their adaptability.
To improve the structure of the LLMs' outputs and facilitate parsing, we incorporate additional instructions in the prompt. By utilizing single input to obtain answers to all five questions in one output, rather than engaging in multiple rounds of interactive prompting, our approach is more efficient and cost-effective than the iterative prompting methods.
Q4 and Q5 are of primary importance, while Q1-Q3 serve auxiliary guidance for the LLMs. If the action history indicates a mistaken action or an unachievable previous subgoal, the response to Q5 refines the answer to Q4 through self-reflection on the fly. This approach differs from the REFLEXXION agent, which only prompts reflective questions at the end of a failed trial, allowing agents to improve their planning in subsequent attempts. In contrast, method detects exceptions and errors each time the agent plans for the next subgoals, enabling earlier correction of the agent's behavior.
Grounding stage. While the answers to Q1-05 provide valuable guidance for agents, they are not directly executable. Converting plans into valid actions that can be accepted by the environment

---

remains a challenge. Previous methods using LLMs over over- generate candidates, and they rely on reranking or filtering based on the action space to select the next action. However, this inefficient and inaccurate for complex tasks with vast action spaces. Additionally, these methods generate single action at time, which can be both costly and ineffective for long-horizon tasks.
To tackle these issues, we first present supported action types using formal style accompanied by remarks. For instance, the action type "pour X into Y" is introduced "POUR (X, Y) pour object X into container Y; e.g., pour red paint into wood cup". More examples:
TELEPORT (room) directly go a room such as TELEPORT (kitchen) PICK (object) : pick up an object and put into your inventory
OPEN (object) : open an object to search or put things in it, e.g., OPEN (freezer).
ACTIVATE (objectt activate / turn on an object such as sink or stove, so that you can use it. DEACTIVATE (o jbct)) deactivate turn off the object
EXAMINE (object) look at object carefully. For example, EXAMINE (light bulb). MOVE (object, place) move/place the object to a place
We then incorporate the LLM's outputs from the planning stage as part of the input for the grounding stage. Furthermore, we provide the recent action history of the past 10 time steps as context. Finally, we prompt LLMs to concentrate on the next subgoal and convert it into a list of actions (rather than a single action) accomplish the next subgoal. Our formatting instructions enable the straightforward splitting and conversion of output actions from LLMs in the grounding stage back to their original action representations. We denote this list of actions generated by LLMs as the action buffer:
B (Á,, Ar+1, ...+,,..., }. One can opt to use only answers to Q4 and Q5 to reduce computational costs. Our small-scale ablation study indicates that incorporating answers to Q1-Q3 in the grounding stage proves beneficial, yielding a gain of about 2 points for short tasks on average.
3.4 Integration of Fast and Slow Thinking
Having described the SWIFT and SAGE modules, we now address the question how to merge both modules and effectively integrate fast and slow thinking within the SWIFTSAGE agent. We establish heuristic algorithm to control the activation and deactivation of the two modules.
Initially, we employ the SWIFT module due to its superior intuitive reasoning capabilities, which facilitate accurate associations between the task description and the environment during the first few actions. We will switch from SWIFT mode to SAGE when any of the following conditions are met:
1) Stuck: There are K=5 consecutive time steps with zero reward (Eil- R, 0).
2) Invalid:
The SWIFT'S prediction for the next action (A²) is invalid in the current environment. 3) Critical: A/, involves a critical decision, e.g., giving the final answer for the experiment result.
4) Unexpected:
The observation of A, suggests that an exception is encountered.
Upon activating the SAGE module, we execute the two-stage prompting process and generate an action buffer. We attempt to execute each predicted action and revert to the SWIFT module when the buffer is empty. This approach enables a seamless integration of both modules, providing an efficient and robust problem-solving process for the SWIFTSAGE agent. The pseudo code for illustrating the SwiftSage framework is shown in Fig. 4 (Appendix).
4 Evaluation
4.1 Evaluation Setup
To evaluate the effectiveness of SWIFTSAGE and other baseline methods in complex interactive reasoning tasks, we use the Science World benchmark. In Section 2.1 and Section 3.1, we introduce the benchmark and problem formulation. Each task type categorized as 'short" (S), 'medium" (M), or 'long' (I.) based on the average length of the oracle truth trajectories. However, the length of the task does not necessarily indicate its level of difficulty as some tasks may require additional commonsense knowledge. Further evaluation details are provided in the appendix.
7

---

Task Type
*enn
DRRN
KGA2C
CALM
TDT
SayCan
ReAct
Reflexion
SwiftSage
1-1 (I)
107.7
3.52
0.0
0.0
0.71
33.06
3.52
4.22
97.04
11)
78,6
3.52
0.0
0.0
0.44
10.39
13.70
10.61
87.04
(1)
88.9
0.0
4.0
0.0
3.88
3.88
7.78
7.78
72.78
(1)
75.2
0.0
0.0
0.0
0.55
0.37
9.88
0.92
100.0
2M)
21.4
6.56
6.0
6.16
26.37
7.19
5.92
99.17
20)
35.2
5.50
11.0
1.0
6.43
8.03
6.10
28.59
88.17
(1)
65.0
6.0
4.0
1.0
19.87
17.41
22.37
22.37
95.73
(S)
13.6
12.0
7.0
5.0
40.55
52.14
56.0
100.0
88.67
(M)
20.8
9.0
4.0
7.0
14.26
22.50
54.33
17.45
55.33
3-3 (M)
25.6
9.05
4.0
2.0
10.16
99.56
76.19
72.54
71.90
(4)
29.0
9.52
4.0
21.65
47.76
70.22
77.86
(9)
14.6
15.0
18.0
10.0
41.93
22.87
26.67
64.93
100.0
42(S)
8.8
45.0
44.0
54.0
55.76
58.18
80.0
87.27
100.0
43((S)
12.6
21.67
16.0
10.0
27.82
20.87
53.33
16.42
91.67
44(9)
14.6
19.17
15.0
8.0
47.15
31.43
27.50
100.0
100.0
(1)
69.5
8.0
6.0
2.0
6.89
9.92
9.06
7.33
74.59
(1)
79.6
14.29
11.0
4.0
11.86
13.93
18.57
13.0
93.93
6M1
33.6
15.77
17.0
3.0
15.10
47.81
51.04
70.35
49,40
(-2
15.1
26.67
19.0
6.0
15.70
39.26
58.89
70.67
100.0
6-3 (M)
23.0
10.37
4.0
3.0
5.25
19.72
40.74
15.77
91.48
(9)
7.0
50.0
43.0
6.0
30.0
80.0
60.0
100.0
95.0
7-2 (S)
7.0
50.0
32.0
10.0
8,43
67.50
67.50
84.37
85.0
(S)
8.0
33.33
23.0
8.34
50.0
50.0
83.0
93.33
(M)
40.0
21.0
5.0
4.0
3.86
20.91
27.67
2.58
89.0
(-2
16.3
8.0
10.0
0.0
8.0
16.0
8.0
8.0
68.50
(1)
97.0
10.0
4.0
0.0
2.53
21.94
40.50
50.63
75.0
(1)
84.9
10.0
4.0
3.0
14.66
32.26
44.0
100.0
70.0
(L)
123.1
10.0
9.12
13.67
70.62
60.0
10-1(L) (L)
130.1
16.80
11.0
2.0
1.51
67.53
25.70
50.90
92.30
10-2 (I)
132.1
17.0
11.0
2.0
1.29
59.45
16.80
23.69
77.60
Short
11.76
28.08
22.70
11.30
28.37
43.83
48.79
71.47
92.22
Medium Long 94.30
28.58
10.85
2.88
10.36
36.58
44.01
35.43
77.79
8.26
4.92
1.33
6.11
23.65
21.07
30.17
83.0
Overall 49.26
15.56
11.37
5.07
14.66
33.82
36.43
45.34
84.68
Table 1: Results on the Science benchmark. *Len is the average length of the oracle agent's trajectories. In addition overall results, also report performance on three groups of *Len (short, medium, long). The last four methods use GPT-4 as the base LLM for prompting. We show more details of these tasks and results on other LLMs in the appendix.
4.2 Baseline Agents
In addition to the baseline methods evaluated in the ScienceWorld paper, such as DRRN, CALM. KG-A2C, and TDT, we incorporate three LLM-based prompting techniques: SAYCAN, REACT, and REFLEXION, as detailed Section 2.3 and Figure 1. This subsection presents the implementation details for adapting these methods to build Science World agents.
SAYCAN necessitates value function from the environment for reranking purposes. We employ SentenceBERT [27] to rank all valid actions (generated by ScienceWorld' APIs) based on their similarity the top generations for A, from SAYCAN. We implemented REACT and REFLEXION in similar manner. Adhering to their released code, we utilized the best single generation and determined the valid action with the minimal edit distance, if required. Both REACT and REFLEXION necessitate subgoal annotations for teaching LLMs to plan with virtual 'think' actions. We annotated such truth subgoals by translating ScienceWorld's APIs into natural language, which was also employed by the oracle agents. For all agents, we incorporated the complete trajectories of one or two training variations from the same task type for in-context learning. Our primary experiments were conducted using OpenAl's GPT-4; however, other LLMs can be readily substituted required.

---

5-15
7-25
7-35
4.2.5
4-3.5
3-15
4.15
445
6.2
0:2.5
3.2
211
6:3
3:3 M
3-4 M
6:1
2:2 M
8-1 M
2:3.L
1-2LL
5-2.L
13L
91L
11L
93L
10 1L
10
Figure 3: Visualizing trajectories of SWIFTSAGE, REACT and ORACLE. X: time steps (0 + 7); Y: scores (0 >> 100). Each figure displays the merged trajectories of testing variations by an agent in each task. Task IDs are shown the bottom-right, and the ordering is based *Len in Tab
4.3 Results and analysis.
Main Results Table compares the performance of various agents across 30 types of tasks. Detailed descriptions of each task type can be found in the ScienceWorkde paper [36] and appendix. It is evident that L.L.-based methods outperform conventional agents due to their superior generalization ability, albeit at higher deployment cost. The behavior cloning model TDT [36, 9] (11b) performs on par with DRRN [14], but with greater efficiency in learning and inference. In contrast, our SWIFT- only agent (770m) achieves an overall performance of 49 22, which we attribute its balanced training data and the use of a sliding window for longer action histories.
REACT demonstrates noticeable improvement over SAYCAN for short and medium tasks, owing to its subgoal annotations for in in-context learning and the inclusion of 'think' actions. REFLEXION surpasses REACT in shorter tasks; however, comparing REFLEXION with other agents is not entirely fair. REFLEXION can run up to four rounds, while the others are limited to one round. This discrepancy is particularly unfair for tasks involving multiple-choice scenarios. Nevertheless, include REFLEXION'S s results to analyze the potential of such methods.
Exception handling. Consider the example in Figure 2, where the stove broken, presenting an exception. Agents like DRRN and TDT often resort repeating meaningless action sequences (e.g., continuously attempting to activate the stove or moving between rooms aimlessly). Although the SWIFT module, when used independently, improves upon this due its larger context window from imitation learning, it still struggles to address exceptions robustly. ReAct and Reflexion occasionally utilize the 'think' action or reflections redirect agents towards alternative solutions, but the generated actions rarely achieve the new subgoals if they are not grounded. In contrast, the plan-and aad--nd--gaen and-ground prompts in our SAGE module handle exceptions more effectively.
Cost-effectiveness. Despite SAGE invoking LLMs APIs twice for inference, its overall cost remains lower, as the result is sequence of actions typically containing about actions. In comparison, SAYCAN and REACT require 1,855.84 and 1,971.03 tokens per action (tpa) respectively, while REFLEXTON necessitates 2,983.46 tpa. SWIFTSAGE on the other hand, only uses 757.07 tpa. Given its superior performance, SWIFTS AGE proves more cost-effective than other M-b.eed methods. This efficiency is primarily attributed to invoking LLMS only when needed (courtesy of our strong SWIFT module) and the action buffer mechanism.
Efficiency. To thoroughly examine the efficiency of agents across all task types, we use Figure to visualize the average trajectories of the first three testing variations for each task involving

---

SWIFTSAGE, REACT, and the oracle agent. We arrange the tasks based on their average lengths of oracle trajectories (*Len in Table 1). We observe that oracle trajectories consistently achieve perfect scores, yet SWIFTSAGE can reach similar scores more efficiently. This is particularly evident in longer tasks (the bottom two rows), although SWIFTSAGE does not achieve a perfect score for few tasks (e.g., 9-2 and 1-3). Interestingly, we find that REACT performs competitively in shorter tasks (e.g., 4-2 and 3-4), but most trajectories plateau at an intermediate score and fail to reach 100.
More analysis. Due to page limit, we have to provide further details and analysis in the appendix, including more detailed analysis on cost-effectiveness and efficiency, additional case studies and ablation studies, sensitivity to LLM choices, and an the evaluation of the SwIFT-only agent.
5 Conclusion
Contributions. We present SWIFTSAGE, generative agent for complex interactive reasoning tasks, inspired by the dual-process theory of human cognition. The agent framework comprises two modules: SWIFT, responsible for fast thinking, and SAGE, dedicated to slow thinking. The SWIFT module is a smaller LM that is fast and specialized, while the SAGE module focuses on prompting LLMs (e.g.. GPT-4) for subgoal planning and reflective thinking. Through extensive experiments on 30 distinct tasks within the ScienceWorks benchmark, SWIFTSAGE outperforms baseline agents, achieving state-of-the-ar performance, increased efficiency, and reduced cost.
Implications. The success of SWIFT SAGE highlights the potential for collaborative frameworks combining smaller LMs and LLMs in complex reasoning tasks. Smaller LMs can be trained more casily to recognize task-specific and environment-specific patterns, fostering effective in-distribution generalization. On the other hand, LLMs demonstrate remarkable zero-shot generalization abilities and deliberate thinking, though grounding their outputs in real-world environments remains chal- lenging. We posit that dual-process agents, harnessing the strengths of both approaches, constitute a crucial step towards addressing complex interactive reasoning tasks and building general AI agents. Additionally, we can regard SWIFTSAGE method within the broader context of utilizing LLMs as controllers or planners for decomposing complex tasks and leveraging APis/tools [19, 13, 29, 28). To this end, we have explored applying SAGE in web tasks and coding for math problems.
Limitations. Our work has been evaluated solely within textual simulator, Science which supports limited set of actions and tasks compared to real-world situations. Also, we did not implement any safeguards to prevent agents from engaging in potentially hazardous actions that could occur in the real world, such as picking up substances from blast furnace. We argue that one important future direction is to develop a true open-ended enviromment, allowing agents to interact with a much wider variety of actions and objects to better emulate real-world scenarios. Besides, the use of LLMs in SAGE may present scalability challenges, LLMs require significant computational resources and may not be feasible in some settings. Future research should explore the generalizability of SWIFTSAGE to other domains and the potential for more lightweight approaches to slow thinking. In addition, we believe it is important to train agents beyond simple supervised fine-tuning and to leam trainable module to decide when to switch between SWIFT and SAGE mode.
Acknowledgements
We thank Peter Jansen, Eric Xingdi Yuan, and Alex-Ale Côté for valuable discussions. We thank members of the INK lab at USC and the Mosaic team at A12 for valuable feedback on this project. Xiang Ren supported in part by the Office of the Director of National Intelligence (ODNI, Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-2207220000 the DARPA MCS program under Contract No. 1966031924033, the Defense Advanced Research Projects Agency with award W911NF-19 19-2071,, NSF IIS 2048211, and gift awards from Google and Amazon. This research was also supported by the DARPA MCS program through NIWC Pacific (N66001-19-2-400))
and Allen Institute for AI. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government.
10

---

References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog. Daniel Ho, Jasmine Hsu, Julian Ibarz. Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, 2022.
[2] Prithviraj Ammanabrolu and Matthew Hausknecht. Graph constrained reinforcement learning for natural language action spaces, In International Conference on Learning Representations, 2020.
[3] Prithviraj Ammanabrolu, Jack Urbanek, Margaret Li, Arthur D. Szlam, Tim Rocktaschel, and Jason Weston. How to motivate your dragon: Teaching goal-driven agents to speak and act in fantasy worlds. In North American Chapter of the Association for Computational Linguistics, 2020.
[4] Gabor Angeli, Melvin Johnson, and Christopher D. Manning. Leveraging linguistic structure for open domain information extraction. In Annual Meeting of the Association for Computational Linguistics, 2015.
[5] Thomas W. Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. ArXiv, abs/ (oss/175508439, 2017.
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[7] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg. Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro. and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv, abs/2303..1211 2023.
[8] Di Chen, Yiwei Bai, Wenting Zhao, Sebastian Ament, J, Gregoire, and Carla Gomes. Deep reasoning networks: Thinking fast and slow. ArXiv, Abs/1996.00085, 2019.
[9] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems, 2021.
[10] Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Fric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction- finettued language models. ArXiv, abs/2210.114411 2022.
[11] Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben A. Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew J. Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworld: A learning environment for text-based games. In CGW@ IJCII, 2018.
11

---

[12] M. B. Ganapini, Murray Campbell, F. Fabiano, L. Horesh, Jonathan Lenchner, Andrea Loreggia, Nicholas Mattei, Francesca Rossi, Biplav Srivastava, and Kristen Brent Venable. Thinking fast and slow in ai: the role of metacognition. In International Conference Machine Learning, Optimization, and Data Science, 2021.
[13] Yingqiang Ge, Wenyue Hua, Jianchao Ji, Juntao Tan, Shuyuan Xu, and Yongfeng Zhang. Openagi: When IIm meets domain experts. arXiv, 2023.
[14] He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with natural language action space. arXiv: Artificial Intelligence, 2015.
[15] Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents, ArXiv, (bss/2001007200 2022.
[16] Daniel Kahneman. Thinking, Fast and Slow. 2011.
[17] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual Al. arXiv, 2017.
[18] Bill Yuchen Lin, Chengsong Huang, Qianchu Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models. ArXiv, abs/2209.00465, 2022.
[19] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song- Chunn Zhu, and Jianfeng Gao. Chameleon: Plug-and- and-play compositional reasoning with large language models. ArXiv, abs/2304.09842, 2023.
[20] Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Think- ing fast and slow: Efficient text-to-visual retrieval with transformers. 2021 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 9821-9831 2021.
[21] Keerthiram Murugesan, Mattia Atzeni, Pavan Kapanipathi, Pushkar Shukla, Sadhana Kumaravel, Gerald Tesauro, Kartik Talamadupula, Mrinmaya Sachan, and Murray Campbell. Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines. In Thirty Fifth AAAI Conference on Artificial Intelligence, 2021.
[22] Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling. In International Conference on Machine Learning (ICML), 2023.
[23] Maxwell Nye, Michael Henry Tessler, Joshua B. Tenenbaum, and Brenden M. Lake. Improv- ing coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In Neural Information Processing Systems, 2021.
[24] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02111 2022.
[25] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67. 2020.
[26] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. generalist agent, ArXiv, has/2205.06173. 2022.
12

---

[27] Nils Reimers and Iryna Gurevych. Sentence-BERG Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language P Processing (EMNN.-IJCNI.P), pages 3982-3992, Hong Kong, China, 2019. Association for Computational Linguistics.
[28] Timo Schick, Jane Dwivedi-Yuu Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXiv, abs/2302.04761, 2023.
[29] Yongliang Shen, Kaitao Song, Xu Tan, Dong Sheng Li, Weiming Lu, and Yue Ting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. ArXiv, abs/2303. 17580, 2023.
[30] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. ArXiv, abs/2303.11366 2023.
[31] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, ArXiv, abs/2010.03768, 2020.
[32] Chan Hee Song, Jiaman Wu, Clay Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. ArXív, abs/2212.040888 2022.
[33] Ilya Sutskever, Oriol Vinyals. and Quoc V. Le. Sequence sequence learning with neural networks. ArXiv, abs/1409.3215, 2014.
2014.
[34] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. ArXiv, Abss/805.01954, 2018.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[36] Ruoyao Wang, Peter Alexander Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Conference on Empirical Methods in Natural Language Processing, 2022.
[37] Zekun Wang. Ge Zhang. Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao, Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xiuying Chen, Qingqing Zhu, Zhenzhu Yang, Adam Nik, Qi Liu, Chenghua Lin, Shi Wang, Ruibo Liu, Wenhu Chen, Ke Xu, Dayiheng Liu, Yike Guo, and Jie Fu. Interactive natural language processing. ArXiv, 2023.
[38] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open worldworld multi-tase agents. ArXiv, abs/2302.01560, 2023.
[39] Peter C Wason and J St BT Evans. Dual processes in reasoning? Cognition, 3(2):141-154, 1974.
[40] Shunyu Yao, Rohan Rao, Matthew J. Hausknecht, and Karthik Narasimhan. Keep calm and explore: Language models for action generation in text-based games. ArXiv, abs/2010.02903, 2020.
[41] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/2210.03629, 2022.
13


Reviews:
[{"review_id": "Eid1cUOM0k", "reviewer": "Reviewer_AXAQ", "summary": "This paper proposes a new agent framework for action planning in complex interactive reasoning tasks. The authors leveraged an imitation-learning trained \"light-weight\" LLM for fast inference and an LLM (e.g. GPT4) for slow reasoning with proper designs of prompts and interaction between fast/slow modules. The proposed model achieve state-of-the-art results on ScienceWorld, outperforming existing models by a large margin.", "questions": "See the Weakness section.", "limitations": "The authors have properly addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This idea of two-system in LLM is a new agent framework that could potentially be extended to many more settings and tasks.\n\n- The proposed model outperforms existing baselines by a large margin on ScienceWorld which proves the effectiveness of the two-stage design. \n", "weaknesses": "- One concern about the current framework is its generalizability to more common tasks. It seems that the current design is strongly related to the ScienceWorld benchmark, it would be best to see if the current prompt/interaction designs could be still effective in customized settings, how to define the prompts for a general task, and also how sensitive they are to for example customized defined states, scores, etc.\n\n- As 770M is already quite a big model before the LLM era, is the current Swift module capable of what we desire of accurate fast reasoning? This is reflected by the swift-only experiments that can already outperform existing RL/IL-based learning methods by a large margin. It makes one wonder to what extent can swift-module handle the reasoning problem, what are its failure cases, when does it need the Large LLM for slow reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer AXAQ,\n\nThank you very much for your thoughtful feedback and comments! We are glad to hear that you believe our paper is well motivated and addresses the important problem of interactive reasoning and planning. We appreciate your concerns and have addressed your two questions below.\n\n---\n\n### Q1: Generalizability to more tasks; how to customize the prompts for general tasks.\n\nThank you for this excellent question. In the newly uploaded one-page PDF, we present a general recipe for applying SwiftSage to other tasks, despite our focus on embodied tasks. The PDF demonstrates the customization of SwiftSage for coding tasks, web tasks, math problems, and complex Q&A.\n\nNotably, the first two task types have **interactive** evaluation environments (recently released) with straightforward observations and rewards. In contrast, the latter two receive limited feedback from existing benchmarking environments, and current datasets are unsuitable for testing agents in complex interactive tasks.\n\nWe are conducting experiments on these benchmarks and will include the results in the next version. Please note that many interactive coding task environments (e.g., InterCode [1]) and web task benchmarks (e.g., WebArena [2]) were released after the NeurIPS deadline. We are working through some technical issues in their codebases and contacting their authors for assistance. Nonetheless, our one-page PDF demonstrates the general method of customizing SwiftSage for complex interactive tasks, and we hope it can answer your question.\n\n### Q2: More error analysis for the Swift module; when does it need the Sage module.\n\nThank you for raising this important question. We previously investigated failure cases for Swift-only experiments. Due to space limitations, we cannot provide full examples here. Instead, we summarize our findings below.\n\n(1). Swift may generate invalid actions, necessitating the choice of one for execution. In preliminary experiments, we tested various decoding algorithms, all of which often generated invalid actions unparseable by ScienceWorld. Without the Sage module, we must either choose less confident yet valid actions or select the most similar valid action from a pool if all top candidates are invalid. In essence, if Swift is unconfident in its predictions or cannot generalize in the current situation, more risky actions must be executed, which can propagate mistakes to future steps.\n\n(2). Swift cannot replan when mistakes occur or exceptions are encountered, often leaving it lost in later time steps. This issue is partly due to Swift's limited context window for memorizing action history. In contrast, Sage can effectively reflect on action history and replan subsequent subgoals and actions, a skill Swift is not trained to perform.\n\n(3). Swift tends to make decisions before identifying correct objects. As a greedy algorithm trained with imitation learning, the Swift module aims to complete tasks quickly. However, it can overlook the absence of necessary objects or proper conditions, potentially leading to incorrect actions and missed score points.\n\n(4). Swift has limited commonsense knowledge. Some task variations require reasoning about object locations, affordances, animal lifespans, color mixtures, etc.\n\nThese scenarios demonstrate the need for Sage and why SwiftSage significantly outperforms the Swift-only module. Our motivation for the four conditions activating Sage stems from these observations. We will include these findings with more concrete examples in the next version.\n\n---\n\nPlease let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.\n\nBest regards,\\\nAuthors of SwiftSage\n"}, {"review_id": "tKZ1sAN8mp", "reviewer": "Reviewer_CSRK", "summary": "This paper proposes a dual-module approach for ScienceWorld which combines an imitation learning agent based on t5, with an llm-based module which activates based on a heuristic to re-plan its actions, using a buffer of actions to be executed.\nThe approach is novel, sensible and well-motivated, drawing from the concept of slow-fast thinking from psychology.\nThe results significantly advance the state of the art on ScienceWorld, and significantly outperform the compared methods which include state of the art LLM methods like Reflexion.", "questions": "See above.", "limitations": "Limitations discussion is adequate.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper significantly advances the state of the art on a difficult and interesting task involving language-based interaction, ScienceWorld. I expect that the proposed model will be a useful benchmark for many tasks of a similar type.\n- The method is sound and well designed. Using seq2seq for a history-aware imitation learning is a sensible model for \"fast thinking\". And using an LLM for \"deliberate thinking\", i.e. object and progress tracking, planning and exception handling is also reasonable. Both components are well crafted, and they are nicely integrated in a complementary manner.\n- The ablations are informative - Ablations show that the Swift module alone establishes a new state of the art, while GPT-4 almost doubles this performance. However chatgpt-turbo, which is comparatively much weaker than GPT-4, also improves Swift-only by about 25%, which is encouraging.", "weaknesses": "- I'm curious about the sensitivity of the method to prompt engineering. Q1~5 are quite comprehensive and capture a lot of the core aspects required to successfully solve the game. This is possible for ScienceWorld but would not be possible for many other tasks. Would it be possible to ablate the Qs to show prompt sensitivity?\n- A weakness of the model is that it relies on a heuristic to decide when to activate the Sage module. I'm curious whether there are no better alternatives or more general solutions to the problem of integrating the two modules?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CSRK,\n\nThank you very much for your thoughtful feedback and comments! We are glad to hear that you believe our method is well designed and significantly advances the state of the art in complex interactive reasoning, as well as finding our evaluation informative. We also appreciate your concerns and suggestions.\n\n---\n\n### Q1: Would it be possible to ablate the Qs to show prompt sensitivity? \n\nWe acknowledge that many other task types cannot directly utilize Qs in the current SwiftSage implementation. For example, there are no “objects” in some tasks, requiring customization of the Qs with task-specific prompts. Changes such as rephrasing Q1 and Q2 by replacing “objects” with task-specific elements are necessary. Q3-5, however, are largely task-general as they involve planning subgoals and self-reflection. We have included examples from other tasks, such as interactive coding and web-based tasks, in the updated PDF. For instance, Q1-2 will be about “files” in bash coding tasks, and “tables” for SQL tasks, etc.\n\nRegarding “ablate the Qs to show prompt sensitivity,” we had two interpretations: 1) maintaining the five questions with different wording and descriptions or 2) retaining only a subset of the questions. Due to the time constraints, we here provide some preliminary findings on a mini-size validation set, and we will provide a comprehensive report in the next version of the paper. \n\n1) We have experimented with various phrasings for each question, finding that longer, more specific, and contextual descriptions are generally beneficial. For instance, we added sentences that set the context and invoked commonsense knowledge (e.g., “You are a teacher who instructs elementary students to do science experiments.” and “Please use commonsense knowledge when it is necessary.”) Removing such detailed descriptions results in a 4.3 point decrease in performance.  We also tried to merge the questions. Merging specific questions, like Q1 & Q2, led to a 2~3 point performance drop.\n\n2) In ablation studies, we removed Q1+Q2 and Q5 to evaluate their importance (Q3 & Q4 are required for planning subgoals). Our results show that both are crucial, with a 9.6 point performance decrease for removing Q1+Q2 and a 13.5 point drop for removing Q5.\n\n### Q2: Any alternatives for integrating Swift and Sage? \n\nThank you for raising this point. We believe exploring alternative methods for integrating Swift and Sage is a promising future direction. One possible straightforward solution is to make Swift as a multi-task model that can also output some meta information about the states, such as if it is very uncertain about its predictions or the current action is rather risky such that it needs Sage to help. However, in order to learn such ability in either multi-task Swift or a dedicated separate module, we need supervision. One possible way to get such supervision can be from running the Swift agents in training variations and performing DAgger-style training. We will add more discussion on this topic in the next version of the paper. Thanks again for the great question.\n\n---\n\nPlease let us know if you have any further questions, and we are more than happy to continue the discussion.\n\n\nThank you very much!\n\nBest regards, \\\nAuthors of SwiftSage\n\n"}, {"review_id": "h5nR8hK3im", "reviewer": "Reviewer_Sgwx", "summary": "The paper proposes SwiftSage that predicts a high-level plan by Swift and prompts LLMs to conduct each high-level subgoal by Sage.\nSwift is trained by imitation learning for action planning as fast intuitive thinking and Sage utilizes LLMs for better generalizable planning and exception handling as deliberate thought processes.\nThe proposed method achieves strong performances on the ScienceWorld benchmark over the recent state-of-the-art.", "questions": "- Q1: The Swift module fine-tunes a T5 model with oracle trajectories of training tasks.\nBut there is some work [14,A] that plans a sequence of actions by providing task-relevant examples to LLMs.\nIs there any reason why the authors take imitation learning rather than prompting LLMs?\nAlthough it is fine-tuning and this is quite efficient compared to training from scratch, it does require additional training.\nIn addition, wouldn't it be that planning with LLMs can generalize better than the imitation-learning models as we can leverage commonsense knowledge of LLMs?\n\n- Q2: It seems that SwiftSage can perfectly recognize when and why it fails based on the feedback from the environment, which might enable Sage to address exceptions effectively.\nCan the proposed method address cases where the failure feedbacks are incorrect?\n\n- Q3: I'm not sure the proposed method can ensure that LLMs always provide helpful exception-handling strategies. If LLMs provide some strategies that the agent cannot conduct (e.g., due to the absence of objects), how can the agent address the exception? Note that this is not about the excitability of plans, which is addressed in the \"Grounding stage\" in Sec. 3.3.\n\n- Q4: In Sec. 3.4, it seems missing the motivation of the conditions to switch to Sage. Why does each condition imply when to switch to Sage? It would solidify the switching conditions to include them.\n\nReferences\\\n[A] Liang et al. Code as Policies: Language model programs for embodied control. ICRA, 2023.\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is generally written well and easy to follow.\n- Utilizing LLMs for exception handling seems reasonable.\n- The overall two-staged framework (intuitive planning -> deliberate planning) is intuitive and sounds sensible.\n- The proposed method achieves strong performance over prior work by large margins in most cases in the ScienceWorld benchmark.", "weaknesses": "- Swift adopts imitation learning to train an encoder-decoder (T5) model but I'm not sure why not just prompting LLMs (see Q1).\n- The environment provides if the agent fails at an action, which might not be always available (see Q2).\n- SwiftSage utilizes LLMs for exception handling but it is not clear if strategies for exception handling provided by LLMs are always useful and can be conducted by agents (Q3).\n- The motivation of each condition of switching to Sage is missing (see Q4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Sgwx,\n\nWe greatly appreciate your thoughtful review and are pleased to learn that you find the SwfitSage well-motivated, yielding strong performance over prior work. We value your four insightful questions (Q1-Q4) regarding the weaknesses of our approach and have provided detailed responses to each below. Furthermore, a one-page pdf addressing these points is also available in our general rebuttal.\n\n---\n\n### Q1: Why not use prompting LLMs for the Swift module?\n\n**Swift** is designed to harness the capabilities of smaller language models (commonly understood as those with less than 1 billion parameters) to learn **environment-specific knowledge** from training trajectories while minimizing resource consumption during LLM inference. For these reasons, LLM prompting may not be optimal for Swift, as larger LLMs are typically more powerful, making their smaller counterparts (e.g., 7B models) less effective with prompting for planning. Additionally, in-context examples relevant to specific tasks can only cover a limited range of actions and scenarios, thus resulting in a lower performance.\n\nWe do recognize the benefits of greater generalizability and enhanced **commonsense knowledge** offered by prompting LLMs, which is why the **Sage** module is employed to encapsulate these strengths. In contrast, Swift serves as an expert model, learning environment-specific knowledge from offline data. Thus, imitation learning is deemed a more efficient and conventional method for achieving this objective.\n\n\n### Q2: Can SwiftSage address cases where the failure feedback is incorrect?\n\nInaccurate failure feedback, particularly from humans, indeed poses a practical challenge, although we could not evaluate SwiftSage in such scenarios with ScienceWorld. If the failure feedback is inaccurate or unavailable, we argue that the agent can still transition to Sage for replanning, albeit less efficiently (for example, when no positive reward is received for K consecutive steps; similar to the situation in Q3 below).\n\nWe appreciate your suggestion and we do plan to include an experiment in the next version of the paper by introducing noise to ScienceWorld's feedback to simulate a more realistic environment.\n\n### Q3: What if the LLMs’ subgoals are not feasible?\n\nUpon receiving an action buffer containing grounded actions, the agent first attempts to execute each action sequentially. Any invalid action (e.g., resulting from missing objects) triggers a failure message, which is stored as an observation. Subsequently, these failure messages become part of the input (as observations in action history) for the next prompts.\n\nThis enables the Sage module to infer that previous subgoals were infeasible in the current environment, allowing it to generate a refined plan in response to the 5th prompting question in the planning stage.\n\n### Q4: What is the motivation for each condition in switching from Swift to Sage?\n\nWe will include a clearer explanation of the motivation behind each condition in the revised Sec 3.4. A summary is provided below, and more further details are available in the newly uploaded pdf.\n\nFour triggering conditions (Line 254) are established to invoke the Sage module, each with their respective motivation:\n\n(1). Swift agents may perform actions that yield no rewards. Thus, activating Sage after K non-reward steps enables reflection and replanning. Also, for complex subgoals necessitating multiple steps, Sage can confirm progress (and/or intervene) with LLM prompting if necessary.\n\n(2). If predicted actions or those in the buffer are inexecutable (i.e., cannot be parsed by the ScienceWorld engine), it often indicates Swift's low confidence or inability to generalize to current out-of-distribution situations. Thus, we should use the Sage module that is engaged for its superior world knowledge for planning and reasoning.\n\n(3). Since the Swift module is generally weaker than Sage, the latter is activated for critical, potentially irreversible actions, such as providing answers in ScienceWorld tasks, submitting forms in web tasks, or removing files/folders in coding tasks. This criterion can be customized to suit the given environment and human preferences.\n\n(4). Analogous to the 2nd condition, when an unexpected observation surfaces following an executable action, the Sage module is prompted to replan based on these unforeseen factors (e.g., broken or missing required objects). Figure 2 exemplifies such a scenario.\n\n---\n\nWe welcome any further questions and are eager to continue the discussion. If our explanations have adequately addressed your concerns, may we kindly request that you consider raising your rating score for our paper? Thank you in advance!\n\nBest regards, \\\nAuthors of SwiftSage\n"}, {"review_id": "zyQbRltumP", "reviewer": "Reviewer_MDXU", "summary": "The paper suggests an approach to solving interactive reasoning and planning tasks inspired by the dual-system theory of human cognition popularized by Kahneman. The approach, SwiftSage, combines a small seq2seq model as System 1 and a LLM as System 2. A heuristic module mitigates between the two. The authors conduct extensive experimentation on a large number of textual reasoning and planning tasks. SwiftSage achieves better performance than previous approaches by a large margin in terms of overall performance, efficiency, and exception handling. ", "questions": "Lines 24-26 – are those the only approaches? What about using classical planners as part of the solution (e.g. [1] above). \n\nLine 205 – I thought that the two stages of planning+grounding are both part of the Sage module. Here you refer to them as SwiftSage. \n \nHeuristic for switching between Swift and Sage. – what are the critical decisions (other than giving the final answer)? \n\nSec. 4.3 – how much of the improvement is attributed to a balanced training set? Has this been checked on other models?\n\n", "limitations": "Limitations are adequately addressed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper tackles an important and well-studied problem of solving interactive reasoning and planning tasks. The authors suggest a nice combination between small seq2seq LMs and LLMs, building on the strengths of each one on its own. While this idea is fairly simple, I consider it a solid contribution. Evaluation is extensive and convincingly demonstrates the superiority of the approach over previous ones in terms of performance, efficiency ad exception handling. Overall the paper is clear to read and well motivated.", "weaknesses": "There are some related works that should be mentioned and perhaps even included in the evaluation. One is about combining LLMs and classical planners [1]. The other is about a different LLM-based dual-system approach [2]. \n\nStatements about LLMs resembling System 2 in human cognition (lines 50-52) are overblown in my opinion. While LLMs sometimes exhibit a behavior that may seem like deliberate reasoning, some may argue that System 2 in [2] is more akin to a humans’ System 2. I suggest to water down such statements or at least address the controversy. \n\n[1] https://escholarship.org/content/qt3qq6w5kx/qt3qq6w5kx_noSplash_2e490248a98936f442c743ad768b65f4.pdf\n[2] https://openreview.net/forum?id=uyKk_avJ-p4\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer MDXU,\n\nThank you very much for your thoughtful feedback and comments! We are pleased to learn that you find our method to be **solid** and that we address an important problem through **extensive experiments**. We also appreciate your valuable suggestions and questions. In the following paragraphs, we have addressed the two mentioned weaknesses and four questions.\n\n---\n\n### W1: Missing two relevant works.\n\nWe appreciate your bringing these two related papers to our attention. We will certainly include a discussion of these works in the next version of our paper. In brief, [1] and [2] have similar motivations to ours in developing human-like planning skills but differ in evaluation and methodology focus.\n\n### W2: Statements about System 2 thinking.\n\nThank you for the suggestion! We concur that the statement about System 2 requires clearer justification and clarification. Our primary intent was to highlight that SwiftSage is inspired by cognitive theory, and the analogy between System 1/2 and Swift/Sage modules might help readers grasp the concept more easily. We will revise the relevant paragraphs to address this concern appropriately.\n\n### Q1: What about using classical planners as part of the solution?\n\nIndeed, classical planners could be applicable as well. However, they might necessitate more human-crafted design and conversion to specialized formal planning languages, and their performance is usually not on par with the three types of baseline methods we mentioned. In recent literature on agents for interactive tasks such as ALFWorld, Jericho, and ScienceWorld, classical planners are seldom implemented as baselines due to their limited generalizability during inference time.\n\nNonetheless, we will revise our descriptions to ensure our claim is more inclusive and cite some classical planning methods in the related works for reference. We appreciate your question!\n\n### Q2: Line 205. Typo of “SwiftSage”.\n\nYou are correct; this is indeed a typo. The proper term should be “Sage” instead of “SwiftSage”. We will fix this in the next version. We appreciate you identifying the typo.\n\n### Q3: What are the critical actions?\n\nIn the ScienceWorld environment, we refer to all \"focus on\" actions as critical actions. Besides deciding the final answer, these actions also cover intermediate key decisions, such as confirming key objects (e.g., \"focus on tin\" when the task involves boiling tin). This condition can be customized based on the given environment and preferences. For example, critical actions may involve removing folders (\"rm xxx\") in script coding tasks like InterCode. Please refer to the newly uploaded pdf in the general rebuttal.\n\nFurthermore, we believe it holds potential to learn a dedicated module for classifying whether a time step is risky enough to activate Sage for additional planning. Hence, future work may focus on better generalizing the switching between the two modules.\n\n### Q4: How much of the improvement is attributed to a balanced training set?\n\nWe appreciate this insightful question. To address it, we have conducted new experiments for ablation analysis. When using the original, full data version, we find that the performance of Swift declines to 39.5. We also tested BART-large and smaller models, such as T5-base and T5-small, for similar comparisons. In all cases, utilizing task-balanced and action-balanced training trajectories results in better performance, increased training efficiency (i.e., obtaining a better model in less time), and smoother loss curves. T5-small exhibits a less significant difference in validation loss before and after balancing the data, we believe using balanced data. Overall, we think balancing the data in terms of task types and action types is a critical trick for imitation learning.\n\n---\n\nPlease do not hesitate to reach out with further questions, as we welcome continuing the discussion. If you feel that we have addressed your concerns sufficiently, would you kindly consider raising the rating score for our paper? Thank you very much in advance!\n\nBest regards,\\\nAuthors of SwiftSage\n"}]

Author rebuttal:
### General response 

Dear Reviewers and AC,

We greatly appreciate your insightful reviews and are delighted that all of you have acknowledged our paper's positive aspects. We briefly summarize the favorable feedback as follows:

- **Well-motivated, innovative, generalizable method**: Sgwx, MDXU, CSRK, AXAQ;
- **Significant performance improvements**: Sgwx, AXAQ, CSRK, MDXU;
- **Extensive evaluation & informative ablation**:  MDXU, CSRK;
- **Well-written and easy to follow**: Sgwx, MDXU, CSRK;

---

We sincerely thank the reviewers for their constructive suggestions and questions to enhance our paper. We trust that our rebuttal has sufficiently addressed the concerns raised by the reviewers. Please do not hesitate to contact us if you have any further questions, and we will be more than happy to continue the discussion. 

---
Aside from the rebuttal text, we are also providing **a one-page pdf** that includes the following content: 

(1). A pseudocode to better illustrate the SwiftSage framework, which we hope clarifies some details of our framework. 

(2). A set of figures demonstrating SwiftSage's customization for other complex interactive tasks such as coding tasks, web tasks, math problems, and complex QA. We believe this will help showcase SwiftSage's potential generalization and serve as a general recipe for extending SwiftSage to other applications. 

Once again, thank you for your valuable input. If our responses have addressed your concerns, please kindly consider increasing your scores. We sincerely appreciate your consideration.

Best regards, \
Authors of SwiftSage




Meta-review: This work proposes a two-stage method for action planning in complex interactive reasoning tasks: a light-weight model for fast intuitive thinking, and a more complex LLM for subgoal planning and grounding. The proposed method is shown to achieve STOA results on ScienceWorld.

Reviewers recognize the new framework, significant improvements over existing methods as well as the extensive experiments. 

The authors addressed the reviewers concerns quite extensively in the rebuttal. Two of the reviewers raised their scores, and the final ratings are unanimously accept (6, 8, 6, 7). 

Overall this is a solid and timely work to show that complex reasoning tasks can be better solved with a two-stage approach, similar to slow-fast thinking from psychology.

Please add the additional experiments/discussions (especially the ablation experiments and the discussions regarding how the method can be extended to more general settings) in the final version.
Decision: Accept (spotlight)