SwiftSage: A Generative Agent with
Fast and Slow Thinking for Complex Interactive Tasks

Bill Yuchen Lin 1 Yicheng Fu 2 Karina Yang 3 Prithviraj Ammanabrolu 1 Faeze Brahman 1 Shiyu Huang 4
Chandra Bhagavatula 1 Yejin Choi 1 5 Xiang Ren 1 3

Abstract

tial items, such as batteries and light bulbs, build a circuit,perform an experiment, and interpret the results. Tacklingsuch a complex interactive task demands agents to exhibit
long-horizon planning, long-term memorization, subgoal
decomposition, spatial reasoning, exception handling, andcommonsense knowledge capabilities (Wang et al., 2023b).

We introduce S WIFT S AGE, a novel agent framework inspired by the dual-process theory of human cognition, designed to excel in action planning for complex interactive reasoning tasks.
S WIFT S AGE integrates the strengths of behavior cloning and prompting large language models(LLMs) to enhance task completion performance.
The framework comprises two primary modules:
the S WIFT module, representing fast and intuitive thinking, and the S AGE module, emulating
deliberate thought processes. The S WIFT module is a small encoder-decoder LM fine-tuned on
the oracle agent’s action trajectories, while the
S AGE module employs LLMs such as GPT-4 forsubgoal planning and grounding. We develop aheuristic method to harmoniously integrate thetwo modules, resulting in a more efficient androbust problem-solving process. In 30 tasks fromthe ScienceWorld benchmark, S WIFT S AGE significantly outperforms other methods such as SayCan, ReAct, and Reflexion, demonstrating its effectiveness in solving complex real-world tasks.

There are three primary approaches to developing agents
capable of addressing complex interactive reasoning tasks:(1) deep reinforcement learning (RL), (2) behavior cloning(BC) (Torabi et al., 2018) through sequence-to-sequence(seq2seq) learning (Sutskever et al., 2014), and (3) prompting large language models (LLMs) (Brown et al., 2020). In
addition to conventional RL methods such as DRRN (Heet al., 2016), interactive reasoning can be framed as a
seq2seq task, where the input text serves as the current state
description and the output text corresponds to the subsequent action (Chen et al., 2021; Ammanabrolu et al., 2021).
By leveraging numerous gold trajectories generated by oracle agents, it becomes feasible to fine-tune Transformer
models (Vaswani et al., 2017), like T5 (Raffel et al., 2020),
to effectively imitate the behavior of these oracle agents. Recent studies have also demonstrated that generative agentsbased on prompting LLMs, such as GPT-4, can produce
reasonable plans and actions (Lin et al., 2022; Huang et al.,2022; Song et al., 2022).

1. Introduction

Although the aforementioned methods exhibit remarkable
performance in relatively simple tasks, their ability to generalize to more complex and demanding tasks is limited. BothRL-based and seq2seq-based BC approaches effectively acquire knowledge from the environment through large-scaleinteractions and learn general action patterns from oracle
agents. However, they face difficulties in decomposing
tasks into subgoals, maintaining long-term memory, generalizing to unseen tasks, and handling exceptions. In contrast,
instruction-tuned LLMs (Ouyang et al., 2022) demonstrate
the ability to generate reasonable high-level plans for complex tasks and adapt their outputs based on human feedback. Yet, grounding their outputs to executable actionsin the environment remains a challenge. These procedures
also lack the capability to efficiently handle environmentspecific exceptions that prevent agents from adhering to

The advancement of artificial general intelligence is largely
dependent on the development of agents that are proficientin complex interactive reasoning tasks. These agents should
be capable of exhibiting problem-solving abilities akin tohumans within dynamic, open-world environments (Reedet al., 2022; Bubeck et al., 2023). For example, the ScienceWorld benchmark (Wang et al., 2022) features a task wherean agent must determine the electrical conductivity of an unknown object. In a simulated environment, the agent mustnavigate to appropriate rooms, locate and acquire essenWebsite: https://yuchenlin.xyz/swiftsage/. Affiliations: 1 Allen
Institute for AI 2 Tsinghua University 3 University of Southern California 4 4Paradigm Inc. 5 University of Washington. Correspondence to: Bill Yuchen Lin <yuchenl@allenai.org>.

1

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks
For Each Timestep t: ↓Demo: An oracle path for thetask of boiling water.
Task: Your task is to boil tin.
Action 1: go to kitchen> you moved to kitchen
Action 2: look around> In this kitchen, you can see …
…..
Action t-1: pick up metal pot> metal pot in inventory now

LLM(e.g., GPT-4)

K best generations for Action t

For Each Timestep t: ↓For Each Round: ↓Demo: oracle path + manuallyannotated subgoals
Task: Your task is to boil tin.
Action 1: go to kitchen

The full action history of theprevious failed round.

LLM> you moved to kitchen
Action 2: look around> In this kitchen, you can see …

reflection>>For Each Timestep t: ↓Demo: ReAct’s version.

…..
Action t-2: think: now I need to

Task: Your task is to boil tin. +the reflection of previously
failed trials in the last round.
Action history (<= t-1) for thisround. (same as ReAct)place the metal pot on a heater.
> OK.
Action t-1: pick up metal pot> metal pot in inventory now

LLM

LLM

Action t: put metal pot on stove

Action t: put metal pot on stove

Action t: put tin into metal pot

SayCan

ReAct

Reflexion

RerankingOracle paths for training data.
offline imitation learning

Small LM
Only when needed: ↓Demo: w/o manual annotations
Task: Your task is to boil tin.
Action history <= t-1
Prompts for object locating,planning & tracking subgoals,
and online exception handling.

Nextsubgoals

LLM
Grounding
Templates

LLM

Action Buffer: Action t,
Action t+1, Action t+2 , …

SwiftSage

Figure 1. Comparing methods of prompting LLMs to build agents for interactive tasks.

the LLM’s plans. Additionally, previous methods such as
S AY C AN (Ahn et al., 2022), R E ACT (Yao et al., 2022) andR EFLEXION (Shinn et al., 2023) require a new inferencewith LLMs for each time step, making them considerably
costly and inefficient (see Figure 1).

the next individual action. This module simulates the fast,
intuitive thinking characteristic of System 1. The S AGE
module, representing the deliberate thinking of System 2,
utilizes LLMs, such as GPT-4, and is structured around two
prompting stages: planning and grounding. In the planning stage, we prompt LLMs to locate necessary items,
plan and track subgoals, as well as detect and fix potential exceptions and mistakes. In the grounding stage, we
focus on utilizing LLMs to transform the output subgoals
derived from the planning stage into a sequence of actions
by demonstrating potential action templates. Unlike priormethods, where LLMs only generate the next immediate action, our procedures engage in longer-term action planning.
To harmoniously integrate the S WIFT and S AGE modules,
we developed a heuristic algorithm that determines whento (de)activate the S AGE module and how to combine theoutputs effectively with an action buffer mechanism.

Inspired by the dual process theory (Wason & Evans, 1974;
Kahneman, 2011), we propose a novel framework that enables agents to closely emulate how humans solve complex,
open-world tasks. The dual-process theory posits that human cognition is composed of two distinct systems: System
1, characterized by rapid, intuitive, and automatic thinking;and System 2, which entails methodical, analytical, anddeliberate thought processes. System 1 is reminiscent of
seq2seq methods, which learn through imitation of oracle
agents and primarily operate utilizing shallow action patterns. Conversely, System 2 bears resemblance to LLMs
that excel in applying commonsense knowledge, engaging instep-by-step reasoning, devising subgoal strategies, and exercising self-reflection. Thus, our proposed method, S WIFTS AGE, is designed to enable both fast and slow thinking incomplex interactive reasoning tasks. It effectively integrates
the strengths of behavior cloning (representing System 1)
and prompting LLMs (emulating System 2), resulting insignificant enhancements in task completion performanceand efficiency.

In a comprehensive evaluation on 30 task types from the
ScienceWorld benchmark, S WIFT S AGE significantly outperforms other methods, achieving a state-of-the-art averagescore of 84.7. In comparison, S AY C AN scores 33.8, R E ACT obtains 36.4, and R EFLEXION reaches 45.3. Moreover,
S WIFT S AGE is more cost-effective and efficient, requiringmuch fewer tokens per action for LLM inference than previous methods. This considerable performance advantage
highlights the effectiveness and efficiency of the S WIFTS AGE framework in addressing complex interactive tasks.

Specifically, S WIFT S AGE consists of two primary modules:
the S WIFT module and the S AGE module. The S WIFT module is a small encoder-decoder LM, fine-tuned on a T5-large(770m) checkpoint using the searched oracle trajectories oftraining tasks. It encodes short-term memory components,
such as previous actions, observations, visited locations,as well as the current environment state. Then, it decodes

2. Background and Related WorkWe first provide a formal introduction to complex interactive reasoning tasks, with a particular emphasis on the2

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

ScienceWorld benchmark. Subsequently, we present a summary of existing methods that can be adapted for interactive
reasoning, which also serve as baseline methods in our experiments. Lastly, we discuss the dual-process theory, which
serves as a significant inspiration for our work.

update these graphs. These graphs guide policy networks
by constraining the combinations of action templates andobjects. This method has been shown to be effective in otherdomains such as for multimodal embodied agents (Nottingham et al., 2023).

2.1. Complex Interactive Reasoning

Behavior cloning for offline imitation learning. Behavior cloning is an imitation learning method that trains a
seq2seq Transformer offline with action transcripts of similar training tasks generated by oracle agents (Torabi et al.,2018; Ammanabrolu et al., 2021). During training, it uses
the previous action, observation at time step t − 1, and thecurrent observation as input and learns to output the next
action. The Text Decision Transformer (TDT) is a textual
variant of the Decision Transformer (Chen et al., 2021),
which also employs behavior cloning and uses the samedata. The primary innovation of TDT is the introductionof reward-to-go as part of the inputs, enabling the model
to learn predicting actions that maximize future expected
rewards.

We define interactive reasoning as the problems where
agents are tasked with accomplishing a goal within an interactive environment, typically simulated by engines suchas AI2Thor (Kolve et al., 2017) and TextWorld (Côté et al.,2018). Our focus lies on the textual environment of ScienceWorld (Wang et al., 2022) and the complex interactive tasks
it supports. Simple interactive tasks, like those created in
ALFWorld (Shridhar et al., 2021) and TWC (Murugesan
et al., 2021), primarily involve searching for and placing
objects as well as performing basic actions within a single location. Many of these simple tasks have been almost
solved by recent works.
In contrast, tasks in ScienceWorld exhibit greater complexity, characterized by more challenging task planning anda significantly larger action space (encompassing 10 locations, 200 types of objects with varying states, and 25 typesof actions). Furthermore, agents may encounter random,
unforeseen obstacles, such as broken stoves or missing soil,
which hinder the execution of planned actions. As a result,
agents must adapt and re-plan accordingly, for example, by
seeking alternative heat sources or using a shovel on theoutside ground to get soil. These challenges demand thatagents possess skills in long-horizon planning, long-term
memory, subgoal decomposition, exception handling, andcommonsense knowledge—capabilities that are not explicitly required for simple interactive tasks.

2.3. Prompting LLMs for Action Planning.
Language models (LLMs) such as GPT-4 have shownpromise for action planning in interactive tasks (Lin et al.,2022; Huang et al., 2022; Song et al., 2022; Wang et al.,2023a). In this paper, we adapt three prominent methods tocomplex interactive reasoning tasks in ScienceWorld: S AYC AN (Ahn et al., 2022), R E ACT (Yao et al., 2022), andR EFLEXION (Shinn et al., 2023).
S AY C AN (Ahn et al., 2022) is a straightforward agent that
integrates an LLM with a value function of underlying policies regarding grounding affordances (i.e., the feasibilityof an action in the environment). We need to provide thehistory and current environment as textual inputs to LLMs
for generating a ranked list of action candidates. This action
list is then reranked based on a value function.

2.2. Reinforcement Learning & Imitation Learning
DRRN. Interactive tasks can naturally be framed as
partially-observable Markov decision processes (POMDPs),
enabling the application of RL-based methods. Deep Reinforced Relevance Network (DRRN) (He et al., 2016) is astandard baseline method to learn agents within text environment. It aims to learn representations of observations andactions separately and train a policy network to select actions from candidates based on feedback from the simulated
environment. CALM (Yao et al., 2020) is a reranking-based
method that combines DRRN with a causal language model(LM) fine-tuned with oracle transcripts. In essence, thecausal LM captures task-specific and environment-specificknowledge through imitation learning, and the DRRN learns
to rerank the predictions from the LM.

R E ACT (Yao et al., 2022) presents a virtual ‘think’ action,
enabling LLMs to generate subgoals during action planning. This approach requires human annotators to supply
examples of correct subgoals for each task type, employing
few-shot in-context learning to teach LLMs when and how
to ‘think’ in order to plan subsequent subgoals, in addition
to providing complete action trajectories.
R EFLEXION (Shinn et al., 2023), a recent work building on
R E ACT, proposes a multi-round approach enabling LLMs
to use the history of previously failed rounds to refine theirplanning for the next round. This self-reflection mechanism
helps LLMs improve after each failed attempt. However,this may not be practical in real-world applications for manytasks, as actions in failed trials can be irrecoverable.

The KG-A2C (Ammanabrolu & Hausknecht, 2020) method
uses an OpenIE technique (Angeli et al., 2015) to represent
environment states with graph structures and dynamically

All three methods require a new LLM inference at each time3

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

step to predict the next immediate action, resulting in inefficient and costly agents. R E ACT and R EFLEXION require
human annotations of correct subgoals for each unseen task
type. Moreover, it is difficult to generalize R EFLEXION toreal-world situations where trial-and-error approaches canbe infeasible for embodied tasks.

ability for effectively handling real-world scenarios.
Interactions. Given a task variation, an agent is provided
with the task description D and the initial environment state(t = 0). The task description D is a text specifying a highlevel goal, e.g., “Your task is to test if an unknown substance
A is electronically conductive.” At each time step t, theagent generates an action At based on a set of supported
action templates (e.g., pick up X, use X on Y). A0 isalways “look around” for showing initial environment
information. Upon receiving an action from the agent, theenvironment produces feedback in four dimensions:2.4. Dual-Process Theory
The dual-process theory (Wason & Evans, 1974; Kahneman,2011) is a cognitive psychological framework proposing
the existence of a fast and a slow thinking systems in thehuman brain. This influential theory has found widespread
applications across various fields, highlighting the criticalrole of both systems in shaping human cognition (Anthony
et al., 2017; Chen et al., 2019; Ganapini et al., 2021; Miechet al., 2021). By integrating the complementary strengths
of both systems, agents can effectively and efficiently handle diverse challenges in real-world scenarios. Inspired bythis, we aim to construct a generative agent that utilizes asmall seq2seq LM as System 1 for associative reasoning via
behavior cloning while developing System 2 for analyticalreasoning by prompting LLMs.

• Observation Ot provides direct feedback on the action At
regarding its effects on the environment or the information
queried. For example, an At of “use thermometeron the substance in metal pot” may resultin an Ot like “The temperature is 80F.”• Environment Et represents the current room in which
the agent is situated and provides details about all visible
objects. Object visibility is based on container states, e.g.,
objects within a closed fridge are not included in Et until
the agent performs an action like “open fridge.”• Inventory It lists objects picked up by the agent, whichis particularly useful when agents collect items from different locations to complete the task.

3. S WIFT S AGE: A Generative Agent with Fastand Slow Thinking• Score St represents the agent’s cumulative score ranging
from 0 to 100. When a required intermediate state isachieved, the score increases with a positive reward.

3.1. Problem Formulation
Environment and tasks. We focus on complex interactive reasoning tasks situated in virtual textual environments
such as ScienceWorld (Wang et al., 2022). ScienceWorld
provides an optimal setting for developing and evaluating
agents in complex tasks, comprising 30 distinct task types
covering 10 topics in science experiments. It features 10
locations, including an art studio, workshop, kitchen, livingroom, bedroom, bathroom, foundry, greenhouse, outdoor
area, and a connecting hallway. The environment includes
200+ object types with multiple states (e.g., open, activated)and supports 25 action templates, resulting in an intractablesearch space. The simulator can generate numerous variations of each task type, providing a rich training ground.
In each variation, the agent and environment initialization,such as the locations and states of objects, will differ. A
plethora of training variations encompassing all task types
are available for training agents. Additionally, it provides ahandcrafted oracle agent to search for successful transcripts
with minimal actions for offline learning.

3.2. S WIFT: The Module for Intuitive and Associative
Thinking via Imitation Learning
Imitation learning is used to construct an agent that learns
to mimic oracle agents in various training scenarios throughseq2seq learning. Previous methods, such as TDT (Wanget al., 2022), mainly employ one-hop history as input context
and learn to output the subsequent action At (Wang et al.,2022). However, these methods exhibit limitations due totheir restricted context of action history and harmful biases
arising from data imbalance. To address these issues, we
introduce our S WIFT module, depicted in Figure 2.
Representation for longer history. We expand the conventional one-hop BC to multi-hop by incorporating a sliding window of observations and rewards for the K = 10 recent actions. Additionally, we include a special field for visited rooms (without duplication). This approach aims to provide agents with a longer context and prevent unnecessary
room navigation. The input format is as follows: “Task: D;
Time: t − 1; Score: St−1 ; Action history: [At−i(+Rt−i ) → Ot−i ] /* i loops from K to 1*/; Current
room: Et−1 ; Inventory: It−1 ; Visited rooms:∗{E1∗ , . . . , Et−1}”. Here, Rt = St − St−1 represents the

Evaluation is done on a set of testing variations with unseen combinations of required objects and situations, thus
substantially different from the training variations. For example, a training variation may involve boiling water, whilea testing variation could require boiling tin. Therefore, it is
crucial to ensure the agent’s compositional generalization4

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

Task: Your task is to melt ice cream.; Time: 14; Score: 35; Action history: <extra_id_0> Action 5 (+5): open fridge--> You opened fridge. In it, you see an ice cream… […] <extra_id_9> Action 14 (+0): move metal pot to stove -> You move the metal pot to the stove. 
 Current environment: This room is kitchen. You see: a fridge(closed) | a sink | an oven (closed, turned off) | a stove (turned off; on it: a metal pot containing ice cream …).
|[….]| Inventory: an orange, … 
 Visited: workshop, hallway, kitchen 
 What should be the next action?
T5-large w/
Swift: (imitation
learning)

Next action:

Obs. 15: The stove appears
broken and can't be activated.

activate stove

Time toswitchSage: Prompting LLMs (GPT-4) for Planning and Grounding the Next Subgoals

Groundopen oven Action buffer
move metal pot to ovenclose ovenactivate oven

LLM

Env.

Plan

Q1 & Q2: Locate Needed Objects
Q3 & Q4: Plan & Track Subgoals
Q5: Detect & Fix Exceptions

LLM

Task + History (t=1→15) + Env.

Plans: Q1: … Q2: … Q3: …
Q4: 1. Find a working heat source
2. Place the metal pot with ice cream on the heat source.
3. Wait until the ice cream melts.
Q5: You tried to use the broken stove as a heat source. To
fix this, you should try using the oven in the kitchen or …
Demo + Task + Env. + History + Plan + Action Types:- POUR (A, B): pour object A into container B; e.g., POUR(water, pot)
- … Convert next subgoals to a sequence of actions:Figure 2. An example of how S WIFT S AGE works with fast and slow thinking. The S WIFT module is offline trained via imitation
learning with a small LM such as T5-large (770m). When it is necessary, for example, encountering an exception, we switch to the S AGE
module that prompts LLMs (e.g., GPT-4) for planning and grounding the next subgoals, resulting in an action buffer.

reward at t, and Et∗ is the location name at t.

3.3. S AGE: The Module for Deliberate and AnalyticalThinking via Prompting LLMs
While the S WIFT module acquires surface knowledge about
the environment and task types through imitation learning, itlacks two key abilities essential for complex interactive reasoning: 1) generalizable planning and tracking of subgoals,
and 2) robust handling of exceptions. Prior research has
shown that LLMs outperform smaller LMs in these abilities.
They can perform step-by-step reasoning to devise concreteplans for tasks and self-refine their outcomes. However,the performance of prior methods remains unsatisfactory incomplex interactive tasks such as those in ScienceWorld.

Balanced imitation learning. To avoid bias caused bydata imbalance for seq2seq learning, we down-sampled specific types of tasks and actions to achieve a more balanced
final dataset for training. We used the T5-large with 770
million parameter and instruction-following ability (Chunget al., 2022), creating an efficient agent that we named
S WIFT. Our empirical results show that the S WIFT module
performs much better than TDT (11 billion) despite being
15x smaller in size.

We introduce a novel two-stage approach, named S WIFTS AGE. This method initially acquires higher-level recommendations from LLMs during the planning stage, followedby their translation into specific action sequences in the
grounding stage. By decoupling the planning and grounding processes, S WIFT S AGE effectively generates a series ofactions for completing the planned subgoals.

The S WIFT module exhibits greater accuracy during initialtime steps, enabling it to attain higher scores in the earlystages of a complex task. However, it often fails to generalize to unseen situations. The module also has a tendency
to repeat meaningless actions when its learned plans yield
exceptions from the environment (e.g., the broken stovein Figure 2). This is partly due to the nature of imitation
learning, which prioritizes emulating the observable actionsof oracle agents rather than their intrinsic planning abilities.
Besides, since the oracle trajectories contain only the shortest, correct actions, it is thus also challenging for the S WIFT
to learn how to fix mistaken actions.

Planning stage. In this stage, we leverage LLMs to plan
based on the current state. Specifically, we prompt LLMs
with a single prompt that includes a summarized version of
the task description and action history, and asks the following five key questions.
Before posing the five planning-related questions, we con5

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

dense the entire action history (A<t and O<t ), and the current environment information Et−1 . Q1 and Q2 pertain toobjects, as acquiring all necessary objects serves as the foundation for effective task planning. By addressing these questions, we ensure that LLMs develop a comprehensive understanding of the current environment. Q3 prompts LLMs toengage in step-by-step planning by decomposing the task
into a series of subgoals. Q4 acts as a follow-up question, allowing the agent to monitor its progress based on the action
history and determine completed subgoals, subsequently
focusing on the remaining tasks. Lastly, Q5 is employed to
identify and address potential exceptions. These questions
can be further tailored with additional environment-specifichints, thereby enhancing their adaptability.
▶

Q1(locate objects): “To complete the task,
which objects do I need to collect? Please list themand their possible locations one by one.”▶

Q2(track objects): “Are there any objects thathave not been collected yet?”▶

Q3(plan subgoals): “To complete the task mostefficiently, what are the important subgoals to achieve?
Please list the subgoals one by one.”▶

Q4(track progress): “Considering these subgoals, what have I already completed? And which
subgoal should I focus on right now?”▶

ous methods using LLMs over-generate candidates, andthey rely on reranking or filtering based on the action space
to select the next action. However, this is inefficient andinaccurate for complex tasks with vast action spaces. Additionally, these methods generate a single action at a time,
which can be both costly and ineffective for long-horizon
tasks.
To tackle these issues, we first present supported action
types using a formal style accompanied by remarks. For instance, the action type “pour X into Y” is introduced
as “POUR(X, Y): pour object X into container Y; e.g.,pour red paint into wood cup”. We then incorporate the
LLM’s outputs from the planning stage as part of the input for the grounding stage. Furthermore, we provide therecent action history of the past 10 time steps as context.
Finally, we prompt LLMs to concentrate on the next subgoal and convert it into a list of actions (rather than a single
action) to accomplish the next subgoal. Our formatting
instructions enable the straightforward splitting and conversion of output actions from LLMs in the grounding stage
back to their original action representations. We denote
this list of actions generated by LLMs as the action buffer:
B = {Ât , Ât+1 , . . . }.
3.4. Integration of Fast and Slow Thinking
Having described the S WIFT and S AGE modules, we now
address the question of how to merge both modules and effectively integrate fast and slow thinking within the S WIFTS AGE agent. We establish a heuristic algorithm to control
the activation and deactivation of the two modules.

Q5(handle exceptions): “Have I made any
mistakes that might prevent me from efficiently completing the next subgoal? If any , how should I fix
them?”

To improve the structure of the LLMs’ outputs and facilitate
parsing, we incorporate additional instructions in the prompt.
By utilizing a single input to obtain answers to all five questions in one output, rather than engaging in multiple rounds
of interactive prompting, our approach is more efficient andcost-effective than the iterative prompting methods.

Initially, we employ the S WIFT module due to its superior
intuitive reasoning capabilities, which facilitate accurate associations between the task description and the environment
during the first few actions. We switch from S WIFT mode
to S AGE when any of the following conditions are met:

Q4 and Q5 are of primary importance, while Q1-Q3 serve
as auxiliary guidance for the LLMs. If the action history
indicates a mistaken action or an unachievable previous
subgoal, the response to Q5 refines the answer to Q4 throughself-reflection on the fly. This approach differs from theR EFLEXION agent, which only prompts reflective questions
at the end of a failed trial, allowing agents to improve theirplanning in subsequent attempts. In contrast, our method
detects exceptions and errors each time the agent plans forthe next subgoals, enabling earlier correction of the agent’s
behavior.

1)PThere are five consecutive time steps with zero rewardt−1( i=t−5 Ri = 0).
2) The S WIFT’s prediction for the next action (A′t ) isinvalid in the current environment.
3) A′t can result in a critical decision, such as giving thefinal answer for the experiment result.
4) The observation of A′t suggests that an exception is
encountered.
Upon activating the S AGE module, we execute the twostage prompting process and generate an action buffer. We
attempt to execute each predicted action and revert to the
S WIFT module when the buffer is empty. This enables aseamless integration of both modules, providing an efficientand robust reasoning process for the S WIFT S AGE agent.

Grounding stage. While the answers to Q1-Q5 provide
valuable guidance for agents, they are not directly executable. Converting plans into valid actions that can be
accepted by the environment remains a challenge. Previ6

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

4. Evaluation
4.1. Evaluation Setup

achieves an overall performance of 49.22, which we attributeto its balanced training data and the use of a sliding window
for longer action histories.

To evaluate the effectiveness of S WIFT S AGE and other baseline methods in complex interactive reasoning tasks, we
use the ScienceWorld benchmark. In Section 2.1 and Section 3.1, we introduce the benchmark and problem formulation. Each task type is categorized as ‘short’ (S), ‘medium’(M), or ‘long’ (L) based on the average length of the oracle truth trajectories. However, the length of the task does
not necessarily indicate its level of difficulty as some tasks
may require additional commonsense knowledge. Further
evaluation details are provided in the appendix.

R E ACT demonstrates a noticeable improvement over S AYC AN for short and medium tasks, owing to its subgoal annotations for in-context learning and the inclusion of ‘think’actions. R EFLEXION surpasses R E ACT in shorter tasks;
however, comparing R EFLEXION with other agents is notentirely fair. R EFLEXION can run up to four rounds, while
the others are limited to one round. This discrepancy isparticularly unfair for tasks involving multiple-choice scenarios. Nevertheless, we include R EFLEXION’s results toanalyze the potential of such methods.

4.2. Baseline Agents

Exception handling. Consider the example in Figure 2,
where the stove is broken, presenting an exception. Agentslike DRRN and TDT often resort to repeating meaninglessaction sequences (e.g., continuously attempting to activate
the stove or moving between rooms aimlessly). Although
the S WIFT module, when used independently, improves
upon this due to its larger context window from imitation
learning, it still struggles to address exceptions robustly. ReAct and Reflexion occasionally utilize the ‘think’ action orreflections to redirect agents towards alternative solutions,but the generated actions rarely achieve the new subgoals
if they are not grounded. In contrast, the plan-and-groundprompts in our S AGE module handle exceptions more effectively.

In addition to the baseline methods evaluated in the ScienceWorld paper, such as DRRN, CALM, KG-A2C, and
TDT, we incorporate three LLM-based prompting techniques: S AY C AN, R E ACT, and R EFLEXION, as detailedin Section 2.3 and Figure 1. This subsection presents theimplementation details for adapting these methods to build
ScienceWorld agents.
S AY C AN necessitates a value function from the environment for reranking purposes. We employ SentenceBERT (Reimers & Gurevych, 2019) to rank all valid actions(generated by ScienceWorld’s APIs) based on their similarity to the top 5 generations for At from S AY C AN. We
implemented R E ACT and R EFLEXION in a similar manner.
Adhering to their released code, we utilized the best single
generation and determined the valid action with the minimal edit distance, if required. Both R E ACT and R EFLEX ION necessitate subgoal annotations for teaching LLMs toplan with virtual ‘think’ actions. We annotated such truth
subgoals by translating ScienceWorld’s APIs into natural
language, which was also employed by the oracle agents.
For all agents, we incorporated the complete trajectories
of one or two training variations from the same task typefor in-context learning. Our primary experiments were conducted using OpenAI’s GPT-4; however, other LLMs canbe readily substituted as required.

Cost-effectiveness. Despite S AGE invoking LLMs APIs
twice for inference, its overall cost remains lower, as theresult is a sequence of actions typically containing about
5 actions. In comparison, S AY C AN and R E ACT require
1,855.84 and 1,971.03 tokens per action (tpa) respectively,
while R EFLEXION necessitates 2,983.46 tpa. S WIFT S AGE,
on the other hand, only uses 757.07 tpa. Given its superior
performance, S WIFT S AGE proves more cost-effective thanother LLM-based methods. This efficiency is primarily
attributed to invoking LLMs only when needed (courtesy of
our strong S WIFT module) and the action buffer mechanism.
Efficiency. To thoroughly examine the efficiency of agentsacross all task types, we use Figure 3 to visualize the average trajectories of the first three testing variations for eachtask involving S WIFT S AGE, R E ACT, and the oracle agent.
We arrange the tasks based on their average lengths of oracle trajectories (*Len in Table 1). We observe that oracle
trajectories consistently achieve perfect scores, yet S WIFTS AGE can reach similar scores more efficiently. This isparticularly evident in longer tasks (the bottom two rows),
although S WIFT S AGE does not achieve a perfect score fora few tasks (e.g., 9-2 and 1-3). Interestingly, we find thatR E ACT performs competitively in shorter tasks (e.g., 4-2

4.3. Results and analysis.
Main Results Table 1 compares the performance of various agents across 30 types of tasks. Detailed descriptionsof each task type can be found in the ScienceWorld paper (Wang et al., 2022) and our appendix. It is evident that
LLM-based methods outperform conventional agents dueto their superior generalization ability, albeit at a higher
deployment cost. The behavior cloning model TDT (Wanget al., 2022; Chen et al., 2021) (11b) performs on par with
DRRN (He et al., 2016), but with greater efficiency in learning and inference. In contrast, our S WIFT-only agent (770m)
7

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

Task Type

*Len

DRRN

KGA2C

CALM

TDT

SayCan

ReAct

Reflexion

SwiftSage

1-1 (L)
1-2 (L)
1-3 (L)
1-4 (L)
2-1 (M)
2-2 (M)
2-3 (L)
3-1 (S)
3-2 (M)
3-3 (M)
3-4 (M)
4-1 (S)
4-2 (S)
4-3 (S)
4-4 (S)
5-1 (L)
5-2 (L)
6-1 (M)
6-2 (S)
6-3 (M)
7-1 (S)
7-2 (S)
7-3 (S)
8-1 (M)
8-2 (S)
9-1 (L)
9-2 (L)
9-3 (L)10-1 (L)10-2 (L)

107.7
78.6
88.9
75.2
21.4
35.2
65.0
13.6
20.8
25.6
29.0
14.6
8.8
12.6
14.6
69.5
79.6
33.6
15.1
23.0
7.0
7.0
8.0
40.0
16.3
97.0
84.9
123.1
130.1
132.1

3.52
3.52
0.0
0.0
6.56
5.50
6.0
12.0
9.0
9.05
9.52
15.0
45.0
21.67
19.17
8.0
14.29
15.77
26.67
10.37
50.0
50.0
33.33
21.0
8.0
10.0
10.0
10.0
16.80
17.0

0.0
0.0
4.0
0.0
6.0
11.0
4.0
7.0
4.0
4.0
4.0
18.0
44.0
16.0
15.0
6.0
11.0
17.0
19.0
4.0
43.0
32.0
23.0
5.0
10.0
4.0
4.0
4.0
11.0
11.0

0.0
0.0
0.0
0.0
1.0
1.0
1.0
5.0
7.0
2.0
2.0
10.0
54.0
10.0
8.0
2.0
4.0
3.0
6.0
3.0
6.0
10.0
4.0
4.0
0.0
0.0
3.0
2.0
2.0
2.0

0.71
0.44
3.88
0.55
6.16
6.43
19.87
40.55
14.26
10.16
21.65
41.93
55.76
27.82
47.15
6.89
11.86
15.10
15.70
5.25
30.0
8.43
8.34
3.86
8.0
2.53
14.66
9.12
1.51
1.29

33.06
10.39
3.88
0.37
26.37
8.03
17.41
52.14
22.50
99.56
47.76
22.87
58.18
20.87
31.43
9.92
13.93
47.81
39.26
19.72
80.0
67.50
50.0
20.91
16.0
21.94
32.26
13.67
67.53
59.45

3.52
13.70
7.78
9.88
7.19
6.10
22.37
56.0
54.33
76.19
88.81
26.67
80.0
53.33
27.50
9.06
18.57
51.04
58.89
40.74
60.0
67.50
50.0
27.67
8.0
40.50
44.0
41.0
25.70
16.80

4.22
10.61
7.78
0.92
5.92
28.59
22.37
100.0
17.45
72.54
70.22
64.93
87.27
16.42
100.0
7.33
13.0
70.35
70.67
15.77
100.0
84.37
83.0
2.58
8.0
50.63
100.0
70.62
50.90
23.69

97.04
87.04
72.78
100.0
99.17
88.17
95.73
88.67
55.33
71.90
77.86
100.0
100.0
91.67
100.0
74.59
93.93
49.40
100.0
91.48
95.0
85.0
93.33
89.0
68.50
75.0
70.0
60.0
92.30
77.60

Short
Medium
Long

11.76
28.58
94.30

28.08
10.85
8.26

22.70
6.88
4.92

11.30
2.88
1.33

28.37
10.36
6.11

43.83
36.58
23.65

48.79
44.01
21.07

71.47
35.43
30.17

92.22
77.79
83.0

Overall

49.26

15.56

11.37

5.07

14.66

33.82

36.43

45.34

84.68

Table 1. Results on the ScienceWorld benchmark. *Len is the average length of the oracle agent’s trajectories. We report performanceon three groups of *Len (short, medium, long). The last four methods use GPT-4 as the base LLM for prompting.

and 3-4), but most trajectories plateau at an intermediate
score and fail to reach 100.

thinking, and S AGE, dedicated to slow thinking. The S WIFT
module is a smaller LM that mimics oracle agents’ behavior,
while the S AGE module focuses on prompting LLMs for
subgoal planning and action sequence grounding. Throughextensive experiments on 30 distinct tasks within the ScienceWorld benchmark, S WIFT S AGE outperforms baseline
agents, achieving state-of-the-art performance, increased
efficiency, and reduced cost.

More analysis. Due to page limit, we have to provide
further details and analysis in the appendix, including moredetailed analysis on cost-effectiveness and efficiency, additional case studies and abalation studies, sensitivity to LLM
choices, and an the evaluation of the S WIFT-only agent.

Implications. The success of S WIFT S AGE highlights thepotential for collaborative frameworks combining smallerLMs and LLMs in complex reasoning tasks. Smaller
LMs can be trained more easily to recognize task-specificand environment-specific patterns, fostering effective indistribution generalization. On the other hand, LLMs

5. Conclusion
Contributions. We present S WIFT S AGE, a novel generative agent for complex interactive reasoning tasks, inspiredby the dual-process theory of human cognition. The framework comprises two modules: S WIFT, responsible for fast8

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

7-1 S

7-2 S

7-3 S

4-2 S

4-3 S

3-1 S

4-1 S

4-4 S

6-2 S

8-2 S

3-2 M

2-1 M

6-3 M

3-3 M

3-4 M

6-1 M

2-2 M

8-1 M

2-3 L

5-1 L

1-4 L

1-2 L

5-2 L

9-2 L

1-3 L

9-1 L

1-1 L

9-3 L

10-1 L10-2 L

Figure 3. Visualizing trajectories of S WIFT S AGE, R E ACT and O RACLE. X: time steps (0 → T ); Y : scores (0 → 100). Each figure
displays the merged trajectories of testing variations by an agent. Task IDs are bottom-right, and the ordering is based on *Len.

References

demonstrate remarkable zero-shot generalization abilities
and deliberate thinking, though grounding their outputs inreal-world environments remains challenging. We positthat dual-process agents, harnessing the strengths of both
approaches, constitute a crucial step towards addressing
complex interactive reasoning tasks and building general AI
agents. We can regard S WIFT S AGE as a method of utilizing
LLMs as controllers or planners for decomposing complex
tasks and leveraging APIs/tools (Lu et al., 2023; Ge et al.,2023; Shen et al., 2023; Schick et al., 2023).

Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O.,David, B., Finn, C., Gopalakrishnan, K., Hausman, K.,
Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan,
A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi,
N. J., Julian, R. C., Kalashnikov, D., Kuang, Y., Lee,
K.-H., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor, P.,
Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D. M.,
Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke,
V., Xia, F., Xiao, T., Xu, P., Xu, S., and Yan, M. Do
as i can, not as i say: Grounding language in robotic
affordances. In Conference on Robot Learning, 2022.
Ammanabrolu, P. and Hausknecht, M. J. Graph constrained
reinforcement learning for natural language action spaces.
In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,
2020. OpenReview.net, 2020.

Limitations. Our work has been evaluated solely within atextual simulator, ScienceWorld, which supports a limited
set of actions and tasks compared to real-world situations.
Also, we did not implement any safeguards to prevent agentsfrom engaging in potentially hazardous actions that could
occur in the real world, such as picking up substances from a
blast furnace. We argue that one important future direction isto develop a true open-ended environment, allowing agentsto interact with a much wider variety of actions and objects
to better emulate real-world scenarios. Besides, the use of
LLMs in S AGE may present scalability challenges, as LLMs
require significant computational resources and may not be
feasible in some settings. Future research should explore thegeneralizability of S WIFT S AGE to other domains and thepotential for more lightweight approaches to slow thinking.

Ammanabrolu, P., Urbanek, J., Li, M., Szlam, A., Rocktäschel, T., and Weston, J. How to motivate your dragon:
Teaching goal-driven agents to speak and act in fantasy
worlds. In Proceedings of the 2021 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.
807–833, Online, 2021. Association for Computational
Linguistics. doi: 10.18653/v1/2021.naacl-main.64.
Angeli, G., Johnson Premkumar, M. J., and Manning, C. D.
Leveraging linguistic structure for open domain infor9

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

mation extraction. In Proceedings of the 53rd Annual
Meeting of the Association for Computational Linguistics
and the 7th International Joint Conference on Natural
Language Processing (Volume 1: Long Papers), pp. 344–
354, Beijing, China, 2015. Association for Computational
Linguistics. doi: 10.3115/v1/P15-1034.

D., Le, Q. V., and Wei, J. Scaling instruction-finetuned
language models. ArXiv preprint, abs/2210.11416, 2022.
Côté, M.-A., Kádár, Á., Yuan, X., Kybartas, B. A., Barnes,
T., Fine, E., Moore, J., Hausknecht, M. J., Asri, L. E.,
Adada, M., Tay, W., and Trischler, A. Textworld: A learning environment for text-based games. In CGW@IJCAI,2018.

Anthony, T., Tian, Z., and Barber, D. Thinking fast andslow with deep learning and tree search. In Guyon, I.,
von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R.,
Vishwanathan, S. V. N., and Garnett, R. (eds.), Advancesin Neural Information Processing Systems 30: Annual
Conference on Neural Information Processing Systems
2017, December 4-9, 2017, Long Beach, CA, USA, pp.
5360–5370, 2017.

Ganapini, M. B., Campbell, M., Fabiano, F., Horesh, L.,
Lenchner, J., Loreggia, A., Mattei, N., Rossi, F., Srivastava, B., and Venable, K. B. Thinking fast and slow in ai:
the role of metacognition. In International Conference
on Machine Learning, Optimization, and Data Science,
2021.
Ge, Y., Hua, W., Ji, J., Tan, J., Xu, S., and Zhang, Y. Openagi: When llm meets domain experts. arXiv, 2023.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.,
Gray, S., Chess, B., Clark, J., Berner, C., McCandlish,
S., Radford, A., Sutskever, I., and Amodei, D. Language
models are few-shot learners. In Larochelle, H., Ranzato,
M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advancesin Neural Information Processing Systems 33: Annual
Conference on Neural Information Processing Systems
2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.

He, J., Chen, J., He, X., Gao, J., Li, L., Deng, L., and Ostendorf, M. Deep reinforcement learning with a natural
language action space. In Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics(Volume 1: Long Papers), pp. 1621–1630, Berlin, Germany, 2016. Association for Computational Linguistics.
doi: 10.18653/v1/P16-1153.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Chaudhuri, K.,
Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato,
S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland,
USA, volume 162 of Proceedings of Machine Learning
Research, pp. 9118–9147. PMLR, 2022.

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J. A.,
Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y.-F., Lundberg, S. M., Nori, H., Palangi, H., Ribeiro, M. T., andZhang, Y. Sparks of artificial general intelligence: Earlyexperiments with gpt-4. ArXiv preprint, abs/2303.12712,
2023.

Kahneman, D. Thinking, Fast and Slow. 2011.
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L.,
Herrasti, A., Gordon, D., Zhu, Y., Gupta, A., and Farhadi,
A. AI2-THOR: An Interactive 3D Environment for Visual
AI. arXiv, 2017.

Chen, D., Bai, Y., Zhao, W., Ament, S., Gregoire, J., andGomes, C. P. Deep reasoning networks: Thinking fastand slow. ArXiv preprint, abs/1906.00855, 2019.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A.,
Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequencemodeling. In Ranzato, M., Beygelzimer, A., Dauphin,
Y. N., Liang, P., and Vaughan, J. W. (eds.), Advancesin Neural Information Processing Systems 34: Annual
Conference on Neural Information Processing Systems
2021, NeurIPS 2021, December 6-14, 2021, virtual, pp.
15084–15097, 2021.

Lin, B. Y., Huang, C., Liu, Q., Gu, W., Sommerer, S., andRen, X. On grounded planning for embodied tasks withlanguage models. ArXiv preprint, abs/2209.00465, 2022.
Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W.,
Wu, Y. N., Zhu, S.-C., and Gao, J. Chameleon: Plug-andplay compositional reasoning with large language models.
ArXiv preprint, abs/2304.09842, 2023.
Miech, A., Alayrac, J., Laptev, I., Sivic, J., and Zisserman, A. Thinking fast and slow: Efficient text-to-visual
retrieval with transformers. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 9826–9836. Computer Vision
Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.
2021.00970.

Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y.,
Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S.,
Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X.,
Chowdhery, A., Valter, D., Narang, S., Mishra, G., Yu,
A. W., Zhao, V., Huang, Y., Dai, A. M., Yu, H., Petrov, S.,
hsin Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou,
10

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

Murugesan, K., Atzeni, M., Kapanipathi, P., Shukla, P.,
Kumaravel, S., Tesauro, G., Talamadupula, K., Sachan,M., and Campbell, M. Text-based RL agents with commonsense knowledge: New challenges, environments andbaselines. In Thirty-Fifth AAAI Conference on Artificial
Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021,
The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February
2-9, 2021, pp. 9018–9027. AAAI Press, 2021.

Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self-reflection.
ArXiv preprint, abs/2303.11366, 2023.
Shridhar, M., Yuan, X., Côté, M., Bisk, Y., Trischler, A., andHausknecht, M. J. Alfworld: Aligning text and embodied
environments for interactive learning. In 9th International
Conference on Learning Representations, ICLR 2021,
Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
2021.
Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao,
W.-L., and Su, Y. Llm-planner: Few-shot grounded planning for embodied agents with large language models.
ArXiv preprint, abs/2212.04088, 2022.

Nottingham, K., Ammanabrolu, P., Suhr, A., Choi, Y.,
Hajishirzi, H., Singh, S., and Fox, R. Do embodied
agents dream of pixelated sheep: Embodied decision
making using language guided world modelling. volume
abs/2301.12050, 2023.

Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Ghahramani,Z., Welling, M., Cortes, C., Lawrence, N. D., and Weinberger, K. Q. (eds.), Advances in Neural Information
Processing Systems 27: Annual Conference on Neural
Information Processing Systems 2014, December 8-13
2014, Montreal, Quebec, Canada, pp. 3104–3112, 2014.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright,
C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K.,
Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E.,
Simens, M., Askell, A., Welinder, P., Christiano, P. F.,
Leike, J., and Lowe, R. J. Training language models tofollow instructions with human feedback. ArXiv preprint,
abs/2203.02155, 2022.

Torabi, F., Warnell, G., and Stone, P. Behavioral cloning
from observation. In Lang, J. (ed.), Proceedings of theTwenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp. 4950–4957. ijcai.org, 2018. doi:
10.24963/ijcai.2018/687.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S.,
Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring thelimits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention
is all you need. In Guyon, I., von Luxburg, U., Bengio,S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N.,
and Garnett, R. (eds.), Advances in Neural Information
Processing Systems 30: Annual Conference on Neural
Information Processing Systems 2017, December 4-9,
2017, Long Beach, CA, USA, pp. 5998–6008, 2017.

Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G.,
Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y.,
Kay, J., Springenberg, J. T., Eccles, T., Bruce, J., Razavi,A., Edwards, A. D., Heess, N. M. O., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. A
generalist agent. ArXiv preprint, abs/2205.06175, 2022.
Reimers, N. and Gurevych, I. Sentence-BERT: Sentence
embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP), pp. 3982–3992, Hong Kong, China,
2019. Association for Computational Linguistics. doi:
10.18653/v1/D19-1410.

Wang, R., Jansen, P., Côté, M.-A., and Ammanabrolu, P.
ScienceWorld: Is your agent smarter than a 5th grader? In
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11279–11298,
Abu Dhabi, United Arab Emirates, 2022. Association for
Computational Linguistics.
Wang, Z., Cai, S., Liu, A., Ma, X., and Liang, Y. Describe,
explain, plan and select: Interactive planning with large
language models enables open-world multi-task agents.
ArXiv preprint, abs/2302.01560, 2023a.

Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli,
M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use
tools. ArXiv preprint, abs/2302.04761, 2023.

Wang, Z., Zhang, G., Yang, K., Shi, N., Zhou, W., Hao, S.,
Xiong, G., Li, Y., Sim, M. Y., Chen, X., Zhu, Q., Yang,Z., Nik, A., Liu, Q., Lin, C., Wang, S., Liu, R., Chen, W.,
Xu, K., Liu, D., Guo, Y., and Fu, J. Interactive natural
language processing. ArXiv, 2023b.

Shen, Y., Song, K., Tan, X., Li, D. S., Lu, W., and Zhuang,
Y. T. Hugginggpt: Solving ai tasks with chatgpt and its
friends in huggingface. ArXiv preprint, abs/2303.17580,
2023.
11

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

Wason, P. C. and Evans, J. S. B. Dual processes in reasoning? Cognition, 3(2):141–154, 1974.
Yao, S., Rao, R., Hausknecht, M., and Narasimhan, K.
Keep CALM and explore: Language models for action generation in text-based games. In Proceedings ofthe 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pp. 8736–8754, Online,
2020. Association for Computational Linguistics. doi:
10.18653/v1/2020.emnlp-main.704.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan,K., and Cao, Y. React: Synergizing reasoning and acting
in language models. ArXiv preprint, abs/2210.03629,
2022.

12

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

C.2. Prompting in S AGE

Appendix

In Section 3.3, we provided an overview of the two-stage
prompting framework: planning and grounding. In thissection, we delve into further details of each stage.

A. Limitations
Our work has been evaluated solely within a textual simulator, ScienceWorld, which supports a limited set of actions
and tasks compared to real-world situations. Also, we did
not implement any safeguards to prevent agents from engaging in potentially hazardous actions that could occur inthe real world, such as picking up substances from a blast
furnace. We argue that one important future direction is todevelop a true open-ended environment, allowing agents tointeract with a much wider variety of actions and objects
to better emulate real-world scenarios. Besides, the use of
LLMs in S AGE may present scalability challenges, as LLMs
require significant computational resources and may not be
feasible in some settings. Future research should explore thegeneralizability of S WIFT S AGE to other domains and thepotential for more lightweight approaches to slow thinking.

Memory augmentation. Since the agent can only perceive objects in its current environment location, objects
from previously visited locations are not displayed unless aprior ‘look around’ action has been executed. To augment
memory for LLMs during planning and grounding, we alsopresent the objects observed in previously visited locations.
Additionally, we include the agent’s location during each action in the action history, e.g., “pick up metal pot [location:
kitchen],” to facilitate spatial reasoning for LLMs.
Connecting the two stages. We conveniently reuse the
LLM output from the first stage (i.e., answers to Q1-Q5)
as part of the input for the second stage. Our experiments
involve using answers to all questions in the grounding stage.
However, one can opt to use only answers to Q4 and Q5
to reduce computational costs. Our small-scale ablation
study indicates that incorporating answers to Q1-Q3 in the
grounding stage proves beneficial, yielding a gain of about
2 points for short tasks on average.

B. Dataset Statistics
Table 2 presents the details of all 30 types of tasks in the ScienceWorld benchmark. To improve the training of S WIFT,
we down-sampled Task 9-x, 10-x, and 3-3 from the originalfull dataset, as their large sizes resulted in a significant data
imbalance. Additionally, we down-sampled less informative
actions, such as ‘close door to kitchen,’ to produce a more
effective dataset for imitation learning.

Grounding with action templates. We previously introduced an action template, ‘POUR(object A, object B)’, inFigure 2. Here, we present several additional templates tofurther illustrate the concept:

Evaluation. To save time while evaluating the numerous
tasks and agents, we only used the first 10 variations fortasks with more than 10 test variations. This resulted in a total of 270 variations for fair and cost-effective comparisonsamong all agents. Some agents may receive a negative scorefrom the engine and be unable to proceed any further due
to their final action violating task requirements and being
irrecoverable. In such cases, we used their last non-negativescores for evaluation.

TELEPORT(room) : directly go to a room such as
TELEPORT(kitchen)
PICK(object) : pick up an object and put it into
your inventory
OPEN(object) : open an object to search or put
things in it, e.g., OPEN(freezer).
ACTIVATE(object) : activate / turn on an object
such as sink or stove, so that you can use it.

C. Implementation Details

DEACTIVATE(object) : deactivate / turn off theobjectC.1. Training Details of S WIFTEXAMINE(object) : look at an object carefully. For
example, EXAMINE(light bulb).

We utilized flan-t5-large (770m) as the base modeland fine-tuned it using the seq2seq action-prediction data(62k) as previously described. A learning rate of 1e-4 andbatch size of 128 were employed for training 500 steps,
selected based on dev loss. Although we experimented withlarger sizes of flan-t5 models, we observed only marginal improvements at a much higher training cost. We believe thisis because the language used to describe the environment
and actions covers a small vocabulary, and the language
complexity does not warrant the use of more parameters.

MOVE(object, place) : move/place the object toa placeIt should be noted that despite explicitly instructing the LLM
to only utilize permitted action types, it may occasionallygenerate actions of disallowed types that cannot be parsed.
These invalid actions will be disregarded in the action buffer,and if necessary, the system will revert to the S WIFT mode.
13

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks
Task Type

Topic

Name

*Lens

#Vars: Train

Dev

Test

# Actions

1-1
1-2
1-3
1-4

MatterMatterMatterMatterChanges of State (Boiling)
Changes of State (Melting)
Changes of State (Freezing)
Changes of State (Any)

107.7
78.6
88.9
75.2

14
14
14
14

7
7
7
7

9
9
9
9

694
427
469
344

2-1
2-2
2-3

Measurement
Measurement
Measurement

Use Thermometer
Measuring Boiling Point (known)
Measuring Boiling Point (unknown)

21.4
35.2
65

270
218
150

10
10
10

10
10
10

4278
6511
9768

3-1
3-2
3-3
3-4

Electricity
Electricity
Electricity
Electricity

Create a circuit
Renewable vs Non-renewable Energy
Test Conductivity (known)
Test Conductivity (unknown)

13.6
20.8
25.6
29

10
10
48
300

5
5
10
10

5
5
10
10

94
169
1341
6974

4-1
4-2
4-3
4-4

Classification
Classification
Classification
Classification

Find a living thing
Find a non-living thing
Find a plant
Find an animal

14.6
8.8
12.6
14.6

150
150
150
150

10
10
10
10

10
10
10
10

1606
756
1458
1606

5-1
5-2

Biology
Biology

Grow a plant
Grow a fruit

69.5
79.6

62
62

10
10

10
10

3675
4283

6-1
6-2
6-3

Chemistry
Chemistry
Chemistry

Mixing (generic)Mixing paints (secondary colours)
Mixing paints (tertiary colours)

33.6
15.1
23

16
18
18

8
9
9

8
9
9

347
224
350

7-1
7-2
7-3

Biology
Biology
Biology

Identify longest-lived animal
Identify shortest-lived animal
Identify longest-then-shortest-lived animal

7
7
8

62
62
62

10
10
10

10
10
10

298
298
360

8-1
8-2

Biology
Biology

Identify life stages (plant)
Identify life stages (animal)

40
16.3

6
4

3
2

5
4

165
31

9-1
9-2
9-3

ForcesForcesForcesInclined Planes (determine angle)
Friction (known surfaces)
Friction (unknown surfaces)

97
84.9
123.1

24
26
23

10
10
10

10
9
10

2733
3644
3284

10-1
10-2

Biology
Biology

Mendelian Genetics (known plants)
Mendelian Genetics (unknown plants)

130.1
132.1

26
24

10
10

10
10

3043
2853

Short (0 < *Len ≤ 20)
Medium (20 < *Len ≤ 50)
Long (*Len > 50)

11.76
28.58
94.30

81.80
110.75
37.75

8.6
8.13
9.00

8.80
8.38
9.58

673.102516.882934.75

Overall (avg)
Overall (sum)

49.26
N/A

71.90
2,157

8.63
259

9
270

2069.43
62,083

Table 2. The statistics of ScienceWorld benchmark. *Len is the average length of the oracle agent’s trajectories. We show the numberof our down-sampled variations in each split. The last column is the number of data points forr action-prediction seq2seq task in training
S WIFT.

C.3. Action Buffer

we attempt to execute each action sequentially. If two consecutive actions are invalid or cause exceptions, we halt andrevert to S WIFT.

In Section 3.4, we presented four conditions for activating
the S AGE module. To detect critical decisions (Condition3), we primarily focus on the ‘focus on’ actions, as manytasks in ScienceWorld necessitate agents to concentrate on
the correct substances and objects in the proper sequence.
A single incorrect ‘focus on’ action can terminate the entire
run. Thus, we restrict the S WIFT module from performing
such actions if S AGE has not yet been activated.

D. Additional Results and Analysis
D.1. Sensitivity to LLMs: GPT-3.5-turbo vs GPT-4
Besides the empirical results in Table 1, we also evaluate
performance using GPT-3.5-turbo instead of GPT-4, whichis considerably larger and more expensive. Other methods
exhibit a significant performance decline, for instance, ReAct’s score drops from 36.43 to 19.76, which is close to
non-LLM methods and even lower than the vanilla method,

For identifying exceptions (Condition 4), we examinephrases like “No known action can match,” “... cannot/doesn’t...,” and so on. When processing an action buffer,

14

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

SayCan. In contrast, S WIFT S AGE maintains a respectableperformance of 62.22, indicating better robustness.
As discussed in Sec. 5 (limitations), we plan to utilize otheropen-source LLMs, such as Alpaca, and investigate distilling the planning ability from closed-source LLMs to opensource and smaller LMs. Nevertheless, a practical challengearises due to the current open-source LLMs having more
restrictive length limits for inputs and outputs.
D.2. Efficiency Analysis
Figure 4 illustrates that most of S WIFT S AGE’s curves are
situated near the top-left corner, indicating that S WIFT S AGE
attains higher scores than oracle agents at a faster rate. Although ReAct is competitive with our method for shortertasks, its trajectories typically plateau at intermediate scoresand do not reach 100. While the O RACLE agent consistently
achieves a perfect score (100.0), its efficiency, particularly
in longer tasks, is often outperformed by S WIFT S AGE.
D.3. Cost-effectiveness
Table 4 presents a comprehensive analysis of the costeffectiveness of LLM-based methods. We examine two
specific metrics: tokens per action (tpa) and scores per action (spa) for SayCan, ReAct, Reflexion, and S WIFT S AGE
across all tasks. Despite S AGE invoking LLM APIs twice
for inference, its overall cost remains lower, as the result
is a sequence of actions typically containing about 5 actions. In contrast, S AY C AN and R E ACT require 1,855.84
and 1,971.03 tokens per action (tpa) respectively, while
R EFLEXION necessitates 2,983.46 tpa. S WIFT S AGE, however, only uses 757.07 tpa. Given its superior performance,
S WIFT S AGE proves to be more cost-effective than other
LLM-based methods. This efficiency primarily stems frominvoking LLMs only when necessary (thanks to our robust
S WIFT module) and the action buffer mechanism.
Interestingly, we observe that S WIFT S AGE has an evenlower tpa for long tasks compared to its tpa in medium
and short tasks. Upon further investigation, we attribute
this finding to longer action buffers and the S WIFT module
being more frequently effective. Additionally, regarding
scores per action (spa), we discover that our S WIFT S AGE is
more cost-effective by utilizing fewer tokens and achieving
higher scores.

15

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

Task Type

Swift-Only

SayCanChatGPTReActChatGPT

ReflexionChatGPT

SwiftSageChatGPT

1-1
1-2
1-3
1-4
2-1
2-2
2-3
3-1
3-2
3-3
3-4
4-1
4-2
4-3
4-4
5-1
5-2
6-1
6-2
6-3
7-1
7-2
7-3
8-1
8-2
9-1
9-2
9-3
10-1
10-2

15.0
24.4
32.2
57.4
9.4
6.7
5.7
70.0
48.3
59.5
69.0
100.0
100.0
94.4
100.0
13.4
44.6
26.2
53.3
11.1
83.3
100.0
77.8
33.0
8.0
73.3
73.3
53.3
17.0
17.0

0.0
0.0
0.0
0.0
1.7
14.1
93.7
19.3
8.7
22.0
36.4
11.7
76.0
11.4
9.5
11.3
75.0
13.5
25.0
58.4
75.0
100.0
31.7
5.6
12.8
38.0
4.2
0.0
1.3
0.3

0.4
8.5
1.1
0.6
4.1
7.2
19.6
16.7
9.7
55.5
36.4
17.5
73.3
20.0
15.8
11.1
18.8
35.0
20.0
16.7
37.5
50.0
31.7
4.2
7.0
28.5
10.0
0.0
24.5
11.7

0.7
0.0
0.0
0.0
2.2
2.5
14.7
20.0
8.6
6.4
30.1
46.5
68.2
19.9
41.0
5.8
47.6
22.4
10.0
40.0
75.0
75.0
28.1
2.8
8.2
100.0
17.5
1.7
1.3
6.0

58.0
58.5
38.5
62.5
47.9
53.3
48.6
72.7
50.3
66.9
78.1
100.0
97.5
58.3
100.0
57.5
50.9
43.2
63.3
27.4
75.0
60.0
68.3
75.6
33.0
54.0
63.3
77.0
76.0
51.1

Short
Medium
Long

78.68
32.90
35.55

37.24
20.06
18.66

28.95
21.09
11.23

39.19
14.37
16.27

72.81
55.34
57.99

Overall

49.22

25.22

19.76

23.40

62.22

Table 3. Additional results on the ScienceWorld benchmark. Different from Table 1, we use gpt-3.5-turbo instead of gpt-4 as
the LLM for evaluating SayCan, ReAct, Relfexion, and our S WIFT S AGE. We also present the results of using S WIFT module only.

16

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

Figure 4. An overview of visualizing trajectories of S WIFT S AGE, R E ACT and O RACLE. X: time steps (0 → T ); Y : scores (0 → 100).
Similar to Figure 3, each curve is a single trajectory by an agent in performing a task variation. A more efficient agent will achieve higherscores in a shorter time, resulting in curves positioned near the top-left corner.

17

SwiftSage: A Generative Agent with Fast & Slow Thinking for Complex Interactive Tasks

Average number of tokens per action (tpa)

Average scores per action (spa)Task Type

SayCan

ReAct

Reflexion

SwiftSage

SayCan

ReAct

Reflexion

SwiftSage

1-1
1-2
1-3
1-4
2-1
2-2
2-3
3-1
3-2
3-3
3-4
4-1
4-2
4-3
4-4
5-1
5-2
6-1
6-2
6-3
7-1
7-2
7-3
8-1
8-2
9-1
9-2
9-3
10-1
10-2

1944.941125.761034.331295.03
1188.46
1862.11939.171713.64
1785.01
1762.131698.85
411.081332.831155.991126.672323.432646.50
1454.652413.991371.50376.50424.53424.731505.393189.80
2066.06
2517.487002.723612.33
3969.621503.60
1339.39
1268.231251.451545.031181.881358.33
1846.91
1754.142441.79
1195.59579.701098.69
1314.74591.152620.662575.111802.622763.66
2860.68495.83478.09564.691155.71741.712642.793031.95
7507.00
4218.44
5401.37

2632.97
3066.703307.30
2439.34
1988.59
1596.03
1753.172677.89
2337.022262.392859.30
1053.57
1250.37
2966.82
1003.18
5091.495864.932344.904342.074551.96813.081180.58
1175.35
2466.59
2886.092652.563606.60
7785.294822.976724.81528.17545.34550.17754.05494.52394.29574.05807.62823.28220.80287.25309.14298.48406.17309.71
168.95536.56
1388.89402.506361.79768.63772.00609.73249.382479.00
307.30314.19366.06466.21218.00

0.10
0.08
0.04
0.00
0.13
0.03
0.73
0.49
0.21
0.53
0.18
1.91
0.20
0.08
0.24
0.02
0.03
0.28
0.18
0.09
5.71
2.11
4.55
0.21
0.12
0.16
0.11
0.04
0.32
0.52

0.05
0.13
0.13
0.09
0.06
0.25
0.93
0.44
0.22
0.18
1.93
1.16
0.69
0.39
1.02
0.02
0.13
0.25
0.14
0.10
3.33
6.14
3.85
0.79
0.47
0.14
0.14
0.18
0.36
0.10

0.00
0.01
0.00
0.00
0.00
0.15
0.12
0.04
0.01
0.10
0.10
0.34
0.47
0.02
0.79
0.00
0.00
0.04
0.02
0.00
0.77
0.36
0.24
0.00
0.02
0.08
0.05
0.04
0.13
0.01

1.49
1.64
0.71
1.69
3.01
2.32
1.71
0.49
0.31
0.84
1.13
4.76
4.76
3.82
4.76
0.22
0.75
0.37
3.33
0.59
11.88
10.63
8.48
2.23
4.03
1.06
0.66
0.65
1.78
1.69

Short
MediumLong

1256.98
1578.512539.781047.52
1742.18
2893.19

1934.902550.854145.68

716.301277.52444.09

1.56
0.21
0.18

1.76
0.47
0.20

0.31
0.05
0.04

5.69
1.35
1.17

Overall

1855.84

1971.032983.46757.07

0.65

0.79

0.13

2.73

Table 4. Cost-effectiveness analysis for LLM-based methods.

18