SlowGuess/NeurIPS_2023_OCR huggingface dataset:
TELEPORT(room) : directly go to a room such as TELEPORT(kitchen)  
PICK(object) : pick up an object and put it into your inventory  
OPEN(object) : open an object to search or put things in it, e.g., OPEN(freezer).  
ACTIVATE(object) : activate / turn on an object such as sink or stove, so that you can use it.  
DEACTIVATE(object) : deactivate / turn off the object  
EXAMINE(object) : look at an object carefully. For example, EXAMINE(light bulb).  
MOVE(object, place) : move/place the object to a place
We then incorporate the LLM's outputs from the planning stage as part of the input for the grounding stage. Furthermore, we provide the recent action history of the past 10 time steps as context. Finally, we prompt LLMs to concentrate on the next subgoal and convert it into a list of actions (rather than a single action) to accomplish the next subgoal. Our formatting instructions enable the straightforward splitting and conversion of output actions from LLMs in the grounding stage back to their original action representations. We denote this list of actions generated by LLMs as the action buffer:  $B = \{\hat{A}_t, \hat{A}_{t+1}, \dots\}$ . One can opt to use only answers to Q4 and Q5 to reduce computational costs. Our small-scale ablation study indicates that incorporating answers to Q1-Q3 in the grounding stage proves beneficial, yielding a gain of about 2 points for short tasks on average.
3.4 Integration of Fast and Slow Thinking
Having described the SWIFT and SAGE modules, we now address the question of how to merge both modules and effectively integrate fast and slow thinking within the SWIFTSAGE agent. We establish a heuristic algorithm to control the activation and deactivation of the two modules.
Initially, we employ the SWIFT module due to its superior intuitive reasoning capabilities, which facilitate accurate associations between the task description and the environment during the first few actions. We will switch from SWIFT mode to SAGE when any of the following conditions are met:
1) Stuck: There are  $\mathrm{K} = 5$  consecutive time steps with zero reward  $(\sum_{i = t - 5}^{t - 1}R_{i} = 0)$ .
2) Invalid: The SwIFT's prediction for the next action  $(A_{t}^{\prime})$  is invalid in the current environment.
3) Critical:  $A_t'$  involves a critical decision, e.g., giving the final answer for the experiment result.
4) Unexpected: The observation of  $A_{t}^{\prime}$  suggests that an exception is encountered.
Upon activating the SAGE module, we execute the two-stage prompting process and generate an action buffer. We attempt to execute each predicted action and revert to the SWIFT module when the buffer is empty. This approach enables a seamless integration of both modules, providing an efficient and robust problem-solving process for the SWIFT SAGE agent. The pseudo code for illustrating the SwiftSage framework is shown in Fig. 4 (Appendix).
To evaluate the effectiveness of SWIFTSAGE and other baseline methods in complex interactive reasoning tasks, we use the ScienceWorld benchmark. In Section 2.1 and Section 3.1, we introduce the benchmark and problem formulation. Each task type is categorized as 'short' (S), 'medium' (M), or 'long' (L) based on the average length of the oracle truth trajectories. However, the length of the task does not necessarily indicate its level of difficulty as some tasks may require additional commonsense knowledge. Further evaluation details are provided in the appendix.
Task Type	*Len	DRRN	KGA2C	CALM	TDT	SayCan	ReAct	Reflexion	SwiftSage
1-1 (L)	107.7	3.52	0.0	0.0	0.71	33.06	3.52	4.22	97.04
1-2 (L)	78.6	3.52	0.0	0.0	0.44	10.39	13.70	10.61	87.04
1-3 (L)	88.9	0.0	4.0	0.0	3.88	3.88	7.78	7.78	72.78
1-4 (L)	75.2	0.0	0.0	0.0	0.55	0.37	9.88	0.92	100.0
2-1 (M)	21.4	6.56	6.0	1.0	6.16	26.37	7.19	5.92	99.17
2-2 (M)	35.2	5.50	11.0	1.0	6.43	8.03	6.10	28.59	88.17
2-3 (L)	65.0	6.0	4.0	1.0	19.87	17.41	22.37	22.37	95.73
3-1 (S)	13.6	12.0	7.0	5.0	40.55	52.14	56.0	100.0	88.67
3-2 (M)	20.8	9.0	4.0	7.0	14.26	22.50	54.33	17.45	55.33
3-3 (M)	25.6	9.05	4.0	2.0	10.16	99.56	76.19	72.54	71.90
3-4 (M)	29.0	9.52	4.0	2.0	21.65	47.76	88.81	70.22	77.86
4-1 (S)	14.6	15.0	18.0	10.0	41.93	22.87	26.67	64.93	100.0
4-2 (S)	8.8	45.0	44.0	54.0	55.76	58.18	80.0	87.27	100.0
4-3 (S)	12.6	21.67	16.0	10.0	27.82	20.87	53.33	16.42	91.67
4-4 (S)	14.6	19.17	15.0	8.0	47.15	31.43	27.50	100.0	100.0
5-1 (L)	69.5	8.0	6.0	2.0	6.89	9.92	9.06	7.33	74.59
5-2 (L)	79.6	14.29	11.0	4.0	11.86	13.93	18.57	13.0	93.93
6-1 (M)	33.6	15.77	17.0	3.0	15.10	47.81	51.04	70.35	49.40
6-2 (S)	15.1	26.67	19.0	6.0	15.70	39.26	58.89	70.67	100.0
6-3 (M)	23.0	10.37	4.0	3.0	5.25	19.72	40.74	15.77	91.48
7-1 (S)	7.0	50.0	43.0	6.0	30.0	80.0	60.0	100.0	95.0
7-2 (S)	7.0	50.0	32.0	10.0	8.43	67.50	67.50	84.37	85.0
7-3 (S)	8.0	33.33	23.0	4.0	8.34	50.0	50.0	83.0	93.33
8-1 (M)	40.0	21.0	5.0	4.0	3.86	20.91	27.67	2.58	89.0
8-2 (S)	16.3	8.0	10.0	0.0	8.0	16.0	8.0	8.0	68.50
9-1 (L)	97.0	10.0	4.0	0.0	2.53	21.94	40.50	50.63	75.0
9-2 (L)	84.9	10.0	4.0	3.0	14.66	32.26	44.0	100.0	70.0
9-3 (L)	123.1	10.0	4.0	2.0	9.12	13.67	41.0	70.62	60.0
10-1 (L)	130.1	16.80	11.0	2.0	1.51	67.53	25.70	50.90	92.30
10-2 (L)	132.1	17.0	11.0	2.0	1.29	59.45	16.80	23.69	77.60
Short	11.76	28.08	22.70	11.30	28.37	43.83	48.79	71.47	92.22
Medium	28.58	10.85	6.88	2.88	10.36	36.58	44.01	35.43	77.79
Long	94.30	8.26	4.92	1.33	6.11	23.65	21.07	30.17	83.0
Overall	49.26	15.56	11.37	5.07	14.66	33.82	36.43	45.34	84.68
Table 1: Results on the ScienceWorld benchmark. *Len is the average length of the oracle agent's trajectories. In addition to overall results, we also report performance on three groups of *Len (short, medium, long). The last four methods use GPT-4 as the base LLM for prompting. We show more details of these tasks and results on other LLMs in the appendix.
4.2 Baseline Agents
In addition to the baseline methods evaluated in the ScienceWorld paper, such as DRRN, CALM, KG-A2C, and TDT, we incorporate three LLM-based prompting techniques: SAYCAN, REACT, and REFLEXION, as detailed in Section 2.3 and Figure 1. This subsection presents the implementation details for adapting these methods to build ScienceWorld agents.
SAYCAN necessitates a value function from the environment for reranking purposes. We employ SentenceBERT [27] to rank all valid actions (generated by ScienceWorld's APIs) based on their similarity to the top 5 generations for  $A_{t}$  from SAYCAN. We implemented REACT and REFLEXION in a similar manner. Adhering to their released code, we utilized the best single generation and determined the valid action with the minimal edit distance, if required. Both REACT and REFLEXION necessitate subgoal annotations for teaching LLMs to plan with virtual 'think' actions. We annotated such truth subgoals by translating ScienceWorld's APIs into natural language, which was also employed by the oracle agents. For all agents, we incorporated the complete trajectories of one or two training variations from the same task type for in-context learning. Our primary experiments were conducted using OpenAI's GPT-4; however, other LLMs can be readily substituted as required.
Figure 3: Visualizing trajectories of SWIFTSAGE, REACT and ORACLE.  $X$ : time steps  $(0 \to T)$ ;  $Y$ : scores  $(0 \to 100)$ . Each figure displays the merged trajectories of testing variations by an agent in each task. Task IDs are shown at the bottom-right, and the ordering is based on *Len in Tab 1.
4.3 Results and analysis.
Main Results Table 1 compares the performance of various agents across 30 types of tasks. Detailed descriptions of each task type can be found in the ScienceWorld paper [36] and our appendix. It is evident that LLM-based methods outperform conventional agents due to their superior generalization ability, albeit at a higher deployment cost. The behavior cloning model TDT [36, 9] (11b) performs on par with DRRN [14], but with greater efficiency in learning and inference. In contrast, our SWIFT-only agent  $(770\mathrm{m})$  achieves an overall performance of 49.22, which we attribute to its balanced training data and the use of a sliding window for longer action histories.
REACT demonstrates a noticeable improvement over SAYCAN for short and medium tasks, owing to its subgoal annotations for in-context learning and the inclusion of 'think' actions. REFLEXION surpasses REACT in shorter tasks; however, comparing REFLEXION with other agents is not entirely fair. REFLEXION can run up to four rounds, while the others are limited to one round. This discrepancy is particularly unfair for tasks involving multiple-choice scenarios. Nevertheless, we include REFLEXION's results to analyze the potential of such methods.
Exception handling. Consider the example in Figure 2, where the stove is broken, presenting an exception. Agents like DRRN and TDT often resort to repeating meaningless action sequences (e.g., continuously attempting to activate the stove or moving between rooms aimlessly). Although the SWIFT module, when used independently, improves upon this due to its larger context window from imitation learning, it still struggles to address exceptions robustly. ReAct and Reflexion occasionally utilize the 'think' action or reflections to redirect agents towards alternative solutions, but the generated actions rarely achieve the new subgoals if they are not grounded. In contrast, the plan-and-ground prompts in our SAGE module handle exceptions more effectively.
Cost-effectiveness. Despite SAGE invoking LLMs APIs twice for inference, its overall cost remains lower, as the result is a sequence of actions typically containing about 5 actions. In comparison, SAYCAN and REACT require 1,855.84 and 1,971.03 tokens per action (tpa) respectively, while REFLEXION necessitates 2,983.46 tpa. SWIFT SAGE, on the other hand, only uses 757.07 tpa. Given its superior performance, SWIFT SAGE proves more cost-effective than other LLM-based methods. This efficiency is primarily attributed to invoking LLMs only when needed (courtesy of our strong SWIFT module) and the action buffer mechanism.
Efficiency. To thoroughly examine the efficiency of agents across all task types, we use Figure 3 to visualize the average trajectories of the first three testing variations for each task involving
SWIFTSAGE, REACT, and the oracle agent. We arrange the tasks based on their average lengths of oracle trajectories (*Len in Table 1). We observe that oracle trajectories consistently achieve perfect scores, yet SWIFTSAGE can reach similar scores more efficiently. This is particularly evident in longer tasks (the bottom two rows), although SWIFTSAGE does not achieve a perfect score for a few tasks (e.g., 9-2 and 1-3). Interestingly, we find that REACT performs competitively in shorter tasks (e.g., 4-2 and 3-4), but most trajectories plateau at an intermediate score and fail to reach 100.
More analysis. Due to page limit, we have to provide further details and analysis in the appendix, including more detailed analysis on cost-effectiveness and efficiency, additional case studies and ablation studies, sensitivity to LLM choices, and an evaluation of the SWIFT-only agent.
Contributions. We present SWIFT SAGE, a generative agent for complex interactive reasoning tasks, inspired by the dual-process theory of human cognition. The agent framework comprises two modules: SWIFT, responsible for fast thinking, and SAGE, dedicated to slow thinking. The SWIFT module is a smaller LM that is fast and specialized, while the SAGE module focuses on prompting LLMs (e.g., GPT-4) for subgoal planning and reflective thinking. Through extensive experiments on 30 distinct tasks within the ScienceWorld benchmark, SWIFT SAGE outperforms baseline agents, achieving state-of-the-art performance, increased efficiency, and reduced cost.
Implications. The success of SWIFT SAGE highlights the potential for collaborative frameworks combining smaller LMs and LLMs in complex reasoning tasks. Smaller LMs can be trained more easily to recognize task-specific and environment-specific patterns, fostering effective in-distribution generalization. On the other hand, LLMs demonstrate remarkable zero-shot generalization abilities and deliberate thinking, though grounding their outputs in real-world environments remains challenging. We posit that dual-process agents, harnessing the strengths of both approaches, constitute a crucial step towards addressing complex interactive reasoning tasks and building general AI agents. Additionally, we can regard SWIFT SAGE as a method within the broader context of utilizing LLMs as controllers or planners for decomposing complex tasks and leveraging APIs/tools [19, 13, 29, 28]. To this end, we have explored applying SWIFT SAGE in web tasks and coding for math problems.
Limitations. Our work has been evaluated solely within a textual simulator, ScienceWorld, which supports a limited set of actions and tasks compared to real-world situations. Also, we did not implement any safeguards to prevent agents from engaging in potentially hazardous actions that could occur in the real world, such as picking up substances from a blast furnace. We argue that one important future direction is to develop a true open-ended environment, allowing agents to interact with a much wider variety of actions and objects to better emulate real-world scenarios. Besides, the use of LLMs in SAGE may present scalability challenges, as LLMs require significant computational resources and may not be feasible in some settings. Future research should explore the generalizability of SWIFT SAGE to other domains and the potential for more lightweight approaches to slow thinking. In addition, we believe it is important to train agents beyond simple supervised fine-tuning and to learn a trainable module to decide when to switch between SWIFT and SAGE mode.
We thank Peter Jansen, Eric Xingdi Yuan, and Marc-Alexandre Côté for valuable discussions. We thank members of the INK lab at USC and the Mosaic team at AI2 for valuable feedback on this project. Xiang Ren is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200006, the DARPA MCS program under Contract No. N660011924033, the Defense Advanced Research Projects Agency with award W911NF-19-20271, NSF IIS 2048211, and gift awards from Google and Amazon. This research was also supported by the DARPA MCS program through NIWC Pacific (N66001-19-2-4031) and Allen Institute for AI. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government.