Language Models can Solve Computer Tasks
Authors: Geunwoo Kim
Pierre Baldi
Stephen Marcus McAleer
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language models
Web Navigation
Foundation Models
Decision Making

Abstract: Agents capable of carrying out general tasks on a computer can improve efficiency and productivity by automating repetitive tasks and assisting in complex problem-solving. Ideally, such agents should be able to solve new computer tasks presented to them through natural language commands. However, previous approaches to this problem require large amounts of expert demonstrations and task-specific reward functions, both of which are impractical for new tasks. In this work, we show that a pre-trained large language model (LLM) agent can execute computer tasks guided by natural language using a simple prompting scheme where the agent \textbf{R}ecursively \textbf{C}riticizes and \textbf{I}mproves its output (RCI). The RCI approach significantly outperforms existing LLM methods for automating computer tasks and surpasses supervised learning (SL) and reinforcement learning (RL) approaches on the MiniWoB++ benchmark. 
We compare multiple LLMs and find that RCI with the InstructGPT-3+RLHF LLM is state-of-the-art on MiniWoB++, using only a handful of demonstrations per task rather than tens of thousands, and without a task-specific reward function. Furthermore, we demonstrate RCI prompting's effectiveness in enhancing LLMs' reasoning abilities on a suite of natural language reasoning tasks, outperforming chain of thought (CoT) prompting with external feedback. We find that RCI combined with CoT performs better than either separately. Our code can be found here: https://github.com/posgnu/rci-agent.

Language Models can Solve Computer Tasks
Geunwoo Kim University California, Irvine kgwOuci.. .edu
Pierre Baldi
University of California, Irvine of pfaadiiics uci edu
Stephen McAleer* Carnegie Mellon University smcaleer@es. emu.edi
Abstract
Agents capable of carrying out general tasks computer can improve efficiency and productivity by automating repetitive tasks and assisting complex problem- solving. Ideally, such agents should be able to solve new computer tasks presented to them through natural language commands. However, previous approaches to this problem require large amounts of expert demonstrations and task-specific reward functions, both of which are impractical for new tasks. In this work, we show that a pre-trained large language model (LLM) agent can execute computer tasks guided by natural language using a simple prompting scheme where the agent Recursively Criticizes and Improves its output (RCI). The RCI approach significantly outperforms existing LLM methods for automating computer tasks and surpasses supervised learning (SI.) and reinforcement learning (RI.) approaches the MiniWoB++ benchmark. We compare multiple I.LMs and find that RCI with the Instruc(GPT-3<RLLEr LLM is state-of-the-a of demonstrations per task rather than tens of thousands, and without task-specific reward function. Furthermore, enhancing LLMs' reasoning abilities on suite of natural language reasoning tasks, outperforming chain of thought (CoT) prompting with external feedback. We find that RCI combined with CoT performs better than either separately. Our code can found here: https: :// ///gttuub.//t///// com/ cwm/ commposgnu/rcii agent
MiniWoB++, using only handful
demonstrate RCI prompting's 's effectiveness
1 Introduction
long-standing goal artificial intelligence has been to create generally-intelligent agents that can accomplish cognitive tasks well as humans. Such agents should be able to solve any computer task a human can by communicating via natural language. By automating repetitive tasks and providing assistance in complex problem-solvine productivity.
emnerall-intteligeen virtual agents may radically increase
Recently, large language models (LLMs) have shown remarkable in-context learning capabilities across variety of domains and tasks [12 69 17
Although L.MMs can impressively previous approaches to using LLMs that directly take keyboard and mouse actions on computers have had difficulty compared to imitation learning and reinforcement learning approaches (24). LLMs that take keyboard and mouse actions on computers face a number obstacles, such as ensuring that generated actions are task-appropriate (task grounding), feasible in the agent's current state (state grounding), and admissible be executed (agent grounding).
26
46
manipulate text and can use high-level API tools [59
48
41
*Correspodding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Use the lerrs beloe io Ber dalite anding
For Baenda dies on
Lhe the teethox thurdeer "Dannie"" Play rg ein gamn preas "Search" than
Popuadd the owetlons on
"Like" button
Terntrat
grasss curazz Pauctbuu
Search
Section #11
#2
Luok leginn The Man
effend.
Guaes between
Clzk onehw furton
Fixd Briee the trash con to deltt
Switch betwean the find and blik link "Gus"
and preas Submii Usa the number
ight
Warling
mobi redertoncia maunts damm
Klersten
Primary
Teb BY
Tash
Tab 43
Pollentasque tr.
Submit
sagata, color ascurion
submit
Marin
Rhongus nutrum amet
sod omare poteril Et
Magna demintuu..
baanii momeuadds panu
Nuñl nuno 3016..
MiniWoB++ environment. Every task contains natural language prompt in yellow. The agent then uses keyboard strokes and mouse clicks to accomplish the task.
Figure 1:
The previous best best-performing approaches for taking actions on computers have not used LLMs. Instead, they have trained networks from scratch to predict actions given prompts and screenshots or DOM information, either via supervised learning (SL) from expert demonstrations, reinforcement learning (RL) on handcrafted reward signal, or both (SL+RL) 30] Although SL+RL works well on number of individual computer tasks, since it requires expert data and reward function for every task, it has not been shown to generalize to novel tasks few-shot setting.
In this work, we show that a pre-trained LLM agent can successfully execute computer tasks guided by natural language. Our method employs simple prompting scheme, which we call Recursive Criticism and Improvement (RCI), that sigmificantly outperforms existing LLM methods for automating computer tasks. RCI works by first having the LLM generate an output based on zero-shot prompting. Then, RCI prompts the LLM to identify problems with the given output. After the LLM has identified problems with the output, RCI prompts the LLM generate an updated output.
When applying RCI computer tasks, we improve task grounding, state grounding, and agent grounding sequentially. Firstly, task grounding prompts the LLM with the task text, instructing it to generate high-level plan. Secondly, state grounding connects high-level concepts derived from the task grounding step with actual HTML elements present in the current state, subsequently outputting the appropriate action. Finally, agent grounding ensures the correct formatting of the action output obtained from the state grounding step. RCI is applied to each of these three steps; however, we find that critiquing the state-grounding step is only necessary once.
We evaluate the RCI approach on the MiniWoB++ benchmark 61), and show it surpasses existing SL, RL, and LLM approaches. Furthermore, it proves itself to state-of-the-ee compared to existing methods, using only a small number of demonstrations per task instead of tens of thousands, and without relying
task-specific reward function. This significant reduction in required demon- strations and the elimination of task-specific reward functions make our method more practical and accessible for new tasks. Furthermore, as the capabilities of LLMs continue improve, one can expect the performance of our method to improve as well.
In addition to its success in automating computer tasks, we also showcase the effectiveness of RCI prompting in enhancing the reasoning abilities of LLMs on a suite of natural language reasoning tasks. When external feedback is given, our method achieves significant performance increase shot prompting and slightly improves upon chain-of-thhe tf-thhuhht [73] (CoT) prompting. Interestingly, RCI and CoT have synergistic effect, and their combination outperforms all other methods.
In summary, our work presents new powerful and practical approach to enabling L.LM agents to execute computer tasks guided by natural language. The RCI prompting scheme not only outperforms

---

Valerie earns $5000 per month, 1/2 of what her brother earns. If their mother earns twice their combined salany, what's the total amount of money they all have together?
Zero-Shot
RCI
A:
Valorie's brother $10,000 month, while Valeria herself earns that amount. Together, they earn total of $1 5,000 per month. Their mother's monthly eaminns amount to $30,000, which the combined satary Valeria her brother. Thus, the total amount money they have together is $30,000 (Wiongg)
<initial output generation>
A: Valerie's brother $10,000 per month, while Valerie herself eams
amount. Together, they cam a total of $15,000 per month. Their mother's monthlly earnings amount to $30,00, which is twice the combined salary Valerie and her brother. Thus, total amount of money they all have together is $30,000.
Few-Shot-CoT
<Few shot exemples>
<Critquee
Review your previous
problems with your answer.
Valerie earns $5000 per month, half beother's earnings,
making his monthly income $10000. Combined, Valerio and her
All individuals, including Valerie, her brother, and mother, should be calculating overall money.
brother earn $15000 per month. Their mother double their
combined salory, amounting to $30000 per month. total, they have $30000 collectively. (Wrong)
<Improve>
Rased the problems you found, improve your
Zero- Prro--hot-CoT
A: think step step. First, Valerie's brother Valerie earns (half her brother's earnings). Next, their mother's Income is twice the combined earnings of Valerie and her brother. in income amounts $30,000. (Wrong)
and
Valerie's brother eams $10,000 per month, while Valerie herself $5,000 Togather, they
total $15,000 month. Their
mother earns double their combined salery, which amounts to $30,000 per month. In total, they have 45,000 collectively, (Correct)
Figure 2: Illustrative examples of explicit RCI prompting and baseline prompting approaches on the GSM8K dataset. RCI prompting effectively addresses logical errors that arise in the baseline prompting approaches. Prompts text is displayed in violet color.
previous methods in computer tasks, but also improves reasoning abilities for LLMs more broadly, making it a significant contribution in the development of intelligent agents.
2
Methods
2.1 RCI Prompting
The self-critiquing ability of LLMs has demonstrated that LLMs can find errors in their own output by themselves [58, 20 B). In light of this, we introduce a simple reasoning architecture called RCI prompting, where prompt LLMs to find problems in their output and improve the output based on what they find. This architecture is designed to further enhance the reasoning ability of LLMs by inserting critique step before generating the final answer. Figure catureeccmmpree example traces of RCI prompting and baseline prompting methods on GSM8K dataset where language models should answer grade school math problems. While baselines elicit answers with single step of prompting RCI consists of two steps: criticize the previous answer (e.g., "Review your previous answer and find problems with your answer") and improve the answer based on the critique (e.g.. "Based on the problems you found, improve your answer"). In this way, RCT prompting finds errors (e.g., the overall sum of money only considered Valerie and her brother) in the previous answer and generates an improved answer (e.g.. money from Valerie's mother is included in the total) conditioned on the critique. The iterative process of RCI can be continued until specific conditions are satisfied, which could include receiving feedback from the environment, reaching the maximum predetermined number of iterations, or adhering to certain heuristics. We define two approaches for achieving RCI: explicit RCI and implicit RCI. Explicit RCI includes the critique in the prompt to generate improved output and implicit RCI updates the previous output directly without sampling a critique explicitly. Examples of explicit RCI and implicit RCI applied to computer tasks are illustrated in Figure where the action plan is improved based on an explicit critique while actions are updated without an explicit critique. This process is described in more detail in the following section.
2.2 RCI for Computer Tasks
In this section we describe the application of RCI to computer tasks via a decomposition of action selection into three reasoning steps: task grounding, state grounding, and agent grounding. The first step, task grounding, involves generating plan for task-solving and conditioning actions on this plan, with RCI being used to improve the plan's 's success rate. The state grounding subsection discusses the importance of grounding actions in the environment for language-based agents and how implicit RCI is used to refine task-grounded actions to be feasible in the current state. Lastly, the agent grounding
3

---

bobow daiioo urdding di
Task
sermiral delete snding with ace actemion
Explicit RCI
Tamiina
les Leteee Thoe Pelp
Plan
Critique
Improve
inver spec
Type ti' commodd in filesi termina
1. Type, 'im (Niaroma]" command dbdate the like unding with "1t".
the in Without krowt ng Reename the
esterdie
Type from [laassmo commond to deles the
sandng with yet
3. Frees tenter
sigot carnau difiia the lice
lyping each commnard to
Task grounding
Initial state
1 step
2 stop
Implicit RCI
3 step
tablie concertly tu
2003
caekure red resileto shenst,
ancording to the current pran, the not proper instaation chruldiiee Typ- [Reename"
(tee specit intruttion for uatving the
[oomudaring oxipur iomeeeeeeeeeeee
Thheelore, the single intruction that matches one the repdar expressiions type code.
Llak stoule by "ype' intruction with command.
State grounding
Agant grounding
Current state
Figure 3: An illustrative execution trace of the agent for terminal tasks with RCI prompting. The language model generates a step-by-step plan for the high-level task described in natural language, which in this case involves using the terminal explicit RCI on this plan, where we sample an improved plan based on the critique and the previous plan, resulting in improvement in the task- bisk-grounding of the plan. For each step, first sample the task- task-grounded action that follows the improved plan, and then the implicit RCI updates the task-grounded actions sequentially provide hate-grounding and agent-grounding. Finally, the agent-grounded action executed by the instruction-followe prompts are highlighted, and the remaining text shows the outputs generated by the language model.
delete file ending with " ".rb". We then
agent on the environment. The
step focuses ensuring that actions are admissible for the computer agent by employing implicit RCI and conditioning agent- sroundgd actions on the current state, task, and other grounded actions, with loop count set to optimize performance.
2.2.1 Problem Setting
We assume that we are given an cestruction-followwee computer agent that can execute a set of admissible actions given some natural language instructions. An instruction that is not part of the admissible actions will be ignored. At every step, we receive high-level natural language task prompt and state of the environment. Given the current state and task, we sample the most probable action from LLMs. The generated natural language action is then fed into the computer agent. Sampling the actions in a fully generative manner presents a challenge, as the actions must consider the given task, feasibility in the current state, and admissibility for the computer agent simultaneously. Therefore, we propose decomposing this action sampling into three reasoning steps each of which considers task grounding. state grounding. and agent grounding. Task grounding improves actions to be more effective in solving the given task, state grounding ensures the feasibility of actions in the current state, and agent grounding considers the executability of actions given the specification of the computer agent. We first sample step-by-step plan to solve the given task which improves the task grounding. Next, the task- task-grounded action is sampled conditioned the current state, task, and the generated plan. The state-grounded actions is generated conditioned on the task-grounded action. If the task-grounded action is not executable by the computer agent, the agent-grounded action is sampled. For each sampling of grounded action, we use RCI prompting to make LLM consider some specific information for grounding.
2.2.2 Grounding Language Model in Computer Tasks
Task grounding. In the action sampling process, the first step involves generating a plan of actionable steps task solving from L.MSs. Subsequently, actions are sampled from the same LLMs, taking into account the present state, task, and generated plan. The benefits of conditioning

---

the plan for improved grounding of actions are twofold. First, it enables LLMs to identify the stage of task solving at which the agent located, serving as a memory module. Second, we can perform explicit RCI on the generated plan to further improve the plan's 's success rate. Although the number of explicit RCT loops can be arbitrary, we observe that a single pass of explicit RCI suffices for most of MiniWoB++ tasks.
State grounding. language-based agents, grounding actions in the environment is a crucial step to enable real-world task performance. The aim of this phase is to enhance the task-grounded actions to be feasible in the current state. Although the actions generated in the preceding phase may align with the task, they may lack the specificity required to be executed in the current context. For example, the assigned task is to forward an email from Bob to Alice and the action obtained from the task grounding phase is to click connection between the abstract concept of "email from Bob" and the concrete element, such as the email heading. in the current webpage state represented by HTML. To achieve this goal, perform the implicit RCI and prompt the LLMs to consider the current state, which subsequently outputs refined state-grounded actions. Moreover, the state-groud state-grounded action is additionally conditioned on the task-grounded action. We avoid repeating the implicit RCI cycle more than once as it does not impact the success rate based our observations.
email from Bob in the email inbox, it is necessary establish
Agent grounding, To ensure the successful integration of language-based methodologies in decision-making processes, it imperative to establish a scalable framework that guarantees the admissibility of actions derived from the language model. While the preceding steps of sampling produce a state-gron sntte-grouned action that both feasible and grounded in the task, it may not be executable by the agent due to issues such as improper formatting. To address this, Implicit RCI is employed, whereby an agent- agent-grounded action is sampled conditioned on the current state, task, sosk-grounded action, and stategrroudde action. The LLMs are prompted to consider specifications of the computer agent. The implicit RCI is repeatedly run until the resulting action is executable, with maximum loop count set to limit the number of iterations. Empirical analysis on Mini WoB+ tasks suggests that setting the loop count to 3 yields optimal performance.
Evaluation
3.1 Reasoning tasks
In our grounding enhancement process, RCI prompts the LLM to criticize its prior output, considering the given context (e.g., current task, state, and agent), which ultimately leads improved output. We first demonstrate the effectiveness of RCI prompts in augmenting the reasoning capabilities of LLMs across a range of reasoning benchmarks. We compare RCI Chain-of-Thought (CoT) prompting, state-of-the-art method recognized for its effectiveness in reasoning tasks.
approach with Few-Shot-CoT [73] where few chain-of-thougle demonstrations are given as examples in prompting, and Sero-Shot-CooT [3] that elicit multiple reasoning steps by simply adding "Let's think step by step" to the prompt. Following Kojima et al. 333, our evaluation is conducted with 8 datasets from two categories of reasoning: arithmetic and commonsense. Please refer to Appendix C.2 for comprehensive depiction of the datasets. We use the same experimental setting with their answer extraction method except that we use InstructGPT-3 + RLHF (gpt-3.5-turbo) as the underlying language model. We use the same prompts that CoT uses and we also use the answer cleansing approach used in CO',, but we only used answer extraction prompting in zero-shot CoT experiments. We also use the same few-shot examples that were introduced in [3 to evaluate Few-Shot CoT's performance on five arithmetic reasoning tasks. A threshold is established by setting the maximum number of RCI loops to two, terminating the loop once the output aligns with the ground-trus data. We observed that in the absence of this external feedback mechanism, the RCI process is prone to false negative critics, subsequently leading to decrease in performance. Experimental results indicate that RCI without external feedback achieves zero-shot performance in half of the benchmark tests, but underperforms in others, as shown Appendix [7
Specifically, we compare
Comparison with Zero-Shot. RCI prompting is better at solving reasoning tasks compared to zero- shot prompting. Table summarizes the accuracy of our approach (Zero-Shot + RCI) and standard zero-shot prompting for each reasoning benchmark. Zero-Shot + RCI substantially outperforms the
5

---

standard prompting in all benchmarks including arithmetic (GSM8K, MultiArith, AddSub, AQUA, SVAMP. SingleEq) and common sense (CommonSenseQAA, StrategyQA) tasks. RCI prompting even achieves score gains from two arithmetic reasoning tasks (Singlefiq and AddSub), which do not require multi-step reasoning. This distinguishes our RCI prompting from the previous CoT prompting methods 773 33 that are not useful in simple reasoning tasks. It is also worth noting that RCI prompting achieves a significant performance gain in commonsense reasoning tasks (CommonSense(4 and StrategyQA), While Wei et al. [73] reported that only a substantially large PaLM (540B) model can benefit from Few-Shot-CoTl RCI prompting can provide performance gain even with smaller InstructGPT-3 RLHF (175B) model.
Arithmetic
Common Seuse
GSMBK
MultiArith
AddSub
SVAMP
SingleEq
AQuA
CommonSenseQA
StrategyQA
Zero-Shot 77.95 85.43
94,48
88.58 89.76
80,70 84.65
86.61 94.49
60.23 67.32
64.56 68.11
48.81
Zemo-Soo Shot RCI
97.64
61.81
Table 1: RCI prompting increases the reasoning capability of LLMs on all of eight reasoning benchmarks.
Comparison with Chain-of-Thought. The performance results of RCI and CoT baselines on arithmetic reasoning tasks are summarized in Table 2 Notably, Zero-Shot + RCI outperforms Zero-Shot CoT and Few-Shot CoT without any CoT prompting in four tasks except MultiArith. In MultiArith tasks, where most of the standard prompting's ss answers are correct (96.06%), RCI prompting does not yield significant performance gains. RCI prompting has synergistic collaborative impact on the two CoT baselines. Namely, Zero-Shot CoT RCI and Few-Shot CoT + RCI attain the highest scores on four out of the five tasks. These findings suggest a promising avenue for future research: combining RCI with other prompting methods for CoT, such as self-consistency (58)
GSMSK
MultiArith
AddSub
SVAMP
SingleTiq
Zero Shot 7ero-Shrt RCI
78.35
96.06
85.83
78.35
91.34
85.43
89.76
84.65
94.49
Zevo-Shot CoT
82,28
96.85
83.86
79.92
89.37
Zevo-Shot Col
86.22
89.88
90.94
Few Shat CoT
80.31
98.82
89.37
83.46
91.73
ShotShat CoT RCI
84.25
99.21
90.55
87.40
93.70
Table 2: Chain-of- prompting exhibits synergistic effect when coupled with RCI prompting in arithmetic reasoning tasks.
3.2 Computer tasks
3.2.1
Setup
MiniWoB++ benchmark suite. The miniwob++ task suite is selected as the main benchmark to evaluate computer agent. MiniWoB++ [36]. extension of MiniWoB 61]. is web-based simulation environment that offers a diverse range of computer tasks, from simple button-clicking to complex compositional tasks requiring advanced reasoning, such as solving math problems. Its shared action space, including keyboard and mouse, and a common state space centered around HTML code enables our proposed agent be thoroughly evaluated in ample tasks. Additionally, the varying levels of complexity between tasks enable a systematic evaluation of our work. The action space consists of two operations each of which controls the keyboard and mouse. The first action enables typing of arbitrary characters or special keys such as Backspace and Enter. The second action involves moving and clicking the mouse, allowing the agent to interact with visible HTML elements on a webpage. All actions can be executed through natural language instructions defined by regular expressions that are presented within the initial prompts provided to the L.MMs. The regular expressions employed in evaluation are presented in Appendix D Our action space definition is similar previous works,

---

MiniWoB++ Average Performance
MiniWoB++ Average Performance
In-oorted Leaning
0.94
0.94
Amount of Data
Average Success
RL
0.57
0.57
Average Success Rate
CC-Net (SL RL)
0.46
Workflow guidad exploration (SL RL)
0.52
WabN-T5-38
WebN-TS-38
CC-Net
Webbl WabN-55338 75-39
(no hesoor)
WebN 73- 38
CC-Net
(no dictorary echene)
Others
CC.tet
CC-Net
St Setd
Skth
SL RL Bold
(no action history)
X
CC-Net (SL)
World (SL RL)
Model
Amount of Demos per Task
(b)
(a)
Figure 4: (a) Average performance comparison with baselines. Our agent with RCI prompting achieves state-of-the-art performance in MiniWoB+++ environment. The tasks that were included in the averaging process are indicated in Table [8] (b) Relationship between performance and amount of expert training data. Our agent displays comparable performance to the current state-of-the-arr scores on the MiniWoB benchmark, despite using the least amount of data.
which clicking actions directly interact with HTML elements. However, for typing actions, we extend beyond simple form-fi foln-filing by using keyboard- typing actions. Instead of relying on dictionary-baae based typing actions [30), where the agent simply chooses from predefined dictionary of texts, our approach requires the agent to predict the proper text input. Our approach, therefore, has a better generalization capability for diverse computer tasks. The state space of our agent consists solely of HTML code.
such 25 32
36
Model choices. For the purpose of evaluating the effectiveness of RCI prompting, multiple lan- guage models are used in our experiments. Specifically, we employ three models, namely, GPT-3 (davinci) [5) InstructGPP-3 (text-davinci-002) [47 72 57], and InstructGPT-3 + RLHF (gpt-3.5- turbo, gpt-4) [47]. Unless otherwise specified, we primarily evaluate our computer agent with the InstructGPT-3 + RLHF models (gpt-3.5-turbo gpt-4). Additionally, we use GPT-3 and InstructGPT-3 models for ablation studies. All the models were obtained through the OpenAl API, and further details can found in Appendix C.T
Evaluated tasks. We employ of 55 tasks to enable fair comparisons with baselines, previous works are only evaluated on subset of tasks consistently. Furthermore, to assess the performance of models challenging tasks, have selected tasks that involve free-form language typing actions, which have been reported to have an almost-zero success rate in previous works (e.g., terminal). Notably, certain commonly evaluated tasks in prior works excluded due the excessive length of HTML code for some UI components, which are described Appendix C.3
Metrics Consistent with prior studies, our main evaluation criterion the success rate, which measures the ability of our agent to actually complete the assigned task. This rate calculated as the proportion of successful episodes, which are defined those in which the agent receives a positive reward. We identified two modes of failure: the production of unexecutable actions and task failure. When the agent generates an unexecutable action following the implicit RCI step, fails immediately, Moreover, an episode is considered unsuccessful when the agent, despite effectively executing the plan generated, is unable accomplish the task and thus receives no reward.
3.2.2 Outperforming baselines on MiniWoB++ task suite
We present Figure 4a which summarizes the average success rate of our agent and baseline models over the MiniWoB++ benchmark. The results demonstrate significant outperformance of our approach over supervised learning models. Specifically, we observe 41 % higher score than the WebN-T5- 3B, which employs a finetuned large language model with 12K expert demonstration data. Our approach also outperforms reinforcement learning approaches that require order of magnitude

---

more interactions with the environment. Among all the baselines, our approach achieves the second highest score. The sole model that surpasses our agent the CC-Net, which involves co-training of reinforcement learning and imitation learning. However, direct comparison with CC-Net is not possible since it uses dictionary-basse typing actions. In other words, CC-Net selects text from predefined list for typing actions in some tasks, while our approach is fully generative. Thus, action) in Figure (4a serves as our appropriate comparison and we
CC-Net (without dictionary-bass outperform it by 6%. The performance data for CC-Net (with no dictionary-based action) is obtained from the ablation study section in their paper (30].
Another comparative analysis is performed to evaluate the performance of our agent in contrast to the state-of-the of-th--rtt agents in three categories, namely supervised learning, reinforcement learning, and a combination of both. To facilitate a fair comparison, we specifically isolate LLM- based state-of- ahe-art approaches, which share similarities with our approach to solving computer tasks. The best per-task performance achieved by each category is then aggregated, and the outcomes are presented as SotA in Figure 4a] The result shows that our agent surpasses SotA by 37 percentage points in supervised learning and by 27 percentage points in reinforcement learning. Notably, proposed RCI prompting method outperforms the SotA LLM approach (24), even when the latter employs both finetuning and few-shot examples in prompts. This outcome highlights the effectiveness of our approach in extracting vital knowledge for computer tasks from language models. Our agent even achieves slight edge over SotA (less than 1 percentage point) in the combined use of supervised and reinforcement learning, which employs significantly more expert data and online interactions. We also provide task-level performance comparisons in Figure T0 where tasks are arranged in ascending order based on the difference between our agent's performance and the baseline. We observed three main failure modes of our agent: (i) underperformance in tasks that require long-horizon planning (e.g., guess-number, search-engine, use-spinner); (ii) difficulty in selecting appropriate actions for tasks that require multi-step reasoning (e.g., tic-tac-toe, use-autocomplete), and (iii) lower scores in tasks that rely on visual rendering of HTML code to solve the task (e.g,, count-shape). These failures are explained in more detail in Appendix P
3.2.3 Lowest sample complexity
Figure provides a comparative analysis of the total number of samples used in several models and their mean performance. We begin by discussing CC-Ner 300 model, which employs 2.4 million expert demonstrations (equivalent to 6,300 hours) collected from 77 human participants across 104 tasks for behavior cloning. This amounts to an average of 23,076 demonstrations per task. In contrast, the WebN-T5-3B (24) model uses 12,000 expert demonstrations to fine-tune its pre-trained T5 model. Rather than directly updating model parameters with demonstration data, our approach involves integrating two to three demonstrations into the prompt for in-context learning, which biases the model output without any parameter updates. This approach allows our agent to generalize to unseen tasks with only a handful of demonstrations. Our results show that our agent achieved a higher success rate than all baselines, requiring 120x fewer samples than WebN-75-33 and 11 11,000x fewer samples than CC-Ner. Given the challenges of obtaining expert demonstrations for computer tasks, our findings demonstrate the practicality of our approach in automating such tasks.
3.2.4 Ablating the groundings
This section examines the impact of grounding improvement on task success rates. We conduct ablations to isolate the contributions of task, state, and agent grounding improvements by eliminating RCI prompting each stage. We categorize tasks by three different difficulty levels to provide more detailed understanding of the effects of grounding improvements across diverse range of tasks. We conducted task grounding ablation by climinating the plan sampling stage. This modification entails generating actions directly from the state, without the need for conditioning on a step-by step-by-step plan. State grounding is evaluated by directly applying the agent-grounding actions. Lastly, we ablate the implicit RCI of the agent grounding by letting the state- srtte-grouned action be the final output of the agent. Figure 5 iillusttraes the performance degradation resulting from each ablation of grounding. Our results indicate that each grounding contribution is essential to solving computer tasks, with each contributing almost equally to the overall success rate. The reason for this is partially due to the fact that the three methods of improving grounding are not mutually exclusive, but rather complementary, with one enhancement in grounding contributing to multiple action groundings. Examples of cross-grounding improvement are provided in Appendix E
update to task-grounded
8

---

Moreover, it has been observed that state grounding plays crucial role in enabling agent to use relevant information during episodes, particularly in scenarios where the initial state does not offer sufficient information to accomplish the task, such as lerminal task. Interestingly, task grounding significantly improves the success rate when task requires a long-horizon action plan, such as the click checkboxes large task. We also observe that agent grounding significantly enhances the feasibility actions. Notably, simpler tasks, the success rate decreases by 60% in contrast to the baseline without the agent grounding. This finding is of particular significance as it distinguishes our work from prior investigations [ 28,, which employ additional trained model components. In contrast, study solely relies on the reasoning ability of language models.
Easy
Medium
Hard
Success rate
1.0
1.0
Success rate
1.0
Success 1
0.8
0.8
0.8
0.2
0.
0.2
0.0
0.0
0.0
Without grounding
Whlthr grourding
Whthos grsunsing
Vitholt
Witter grourding
Withost grandinng
Baxel
Vittoul grounding
Wittor
Without grounding
grounding
grounding
Figure 5: Ablation analysis on the different types of grounding across tasks with varying degrees of difficulty. The experimental design employs the use of InstructGPT-3 RLHF model (gpt-3.5-turbo).
3.2.5 Ablating the language model
The performance of agent is contingent on the quality of the underlying pre- pre-trained language models used, so enhancing language models can lead an improvement the agent's performance. In this section, present comparison of the agent's performance using three distinct language models: GPT-3, InstructGPT-3, and InstructGPT-3 + RLHF (gpt-3.5-uurbo). Our objective is to investigate the relationship between LLMs' capability and their ability solve MiniWoB++ tasks. The experimental setting employed in Section 3.2.4 is replicated in this study. Figure 6 depicts the average success rate of three language models tasks of varying difficulty levels. Our results reveal that LLMs struggle to effectively complete tasks without instruction fine-tuning. This may be attributed the absence of intricate prompt engineering, that GPT-3 displays sufficient competence in comprehending HTML code, regular expressions, and engaging in reasoning.
observations have indicated
Easy
Medium
Hard
Success rate
1.00
1.00
LUCO 65 raaee
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.80
0.00
0.00
HI.MI (Bhealin)
HRUF (passina)
+NLFF (Bhaal
Figure 6: Ablation study on different language models across tasks of varying degrees of difficulty.
4 Limitations
In the course of our work, several limitations became apparent that may serve potential avenues for further research. One central concern our primary focus on the InstructGPT-3- RLHF models (gpt-3.5-turbo, gpt-4), leaving the generalization ability of RCI to other models unexplored. The versatility RCI across diverse models remains pertinent question, suggesting that future studies should expand their scope to determine the robustness and adaptability of RCI. Handling lengthy HTML presents another challenge. The current model grapples with extensive HTML states. While has been suggested that efficiency might be bolstered by pruning HTML states exclude non-critical

---

elements, the task itself is non-trivial. A fundamental constraint of LLMs is the limited context length, which can hamper handling extensive HTML states effectively. Addressing this may require architectural adjustments or novel parsing methods. Our agent's action space, mainly restricted to clicks and typing, limits its web navigation capabilities. There's need to diversify its actions for a more seamless experience. Furthermore, The agent's 's focus on short-term decisions overlooks the necessity for long-term strategy, especially in tasks requiring coordinated sequences. Broadening this focus is essential for versatile applications. Lastly, the intricate UI components populating contemporary websites present a challenge for LLMs to fully understand the HTML states. The subtle nuances of such components, which may not be discemible through HTML alone, underscore the need for adding more modalities to the state definition. Addressing these issues is crucial to enhance the RCI agent, making it more adaptable and efficient in practical applications.
5
Discussion
This work is part of a growing literature showing that LLMs might be you need for hard decision- making problems [76). In contrast to imitation learning and reinforcement learning approaches, LLMs can solve novel tasks in zero-shot or few-shot manner, and don't 'tt require task-dependent expert data or a reward function. Furthermore, we expect that as the capabilities of L.MSs and foundation models increase, our method will naturally improve as well. However, we find that current capabilities of LLMs aren't 't as powerful as task-dependent SL+RL approaches on some computer tasks. Also, RCI is more expensive to run compared approaches that just sample once from the LLM. There are many avenues for future research in increasing the capacity of LLMs in decision-making tasks. First, our experiments use LLMs on HTML code, but ideally methods based on multimodal foundation models [16
55 2 46 will be able to take actions based on text, images, audio, and video as input 18 44 71. Second, the results presented in this paper all use pre- pre-trained LLMs. We expect the performance of our method to increase when using LLMs fine-tuned to solve computer tasks.
Importantly, current LLMs are poor at reasoning tasks, such as playing tic-tac-toe, because they do not think ahead. Although RCI improves reasoning capabilities in LLMs, there exists much work to be done on increasing the reasoming capabilities in LLMs. This will be crucial to accomplish hard cognitive tasks on computers that require thinking ahead. Similar to other prompting-bee approaches for reasoning in LLMs, RCI can be viewed as using the LLM's s output to write to an external memory, which is later retrieved to choose an action. LLMs with memory have been demonstrated to be computationally universal [60], mcaning that in principle all that is needed to run arbitrary programs is the right prompt. Since RCI represents a basic version of this powerful framework, anticipate the development of more advanced RCI variations in the future. There is a vast array of potential methods that repeatedly feed the output of particular prompts into the LLM. For example, multiple different LLMs simulate the information exchange between team members in an organization. This would enable the merging of diverse perspectives to tackle complex problems. In such context, incorporating game theory and multi-agent systems research could significantly enhance the overall performance. Reinforcement learning could be used to discover effective structures involving loops and prompts [8T], either through human feedback reward function. This optimization process can be further refined by exploring the space of potential loop and prompt structures, identifying those that yield the best results, and fine-tuning the model accordingly [75].
given
Acknowledgement
This material is based upon work supported by the National Science Foundation under Grant #2127309 to the Computing Research Association for the CIFellows 2021 Project
References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as I can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv: 2204:22200.01911, 2022.
10

---

[2]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[3] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini arXiv:2212.080735 2022.
[4] Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt); Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melamic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
[6] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprini arXiv:2303. /2712, 2023.
[7] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning, arXiv preprint arXiv:2302.02660 2023.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint vxXiv22110.14166 2021.
[10] Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models. arXiv
preprint arXiv:2208. 14271, 2022.
[11] Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inferente Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022.
[12] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? Language models secretly perform gradient descent as meta optimizers. arXiv preprint 2rXiv:2212.10552 2022.
[13] Ishita Dasgupta, Christine Kaeser-Ched Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, and Rob Fergus. Collaborating with language models for embodied reasoning. In Second Workshop on Language and Reinforcement Learning, 2022.
[14] David Dohan, Winnie Xu, Aitor Lewkowyez, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-Dickstein, al. Language model cascades. arXiv preprint 2rX.v 10342, 2022.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representations, 2020.
[16] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03377 2023.
[17] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient scaling of language models with mixture-of-expee ofeexeerss. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.

---

[18] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended em- bodied agents with internet-scaee knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[19] Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. instruction-finetunel foundation models for multimodal web navigation. In Workshop on Reincarnating Reinforcement Learning at ICLR, 2023.
[20] Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459 2023.
[21] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
[22] Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.1437. 2022.
[23] Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning. Advances in Neural Information Processing Systems, 34:4157-4166 2021.
[24] Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowd- hery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding HTML with large language models. arXiv preprint arXiv:2210.0394, 2022.
[25] Izzeddin Gur, Uirich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. Learning to navigate the web. In International Conference on Learning Representations, 2019.
[26] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 1530016-30034 2022.
[27] Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. EMNLP, pages 523-533, 2014.
[28] Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
[29] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning. 2022.
[30] Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In International Conference on Machine Learning, pages 9466-9482. PMLR, 2022.
[31] Taichi Iki and Akiko Aizawa. Do BERTs learn to use browser user interface? Exploring multi-step tasks with unified vision-and-language berts. arXiv preprint arXii:2203.07882, 2022.
[32] Sheng Jia, Jamie Ryan Kiros, and Jimmy Ba. DOM-Q-NET: Grounded RL structured language. In International Conference on Learning Representations, 2019.
[33] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 2022.
12

---

[34] Rik mas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Compulational Linguistics, 3:585-597, 2015.
Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Du-
[35] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. Proceedings of ACI, 2017.
[36] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang, Reinforcement learning on web interfaces using workflow-guide exploration. In International Conference on Learning Representations, 2018
[37] Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M Dai. Mind's eye: Grounded language model reasoning through simulation. In International Conference on Learning Representations, 2023.
[38] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 7655,, 2023.
[39] Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07688 2022.
[40] Jacob Menick, Maja Trebacz. Vladimir Mikulik, John Aslanides, Francis Song, Martin Chad- wick, Mia Glaese, Susannah Young. Lucy Gamphell-Gillingaane Geoffrey Irving, et Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203. 147, 2022.
[41] Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: survey. arXiv preprint 2302:::2000.78444 2023.
[42] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser-assisted question-answeringg with human feedback. arXiv preprint 4rXiv:2112.09332, 2021.
[43] Rodrigo Nogueira and Kyunghyun Cho. End-to-end goal-driven web navigation. Advances in Neural Information Processing Systems, 29, 2016.
[44] Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. arXiv preprint arXiv:2301.12050 2023.
[45] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop al ICLR, 2022.
[46] OpenAl. Gpt-4 technical report, 2023.
[47] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744, 2022.
[48] Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. ART: Automatic multi-step reasoning and tool-use for large language models, arXiv preprint arXiv:2303. 09014, 2023.
[49] Panupong Pasupat, Tian-Shun Jiang, Evan Liu, Kelvin Guu, and Percy Liang. Mapping natural language commands web elements. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4970-4976, 2018
13

---

[50] Panupong Pasupat and Percy Liang, Zero-shot entity extraction from web pages. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 391-401, 2014.
[51] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nip models really able solve simple math word problems? Proceedings of NAACL, 2021.
[52] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
[53] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
[54] Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re- promppitgg Foundation Models for Decision Making workshop at NeurlPS, 2022.
[55] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research, 2022.
[56] Subhro Roy and Dan Roth. Solving general arithmetic word problems. EMNLP, 2016.
[57] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnad Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[58] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022.
[59] Timo Schick, Jane Dwivedi-Yi Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476. 2023.
[60] Dale Schuurmans. Memory augmented large language models are computationally universal. arXiv preprint arXiv:2301.04589 2023.
[61] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hermandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135-3144. PMLR, 2017.
[62] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 13666 2023.
[63] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.
CLIPort: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[64] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye. George Zerveas, Vijay Korthikanti, al. Using deepspeed and megatron to train megatron-turin turing NLG 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990, 2022.
[65] Shashank Srivastava, Oleksandr Polozov, Nebojsa Jojic, and Christopher Meek. Learning web- based procedures by reasoning over explanations and demonstrations in context. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7652-7662, 2020.
(66) Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmeenee language models. In International Conference Learning Representations, 2023,
14

---

[67] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ:: A question answering challenge targeting commonsense knowledge. Proceedings of NAACL-HLT. 2019.
[68] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201 N.X.02200.00239, 2022.
[69] Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mord- vintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv preprini (at.v::2212.076777 2022.
[70] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le,, Ed Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. In International Confer- ence on Learning Representations, 2023.
[71] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang, Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.012 2023.
[72] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2022.
[73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
[74] Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint arX0022211.0005,, 2022.
[75] Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain of thought im- itation with procedure cloning. Advances in Neural Information Processing Systems, 35:36366- 36381, 2022.
(76] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04123 2023.
[77] Shunyu Yao, Howard Chen, John Yang. and Karthik R Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, 2022.
[78] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.
[79] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:15476-15488, 2022.
[80] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, al. Socratic models:
Composing zero-shot multimodal reasoning with language. In International Conference on Learning Representations, 2023.
[81] Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. TEM- PERA: Test-time prompt editing via reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[82] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Seales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning large language models. In The Eleventh International Conference on Learning Representations, 2023.
15

---

Appendix
A
Broader Impacts
Although the results presented in this paper are only on a research benchmark, if we extrapolate forward the capabilities of these models and methods, we anticipate vast broader impacts that have the potential to revolutionize numerous industries. By allowing LLMs to execute tasks on computers, our approach can enhance the capabilities of Al assistants and automation tools. This could lead to increased efficiency, reduced labor costs, and improved user experiences across any sector which uses computers to work. We are most excited about gains in productivity in science and education, including Al research, which will lead to even faster development of new beneficial technologies and treatments.
However, there are many potential misuses and unintended consequences associated with allowing these models take actions in the world. Malicious actors may leverage LLMs to automate cyber- attacks, manipulate information, or propagate disinformation on a large scale. Additionally, the potential loss of jobs due widespread automation could lead to economic disruption and increased income inequality. There are also obvious security risks of running LLMs on computers (or even virtual machines) such as prompt injection attacks. Perhaps most alarming, future LLMs taking actions on computers may lead to catastrophic runaway chains of events, especially if LLMs are integrated widely in the economy.
To mitigate these risks, is crucial for researchers, policymakers, and industry leaders to work together to establish regulations and ethical guidelines that govern the development and deployment of such technologies. Ensuring transparency, accountability, and fairness in Al systems will be vital in harnessing the benefits while minimizing potential harm. We also believe that the time has come where we as research community must discuss possible ways to coordinate to slow down the pace of developing highly-disruptive technology, necessary.
16

---

B Related Works
B.1 Automated computer tasks
The automation of computer tasks is an important topic for both information retrieval and natural
language processing 43 500
49.
31 65]- Recent efforts have focused on the development of
reinforcement learning agents that interact with websites using raw mouse and keyboard actions [6]]. MiniWoB, a benchmark proposed in [61], has been extended in MiniWoB++ [36], which has become widely-used platform for studying models for computer tasks. Reinforcement learning and imitation learning have been employed in several studies to tackle MiniWoB++ tasks [36] 25 32 233. However, achieving human-level performance requires a significant amount of expert demonstration data (6,300 hours), as demonstrated in [30]. Recent work (24] 19] has suggested the use of large language models (LLMs) to comprehend HTML code and vision transformer [13] to extract screenshot image features, with few-shot in in-context approach showing promising results without extensive RL exploration. Nevertheless, significant amounts of expert demonstration data are still required to finetune L.MMs. On the contrary, the agent we suggest needs less than two demonstrations per task on average and doesn't 't necessitate any finetuning. WebGPT [42] and WebShop [77] show that LLMs can automate some web-based tasks by introducing a handful of custom commands such as Search <query> and Next Page. As result, these methods are limited in scope and do not work on general computer tasks which require keyboard strokes and mouse clicks. In contrast, approach can tackle open-domain tasks at scale.
B.2 LLMs with actions
In recent years, there have been significant advancements in large language models (LLMs), leading to new possibilities for utilizing natural language for decision-making tasks. One approach involves augmenting LLMSs with executable actions [1). Huang et al. (2 showed that L.MSs can be used to plan and achieve simple household tasks, utilizing method for grounding the actions generated
by LLMs by comparing their embeddings with predefined list of admissible actions. However, their work did not consider state grounding. Another study by Ahn et al. [1] proposed SayCan, which grounded the actions by multiplying each candidate action's 's probability under FLAN [72] with the action's 's value function, serving
indicator for the suitability of actions. Huang et al. (29) proposed an extension to the SayCan model called Inner Monologue, which incorporates a feedback loop for state grounding. However, Inner Monologue still requires a pre- t-tinrrinedd language- conditioned robot policy with underlying reasoning capabilities that are not free-formed and flexible, thereby hindering generalization to diverse task domains. Similarly, Zeng et al. (80) employed combination of L.MMs with a visual-language model (VLM) and a pre-trained language-conditione robot policy 663 to perform open vocabulary pick-and-place robotic tasks. Mcanwhile, Dasgupta et al. [13] used Chinchilla 26) as a planner for an agent in the PycoLab environment, but their actor module requires pre-training with reinforcement learning (RL) to follow natural language instructions. In a related line of research, Carta et al. (7) employed online RL fine-tuning to achieve functional grounding of LLMs in the BabyAI-Text environment. In contrast to these previous approaches, our method does not rely on additional model components beyond LLMs for grounding actions. Instead, we propose the use of RCI prompting, which enables LLMs update their actions to be grounded autonomously. As a result, our approach can scale to a wider range of action spaces, including keyboard and mouse actions. Furthermore, prior approaches have been limited by the need for fine-tuning. In contrast, our RCI prompting method is zero-shot approach that overcomes these limitations. More recently, an approach to improve the efficacy of LLMs involves their integration with APIs, allowing them to use external tools such as information retrieval systems, code interpreters, and web browsers [59] 68 40 22). Notably, these external tools necessitate manual engineering and may be constrained in their functionality. In contrast, our agent is equipped with general computer interface, enabling it to access a wide range of functionalities offered by computers.
B.3 LLMs with reasoning
Recent research has also demonstrated that large language models (LLMs) exhibit enhanced per- formance in compositional tasks when they produce traces of the underlying reasoning process
along with the final answer, as evidenced by studies such as [73 45 33 52 This discovery has led
to the emergence of new line of research where reasoning capabilities are used to address tasks
17

---

beyond reasoning 78 29), or enhance reasoning proficiency [33 37 70 39 79 53 7 66 14.
Furthermore, various reasoning architectures have been proposed. expanding from naive prompting. such as Selection-Inference [i]], Least-to-Most [22 and Faithful reasoning [TO]. In the existing
literature, a work closely related to our research ReAct 78 which interleaves reasoning and action for resolving the issue of hallucination and error propagation well helping the model induce, track, and update action plans. An alternative method, Reflexion [62], extends ReAct by improving its performance by allowing LLMs consider previous trial and enror experiences. Nevertheless, due to the necessity multiple rounds of explicit task-specific success feedback from trial and error, this approach may not scale effortlessly ours because it requires task task-specific success feedback. Similarly. Corrective Re- Re-prompting, as proposed by Raman et al. [54] necessitates the establishment of task-specific preconditions. RCI pertains to extended reasoning architecture where LL are instructed to find errors in their outputs and improve them accordingly, which can further be used to ground actions generated from LLMMs in decisionnmakkin problems. Saunders al. [58] used a similar approach to ours by utilizing the self-critiquing ability of L.LMs to generate summaries produced by LLMs. The aim accelerate the human evaluation process by uncovering possible errors in the generated summaries, Likewise, Ganguli et al. [20), employed LLMs morally self-correct their outputs prevent the generation of harmful content. The most recent work that is in the same vein with RCI is Self-Refine [38] which uses localized and aspect-based basset feedback to iteratively refine outputs from LLMs. However, our work is, to the best of our knowledge, the first to demonstrate the self- self-critiquing capability of LLMs only with implicit feedback (e.g., "Find problems with this plan") in enhancing reasoning proficiency.
critical feedback
18

---

C Experimental setup
C.1 Language models
In our evaluation, various pre-trained language models were used. RCI prompting reasoning tasks evaluated using gpt-3.5-turbb which is presented in Tand T Our primary evaluation on MiniWoB++ tasks is conducted using gpt-3.5-turbe and gpt-4, as shown in Figure and Figure 10 We also used davinci, lext-davinci-002, and gpt-3.--turbo for our ablation study on MiniWoB++ tasks. For all model usage, maximum token length of 256 and temperature value of O, indicating greedy decoding, are used. All models are accessed through the OpenAl API between January 2023 and March 2023.
Language model
of parameters
Max. tokens
API provider
API name
GPT-3
175 B(+1)
2.049
OpenAl API
davinci
InstructGPT-3
(2))
4,097
OpenAl API
text-davinci-002
InstructGPT-33 RLHF
+2)
4,096
OpenAl API
gpt-3.5-turbo
InstructGPT-3 + RLHF
8,192
OpenAl API
gpt-4
Table 3: Description of language models. (+1) We identify the model size of GPT-3 by referring to the htt.a/oaa.....cccooooooooooooooo cow//ocssmooellinneeoo/oo/o///oe htwps//mooo--ddd-o-----eeeee (orreeecrorrrsrrheerss))) official document that OpenAl provides (https://beta.og) opw.taippppnii..o
(+2) The size of InstructGPT-base models remains undisclosed by its provider.
C.2 Reasoning tasks
We conducted an evaluation of RCI prompting on eight datasets, encompassing two categories of reasoning tasks: arithmetic and commonsense. In the domain arithmetic reasoning, we considered six datasets: SingleEi [34), AddSub (27). MultiArith (56), AQuA 35,, GSM8K (9, and SVAMP [51]. For commonsense reasoning, we utilized the CommonsenseQA A dataset (67 and the Strategy dataset [21]. To ensure fair comparison with baselines, specifically selected tasks that were employed in the work of Kojima et [33]. In the experiment on reasoning tasks, we enable RCI loop to get implicit feedback to correct outputs. We fix the maximum number of loops to 2. Following previous works [38] 62 [4], we use the correct label to decide when to stop the RCI loop. In our
setting, consider the correct label to be another source of feedback (label feedback).
C.3 MiniWoB++ task selection
In order to ensure a fair and comprehensive evaluation, a subset of MiniWoB++ tasks we use in the evaluation is selected from the evaluation of WebN-T5-3B (24), the most recent work on MiniWoB++ tasks, which employs LLMs. However, certain tasks such as book-flight, choose-date-easy, choose- date-medium, choose-date, and click-pie have been excluded from our evaluation due to their HTML code exceeding the maximum context length language models. On the other hand, some of the challenging tasks such terminal and simple-algebra have been included in the evaluation. The choice of these tasks is determined by the suboptimal performance CC-Ner [30], which currently represents the state-of-the- the potential of leveraging LLMs in computer tasks, in contrast to the conventional approaches of Supervised Learning (SL) and Reinforcement Learning (RL). While agent has not been evaluated on tasks that necessitate additional actions, such as drag and copy & paste, we posit that their inclusion can be readily achieved through the expansion of the actions space specification within the prompts.
model in the field. The purpose of this inclusion is to showcase
C.4 MiniWoB++ task selection for ablation studies
In ablation studies, we categorize the tasks based on the success rate achieved by our agent with gpt-3.5-turbo. We select subset of tasks from three levels of difficulty, as depicted in Table
19

---

click shape click widget
click-shupe
0.98
easy (0.9, 1]
enter-date
0.96
click-chockkooosssott click-collapsible-2
medium [0.6, 0.9)
0.62
click tah
click-tab-2-hard
0.56
hard 0.6)
count-shape
0.4
guess-number
0.2
Table 4: The tasks used in the ablation study are classified according to their level of difficulty.
C.5 Modifications on MiniWoB++ tasks
In T we outline several modifications that were incorporated into the MiniWoB++ benchmark for the purpose of our evaluation with language models that have limited context length.
Tisks
Modifications
social-media-all
social-media
quantity media components ranging from three
social-men some
email inhox forward- nl turk email-inhox--fon fooo--forward-ll cmail-inbox-nl-turk
The quantity of randomly generated emails has been restricted to range of three to six.
Table 5: Modifications on MiniWoB++ tasks.
20

---

D Prompts for MiniWoB++ tasks
We have autonomous computer control agent that can perform a of instructions control computers.
First, given the instruction that matches the regular expression. <type regex>, can type list of characters via the keyboard. This instruction should specify the target keyboard input for the agent to type. Before this typing instruction, you should first locate the cursor by clicking the input box with the click instruction.
Second, given the instruction that matches the regular expression, <press regex>, can press specific key on the keyboard.
Third, given the instruction that matches the regular expression, <clickoption regex>, it can click an option HTML element in a list with XPath that is visible on the webpage. The target of this instruction should valid XPath.
Fourth, given the instruction that matches the regular expression, <movemouse regex>, it can move the mouse cursor on an HTML element with an XPath that is visible on the webpage. Lastly, given the instruction that matches the regular expression, <clickxpath regex>, it can click an HTML element with an XPath that is visible on the webpage. The target of this instruction should be valid XPath.
Table 6: Agent specification.
<type regex> "ctype's. {1,}$"
arrowleft /arrowright arrowup arrowdown <press regex> = ppress)s(enteet wtroolefft
backspace)
<clickoption regex> = *rclickoptionns.(1..
<movemouse regex> 'mmoveouse'ss. {1,}$"
<clickxpath regex> c.lickxpathls.(1111
Listing 1: Regular expressions for specifying the admissible actions.
Based on the current plan and task, the next proper instruction should be
Table 7: Task grounding.
Considering the output the webpage, the specific instruction should be
Table 8: State grounding.
This action does not match the regular expressions. The updated instruction that matches one of the regular expressions is
Table Agent grounding.
21

---

E Examples of cross grounding
task:
click on the "ok" button on the webpage.
plan:
1. type "sit areu nullam:"
2. clickxpath //button| @data-wob wottta--bbreff"""
We have history of instructions that have been already executed by the autonomous agent so far.
No instruction has been executed yet.
Based on the current plan and the task, the first instruction should be "type "sit arcu nullam:"", Therefore, considering the output on the webpage, this instruction for solving the task should "clickxpath //button| @ "Sata-wobrref=""""
Table 10:
The state grounding optimizes the action plan. In the initial plan, the inclusion of the type instruction is deemed superfluous for accomplishing the task.
task: Find the email by Clarinda and reply to them with the text "Vel id at.".
plan:
1. clickxpath //divi cecass='email--endeer and text()"Clarinda')
2. clickxpath //span| @class== email-reply']
clickxpath //*[ @ (iid=""rrppll idd"reely-text"
4. type Vel id at. clickxpath l/*| iidd"sedd-eepll""
We have a history of instructions that have been already executed by the autonomous agent so far.
O: clickxpath //div[ @class=' @cllas='cmmall-seeeeesss email-sender' and text(="Clarindaa]
I: clickxpath //span| @class=' (clas=="email-rpplyy"") email-reply"]
2: clickxpath //*[ @idd""d=""" "idd="rrpyytext"]
Based on the current plan and the task, the next proper instruction after the history should "type Vel id at'. Therefore, considering the output the webpage, this instruction should be "type Vel id at."
Table 11: The state grounding corrects actions to grounded in the task. Although the target of typing action in the action plan is "Vel id at." the generated task uusk-grounded action lacked the last punctuation mark. Nonetheless, the state grounding update was successful remedying this discrepancy, resulting in the state-grae gaatu-gronned action being "Vel id at." Consequently, the state-grounded action now matches the typing action specified in the action plan.
22

---

Failure modes on MiniWoB++ tasks
Click builon in the dialog bux labaled "OK"
Select Kastey--aame--Csse
How many ys are there?
Kasey
X
Dulci
7
Vel, Fells trietique.
Cancel
OK
y
3
8
9
7
(a) click-dialog-2
(b) elick-menu
(c) count-shape
Figure 7: Screenshots.
F.1 click-dialog-2
The state-groud haat-ggounning process is occasionally inadequate for maintaining task soos-goonnding of actions. Specifically, in the click-dialog-2 task, grounding actions in the current state results in iask-irrelevant actions. Consequently, we opt to exclude the state grounding step for this particular task.
task: click the button in the dialog box labeled "x" plan:
1. Clickxpath //div[ @ewe@ecalui-deeeee didiioog-iiiallottttlbbbrr ui-corner-all ui-widget-header ui-helper-clearfix ui-draggable-handle |/button[ @eitle="Closse
We have history of instructions that have been already executed by the autonomous agent so far.
No instruction has been executed yet.
Based on the current plan and task, the first instruction should be "clickxpath //div[ doclass='uiddiaogge handle") ]/button[ title= Close'T. Therefore, considering the output on the webpage, this instruction should be clickxpath //div[ bialasss"--dillo--bbee click the button the dialog box labeled "x".
ui-corner-all hi-wieget-heaaer ui-helper-clearfix ui-draggable-
|/button[ @isoon[[@till @title='OKTT to
Table 12:
Example trajectory on click 2ialo-.. The state grounding updates the task uisk-grounded actions to be irrelevant to the task.
F22 click-menu
We find that including explanatory trajectories in the few-shot examples can significantly reduce the number of required few-shot examples. Specifically, in the click-menu task, the selection of the target component necessitates the mouse hovering over the expandable menu. Therefore, we provided explanations regarding the mouse hovering technique in the few-shot examples.
F33 count-shape
This task necessitates an agent to visually perceive rendered images on canvas HTML element and enumerate the relevant shapes. Unfortunately, our agent encounters difficulties in tackling such tasks which entail the need for perceiving visual representations, as solely depends on HTMI. observation.
23

---

task:
select Leonelle Jane > Amabel from the HTML code provided.
plan:
1. movemouse [/acs//e// // we need hover the mouse to expand the menu
2. movemouse
// we need to hover the mouse to expand the menu 3. clickxpath //
task: select the option "Joye Phylis" from the HTML code provided. plan:
1. movemouse //M[ext((=""ooeee" // we need to hover the mouse expand the menu
2. clickxpath (Maxx)"ahh//""exxxx""""hhhi'""
Table 13: Example trajectory on click-menu. Providing explanatory trajectory enhances the sample-
efficiency of few-shot examples.
task: answer the question "How many large items are there?".
plan:
1. Clickxpath //buttoon|(eexx(="""]
Table 14:
Example trajectory on count-shape. The agent struggle to solve tasks that requires visual rendering of HTML.
Guees the number between 0-9 and press Submit, Use the feerbbrkk helnw right
Erieer the username "emile" and the possword "yC" into the text fields and login.
Playing as 'X. win çame of lic- tac-toe.
Enter item that starts with "Iran"
Tags:
Waiting
Your session expiring
Submit
Submit
Exit home page?
OK
Cancel
(a) guess-number
(b) login-user-popup
(c) tic-tac-toe
(d) use-autocomplete
Figure Screenshots.
F.4 guess-number
When considering decision-making that involves a long-term perspective, our agent encounters difficulties generating actions that are grounded in the current task. For instance, in this guess- number task, the plan generated by our agent involves the sequential typing of numbers from 0 to in order to solve the task. However, due to the high number of steps required (over 50), our agent occasionally produces actions that deviate from the original plan, ultimately resulting in irreparable failure.
F.5 login-user-popup
This task involves the identification of an optimal strategy for an agent to handle an unpredictable pop-up window that emerges during a task. Due to the unexpected nature of the pop-up window, pre-planning the closure of the window is not feasible. Our agent designed to adapt the agent's actions to the current state, so it should generate appropriate instruction to close the pop-up window the state-grounding step. Nevertheless, there are instances where it unsuccessful executing the pop-up window closure, leading to task failure.

---

task: Guess the number between 0-9 and press Submit. Use the feedback below find the right number.
plan:
1. clickxpath [/"[@i==ttt"]
type
3. clickxpath //*| "eiddddddbbb"")
4. clickxpath //*1 @id="t"]
5. press backspace type
7. clickxpath /*[bid="subbtn") clickxpath //*
9. press backspace
10. type 2
11. clickxpath /
12. clickxpath //*[ [@id="t""]]]
13. press backspace
14. type
15. clickxpath [#[[i=='subbun]
16. clickxpath //*1 @ @id="tt"]
17. press backspace
18. type 4
19. clickxpath //"[@id="ubbbun)
20. clickxpath ["[@d="tt"]
21. press backspace
22. type 5
23. clickxpath 1/*[@/==="" (id==""d=bbbb")
24. clickxpath //*[ @ @id="tt"]
press backspace
26.
We have history of instructions that have been already executed by the autonomous agent far.
0: clickxpath //* [/[@itt"ttt]
1: type
2: clickxpath //[@id="sbbbun]
3: press backspace
4: type
5: clickxpath //*[ @id="subbtn"]
Based on the current plan and the task, the next proper instruction should be "type 2".
Table 15: Example trajectory on guess-number The agent fails generate task-grounded action when long- long-horizon decision making is required.
F.6 tic-tac-toe
We also examine the causes of the poor success in playing tic-tac-toe. Two distinct factors are identified responsible for its failure. The first factor is that our agent is unable to adopt a defensive strategy when its opponent is only one move away from victory and there is immediate opportunity to win the game. The second factor relates to the inability to consider the possibility of its attack being blocked by the opponent. A case point is the bottom-left illustration in Figure where our agent's is fourth move is ineffective since its diagonal direction is already obstructed by the opponent.
F.7 use-autocomplete
In use-autocomplete task, our agent demonstrated an ability select words beginning with specific characters. However, it struggles when it comes to selecting words that ended with particular
25

---

1
3
5
2
6
2
4
2
1
3
LASP
Opponent
4
4
3
5
4
6
2
3
2
1
6
2
5
6
1
6
7
3
7
1
4
5
3
5
4
3
4
Figure 9: Failure examples of tic-tac-toe task.
characters. This difficulty arose from the need to identify the correct word from the autocomplete list, a process that required the agent press the down arrow button the exact number of times needed locate the desired word within the list.
task: Enter an item that starts with "Se" and ends with "ia".
plan:
1. click the input field (e.g., clickxpath //*[@id='tags")
2. type the starting word (e.g.. type Se)
press the down arrow key to select the word ends with "ia" (e.g.. press arrowdown)
5. select the word (e.g., press enter)
click the submit button (e.g., clickxpath (aiiatt//dddddidbbbb"""
Table 16: Example trajectory on use-autocomplete. The agent fails identify how many times has to press the down-arrow key.
26

---

G
Additional results
Arithmetic
Common Sense
GSM8K
MultiArith
AddSub
SVAMP
Singletig
AQuA
CommonSenseeQ
Stratlggy
Zero-Shot Zero-Shot RC1
77.95
80.70
72.83
86.61
83.07
79.13
83,07
59.84
46.06
48.81
Zero-Shot Col
n/a
Zeno Shat CoT RCT
74.40
87.79
83.07
79.13
83.46
n/a
n/a
n/u
Few-Shot CoT Few Shot CoT + RCI
80.31
98.82
89.37
83,46
91.73
a/a
n/a
n/a
71.65
93.70
83.46
78.74
83.46
m/a
n/a
n/a
Table 17: In the absence of external feedback, RCI prompting on reasoning benchmarks exhibits performance equivalent to, or below that of zero-shot approach.
27

---

compartion with LLM SotA
0.75
0.50
Residual
0.25
0.00
-0.8 -
-050
-0.75
1.00
bor pedlent und
you
Comparison with SL SotA
077
0.50-
Residual
0:25
0.00
0.50
0.76
1.00
del suiien Suttiii deluonn
Comparison with RL SotA
100
030
Residual
0.25
0.00
-0
-075
m
Comparison with SL + RL SotA
100
077
0.50-
Residual
0.00
-025
0.75
1000
the
the uoe
the
Figure 10: The task- level performance comparison with the ofate-of-the-a (SotA) baselines. The values, M y-axis represents the residual values, which ahiihhrre are from our agent's performance. mnrormancc..
are obtained by subtracting the performance of SotA
28

---

TASK
Ours
Ours owf
WebN- 15-38
CC-Ne:
OCNNer
CC-Net (SL)
Others (SL
SarA
SotA
SarA
GPFT-)
ISL
(RL)
(SL.)
(KL)
(SL
RE.
bisect angle
1/a
0.97
1.00
0.29
0.80
0.29
1.00
0.97
book-flight
IV8
1/a
chase circle
W/
0.95
0.93
0.80
1.00
0.50
0.93
1.00
choose date-cwy
1V6
0.03
0.99
0.99
0.42
0042
0.99
0.99
choose cate- date-medium
0.26
choose-list
1.00
1,00
0.26
0.99
0.99
0.19
0.26
10.66
0..99
0.99
circle center
026
D/a
0.97
1,00
0.36
0.98
10.56
1,00
0.95
click bution sequence
1.00
1000
1,00
0.47
1,00
1,00
0.78
1,00
1,00
click checkboses large
0.54
0.94
0.22
0.71
0.000
0.00
0.84
1.22
01.00
0.71
click checkboses soft
0.72
0.95
0.54
0.95
0.12
0.04
0.94
10.54
0.12
0.95
click--cha sedk-cchckhone--tnnee
0.36
checkboses
0.96
clict-oolapsible-2
0.62
1.00
0.00
0.9%
0.38
D.17
0.99
11.17
0.888
0.98
click-coliee dickk-oollassiee
1.00
1,00
0.00
1,00
1,00
0.81
1.00
10.11
1,00
1.00
click color
0.82
0,82
1,00
click-dinlog-2
1.00
0.88
0.88
1.00
click-dinlog
1.00
1.00
1.00
1.00
1.00
0.95
1.00
1:00
1.00
1.00
click link
100
1,00
1000
0.99
0.94
0.59
1,00
1.00
1.00
1.00
click-menu-2
0-48
click-opiinn
100
1.00
0.87
0.99
0.78
0.21
1.00
1187
1/00
1.00
click-pic
0.51
0.97
0.92
0.15
1.00
12.11
1.00
0.97
click-scrolllistt
0.50
004
0/03
1.00
click-shage
098
0.98
0.53
0.95
0.50
0.11
0.64
1154
0.50
0.95
click- tnb 2-ensy
IVa
0.99
0.94
0.61
nra
1221
01.94
0.99
click-taih--iand
0.19
0.98
click-tah---meediu
I/a
1254
0.996
click-tab-2
0.74
1.00
0.18
0.98
0.91
0.27
1.00
0.27
1.00
0.98
click-talh
1.00
1,00
0.74
1,00
1.00
0.95
1.00
1.00
1,00
1.00
clicli-test-22
1.00
1.00
click-test-trmsserr
n/a
1.00
0.94
nia
0.4
1,00
1.00
click-teest
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
click--widget
058
0.98
1000
1,00
1.00
0.56
1.00
1.00
1.00
1.00
copy- copy-paste-2
cope-paste
0014
0.79
couné-shige shige
0.40
0.40
0.41
0.85
0.70
0.21
0.76
0.43
0.70
0.85
count-sidex
nia
1.00
1,00
0.74
0.30
1.74
1,00
1.00
drag-hot
1.00
drag-cube
n/a
0.79
0.95
0.23
0.23
0.95
0.79
drag-item
n/3
n/a
nda
1.00
0.00
0.61
n.61
0.00
1.00
drag-itens-grid
AV6
nta
0.98
0.00
0
0:01
DAS
01.00
0.95
drag-shapes
I/a
n/a
0.26
0.23
0.99
drag sort numbers
1/5
n/a
maa
0.97
0.00
0.11
0.66
0.11
0.00
0.97
email-inhar rnbbx-delet
n/a
1.00
1,00
0.22
1,00
0.22
1,00
1,00
1.00
email- inbux: forward-nl
1.00
1.00
0.60
1.00
0.00
0.00
m/a
0.60
0.00
1.00
email-inke Seaiiil-ibu----ee furward
m/a
1.00
0.00
0.01
m/a
0.01
0.00
1.00
email-inbax--iie
nta
1.00
1.00
0.30
nia
0.50
1,00
1.00
email- nosemoll
1/6
n/a
n/a
0.13
mia
0.13
0.48
1.00
mnix---ibbx--eppll
0/6
n/a
1.00
0.00
0.00
0.00
0.00
1.00
email- inbux
m/a
0.47
0.11
m/a
0.11
0.47
1,00
-unil--ibbxx
1.00
enter date
0.96
0.96
0.00
L.00
1,00
0.02
1.00
0.02
1,00
1.00
enter passwurd
1.00
1.00
0.97
1.00
0.01
0.02
1.00
0.97
1.00
1.00
entor-test-2
m/a
0.98
0.00
0.04
0.00
0.04
0.00
0.96
dynamic
enlor-texl
1.00
0.89
1,00
0.35
1.00
0.99
1,00
entee-time
1.00
1.00
0.00
0.97
0.80
0.04
0:90
0.04
0.89
0.97
find midpoint
1/a
0.97
0.35
0.35
0.97
0.97
word
docus-teas 1ext-2
1.00
1,00
1.00
L.00
1,00
0.96
1.00
1,00
1,00
1.00
focus
1.00
1.00
1.00
1.00
1,00
0.99
1.00
1.00
1,00
1.00
grid-ccomlimiie
1.00
1,00
0.49
1.00
0.00
0.66
1,00
0.66
0.02
1.00
guess
highlightt text-2
m/a
m/a
1.00
0.34
0.40
0.13
0.60
0.34
1.00
highlight-tunt
m/a
n/a
1.00
1,00
0.51
0.90
0.51
1.00
1.00
shape
0.76
1,00
0.68
0.89
1.00
usee popup
login user
1.00
1.00
0.82
1.00
0.00
0.00
1.00
0.82
1.00
1.00
moving stems
4/2
n/a
0.88
0.69
0.13
0.78
0.13
0..99
0.88
mulli layouts
0.72
0.95
0.83
1.00
0.00
0.00
1.00
0.83
0.00
1.00
orderings
1.00
navigate trce
0.36
1.00
0.91
0.99
0.94
0.32
1.00
0.99
1,00
0.99
number numberchrekboose checkboses
1/5
m/a
m/a
0.99
0.00
0.00
0.16
0,00
0.00
0.90
read cable-2
0,00
B/a
textaee
3/6
1/6
1.00
0.68
0.27
0.11
0.27
01.68
1.00
right angle
1/2
B/a
0.98
0.9%
0.26
0.38
1026
01.98
0.98
scroll text
1.00
0.88
0.88
1.00
scroll--ce
search engine
1.00
1.00
0.34
1.00
0.01
0.15
1.00
0.34
1.00
1,00
sioon
1/2
m/a
0.00
0.00
0.02
0.28
0.02
0.00
0.28
29

---

algebra
simpl-arithheric
16
0.38
9338
0.86
soeial-media
1.0
1.00
010
0.35
000
0000
1:00
0.75
sociali media somne
0.9
0.96
000
0.85
0,22
100
1222
1881
0.85
sccial-media
text ealto
0.98
0.00
0.11
0.01
1111
0.00
0.98
lext-tmmnform
0.50
0.80
0.60
0.10
0.19
6.00
1015
0.00
0.50
unioode-test
105
use-aulic amplee
0.58
0.58
0.22
1.00
1.00
0.07
0.98
10.22
1.00
1.00
Use 1se-colarhheel22
DV6
D/a
03355
0.35
0.38
1,00
10.58
11.85
1.00
use-oolerwhel
use-dlider-2
IV8
use sllider
2V6
I/6
0.91
0.47
0.18
0..1
0.18
0.44
0.91
Use-spiencr
0.58
0.96
0.07
1,00
0.07
0.47
0.17
1047
0.402
1.00
visual-addttio
Table 18: Comprehensive task-level success evaluation of baseline models in MiniWoB++ tasks. Ours (w/ GPT-4) depicts the performance outcomes obtained through the use of the GPT-4 model for some tasks, which are visually highlighted in the color blue. The performance of baseline models has been sourced from prior studies (3 24]. The average success rates of the tasks highlighted with violet color are shown in Figure The state-of-the-ar (SotA) supervised learning (SL) is represented by the works of [30
24 while the SotA in reinforcement learning (RL) includes the studies of 30 25 32.. Furthermore, the SotA in the combined application of SL and RL consists of the contributions of 30 61 [6]. Combined result of models proposed prior to CC-Net [30 is denoted as Others, which include [6]
32.. This corresponds to Aggregated SotA (Augmented) baseline in previous works 30 We generously estimate the performance of CC-Net (RL) based on their figures.
30


Reviews:
[{"review_id": "so59TqYk6N", "reviewer": "Reviewer_kDG8", "summary": "This paper proposes an algorithm for executing tasks using LLMs. While this can in principle be applied to any sequential task with language input and output spaces, they tested primarily on computer control tasks.\n\nThe method, RCI, works as follows:\n- The environment state, action space, and task are represented in language (e.g. HTML code for the state).\n- Based on the task description, the LLM generates a high-level plan of how to solve the task.\n- The LLM writes a critique of its plan and improves it. This can happen multiple times.\n- Conditioned on the plan and the current state, the LLM is asked to find the webpage elements necessary to execute the plan.\n- Based on the above, ask the LLM to generate an executable action.\n\nThey show that on a large subset of Miniwob++ tasks, this method is able to outperform past LLM SOTA and is able to match past IL+RL SOTA without needing any training. They also show improvements over other prompting methods on various math and reasoning tasks.", "questions": "**Major questions**\n* The description of Figure 4a doesn't seem to quite match the numbers in the table. \"our agent surpasses SotA in supervised learning by 34% and in reinforcement learning by 24%\".  (I'm guessing you meant 37 and 27. Also, please be clear whether you're talking about *percent improvement* or *percentage point improvement.*). Also, the text says \"our agent underperforms in comparison to SotA in the combination of supervised and reinforcement learning...\" but the graph currently shows they match. Which is correct?\n\n**More minor**\n* Nit: Typo L196 \"wich\"\n* Nit: L322 \"exist\" -> \"exists\"\n* Explain more clearly what \"dictionary-based typing actions\" are.\n* In Figure 4a, it would be helpful if the caption stated which models are the \"SL SotA\", \"RL SotA\", and \"SL + RL SotA\". Currently it isn't clear.\n", "limitations": "Limitations are discussed, although additional limitations (such as lack of history, limited context window, and inability to improve through accumulated agent experience) are not discussed much.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method gives a substantial performance boost compared to all Miniwob++ baselines (except one which uses far more demonstration data). It also shows consistent improvements in math and reasoning tasks. This is useful since it suggests that for many computer control tasks collecting lots of demo or RL data is unnecessary.\n- Each component of the pipeline includes intuition for why it's necessary + ablations showing that performance drops significantly without it.\n- It's hard to judge novelty given the recent explosion in works prompting LLMs for tasks with various forms of self-reflection. I'm inclined to say it's all concurrent work.", "weaknesses": "- See the \"Questions\" section for discussion on figure/discussion mismatch.\n- This method has limitations, including (a) it is hard to solve tasks where the HTML description exceeds the context width and (b) the agent is currently unable to handle tasks which involve dragging or other complex UI manipulation. Currently, these tasks are excluded from the set of evaluation tasks. On the other hand, tasks where the prior SotA did poorly were stated to be intentionally included. This seems like a bit of an unfair comparison.  (Though I think many of these problems could be addressed by modifying the action space and getting a longer-context LM). Even if the additional tasks are not reported in the main results I think it should at least be more prominently emphasized that tasks where RCI has insufficient context were modified or excluded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' valuable feedback on our paper. We have carefully considered the comments and suggestions and would like to address each of the concerns raised.\n\n1. Task selection:\nWe understand the concern about task selection for our evaluation. We followed the WebN-T5-3B's [1] task set selection as it was the most recent work at the time and the first to use LLMs for solving computer tasks. We acknowledge that this approach might have resulted in a limited comparison. To address this, we will explicitly mention in the paper that the task set was selected to provide a direct comparison with the most relevant previous work at the time. We also acknowledge the importance of considering other relevant tasks and will make this point clear in the revised manuscript.\n\n2. Context length limitation for lengthy HTML state:\nWe agree that HTML descriptions exceeding the context length pose a challenge. While we used the original HTML code, we recognize the possibility of reducing the HTML state size by removing irrelevant code, such as CSS and JS code. This optimization could improve the model's ability to handle tasks with lengthy HTML effectively. We will incorporate this suggestion into the discussion section of the revised paper.\n\n3. Limited action space:\nThe reviewer correctly pointed out that the action space definition is limited. We acknowledge the importance of a more extensive action space to support diverse computer tasks. However, as the reviewer already mentioned, we expect that it can be achieved by adding more feasible actions to the prompt. This modification should enhance the model's capability to handle tasks involving dragging or other complex UI manipulations.\n\n4. Mismatch between figures and explanation:\nWe apologize for the oversight in the description of Figure 4a. The discrepancy in the numbers appeared in the figure and the text will be rectified in the revised version. \n\nFurthermore, we will address the minor nitpicks mentioned, including the typos in Lines 196 and 322, as well as provide a clearer explanation of \"dictionary-based typing actions.\"\n\nLastly, we appreciate the reviewer pointing out the limitations not fully discussed in the paper. Our agent keeps the history of actions in the prompt, but we will make sure that other limitations such as limited context window, and inability to improve through accumulated agent experience will be properly discussed.\n\nIn conclusion, we thank the reviewers for their insightful comments, which have helped us improve the quality of our work.\n\n[1] UNDERSTANDING HTML WITH LARGE LANGUAGE MODELS"}, {"review_id": "7AKc1GZd0f", "reviewer": "Reviewer_6D7X", "summary": "This paper proposes a prompting approach that utilizes a critic to improve the web navigation performance. Within the same context, the model reviews a candidate answer and improves it based on reasoning steps. For web navigation, there are two phases: explicit and implicit RCI. Explicit RCI takes a task and generates a plan with step-by-step actions. This plan is fed to the implicit RCI to ground on the state and the agent. Resulting action is executed in the environment to retrieve a new state. Compared to previous SL and RL approach, RCI achieves competitive to the SoTA while using only few-shot prompting.", "questions": "1. How do you compare to more recent reasoning approaches?\n\n2. Could RLHF training be using some web navigation tasks or even MiniWoB?", "limitations": "The paper discusses some of the limitations but more detailed discussion on generalization to real websites would be helpful.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel prompting approach with planning and iterative improvement. It performs favorably to CoT prompting and outperforms previous finetuning approaches while using significantly less data. It also shows the general capabilities of LLMs in decision making tasks and illustrates limits of the MiniWoB benchmark.", "weaknesses": "Some questions regarding better baselines and RLHF performance.\n\n- More detailed comparison with recent approaches such as ReACT or InnerMonologue is missing. What is the main novelty of RCI and why would it be a better candidate for computer tasks compared to ReACT which also studies WebShop benchmark tasks.\n\n- Instruction tuning and RLHF are crucial for achieving good results, otherwise the success rate drops to 50%. Given that training data for RLHF training is undisclosed, I wonder if similar tasks (or even MiniWoB data) are used during training. Is there any empirical evidence for the case of InstructGPT-3+RLHF model?\n\n- WebN-T5 doesn't use in-context examples. It uses the history of actions within the same episode. (referring to Figure-4 in your paper)\n\n- WebGUM and WebN-T5 also don't use dictionary based typing actions. WebGUM is missing from comparison.\n\n- Please clarify Line 33 as there are previous models using LLMs that you also compare.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for their valuable feedback, which has significantly contributed to identifying areas for improvement in our paper. Below, we address the points raised to enhance the clarity of our content.\n\n1. Comparison with ReACT and InnerMonologue on computer tasks\nRegarding the comparison with ReACT and InnerMonologue on computer tasks, it is important to note that Webshop has a more limited action space, defining a single task, which only involves shopping on the internet. We consider it analogous to the tasks implemented in the MiniWoB++ benchmark.\nDirect feedback from LLMs, which includes information about invalid actions that cannot be executed by the agent or is invalid in the current state, proves to be highly beneficial in the RCI process. Critics, with their explicit explanation, offer valuable guidance for improvement, making the RCI approach advantageous compared to mere reasoning without any clear direction for enhancement. While ReACT can be integrated into the RCI process, we firmly believe that critics play a crucial role in grounding actions for the state, task, and agent, leading to more effective outcomes.\nReflexion, a more similar work to ours, builds upon ReACT and employs reasoning steps to enhance task and state grounding of action generations. However, it relies on external feedback, indicating a continued need for a reward function. In contrast, the RCI agent does not depend on any external reward function.\nWe would also like to note that a more comprehensive comparison with related works, including ReACT and InnerMonologue, has been provided in the related works section of the appendix.\n\n\n2. Empirical evidence for InstructGPT-3+RLHF model training data\nThe reviewer inquired about empirical evidence that the InstructGPT-3+RLHF model is not trained with similar tasks. While we do not have specific evidence regarding the exact training data, we believe it is unlikely that expert demonstration data from MiniWoB++ was used for the training of InstructGPT-3+RLHF. The use of HTML code for training is likely, but we believe that extensive demonstration data specific to MiniWoB++ tasks is improbable because to our knowledge the RLHF was done in the context of a chatbot, not a computer agent. We do not know of experiments that can be run to determine if InstructGPT-3+RLHF used MiniWoB demonstrations, but are open to suggestions. Unfortunately, we will not definitively know the answer and we acknowledge the downsides of doing academic research on closed-source LLMs. In our revised manuscript we will include discussion on this topic.\n\n\n3. Clarification\nWe appreciate the reviewer pointing out the confusion in Line 33 regarding the best-performing approach. Indeed, CC-Net, which does not use LLMs, was the best-performing model in our comparisons. We apologize for the oversight and will make the necessary corrections in the revised version. Furthermore, we will accurately represent the usage of dictionary-based actions in the models for a more coherent comparison.\n\n4. Comparison with WebGUM\nAs the WebGUM paper was published after the deadline of our paper, we weren’t able to include it. We will cite this paper as contemporaneous work in the camera-ready version, but we note that WebGUM performs worse than our approach. \n\n5. Comparison to more recent reasoning approaches\nIn the camera-ready version of the paper we can include comparisons with newer reasoning approaches such as Tree of Thoughts and Self-Notes, but since these works came out after we submitted the paper we do not include these comparisons in the current version. \n\n6. Does RLHF training use web navigation?\nAs discussed above, we will include qualifications and limitations that we do not know for certain how these models are trained, which is a major drawback of doing academic research on closed-source LLMs. \n\n7. Limitations: Generalization to real websites\nThe reviewer also raised a concern regarding the generalization of this work to real websites. We acknowledge that there are certain limitations, such as the restricted context length and limited action space, which may hinder the direct application of the RCI agent to real websites. Additionally, the presence of complex UI components that cannot be fully understood by HTML code might also impede further generalization. Nevertheless, it is important to note that our approach is intended to be viewed as complementary and independent from these orthogonal challenges that are left for future work.\n"}, {"review_id": "7CWkx1HtId", "reviewer": "Reviewer_qKku", "summary": "The paper presents a new method to enable LLMs to execute computer tasks guided by natural language. The method uses a simple prompting scheme called Recursive Criticism and Improvement (RCI), which prompts the LLMs to find and fix problems in their output. Without large amounts of expert demonstrations and task-specific reward functions, RCI improves the performance of LLMs on the MiniWoB++ benchmark, a web-based simulation environment that offers a diverse range of computer tasks. The paper also demonstrates that RCI enhances the reasoning abilities of LLMs on a suite of natural language reasoning tasks. The paper claims that RCI is a significant contribution to the development of intelligent agents that can solve any computer task by communicating via natural language.", "questions": "1. The model name “InstructGPT-3 + RLHF” in the experiment part of the main paper is quite confusing since there are 3 different models named after it in the appendix. Also, InstructGPT is also tuned with RLHF. I suggest replacing them with corresponding API names.\n2. Why are benchmark AQuA, CommonSenseQA, and StrategyQA evaluated on the zero-shot setting but not evaluated on the few-shot setting?\n3.“Specifically, we employ three models, namely, GPT-3 (davinci), InstructGPT-3 (text-davinci-002) , and InstructGPT-3 + RLHF (text-davinci-003, gpt-3.5-turbo, gpt-4).” Which InstructGPT-3 + RLHF model is used in section 3.2?\n4. Line 246 – 252: “Our findings reveal that our agent surpasses SotA in supervised learning by 34% and in reinforcement learning by 24%. Notably, our proposed RCI prompting method outperforms the SotA LLM approach, even when the latter employs both finetuning and few-shot examples in prompts. This outcome highlights the effectiveness of our approach in extracting vital knowledge for computer tasks from language models. However, our agent underperforms in comparison to SotA in the combination of supervised and reinforcement learning, which employs significantly more expert data and online interactions.” This result is inconsistent with Figure 4 (a). I guess there is a typo in Figure 4 (a) where “Ours” should be 0.91 instead of 0.94.", "limitations": "The authors have not discussed their limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is original and insightful in tackling the grounding problem of LLMs by introducing RCI which uses the self-criticism and self-improvement abilities of LLMs to finish computer tasks and reasoning tasks. \n2. It seems that RCI is general and flexible enough to be integrated with other techniques like CoT.\n3. The paper applies RCI to a diverse range of web-based computer tasks and natural language reasoning tasks and shows promising results.\n4. The paper is well-written and clear in presenting the main ideas, contributions, and results. ", "weaknesses": "1. The impact of the RCI iteration steps should be presented. Will RCI iteration always increase the performance?\n2. There is no explicit criterion for terminating RCI, and it relies on multiple attempts on the dataset. However, this kind of trial may not be feasible for new tasks in the real world. A simple approach might be to iterate as much as possible, but this would increase computational and time costs significantly.\n3. Although RCI performs well in various reasoning tasks, considering the wide variety of models available now, it is recommended that the author conduct tests on several different models to demonstrate the effectiveness of RCI. For example, zero-shot CoT [1] has measured 17 different models in total. Another example is that few-hot CoT [2] has also measured models with different sizes (see their Table 4). Also, newly proposed models like Vicuna should be measured if possible.\n\n[1] Large Language Models are Zero-Shot Reasoners\n\n[2] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and valuable feedback. We appreciate the reviewer's careful examination of our submission. Below is our rebuttal addressing the raised concerns:\n\n1. Impact of RCI iteration steps\nWe agree that the impact of RCI iteration steps should have been explicitly presented in the paper. While it is true that RCI iteration does not always increase performance because of false negative critics, we found that if we properly set the maximum iteration number, it didn’t cause a notable performance drop. We also acknowledge that the lack of an explicit criterion for terminating RCI is not well specified. In our evaluation, we could find that empirically stopping the interaction after 2 showed the best performance, but more heuristics and advanced methods can further improve this process.\n\n2. Evaluation on multiple models \nAlthough we didn’t have enough budget to run on multiple models for this submission, this is an excellent suggestion, and we will try to include additional models for the camera-ready version. \n\n3. Confusing model naming \nWe apologize for the confusion caused by the naming of the models in the experiment section. In the revised paper, we will make clear which API is used for each section of the evaluations to avoid any ambiguity.\n\n4. Benchmarks for few-shot setting \nThe reason we didn’t evaluate AQuA, CommonSenseQA, and StrategyQA on the few-shot setting was that the previous works on the few-shot setting were not evaluated on the tasks. Since the few-shot approach highly relies on the few-shot examples, it was hard to set up a fair experiment setting without those data.\n\n5. InstructGPT-3 + RLHF model used in section 3.2\nWe apologize for the oversight. In section 3.2, we used the \"gpt-3.5-turbo\" variant of the InstructGPT-3 + RLHF model for the experiments. We will update the paper to clarify this.\n\n6. Inconsistency between text and Figure 4(a)\nWe thank the reviewer for properly pointing out the mismatch between the text and figure 4(a). There seems to be an obsolete explanation for Figure 4(a) regarding the performance comparison. We apologize for this mistake. In the revised version, we will correct the text to be aligned with the figure.\n\n7. Limitations:\nThank you for bringing this to our attention. We acknowledge that our initial discussion of limitations may have been inadequate, and we appreciate the opportunity to address this concern in our revised version. In the revised version, we will address the following main limitations of our work:\n\n* Limited Context Length: Due to the maximum context length of LLMs, lengthy HTML states hinder the use of LLMs for web navigation.\n* Limited Actions Space: Currently, our model only supports click and keyboard typing actions. \n* Lack of Long-Term Strategies: Our model focuses on short-term decisions and lacks long-term strategic planning capabilities.\n\nOnce again, we appreciate the reviewer's thorough evaluation and constructive feedback. We will address all the concerns raised in the paper's revised version to strengthen the quality and clarity of our work.\n"}, {"review_id": "nB9bGiMuAq", "reviewer": "Reviewer_s5S9", "summary": "The paper introduces a novel approach, Recursively Criticizes and Improves (RCI), for guiding large language models (LLMs) to execute computer tasks using natural language commands. The RCI method outperforms existing methods and sets a new state-of-the-art on the MiniWoB++ benchmark when combined with the InstructGPT-3+RLHF LLM. The authors also demonstrate that RCI enhances LLMs' reasoning abilities on natural language reasoning tasks, and performs better when combined with the chain of thought (CoT) prompting.", "questions": "I am wondering how RCI performs compared with other LLMs-based methods that leverages environment feedback and self-evaluation.\n", "limitations": "The authors include the limitations in the model failure part.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is overall well-written and easy to follow.\n2. The paper introduces a new method, Recursively Criticizes and Improves (RCI), which guides large language models (LLMs) to execute computer tasks using natural language commands. The idea of explicit and implicit RCI is interesting.\n3. Abundant experiments have shown that RCI not only shows great reasoning ability on different tasks, but also achieves same performance with fully-supervised methods on real-world computer tasks.", "weaknesses": "There are a lot of works that leverage environmental feedback or model self-reflection to improve the plans, like self-debugging, reflexion, self-refine, etc. The authors have briefly introduced some of them in the appendix, but a detailed comparison of these methods is still needed. The baselines included in the reasoning part of experiments are not enough as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback on our submission. We appreciate the time you took to review our work. We have carefully considered your comments and have made improvements to address the weaknesses and address your questions and concerns.\n\n1. More comparison with other methods (e.g. self-debugging, reflexion, self-refine) that leverage environmental feedback \n\nSelf-debugging, Reflexion, and Self-refine require external feedback from the environment to enhance their outputs. However, this implies that in the domain of decision-making, a reward function is needed. One of the main contributions of the RCI agent is removing the reliance on the reward function for various computer control tasks. Although the core concept of using self-feedback is similar, the problem we aim to solve using self-feedback is distinct from these methods. Thank you for pointing out that this distinction between our method and these related works was not clear. We will be sure to make this point clearer in the camera-ready version of the paper. \n\n2. More baselines for reasoning tasks\n\nWe disagree that the baselines included in our reasoning experiments are not enough. We use two baselines, CoT and Zero-Shot CoT which were the SoTA at the time of submission on the reasoning benchmarks we used. We carefully argue that it is sufficient to show the effectiveness of RCI by performing better than SoTA on a subset of reasoning benchmarks. Are you aware of another approach that outperformed CoT and Zero-Shot CoT on these benchmarks at the time of submission? If so we can happily include comparisons for the camera-ready version of the paper. \n"}]

Author rebuttal:

Meta-review: This work proposes using a simple prompting scheme where an LLM agent recursively criticizes and improves its output (RCI). It is shown that RCI outperforms SL and RL on the MiniWoB++ benchmark. It is interesting and impressive to see that this best performing method requires no training.  

The authors addressed most of the concerns raised by the reviewers. After the rebuttal, the reviewers unanimously recommended (weak) accept.

Reviewers recognize the performance of the proposed method as well as the clear presentations and intuitions.

The only remaining concern is regarding the generalization ability of the model to other tasks (6D7X, qKku). We think that without detailed analysis on generalization ability, the work is still of great interest to the community. That said, please try to provide more discussions in the final version.
Decision: Accept (poster)