A Zero-Shot Language Agent for Computer Control with Structured Reflection
Authors: Tao Li
Gang Li
Zhiwei Deng
Bryan Wang
Yang Li
Venue: EMNLP 2023 Findings (2023)
Keywords: planning
reflection
action
grounding
TL;DR: We propose the first zero-shot language agent for computer control tasks using structured reflection.

Abstract: Large language models (LLMs) have shown increasing capacity at planning and executing a high-level goal in a live computer environment (e.g. MiniWoB++). To perform a task, recent works often require a model to learn from trace examples of the task via either supervised learning or few/many-shot prompting. Without these trace examples, it remains a challenge how an agent can autonomously learn and improve its control on a computer, which limits the ability of an agent to perform a new task. We approach this problem with a zero-shot agent that requires no given expert traces. Our agent plans for executable actions on a partially observed environment, and iteratively progresses a task by identifying and learning from its mistakes via self-reflection and structured thought management. On the easy tasks of MiniWoB++, we show that our zero-shot agent often outperforms recent SoTAs, with more efficient reasoning. For tasks with more complexity, our reflective agent performs on par with prior best models, even though previous works had the advantages of accessing expert traces or additional screen information.

A Zero-Shot Language Agent for Computer Control with Structured Reflection
Tao Li"
Gang Li' Zhiwei Deng" Bryan Wang Yang Li* 'Google Research, Mountain View, U.S.A.
Uniiersityy of Toronto, Ontario, Canada (elini,, zhiwei deng, iiiyngg@google.cooo bryanwedgp. toronto.
Abstract
several rounds of self-reflection can substantially improve task completion, e.g.. REFLEXION (Shinn et al., 2023), SELF-REFINE (Madaan et al., 2023). During the process, LLMs are prompted to update a prior execution plan according to feedback from the environment. Such updates become part of the prompt for the action generator in the next round. Recently, MINIWOB++ has been used as a testbed for LLM's capacity at modularized com- puter tasks. To learn task, a common approach is the use of extensive trace examples of the task for direct supervision (e.g., CC-Net (Humphreys et al., 2022), WebGUM (Furuta et al.. 2023)). self-supervision (Gur et al., 2023), or few/many- shot prompting (e.g., RCI (Kim et al., 2023), SYNAPSE (Zheng et al., 2023)). They have achieved more than 90% task completion rate on dozens of computer tasks, scemingly to have solved the computer control problem.
Large language models (LLMs) have shown increasing capacity at planning and executing a high-level goal live computer environ- ment (e.g. MINIWOB++). To perform task, recent works often require a model to learn from trace examples of the task via either pervised learning or few/many-shi prompt- ing. Without these trace examples, it remains a challenge how agent can autonomously learn and improve its control on a computer, which limits the ability of an agent perform a new task. We approach this problem with zero-shot agent that requires no given expert traces. Our agent plans for executable actions on a partially observed environment, and iter- atively progresses task by identifying and learning from its mistakes via self-reflection and structured thought management. On the easy tasks of MINIWOB++, we show that our zero-shot agent often outperforms recent So- TAs, with more efficient planning. For tasks with more complexity, our reflective agent per- forms par with prior best models, even though previous works had the advantages of accessing expert traces or additional screen in- formation.
However, the requirement of expert traces for learning to perform a task limits the agent's s ability on new tasks. Without using carefully selected traces as guidance, can agent autonomously learn and improve its control a computer? To address this question, we propose zero-shot agent. We build our agent on top of PaLM2 (Anil et al., 2023), recent LLM, and our agent employs a unified instruction prompt set across different tasks, without extensive tailoring for individual tasks.
1 Introduction
Prior works have shown promises in using large language models (LLMs) for action generation, e.g. SAYCAN (Brohan et al., 2023), REACT (Yao et al., 2023). TOOLFORMER (Schick et al., 2023), and SWIFTSAGE (Lin et al., 2023)) over variety of live environments (e.g. MINIWOB ++ (Shi et al., 2017: Liu al., 2018), ALFWORLD (Shridhar al., 2021), and ALPHACODE (Li et al., 2022). A shared approach is use LLMs to follow expert traces, comprehend environment changes, plan fu- ture actions, and execute an action by composing API calls; all in the form of text. Some works have shown that iteratively attempting task with
In addition, recent works, e.g., RCI (Kim et al., 2023), ADAPLANNER (Sun et al., 2023), and SYNAPSE (Zheng et al., 2023), utilize screen rep- resentation that may include much more informa- tion than what is presented to user on the screen. For instance, Fig. shows an example of elements that are not shown the screen yet present in the HTML that fed the LLM. Using such addi- tional information arbitrarily reduces the difficulty for the agent to perform the task. Yet in general use cases, such information might not be readily available, and relying on such information can po-
Work done during internship at Google Reseurch.

---

goal. These models have exhibit high capacity follow in-context traces to solve complex tasks.
Planning & Reflection REACT (Yao et al., 2023) used intermediate thoughts to guide long- chain of action planning in a live environment. Be- yond one trial of planning, REFLEXION (Shinn ct al., 2023) and SELF-REFINE (Madaan et al., 2023) recently found the ability of LLM to self- criticize and self-improve can iteratively learn and solve given task via multiple trials. Neverthe- less, recent planning works require extensive and customized trace prompts for the LLM agent to learn accurate planning. SWIFTSAGE (Lin et al.. 2023) reduces the extensive planning calls to LLM by training small and fast planning module to fa- cilite long-chain of actions. In similar motivation, our zero-shot agent is based efficient staged planner. We list a detailed comparison in Tab. 1.
11
Perter Tooe
thon
(a) Before
(b) After (c) HTML of before
sedurrs ogen sleun slart coreeeee doshiuee mean
comer pelpoluuee or aiter musm
welett prttiiigge dess egeito sed pore
1pst, Socuta wow Linseaal Atwllicanhhiee
Coe b-to Debie 1930
wate in cverre aapper van cheonta inatttatam (lasin crree Leme,
due
(d) Before
(c) HTMI. of before
Inconsistent screen representations. Fig. la-Ic: be- fore and after clicking "more" button the social-media task (seed=2). HTML before clicking already revealed the content. Fig. Id- le Similar issue in the click-tab-2 (seed=0).
Figure I:
tentially hamper the applicability of the agent. We manually examined 13 relatively challenging tasks on MINIWOB+ that are supposed to span multi- ple screens, and found 5 of them contained such cntematioonmulli-seeee
Reflection
Planning
Unsup
Trace
Efficient
Feedback
mem
RCI
sparse
ADAPLANNER REFLEXION
few
detailed
information in a single observation-in their HTML.
few
X
detailed
3
SWIFTSAGE
X
detailed
Our contributions follows: Firstly, we ploy compact screen representation that assumes much less information than what used by pre- vious works, thus resulting more general and realistic test environment. Secondly, we propose a simple yet efficient action planner that can accu- rately plan out exccutable actions on a state in one pass. With recent capacity of LLM, we show that such "naive" strategy can solve almost all the easy tasks the MINIWOB++ benchmark. For more challenging tasks, take inspiration from Reflex- ion (Shinn al., 2023) and propose a structured thought management strategy to facilitate reflec- tion, allowing the agent to effectively learn and im- prove from exploration failures. With few rounds of attempts, our agent achieves comparable perfor- mance with prior few/many-shot state-of-the-arr. To the best of our knowledge, agent is the first zero-shot design for computer control tasks"
Ours
sparse
N
Table 1: Comparison with prior work on reflective planning.
N: number of actions complete the task, Efficient X: quires planner call for each action.
Consistent
MINIWOB++
Unsup
Trace
Efficient
Screen
CC-NET
large
PIXZACT
large
RCI ADAPLANNER
few/many
X
x
few
v
x
Ours
Table 2: Comparison with prior work on MiniWoB, For
discussion on consistent screen, please refer to Sec. 3.1.
Language/Vision Agent for Computer Control MINIWOB+ (Shi al., 2017) has several dozens of fundamental computer tasks hosted as live en- vironments. Recent advances on this benchmark have benefited from extensive human annotation to facilitate behavior cloning and reinforcement learn- ing, such as CC-NET (Humphreys et al., 2022) and ACT (Shaw et al., 2023). Beside these models that rely on multimodal or vision input, another
2 Background
LLMs have become an emerging tool for planning and executing necessary steps to fulfill a tepelevel
CCode and notebook: https://github... com/
google-resean sille--reearcchggee wowphhgooolleeeeeeeo http:/ggggleererrcceeeeeeeoeaeee sessearcch/rree/matte.e
NNo public source for CC-NET's screen representation at the time of writing this paper.

---

track is using LLMs as an off-the-shelf agent, and use prompted inference for action generation, such as RCI (Kim et al., 2023), ADAPLANNER (Sun et al., 2023), and SYNAPSE (Zheng et al., 2023). We highlight our technical differences in Tab. 2.
For "Shere via DM butter.
Karte Seraa 10h at
Jaimie Guaann 10h ago Aanean st quik.
Georgeta Basque 20 ago Llbimers (a) Screen
50 clase
3 Environment Interface
(b) Simplified HTML
The role of language agent is to comprehend screen representations (Sec. 3.18322), execute ac- tions on the environment (Sec. 3.3), and react to environment feedback (See. 3.4).
Figure 2: Example of compact screen representation.
3.3 Action Space
For each action, our agent model outputs com- mands in format that is specific to the action type. Specifically, we use three types of actions as shown in Tab. 3. Recent L.Mss such as PaLM- 2 (Anil et al., 2023) are good at following such output format. More prompt details given in Appx. To deterministically ground an action command on MINIWOB+ environment, we fol- low the approach in prior language-only works (e.g. Kim et al., 2023) to access HTML elements by XPATH pattern. When grounding click actions on the actual environment, we use the compact ele- ment id (Sec. 3.2) which is aligned to the actual HTML element in the raw HTML. For the type tion, decompose it into click action followed by series of keyboard presses for the text.
3.1 Treatment of Screens
The definition of screen observation varies by modality in recent works. The screen observation for a vision-based model (e.g. Humphreys et al., 2022; Shaw et al., 2023) can be constrained by vari- ous viewport configurations. For a language model, specifically those taking in HTML code, a screen observation is often the page source of screen, c.g., ones instantiated from a MINIWOB++ task template with a given random seed. When HTML elements are not constrained by viewport configura- tion, the need for scrolling action gone. However, as discussed Sec. we do not immediately use the expanded HTML if the expansion requires UI action: we only expand the HTML representation when the agent actually takes the action. The de- sign relaxes the assumption about the environment, and forces the agent to learn to behave rationally when it is given limited information.
click
type
special key
click id=6 enter "text" to id=10 press ARROWDOWN x
Table 3: Action types and example commands.
3.2 Compact Screen Representation
3.4 Environment Feedback
Raw HTML code tends to be verbose, which poses a practical challenge for LLMs that often have an inherent limit on the input context length. Zheng al. (2023) designed a technique for structured prompting example selection to extract more infor- mative trace examples, as a way to reduce the input content to LLMs. MindAct (Deng et al., 2023) ranked and consolidated element of interests to rep- resent web page snippet. Alternatively, we take inspiration from Wang et al. (2023a) to heuristi- cally simplify the HTML code of each screen, re- taining key attributes for each leaf element, i.e., id, class, text, placeholder, value, and position on 3x3 screen grid. Such simplification has shown to give compelling results on UI understanding tasks. An example is shown in Fig. 2.
MINIWOB+ differs from TEXTWORLD-lik en- vironment (Shridhar et al., 2021) in that state change from taking an action is not naturally phrased out. Instead, an agent will need ob- serve the entire screen change implicitly, making less convenient for the agent to adopt Chain-of- Thought (Wei et al., 2022) reasoning. We broadly categorize trial ending conditions into: 1) correct, 2) cycle, 3) no change, 4) incomplete, 5) exception, and 6) failed. Condition 2) and 3) compare the HTML code of the current screen to those of prior screens. Condition 5) happens when grounding action can not be successfully executed. During multiple trials, each ending condition associated with a prompt for the reflection inference.

---

Enter "An"' to the teput.
Press ARROWWDOW 3 "Aur ands with Youa"
press EXTER
cl the submit
roflection
'er' and wich aik "eal"
TAal and enis with "oua"
Locrnse acCIESI PALCEO
and
COM LACK
CeACe cE
BOT0IC0 Lid lot
Arguilaa
yoek.
Andorna
Andorma
sluckbyy the ORCLANE CeELLORs
Submit
<tep slace
Angaia
Angole
Angullia
Annguilla
Antarctica
Antarctica
Tear suppesiten
Por extlon Codteed,
Antigua and
Antigua and
"Ant the Imput.
Press ARRONDOWW x33
Press AIRRSVIDOINN
prese ENTER
dlick submit
Crtar with wit "oua"
Crrar "Au" unds "oua""
Erter
unas You?"
ands with "oua"
Arguila
Antarttica
Antarttiaa
Andorra
Andorra
Andorra
Submit
Angoia
Angota
Angola
Angullia
Antarctica
Antarctica
Antarctica
Antigua
Figure 3: An example of successful reflection trial by our zero-shot agent on MINIWOB++ task use-oudocomplete with sced=0. Step actions are paraphrased from the actual executable ones for readability.
4 Planning Strategies
where zg adapts those initial steps to be executable actions (ā ()) the environment. In practice, both Ty and 2g use the same LLM.
this section, we summarize the planning strate- gies used recent motivate our staged planning. With a given goal. an agent model is to issue actions based on prior interaction with the environment. For brevity, let denote the interaction sequence of state and action pairs (8;; az).
planning works to
Conceptually, this is similar to zero-shot plan- ner (Huang et al., 2022) and ReAct (Yao et al., 2023) that intermediate thought can help plan long chain of actions. The downside though is that the agent needs to follow carefully crafted few- shot trace examples make a good initial plan, ADAPLANNER (Sun et al., 2023) addresses this is- sue with an adaptive plan refiner that monitors the state-action compatibility and issues refined actions when there is mismatch. This line of planning often needs to deal with hallucinations in the initial plan since, after all, the agent model only observes so but needs to plan for unobserved states.
4.1 Iterative Planning
In iterative planning (e.g. Yao et al., 2023; Madaan et al., 2023), the agent model loops over generating an "atomic" action a,, grounding it on the environ- ment for execution, and then observing for the next state 8;- That is,
ag To(a)3; Si, 1, 1, 1,...ap. sp)
(1)
4.3 Staged Plan-And-Folies
where Ta denotes the planning model. Such plan- ning a common choice for environments that require observation by exploration. With respon- sive environments (e.g. Côté et al., 2018; Shridhar et al., 2021), such an agent can benefit from a long history of interaction that can be easily connected to Thouught reasoning (Wei al., 2022).
Prior works essentially add on extra planning com- ponents to the agent. Instead, we adopt a sim- pler planning strategy. For computer environment, agents often sees a state where multiple actions can be executed on, without the need to observe nuanced state changes, e.g., multiple selection on list. In such cases, iterative planning on single screen can be less efficient, and often, unnecessary. On the other hand, plan-then-oe then-adapt generates ac- tions beyond executable ones that could confuse the LLM agent during the adaptation step. Further- more, both approaches require the agent to itera- tively generate the next action, requiring LLM to have large context window.
4.2 Plan- Tlan-Then-Adapt
Recently, Kim et al. (2023) observed that an initial, and yet rough, plan could help iterative planning. Formally.
(ag, a1, o.)) To (a(a()))
ā ~ =o(a|s; Tj-1, 8i-1, (0o, a1 ...))))))
(3)
To address these issues, we take a step in the mid-

---

dle by maximally planning actions that are visible on the current state all at once. After the plan- ning, the agent only needs strictly follow the gener- ated plan, and such process repeats over multiple screens. Formally,
it increases the input length of LLM. Moreover, it requires the LLM agent to handle the thought struc- tures through multiple trials. practice, Shinn et al. (2023) limited the memory size E [1,3].
5.1 Structured Thought Management
(af....ff ~ s; 3j 2. 2.)..a))
(4)
In a zero-shot setting, reflection puts a heavy bur- den on LLM's capacity to follow a complex in- struction set (in addition to those in Sec. 3.3&4.3). After all, in this case. LLM has no expert traces to follow, thus need to learn from trials. With in- creasing the number of trials, reflection memory essentially forms a combinatorial problem for the agent to solve. For a time step i, there can be mul- tiple failed actions in historical trials, thus should be avoided. For a trial L, if the agent identified a critical mistake at time i, reflections on later time steps can be considered outdated.
where each stage is essentially generating k ex- ecutable actions for state Note that, we also omit former states in Eq. 4 to make inference more efficient. In practice, found that simple state- ment, in natural language, that summarizes what functionality achieved by the action, is a good representation for the state-action pair (s;, (s),a:).
Implementation details. We once again rely on the underlying LLM to follow execution instruc- tions as in Eq. 4. Prompt details are in Appx. B- E. In occasions, agent model predicts fewer steps (e.g., forgetting submit) more steps (e.g.. hallucinating non-executable actions) than needed. For the former, we loop over the planning in Eq. until no further plan is generated. For the latter, we halt the execution of the current plan and resort to self-reflection for correction.
In our preliminary study, we found it is important to devise way to help an LLM agent maintain this memory. Otherwise, when a reflection entry (a,, df) is given, even a state-of-the-ari LLM can still 1) repeat the same mistake a,, 2) fail to follow the reflected plan to even reach 8;, and 3) bounce between wrong actions it collected in prior trials.
To make reflection run more reliably and effi- ciently, we propose structured self-reflection in Algo. When a suggested action af is given by the reflection agent, we enforce our agent to plan exactly of at step i. Moreover, to avoid looping over two failed actions at the same step, we use a disabled action set 1) to memorize them and jointly disable these actions in the environment. Finally, clear reflection entries for future steps if an early entry is updated. With this management, our agent is no longer bounded by LLM's input limit, and has memory size N.
5 Structured Self-Reflection
practice, high-level human instruction goal can be ambiguous, and an environment can be par- tially hidden. Therefore, agents are prone to mak- ing mistakes; this is even true for a human user when executing a task (Humphreys et al., 2022). Once negative signal is given, such as no change failed (Sec. 3.4), we ask the agent to reflect on its past action trajectory, suggest an improved version, and then retry the task. An example of successful reflection from our agent is shown in Fig. 3.
In recent works (Shinn et al., 2023; Madaan et al., 2023), reflection is conducted at the end of each trial by accumulating a text entry. The entry is essentially natural language statement about what should have been done instead. At trial
Algorithm 1: Structured Thought Management
I: R [Ø] + N; 12 [2] N;
2: for f é [0)
3: for i N):
4:
if R[] and R[ a' DMV: // if has reflection
5:
0, =R[]aaa
f/ action from reflection
aj re(a|sis aj 1,...00, 80; Rs)
(5)
6:
else:
(a(a)...
Il regular planning
7:
if needToReftect.
Il if error happens
R&11 REFLO((n, Sn ... ... Si. Re)
(6)
8:
(a,.a() ~ REFLo(..)
/| reflect
9:
if RLJ # Ø:
where R2 consists of list of (a,, a() pairs, each denotes update the wrong action a, to a/. In the followup trial i + 1, accumulated entries in R are prefixed to the prompt for the agent planner Ta- The amount of entries maintained in the reflec- tion memory is limited by multiple factors. For one,
10:
DD DljjadddRR]]].
#/ record wrong click
11:
RDJ (a,,d,)
If record verbal reflection
12:
R
Ø: D + :]= Ø // clear mem
Note that in line 6, we use the staged planner in Eq. 4.3 which does not depend on the iteratively up-

---

dated R. thus is different from recent works (Liq. 6).
experiment with our staged planning in Sec. 6.3- 6.4. Finally, we examine if our zero-shot agent can learn from mistakes in Sec. 6.5. Our prompts are in Appx. A-E. Complete results are in Appx. F.
Interplay with Staged Planning. Suppose the staged planner predicted [ao....i, but exe- cution had failed, and the reflection step identified Oj the carliest mistake, thus suggested at. In the next trial, we will repeat the executions from on az 3 and intercept the agent planner at step to enforce the execution of d4. For steps after i, we bring our planner to the next stage. In the worst case where an agent fails at every step, our staged planning essentially falls back to the plan- then-adapt (Sec. 4.2), except having no initial plan.
6.1 Setup
We focus on 43 MINIWOB+ tasks that are suit- able for evaluating language-based models. This differs from prior work since excluded those 1) require visual signal to solve (e.g.. count-shape and grid-coordinate); and 2) expose insufficient language API to operate (c.g., enter-date and enter- time); The motivation for this filtering is simple: even if some filtered tasks can be solved by an LLM agent, it does not generalize. Furthermore, we do not include ferminal as the synthetic console supports very limited set of commands while the LLM, in our preliminary experiment, tends to use more sophisticated ones.
5.2 Constraining Action Space
For updated action a' at reflection trial t. enforce it to be taken at the associated time step if and only if a' is not an failed attempt before trial t. It can be tricky to prompt LLM to follow such sim- ple combinatorial constraint in text, especially as a mixture of positive and negative signal surrounded by other instructions. Therefore, we found it is crucial to explicitly disable those previously failed actions in the corresponding screen representation. This, however, does not mean removing the cor- responding element from the HTML pseudo code. We instead only remove the id attribute, and still allow the element information to be presented to LLM. We only do so for click-type actions.
We separate these 43 tasks into three categories: 1) 1-screen-l-step, 2) 1-screen-n-step, and 3) n- screen-n-step. If the task involves state update (e.g. expanding dropdown list or openning hidden tab), the task is n-screen. If the task can be solved by just one action, it is I-step: otherwise n-step. The task distribution is reported in Tab. 4.6
For each task, we evaluate with 25 different ran- dom seeds, starting from secd=1000, similar to Pix2Act (Shaw et al., 2023). Performances are re- ported as the correct completion rate over multiple runs. For validation and prompt design, use seed [0, 10]. For the LLM agent, we use the FLAN-PaLM22 L. (Anil et al., 2023) with tempera- ture across all evaluations for better reproducibil- ity.
For non-click actions, the disable set D cannot be easily enforced on the environment and the LLM agent. We can indeed prompt the underlying LLM saying certain special keys are invalid or certain texts not to input. However, we did not observe positive impact from doing so in our preliminary experiment*. Thus, we fallback to only determinis- tically generate the df at time step i.5 We locate the time step by prompting the reflection agent to out- put in format: "For action index=i, you should ag". This differs from prior work (Shinn et al., 2023) which uses reflection memory as sticky entries in LLM prompts across all time steps.
1-screen-1-step
1-sereen-n-step n-screen-n-step
#Task
10
20
13
Table 4: Task distribution for each category MINIWOB++.
6 Experiments
6.2 Model Comparison
For each task category, we compare with prior best models that rely on language as input sig- nal, including supervised models, i.e., WEEN- T5 (Gur et al., 2022) and CC-NET (Humphreys et al., 2022), and agents based on prompted infer- ence, i.e., RCI (Kim et al., 2023) with GPT-3.5
We start with categorizing tasks by their planning complexities to have an isolated testbed. Then we
UUp this point, the work flow is similar the refine- then-resume in ADAPLANNER (Sun et al., 2023).
'A possible reason is that the instruction set LLM prompt is already dense and reflection prompt tends to be long. thus such nuanced requirements sometimes get ignored.
"The downside is that agent can potentially loop over two non-click actions across multiple reflection trials.
"Based on our categorization, the screen issue (Sec. 1) impacts the n-screen-n-step category.

---

Fun 1
rum, 31ai-3
Supervised
Few/N-shot
Zero-shot (Ours)
WebN-TS
CC-Net
RCI
AdaPln
T
T
Compless
60.4
95.1
96.1
96.5
95.3
97.3
Table 5: Average performance on T-screen-n-step tasks, 16 shared across all models. T: number of trials.
dellp
Butter
dlac rest
dickrest
aick-addeer
Ascnntestt
bron-test22
Fun brial1
num, 3rial3
Completed
Figure 4: Performance I-screen- 1-nn----ssss 1-step tasks, click-widger contains ambiguous task objective, thus reflection helps.
and ADAPLANNER (Sun et al., 2023). For few- shot models, we focus comparing with agents that have reflection capacity. Non-reflective agents, such SYNAPSE (Zheng et al.. 2023), have tech- niques that are orthogonal our work, and thus can potentially be combined with ours. Furthermore, we notice each work often used slightly different set of tasks. For fair comparison, we will also report performances on the shared set of tasks.
dlick Meettutton sequeecce
drd optioo di
Lattl
did cheet checo cortine ond
aiiii th
spccil
ester
Figure 5: Performance on I I-screen-n stsp tasks,
screen-n-step tasks in Fig. 5, Our agent achieved
94.0% completion in 1 trial, and 96.2% in 3 trials.
6.3 Single Step Tasks
6.5 Reflective Planning on Challenging Tasks
We compare our zero-shot agent on the easiest cat- egory (I-screen-L-step) tasks against recent state- of-the-art. As shown in Fig. 4, our agent achieves 100% accuracy on correctly finishing tasks, even without reflection. One exception is the ambigu- ous click-widgel which, without in-context trace prompt, can be easily failed. For instance, the task could ask agent to click text widget, however, in- put text and text area not deemed correct. With 3 rounds of reflection trials, our agent achieved 96% completion rate on it. Overall, we have 96.4% with trial, and 99.6% with 3 trials. compari- son, with few-shot trace examples, RCI (Kim et al., 2023) achieved 99.8% with round of reflection (at the plan level).
Here, we move on to more challenging (n-screen- n-step) tasks to show the impact of our efficient reflection. Task-wise completion rate is reported in Fig. 6. Firstly, we observe without examplar traces, zero-shot agent tends fail on the first trial. This happens often tasks that requires exploring across multiple screens, e.g., click-menu-2, click- tab-2-hard, and search-engine. After a few rounds of exploration, our agent achieved substantially bet- ter completion rate by avoiding previous negative signals recorded in the memory. Our agent contin- ues to improve even with T 5, suggesting more efficient reflection than prior work e.g., RCI only capable of one round of reflection plan level.
6.4 Iterative Planning v.s. Staged Planning
Supervised
Few/N-shot
Zero-shot (Ours)
WebN-TS CC-Net
RCI
AdaPIn
T
T
We compare these two approaches using 1-screen- n-step tasks. We hope these experiments can an- swer that, with a given state, whether one should query the agent for actions one at a time or once for all. We compare the prior state-of-the-art works with our staged planning in Tab. 5, showing that one can simply plan out all executable actions screen and "blindly" execute them. Doing can substantially reduce LLM queries and still achieve high completion rate.
31.0
97.2
85.8
89.7
73.5
87.3
Table 6: Comparison on 11 shared tasks across different models in the n-screen-n-step category. T: number of trials.
Again, we compare with prior best models in Tab. 6. The few-shot models exploited inconsistent screens (as discussed in Sec. 1). thus our work in an unfair disadvantage against them. Despite such disadvantage, our agent achieved performance com- parable to them. Importantly, our agent does not re-
We report detailed completion rate on all 20

---

is turned off. This setting is the comparison be- tween Both v.s. w/o structured mem in Fig. where we select challenging tasks and run 25 seeds for each setting. Clearly, our structured reflection is a beneficial add-on.
Fum Irill1
num, 3
num, trialeS
Completed
60
20
7.2 Ablation on Action Constraint
dick meno
dick Aac--aab-
dick bicktabb22 hard
maalintor
enast mestinttor-ddto cutur
email latoo foreord
erant enast wow.orrooooooooddoooooo elttork
navigate tree
search engine
social mesia
use sen-autucommiitte
disk
A useful technique we proposed in Sec. 5.2 is to delete the id field in the HTML pseudo code to heuristically discourage LLM agent from issuing the corresponding action. This is essentially mini- mal change to the input. In Fig. 7, we ablate on this small change by comparing Both v.s. w/o action constraint and show that it is beneficial to apply the action constraint.
Figure 6: Performance on n-screen-n--ee stsep tasks.
quire in-context trace examples for few-shot, some- times many-shot, prompting, and no customized and detailed environment feedback. Finally, we note that the gap complex tasks between super- vised model and unsupervised ones is still large.
7.3 Statistical Significance over Trials
We evaluate statistical significance across various trials on the n-screen-n-stee step tasks. For each task, consider all 25 example predictions. This gives us 13 x 25 325 samples for each comparison. Using t-test (Dror et al., 2018), the results are in- deed significant (p < 0.05) as shown in Tab. 7. For task-wise significance, see Appx. G.
7 A Analysis& & Discussions
7.1 Ablation on Reflection Strategies
Here, we compare our structured reflection against the "'riiiiaa" reflection mechanism. We should note that reflection is a general scope that has dif- ferent formations (e.g. Shinn et al., 2023; Madaan et al., 2023) and was introduced on environments (e.g., ALFWORLD) that are significantly different from MINIWOB ++. Moreover, it was often used along with iterative planning strategy, which is not directly compatible with our staged planning.
Baseline Hypothesis p-value
T=1
T=3
2e-10
T=3
T=5
0.002
T=1
T=5
2e-12
Table 7: Significance test using t-test comparing different number of trials.
structured mem
w/o action comstrant
Both
7.4 Planning Call Reduction
Compltted
In Tab. 8, we highlight the efficiency boost by su- ing our staged planning formulation. We illustrate the result on I-screen-n-siep tasks that require rela- tively long action traces (> 7 actions) on single screen, and compare the number of planning calls for completed traces as well as failed traces.
m
aik-tab22
dick 18-22222. Rard
Bed word
7.5 Compact Screen & Input Length Limit
Figure? 7: Comparison of reflection strategies with T 3.
Representing user interface HTML puts high demand on LLM's context capacity. One instance the
Therefore, use adapted version for com- parison: an agent that uses structurally managed timestep? while structurally thought management
task that can span more than a dozen candidates, each with multiple options. As a result, flattening the complete set of state- action pairs can easily run over the input limit for the reflection agent, since it needs to observe the entire trace. On this task, we noticed that nuanced
"We insert reflection thought at the corresponding time step so that actions before this time step can be deterministically replayed for better efficiency.
*On top of the structurally managed timestep, we also manage the expiration of thoughts over multiple trials, as well
as constraining action space.

---

T=I (Success-only)
T=3
screen representation for this task is relatively sim- ple as each checkbox corresponds to a line of text, This differs from the social-media task where each candidate has multiple actionable, sometimes am- biguous, items, thus putting a stronger requirement to LLM for disambiguation. We observe a pattern in Tab. 9 that with flat and less ambiguous screen, LLM has high capacity to accurately plan out mul- tiple steps in one inference call. In such case, one could just execute all planned steps without need- ing repetitive planning calls. But with complex screen constructions, the capacity of one-pass plan- ning is reduced by a large margin. Prior work (i.e. RCI) constrained the number of candidates in the social-media task to [3. 6]. We observe that relax- ing such constraint introduces significant difficulty for planning. Therefore, multiple trials of reflection can help the agent in these complex scenarios.
Task
IP
SP
IP
[SP]
click-checkhooce larees-laoooee-rrggee
234
24
89.7%
270
31
88.5%
clickccheekbones-sott
81
19
76.5%
167
64
61.7%
multi-layout
175
25
85.7%
175
25
85.7%
chek-chhecbones-soll
114
20
82.5%
224
64
71.4%
Table 8: Planning call reduction by staged planning. Compar- isons are on the successful first trials and all trials when T = 3, using 25 examples per task. IP: number of planning calls re- quired for iterative planning. SP: number of planning calls in staged planning. 1: pereentage of planning calls reduced by staged planning.
actions does not substantially change the screen. Therefore, we always stick to the first screen when constructing prompt for the reflection agent. A more autonomous method can be state filtering in SYNAPSE (Zheng et al., 2023).
Lining up the separation of what HTML ele- ments expose LLM is important for evalua- tion. As we have seen that many of the MiniWoB++ tasks are already easy for today's LLM. Exposing more unseen elements risks hiding the actual chal- lenge in the navigation tasks. For instance, expos- ing unseen elements basically simplifies n-screen- n-step tasks into 1-screen-n-step ones. However, our experiment shows that n-screen-n-steg ones actually much harder to deal with.
Conclusions
We proposed the first zero-shot agent for computer control tasks. Our agent design generalizes the workflow for easy and complex tasks via efficient planning and structured thought management. We evaluated our agent on the MINIWOB++ bench- mark, showing that our agent, with often one pass of planning query, outperforms the best iterative planning agent as well as supervised state-of-the. art simple tasks. For complex tasks, we show that agent design performs on par with the best LLM-based model via more efficient planning and reflection, without requiring manually crafted trace prompts and ad-hoc environment feedback.
7.6 Capacity of Staged Planning
To better understand the planning capacity and limit of our staged planning, we experiment with I I-screen-n-step tasks that have extensive number of candidates. Namely, we use click-checkboxes and social-media as probe tasks, and report in Tab. 9, Both tasks are multi-choice, but differ in their can- didate structure complexity.
Limitations
9.1 Other LLM Choices
We focused evaluations based on PaLM-2. Re- cent advances in LLM agents (c.g., Wei et al., 2022;
Task
#Gold
Completion
click-checkhoxes
10
100
Yao al., 2023; Shinn al., 2023) have shown that different LLMs (e.g.. PaLMM, GPT-3/4, Codex) gen- erally exhibit a common capacity to benefit from intermediate thoughts and self-criticism. We be- lieve there is a reasonable adaptation of our findings on other LLMs.
click-checkhoxes
90
Task
#Candidate Completion rate
10
80
Jucial-meddia-all
>
40
Table 9: Impact of number of candidate/gold actions on task completion. We evaluated 20 examples for each setting, and T 1.
9.2 Other Modalities of Input
Large multimodal models can take additional in- puts such as screen images, and prior works (e.g.. CC-Net (Humphreys et al., 2022)) have shown that extra modality can indeed be beneficial. However,
For the click-checkboxes task, we separate ex- amples by their number of actions required". The
"which can he heuristically parsed from the task command.

---

even with recent designs of large multimodal mod- cls, explicit reasoning still takes place in the form of language. Therefore, our proposal could benefit in such multimodal use cases.
Acknowledgements
We thank the reviewers of EMNLP for constructive comments and pointers.
9.3 Integration Zero-shot Chain-of-Thought
References
Prior zero-shot works (e.g., Huang et al., 2022; Wang et al., 2023b; Crispino et al., 2023) discov- ered LLMs can be used to expand prompts with prior knowledge and intermediate steps to work in zero-shot manner. Theoretically, this line of works can also be integrated into our reflective agent to promote completion rate on the first trial. One potential challenge is that computer control tasks, looking at the input texts, are quite different from those in general domain (e.g., sentiment clas- sification, numerical reasoning). Thus, the quality of extracted prior knowledge needs to be evaluated. We leave this direction be explore in future work.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.1040:
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Ju- lian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, al. 2023. Do as i can, not as i say: Grounding language in robotic affordances. In Conference Robot Learning, pages 287-318, PMLR.
Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben A. Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew J. Hausknecht, Layla EI Asri, Mah- moud Adada, Wendy Tay, and Adam Trischler. 2018. Textworld: A learning environment for cest-based games. In CGW@IJCAL.
9.4 Constraining Space for Non-click Actions
Sec. 5.2, we let the reflection module to interact with the environment, explicitly disabling failed click actions by removing the "id" field on respec- tive elements. This often helps our agent avoid repeating the same mistakes, but only for click ac- tions.
Nicholas Crispino, Kylc Montgomery, Fankun Zeng, Dawn Song, and Chenguang Wang. 2023. Agent instructs large language models to be general zero- shot reasoners. arXiv preprint 23X.0:2310.03710.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards generalist agent for the web.
9.5 More End-to-end Tasks
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Re- ichart, 2018. The hitchhiker's guide testing statis- tical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguisties (Volume 1: Long Papers), pages 1383-1392, Melbourne, Aus- tralia. Association for Computational Linguistics.
Recent few-shot works have used techniques to extract informative reference traces, either from expert or agent exploration (Zheng et al., 2023), to progress more end-to-end computer tasks, such book-flight. We observe such end-to-end tasks remains significant challenge to zero-shot agent.
Hiroki Furuta, Ofir Nachum, Kuang- Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. 2023, instruction-finttune foundation models for multi- modal web navigation. In ICLR 2023 Workshop Muthematical and Empirical Understanding of Foundation Models.
9.6 Higher-order Action Cycle
In Sec. 5, we proposed a structured thought man- agement to facilitate agent's self-reftection. While this module can effectively help L.M agent avoid repeating prior mistakes, there are corner cases need to be covered. In rare cases, we observed the agent can loop over two failed and different traces by accidentally clearing up prior reflection mem-- ory. This is because our agent considers reflections on later time steps outdated once there is a reflec- tion entry for earlier time step. Future work can use additional trace memory to avoid such corner cases.
Izzeddin Gur, Hiroki Furuta, Austin Huang. Mustafa Safdari, Yutaka Matsuo. Douglas Eck, and Aleksan- dra Faust. 2023. A real-world webagent with plan- ning, long context understanding, and program syn- thesis. arXiv preprint 2307 2007..22522 12856.
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sha- Narang, Noah Fiedel, and Aleksandra Faust. 2022. Understanding html with large language mod- els. arXiv preprint arXiv:2210.0394.

---

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 91 9118-9147. PMLR.
Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representa- tions (ICLR).
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thortoo,, Rachita Chhaparia, Alistair Mul- dal, Josh Abramson, Petko Georgiev, Adam San- toro, and Timothy Lillicrap. 2022. data-driven approach for learning to control computers. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9466-9482, PMLR.
Haotian Sun, Yuchen Zhuang. Lingkai Kong. Bo Dai, and Chao Zhang. 2023. Adaplanner: Adaptive plan- ning from feedback with language models. arXiv preprint arXiv:2305. 16653.
Bryan Wang, Gang Li, and Yang Li. 2023a. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference Human Factors in Computing Sys- tems, pages 1-17.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. arXiv preprint arXiv:2303. 17491.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ec-Peng Lim. 2023b. Plan-and-solve prompting: Improving zero- shot chain-of-thoughe reasoning by large language models. In Proceedings of the 61st Anmual Meet- ing of the Association for Computational Linguis- rics (Volume 1: Long Papers), pages 2609-2634, Toronto, Canada. Association for Computational Linguistics.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition- levell code generation with alphacode. Science, 378(6624):1092 1097. 1097.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman. Shiyu Huang, Chan- dra Bhagavatula, Yejin Choi, and Xiang Ren. 2023. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. In Advances Neural Information Processing Systems.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages Inc.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tian- lin Shi, and Percy Liang. 2018. Reinforcement leamn ing on web interfaces using worktlow- guided explo- ration. arXiv preprint 1rXiv:18022 08802.
Curran Associates,
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv arXiv:2303. 17651.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu. Maria Lomeli, Luke Zettlemoyer. Nicola Cancedla, and Thomas Seialom. 2023. Tool- former: Language models can teach themselves to use tools. arXiv preprint 2302::3302.0776
Longtao Zheng, Rundong Wang. and Bo An. 2023. Synapse: Leveraging few-shot exemplars for human- computer control. arXiv preprint 23Xiv:2306.0786.
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Be- rant, Panupong Pasupat, Hexiang Hu, Urvashi Khan- delwal, Kenton Lee, and Kristina Toutanova. 2023. From pixels actions: Learning to follow instruc- tions via graphical user interfaces. arXiv preprint arXiv:2306.0022 300.0025..
A Prompt for Action Space
Fig. 8 contains the prompt prefix that lines out three action types.
B Prompt for Staged Planning
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Her- nandez, and Percy Liang. 2017. World of bits: An open-domain platform web-based agents. In In- ternational Conference on Machine Learning, pages 3135-3144, PMLR.
Fig. 9 contains the prompt prefix for maximally plan out actions on each screen in one pass.
C Prompt for Action Paraphrase
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: autonomous agent with dy- namic memory and self-refiection. arXiv preprint arXiv:2303. 1136.
Fig. 10 contains the prompt prefix for maximally plan out actions on each screen in one pass.

---

You can generate series of atomic actions to fulfill a top-level goal. There are three types of atomic actions you can perform. Firstly, you can click an object by referring its id, such as "click id=... Secondly, you can enter text to input field, such "enter" "..." id=... Specifically, you should always wrap the text you want type in with double quotes. Lastly, you can operate special keys the keyboard, such "hold CTRL" and "release CTRL" before and after multiple selections. If dropdown list available. you can "press ARROWUP x N" "press ARROWDOWN N" to press the arrow key N times to iterate over list items, and then "press ENTER" to select the current item.
Figure 8: Prompt that defines three action types.
Now, you need to plan actions that executable and only this screen. For actions that executable this sereen, should leave them future planning. Your plan should consist of a list of atomic actions on the screen. Please separate them by newline.
Figure 9: Prompt that promotes staged planning.
D Status Prompt Prefix
Fig. II contains prompt component we use to com-
pose reflection prompt for each type of environ- ment feedback.
E
Prompt for Reflection
Fig. 12 contains the prompt prefix for our structured reflection.
F
Completion Rate Tables
We report the completion rate 43 tasks in MINI- WoB++ in categories. Performances in 1-screen- I-step tasks are Tab. 10, I-screen-n-step task in Tab. 11, and r-screen-n-step tasks in Tab. 12.
Task-wise Statistical Significance
In Tab. 13, compare T v.s. T = 5 on the n-screen-n-step tasks. We use the one-tailed McNemar (matched chi-square) for the test.

---

You are capable of describing actions taken computer. The computer screen is represented the following HTML pscudo code: <screen> [html] </screen> And the action taken is: (action_str) Now, in plain language, please summarize what has been done. You should describe the specific purpose for the action, instead of simply referring to the element id or position of the element. Summary:"
Figure 10: Prompt for paraphrasing executed action. Newlines are filtered.
STATUS.FAILED. "However, your actions did not complete the goal. Now, you need to identify the earliest critical step where you made mistake, and suggest correction."
STATUS.CYCLE: "However, your actions led you loop that did not progress the task. Now, you need to identify the carliest critical step where you made mistake, and suggest correction.
STATUS. SAATUS.NOCHAANGEE "However, your last action did not cause anything to change on the last screen. You prob- ably used the wrong action type. Now, you need to identify the carliest critical step where you made mistake, and suggest a correction.
STATUS.IN COMPLETE: "However, your actions did not finish lask, likely more steps are needed. Now, you need identify earliest critical step where you made mistake, and suggest a correction." "
"However, you took too many steps and yet still did finish the task. Now, you need identify the earliest critical step where you made mistake, and suggest correction." "
STATUS.IN PROGRESS:
STATÜS.EXCEPTION "However, your last action is invalid. You should avoid doing that again and try different action."
Figure Il: Prompt component that describes environment status.
You ure operating computer for lask: [lask name). You went series of screens and executed actions fulfill top-level goal. Your action trajectory is as follows: The index=k screen: {screen_hmi at k} Your index=k action: [action at k) You conducted the above actions for the top-level goal: goal (status str) Your suggestion should be in this format: "For action index=A, you should B. where A the action index, and is the suggested action you should have taken. Your suggestion:
Figure 12: Prompt for reflection. Newlines are filtered. status_str _tr is defined in Fig. 11.
Task T=1
T=3
T=5
click-dialog
2
100
100
100
click-link
100
100
100
click-button
100
100
100
click-tab
100
100
100
click-test
100
100
100
click-test-2
100
100
100
click widget
focus-text
100
100
100
focus-text-2
100
100
100
Table 10: Completion rate 1-screen-I-ster tasks.

---

Task
T=1
T=3
T=5
clickhhutton-squenee
100
100
100
click-checkboxes
100
100
100
click-checkhoxee--lrge
slick-checkhoxes-softt
68
84
84
seik-chhcchoxe-traastee
100
100
100
clic--collapsible
click-option
100
100
100
click-scroll-ist
100
100
100
enter-password
100
100
100
enter-text
enter-text-2
100
100
100
enler-lex--dymmmic
100
100
100
find-word
login-user
100
100
100
multi-layouts
100
100
100
multi-ordering
100
100
100
table
100
100
read-table-2
100
100
100
melial-mdia-all
68
80
80
sicial-media-aao some
Table 11: Completion rate on 1-screen-n-step tasks.
Task
T=1 T=3
T=5
click-collapsible-2
88
88
88
2lick-menu-2
36
72
76
click-pie
96
96
96
click-tab-2
44
60
68
click-tab-2-hun
32
56
64
email-inbox
100
100
cmail-inbox-nl-turk
76
92
92
100
100
100
email-inbox-forvaad----mo
100
100
100
mavigate-tree
96
96
96
seurch-engine
20
64
72
social-media
84
84
84
use-autocomplete
72
96
96
Table 12: Completion rate on n-screen-n-step tasks.
Task #Completion (T=1)
#Completion (T=5)
p-value
click-collapsible-2
22
22
click-menu-2
0.008
click-pie
24
click-tab-2
11
17
0.007
click-lab-2-hurd
8
16
0.0024
email-inhox
24
0.159
email-inbox-nl-tmre
19
23
0.042
email-inbox forwwardr
25
25
email-inbee in -l--unl-uumm
25
25
navigate-tree
24
24
search-engine
5
18
0.000016
social-media
21
21
use-autocomplete
18
24
0.007
Table 13: Significance test using one-tailed McNemar for each n-screen-n-step task.


Reviews:
[{"review_id": "7d4bplbbMu", "reviewer": "Reviewer_zQff", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough reviews. We address your concerns here, and we believe all of your questions can be addressed in the revision.\n\n> Experimental results on the MINIWOB++ benchmark reveal that while the proposed method matches state-of-the-art (SOTA) performance in simpler tasks, it tends to fall short in more complex ones.\n- Our work presents the first *zero-shot* agent for computer control problems. Zero-shot agents inherently face challenges due to the lack of expert demonstrations, thereby generally expected to underperform supervised or few-shot approaches. Nonetheless, our method impressively achieved state-of-the-art results across a variety of tasks, highlighting the significance of our contribution. Our future work will further explore zero-shot agents for more complex tasks.\n\n> The rationale behind using a staged planner with LLMs for the give task could be better articulated.\n- We will improve the presentation. The main motivation of using a staged planner is efficiency. In our experiments, staged planner performs on par with prior iterative planner approaches while our query efficiency is substantially reduced (evidences are in the “Efficiency” bullet point).\n\n> Experimental controls for random noise appear insufficient. Reporting standard deviations, repetitions of measurements, or carrying out statistical analyses would strengthen the work.\n- We use 0 temperature during decoding, following prior works on MiniWoB (e.g., RCI) to have deterministic and reproducible results.\n- Agent performances are reported as the average of 25 randomly instantiated examples for each task. Even for the same task, different random instantiations could have different complexities, and each instantiation is evaluated in either 1 (for successful completion) or 0 (incomplete for any reason). Thus measuring standard deviation of completion rate does not tell much.\n- For statistical significance over the number of trials, we ran the t-test implemented in [1] over 13 n-screen-n-step tasks as a whole (13x25=325 examples in total). The results are indeed significant (p<0.05):\n  - From num_trial=1 to 3, p-value is 2e-10\n  - From num_trial=3 to 5, p-value is 0.002\n  - From num_trial=1 to 5, p-value is 2e-12\n  - We will add these analyses to the draft.\n\n[1]: The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. Dror et. al. (ACL18)\n\n> On line 465, the authors posit that the staged planner requires fewer LLM queries than the interactive planner, yet fail to provide supporting experimental evidence.\n- We use the 1-screen-n-step category to demonstrate the efficiency improvement. We select tasks that require relatively longer action traces (>= 7 actions) on a single screen. Specifically, we experiment with num_trial=3 and count the planning call reduction for successfully completed traces as well as including failed traces. The result is measured for 25 examples for each task:\n| | 1st successful trial (25 ex per task) || Up to 3 trials (25 ex per task) ||\n|-----|-----|-----|-----|-----|\n| Task | #actions | Planing call reduction | #action | Planning call reduction |\n| click-checkboxes-large | 234 | 89.7% | 270 | 88.5% |\n| click-checkboxes-soft | 81 | 76.5% | 167 | 61.7% |\n| multi-layout | 175 | 85.7% | 175 | 85.7% |\n| social-media-all | 114 | 82.5% | 224 | 71.4%|\n- Specifically, the planning call reduction is calculated by comparing the number of staged planner calls vs the number of iterative planner calls. The number of iterative planner calls is essentially the number of actions since each action requires a round of LLM inference. Clearly, our staged planner offers a substantial planner call reduction. **Thus, it is efficient as we claimed.** We will add this result to the final draft.\n(The numbers are the same for task multi-layout, since our agent achieved 100% completion rate on the first trial.)\n- We want to clarify that the argument we want to make on staged planning is that, with such large efficiency improvement and similar completion rate (both staged and iterative ones ceiling at ~100%), why not just use the more efficient one.\n\n> Both the Structured Thought Management and Constraining Action Space seem to boost performance alongside the number of trials. Ablation studies to individually assess their impacts would be beneficial.\n- Here we do an A/B test on the action constraining component. Specifically, we ablate whether to deterministically remove the element id from HTML so that the agent will avoid issuing actions on it. The below result shows that heuristically constraining action space (in addition to pure verbal instruction) indeed helps the agent model.\n- We use challenging tasks in Table 7, except the find-word task since it’s not centered on click action. Completion rate is measured over 25 examples for each task.  Note that action constraining is a small module in our agent construction thus the relative small improvement is expected. We will add this result to the draft.\n| Task (num_trial=3) | no constraining action space | with constraining action space |\n|-----|-----|-----|\n| click-menu-2 | 72%  | 72% |\n| click-tab-2 | 56% | 60% |\n| click-tab-2-hard | 52% | 56% |\n| search-engine | 60% | 64% |\n- For ablation study on the structured thought management, we already have a comparison in Fig 7 showing that the thought management is a beneficial add-on.\n\n> In the results for 1-screen-1-step and 1-screen-n-step tasks, there's a noticeable ceiling effect, with most tasks achieving a 100% completion rate. This could bias average performance comparisons across models. Focusing analyses on n-screen-n-step tasks' results might offer a more balanced perspective.\n- This is why we separate the testing on different complexity category, unlike prior works that average over all tasks which included a lot of easy tasks.\n- Reporting results on easy tasks is essential to support our claim that a staged planner is enough for easy tasks. Prior works on these types of tasks exhibit similar ceiling effects, therefore matching similar performance is the bottom line.\n- Our proposal of the structured reflection is indeed focusing on more challenging tasks (i.e., n-screen-n-step).\n\n> The adapted comparison mentioned in line 508 is unclear. Could the authors provide further details or clarification?\n- We will improve the presentation on this ablation study. The two variants for comparison are:\n- **Structurally managed timestep**: The original reflection mechanism (as introduced in the Reflexion paper) inserts reflection memory entries to the planning prompt regardless of what timestep the identified error occurred. In our case, we only insert the memory at the corresponding time step so that actions before this time step can be deterministically repeated for better efficiency.\n- **Structurally managed thoughts**: On top of the structurally managed timestep, we also manage the expiration of thoughts over multiple trials, as well as constraining the action space associated with the thoughts. This is essentially Algo 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The exploration of differences between interactive and staged planning is compelling.\n\nThe study presents a novel approach to enhancing LLM agent planning in real-world interactive tasks.", "reasons_to_reject": "The rationale behind using a staged planner with LLMs for the give task could be better articulated. For instance, could RL agents or classic planners tackle MINIWOB++ with interactive planning?\n\nExperimental controls for random noise appear insufficient. Reporting standard deviations, repetitions of measurements, or carrying out statistical analyses would strengthen the work.\n\nOn line 465, the authors posit that the staged planner requires fewer LLM queries than the interactive planner, yet fail to provide supporting experimental evidence.\n\nBoth the Structured Thought Management and Constraining Action Space seem to boost performance alongside the number of trials. Ablation studies to individually assess their impacts would be beneficial.\n\nIn the results for 1-screen-1-step and 1-screen-n-step tasks, there's a noticeable ceiling effect, with most tasks achieving a 100% completion rate. This could bias average performance comparisons across models. Focusing analyses on n-screen-n-step tasks' results might offer a more balanced perspective.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a zero-shot LLM agent designed for web interaction tasks. Enhanced with a self-reflection and structured thought management module, the agent can generate actionable plans in a staged manner. Experimental results on the MINIWOB++ benchmark reveal that while the proposed method matches state-of-the-art (SOTA) performance in simpler tasks, it tends to fall short in more complex ones.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The blending of the proposed methodology with prior research is confusing. I would suggest putting previous work in a related work section, and describing your own work in a method section.\n\nThe task environment should be delineated more than just referencing the MINIWOB++ paper. Terms like \"screens,\" \"candidates,\" \"trials,\" and \"steps\" are used in sections 4 and 5 without prior introduction.\n\nThe adapted comparison mentioned in line 508 is unclear. Could the authors provide further details or clarification?"}}, {"review_id": "eVSg8XLrSr", "reviewer": "Reviewer_wto2", "summary": "", "questions": "A. Algorithm 1 is very very hard to follow; can we add a more annotated version of the algorithm (or better, a visual that captures exactly what’s happening in the structured reflection module)?\n\nB. It’s not immediately intuitive why the staged action prediction is better than closed loop (predict action iteratively, one after the other) in this approach? Can you provide more intuition or motivating examples that show when closed-loop action prediction fails, and staged action prediction succeeds?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and constructive feedback! We address your comments here and will revise our paper accordingly.\n\n> As hinted at above, I think there are several design decisions that are not properly ablated.\n> ...\n> Following the above general criticism, it would be nice to ablate specific components of the proposed approach vs. existing work. How would a method like RCI work if it used the structured reflection module instead of the naive single-step reflection feedback? How would AdaPlanner perform with the staged action prediction component?\n- It is nontrivial to conduct strictly clean comparison against prior works, and we seriously considered a number of options for doing so. One observation of recent LLM-based works on the problem (including our own) is that each component is tailored to work within the proposed paradigm. This includes RCI, Adaplanner, ours, and other cited works; let alone to say that our work is the first zero-shot agent which is quite different from prior few-shot ones. Therefore, our ablation studies were conducted within our proposed paradigm.\n- To illustrate, let us consider the component differences between RCI and ours in details:\n| | Number of shots | Consistent screen | Planner | #Reflection |\n|-----|-----|-----|-----|-----|\n| RCI | Few to many | No | Iterative | 1 |\n| Ours | 0 | Yes | Staged | N |\n  - **On the number of shots and screen consistency**. There is likely no easy approach to reduce RCI from few-shot to zero-shot. This correlates to their use of inconsistent screen representations. We manually inspected RCI’s shots on their Github repo and found that the agent relies on those given demonstrations to predict the id of an element on an unseen screen, so that the generated action can be properly grounded on the environment. That is, the existing use of few-shot prompts relies on the use of inconsistent screens. These two components are bonded together.\n  - **On the choice of planner**. There might be performance differences to tell but the completion rate on 1-screen-n-step tasks is already saturated, the gap is likely minimal. Our argument on this component is actually to simplify the process: staged planning is already good enough when you need multiple steps on a single screen, so one doesn't need iterative planning due to the efficiency cost (evidence is in our response to your question B).\n  - **On the rounds of reflection**. Broadly speaking, RCI could align to our agent with at most 1 round of reflection while our agent can do more rounds. Beyond this difference is the structured reflection (ours) vs plain reflection (RCI). On this component, we already have ablation study in the paper and the performance gap is substantial (i.e., num_trial=1 vs num_trial=3 or 5).\n  - **On the reflection strategy**. We have presented ablation study against an adaptation of the original reflection (from Reflexion 2023 paper) in Fig 7. The conclusion is that our structured thought management is a beneficial add-on component.\n\n> Indeed, given PaLM-2’s general lack of availability to the larger community, it’s not clear at all whether the gains come from the underlying LLM’s capability, or from the contributions described above.\n- Recent advances in LLM agents (e.g., CoT (NeurIPS22), ReACT (ICLR23), and Reflexion (arXiv 23)) have shown that different LLMs (i.e., PaLM, GPT-3/4, Codex) generally exhibit a common capacity to benefit from intermediate thoughts and self-reflection. We believe there is a reasonable adaptation of our findings on other LLMs.\n- There are probably some extents of prompt dependency on LLM. We believe this is expected since switching LLMs sometimes requires adapting prompt/instruction. But we want to emphasize that the proposed methodology is conceptually orthogonal to LLM choices.\n\n> It feels as if this is a paper that is presenting a series of “modules” that can be grafted on to many existing LLM-based web agents; if the contribution of this paper (as I see it) are these modules, then more work needs to be done to verify that these modules are generally helpful in settings beyond just the current system implementation.\n- We believe the answer is it depends on the use case. This work is actually **the first zero-shot agent** in the line. Our design considerations are focused on zero-shot use case which is different from prior few-shot works.\n- With recent LLMs generally showing a degree of capacity to reflect and use intermediate thoughts, we believe our structured reflection has a good promise to generalize across different LLMs.\n\n> As a minor weakness — I’m not sure that I agree about the “compact” screen representation relaxing assumptions on the environment, or forcing the agent to behave more/less rationally when given limited information.\n- We will improve the presentation. The motivation mainly has 3 aspects:\n  1) The compact representation is essentially a simplification of the original HTML which tends to be long and verbose. This poses a practical limitation to existing LLM input length. In our case, reflection requires seeing the complete trajectory that could span over multiple screens. This often runs out the input length limit for PaLM-2. Thus, shorter screen code is preferred.\n  2) Following [1], compact screen HTML codes can give compelling results on UI understanding tasks, suggesting it’s a good alternative. We have this citation in the reference.\n  3) Lining up the separation of valid/invalid screen HTML is important for evaluation. As we have seen that many of the MiniWoB++ tasks are already easy for today’s LLM. Exposing more unseen elements risks hiding the actual challenge in the navigation tasks. For instance, exposing unseen elements basically simplifies n-screen-n-step tasks into 1-screen-n-step ones. However, our experiment shows that n-screen-n-step ones are actually much harder to deal with.\n\n[1]: Enabling Conversational Interaction with Mobile UI using Large Language Models. Wang et. al. CHI 2023\n\n> Question A. Algorithm 1 is very very hard to follow\n- We will add an annotated version of Algo 1 to the draft. The algorithm works in an intuitive way. It starts with a loop over a number of trials. At each trial,\n  - Line 4: it checks if there is a reflection statement from previous trials. \n  - Line 5-6: If there is any, it identifies if there is a suggested action to heuristically take, or sample it from LLM.\n  - Line 7: Such action is executed on the environment. If an error occurs,\n  - Line 8: it starts to reflect by observing the entire action trajectory, and output a statement consists of (a, a’).\n  - Line 9-10: the wrong action is recorded and heuristically disabled on the environment, if applicable.\n  - Line 11: the verbal form of reflection statement is also recorded, so that LLM can use it as input during planning.\n  - Line 12: with a new action a’ suggested for timestep j, previous suggestions after timestep j can be considered outdated, thus memory is cleared accordingly.\n\n> Question B. It’s not immediately intuitive why the staged action prediction is better than closed loop (predict action iteratively, one after the other) in this approach? Can you provide more intuition or motivating examples that show when closed-loop action prediction fails, and staged action prediction succeeds?\n- For intuition on concrete examples, we recommend to try out this [demo](https://miniwob.farama.org/environments/click-checkboxes-soft/). We want to clarify that the major motivation of our staged planning is actually **efficiency** instead of accuracy.\n  - In iteratively planning, an agent will need to query LLM for each click on the screen.\n  - In our staged planning, all items to select are issued in one query.\n  - These two approaches both ceil at ~100% on 1-screen-n-step tasks. Therefore we propose to do planning more efficiently.\n-  Here, we show our staged planner has substantially better efficiency:\n| | 1st successful trial (25 ex per task) || Up to 3 trials (25 ex per task) ||\n|-----|-----|-----|-----|-----|\n| Task | #actions | Planing call reduction | #action | Planning call reduction |\n| click-checkboxes-large | 234 | 89.7% | 270 | 88.5% |\n| click-checkboxes-soft | 81 | 76.5% | 167 | 61.7% |\n| multi-layout | 175 | 85.7% | 175 | 85.7% |\n| social-media-all | 114 | 82.5% | 224 | 71.4%|\n- The planning call reduction is calculated by comparing the number of staged planner calls vs the number of iterative planner calls. The number of iterative planner calls is essentially the number of actions since each action requires a round of LLM inference. Clearly, our staged planner offers a substantial planner call reduction. Thus, it is efficient as we claimed. We will add this result to the final draft to better motivate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The systems contribution in this work is strong; each of the proposed components is well-motivated (if not properly ablated), and the core nugget in the results — that the proposed agent can solve these tasks without finetuning or in-context examples — is clearly impressive and a step forward.\n\nI think this work also does a great job of codifying the prior work in this area, and different assumptions made in building LLM-based web agents.", "reasons_to_reject": "As hinted at above, I think there are several design decisions that are not properly ablated. One general criticism I have for recent work in LLM-based agents that rely on prompting approaches is verifying that new methods work for the same LLMs used in prior work (where possible). This work makes the choice to use PaLM-2; however, all prior methods cited as baselines in the tables are (I believe?) not run with PaLM-2, instead reporting results from the original papers, and original LLMs (e.g., GPT-3.5, text-davinci-002). Indeed, given PaLM-2’s general lack of availability to the larger community, it’s not clear at all whether the gains come from the underlying LLM’s capability, or from the contributions described above.\n\nFollowing the above general criticism, it would be nice to ablate specific components of the proposed approach vs. existing work. How would a method like RCI work if it used the structured reflection module instead of the naive single-step reflection feedback? How would AdaPlanner perform with the staged action prediction component? \n\nIt feels as if this is a paper that is presenting a series of “modules” that can be grafted on to many existing LLM-based web agents; if the contribution of this paper (as I see it) are these modules, then more work needs to be done to verify that these modules are generally helpful in settings beyond just the current system implementation. \n\n---\n\nAs a minor weakness — I’m not sure that I agree about the “compact” screen representation relaxing assumptions on the environment, or forcing the agent to behave more/less rationally when given limited information. I think these claims need to be tempered a bit in the paper — the existing LLM agents are already “unnatural” in that they are operating over HTML vs. visual screens... not sure we should further restrict what HTML is “valid/invalid” for an agent to see (dumping page source, depending on webpage, can result in many different plausible elements). I think it's better instead to scope down this claim to the results observed in evaluating the proposed system with it’s compact representation.\n\n---\nEDIT (Post-Rebuttal): Given the authors clarifications and extra discussion provided during the rebuttal, I am revising my soundness score upwards to a 4!", "questions_to_authors": "A. Algorithm 1 is very very hard to follow; can we add a more annotated version of the algorithm (or better, a visual that captures exactly what’s happening in the structured reflection module)?\n\nB. It’s not immediately intuitive why the staged action prediction is better than closed loop (predict action iteratively, one after the other) in this approach? Can you provide more intuition or motivating examples that show when closed-loop action prediction fails, and staged action prediction succeeds?", "paper_topic_and_main_contributions": "This paper presents a systems contribution in a novel LLM-based agent for interacting with computer environments (specifically, internet web pages a la the MiniWoB++ benchmark) zero-shot, solely learning through the feedback received from the web interface/sparse task success. \n\nThe proposed approach built on top of the PaLM-2 language model is novel in that it requires no finetuning or in-context examples to perform tasks, and is able to obtain high task success rate through a combination of the following contributions:\n\n- A compact (reparameterized) screen representation that simplifies the HTML visible by the LLM as context.\n- A staged action planner that allows the LLM to output *multiple* actions at a given timestep/context in a way that enables better temporal consistency.\n- A structured reflection module that uses an external structure to logically prune out parts of historical action traces whenever a task is failed, aiding in replanning the next task attempt.\n\nWhile a generally complex paper to follow (echoing the complexity of the underlying system), the results show that the proposed approach can not only obtain high success rates on “simple” tasks from MiniWOB++ (requiring only a single step to solve), but that the structured reflection + staged planning components enable strong performance on more complex tasks (more contextual, more turns), beating or coming close to the performance of baseline approaches that are trained with supervision or are enriched with few-shot examples.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "xVSVnSV7mI", "reviewer": "Reviewer_7sGp", "summary": "", "questions": "- Question A: You describe the 3 action types, but how many actions are on average necessary to solve an environment? Does 3 trials performance improve (in some cases) so drastically, because maybe only a (correct) single action is left after 2 trials?\n- Question B: Does the good performance of the programmatic reflexion approach mean that this is what current models lack? A memory module that keeps track of what is going wrong during an interaction?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging remarks and insightful feedback. We believe addressing the points you raised will improve the clarity our paper and strengthen our contributions in the revision.\n\n> the contribution would be stronger, when the \"thought management\" (excluding failed actions) would be learnt (and not programmed)\n- We do agree that a more end-to-end neural model for thought management is very interesting. Our work essentially shows that proper thought management is a beneficial add-on component. Thus, learning such neural module can be the next step. Such neural module can inform LLM towards finishing a given task in one trial. This direction still an open question in the line of reflection works.\n\n> Question A: You describe the 3 action types, but how many actions are on average necessary to solve an environment? \n- We coarsely bucket the expected number of actions to finish a task into [0, 4), [4, 7), and [7, +). In the below table, we show that there is indeed a pattern between action length and task completion rate.\n(Note that 1-screen-1-step tasks are omitted since they only require 1 action.)\n| | <=4 | [4, 7) | >=7 |\n|-----|-----|-----|-----|\n| task | 13 | 15 | 5 |\n| Completion rate | 94% | 91.6% | 90.4% |\n- However, we should also note that required action length is only one of the factors in task complexity. For instance, in the search-engine task, an average of 6 steps is required. But the agent performance is substantially lower than the click-checkboxes-large task which requires 10+ steps to finish. This is due to search-engine involving unseen screens so that agent has to explore and plan on the fly.\n\n> Questions A: Does 3 trials performance improve (in some cases) so drastically, because maybe only a (correct) single action is left after 2 trials?\n- We believe this is not the case since that, during each trial, our agent will maximally attempt on the task until a failure signal is raised. It does not stop in the middle of execution if actions are properly grounded on the environment.\n- The drastic improvements come from our reflection module. The agent learns (verbally) to identify prior errors and correct them across multiple trials.\n\n> Question B: Does the good performance of the programmatic reflexion approach mean that this is what current models lack? A memory module that keeps track of what is going wrong during an interaction?\n- We genuinely think so too. A helper memory management module for LLM is essentially what we proposed in the draft.\n- A future direction is to benefit from prior errors and corrections for better completion rate in one trial. And yes, this can be potentially managed by a separately trained memory module. In general, it is an interesting and still open question in the line of reflection works.\n\n> References: could need another round for camera-ready (resolve arxiv refs if possible)\n- We will improve citations in the revision.\n\n> Reproducibility: seeds are not reported\n- As mentioned in Sec 6, we use seeds 1000-1024 following prior works (e.g., Pix2Act) to start seeds from 1000 for testing.\n\n> Reproducibility: I assume Palm-2 was accessed via an API, so the version and/or date should be reported\n- We will report the version of PaLM and date it was ready for access. We want to emphasize that, while different LLMs are known to behave differently, our proposed method is conceptually orthogonal to LLM choices.\n\n> Are always all trials used?:\n- Not aways. The number of trial is the maximum trials our agent will attempt on a task. Once a task is completed by a trial, later trials will not be conducted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a well-written and easy to follow study on zero-shot usage of a pre-trained LLM for computer control tasks\n- manual evaluation of the existing MiniWOB++ tasks which reveal problems in the way these environments have been approached before\n- the paper fits nicely into the (well explained) research field of computer control and indicates further directions for the community (\"thought management\")", "reasons_to_reject": "- the contribution would be stronger, when the \"thought management\" (excluding failed actions) would be learnt (and not programmed)\n- only results for a single proprietary model (PaLM-2) and none for open-access models (from e.g. huggingfaces)", "questions_to_authors": "- Question A: You describe the 3 action types, but how many actions are on average necessary to solve an environment? Does 3 trials performance improve (in some cases) so drastically, because maybe only a (correct) single action is left after 2 trials?\n- Question B: Does the good performance of the programmatic reflexion approach mean that this is what current models lack? A memory module that keeps track of what is going wrong during an interaction?", "paper_topic_and_main_contributions": "This work investigates the use of a PLM (PaLM-2) for computer control tasks (MiniWOB++) in a zero-shot setting (no further training). \n\nThe authors find in a manual evaluation of 13 MiniWOB++ tasks/environments that at least 5 of these could be solved by using information that is contained in the HTML (of the webpage) although the information is not visible (in the browser). Thus the authors suggest to expand elements in the HTML only when they are actually accessible in the UI (still text is the only modality) to make the tasks more aligned with human perception (and more difficult for LLMs).\n\nIn addition the authors \n- (a) suggest to produce actions only for the screen at hand (\"staged plan-and-follower\", in contrast to other work that produces actions for the whole problem) and \n- (b) propose a programmatic extension to the Reflexion approach (Shinn et al. 2023) so that actions (click element with id, insert text to element with id, press key) that have lead to unsuccessful outcomes are disabled for later trials.\n\nThey test their modifications with the PaLM-2 model for different number of trials (1, 3, 5) on a sub-selection of 43 MiniWOB++ environments (excluded are those that require vision) and compare the success rates with other works (WebN-T5, CC-Net, RCI, AdaPlanner). The authors find that their approach can solve easy (1-screen:1-step) and medium (1-screen:n-step) environments in all cases (except an ambiguous case). And is performing similarily to few-show approaches (RCI, AdaPln) for the hard (n-screen:n-steps) environments. Still, the supervised approach (CC-Net) is reaching the best performance.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L196: actions could be defined more precisely (or pointed to the appendix)\n- L445: \"the 9 tasks\" -- better \"9 tasks\" because this confuses with Table 4 where 10 tasks are indicated\n- L490: \"out\" -- \"our\"\n- 7.3. social media task could need a better task description (what has to be done) to better understand the numbers in Table 7\n- Table 7 is not intuitively to read (10+?), captions more elaborate on this\n- performance should be somehow weighted by the number of necessary trials (or are always alls trials used?)\n- The proposed approach reminds mind me of Hierarchical RL: Learning a (sub-) policy for each room (screen) in an environment\n\nReferences: \n- could need another round for camera-ready (resolve arxiv refs if possible)\n\nReproducibility:\n- seeds are not reported\n- I assume Palm-2 was accessed via an API, so the version and/or date should be reported"}}, {"review_id": "eNgRjDHSc9", "reviewer": "Reviewer_HJRY", "summary": "", "questions": "Table 5 does not reflect the situation of iterative planning v.s. staged planning very well, but it does show that trying three times is better than trying once?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We believe most of your concerns are not due to the limitations of our work, and they can be adequately addressed by improving our writing in the revision. \n\n> For the MINIWOB++ environment, the author removed tasks that require visual signals and clicks to complete. The obvious drawback is the lack of image modality information. As of mid-2023, there are not many commercially available LLM APIs or open-source LLMs that can handle large-scale image input capabilities. Once a more powerful multimodal LLM appears, this type of pure text input work will become meaningless.\n\nWhile we understand multimodal LLM can take additional input such as images, we respectfully disagree that our approach \"will become meaningless\". Our approach for enabling zero-shot capability for agents via structured self reflection is independent of specific input modalities available. Because this is the first work on the topic, we base our work on a language agent because language-based LLM affords a mature base model. Meanwhile, the lack of compelling performance from recent multimodal LLM is a practical limitation, so at the time of this submission, LLM is still a strong choice. Even with recent multimodal design, explicit reasoning still takes place in the form of language. Thus our proposal could benefit such models that are still on the way. We will add such discussion to the final draft.\n\n> Furthermore, since this work does not involve LLM training or reinforcement learning, the design of these mechanisms to manipulate computers is limited by the LLM used. In this paper, the author used PaLM2-L, which can only reflect the ability of PaLM2-L in this task. \n- Our work is actually in the line of the emerging in-context learning and verbal-formed reinforcement learning, such as CoT (NeurIPS22),  ReACT (ICLR23), and Reflexion (arXiv 23). We believe this line of research has been highly valued by the community.\n- There is a shared common capacity to reflect and use intermediate thoughts in recent LLMs. This pattern suggests that our reflection strategy can benefit other LLMs too. While our experiments are focused on PaLM2, the conceptual design of our agent is independent of specific LLM choice.\n\n> LLM can try and error in the environment, but cannot continuously learn from trial and error to avoid errors in the first attempt next time. This mechanism designs themselves cannot long-term improve LLM's ability to manipulate computers. This makes the article look like just a report for a prompt engineering attempt.\n- Long-term or lifetime learning for LLM is still an open question. We definitely did not argue we can solve it in this paper. Our work can be considered a step towards that goal since we showed that an agent model can already learn and improve itself in a short-term period of task execution.\n- Using reflections towards better performance on the first trial is still an open question in the line of reflection works. We genuinely believe the community is on the right path to solve this problem. Firstly, we need to show agents can learn from their own mistakes in computer control problems. Next, we can explore ways to benefit from such mistakes towards better execution on the first trial.\n- Our core contributions (as in the Intro section) is the first zero-shot agent design with structured reflection module. Prompt engineering is part of our work but is not the core.\n\n> At the same time, some mechanisms designed in the article are not very elegant. For example, to prevent repeating errors that have already occurred in a task, the author \"only remove the id attribute, and still allow the element information to be presented to LLM.\" This approach has a certain degree of randomness and no rationale, and cannot be used as a broad conclusion.\n- Our design follows the intuition behind induction heads [1]. Removing the id attribute from the input essentially discourages the LLM to issue actions on that element, therefore the same errors can be better avoided.\n- Removing the entire element could make the screen representation incomplete thus bringing noise and difficulty for LLM to understand the screen context. Our design choice is therefore only to remove the id fields as it is a **minimal** change to the input.\n- We conduct an ablation on the action constraining component below, showing it's a beneficial add-on. We focus on the challenging n-screen-n-step category and the tasks in Fig 7 except the find-word task since it's not centered on click action.\n| Task (num_trial=3) | no constraining action space | with constraining action space |\n|-----|-----|-----|\n| click-menu-2 | 72%  | 72% |\n| click-tab-2 | 56% | 60% |\n| click-tab-2-hard | 52% | 56% |\n| search-engine | 60% | 64% |\n\n[1]: In-context Learning and Induction Heads. Olsson et. al. 22\n\n> The sections describing the proposed methods in the article, Section 4 and Section 5, are interrupted by one reference article after another. The authors’ original intention was to compare their own methods with others, but this breaks the coherence of reading. In addition, the section proposing the method should add an overall figure to describe the relationship between these subsections and their help in completing different aspects of the task.\n\n- Thank you for the suggestions for improving the presentation. There is a wide spectrum of related works for the planning strategy, which we feel important to overview and thus motivate our proposed planning strategy. We will incorporate your suggestions for our revision.\n\n> Question: Table 5 does not reflect the situation of iterative planning v.s. staged planning very well, but it does show that trying three times is better than trying once?\n- Yes num_trial=3 is better than num_trial=1.\n-  The major motivation behind our staged planner is the efficiency (instead of accuracy since both planning strategies ceil at close to 100%). Here, we show our staged planner has substantially better efficiency:\n| | 1st successful trial (25 ex per task) || Up to 3 trials (25 ex per task) ||\n|-----|-----|-----|-----|-----|\n| Task | #actions | Planing call reduction | #action | Planning call reduction |\n| click-checkboxes-large | 234 | 89.7% | 270 | 88.5% |\n| click-checkboxes-soft | 81 | 76.5% | 167 | 61.7% |\n| multi-layout | 175 | 85.7% | 175 | 85.7% |\n| social-media-all | 114 | 82.5% | 224 | 71.4%|\n- The planning call reduction is calculated by comparing the number of staged planner calls vs the number of iterative planner calls. The number of iterative planner calls is essentially the number of actions since each action requires a round of LLM inference. Clearly, our staged planner offers a substantial planner call reduction. Thus, it is efficient as we claimed. We will add this result to the final draft to better motivate it.\n\n>  In Table 6, the author attempted to explain whether having LLM reflect on performance could improve it. At most, after trying five times, although it did not surpass the best existing supervised method, it did not perform significantly better than the Few/N-shot method. Can these subtle differences be attributed to the different abilities of LLM itself, and it is difficult to say how much the Reflection Strategies designed by the author play a role in this.\n- Our proposed reflection strategy plays a substantial role. This comparison is essentially the num_trial>1 vs num_trial=1 across Sec 6. We also adapted the original verbal reflection to compare with our structured thought management. Results in Fig 7 suggests that our structured reflection strategy is a beneficial add-on.\n- We also want to point out that moving from few/N-shot to zero-shot is a substantial challenge. We presented performances from supervised and few/n-shot agents to show where our agent is on the benchmark, however, they are not in direct comparison since the strategies are different.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Similar work on input mechanism design includes process design and prompt design. The author successfully designed a process with CoT and Reflection, enabling LLM to find a feasible action sequence from trial and error.", "reasons_to_reject": "For the MINIWOB++ environment, the author removed tasks that require visual signals and clicks to complete. The obvious drawback is the lack of image modality information. As of mid-2023, there are not many commercially available LLM APIs or open-source LLMs that can handle large-scale image input capabilities. Once a more powerful multimodal LLM appears, this type of pure text input work will become meaningless.\n\nFurthermore, since this work does not involve LLM training or reinforcement learning, the design of these mechanisms to manipulate computers is limited by the LLM used. In this paper, the author used PaLM2-L, which can only reflect the ability of PaLM2-L in this task. LLM can try and error in the environment, but cannot continuously learn from trial and error to avoid errors in the first attempt next time. This mechanism designs themselves cannot long-term improve LLM's ability to manipulate computers. This makes the article look like just a report for a prompt engineering attempt.\n\nAt the same time, some mechanisms designed in the article are not very elegant. For example, to prevent repeating errors that have already occurred in a task, the author \"only remove the id attribute, and still allow the element information to be presented to LLM.\" This approach has a certain degree of randomness and no rationale, and cannot be used as a broad conclusion.\n\nThe sections describing the proposed methods in the article, Section 4 and Section 5, are interrupted by one reference article after another. The authors’ original intention was to compare their own methods with others, but this breaks the coherence of reading. In addition, the section proposing the method should add an overall figure to describe the relationship between these subsections and their help in completing different aspects of the task.", "questions_to_authors": "Table 5 does not reflect the situation of iterative planning v.s. staged planning very well, but it does show that trying three times is better than trying once?", "paper_topic_and_main_contributions": "This work is a type of input mechanism design for LLM. Since the input length of LLM is limited, the content of the input should be organized more effectively.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 4 and Figure 5 are not vector graphics and some are blurry.\nIn Table 6, the author attempted to explain whether having LLM reflect on performance could improve it. At most, after trying five times, although it did not surpass the best existing supervised method, it did not perform significantly better than the Few/N-shot method. Can these subtle differences be attributed to the different abilities of LLM itself, and it is difficult to say how much the Reflection Strategies designed by the author play a role in this."}}]

Author rebuttal:

Meta-review: Strengths of the paper include the systems contribution with well-motivated individual components. The paper is also written well and clear and the reviewers thought the ideas themselves are useful in guiding the field and discovering novel problems. A potential drawback of the paper is that it was conducted with a proprietary LLM rather than an open-source one which would make it difficult to reproduce results.
Decision: Accept-Findings