SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conducted an evaluation of RCI prompting on eight datasets, encompassing two categories of reasoning tasks: arithmetic and commonsense. In the domain of arithmetic reasoning, we considered six datasets: SingleEq [34], AddSub [27], MultiArith [56], AQuA [35], GSM8K [9], and SVAMP [51]. For commonsense reasoning, we utilized the CommonsenseQA dataset [67] and the StrategyQA dataset [21]. To ensure a fair comparison with baselines, we specifically selected tasks that were employed in the work of Kojima et al. [33]. In the experiment on reasoning tasks, we enable RCI loop to get implicit feedback to correct outputs. We fix the maximum number of loops to 2. Following previous works [38, 62, 74], we use the correct label to decide when to stop the RCI loop. In our setting, we can consider the correct label to be another source of feedback (label feedback).
C.3 MiniWoB++ task selection
In order to ensure a fair and comprehensive evaluation, a subset of MiniWoB++ tasks we use in the evaluation is selected from the evaluation of WebN-T5-3B [24], the most recent work on MiniWoB++ tasks, which employs LLMs. However, certain tasks such as book-flight, choose-date-easy, choose-date-medium, choose-date, and click-pie have been excluded from our evaluation due to their HTML code exceeding the maximum context length of language models. On the other hand, some of the challenging tasks such as terminal and simple-algebra have been included in the evaluation. The choice of these tasks is determined by the suboptimal performance of CC-Net [30], which currently represents the state-of-the-art model in the field. The purpose of this inclusion is to showcase the potential of leveraging LLMs in computer tasks, in contrast to the conventional approaches of Supervised Learning (SL) and Reinforcement Learning (RL). While our agent has not been evaluated on tasks that necessitate additional actions, such as drag and copy & paste, we posit that their inclusion can be readily achieved through the expansion of the actions space specification within the prompts.
C.4 MiniWoB++ task selection for ablation studies
In ablation studies, we categorize the tasks based on the success rate achieved by our agent with gpt-3.5-turbo. We select a subset of tasks from three levels of difficulty, as depicted in Table 4.
easy [0.9, 1]	click-shape	0.98
click-widget	0.98
enter-date	0.96
medium [0.6, 0.9)	click-checkboxes-soft	0.72
click-collapsible-2	0.62
click-tab-2	0.74
hard [0, 0.6)	click-tab-2-hard	0.56
count-shape	0.4
guess-number	0.2
Table 4: The tasks used in the ablation study are classified according to their level of difficulty.
C.5 Modifications on MiniWoB++ tasks
In Table 5, we outline several modifications that were incorporated into the MiniWoB++ benchmark for the purpose of our evaluation with language models that have a limited context length.
Tasks	Modifications
social-media-all	We constrain the quantity of media components ranging from three to six.
social-media	We constrain the quantity of media components ranging from three to six.
social-media-some
email-inbox-forward-nl-turk	The quantity of randomly generated emails has been restricted to a range of three to six.
email-inbox-forward-nl
email-inbox-nl-turk
Table 5: Modifications on MiniWoB++ tasks.
D Prompts for MiniWoB++ tasks
We have an autonomous computer control agent that can perform a set of instructions to control computers.
First, given the instruction that matches the regular expression, <type regex>, it can type a list of characters via the keyboard. This instruction should specify the target keyboard input for the agent to type. Before this typing instruction, you should first locate the cursor by clicking the input box with the click instruction.
Second, given the instruction that matches the regular expression, <press regex>, it can press a specific key on the keyboard.
Third, given the instruction that matches the regular expression, <clickoption regex>, it can click an option HTML element in a list with an XPath that is visible on the webpage. The target of this instruction should be a valid XPath.
Fourth, given the instruction that matches the regular expression, <movemouse regex>, it can move the mouse cursor on an HTML element with an XPath that is visible on the webpage.
Lastly, given the instruction that matches the regular expression, <clickxpath regex>, it can click an HTML element with an XPath that is visible on the webpage. The target of this instruction should be a valid XPath.
Table 6: Agent specification.
Listing 1: Regular expressions for specifying the admissible actions.
<type regex> = "~type\s.{1,}$
<press regex> = "~press\s{enter|arrowleft|arrowright|arrowup|arrowdown}
backspace)$"
<clickoption regex> = "~clickoption\s.{1,}$
<movemouse regex> = "~movemouse\s.{1,}$
<clickxpath regex> = "~clickxpath\s.{1,}$
</clickoption regex>
Table 7: Task grounding.
Based on the current plan and task, the next proper instruction should be
Table 8: State grounding.
Considering the output on the webpage, the specific instruction should be
Table 9: Agent grounding.
This action does not match the regular expressions. The updated instruction that matches one of the regular expressions is
E Examples of cross grounding
task: click on the "ok" button on the webpage.
plan:
1. type "sit arcu nullam:"
2. clickxpath //button [@data-wob_ref="6"]
We have a history of instructions that have been already executed by the autonomous agent so far.
No instruction has been executed yet.
Based on the current plan and the task, the first instruction should be `type "sit arcu nullam:"`. Therefore, considering the output on the webpage, this instruction for solving the task should be `clickxpath //button [@data-wob_ref="6"]`.
Table 10: The state grounding optimizes the action plan. In the initial plan, the inclusion of the type instruction is deemed superfluous for accomplishing the task.
task: Find the email by Clarinda and reply to them with the text "Vel id at". plan:
1. clickxpath //div [@class='email-sender' and text()='Clarinda']
2. clickxpath //span [@class='email-reply']
3. clickxpath /*[@id="reply-text"]
4. type Vel id at.
5. clickxpath /*[@id="send-reply"]
0: clickxpath //div [@class='email-sender' and text()='Clarinda']
1: clickxpath //span [@class='email-reply']
2: clickxpath /* [@id="reply-text"]
Based on the current plan and the task, the next proper instruction after the history should be `type Vel id at`. Therefore, considering the output on the webpage, this instruction should be `type Vel id at`
Table 11: The state grounding corrects actions to be grounded in the task. Although the target of typing action in the action plan is "Vel id at." the generated task-grounded action lacked the last punctuation mark. Nonetheless, the state grounding update was successful in remedying this discrepancy, resulting in the state-grounded action being "Vel id at." Consequently, the state-grounded action now matches the typing action specified in the action plan.
F Failure modes on MiniWoB++ tasks
(a) click-dialog-2
(b) click-menu
Figure 7: Screenshot.
(c) count-shape
F.1 click-dialog-2
The state-grounding process is occasionally inadequate for maintaining task-grounding of actions. Specifically, in the click-dialog-2 task, grounding actions in the current state results in task-irrelevant actions. Consequently, we opt to exclude the state grounding step for this particular task.
task: click the button in the dialog box labeled "x".
1. Clickxpath //div [@class='ui-dialog-titlebar ui-corner-all ui-widget-header ui-helper-clearfix ui-draggable-handle']/button [@title='Close']
Based on the current plan and task, the first instruction should be `clickxpath //div [@class='ui-dialog-titlebar ui-corner-all ui-widget-header ui-helper-clearfix ui-draggable-handle']/button [@title='Close']`. Therefore, considering the output on the webpage, this instruction should be `clickxpath //div [@class='ui-dialog-buttonset']/button [@title='OK']` to click the button in the dialog box labeled "x".
Table 12: Example trajectory on click-dialog-2. The state grounding updates the task-grounded actions to be irrelevant to the task.
F.2 click-menu
We find that including explanatory trajectories in the few-shot examples can significantly reduce the number of required few-shot examples. Specifically, in the click-menu task, the selection of the target component necessitates the mouse hovering over the expandable menu. Therefore, we provided explanations regarding the mouse hovering technique in the few-shot examples.
F.3 count-shape
This task necessitates an agent to visually perceive rendered images on a canvas HTML element and enumerate the relevant shapes. Unfortunately, our agent encounters difficulties in tackling such tasks which entail the need for perceiving visual representations, as it solely depends on HTML observation.
task: select Leonelle  $>$  Jane  $>$  Amabel from the HTML code provided.
1. movemouse /*[text()="Leonelle"］// we need to hover the mouse to expand the menu
2. movemouse /*[text()="Jane" // we need to hover the mouse to expand the menu
3. clickxpath /*[text()="Amabel"]
task: select the option "Joye > Phylis" from the HTML code provided.
1. movemouse /*[text()="Joye" // we need to hover the mouse to expand the menu
2. clickxpath /*[text()="Phylis"]
Table 13: Example trajectory on click-menu. Providing explanatory trajectory enhances the sample-efficiency of few-shot examples.
task: answer the question "How many large items are there?"
1. Clickxpath //button[text()="2"]
Table 14: Example trajectory on count-shape. The agent struggle to solve tasks that requires visual rendering of HTML.
Figure 8: Screenshot.
F.4 guess-number
When considering decision-making that involves a long-term perspective, our agent encounters difficulties in generating actions that are grounded in the current task. For instance, in this guess-number task, the plan generated by our agent involves the sequential typing of numbers from 0 to 9 in order to solve the task. However, due to the high number of steps required (over 50), our agent occasionally produces actions that deviate from the original plan, ultimately resulting in irreparable failure.
F.5 login-user-popup
This task involves the identification of an optimal strategy for an agent to handle an unpredictable pop-up window that emerges during a task. Due to the unexpected nature of the pop-up window, pre-planning the closure of the window is not feasible. Our agent is designed to adapt the agent's actions to the current state, so it should generate an appropriate instruction to close the pop-up window in the state-grounding step. Nevertheless, there are instances where it is unsuccessful in executing the pop-up window closure, leading to task failure.
task: Guess the number between 0-9 and press Submit. Use the feedback below to find the right number.
1. clickxpath /*[@id="tt"]
2. type 0
3. clickxpath /* [@id="subbtn"]
4. clickxpath /*[:id="tt"]
5. press backspace
6.type 1
7. clickxpath /* [@id="subbtn"]
8. clickxpath /*
9. press backspace
10. type 2
11. clickxpath /*[@id="subbtn"]
12. clickxpath /*[@id="tt"]
13. press backspace
14. type 3
15. clickxpath /* [@id="subbtn"]
16. clickxpath /*[@id="tt"]
17. press backspace
18. type 4
19. clickxpath /*[@id="subbtn"]
20. clickxpath /*[@id="tt"]
21. press backspace
22. type 5
23. clickxpath /* [@id="subbtn"]
24. clickxpath /*[@id="tt"]
25. press backspace
26. ...
0: clickxpath /*[@id="tt"]
1: type 0
2: clickxpath /* [@id="subbtn"]
3: press backspace
4: type 1
5: clickxpath /* [@id="subbtn"]
Based on the current plan and the task, the next proper instruction should be `type 2`
Table 15: Example trajectory on guess-number. The agent fails to generate task-grounded action when long-horizon decision making is required.
F.6 tic-tac-toe
We also examine the causes of the poor success rate in playing tic-tac-toe. Two distinct factors are identified as responsible for its failure. The first factor is that our agent is unable to adopt a defensive strategy when its opponent is only one move away from victory and there is no immediate opportunity to win the game. The second factor relates to the inability to consider the possibility of its attack being blocked by the opponent. A case in point is the bottom-left illustration in Figure 9, where our agent's fourth move is ineffective since its diagonal direction is already obstructed by the opponent.
F.7 use-autocomplete
In use-autocomplete task, our agent demonstrated an ability to select words beginning with specific characters. However, it struggles when it comes to selecting words that ended with particular
Figure 9: Failure examples of tic-tac-toe task.
characters. This difficulty arose from the need to identify the correct word from the autocomplete list, a process that required the agent to press the down arrow button the exact number of times needed to locate the desired word within the list.
task: Enter an item that starts with "Se" and ends with "ia".
1. click the input field (e.g., clickxpath /*[@id="tags"])
2. type the starting word (e.g., type Se)
3. press the down arrow key to select the word ends with "ia" (e.g., press arrowdown)
5. select the word (e.g., press enter)
6. click the submit button (e.g., clickxpath /*[@id="subbtn"])
Table 16: Example trajectory on use-autocomplete. The agent fails to identify how many times it has to press the down-arrow key.
G Additional results
Arithmetic	Common Sense
GSM8K	MultiArith	AddSub	SVAMP	SingleEq	AQuA	CommonSenseQA	StrategyQA
Zero-Shot	77.95	94.48	88.58	80.70	86.61	60.23	64.56	48.42
Zero-Shot + RCI	72.83	86.61	83.07	79.13	83.07	59.84	46.06	48.81
Zero-Shot CoT	82.28	96.85	83.86	79.92	89.37	n/a	n/a	n/a
Zero-Shot CoT + RCI	74.40	87.79	83.07	79.13	83.46	n/a	n/a	n/a
Few-Shot CoT	80.31	98.82	89.37	83.46	91.73	n/a	n/a	n/a
Few-Shot CoT + RCI	71.65	93.70	83.46	78.74	83.46	n/a	n/a	n/a
Table 17: In the absence of external feedback, RCI prompting on reasoning benchmarks exhibits performance equivalent to, or below that of a zero-shot approach.
Figure 10: The task-level performance comparison with the state-of-the-art (SotA) baselines. The y-axis represents the residual values, which are obtained by subtracting the performance of SotA from our agent's performance.