SlowGuess/NeurIPS_2023_OCR huggingface dataset:
not recorded and primarily serves to familiarize the annotator with the website and task, as well as to prepare the website to prevent future pop-ups, thereby ensuring a clean, streamlined set of final recorded actions. After exploration, the annotator is directed to return to the homepage and reset any altered values, allowing us to begin the demonstration in a fresh state. During the demonstration, the annotator will illustrate how to accomplish the task step-by-step using both the browser and the dialogue window. To ensure a clean set of annotated actions, annotators are restricted from directly engaging with the browser during the demonstration phase. Instead, we divide each action step into two stages: Element selection and operation selection. At each step, the annotator first selects the target element by clicking it in the browser. We will highlight the selected element in the browser window but block the actual click event. The annotator is then prompted to select the operation to perform within the dialogue window, which is then carried out by the annotation tool in the browser. We provide 6 operations: Click, Type, Hover, Press Enter, Click (Fake) and Ignore. For the Type operation, the annotator is additionally required to supply the value as shown in Figure 9. If the chosen element is a select HTML element, and the annotator opts for Click, it translates to a Select Option operation and we will prompt the annotator to select one of the options as shown in Figure 10. To avoid ambiguity, the Click, Hover and Press Enter operations are all mapped to Click in the final dataset. Click (Fake) is a special operation. It will be recorded the same as a normal Click but will not get executed in the browser. This is designed for safeguarding against state-changing actions (i.e., actions that produce side effects to the world), such as posting a comment or scheduling an appointment, since it will interfere with other real users of the website. In practice, once a model predicts Click (Fake), it may prompt the user for confirmation before executing such state-changing actions. Finally, the annotator can also choose Ignore in case they select a wrong element. Once all the actions have been annotated, the annotator can choose to complete the task. They will then be asked to confirm the task description again and make any necessary modifications.
Pop-ups and CAPTCHAs. In this study, we emphasize on clean and direct task execution actions, intentionally omitting extraneous steps like pop-ups and CAPTCHAs that might introduce ambiguity in evaluation. We carefully select only those websites that pose no access issues when used with our tool. Before recording the task demonstration, annotators are requested to familiarize themselves with the website, and preemptively close pop-up windows and clear CAPTCHAs to avoid their recurrence during the actual demonstration. Annotators are further guided not to engage in extra steps such as closing ads unless necessary during the task demonstration. In the final task verification, we revisit the actions and filter out those unrelated to direct task execution. At the same time, we acknowledge that these instances constitute a significant aspect of the dynamic web environment in the real world. Enhancing systems to robustly tackle such scenarios on-the-go could form an interesting avenue for future research.
Mitigating Disruptions on the Websites. The annotator are advised against actions that could potentially interfere the normal operation of the website. To handle tasks such as scheduling appointments, we introduce a Click (Fake) operation that annotators can utilize to indicate the action without actually executing it on the website.
B.4 Task Verification
All collected data undergoes an additional verification process conducted by the authors, as demonstrated in Figure 11. The verification interface is shown in Figure 11. This verification consists of three tasks. Firstly, we evaluate whether a task should be discarded due to its low quality. Secondly, we examine each step to determine if any action should be discarded. This includes reviewing the
Figure 11: Illustration of our verification tool.
initial and final actions of the task, and excluding any additional actions (e.g., closing ads) that are not outlined in the task description to ensure consistency across task annotations. Finally, we verify the task description to confirm that all actions are accurately represented and make modifications if necessary. If there is uncertainty regarding any action, the verifier can opt for the 'unsure' option, prompting a re-evaluation by the first author.
C.1 Evaluation
One complication that arises during evaluation on real-world websites is that multiple elements on a webpage may induce the same effect. For instance, a button might house a text span within it, both of which, when clicked, yield identical results. To enhance the robustness of our evaluation, we employ heuristics to detect elements equivalent to the ground truth. We first examine the ancestors of the labeled element to identify potential higher-level elements acceptable for the current action. We employ a straightforward heuristic that locates the nearest clickable element to the ground truth, including itself. After identifying the top-level acceptable element, we include all its visible descendants that are located within its post-rendering bounding box as acceptable as well. Manual checking on 100 instances where the heuristic identifies a top-level element other than the ground truth confirms the validity of the approach. For both training and evaluation stages, all acceptable elements are considered positive.
C.2 Model Implementation Details
Candidate Generation. We use the Cross-Encoder implementation from Sentence-Transformers  ${}^{4}$  and use DeBERTa as the backbone model. More specifically, we use DeBERTa-v3-base  ${}^{5}$  for our experiments.
Action Prediction. We use the Seq2Seq model implementation from Transformers [38]. We experiment with the base  $^{6}$ , large  $^{7}$  and x1  $^{8}$  versions of Flan-T5 [10].
<sup>4</sup>https://www.sbert.net/examples/applications/cross-encoder/README.html
<sup>5</sup>https://huggingface.co/microsoft/deberta-v3-base
<sup>6</sup>https://huggingface.co/google/flan-t5-base
<sup>7</sup>https://huggingface.co/google/flan-t5-large
<sup>8</sup>https://huggingface.co/google/flan-t5-x1
Table 4: Hyperparameters used in experiments.
Method	Model	Hyperparamerters
Candidate Generation	deberta-v3-base	batch_size:32, epoch:5, learning_rate:3e-5
Action Prediction Generation	flan-t5-base	batch_size:32, epoch:5, learning_rate:5e-5
MINDACT	flan-t5-base, flan-t5-large, flan-t5-x1 gpt-3.5-turbo, gpt-4	batch_size:32, epoch:5, learning_rate:5e-5 temperature:0, # demonstrations:3
Table 5: Step Success Rate for Flan-T5 models with different groups of options. Here we shown mean and standard deviation of 5 runs with different random seeds.
Cross-Task	Cross-Website	Cross-Domain
Flan-T5B	41.5±0.7	30.0±0.8	31.3±0.5
Flan-T5L	49.9±0.2	35.7±0.5	36.7±0.3
Flan-T5XL	51.9±0.8	39.5±0.2	39.6±0.2
LLM In-context Learning. We use the OpenAI API for in-context learning with LLMs. We experiment with two versions of GPT models: gpt-3.5-turbo and gpt-4. We include three demonstration examples for in-context learning. The complete prompt is shown in Table 8.
Training Details. The flan-t5-x1 and flan-t5-large models are trained on servers with 4*A100 80GB cards provided by Ohio Supercomputer Center [6]. All other models are trained with single A6000 48GB cards.
Please see Table 4 for all hyperparameters used in our experiments.
D.1 Effect of Random Grouping Elements for Action Prediction
For both training and inference, we shuffle the elements in the webpage and randomly group them into multi-choice questions. The model might give different predictions when presented with different sets of choices, and leads to slightly different final evaluation scores. Here we show the average and standard deviation of 5 runs with different random seeds to show the effect of random grouping. As we can see from Table 5, the selection of choices only lead to small changes in overall performance with standard deviation less than 1 for all runs.
D.2 Zero-shot Results for Flan-T5XL
Since Flan-T5 is tuned with multi-choice format, it can also do element selection in zero-shot. However, as we can see from Table 6, while the model still gets some elements correct, it is much lower compared to the fine-tuned model, and 3-shot GPT 3.5/4. This is expected, since Flan-T5 is not tuned for HTML and coding related tasks.
Table 6: Zero-shot element selection results for  $\mathrm{{Flan}} - {\mathrm{T}}_{5\mathrm{{XL}}}$  compared with the fine-tuned counterpart.
Cross-Task	Cross-Website	Cross-Domain
Flan-T5XL Zero-Shot	10.8	7.8	11.7
Flan-T5XL Fine-Tuned	52.0	38.9	39.6
Table 7: Step Success Rate for all methods on the 50 tasks subsets we used to evaluate GPT-4. Numbers in parentheses are the results on the full test set (same as Table 2)
Cross-Task	Cross-Website	Cross-Domain
Flan-T5B	43.3 (41.0)	25.3 (29.5)	28.1 (31.6)
Flan-T5L	48.1 (50.3)	30.8 (35.3)	27.6 (37.3)
Flan-T5XL	47.9 (52.0)	33.3 (38.9)	34.6 (39.6)
GPT-3.5	15.2 (17.4)	15.1 (16.2)	16.7 (18.6)
GPT-4	36.2	30.1	26.4
D.3 Results on the 50 task subsets
Due to budget constraint, we only run GPT-4 on 50 tasks for each setting. Here we show the step success rate results for other methods on the same 50 examples that GPT-4 is tested on. As we can see from Table 7, the results on the 50 tasks subsets are consistent with the results on the respective full test set, and the relative performance across methods and splits remains the same.
Table 8: Prompt for action prediction in MINDACT with GPT models. Only part of the HTML snippet is shown here to save space.
Role	Content
system	You are a helpful assistant that is great at website design, navigation, and executing tasks for the user
user	...
<html><div><div><a tock home page /> <button id=0 book a reservation. toggle open> <span> Book a reservation </span> </button> <button book a reservation. toggle open> </button> </div> <div><select id=1 type> <option reservations true> Dine in </option> ... </html>
Based on the HTML webpage above, try to complete the following task:
Task: Check for pickup restaurant available in Boston, NY on March 18, 5pm with just one guest
Previous actions:
None
What should be the next action? Please select from the following choices (If the correct action is not in the page above, please select A. ‘None of the above’):
A. None of the above
B. <button id=0 book a reservation. toggle open> <span> Book a
C. <select id=1 type> <option reservations true> Dine in </option> <option
D. <div id=2> <p> Celebrating and supporting leading women shaking up
assistant	Answer: C.
Action: SELECT
Value: Pickup
user	...
<html><div><main main><section tabpanel><div><ul tablist><li tab heading level 3 search and> </li><li id=0 tab heading level 3 search and> </span> Hotel </span> </li> <li tab heading level 3 search and> </li><li tab heading level 3 search and> </ul> <div tabpanel><div id=1> <div><span> Dates* </span>
<button button clear dates /> </div> ... </html>
Based on the HTML webpage above, try to complete the following task:
Task: Compare the fare types to book a I-adult ticket from Springfiels, IL to Austin, TX for April 29th 2023
Previous actions:
[combobox] Enter your departing city, airport name, or airpor... -> TYPE: SPRINGFIELD
[button] Springfield, IL, US (SPI) -> CLICK
[combobox] Enter your destination city, airport name, or airp... -> TYPE: AUSTIN
[button] Austin, TX, US (AUS) -> CLICK
What should be the next action? Please select from the following choices (If the correct action is not in the page above, please select A. ‘None of the above’:
A. None of the above
B. <li id=0 tab heading level 3 search and> <span> Hotel
C. <div id=1> <div> Dates* </span> <button button clear dates
D. <ul id=2> <a mobile tools> </a> <a open united's tiktok
assistant	Answer: A.
Continued on next page
Table 8 - continued from previous page
Role	Content
user	...
<html><div><nav main menu><ul><div button> Car Sales </div><div id=0><div><div> Buy A Car </div><div> Plan Your Purchase </div></div><div><h4> Its Tax Refund Time. Treat Yourself to an Upgrade.</h4></p> With a variety of options, invest your refund in what you really want - a quality, used vehicle from Enterprise.</p> ... </html>
Based on the HTML webpage above, try to complete the following task: Task: Find a mini van at Brooklyn City from April 5th to April 8th for a 22 year old renter. Previous actions: [searchbox] Pick-up & Return Location (ZIP, City or Airport) (... -> TYPE: Brooklyn [option] Brooklyn, NY, US Select -> CLICK What should be the next action? Please select from the following choices (If the correct action is not in the page above, please select A. 'None of the above'): A. None of the above B. <div id=0><div><div> Buy A Car </div><div> C. <div id=1> Enterprise Fleet Management </div> D. <button id=2 selected pick-up date 03/19/2023><span> 19 </span>
assistant	Answer: D. Action: CLICK