SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To understand the impact of rollouts and value estimates using the value network, in Table 3 we compare mean scores over 12 challenging MiniWob++ tasks for different values of  $\lambda$ : 0 (rollout only), 0.1 (both rollout and value network), and 1 (value network only). We also include the mean score using the greedy policy for reference. These results use the policy network and value network trained on the human demonstrations. The results show that using a combination of rollouts and the value network gives the best results. The value network is primarily useful for challenging tasks that require longer trajectories, such as number-checkboxes, relative to using rollouts only.
Greedy Policy	λ = 0 (rollout only)	λ = 0.1	λ = 1 (value network only)
28.8	74.2	78.3	57.4
Table 3: Mean scores for different policies over 12 challenging MiniWob++ tasks.
Once we have completed  $K$  rounds,  $\pi_{\theta}^{*}(s)$  selects the most visited action  $a$  for state  $s$ , and we begin the process again at the subsequent state. We reuse the search tree for subsequent time steps for efficiency, so we require only  $K - n(s,a)$  additional rounds for the subsequent state.
Policy improvement We can sample trajectories with  $\pi_{\theta}^{*}$ , then update  $\theta$  by training  $\pi_{\theta}(s)$  to approximate  $\pi_{\theta}^{*}(s)$  for each  $s$  in the sampled trajectories. This then also improves  $\pi_{\theta}^{*}(s)$ , as  $\theta$  informs how the search space is constrained and prioritized. Therefore, we can continue to iteratively improve  $\pi_{\theta}(s)$ . To produce these trajectories, we randomly sample MiniWob++ tasks and seeds, and select
actions according to  $\pi_{\theta}^{*}$ . We then filter trajectories where the raw reward is  $< 0.8$ . We then tune  $\theta$  on these new trajectories. For simplicity, we keep the value network (i.e.  $\phi$ ) fixed.
We initially found that tuning on trajectories from MCTS could be unstable, leading to an early loss spike. To resolve this, we slightly decreased the learning rate (from  $1e - 3$  to  $5e - 4$ ) and increased the number of warmup steps (from 1000 to 4000) relative to the hyperparameters used for behavioral cloning.
B.3 Compute Details
We fine-tuned models using 64 Google Cloud TPU v3 cores.
C.1 Variance Estimates
We evaluated results for MiniWob++ based on 100 randomly selected seeds for each of the 59 tasks. To understand how results vary depending on which 100 seeds per task are used for evaluation, we ran 3 trials with different evaluation seeds for the strongest Pix2ACT model reported in Table 3, yielding mean scores of 96.2, 96.4, and 96.1; the standard deviation across these trials was 0.15. For WebShop, there is a standard test set consisting of 500 instances, so selecting seeds for evaluation is not necessary.
C.2 MiniWob++ Results Per Task
We show the performance of  $\mathrm{PIX2ACT}$  (ours) on each of the 59 MiniWob++ tasks we study, compared to other approaches, in Table 4. We compare with human crowdworker performance reported by Humphreys et al. (2022), CC-Net (Humphreys et al., 2022), DOM-Q-Net (Jia et al., 2019), DOMNET with workflow-guided execution (Liu et al., 2018), QWeb (Gur et al., 2018), RCI (Kim et al., 2023), WebN-T5-3B (Gur et al., 2022), and WebGUM (Furuta et al., 2023a). We also report scores for  $\mathrm{PIX2ACT}$  and CC-Net with behavioral cloning (BC) only. We do not include scores for GlobalCNN (Shi et al., 2017), which reported only human normalized success rates. Other than Humphreys et al. (2022), prior work has primarily reported success rate (i.e. the percentage of episodes with positive rewards), which can be equivalently mapped to the scores we report for tasks without partial rewards.
Task	Ours	Ours (BC)	Human	CC-Net	CC-Net (BC)	DOMNET	DOM-Q-Net	QWeb	RCI	WebN-T5	WebGUM
bisect-angle	96	32	92	97	29	—	—	—	—	—	—
choose-date	79	6	97	97	12	0	100	—	—	0	13
circle-center	96	52	96	97	36	—	—	—	—	—	—
click-button	99	32	98	100	78	100	100	100	100	100	100
click-button-sequence	99	100	94	100	47	100	100	—	100	100	100
click-checkboxes	100	99	97	98	32	100	100	—	100	96	100
click-checkboxes-large	99	100	87	71	0	84	—	—	94	22	99
click-checkboxes-soft	61	91	73	95	4	94	—	—	72	54	98
click-checkboxes-transfer	100	76	98	99	36	64	—	—	100	63	99
click-collapsible-2	97	31	97	98	17	99	—	—	62	0	95
click-collapsible	94	80	99	100	81	100	—	100	100	0	98
click-color	99	88	97	100	82	100	—	—	100	27	34
click-dialog	100	12	100	100	95	100	100	100	100	100	100
click-dialog-2	100	73	99	100	88	100	—	—	100	24	43
click-link	98	86	99	99	59	100	100	100	100	100	100
click-option	100	0	99	99	21	100	100	—	100	87	100
click-pie	99	81	98	97	15	32	—	100	—	51	99
click-shades	99	76	91	100	4	99	—	—	100	0	0
click-shape	94	19	88	95	11	64	—	—	98	53	72
click-tab	100	54	99	100	95	100	100	100	100	74	100
click-tab-2	98	42	97	98	27	98	100	—	74	18	95
click-tab-2-easy	99	77	99	99	61	—	—	—	—	—	—
click-tab-2-hard	97	0	96	98	19	—	—	—	76	12	95
click-tab-2-medium	100	7	97	99	54	—	—	—	—	—	—
click-test	100	100	100	100	100	100	100	—	100	100	100
click-test-2	100	100	99	100	95	100	100	—	100	100	100
click-test-transfer	100	100	99	100	94	—	—	—	—	—	—
click-widget	100	87	83	100	56	93	100	—	98	100	100
count-shape	70	0	82	85	21	76	—	—	40	41	68
count-sides	100	38	98	100	74	—	—	—	—	—	—
drag-box	99	100	99	100	61	—	—	—	—	—	—
drag-item	100	85	98	100	61	—	—	—	—	—	—
drag-items	100	64	93	99	13	—	—	—	—	—	—
drag-items-grid	89	60	87	98	5	—	—	—	—	—	—
drag-shapes	98	96	96	99	26	—	—	—	—	—	—
drag-sort-numbers	95	8	92	97	11	—	—	—	—	—	—
email-inbox-delete	100	99	99	100	22	—	100	—	—	—	—
email-inbox-important	100	99	99	100	30	—	—	—	—	—	—
enter-date	100	59	97	100	2	96	—	100	96	0	100
enter-text-2	97	100	91	98	4	—	—	—	—	—	—
enter-time	100	78	98	97	4	90	—	—	100	0	0
find-midpoint	96	74	94	97	35	—	—	—	—	—	—
grid-coordinate	92	97	87	100	66	100	—	—	100	49	100
identify-shape	100	94	98	100	68	100	—	—	76	88	100
navigate-tree	99	7	98	99	32	99	100	100	86	91	100
number-checkboxes	84	26	96	99	0	—	—	—	—	—	—
resize-textarea	99	100	94	100	27	—	—	—	—	—	—
right-angle	97	100	87	98	26	—	—	—	—	—	—
simple-algebra	100	99	86	75	3	—	—	—	100	—	—
simple-arithmetic	100	67	96	86	38	—	—	—	—	—	—
text-transform	92	91	86	60	19	—	—	—	80	—	—
tic-tac-toe	83	76	71	83	32	47	—	—	56	48	56
unicode-test	100	64	99	100	86	—	—	—	—	—	—
use-autocomplete	99	95	98	100	7	98	—	—	58	22	98
use-colorwheel	97	98	90	98	68	—	—	—	—	—	—
use-colorwheel-2	95	100	94	95	38	—	—	—	—	—	—
use slider	92	69	98	91	18	—	—	—	—	—	—
use slider-2	100	9	97	95	3	—	—	—	—	—	—
visual-addition	100	68	97	99	36	—	—	—	—	—	—
average	96.2	66.5	94.3	96.3	38.7	—	—	—	—	—	—
Table 4: Mean scores across 59 MiniWob++ tasks.