SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Custom Datasets We generate the datasets by training policies in the environment using SAC [Haarnoja et al., 2018] and take the final policy at the end of training as  $\pi_{\mathrm{e}}$  and we use an earlier policy with lower performance as the behavior policy. The expected discounted return of the policies and datasets for each domain is given in Table 2 ( $\gamma = 0.99$ ). The values for the evaluation and behavior policies were computed by running each for 300 rollout trajectories, which was more than a sufficient amount for the estimate to converge, and averaging the discounted return (note that Chang et al. [2022] use 200 rollout trajectories).
ρ(πe)	ρ(πb)
Humanoid Standup	14500	13000
Swimmer	43	31
HalfCheetah	544	308
Table 2: Policy values of the evaluation policy and behavior policy.
D4RL Datasets Due to known discrepancy issues between newer environments of gym<sup>7</sup>, we generate our datasets instead of using the publicly available ones. To generate the datasets, we use the publicly available policies <sup>8</sup>. For each domain, the expert and evaluation policy was the 10th (last policy) from training. The medium and behavior policy was the 5th policy. We added a noise of 0.1 to the policies.
D.2.2 FQE Training Iteration Curves for D4RL Datasets
In this section, we include the remaining FQE training iteration curves (OPE error vs. gradient steps) for the D4RL dataset (Figure 6). We can see that FQE diverges in multiple settings while ROPE is very stable. While FQE-CLIP does not diverge, it is still highly unstable.
D.2.3 Ablation: ROPE Hyperparameter Sensitivity
Similar to the results in Section 4.2.3, we show ROPE's hyperparameter sensitivity on all the custom and D4RL datasets. In general, we find that ROPE is robust to hyperparameter tuning, and it produces more data-efficient OPE estimates than FQE for a wide variety of its hyperparameters. See Figures 7 to 10.
<sup>7</sup>https://github.com/Farama-Foundation/D4RL/tree/master
<sup>8</sup>https://github.com/google-research/deep_ope
(a) HalfCheetah-random
(b) HalfCheetah-medium
(c) HalfCheetah-medium-expert
(d) Walker2D-random
(e) Walker2D-medium
(f) Walker2D-medium-expert
(g) Hopper-random
(h) Hopper-medium
(i) Hopper-medium-expert
Figure 6: RMAE vs. training iterations of FQE on the D4RL datasets. IQM of errors for each domain were computed over 20 trials with  $95\%$  confidence intervals. Lower is better.
Note that in the bar graphs, we limit the vertical axis to 1. In the Hopper and Walker D4RL experiments, FQE diverged and had an error significantly larger than 1.
D.2.4 Ablation: RMAE Distributions
In this section, show the remaining RMAE distribution curves [Agarwal et al., 2021b] of each algorithm on all datasets. We reach the similar conclusion that on very difficult datasets, ROPE significantly mitigates the divergence of FQE and that to avoid FQE divergence it is necessary to clip the bootstrapping target. See Figures 11 to 14.
D.2.5 Training Loss Curves for ROPE and FQE
In this section, we include the training loss curves for ROPE's training, FQE's training using ROPE representations as input, and normal FQE and FQE-CLIP. The training curves are a function of the algorithms hyperparameters (learning rate for FQE,  $\beta$  and representation output dimension for ROPE). We can see that on difficult datasets, the loss of FQE diverges. On the other hand, with ROPE, FQE's divergence is significantly mitigated. Note that ROPE does not eliminate the divergence. See Figures 15 to 18.
(c) Humanoid Standup
Figure 7: FQE vs. ROPE when varying ROPE's encoder output dimension (top) and  $\beta$  (bottom) on the custom datasets. IQM of errors are computed over 20 trials with  $95\%$  confidence intervals. Lower is better.
(c) HalfCheetah-medium-expert
Figure 8: FQE vs. ROPE when varying ROPE's encoder output dimension (top) and  $\beta$  (bottom) on the D4RL datasets. IQM of errors are computed over 20 trials with  $95\%$  confidence intervals. Lower is better.
(a) Walker2D-random
(b) Walker2D-medium
(c) Walker2D-medium-expert
Figure 9: FQE vs. ROPE when varying ROPE's encoder output dimension (top) and  $\beta$  (bottom) on the D4RL datasets. IQM of errors are computed over 20 trials with  $95\%$  confidence intervals. Lower is better.
(b) Hopper-medium
(c) Hopper-medium-expert
Figure 10: FQE vs. ROPE when varying ROPE's encoder output dimension (top) and  $\beta$  (bottom) on the D4RL datasets. IQM of errors are computed over 20 trials with  $95\%$  confidence intervals. Lower is better.
(c) Humanoid Standup
Figure 11: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in  $\geq 20$  runs for each algorithm. Shaded region is  $95\%$  confidence interval. Larger area under the curve is better.
(a) HalfCheetah-random
Figure 12: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in  $\geq 20$  runs for each algorithm. Shaded region is  $95\%$  confidence interval. Larger area under the curve is better.
(c) Walker2D-medium-expert
Figure 13: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in  $\geq 20$  runs for each algorithm. Shaded region is  $95\%$  confidence interval. Larger area under the curve is better.
(a) Hopper-random
Figure 14: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in  $\geq 20$  runs for each algorithm. Shaded region is  $95\%$  confidence interval. Larger area under the curve is better.
(c) Hopper-medium-expert
(a) FQE HalfCheetah-random
(b) FQE-CLIP random
HalfCheetah-(c) FQE w/ ROPE HalfCheetah-random
(d) FQE HalfCheetah-medium
(e) FQE-CLIP medium
HalfCheetah-(f) FQE w/ ROPE HalfCheetah-medium
(g) FQE expert
FQE-CLIPum-expert
HalfCheetah-(i) FQE w/ ROPE HalfCheetah-medium-expert
Figure 15: FQE training loss vs. training iterations on the D4RL datasets. IQM of errors for each domain were computed over 20 trials with  $95\%$  confidence intervals. Lower is better. Vertical axis is log-scaled.
D.2.6 Understanding the ROPE Representations
In this section, we try to understand the nature of the ROPE representations. We do so by plotting the mean of the: 1) mean feature dimension and 2) standard deviation feature dimension. For example, if there  $N$  state-action pairs, each with dimension  $D$ , we compute the mean and standard deviation feature dimension for each of the  $D$  dimensions across the  $N$  examples, and then compute the mean along the  $D$  dimensions. If the standard deviation value is close 0, it indicates that there may be a representation collapse. See Figure 19.
D.3 Hardware For Experiments
For all experiments, we used the following compute infrastructure:
- Distributed cluster on HTCondor framework
Intel(R) Xeon(R) CPU E5-2470 0 @ 2.30GHz
RAM: 7GB
(a) FQE Walker2D-random
(b) FQE-CLIP Walker2D-random
(c) FQE w/ ROPE Walker2D-random
(d) FQE Walker2D-medium
(e) FQE-CLIP Walker2D-medium
(f) FQE w/ ROPE Walker2D-medium
(h) FQE-CLIP Walker2D-medium-(i) expert
(i) FQE w/ FQE Walker2D-medium-expert
Figure 16: FQE training loss vs. training iterations on the D4RL datasets. IQM of errors for each domain were computed over 20 trials with  $95\%$  confidence intervals. Lower is better. Vertical axis is log-scaled.
- Disk space: 4GB
(a) FQE Hopper-random
(b) FQE-CLIP Hopper-random
(c) FQE w/ ROPE Hopper-random
(d) FQE Hopper-medium
(e) FQE-CLIP Hopper-medium
(f) FQE w/ ROPE Hopper-medium
(g) FQE Hopper-medium-expert
(h) FQE-CLIP Hopper-medium-(i) FQE w/ ROPE Hopper-medium-expert expert
Figure 17: FQE training loss vs. training iterations on the D4RL datasets. IQM of errors for each domain were computed over 20 trials with  $95\%$  confidence intervals. Lower is better. Vertical axis is log-scaled.
(i) Hopper-medium-expert
Figure 18: ROPE training loss vs. training iterations on the D4RL datasets. IQM of errors for each domain were computed over 20 trials with  $95\%$  confidence intervals. Lower is better.
(a) Mean random
(b) Mean medium
(c) Mean medium-expert
(d) Std random
(e) Std medium
(f) Std medium-expert
Figure 19: Mean of feature dimension stats vs. training iterations on the D4RL Hopper dataset. IQM of errors for each domain were computed over 20 trials with  $95\%$  confidence intervals.