SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \beta \mathcal {E} _ {\mathcal {D}} (\pi_ {1}, f _ {1}) = \beta \mathbb {E} _ {\mu} [ ((f _ {1} - r) (a)) ^ {2} ] = \beta \mu (a _ {2}) \left(\frac {1}{2}\right) ^ {2} = \frac {\beta \mu (a _ {2})}{4}, \\ \beta \mathcal {E} _ {\mathcal {D}} \left(\pi_ {1}, f _ {2}\right) = \beta \mathbb {E} _ {\mu} \left[ \left(\left(f _ {2} - r\right) (a)\right) ^ {2} \right] \\ = \beta \mu (a _ {2}) \left(\hat {r} (a _ {2}) \left(\frac {1}{2} - 2 \Delta\right) ^ {2} + (1 - \hat {r} (a _ {2})) \left(\frac {1}{2} + 2 \Delta\right) ^ {2}\right) \\ \leq \beta \mu (a _ {2}) \left(\left(\frac {1}{2} + 2 \Delta\right) \left(\frac {1}{2} - 2 \Delta\right) ^ {2} + \left(\frac {1}{2} - 2 \Delta\right) \left(\frac {1}{2} + 2 \Delta\right) ^ {2}\right) \\ \leq \beta \mu \left(a _ {2}\right) \left(\left(\frac {1}{2}\right) ^ {2} - (2 \Delta) ^ {2}\right) \\ = \frac {\beta \mu \left(a _ {2}\right)}{4} - 4 \frac {\beta}{N}. \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D}} \left(\pi_ {2}, f _ {1}\right) = \mathbb {E} _ {\mu} \left[ f _ {1} \left(\pi_ {2}\right) - f _ {1} (a) \right] = f _ {1} \left(\pi_ {2}\right) - \mu \left(a _ {1}\right) f _ {1} \left(a _ {1}\right) - \mu \left(a _ {2}\right) f _ {1} \left(a _ {2}\right) \\ = \mu \left(a _ {1}\right) \left(f _ {1} \left(a _ {2}\right) - f _ {1} \left(a _ {1}\right)\right) = - \left(1 - \frac {1}{N \Delta^ {2}}\right) \Delta , \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D}} \left(\pi_ {2}, f _ {2}\right) = \mathbb {E} _ {\mu} \left[ f _ {2} \left(\pi_ {2}\right) - f _ {2} (a) \right] = f _ {2} \left(\pi_ {2}\right) - \mu \left(a _ {1}\right) f _ {2} \left(a _ {1}\right) - \mu \left(a _ {2}\right) f _ {2} \left(a _ {2}\right) \\ = \mu \left(a _ {1}\right) \left(f _ {2} \left(a _ {2}\right) - f _ {2} \left(a _ {1}\right)\right) = \left(1 - \frac {1}{N \Delta^ {2}}\right) \Delta , \\ \end{array}
$$
$$
\beta \mathcal {E} _ {\mu} (\pi_ {2}, f _ {1}) = \beta \mathcal {E} _ {\mathcal {D}} (\pi_ {1}, f _ {1}) = \frac {\beta \mu (a _ {2})}{4},
$$
$$
\beta \mathcal {E} _ {\mu} (\pi_ {2}, f _ {2}) = \beta \mathcal {E} _ {\mathcal {D}} (\pi_ {1}, f _ {2}) \leq \frac {\beta \mu (a _ {2})}{4} - 4 \frac {\beta}{N}.
$$
Since  $\beta / N \geq \Delta$ , we can obtain that  $\mathcal{L}_{\mathcal{D}}(\pi_2, f_1) + \beta \mathcal{E}_{\mathcal{D}}(\pi_2, f_1) > \mathcal{L}_{\mathcal{D}}(\pi_2, f_2) + \beta \mathcal{E}_{\mathcal{D}}(\pi_2, f_2)$ , which implies that  $f^{\pi_2} = f_2$ . Finally, note that
$$
\mathcal {L} _ {\mathcal {D}} (\pi_ {2}, f ^ {\pi_ {2}}) = \mathcal {L} _ {\mathcal {D}} (\pi_ {2}, f _ {2}) > \frac {\Delta}{2} \geq \frac {1}{N \Delta} = | \mathcal {L} _ {\mathcal {D}} (\pi_ {1}, f ^ {\pi_ {1}}) |,
$$
we have  $\hat{\pi} = \pi_2$  by ATAC algorithm. Note that  $J(\pi_1) - J(\hat{\pi}) = r(\pi_1) - r(\pi_2) = \Delta \gg \tilde{\Omega}(1/\sqrt{N})$ . Therefore, for any  $\alpha > 1/2$ , there exists an instance s.t. ATAC cannot achieve the optimal rate  $O(1/\sqrt{N})$ . In particular, when  $\beta = \Theta(N^{2/3})$ , we have  $J(\pi_1) - J(\hat{\pi}) = \Delta = \Omega(1/N^{1/3})$ .
E.2 Complementary lemma
Lemma 6 (Anti-concentration of Binomial distribution, adapted from Proposition 7.3.2 of Matoušek and Vondrák (2001)). Let  $X \sim \operatorname{Bin}(n, \frac{1}{2})$  be a binomial random variable with mean  $\mu = \frac{n}{2}$ . Then we have that for any  $t \in [0, \frac{1}{8}]$  and universal constants  $c_1, c_2$ ,
$$
\Pr \left(X \geq \mu + n t\right) \geq c _ {1} e ^ {- c _ {2} t ^ {2} n}.
$$
F Additional Experimental Details and Results
F.1 More implementation details
Our implementation is nearly identical to ATAC, except that we use a different regularizer for the critic. More details can be found in Cheng et al. (2022).
(a) walker2d-med
(b) walker2d-med-replay
(c) walker2d-med-exp
(d) halfcheetah-rand
(e) halfcheetah-med
(f) halfcheetah-med-exp
(g) hoper-rand
(h) hopper-med
(i) hopper-med-replay
Figure 3: Comparison of A-Crab and ATAC in more different settings.
F.2 Comparison of A-Crab and ATAC in different settings
In this section, we show comparisons of A-Crab and ATAC in more different settings in Figure 3.
F.3 Value of  $\beta$  and  $C_{\infty}$  for each setting
In this section, we show our choices of  $\beta$  and  $C_{\infty}$  for each setting in Table F.3. Note that we directly choose the same value of  $\beta$  as in ATAC, and select  $C_{\infty}$  by a grid search over  $\{1,2,5,10,20,50,100,200\}$ .
Table 2: Choices of  $\beta$  and  $C_{\infty}$  for each setting. The value of  $\beta$  is the same as in ATAC (Cheng et al., 2022) and  $C_{\infty}$  is chosen by a grid search.
β	C∞
walker2d-random	64	50
walker2d-medium	64	1
walker2d-medium-replay	64	5
walker2d-medium-expert	64	2
hopper-random	64	2
hopper-medium	64	1
hopper-medium-replay	16	1
hopper-medium-expert	1	1
halfcheetah-random	16	10
halfcheetah-medium	4	1
halfcheetah-medium-replay	16	2
halfcheetah-medium-expert	0.062	5