1. Optimal statistical rate in competing with the best covered policy: In Theorem 1, we prove that our A-Crab algorithm, which uses average Bellman error, enjoys an optimal statistical rate of  $1 / \sqrt{N}$ . In contrast, we prove in Proposition 4.1 that the ATAC algorithm, which uses the squared Bellman error, fails to achieve the optimal rate in certain offline learning instances. As Cheng et al. (2022) explains, the squared Bellman-error regularizer appears to be the culprit behind the suboptimal rate of ATAC being  $1 / N^{1/3}$ . Moreover, our algorithm improves over any policy covered in the data. This is in contrast to the recent work Rashidinejad et al. (2022), which proposes an algorithm based on the MIS framework that achieves the  $1 / \sqrt{N}$  rate only when the optimal policy (i.e. the policy with the highest expected rewards) is covered in the data.
2. A weaker notion of data coverage that exploits visitation structure: As we discussed earlier,  $\ell_{\infty}$  concentrability notion is used in many prior works (Rashidinejad et al., 2021; Chen and Jiang, 2022; Ozdaglar et al., 2022; Rashidinejad et al., 2022). Our importance-weighted average Bellman error as well as using Bernstein inequality in the proof relies on guarantees in terms of an  $\ell_{2}$  single-policy concentrability notion that is weaker than the  $\ell_{\infty}$  variant. In particular, we have  $C_{\ell_{\infty}}^{\pi} = \| w^{\pi}\|_{\infty}$  and  $C_{\ell_2}^\pi = \| w^\pi \|_{2,\mu}$ , where  $\| \cdot \|_{2,\mu}$  is the weighted 2-norm w.r.t. the dataset distribution  $\mu$ . The latter implies that the coverage coefficient only matters as much as it is covered by the dataset. Moreover, by the definition of  $w^{\pi}$ , we can obtain that  $(C_{\ell_2}^{\pi})^2 = \mathbb{E}_{d^{\pi}}[w^{\pi}(s,a)]$ , which provides another explanation of  $\ell_{2}$  concentrability that the coverage coefficient only matters as much as it is actually visited by the policy. There are also other notions of single-policy concentrability exploiting function approximation structures to make the coverage coefficient smaller (e.g.,  $C_{\mathrm{Bellman}}^{\pi}$  in Xie et al. (2021)). However, these notions degenerate to  $C_{\ell_{\infty}}^{\pi}$  as the function class gets richer (see Figure 1 for an intuitive comparison of different notions of concentrability and Section 2.4 for a rigorous proof).
3. Robust policy improvement: Policy improvement (PI) refers to the property that an offline RL algorithm (under a careful choice of specific hyperparameters) can always improve upon the behavior policy that is used to collect the data. In particular, robust policy improvement (RPI) means the PI property holds under a wide range of the choice of specific hyperparameters (Cheng et al., 2022; Xie et al., 2022). Similar to the ATAC algorithm in Cheng et al. (2022), our approach enjoys the robust policy improvement guarantee as shown in Theorem 2.
4. Inheriting many other benefits of adversarially-trained actor-critic: Since our algorithm is based on ATAC with a different choice of regularizer, it can be easily implemented as ATAC to be applied to practical scenarios, and we provide experimental results in Section 6. Also, our algorithm is robust to model misspecification and does not require the completeness assumption (Assumption 2 in Cheng et al. (2022)) on the value function class, which makes it more practical. Moreover, our algorithm can learn a policy that outperforms any other policies well covered by the dataset.