SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[51] R. S. Sutton and A. G. Barto. Reinforcement learning - an introduction. Adaptive computation and machine learning. MIT Press, 1998.
[52] A. Swaminathan and T. Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. J. Mach. Learn. Res., 16:1731-1755, 2015.
[53] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In Proceedings of the 24th IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, Vilamoura, Portugal, 2012.
[54] C. Voloshin, N. Jiang, and Y. Yue. Minimax model learning. In A. Banerjee and K. Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1612–1620, Virtual Event, 2021.
[55] T. Wang, T. Qin, and Z. Zhou. Estimating possible causal effects with latent variables via adjustment. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202, pages 36308-36335, 2023.
[56] T.-Z. Wang, X.-Z. Wu, S.-J. Huang, and Z.-H. Zhou. Cost-effectively identifying causal effects when only response variable is observable. In Proceedings of the 37th International Conference on Machine Learning, pages 10060–10069, 2020.
[57] S. Yang, S. Zhang, Y. Feng, and M. Zhou. A unified framework for alternating offline model training and policy learning. CoRR, abs/2210.05922, 2022.
[58] J. Yoon, J. Jordon, and M. van der Schaar. GANITE: estimation of individualized treatment effects using generative adversarial nets. In 6th International Conference on Learning Representations, 2018.
[59] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. MOPO: model-based offline policy optimization. In Advances in Neural Information Processing Systems 33, virtual event, 2020.
[60] R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
[61] W. Y. Zou, S. Du, J. Lee, and J. O. Pedersen. Heterogeneous causal learning for effectiveness optimization in user marketing. CoRR, abs/2004.09702, 2020.
A Proof of Theoretical Results 16
A.1 Proof of Lemma A.1 18
A.2 Proof of Eq.(9) 19
A.3 Proof of Thm. A.2 20
A.4 Proof of the Tractable Solution 23
B Discussion and Limitations of the Theoretical Results 25
C Societal Impact 25
D AWRM-oracle Pseudocode 25
E Implementation 25
E.1 Details of the GALILEO Implementation 25
E.2 Connection with Previous Adversarial Algorithms 28
F Additional Related Work 28
GExperiment Details 30
G.1 Settings 30
G.2 Baseline Algorithms 33
G.3 Hyper-parameters 33
G.4 Computation Resources 33
H Additional Results 34
H.1 Test in Single-step Environments 34
H.2 All of the Result Table 35
H.3 Ablation Studies 35
H.4 Worst-Case Prediction Error 35
H.5 Detailed Results in the MuJoCo Tasks 36
H.6 Off-policy Evaluation (OPE) in the MuJoCo Tasks 37
H.7 Detailed Results in the BAT Task 42
A Proof of Theoretical Results
In the proof section, we replace the notation of  $\mathbb{E}$  with an integral for brevity. Now we rewrite the original objective  $\bar{L} (\rho_{M^*}^\beta ,M)$  as:
$$
\min  _ {M \in \mathcal {M}} \max  _ {\beta \in \Pi} \int_ {\mathcal {X}, \mathcal {A}} \rho_ {M ^ {*}} ^ {\mu} (x, a) \omega (x, a) \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) (- \log M \left(x ^ {\prime} | x, a\right)) d x ^ {\prime} d a d x - \frac {\alpha}{2} \| \rho_ {M ^ {*}} ^ {\beta} (\cdot , \cdot) \| _ {2} ^ {2}, \tag {7}
$$
where  $\omega(x, a) = \frac{\rho_{M^*}^\beta(x, a)}{\rho_{M^*}^\mu(x, a)}$  and  $\|\rho_{M^*}^\beta(\cdot, \cdot)\|_2^2 = \int_{\mathcal{X}, \mathcal{A}} \rho_{M^*}^\beta(x, a)^2 \mathrm{d}a \mathrm{d}x$ , which is the squared  $l_2$ -norm. In an MDP, given any policy  $\pi$ ,  $\rho_{M^*}^\pi(x, a, x') = \rho_{M^*}^\pi(x) \pi(a|x) M^*(x'|x, a)$  where  $\rho_{M^*}^\pi(x)$  denotes the occupancy measure of  $x$  for policy  $\pi$ , which can be defined [51, 25] as  $\rho_{M^*}^\pi(x) := (1 - \gamma) \mathbb{E}_{x_0 \sim \rho_0} \left[ \sum_{t=0}^{\infty} \gamma^t \operatorname{Pr}(x_t = x|x_0, M^*) \right]$  where  $\operatorname{Pr}^\pi[x_t = x|x_0, M^*]$  is the state visitation probability that  $\pi$  starts at state  $x_0$  in model  $M^*$  and receive  $x$  at timestep  $t$  and  $\gamma \in [0,1]$  is the discount factor.
Figure 8: The overall pipeline to model the tractable solution to AWRM.  $f$  is a generator function defined by  $f$ -divergence [37].  $\kappa$  is an intermediary policy introduced in the estimation.
The overall pipeline to model the tractable solution to AWRM is given in Fig. 8. In the following, we will summarize the modelling process based on Fig. 8. We first approximate the optimal distribution  $\rho_{M^*}^{\bar{\beta}^*}$  via Lemma. A.1.
Lemma A.1. Given any  $M$  in  $\bar{L} (\rho_{M^*}^\beta ,M)$ , the distribution  $\rho_{M^*}^{\bar{\beta}^*}(x,a)$  of the ideal best-response policy  $\bar{\beta}^*$  satisfies:
$$
\frac {1}{\alpha_ {M}} \left(D _ {K L} \left(M ^ {*} (\cdot | x, a), M (\cdot | x, a)\right) + H _ {M ^ {*}} (x, a)\right), \tag {8}
$$
where  $D_{KL}(M^{*}(\cdot |x,a),M(\cdot |x,a))$  is the Kullback-Leibler (KL) divergence between  $M^{*}(\cdot |x,a)$  and  $M(\cdot |x,a)$ ,  $H_{M^{*}}(x,a)$  denotes the entropy of  $M^{*}(\cdot |x,a)$ , and  $\alpha_{M}$  is the regularization coefficient  $\alpha$  in Eq. (4) and also as a normalizer of Eq. (8).
Note that the ideal best-response policy  $\bar{\beta}^*$  is not the real best-response policy  $\beta^*$ . The distribution  $\rho_{M^*}^{\bar{\beta}^*}$  is an approximation of the real optimal adversarial distribution. We give a discussion of the rationality of the ideal best-response policy  $\bar{\beta}^*$  as a replacement of the real best-response policy  $\beta^*$  in Remark A.4. Intuitively,  $\rho_{M^*}^{\bar{\beta}^*}$  has larger densities on the data where the divergence between the approximation model and the real model (i.e.,  $D_{KL}(M^{*}(\cdot |x,a),M(\cdot |x,a)))$  is larger or the stochasticity of the real model (i.e.,  $H_{M^*}$ ) is larger.
However, the integral process of  $D_{KL}$  in Eq. (8) is intractable in the offline setting as it explicitly requires the conditional probability function of  $M^{*}$ . Our solution to solve the problem is utilizing the offline dataset  $\mathcal{D}_{\mathrm{real}}$  as the empirical joint distribution  $\rho_{M^*}^\mu (x,a,x')$  and adopting practical
techniques for distance estimation on two joint distributions, like GAN [19, 37], to approximate Eq. (8). To adopt that solution, we should first transform Eq. (8) into a form under joint distributions. Without loss of generality, we introduce an intermediary policy  $\kappa$ , of which  $\mu$  can be regarded as a specific instance. Then we have  $M(x'|x,a) = \rho_M^\kappa(x,a,x') / \rho_M^\kappa(x,a)$  for any  $M$  if  $\rho_M^\kappa(x,a) > 0$ . Assuming  $\forall x \in \mathcal{X}, \forall a \in \mathcal{A}, \rho_{M^*}^\kappa(x,a) > 0$  if  $\rho_{M^*}^{\hat{\beta}^*}(x,a) > 0$ , which will hold when  $\kappa$  overlaps with  $\mu$ , then Eq. (8) can transform to:
$$
\frac {1}{\alpha_ {0} (x , a)} \left(\int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) \log \frac {\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M} ^ {\kappa} (x , a , x ^ {\prime})} \mathrm {d} x ^ {\prime} - \rho_ {M ^ {*}} ^ {\kappa} (x, a) \left(\log \frac {\rho_ {M ^ {*}} ^ {\kappa} (x , a)}{\rho_ {M} ^ {\kappa} (x , a)} - H _ {M ^ {*}} (x, a)\right)\right),
$$
where  $\alpha_0(x,a) = \alpha_M\rho_{M^*}^\kappa (x,a)$ . We notice that the form  $\rho_{M^*}^\kappa \log \frac{\rho_{M^*}^\kappa}{\rho_{M}^\kappa}$  is the integrated function in reverse KL divergence, which is an instance of  $f$  function in  $f$ -divergence [2]. Replacing that form with  $f$  function, we obtain a generalized representation of  $\rho_{M^*}^{\bar{\beta}^*}$ :
$$
\bar {\rho} _ {M ^ {*}} ^ {\bar {\beta} ^ {*}} := \frac {1}{\alpha_ {0} (x , a)} \left(\int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) f \left(\frac {\rho_ {M} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) \mathrm {d} x ^ {\prime} - \rho_ {M ^ {*}} ^ {\kappa} (x, a) \left(f \left(\frac {\rho_ {M} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) - H _ {M ^ {*}} (x, a)\right)\right), \tag {9}
$$
where  $f:\mathbb{R}_{+}\to \mathbb{R}$  is a convex and lower semi-continuous (l.s.c.) function.  $\bar{\rho}_{M^{*}}^{\beta^{*}}$  gives a generalized representation of the optimal adversarial distribution to maximize the error of the model. Based on Eq. (9), we have a surrogate objective of AWRM which can avoid querying  $M^{*}$  to construct  $\rho_{M^{*}}^{\beta^{*}}$ :
Theorem A.2. Let  $\bar{\rho}_{M^*}^{\bar{\beta}^*}$  as the data distribution of the best-response policy  $\bar{\beta}^*$  in Eq. (4) under model  $M_{\theta}$  parameterized by  $\theta$ , then we can find the optimal  $\theta^*$  of  $\min_{\theta} \max_{\beta \in \Pi} \bar{L}(\rho_{M^*}^{\beta}, M_{\theta})$  (Eq. (4)) via iteratively optimizing the objective  $\theta_{t+1} = \min_{\theta} \bar{L}(\bar{\rho}_{M^*}^{\bar{\beta}^*}, M_{\theta})$ , where  $\bar{\rho}_{M^*}^{\bar{\beta}^*}$  is approximated via the last-iteration model  $M_{\theta_t}$ . Based on Corollary A.9, we derive an upper bound objective for  $\min_{\theta} \bar{L}(\bar{\rho}_{M^*}^{\bar{\beta}^*}, M_{\theta})$ :
$$
\theta_ {t + 1} = \min  _ {\theta} \mathbb {E} _ {\rho_ {M ^ {*}} ^ {\kappa}} \left[ \frac {- 1}{\alpha_ {0} (x , a)} \log M _ {\theta} (x ^ {\prime} | x, a) \underbrace {\left(f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) - f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) + H _ {M ^ {*}} (x , a)\right)} _ {W (x, a, x ^ {\prime})} \right],
$$
where  $\mathbb{E}_{\rho_{M^*}^\kappa}[\cdot ]$  denotes  $\mathbb{E}_{x,a,x'\sim \rho_{M^*}^\kappa}[\cdot ]$ ,  $f$  is a l.s.c function satisfying  $f^{\prime}(x)\leq 0,\forall x\in \mathcal{X}$ , and  $\alpha_0(x,a) = \alpha_{M_{\theta_t}}\rho_{M^*}^\kappa (x,a)$ .
Thm. A.2 approximately achieve AWRM by using  $\kappa$  and a pseudo-reweighting module  $W$ .  $W$  assigns learning propensities for data points with larger differences between distributions  $\rho_{M_{\theta_t}}^\kappa$  and  $\rho_{M^*}^\kappa$ . By adjusting the weights, the learning process will exploit subtle errors in any data point, whatever how many proportions it contributes, to correct potential generalization errors on counterfactual data.
Remark A.3. In practice, we need to use real-world data to construct the distribution  $\rho_{M^*}^\kappa$ . In the offline model-learning setting, we only have a real-world dataset  $\mathcal{D}_{\mathrm{real}}$  collected by the behavior policy  $\mu$ , which is the empirical distribution of  $\rho_{M^*}^\mu$ . Let  $\kappa = \mu$ , we have
$$
\theta_ {t + 1} = \min _ {\theta} \mathbb {E} _ {\rho_ {M ^ {*}} ^ {\mu}} \left[ \frac {- 1}{\alpha_ {0} (x , a)} \log M _ {\theta} (x ^ {\prime} | x, a) \underbrace {\left(f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\mu} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\mu} (x , a , x ^ {\prime})}\right) - f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\mu} (x , a)}{\rho_ {M ^ {*}} ^ {\mu} (x , a)}\right) + H _ {M ^ {*}} (x , a)\right)} _ {W (x, a, x ^ {\prime})} \right],
$$
which is Eq. (5) in the main body.
In Thm. A.2, the terms  $f(\rho_{M_{\theta_t}}^\kappa (x,a,x') / \rho_{M^*}^\kappa (x,a,x')) - f(\rho_{M_{\theta_t}}^\kappa (x,a) / \rho_{M^*}^\kappa (x,a))$  are still intractable. Thanks to previous successful practices in GAN [19] and GAIL [25], we achieve the objective via a generator-discriminator-paradigm objective through similar derivation. We show the results as follows and leave the complete derivation in Appx. A.4. In particular, by introducing two discriminators  $D_{\varphi_0^*}(x,a,x')$  and  $D_{\varphi_0^*}(x,a)$ , letting  $\kappa = \mu$ , we can optimize the surrogate objective Eq. (5) via:
$$
\theta_ {t + 1} = \max _ {\boldsymbol {\theta}} \left(\mathbb {E} _ {\rho_ {M _ {\theta_ {t}}} ^ {\hat {\mu}}} \left[ A _ {\varphi_ {0} ^ {*}, \varphi_ {1} ^ {*}} (x, a, x ^ {\prime}) \log M _ {\boldsymbol {\theta}} (x ^ {\prime} | x, a) \right] + \mathbb {E} _ {\rho_ {M ^ {*}} ^ {\mu}} \left[ (H _ {M ^ {*}} (x, a) - A _ {\varphi_ {0} ^ {*}, \varphi_ {1} ^ {*}} (x, a, x ^ {\prime})) \log M _ {\boldsymbol {\theta}} (x ^ {\prime} | x, a) \right]\right)
$$
s.t.  $\varphi_0^* = \arg \max_{\varphi_0}\left(\mathbb{E}_{\rho_{M^*}^\mu}\left[\log D_{\varphi_0}(x,a,x')\right] + \mathbb{E}_{\rho_{M_\theta_t}^\hat{\mu}}\left[\log (1 - D_{\varphi_0}(x,a,x'))\right]\right)$
$$
\varphi_ {1} ^ {*} = \arg \max _ {\varphi_ {1}} \left(\mathbb {E} _ {\rho_ {M ^ {*}}} ^ {\mu} [ \log D _ {\varphi_ {1}} (x, a) ] + \mathbb {E} _ {\rho_ {M _ {\theta_ {t}}}} ^ {\tilde {\mu}} [ \log (1 - D _ {\varphi_ {1}} (x, a)) ]\right),
$$
A.1 Proof of Lemma A.1
Proof. Given a transition function  $M$  of an MDP, the distribution of the best-response policy  $\beta^{*}$  satisfies:
$$
\begin{array}{l} \rho_ {M ^ {*}} ^ {\beta^ {*}} = \arg \max  _ {\rho_ {M ^ {*}} ^ {\beta}} \int_ {\mathcal {X}, \mathcal {A}} \rho_ {M ^ {*}} ^ {\mu} (x, a) \omega (x, a) \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) (- \log M \left(x ^ {\prime} | x, a\right)) \mathrm {d} x ^ {\prime} \mathrm {d} a \mathrm {d} x - \frac {\alpha}{2} \| \rho_ {M ^ {*}} ^ {\beta} (\cdot , \cdot) \| _ {2} ^ {2} \\ = \arg \max  _ {\rho_ {M ^ {*}} ^ {\beta}} \int_ {\mathcal {X}, \mathcal {A}} \rho_ {M ^ {*}} ^ {\beta} (x, a) \underbrace {\int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} \mid x , a\right) (- \log M \left(x ^ {\prime} \mid x , a\right)) \mathrm {d} x ^ {\prime}} _ {g (x, a)} \mathrm {d} a \mathrm {d} x - \frac {\alpha}{2} \| \rho_ {M ^ {*}} ^ {\beta} (\cdot , \cdot) \| _ {2} ^ {2} \\ = \arg \max  _ {\rho_ {M ^ {*}} ^ {\beta}} \frac {2}{\alpha} \int_ {\mathcal {X}, \mathcal {A}} \rho_ {M ^ {*}} ^ {\beta} (x, a) g (x, a) d a d x - \| \rho_ {M ^ {*}} ^ {\beta} (\cdot , \cdot) \| _ {2} ^ {2} \\ = \arg \max  _ {\rho_ {M ^ {*}} ^ {\beta}} \frac {2}{\alpha} \int_ {\mathcal {X}, \mathcal {A}} \rho_ {M ^ {*}} ^ {\beta} (x, a) g (x, a) d a d x - \| \rho_ {M ^ {*}} ^ {\beta} (\cdot , \cdot) \| _ {2} ^ {2} - \frac {\| g (\cdot , \cdot) \| _ {2} ^ {2}}{\alpha^ {2}} \\ = \arg \max  _ {\rho_ {M ^ {*}} ^ {\beta}} - \left(- 2 \int_ {\mathcal {X}, \mathcal {A}} \rho_ {M ^ {*}} ^ {\beta} (x, a) \frac {g (x , a)}{\alpha} d a d x + \| \rho_ {M ^ {*}} ^ {\beta} (\cdot , \cdot) \| _ {2} ^ {2} + \frac {\| g (\cdot , \cdot) \| _ {2} ^ {2}}{\alpha^ {2}}\right) \\ = \operatorname * {a r g   m a x} _ {\rho_ {M ^ {*}} ^ {\beta}} - \| \rho_ {M ^ {*}} ^ {\beta} (\cdot , \cdot) - \frac {g (\cdot , \cdot)}{\alpha} \| _ {2} ^ {2}. \\ \end{array}
$$
We know that the occupancy measure  $\rho_{M^*}^\beta$  is a density function with a constraint  $\int_{\mathcal{X}}\int_{\mathcal{A}}\rho_{M^*}^\beta (x,a)\mathrm{d}a\mathrm{d}x = 1$ . Assuming the occupancy measure  $\rho_{M^*}^\beta$  has an upper bound  $c$ , that is  $0\leq \rho_{M^{*}}^{\beta}(x,a)\leq c,\forall a\in \mathcal{A},\forall x\in \mathcal{X}$ , constructing a regularization coefficient  $\alpha_{M} = \int_{\mathcal{X}}\int_{\mathcal{A}}(D_{KL}(M^{*}(\cdot |x,a),M(\cdot |x,a)) + H_{M^{*}}(x,a))\mathrm{d}x\mathrm{d}a$  as a constant value given any  $M$ , then we have
$$
\begin{array}{l} \rho_ {M *} ^ {\beta *} (x, a) = \frac {g (x , a)}{\alpha_ {M}} \\ = \frac {\int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x , a\right) \log \frac {M ^ {*} \left(x ^ {\prime} | x , a\right)}{M \left(x ^ {\prime} | x , a\right)} \mathrm {d} x - \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x , a\right) \log M ^ {*} \left(x ^ {\prime} | x , a\right) \mathrm {d} x}{\alpha_ {M}} \\ = \frac {D _ {K L} (M ^ {*} (\cdot | x , a) , M (\cdot | x , a)) + H _ {M ^ {*}} (x , a)}{\alpha_ {M}} \\ \propto \left(D _ {K L} \left(M ^ {*} (\cdot | x, a), M (\cdot | x, a)\right) + H _ {M ^ {*}} (x, a)\right), \\ \end{array}
$$
which is the optimal density function of Eq. (7) with  $\alpha = \alpha_{M}$ .
Note that in some particular  $M^*$ , we still cannot construct a  $\beta$  that can generate an occupancy specified by  $g(x, a) / \alpha_M$  for any  $M$ . We can only claim the distribution of the ideal best-response policy  $\bar{\beta}^*$  satisfies:
$$
\rho_ {M ^ {*}} ^ {\bar {\beta} ^ {*}} (x, a) = \frac {1}{\alpha_ {M}} \left(D _ {K L} \left(M ^ {*} (\cdot | x, a), M (\cdot | x, a)\right) + H _ {M ^ {*}} (x, a)\right), \tag {10}
$$
where  $\alpha_{M}$  is a normalizer that  $\alpha_{M} = \int_{\mathcal{X}}\int_{\mathcal{A}}(D_{KL}(M^{*}(\cdot |x,a),M(\cdot |x,a)) + H_{M^{*}}(x,a))\mathrm{d}x\mathrm{d}a$  . We give a discussion of the rationality of the ideal best-response policy  $\bar{\beta}^*$  as a replacement of the real best-response policy  $\beta^{*}$  in Remark A.4.
Remark A.4. The optimal solution Eq. (10) relies on  $g(x, a)$ . In some particular  $M^*$ , it is intractable to derive a  $\beta$  that can generate an occupancy specified by  $g(x, a) / \alpha_M$ . Consider the following case: a state  $x_1$  in  $M^*$  might be harder to reach than another state  $x_2$ , e.g.,  $M^*(x_1|x, a) < M^*(x_2|x, a), \forall x \in \mathcal{X}, \forall a \in \mathcal{A}$ , then it is impossible to find a  $\beta$  that the occupancy satisfies  $\rho_{M^*}^\beta(x_1, a) > \rho_{M^*}^\beta(x_2, a)$ . In this case, Eq. (10) can be a sub-optimal solution. Since this work focuses on task-agnostic solution derivation while the solution to the above problem should rely on the specific description of  $M^*$ , we leave it as future work. However, we point out that Eq. (10) is a reasonable re-weighting term even as a sub-optimum:  $\rho_{M^*}^\beta$  gives larger densities on the data where the distribution distance between the approximation model and the real model (i.e.,  $D_{KL}(M^*, M)$ ) is larger or the stochasticity of the real model (i.e.,  $H_{M^*}$ ) is larger.
A.2 Proof of Eq. (9)
The integral process of  $D_{KL}$  in Eq. (8) is intractable in the offline setting as it explicitly requires the conditional probability function of  $M^{*}$ . Our motivation for the tractable solution is utilizing the offline dataset  $\mathcal{D}_{\mathrm{real}}$  as the empirical joint distribution  $\rho_{M^*}^\mu (x,a,x')$  and adopting practical techniques for distance estimation on two joint distributions, like GAN [19, 37], to approximate Eq. (8). To adopt that solution, we should first transform Eq. (8) into a form under joint distributions. Without loss of generality, we introduce an intermediary policy  $\kappa$ , of which  $\mu$  can be regarded as a specific instance. Then we have  $M(x'|x,a) = \rho_M^\kappa (x,a,x') / \rho_M^\kappa (x,a)$  for any  $M$  if  $\rho_M^\kappa (x,a) > 0$ . Assuming  $\forall x\in \mathcal{X},\forall a\in \mathcal{A},\rho_{M^*}^\kappa (x,a) > 0$  if  $\rho_{M^*}^{\bar{\beta}^*}(x,a) > 0$ , which will hold when  $\kappa$  overlaps with  $\mu$ , then Eq. (8) can transform to:
$$
\begin{array}{l} \rho_ {M ^ {*}} ^ {\bar {\beta} ^ {*}} (x, a) = \frac {D _ {K L} (M ^ {*} (\cdot | x , a) , M (\cdot | x , a)) + H _ {M ^ {*}} (x , a)}{\alpha_ {M}} \\ = \frac {1}{\alpha_ {M}} \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) \left(\log \frac {M ^ {*} \left(x ^ {\prime} | x , a\right)}{M \left(x ^ {\prime} | x , a\right)} - \log M ^ {*} \left(x ^ {\prime} | x, a\right)\right) d x ^ {\prime} \\ = \frac {1}{\alpha_ {M} \rho_ {M ^ {*}} ^ {\kappa} (x , a)} \int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a) M ^ {*} \left(x ^ {\prime} | x, a\right) \left(\log \frac {M ^ {*} \left(x ^ {\prime} | x , a\right)}{M \left(x ^ {\prime} | x , a\right)} - \log M ^ {*} \left(x ^ {\prime} | x, a\right)\right) d x ^ {\prime} (11) \\ = \frac {1}{\alpha_ {M} \rho_ {M ^ {*}} ^ {\kappa} (x , a)} \int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) \left(\log \frac {\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M} ^ {\kappa} (x , a , x ^ {\prime})} + \log \frac {\rho_ {M} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)} - \log M ^ {*} (x ^ {\prime} | x, a)\right) \mathrm {d} x ^ {\prime} \\ = \frac {1}{\alpha_ {M} \rho_ {M ^ {*}} ^ {\kappa} (x , a)} \left(\int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) \log \frac {\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M} ^ {\kappa} (x , a , x ^ {\prime})} d x ^ {\prime} - \right. \\ \rho_{M^{*}}^{\kappa}(x,a)\log \frac{\rho_{M^{*}}^{\kappa}(x,a)}{\rho_{M}^{\kappa}(x,a)}\underbrace{\int_{\mathcal{X}}M^{*}(x^{\prime}|x,a)\mathrm{d}x^{\prime}}_{= 1} - \rho_{M^{*}}^{\kappa}(x,a)\int_{\mathcal{X}}M^{*}(x^{\prime}|x,a)\log M^{*}(x^{\prime}|x,a)\mathrm{d}x^{\prime}\Bigg) \\ = \frac {1}{\alpha_ {0} (x , a)} \left(\int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) \log \frac {\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M} ^ {\kappa} (x , a , x ^ {\prime})} d x ^ {\prime} - \rho_ {M ^ {*}} ^ {\kappa} (x, a) \log \frac {\rho_ {M ^ {*}} ^ {\kappa} (x , a)}{\rho_ {M} ^ {\kappa} (x , a)} + \rho_ {M ^ {*}} ^ {\kappa} (x, a) H _ {M ^ {*}} (x, a)\right) (12) \\ \end{array}
$$
where  $\alpha_0(x,a) = \alpha_M\rho_{M^*}^\kappa (x,a)$
Definition A.5 ( $f$ -divergence). Given two distributions  $P$  and  $Q$ , two absolutely continuous density functions  $p$  and  $q$  with respect to a base measure  $\mathrm{d}x$  defined on the domain  $\mathcal{X}$ , we define the  $f$ -divergence [37],
$$
D _ {f} (P \| Q) = \int_ {\mathcal {X}} q (x) f \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x, \tag {13}
$$
where the generator function  $f: \mathbb{R}_+ \to \mathbb{R}$  is a convex, lower-semicontinuous function.
We notice that the terms  $\rho_{M^*}^\kappa (x,a,x')\log \frac{\rho_{M^*}^\kappa(x,a,x')}{\rho_M^\kappa(x,a,x')}$  and  $\rho_{M^*}^\kappa (x,a)\log \frac{\rho_{M^*}^\kappa(x,a)}{\rho_M^\kappa(x,a)}$  are the integrated functions in reverse KL divergence, which is an instance of  $f$  function in  $f$ -divergence (See Reverse
KL divergence of Tab.1 in [37] for more details). Replacing that form  $q \log \frac{q}{p}$  with  $q f\left(\frac{p}{q}\right)$ , we obtain a generalized representation of  $\rho_{M^*}^{\bar{\beta}^*}$ :
$$
\bar {\rho} _ {M ^ {*}} ^ {\bar {\beta} ^ {*}} := \frac {1}{\alpha_ {0} (x , a)} \left(\int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) f \left(\frac {\rho_ {M} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) \mathrm {d} x ^ {\prime} - \rho_ {M ^ {*}} ^ {\kappa} (x, a) \left(f \left(\frac {\rho_ {M} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) - H _ {M ^ {*}} (x, a)\right)\right), \tag {14}
$$
A.3 Proof of Thm. A.2
We first introduce several useful lemmas for the proof.
Lemma A.6. Rearrangement inequality The rearrangement inequality states that, for two sequences  $a_1 \geq a_2 \geq \ldots \geq a_n$  and  $b_1 \geq b_2 \geq \ldots \geq b_n$ , the inequalities
$$
a _ {1} b _ {1} + a _ {2} b _ {2} + \dots + a _ {n} b _ {n} \geq a _ {1} b _ {\pi (1)} + a _ {2} b _ {\pi (2)} + \dots + a _ {n} b _ {\pi (n)} \geq a _ {1} b _ {n} + a _ {2} b _ {n - 1} + \dots + a _ {n} b _ {1}
$$
hold, where  $\pi (1),\pi (2),\ldots ,\pi (n)$  is any permutation of  $1,2,\dots ,n$
Lemma A.7. For two sequences  $a_1 \geq a_2 \geq \ldots \geq a_n$  and  $b_1 \geq b_2 \geq \ldots \geq b_n$ , the inequalities
$$
\sum_ {i = 1} ^ {n} \frac {1}{n} a _ {i} b _ {i} \geq \sum_ {i = 1} ^ {n} \frac {1}{n} a _ {i} \sum \frac {1}{n} b _ {i}
$$
Proof. By rearrangement inequality, we have
$$
\sum_ {i = 1} ^ {n} a _ {i} b _ {i} \geq a _ {1} b _ {1} + a _ {2} b _ {2} + \dots + a _ {n} b _ {n}
$$
$$
\sum_ {i = 1} ^ {n} a _ {i} b _ {i} \geq a _ {1} b _ {2} + a _ {2} b _ {3} + \dots + a _ {n} b _ {1}
$$
$$
\sum_ {i = 1} ^ {n} a _ {i} b _ {i} \geq a _ {1} b _ {3} + a _ {2} b _ {4} + \dots + a _ {n} b _ {2}
$$
$$
\sum_ {i = 1} ^ {n} a _ {i} b _ {i} \geq a _ {1} b _ {n} + a _ {2} b _ {1} + \dots + a _ {n} b _ {n - 1}
$$
$$
n \sum_ {i = 1} ^ {n} a _ {i} b _ {i} \geq \sum_ {i = 1} ^ {n} a _ {i} \sum_ {i = 1} ^ {n} b _ {i}
$$