SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now we extend Lemma A.7 into the continuous integral scenario:
Lemma A.8. Given  $\mathcal{X} \subset \mathbb{R}$ , for two functions  $f: \mathcal{X} \to \mathbb{R}$  and  $g: \mathcal{X} \to \mathbb{R}$  that  $f(x) \geq f(y)$  if and only if  $g(x) \geq g(y)$ ,  $\forall x, y \in \mathcal{X}$ , the inequality
$$
\int_ {\mathcal {X}} p (x) f (x) g (x) \mathrm {d} x \geq \int_ {\mathcal {X}} p (x) f (x) \mathrm {d} x \int_ {\mathcal {X}} p (x) g (x) \mathrm {d} x
$$
holds, where  $p:\mathcal{X}\to \mathbb{R}$  and  $p(x) > 0,\forall x\in \mathcal{X}$  and  $\int_{\mathcal{X}}p(x)\mathrm{d}x = 1$
Proof. Since  $(f(x) - f(y))(g(x) - g(y))\geq 0,\forall x,y\in \mathcal{X}$ , we have
$$
\begin{array}{l} \int_ {x \in \mathcal {X}} \int_ {y \in \mathcal {X}} p (x) p (y) (f (x) - f (y)) (g (x) - g (y)) d y d x \geq 0 \\ \int_ {x \in \mathcal {X}} \int_ {y \in \mathcal {X}} p (x) p (y) f (x) g (x) + p (x) p (y) f (y) g (y) - p (x) p (y) f (x) g (y) - p (x) p (y) f (y) g (x) d y d x \geq 0 \\ \int_ {x \in \mathcal {X}} \int_ {y \in \mathcal {X}} p (x) p (y) f (x) g (x) + p (x) p (y) f (y) g (y) d y d x \geq \int_ {x \in \mathcal {X}} \int_ {y \in \mathcal {X}} p (x) p (y) f (x) g (y) + p (x) p (y) f (y) g (x) d y d x \\ \int_{x\in \mathcal{X}}\left(\int_{y\in \mathcal{X}}p(x)p(y)f(x)g(x)\mathrm{d}y + \int_{y\in \mathcal{X}}p(x)p(y)f(y)g(y)\mathrm{d}y\right)\mathrm{d}x\\ \geq \int_{x\in \mathcal{X}}\int_{y\in \mathcal{X}}p(x)p(y)f(x)g(y) + p(x)p(y)f(y)g(x)\mathrm{d}y\mathrm{d}x \\ \int_ {x \in \mathcal {X}} \left(p (x) f (x) g (x) + \int_ {y \in \mathcal {X}} p (x) p (y) f (y) g (y) \mathrm {d} y\right) \mathrm {d} x \geq \int_ {x \in \mathcal {X}} \int_ {y \in \mathcal {X}} p (x) p (y) f (x) g (y) + p (x) p (y) f (y) g (x) \mathrm {d} y \mathrm {d} x \\ \int_ {x \in \mathcal {X}} p (x) f (x) g (x) \mathrm {d} x + \int_ {x \in \mathcal {X}} \int_ {y \in \mathcal {X}} p (x) p (y) f (y) g (y) \mathrm {d} y \mathrm {d} x \geq \int_ {x \in \mathcal {X}} \int_ {y \in \mathcal {X}} p (x) p (y) f (x) g (y) + p (x) p (y) f (y) g (x) \mathrm {d} y \mathrm {d} x \\ \int_ {x \in \mathcal {X}} p (x) f (x) g (x) d x + \int_ {y \in \mathcal {X}} p (y) f (y) g (y) d y \geq \int_ {x \in \mathcal {X}} \int_ {y \in \mathcal {X}} p (x) p (y) f (x) g (y) + p (x) p (y) f (y) g (x) d y d x \\ 2 \int_ {x \in \mathcal {X}} p (x) f (x) g (x) d x \geq 2 \int_ {y \in \mathcal {X}} \int_ {x \in \mathcal {X}} p (x) p (y) f (x) g (y) d y d x \\ 2 \int_ {x \in \mathcal {X}} p (x) f (x) g (x) d x \geq 2 \int_ {x \in \mathcal {X}} p (x) f (x) d x \int_ {x \in \mathcal {X}} p (x) g (x) d x \\ \int_ {x \in \mathcal {X}} p (x) f (x) g (x) \mathrm {d} x \geq \int_ {x \in \mathcal {X}} p (x) f (x) \mathrm {d} x \int_ {x \in \mathcal {X}} p (x) g (x) \mathrm {d} x \\ \end{array}
$$
Corollary A.9. Let  $g\left(\frac{p(x)}{q(x)}\right) = -\log \frac{p(x)}{q(x)}$  where  $p(x) > 0, \forall x \in \mathcal{X}$  and  $q(x) > 0, \forall x \in \mathcal{X}$ , for  $v > 0$ , the inequality
$$
\int_ {\mathcal {X}} q (x) f \left(v \frac {p (x)}{q (x)}\right) g \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x \geq \int_ {\mathcal {X}} q (x) f \left(v \frac {p (x)}{q (x)}\right) \mathrm {d} x \int_ {\mathcal {X}} q (x) g \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x,
$$
holds if  $f'(x) \leq 0, \forall x \in \mathcal{X}$ . It is not always satisfied for  $f$  functions of  $f$ -divergence. We list a comparison of  $f$  on that condition in Tab. 3.
Proof.  $g'(x) = -\log x = -\frac{1}{x} < 0, \forall x \in \mathcal{X}$ . Suppose  $f'(x) \leq 0, \forall x \in \mathcal{X}$ , we have  $f(x) \geq f(y)$  if and only if  $g(x) \geq g(y), \forall x, y \in \mathcal{X}$  holds. Thus  $f(\upsilon \frac{p(x)}{q(x)}) \geq f(\upsilon \frac{p(y)}{q(y)})$  if and only if  $g(\frac{p(x)}{q(x)}) \geq g(\frac{p(y)}{q(y)})$ ,  $\forall x, y \in \mathcal{X}$  holds for all  $\upsilon > 0$ . By defining  $F(x) = f(\upsilon \frac{p(x)}{q(x)})$  and  $G(x) = g(\frac{p(x)}{q(x)})$  and using Lemma A.8, we have:
$$
\int_ {\mathcal {X}} q (x) F (x) G (x) \mathrm {d} x \geq \int_ {\mathcal {X}} q (x) F (x) \mathrm {d} x \int_ {\mathcal {X}} q (x) G (x) \mathrm {d} x.
$$
Then we know
$$
\int_ {\mathcal {X}} q (x) f \left(v \frac {p (x)}{q (x)}\right) g \left(\frac {p (x)}{q (x)}\right) d x \geq \int_ {\mathcal {X}} q (x) f \left(v \frac {p (x)}{q (x)}\right) d x \int_ {\mathcal {X}} q (x) g \left(\frac {p (x)}{q (x)}\right) d x
$$
holds.
Now, we prove Thm. A.2. For better readability, we first rewrite Thm. A.2 as follows:
Theorem A.10. Let  $\bar{\rho}_{M^*}^{\bar{\beta}^*}$  as the data distribution of the best-response policy  $\bar{\beta}^*$  in Eq. (4) under model  $M_{\theta}$  parameterized by  $\theta$ , then we can find the optimal  $\theta^*$  of  $\min_{\theta} \max_{\beta \in \Pi} \bar{L}(\rho_{M^*}^{\bar{\beta}}, M_{\theta})$  (Eq. (4)) via iteratively optimizing the objective  $\theta_{t+1} = \min_{\theta} \bar{L}(\bar{\rho}_{M^*}^{\bar{\beta}^*}, M_{\theta})$ , where  $\bar{\rho}_{M^*}^{\bar{\beta}^*}$  is approximated via the last-iteration model  $M_{\theta_t}$ . Based on Corollary A.9, we have an upper bound objective for
Table 3: Properties of  $f^{\prime}(x) \leq 0, \forall x \in \mathcal{X}$  for  $f$ -divergences.
Name	Generator function f(x)	If f'(x) ≤ 0, ∀x ∈ X
Kullback-Leibler	x log x	False
Reverse KL	- log x	True
Pearson χ2	(x-1)2	False
Squared Hellinger	(√x - 1)2	False
Jensen-Shannon	-(x+1) log 1+x/2 + x log x	False
GAN	x log x - (x+1) log(x+1)	True
$\min_{\theta}\bar{L} (\bar{\rho}_{M^{*}}^{\bar{\beta}^{*}},M_{\theta})$  and derive the following objective
$$
\theta_ {t + 1} = \arg \max  _ {\theta} \mathbb {E} _ {\rho_ {M ^ {*}} ^ {\kappa}} \left[ \frac {1}{\alpha_ {0} (x , a)} \log M _ {\theta} (x ^ {\prime} | x, a) \underbrace {\left(f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) - f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) + H _ {M ^ {*}} (x , a)\right)} _ {W (x, a, x ^ {\prime})} \right],
$$
where  $\alpha_0(x,a) = \alpha_{M_{\theta_t}}\rho_{M^*}^\kappa (x,a),\mathbb{E}_{\rho_{M^*}}^{\kappa}[\cdot ]$  denotes  $\mathbb{E}_{x,a,x^{\prime}\sim \rho_{M^{*}}^{\kappa}}[\cdot ]$ $f$  is the generator function in  $f$  -divergence which satisfies  $f^{\prime}(x)\leq 0,\forall x\in \mathcal{X}$  , and  $\theta$  is the parameters of  $M$  .  $M_{\theta_t}$  denotes a probability function with the same parameters as the learned model (i.e.,  $\bar{\theta} = \theta$  ) but the parameter is fixed and only used for sampling.
Proof. Let  $\bar{\rho}_{M^*}^{\beta^*}$  as the data distribution of the best-response policy  $\bar{\beta}^*$  in Eq. (4) under model  $M_{\theta}$  parameterized by  $\theta$ , then we can find the optimal  $\theta_{t+1}$  of  $\min_{\theta} \max_{\beta \in \Pi} \bar{L}(\rho_{M^*}^{\beta}, M_{\theta})$  (Eq. (4)) via iteratively optimizing the objective  $\theta_{t+1} = \min_{\theta} \bar{L}(\bar{\rho}_{M^*}^{\bar{\beta}^*}, M_{\theta})$ , where  $\bar{\rho}_{M^*}^{\bar{\beta}^*}$  is approximated via the last-iteration model  $M_{\theta_t}$ :
$$
\theta_ {t + 1} = \min  _ {\theta} \int_ {\mathcal {X}, \mathcal {A}} \bar {\rho} _ {M ^ {*}} ^ {\beta^ {*}} (x, a) \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) (- \log M _ {\theta} \left(x ^ {\prime} | x, a\right)) d x ^ {\prime} d a d x \tag {15}
$$
$$
\begin{array}{l} = \min  _ {\theta} \int_ {\mathcal {X}, \mathcal {A}} \frac {1}{\alpha_ {0} (x , a)} \left(\int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) d x ^ {\prime} \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) (- \log M _ {\theta} \left(x ^ {\prime} | x, a\right)) d x ^ {\prime} \right. \\ \left. - \rho_ {M ^ {*}} ^ {\kappa} (x, a) \left(f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) - H _ {M ^ {*}} (x, a)\right) \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) \left(- \log M _ {\theta} \left(x ^ {\prime} | x, a\right)\right) d x ^ {\prime}\right) d a d x \\ \end{array}
$$
$$
\begin{array}{l} = \min  _ {\theta} \int_ {\mathcal {X}, \mathcal {A}} \frac {1}{\alpha_ {0} (x , a)} \left(\int_ {\mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) d x ^ {\prime} \left(\int_ {\mathcal {X}} M ^ {*} (x ^ {\prime} | x, a) (- \log \frac {M _ {\theta} (x ^ {\prime} | x , a)}{M ^ {*} (x ^ {\prime} | x , a)}) d x ^ {\prime} + H _ {M ^ {*}} (x, a)\right) \right. \\ \left. - \rho_ {M ^ {*}} ^ {\kappa} (x, a) \left(f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) - H _ {M ^ {*}} (x, a)\right) \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) \left(- \log M _ {\theta} \left(x ^ {\prime} | x, a\right)\right) d x ^ {\prime}\right) d a d x \\ \end{array}
$$
$$
\begin{array}{l} \leq \min  _ {\theta} \int_ {\mathcal {X}, \mathcal {A}} \frac {1}{\alpha_ {0} (x , a)} \left(\underbrace {\rho_ {M ^ {*}} ^ {\kappa} (x , a) \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x , a\right) f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) (- \log \frac {M _ {\theta} \left(x ^ {\prime} | x , a\right)}{\bar {M} ^ {*} \left(x ^ {\prime} | x , a\right)}) \mathrm {d} x ^ {\prime}} _ {\text {b a s e d o n C o r o l l a r y A . 9}}\right) \\ \left. - \rho_ {M ^ {*}} ^ {\kappa} (x, a) \left(f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) - H _ {M ^ {*}} (x, a)\right) \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) \left(- \log M _ {\theta} \left(x ^ {\prime} | x, a\right)\right) d x ^ {\prime}\right) d a d x \\ \end{array}
$$
$$
\begin{array}{l} = \min  _ {\theta} \int_ {\mathcal {X}, \mathcal {A}} \frac {1}{\alpha_ {0} (x , a)} \left(\rho_ {M ^ {*}} ^ {\kappa} (x, a) \int_ {\mathcal {X}} \left(M ^ {*} \left(x ^ {\prime} | x, a\right) f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) (- \log M _ {\theta} \left(x ^ {\prime} | x, a\right))\right) d x ^ {\prime} \right. \\ \left. - \rho_ {M ^ {*}} ^ {\kappa} (x, a) \left(f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) - H _ {M ^ {*}} (x, a)\right) \int_ {\mathcal {X}} M ^ {*} \left(x ^ {\prime} | x, a\right) \left(- \log M _ {\theta} \left(x ^ {\prime} | x, a\right)\right) d x ^ {\prime}\right) d a d x \\ = \max  _ {\theta} \int_ {\mathcal {X}, \mathcal {A}, \mathcal {X}} \frac {1}{\alpha_ {0} (x , a)} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) \log M _ {\theta} \left(x ^ {\prime} \mid x, a\right) \left(f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) - f \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) + H _ {M ^ {*}} (x, a)\right) d x ^ {\prime} d a d x, \tag {16} \\ \end{array}
$$
where  $M_{\theta_t}$  is introduced to approximate the term  $\vec{\rho}_{M^*}^{\vec{\beta}^*}$  and fixed when optimizing  $\theta$ . In Eq. (15),  $\| \rho_{M^*}^\beta (\cdot ,\cdot)\| _2^2$  for Eq. (7) is eliminated as it does not contribute to the gradient of  $\theta$ . Assume  $f^{\prime}(x)\leq 0,\forall x\in \mathcal{X}$ , let  $v(x,a)\coloneqq \frac{\rho_{M_{\theta_t}}^\kappa(x,a)}{\rho_{M^*}^\kappa(x,a)} >0$ ,  $p(x^{\prime}|x,a) = M_{\theta}(x^{\prime}|x,a)$ , and  $q(x^{\prime}|x,a) = M^{*}(x^{\prime}|x,a)$ , the first inequality can be derived by adopting Corollary A.9 and eliminating the first  $H_{M^*}$  since it does not contribute to the gradient of  $\theta$ .
A.4 Proof of the Tractable Solution
Now we are ready to prove the tractable solution:
Proof. The core challenge is that the term  $f\left(\frac{\rho_{M_{\theta_t}}^\kappa(x,a,x')}{\rho_{M^*}^\kappa(x,a,x')}\right) - f\left(\frac{\rho_{M_{\theta_t}}^\kappa(x,a)}{\rho_{M^*}^\kappa(x,a)}\right)$  is still intractable. In the following, we give a tractable solution to Thm. A.2. First, we resort to the first-order approximation. Given some  $u \in (1 - \xi, 1 + \xi), \xi > 0$ , we have
$$
f (u) \approx f (1) + f ^ {\prime} (u) (u - 1), \tag {17}
$$
where  $f'$  is the first-order derivative of  $f$ . By Taylor's formula and the fact that  $f'(u)$  of the generator function  $f$  is bounded in  $(1 - \xi, 1 + \xi)$ , the approximation error is no more than  $\mathcal{O}(\xi^2)$ . Substituting  $u$  with  $\frac{p(x)}{q(x)}$  in Eq. (17), the pattern  $f\left(\frac{p(x)}{q(x)}\right)$  in Eq. (16) can be converted to  $\frac{p(x)}{q(x)} f'\left(\frac{p(x)}{q(x)}\right) - f'\left(\frac{p(x)}{q(x)}\right) + f(1)$ , then we have:
Note that the part  $\rho_{M^*}^\kappa(x, a)$  in  $\rho_{M^*}^\kappa(x, a, x')$  can be canceled because of  $\alpha_0(x, a) = \alpha_{M_{\theta_t}} \rho_{M^*}^\kappa(x, a)$ , but we choose to keep it and ignore  $\alpha_0(x, a)$ . The benefit is that we can estimate  $\rho_{M^*}^\kappa(x, a, x')$  from an empirical data distribution through data collected by  $\kappa$  in  $M^*$  directly, rather than from a uniform distribution which is harder to be generated. Although keeping  $\rho_{M^*}^\kappa(x, a)$  incurs extra bias in theory, the results in our experiments show that it has not made significant negative effects in practice. We
leave this part of modeling in future work. In particular, by ignoring  $\alpha_0(x,a)$ , we have:
$$
\begin{array}{l} \theta_ {t + 1} = \arg \max  _ {\theta} \int_ {\mathcal {X}, \mathcal {A}, \mathcal {X}} \rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x, a, x) \left(f ^ {\prime} \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) - f ^ {\prime} \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right)\right) \log M _ {\theta} \left(x ^ {\prime} | x, a\right) d x ^ {\prime} d a d x + (18) \\ \int_ {\mathcal {X}, \mathcal {A}, \mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) \left(f ^ {\prime} \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a)}{\rho_ {M ^ {*}} ^ {\kappa} (x , a)}\right) - f ^ {\prime} \left(\frac {\rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x , a , x ^ {\prime})}{\rho_ {M ^ {*}} ^ {\kappa} (x , a , x ^ {\prime})}\right) + H _ {M ^ {*}} (x, a)\right) \log M _ {\theta} \left(x ^ {\prime} | x, a\right) d x ^ {\prime} d a d x. (19) \\ \end{array}
$$
We can estimate  $f' \left( \frac{\rho_{M_{\theta_t}}^\kappa(x, a)}{\rho_{M^*}^\kappa(x, a)} \right)$  and  $f' \left( \frac{\rho_{M_{\theta_t}}^\kappa(x, a, x')}{\rho_{M^*}^\kappa(x, a, x')} \right)$  through Lemma A.11.
Lemma A.11  $(f'(\frac{p}{q})$  estimation [36]). Given a function  $T_{\varphi}:\mathcal{X}\to \mathbb{R}$  parameterized by  $\varphi \in \Phi$ , if  $f$  is convex and lower semi-continuous, by finding the maximum point of  $\varphi$  in the following objective:
$$
\varphi^ {*} = \arg \max _ {\varphi} \mathbb {E} _ {x \sim p (x)} \left[ T _ {\varphi} (x) \right] - \mathbb {E} _ {x \sim q (x)} \left[ f ^ {*} (T _ {\varphi} (x)) \right],
$$
we have  $f'(\frac{p(x)}{q(x)}) = T_{\varphi^*}(x)$ .  $f^*$  is Fenchel conjugate of  $f$  [23].
$$
\begin{array}{l} \varphi_ {0} ^ {*} = \arg \max  _ {\varphi_ {0}} \mathbb {E} _ {x, a, x ^ {\prime} \sim \rho_ {M ^ {*}} ^ {\kappa}} \left[ T _ {\varphi_ {0}} (x, a, x ^ {\prime}) \right] - \mathbb {E} _ {x, a, x ^ {\prime} \sim \rho_ {M _ {\theta_ {t}}} ^ {\kappa}} \left[ f ^ {*} \left(T _ {\varphi_ {0}} (x, a, x ^ {\prime})\right) \right] \\ \varphi_ {1} ^ {*} = \arg \max _ {\varphi_ {1}} \mathbb {E} _ {x, a \sim \rho_ {M ^ {*}} ^ {\kappa}} [ T _ {\varphi_ {1}} (x, a) ] - \mathbb {E} _ {x, a \sim \rho_ {M _ {\theta_ {t}}} ^ {\kappa}} [ f ^ {*} (T _ {\varphi_ {1}} (x, a)) ], \\ \end{array}
$$
then we have  $f^{\prime}\left(\frac{\rho_{M_{\theta_t}}^{\kappa}(x,a,x^{\prime})}{\rho_{M^*}^{\kappa}(x,a,x^{\prime})}\right) \approx T_{\varphi_0^*}(x,a,x^{\prime})$  and  $f^{\prime}\left(\frac{\rho_{M_{\theta_t}}^{\kappa}(x,a)}{\rho_{M^*}^{\kappa}(x,a)}\right) \approx T_{\varphi_1^*}(x,a)$ . Given  $\varphi_0^*$  and  $\varphi_1^*$ , let  $A_{\varphi_0^*,\varphi_1^*}(x,a,x') = T_{\varphi_0^*}(x,a,x') - T_{\varphi_1^*}(x,a)$ , then we can optimize  $\theta$  via:
$$
\begin{array}{l} \theta_ {t + 1} = \arg \max _ {\theta} \int_ {\mathcal {X}, \mathcal {A}, \mathcal {X}} \rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x, a, x) \left(T _ {\varphi_ {0} ^ {*}} (x, a, x ^ {\prime}) - T _ {\varphi_ {1} ^ {*}} (x, a)\right) \log M _ {\theta} (x ^ {\prime} | x, a) d x ^ {\prime} d a d x + \\ \int_ {\mathcal {X}, \mathcal {A}, \mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) \left(T _ {\varphi_ {1} ^ {*}} (x, a) - T _ {\varphi_ {0} ^ {*}} (x, a, x ^ {\prime}) + H _ {M ^ {*}} (x, a)\right) \log M _ {\theta} (x ^ {\prime} | x, a) d x ^ {\prime} d a d x \\ = \arg \max _ {\theta} \int_ {\mathcal {X}, \mathcal {A}, \mathcal {X}} \rho_ {M _ {\theta_ {t}}} ^ {\kappa} (x, a, x) A _ {\varphi_ {0} ^ {*}, \varphi_ {1} ^ {*}} (x, a, x ^ {\prime}) \log M _ {\theta} (x ^ {\prime} | x, a) \mathrm {d} x ^ {\prime} \mathrm {d} a \mathrm {d} x + \\ \int_ {\mathcal {X}, \mathcal {A}, \mathcal {X}} \rho_ {M ^ {*}} ^ {\kappa} (x, a, x ^ {\prime}) (- A _ {\varphi_ {0} ^ {*}, \varphi_ {1} ^ {*}} (x, a, x ^ {\prime}) + H _ {M ^ {*}} (x, a)) \log M _ {\theta} (x ^ {\prime} | x, a) \mathrm {d} x ^ {\prime} \mathrm {d} a \mathrm {d} x. \\ \end{array}
$$
Based on the specific  $f$ -divergence, we can represent  $T$  and  $f^{*}(T)$  with a discriminator  $D_{\varphi}$ . It can be verified that  $f(u) = u\log u - (u + 1)\log (u + 1)$ ,  $T_{\varphi}(u) = \log D_{\varphi}(u)$ , and  $f^{*}(T_{\varphi}(u)) = -\log (1 - D_{\varphi}(u))$  proposed in [37] satisfies the condition  $f^{\prime}(x)\leq 0,\forall x\in \mathcal{X}$  (see Tab. 3). We select the former in the implementation and convert the tractable solution to:
$$
\theta_ {t + 1} = \arg \max _ {\theta} \mathbb {E} _ {\rho_ {M _ {\theta_ {t}}} ^ {\kappa}} \left[ A _ {\varphi_ {0} ^ {*}, \varphi_ {1} ^ {*}} (x, a, x ^ {\prime}) \log M _ {\theta} (x ^ {\prime} | x, a) \right] + \mathbb {E} _ {\rho_ {M ^ {*}} ^ {\kappa}} \left[ (H _ {M ^ {*}} (x, a) - A _ {\varphi_ {0} ^ {*}, \varphi_ {1} ^ {*}} (x, a, x ^ {\prime})) \log M _ {\theta} (x ^ {\prime} | x, a) \right]
$$
$$
\begin{array}{l} s. t. \qquad \varphi_ {0} ^ {*} = \arg \max _ {\varphi_ {0}} \mathbb {E} _ {\rho_ {M} ^ {\kappa}} \left[ \log D _ {\varphi_ {0}} (x, a, x ^ {\prime}) \right] + \mathbb {E} _ {\rho_ {M} ^ {\kappa} _ {\theta_ {t}}} \left[ \log (1 - D _ {\varphi_ {0}} (x, a, x ^ {\prime})) \right] \\ \varphi_ {1} ^ {*} = \arg \max  _ {\varphi_ {1}} \mathbb {E} _ {\rho_ {M ^ {*}}} ^ {\kappa} [ \log D _ {\varphi_ {1}} (x, a) ] + \mathbb {E} _ {\rho_ {M _ {\theta_ {t}}} ^ {\kappa}} [ \log (1 - D _ {\varphi_ {1}} (x, a)) ], \tag {20} \\ \end{array}
$$
where  $A_{\varphi_0^*,\varphi_1^*}(x,a,x') = \log D_{\varphi_0^*}(x,a,x') - \log D_{\varphi_1^*}(x,a),\mathbb{E}_{\rho_{M_{\theta_t}}^\kappa}[\cdot ]$  is a simplification of  $\mathbb{E}_{x,a,x'\sim \rho_{M_{\theta_t}}^\kappa}[\cdot ]$
Remark A.12. In practice, we need to use the real-world data to construct the distribution  $\rho_{M^*}^\kappa$  and the generative data to construct  $\rho_{M_{\theta_t}}^\kappa$ . In the offline model-learning setting, we only have a real-world dataset  $\mathcal{D}_{\mathrm{real}}$  collected by the behavior policy  $\mu$ . We can learn a policy  $\hat{\mu} \approx \mu$  via imitation learning based on  $\mathcal{D}_{\mathrm{real}}$  [40, 25] and let  $\hat{\mu}$  be the policy  $\kappa$ . Then we can regard  $\mathcal{D}_{\mathrm{real}}$  as the empirical
data distribution of  $\rho_{M^*}^\kappa$  and the trajectories collected by  $\hat{\mu}$  in the model  $M_{\theta_t}$  as the empirical data distribution of  $\rho_{M_{\theta_t}}^\kappa$ . Based on the above specializations, we have:
$$
\theta_ {t + 1} = \max _ {\theta} \left(\mathbb {E} _ {\rho_ {M _ {\theta_ {t}}} ^ {\hat {\mu}}} \left[ A _ {\varphi_ {0} ^ {*}, \varphi_ {1} ^ {*}} (x, a, x ^ {\prime}) \log M _ {\theta} (x ^ {\prime} | x, a) \right] + \mathbb {E} _ {\rho_ {M ^ {*}} ^ {\mu}} \left[ \left(H _ {M ^ {*}} (x, a) - A _ {\varphi_ {0} ^ {*}, \varphi_ {1} ^ {*}} (x, a, x ^ {\prime})\right) \log M _ {\theta} (x ^ {\prime} | x, a) \right]\right)
$$
$$
s. t. \quad \varphi_ {0} ^ {*} = \arg \max  _ {\varphi_ {0}} \left(\mathbb {E} _ {\rho_ {M ^ {*}} ^ {\mu}} \left[ \log D _ {\varphi_ {0}} \left(x, a, x ^ {\prime}\right) \right] + \mathbb {E} _ {\rho_ {M _ {\theta_ {t}}} ^ {\hat {\mu}}} \left[ \log \left(1 - D _ {\varphi_ {0}} \left(x, a, x ^ {\prime}\right)\right) \right]\right)
$$
$$
\varphi_ {1} ^ {*} = \arg \max _ {\varphi_ {1}} \left(\mathbb {E} _ {\rho_ {M ^ {*}}} ^ {\mu} [ \log D _ {\varphi_ {1}} (x, a) ] + \mathbb {E} _ {\rho_ {M _ {\theta_ {t}}} ^ {\hat {\mu}}} [ \log (1 - D _ {\varphi_ {1}} (x, a)) ]\right),
$$
which is Eq. (6) in the main body.
B Discussion and Limitations of the Theoretical Results
We summarize the limitations of current theoretical results and future work as follows:
1. As discussed in Remark A.4, the solution Eq. (10) relies on  $\rho_{M^*}^\beta(x,a) \in [0,c], \forall a \in \mathcal{A}, \forall x \in \mathcal{X}$ . In some particular  $M^*$ , it is intractable to derive a  $\beta$  that can generate an occupancy specified by  $g(x,a) / \alpha_M$ . If more knowledge of  $M^*$  or  $\beta^*$  is provided or some mild assumptions can be made on the properties of  $M^*$  or  $\beta^*$ , we may model  $\rho$  in a more sophisticated approach to alleviating the above problem.
2. In the tractable solution derivation, we ignore the term  $\alpha_0(x,a) = \alpha_{M_{\theta_t}}\rho_{M^*}^\kappa (x,a)$  (See Eq. (19)). The benefit is that  $\rho_{M^{*}}^{\kappa}(x,a,x^{\prime})$  in the tractable solution can be estimated through offline datasets directly. Although the results in our experiments show that it does not produce significant negative effects in these tasks, ignoring  $\rho_{M^{*}}^{\kappa}(x,a)$  indeed incurs extra bias in theory. In future work, techniques for estimating  $\rho_{M^{*}}^{\kappa}(x,a)$  [33] can be incorporated to correct the bias. On the other hand,  $\alpha_{M_{\theta_t}}$  is also ignored in the process.  $\alpha_{M_{\theta_t}}$  can be regarded as a global rescaling term of the final objective Eq. (19). Intuitively, it constructs an adaptive learning rate for Eq. (19), which increases the step size when the model is better fitted and decreases the step size otherwise. It can be considered to further improve the learning process in future work, e.g., cooperating with empirical risk minimization by balancing the weights of the two objectives through  $\alpha_{M_{\theta_t}}$ .
C Societal Impact
This work studies a method toward counterfactual environment model learning. Reconstructing an accurate environment of the real world will promote the wide adoption of decision-making policy optimization methods in real life, enhancing our daily experience. We are aware that decision-making policy in some domains like recommendation systems that interact with customers may have risks of causing price discrimination and misleading customers if inappropriately used. A promising way to reduce the risk is to introduce fairness into policy optimization and rules to constrain the actions (Also see our policy design in Sec. G.1.3). We are involved in and advocating research in such directions. We believe that business organizations would like to embrace fair systems that can ultimately bring long-term financial benefits by providing a better user experience.
D AWRM-oracle Pseudocode
We list the pseudocode of AWRM-oracle in Alg. 2.
E Implementation
E.1 Details of the GALILEO Implementation
The approximation of Eq. (17) holds only when  $p(x) / q(x)$  is close to 1, which might not be satisfied. To handle the problem, we inject a standard supervised learning loss