SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\bar {A} _ {\pi} := \left\{a: a \sim \pi (s), s \in \bar {S} _ {\pi} \right\} \quad \text {a n d} \quad \bar {A} _ {\pi_ {\text {r e f}}} := \left\{a: a \sim \pi_ {\text {r e f}} (s), s \in \bar {S} _ {\pi_ {\text {r e f}}} \right\}
$$
$$
\mathcal {D} _ {\text {m o d e l}} := \mathcal {D} _ {\text {m o d e l}} \cup \{\bar {S} _ {\pi}, \bar {A} _ {\pi} \} \cup \{\bar {S} _ {\pi_ {\text {r e f}}}, \bar {A} _ {\pi_ {\text {r e f}}} \}
$$
$$
\bar {S} _ {\pi} \leftarrow \{s ^ {\prime} \mid s ^ {\prime} \sim \operatorname {d e t a c h} \left(P _ {M} (\cdot \mid s, a)\right), s \in \bar {S} _ {\pi}, a \in \bar {A} _ {\pi} \}
$$
$$
\bar {S} _ {\pi_ {\text {r e f}}} \leftarrow \left\{s ^ {\prime} \mid s ^ {\prime} \sim \operatorname {d e t a c h} \left(P _ {M} (\cdot \mid s, a)\right), s \in \bar {S} _ {\pi_ {\text {r e f}}}, a \in \bar {A} _ {\pi_ {\text {r e f}}} \right\}
$$
9: end for
in the inner minimization of Eq. (1) to a regularization term and introducing an additional critic function. To clearly elaborate this, we first present the regularized objective in its complete form, and subsequently derive it from Eq. (1).
Let  $\mathcal{F}:\{f:S\times \mathcal{A}\to [0,V_{\mathrm{max}}]\}$  be a class of critic functions. The regularized objective is given as
$$
\tilde {\pi} \in \underset {\pi \in \Pi} {\operatorname {a r g m a x}} \mathcal {L} _ {d _ {M} ^ {\pi_ {\text {r e f}}}} (\pi , f) \tag {6}
$$
$$
\text {s . t .} f ^ {\pi} \in \operatorname * {a r g m i n} _ {M \in \mathcal {M}, f \in \mathcal {F}} \mathcal {L} _ {d _ {M} ^ {\pi_ {\mathrm {r e f}}}} (\pi , f) + \beta \left(\mathcal {E} _ {\rho_ {\pi_ {\mathrm {r e f}}, \pi}} (\pi , f, M) + \lambda \mathcal {E} _ {\mathcal {D}} (M)\right)
$$
where  $\mathcal{E}_{\mathcal{D}}(M) = \sum_{\mathcal{D}} - \log P_M(s' \mid s, a) + (R_M(s, a) - r)^2 / V_{\max}^2$  is the model-fitting error,  $\mathcal{L}_{d_M^{\pi_{\mathrm{ref}}}}(\pi, f) \coloneqq \mathbb{E}_{d_M^{\pi_{\mathrm{ref}}}}[f(s, \pi) - f(s, \pi_{\mathrm{ref}})]$  is equal to the performance difference  $(1 - \gamma)(J_M(\pi) - J_M(\pi_{\mathrm{ref}}))$ ,  $\mathcal{E}_{\rho_{\pi_{\mathrm{ref}}, \pi}}(\pi, f, M)$  denotes the squared Bellman error on the distribution  $\rho_{\pi_{\mathrm{ref}}, \pi}$  that denotes the distribution generated by first running  $\pi_{\mathrm{ref}}$  and then rolling out  $\pi$  in  $M$  (with a switching time sampled from a geometric distribution of  $\gamma$ ), and  $\beta, \lambda$  act as the Lagrange multipliers.
This regularized formulation in Eq. (6) can be derived as follows. Assuming  $Q_M^\pi \in \mathcal{F}$ , and using the facts that  $J_M(\pi) = \mathbb{E}_{d_0}[Q_M^\pi (s,\pi)]$  and the Bellman equation  $Q_{M}^{\pi}(s,a) = r_{M}(s,a) + \gamma \mathbb{E}_{s^{\prime}\sim P_{M}(s,a)}[Q_{M}^{\pi}(s^{\prime},\pi)]$ , we can rewrite Eq. (1) as
$$
\max  _ {\pi \in \Pi} \min  _ {M \in \mathcal {M}, f \in \mathcal {F}} \quad \mathbb {E} _ {d _ {M} ^ {\pi_ {\text {r e f}}} \left[ f (s, \pi) - f (s, \pi_ {\text {r e f}}) \right]}. \tag {7}
$$
$$
\text {s . t .} \quad \mathcal {E} _ {\mathcal {D}} (M) \leq \alpha + \min  _ {M ^ {\prime} \in \mathcal {M}} \mathcal {E} _ {\mathcal {D}} \left(M ^ {\prime}\right)
$$
$$
\forall s, a \in \operatorname {s u p p} \left(\rho_ {\pi_ {\text {r e f}}, \pi}\right), \quad f (s, a) = r _ {M} (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P _ {M} (s, a)} [ f (s ^ {\prime}, \pi) ]
$$
We then convert the constraints in Eq. (7) into regularization terms in the inner minimization by introducing Lagrange multipliers  $(\beta, \lambda)$ , following (Xie et al., 2021a; Cheng et al., 2022), and drop the constants not affected by  $M, f, \pi$ , which results in Eq. (6).
4.2 Algorithm Details
Algorithm 1 is an iterative solver for approximating the solution to Eq. (6). Here we further approximate  $d_{M}^{\pi_{\mathrm{ref}}}$  and  $\rho_{\pi_{\mathrm{ref}},\pi}$  in Eq. (6) using samples from the state-action buffer  $\mathcal{D}_{\mathrm{model}}$ . We want ensure that  $\mathcal{D}_{\mathrm{model}}$  has a larger coverage than both  $d_{M}^{\pi_{\mathrm{ref}}}$  and  $\rho_{\pi_{\mathrm{ref}},\pi}$ . We do so heuristically, by constructing the model replay buffer  $\mathcal{D}_{\mathrm{model}}$  through repeatedly rolling out  $\pi$  and  $\pi_{\mathrm{ref}}$  with the adversarially trained MDP model  $M$ , such that  $\mathcal{D}_{\mathrm{model}}$  contains a diverse training set of state-action tuples.
Specifically, the algorithm takes as input an offline dataset  $\mathcal{D}_{\mathrm{real}}$ , a policy  $\pi$ , an MDP model  $M$  and two critic networks  $f_{1}, f_{2}$ . At every iteration, the algorithm proceeds in two stages. First, the adversary is optimized to find a data-consistent model that minimizes the performance difference with the reference policy. We sample mini-batches of only states and actions  $\mathcal{D}_{\mathrm{real}}^{\mathrm{mini}}$  and  $\mathcal{D}_{\mathrm{model}}^{\mathrm{mini}}$  from the real and model-generated datasets respectively (Line 4). The MDP model  $M$  is queried on these mini-batches to generate next-state and reward predictions. The adversary then updates the model and Q-functions (Line 5) using the gradient of the loss described in Eq. (4), where
$$
\mathcal {L} _ {\mathcal {D} _ {M}} (f, \pi , \pi_ {\text {r e f}}) := \mathbb {E} _ {\mathcal {D} _ {M}} [ f (s, \pi (s)) - f (s, \pi_ {\text {r e f}} (s) ]
$$
$$
\mathcal {E} _ {\mathcal {D} _ {M}} ^ {w} (f, M, \pi) := (1 - w) \mathcal {E} _ {\mathcal {D}} ^ {t d} (f, f, M, \pi) + w \mathcal {E} _ {\mathcal {D}} ^ {t d} (f, \bar {f}, M, \pi)
$$
$$
\mathcal {E} _ {\mathcal {D} _ {\mathrm {r e a l}} ^ {\mathrm {m i n i}}} (M) := \mathbb {E} _ {\mathcal {D} _ {\mathrm {r e a l}} ^ {\mathrm {m i n i}}} \left[ - \log P _ {M} \left(s ^ {\prime} \mid s, a\right) + \left(R _ {M} (s, a) - r\right) ^ {2} / V _ {\max } ^ {2} \right]
$$
$\mathcal{L}_{\mathcal{D}_M}$  is the pessimistic loss term that forces the  $f$  to predict a lower value for the learner than the reference on the sampled states.  $\mathcal{E}_{\mathcal{D}_M}^w$  is the Bellman surrogate to encourage the Q-functions to be consistent with the model-generated data  $\mathcal{D}_M$ . We use the double Q residual algorithm loss similar to Cheng et al. (2022), which is defined as a convex combination of the temporal difference losses with respect to the critic and the delayed target networks,  $\mathcal{E}_{\mathcal{D}}^{td}(f,f',M,\pi)\coloneqq \mathbb{E}_{\mathcal{D}}\left[(f(s,a) - r - \gamma f'(s',\pi))^2\right]$ .  $\mathcal{E}_{\mathcal{D}}(M)$  is the model-fitting loss that ensures the model is data-consistent.  $\beta$  and  $\lambda$  control the effect of the pessimistic loss, by constraining Q-functions and models the adversary can choose. Once the adversary is updated, we update the policy (Line 6) to maximize the pessimistic loss as defined in Eq. (5). Similar to Cheng et al. (2022), we choose one Q-function and a slower learning rate for the policy updates ( $\eta_{\mathrm{fast}}\gg \eta_{\mathrm{slow}}$ ).
We remark that  $\mathcal{E}_{\mathcal{D}_M}^w$  not only affects  $f_1, f_2$ , but also  $M$ , i.e., it forces the model to generate transitions where the Q-function is Bellman consistent. This allows the pessimistic loss to indirectly affect the model learning, thus making the model adversarial. Consider the special case where  $\lambda = 0$  in the loss of Line 4. The model here is no longer forced to be data consistent, and the adversary can now freely update the model via  $\mathcal{E}_{\mathcal{D}_M}^w$  such that the Q-function is always Bellman consistent. As a consequence, the algorithm becomes equivalent to IL on the model-generated states. We empirically study this behavior in our experiments (Section 5).
Lines 7 and 8 describe our model-based rollout procedure. We incrementally rollout both  $\pi$  and  $\pi_{\mathrm{ref}}$  from states in  $\mathcal{D}_{\mathrm{real}}^{\mathrm{mini}}$  for a horizon  $H$ , and add the generated transitions to  $\mathcal{D}_{\mathrm{model}}$ . The aim of this strategy is to generate a distribution with large coverage for training the adversary and policy, and we discuss this in detail in the next section.
Finally, it is important to note the fact that neither the pessimistic nor the Bellman surrogate losses use the real transitions; hence our algorithm is completely model-based from a statistical point of view, that the value function  $f$  is solely an intermediate variable that helps in-model optimization and not directly fit from data.
We test the efficacy of ARMOR on two major fronts: (1) performance comparison to existing offline RL algorithms, and (2) robust policy improvement over a reference policy that is not covered by the dataset, a novel setting that is not applicable to existing works. We use the D4RL (Fu et al., 2020) continuous control benchmarks datasets for all our experiments and the code will be made public.
Experimental Setup: We parameterize  $\pi, f_1, f_2$  and  $M$  using feedforward neural networks, and set  $\eta_{\mathrm{fast}} = 5e - 4$ ,  $\eta_{\mathrm{slow}} = 5e - 7$ ,  $w = 0.5$  similar to Cheng et al. (2022). In all our experiments, we vary only the  $\beta$  and  $\lambda$  parameters which control the amount of pessimism; others are fixed. Importantly, we set the rollout horizon to be the max episode horizon defined in the environment.
4In Appendix F we empirically show how imitation learning can be obtained as a special case of ARMOR
Dataset	ARMOR	MoREL	MOPO	RAMBO	COMBO	ATAC	CQL	IQL	BC
hopper-med	101.4	95.4	28.0	92.8	97.2	85.6	86.6	66.3	29.0
walker2d-med	90.7	77.8	17.8	86.9	81.9	89.6	74.5	78.3	6.6
halfcheetah-med	54.2	42.1	42.3	77.6	54.2	53.3	44.4	47.4	36.1
hopper-med-replay	97.1	93.6	67.5	96.6	89.5	102.5	48.6	94.7	11.8
walker2d-med-replay	85.6	49.8	39.0	85.0	56.0	92.5	32.6	73.9	11.3
halfcheetah-med-replay	50.5	40.2	53.1	68.9	55.1	48.0	46.2	44.2	38.4
hopper-med-exp	103.4	108.7	23.7	83.3	111.1	111.9	111.0	91.5	111.9
walker2d-med-exp	112.2	95.6	44.6	68.3	103.3	114.2	98.7	109.6	6.4
halfcheetah-med-exp	93.5	53.3	63.3	93.7	90.0	94.8	62.4	86.7	35.8
pen-human	72.8	-	-	-	-	53.1	37.5	71.5	34.4
hammer-human	1.9	-	-	-	-	1.5	4.4	1.4	1.5
door-human	6.3	-	-	-	-	2.5	9.9	4.3	0.5
relocate-human	0.4	-	-	-	-	0.1	0.2	0.1	0.0
pen-cloned	51.4	-	-	-	-	43.7	39.2	37.3	56.9
hammer-cloned	0.7	-	-	-	-	1.1	2.1	2.1	0.8
door-cloned	-0.1	-	-	-	-	3.7	0.4	1.6	-0.1
relocate-cloned	-0.0	-	-	-	-	0.2	-0.1	-0.2	-0.1
pen-exp	112.2	-	-	-	-	136.2	107.0	-	85.1
hammer-exp	118.8	-	-	-	-	126.9	86.7	-	125.6
door-exp	98.7	-	-	-	-	99.3	101.5	-	34.9
relocate-exp	96.0	-	-	-	-	99.4	95.0	-	101.3
Table 1: Performance comparison of ARMOR against baselines on the D4RL datasets. The values for ARMOR denote last iteration performance averaged over 4 random seeds, and baseline values were taken from their respective papers. The values denote normalized returns based on random and expert policy returns similar to Fu et al. (2020). Boldface denotes performance within  $10\%$  of the best performing algorithm. We report results with standard deviations in Appendix F.
The dynamics model is pre-trained for 100k steps using model-fitting loss on the offline dataset. ARMOR is then trained for 1M steps on each dataset. Refer to Appendix F for more details.
5.1 Comparison with Offline RL Baselines
By setting the reference policy to the behavior-cloned policy on the offline dataset, we can use ARMOR as a standard offline RL algorithm. Table 1 shows a comparison of the performance of ARMOR against SoTA model-free and model-based offline RL baselines. In the former category, we consider ATAC (Cheng et al., 2022), CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2021), and for the latter we consider MoREL (Kidambi et al., 2020), MOPO (Yu et al., 2020), and RAMBO (Rigter et al., 2022). We also compare against COMBO (Yu et al., 2021) which is a hybrid model-free and model-based algorithm. In these experiments, we initially warm start the optimization for 100k steps, by training the policy and Q-function using behavior cloning and temporal difference learning respectively on the offline dataset to ensure the learner policy is initialized to be the same as the reference. Overall, we observe that ARMOR consistently outperforms or is competitive with the best baseline algorithm on most datasets. Specifically, compared to other purely model-based baselines (MoREL, MOPO and RAMBO), there is a marked increase in performance in the walker2d-med, hopper-med-exp and walker2d-med-exp datasets. We would like to highlight two crucial elements about ARMOR, in contrast to other model-based baselines - (1) ARMOR achieves SoTA performance using only a single neural network to model the MDP, as opposed to complex network ensembles employed in previous model-based offline RL methods (Kidambi et al., 2020; Yu et al., 2021, 2020; Rigter et al., 2022), and (2) to the best of our knowledge, ARMOR is the only purely model-based offline RL algorithm that has shown performance comparable with model-free algorithms on the high-dimensional Adroit environments. The lower performance compared to RAMBO on halfcheetah-med and halfcheetah-med-replay may be attributed to that the much larger computational budget used by RAMBO is required for convergence on these datasets.
5.2 Robust Policy Improvement
Next, we test whether the practical version of ARMOR demonstrates RPI of the theoretical version. We consider a set of 14 datasets comprised of the medium and medium-replay versions of D4RL locomotion tasks, as well as the human and cloned versions of the Adroit tasks, with the reference policy set to be the stochastic behavior cloned policy on the expert dataset. We chose these combi
5The variation in performance of the reference for different dataset qualities in the same environment is owing to different random seeds.
Figure 3: Verification of RPI over the reference policy for different  $\beta$  (purple). ORL denotes the performance of offline RL with ARMOR (Table 1), and REF is the performance of reference policy.
nations of dataset quality and reference, to ensure that the reference policy takes out-of-distribution actions with respect to the data. Unlike Sec. 5.1 here the reference policy is a black-box given as a part of the problem definition. This opens the question of how the learner should be initialized, since we can not trivially initialize the learner to be the reference as in the previous experiments. $^6$  In a similar spirit to Sec. 5.1, one might consider initializing the learner close to the reference by behavior cloning the reference policy on the provided dataset during warmstart, i.e., by replacing the dataset actions with reference actions. However, when the reference chooses out of support actions, this procedure will not provide a good global approximation of the reference policy, which can make the optimization problem harder. Instead, we propose to learn a residual policy where the learned policy outputs an additive correction to the reference (Silver et al., 2018). This is an appropriate choice since ARMOR does not make any restrictive assumptions about the structure of the policy class. Figure 3 shows the normalized return achieved by ARMOR for different  $\beta$ , with fixed values for remaining hyperparameters. We observe that ARMOR is able to achieve performance comparable or better than the reference policy for a range of  $\beta$  values uniformly across all datasets, thus verifying the RPI property in practice. Specifically, there is significant improvement via RPI in the hammer, door and relocate domains, where running ARMOR as a pure offline RL algorithm (Section 5.1) does not show any progress $^7$ . Overall, we note the following metrics:
- In 14/14 datasets, ARMOR shows RPI (i.e., ARMOR policy is no worse than the reference when measured by overlap of confidence intervals). Further, considering the difference between ORL and REF as a rough indication of whether the reference is within data support, we note that in 12/14 cases REF is strictly better than ORL, and in all those cases ARMOR demonstrates RPI.
- In 5/14 datasets, the ARMOR policy is strictly better than the reference. (Criterion: the lower confidence of ARMOR performance is better than upper confidence of REF). It is important to note that this metric is highly dependent on the quality of the reference policy. Since the reference is near-expert, it can be hard for some environments to improve significantly over it.
The RPI of ARMOR is highly valuable as it allows easy tuning of the pessimism hyperparameter without performance degradation. We believe that leveraging this property can pave the way for real-world deployment of offline RL. Thus, we next present a discussion of RPI. $^{8}$
When does RPI actually improve over the reference policy?
Given ARMOR's ability to improve over an arbitrary policy, the following question naturally arises: Can ARMOR nontrivially improve the output policy of other offline algorithms, including itself? If this were true, can we repeatedly run ARMOR to improve over itself and obtain the best policy any algorithm can learn offline? Unfortunately, the answer is negative. Not only can ARMOR not
In Appendix F.5 we provide further experiments for different choices of reference policies.
We provide comparisons when using a behavior cloning initialization for the learner in Appendix F.
Due to space limit, we defer the complete discussion to Appendix D and only provide salient points here.
improve over itself, but it also cannot improve over a variety of algorithms (e.g., absolute pessimism or minimax regret). In fact, the optimal policy of an arbitrary model in the version space  $\mathcal{M}_{\alpha}$  is provably unimprovable ( Corollary 10; Appendix D). With a deep dive into when RPI gives nontrivial improvement (Appendix D), we found some interesting observations, which we highlight here.
Return maximization and regret minimization are different in offline RL. These objectives generally produce different policies, even though they are equivalent in online RL. Their equivalence in online RL relies on the fact that online exploration can eventually resolve any uncertainty. In offline RL with an arbitrary data distribution, there will generally be model uncertainty that cannot be resolved, and the worst-case reasoning over such model uncertainty (i.e.,  $\mathcal{M}_{\alpha}$ ) leads to definitions that are no longer equivalent. Moreover, it is impossible to compare return maximization and regret minimization and make a claim about which is better. They are not simply an algorithm design choice, but are definitions of the learning goals and guarantees themselves—and are thus incomparable: if we care about obtaining a guarantee for the worst-case return, the return maximization is optimal by definition; if we are more interested in a guarantee for the worst-case regret, then regret minimization is optimal. We also note that analyzing algorithms under a metric that is different from the one they are designed for can lead to unusual conclusions, e.g., Xiao et al. (2021) show that optimistic/neutral/pessimistic algorithms are equally minimax-optimal in terms of their regret guarantees in offline multi-armed bandits. However, the algorithms they consider are optimistic/pessimistic with respect to the return (as commonly considered in the offline RL literature) not the regret which is the performance metric they are interested in analyzing.
$\pi_{\mathrm{ref}}$  is more than a hyperparameter—it defines the performance metric and learning goal. Corollary 10 in Appendix D shows that ARMOR has many different fixed points: when  $\pi_{\mathrm{ref}}$  is chosen from these fixed points, the solution to Eq. (1) is also  $\pi_{\mathrm{ref}}$ . Furthermore, some of them may seem quite unreasonable for offline learning (e.g., the greedy policy to an arbitrary model in  $\mathcal{M}_{\alpha}$  or even the optimistic policy). This is not a defect of the algorithm. Rather, because of the unresolvable uncertainty in the offline setting, there are many different performance metrics/learning goals that are generally incompatible/incomparable, and the agent designer must make a conscious choice among them and convey the intention to the algorithm. In ARMOR, such a choice is explicitly conveyed by  $\pi_{\mathrm{ref}}$ , which makes ARMOR subsume return maximization and regret minimization as special cases.
We have presented a model-based offline RL framework, ARMOR, that can improve over arbitrary reference policies regardless of data coverage, by using the concept of relative pessimism. ARMOR provides strong theoretical guarantees with general function approximators, and exhibits robust policy improvement over the reference policy for a wide range of hyper-parameters. We have also presented a scalable deep learning instantiation of the theoretical algorithm. Empirically, we demonstrate that ARMOR indeed enjoys the RPI property, and has competitive performance with several SoTA model-free and model-based offline RL algorithms, while employing a simpler model architecture (a single MDP model) than other model-based baselines that rely on ensembles. This also opens the opportunity to leverage high-capacity world models (Hafner et al., 2023) with offline RL in the future. However, there are also some limitations. While RPI holds for the pessimism parameter, the others still need to be tuned. In practice, the non-convexity of the optimization can also make solving the two-player game challenging. For instance, if the adversary is not strong enough (i.e., far from solving the inner minimization), RPI would break. Further, runtime of ARMOR is slightly slower than model-free algorithms owing to extra computations for model rollouts.
Nan Jiang acknowledges funding support from NSF IIS-2112471 and NSF CAREER IIS-214178.
Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in Neural Information Processing Systems, 33:20095-20107, 2020.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems, 34:6683-6694, 2021a.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems, 34:27395-27407, 2021b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34:28954-28967, 2021.
Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Tong Zhang. From  $\varepsilon$ -entropy to kl-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics, 34(5):2180-2210, 2006.
A Proofs for Section 3
A.1 Technical Tools
Lemma 4 (Simulation lemma). Consider any two MDP model  $M$  and  $M'$ , and any  $\pi : \mathcal{S} \to \Delta(\mathcal{A})$ , we have
$$
| J _ {M} (\pi) - J _ {M ^ {\prime}} (\pi) | \leq \frac {V _ {\max}}{1 - \gamma} \mathbb {E} _ {d ^ {\pi}} \left[ D _ {\mathrm {T V}} (P _ {M} (\cdot | s, a), P _ {M ^ {\prime}} (\cdot | s, a)) \right] + \frac {1}{1 - \gamma} \mathbb {E} _ {d ^ {\pi}} \left[ | R _ {M} (s, a) - R _ {M ^ {\prime}} (s, a) | \right].
$$
Lemma 4 is the standard simulation lemma in model-based reinforcement learning literature, and its proof can be found in, e.g., Uehara and Sun (2021, Lemma 7).
A.2 MLE Guarantees
We use  $\ell_{\mathcal{D}}(M)$  to denote the likelihood of model  $M = (P,R)$  with offline data  $\mathcal{D}$ , where
$$
\ell_ {\mathcal {D}} (M) = \prod_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} P _ {M} \left(s ^ {\prime} \mid s, a\right). \tag {8}
$$