SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mohak Bhardwaj*
mohakb@cs.washington.edu
Tengyang Xie*
Microsoft Research & UW-Madison
tx@cs.wisc.edu
Byron Boots
University of Washington bboots@cs.washington.edu
Nan Jiang
nanjiang@illinois.edu
Ching-An Cheng
Microsoft Research, Redmond
chinganc@microsoft.com
We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage. ARMOR is designed to optimize policies for the worst-case performance relative to the reference policy through adversarially training a Markov decision process model. In theory, we prove that ARMOR, with a well-tuned hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with any admissible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design a scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensembles in contrast to typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and model-based RL algorithms and can robustly improve the reference policy over various hyperparameter choices.
Offline reinforcement learning (RL) is a technique for learning decision-making policies from logged data (Lange et al., 2012; Levine et al., 2020; Jin et al., 2021; Xie et al., 2021a). In comparison with alternate learning techniques, such as off-policy RL and imitation learning (IL), offline RL reduces the data assumption needed to learn good policies and does not require collecting new data. Theoretically, offline RL can learn the best policy that the given data can explain: as long as the offline data includes the scenarios encountered by a near-optimal policy, an offline RL algorithm can learn such a near-optimal policy, even when the data is collected by highly sub-optimal policies and/or is not diverse. Such robustness to data coverage makes offline RL a promising technique for solving real-world problems, as collecting diverse or expert-quality data in practice is often expensive or simply infeasible.
The fundamental principle behind offline RL is the concept of pessimism, which considers worst-case outcomes for scenarios without data. In algorithms, this is realized by (explicitly or implicitly) constructing performance lower bounds in policy learning which penalizes uncertain actions.
2Open source code is available at: https://sites.google.com/view/armoroffliner/.
Figure 1: Robust Policy Improvement: ARMOR can improve performance over the reference policy (REF) over a broad range of pessimism hyperparameter (purple) regardless of data coverage. ORL denotes best offline RL policy without using the reference policy, and reference is obtained by behavior cloning on expert dataset.
Various designs have been proposed to construct such lower bounds, including behavior regularization (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021), point-wise pessimism based on negative bonuses or truncation (Kidambi et al., 2020; Jin et al., 2021), value penalty (Kumar et al., 2020; Yu et al., 2020), or two-player games (Xie et al., 2021a; Uehara and Sun, 2021; Cheng et al., 2022). Conceptually, the tighter the lower bound is, the better the learned policy would perform; see a detailed discussion of related work in Appendix C.
Despite these advances, offline RL still has not been widely adopted to build learning-based decision systems beyond academic research. One important factor we posit is the issue of performance degradation: Usually, the systems we apply RL to have currently running policies, such as an engineered autonomous driving rule or a heuristic-based system for diagnosis, and the goal of applying a learning algorithm is often to further improve upon these baseline reference policies. As a result, it is imperative that the policy learned by the algorithm does not degrade the base performance. This criterion is especially critical for applications where poor decision outcomes cannot be tolerated.
However, running an offline RL algorithm based on pessimism, in general, is not free from performance degradation. While there have been algorithms with policy improvement guarantees (Laroche et al., 2019; Fujimoto et al., 2019; Kumar et al., 2020; Fujimoto and Gu, 2021; Cheng et al., 2022), such guarantees apply only to the behavior policy that collects the data, which might not necessarily be the reference policy. In fact, quite often these two policies are different. For example, in robotic manipulation, it is common to have a dataset of activities different from the target task. In such a scenario, comparing against the behavior policy is meaningless, as these policies do not have meaningful performance in the target task.
In this work, we propose a novel model-based offline RL framework, called Advesarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies that improve upon an arbitrary reference policy by adversarially training a Markov decision process (MDP) model, regardless of the data quality. ARMOR is designed based on the concept of relative pessimism (Cheng et al., 2022), which aims to optimize for the worst-case relative performance over uncertainty. In theory, we prove that, owing to relative pessimism, the ARMOR policy never degrades the performance of the reference policy for a range of hyperparameters which is given beforehand, a property known as Robust Policy Improvement (RPI) (Cheng et al., 2022). In addition, when the right hyperparameter is chosen, and the reference policy is covered by the data, we prove that the ARMOR policy can also compete with any policy covered by the data in an absolute sense. To our knowledge, RPI property of offline RL has so far been limited to comparing against the data collection policy (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022). In ARMOR, by adversarially training an MDP model, we extend the technique of relative pessimism to achieve RPI with arbitrary reference policies, regardless of whether they collected the data or not (Fig. 1).
In addition to theory, we design a scalable deep-learning implementation of ARMOR to validate these claims that jointly trains an MDP model and the state-action value function to minimize the estimated performance difference between the policy and the reference using model-based rollouts. Our implementation achieves state-of-the-art (SoTA) performance on D4RL benchmarks (Fu et al., 2020), while using only a single model (in contrast to ensembles used in existing model-based offline RL works). This makes ARMOR a better framework for using high-capacity world models (e.g. (Hafner et al., 2023)) for which building an ensemble is too expensive. We also empirically validate the RPI property of our implementation.
Markov Decision Process We consider learning in the setup of an infinite-horizon discounted Markov Decision Process (MDP). An MDP  $M$  is defined by the tuple  $\langle S, \mathcal{A}, P_M, R_M, \gamma \rangle$ , where  $S$  is the state space,  $\mathcal{A}$  is the action space,  $P_M: S \times \mathcal{A} \to \Delta(S)$  is the transition dynamics,  $R_M: S \times \mathcal{A} \to [0,1]$  is a scalar reward function and  $\gamma \in [0,1)$  is the discount factor. A policy  $\pi$  is a mapping from  $S$  to a distribution on  $\mathcal{A}$ . For  $\pi$ , we let  $d_M^\pi(s, a)$  denote the discounted state-action distribution obtained by running  $\pi$  on  $M$  from an initial state distribution  $d_0$ , i.e.  $d_M^\pi(s, a) = (1 - \gamma) \mathbb{E}_{\pi, M}[\sum_{t=0}^{\infty} \gamma^t \mathbb{1}(s_t = s, a_t = a)]$ . Let  $J_M(\pi) = \mathbb{E}_{\pi, M}[\sum_{t=0}^{\infty} \gamma^t r_t]$  be the expected discounted return of policy  $\pi$  on  $M$  starting from  $d_0$ , where  $r_t = R_M(s_t, a_t)$ . We define the value function as  $V_M^\pi(s) = \mathbb{E}_{\pi, M}[\sum_{t=0}^{\infty} \gamma^t r_t | s_0 = s]$ , and the state-action value function (i.e., Q-function) as  $Q_M^\pi(s, a) = \mathbb{E}_{\pi, M}[\sum_{t=0}^{\infty} \gamma^t r_t | s_0 = s, s_0 = a]$ . By this definition, we note  $J_M(\pi) = \mathbb{E}_{d_0}[V_M^\pi(s)] = \mathbb{E}_{d_0, \pi}[Q_M^\pi(s, a)]$ . We use  $[0, V_{\max}]$  to denote the range of value functions, where  $V_{\max} \geq 1$ . We denote the ground truth MDP as  $M^\star$ , and  $J = J_M^\star$
Offline RL The aim of offline RL is to find the policy that maximizes  $J(\pi)$ , while using a fixed dataset  $\mathcal{D}$  collected by a behavior policy  $\mu$ . We assume the dataset  $\mathcal{D}$  consists of  $\{(s_n, a_n, r_n, s_{n+1})\}_{n=1}^N$ , where  $(s_n, a_n)$  is sampled from  $d_{M^{\star}}^{\mu}$  and  $r_n, s_{n+1}$  follow  $M^{\star}$ ; for simplicity, we also write  $\mu(s, a) = d_{M^{\star}}^{\mu}(s, a)$ .
We assume that the learner has access to a Markovian policy class  $\Pi$  and an MDP model class  $\mathcal{M}$ .
Assumption 1 (Realizability). We assume the ground truth model  $M^{\star}$  is in the model class  $\mathcal{M}$ .
In addition, we assume that we are provided a reference policy  $\pi_{\mathrm{ref}}$ . In practice, such a reference policy represents a baseline whose performance we want to improve with offline RL and data.
Assumption 2 (Reference policy). We assume access to a reference policy  $\pi_{\mathrm{ref}}$ , which can be queried at any state. We assume  $\pi_{\mathrm{ref}}$  is realizable, i.e.,  $\pi_{\mathrm{ref}} \in \Pi$ .
If  $\pi_{\mathrm{ref}}$  is not provided, we can still run ARMOR as a typical offline RL algorithm, by first performing behavior cloning on the data and setting the cloned policy as  $\pi_{\mathrm{ref}}$ . In this case, ARMOR has RPI with respect to the behavior policy.
Robust Policy Improvement RPI is a notion introduced in Cheng et al. (2022), which means that the offline algorithm can learn to improve over the behavior policy, using hyperparameters within a known set. Algorithms with RPI are more robust to hyperparameter choices, and they are often derived from the principle of relative pessimism (Cheng et al., 2022). In this work, we extend the RPI concept to compare with an arbitrary reference (or baseline) policy, which can be different from the behavior policy and can take actions outside data support.
3 Adversarial Model for Offline Reinforcement Learning (ARMOR)
ARMOR is a model-based offline RL algorithm designed with relative pessimism. The goal of ARMOR is to find a policy  $\widehat{\pi}$  that maximizes the performance difference  $J(\widehat{\pi}) - J(\pi_{\mathrm{ref}})$  to a given reference policy  $\pi_{\mathrm{ref}}$ , while accounting for the uncertainty due to limited data coverage. ARMOR achieves this by solving a two-player game between a learner policy and an adversary MDP model:
$$
\widehat {\pi} = \underset {\pi \in \Pi} {\operatorname {a r g m a x}} \min  _ {M \in \mathcal {M} _ {\alpha}} J _ {M} (\pi) - J _ {M} \left(\pi_ {\text {r e f}}\right) \tag {1}
$$
based on a version space of MDP models
$$
\mathcal {M} _ {\alpha} = \left\{M \in \mathcal {M}: \mathcal {E} _ {\mathcal {D}} (M) - \min  _ {M ^ {\prime} \in \mathcal {M}} \mathcal {E} _ {\mathcal {D}} \left(M ^ {\prime}\right) \leq \alpha \right\}, \tag {2}
$$
where we define the model fitting loss as
$$
\mathcal {E} _ {\mathcal {D}} (M) := - \sum_ {\mathcal {D}} \log P _ {M} \left(s ^ {\prime} \mid s, a\right) + \left(R _ {M} (s, a) - r\right) ^ {2} / V _ {\max } ^ {2} \tag {3}
$$
and  $\alpha \geq 0$  is a bound on statistical errors such that  $M^{\star} \in \mathcal{M}_{\alpha}$ . In this two-player game, ARMOR is optimizing a lower bound of the relative performance  $J(\pi) - J(\pi_{\mathrm{ref}})$ . This is due to the construction that  $M^{\star} \in \mathcal{M}_{\alpha}$ , which ensures  $\min_{M \in \mathcal{M}_{\alpha}} J_M(\pi) - J_M(\pi_{\mathrm{ref}}) \leq J_{M^{\star}}(\pi) - J_{M^{\star}}(\pi_{\mathrm{ref}})$ .
One interesting property that follows from optimizing the relative performance lower bound is that  $\widehat{\pi}$  is guaranteed to always be no worse than  $\pi_{\mathrm{ref}}$ , for a wide range of  $\alpha$  and regardless of the relationship between  $\pi_{\mathrm{ref}}$  and the data  $\mathcal{D}$ .
Figure 2: A toy MDP illustrating the RPI property of ARMOR. (Top) The true MDP has deterministic dynamics where taking the left  $(a_{l})$  or right  $(a_{r})$  actions takes the agent to corresponding states; start state is in yellow. The suboptimal behavior policy visits only the left part of the state space, and the reference policy demonstrates optimal behavior by always choosing  $a_{r}$ . (Bottom) A subset of possible data-consistent MDP models in the version space. The adversary always chooses the MDP that makes the reference maximally outperform the learner. In response, the learner will learn to mimic the reference outside data support to be competitive.
Proposition 1. For any  $\alpha$  large enough such that  $M^{\star} \in \mathcal{M}_{\alpha}$ , it holds that  $J(\widehat{\pi}) \geq J(\pi_{\mathrm{ref}})$ .
This fact can be easily reasoned: Since  $\pi_{\mathrm{ref}} \in \Pi$ , we have  $\max_{\pi \in \Pi} \min_{M \in \mathcal{M}_{\alpha}} J_M(\pi) - J_M(\pi_{\mathrm{ref}}) \geq \min_{M \in \mathcal{M}_{\alpha}} J_M(\pi_{\mathrm{ref}}) - J_M(\pi_{\mathrm{ref}}) = 0$ . In other words, ARMOR achieves the RPI property with respect to any reference policy  $\pi_{\mathrm{ref}}$  and offline dataset  $\mathcal{D}$ .
This RPI property of ARMOR is stronger than the RPI property in the literature. In comparison, previous algorithms with RPI (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022) are only guaranteed to be no worse than the behavior policy that collected the data. In Section 3.2, we will also show that when  $\alpha$  is set appropriately, ARMOR can provably compete with the best data covered policy as well, as prior offline RL works (e.g., Xie et al., 2021a; Uehara and Sun, 2021; Cheng et al., 2022).
3.1 An Illustrative Toy Example
Why does ARMOR have the RPI property, even when the reference policy  $\pi_{\mathrm{ref}}$  is not covered by the data  $\mathcal{D}$ ? While we will give a formal analysis soon in Section 3.2, here we provide some intuitions as to why this is possible. First, notice that ARMOR has access to the reference policy  $\pi_{\mathrm{ref}}$ . Therefore, a trivial way to achieve RPI with respect to  $\pi_{\mathrm{ref}}$  is to just output  $\pi_{\mathrm{ref}}$ . However, this naive algorithm while never degrading  $\pi_{\mathrm{ref}}$  cannot learn to improve from  $\pi_{\mathrm{ref}}$ . ARMOR achieves these two features simultaneously by 1) learning an MDP Model, and 2) adversarially training this MDP model to minimize the relative performance difference to  $\pi_{\mathrm{ref}}$  during policy optimization.
We illustrate this by a one-dimensional discrete MDP example with five possible states as shown in Figure 2. The dynamic is deterministic, and the agent always starts in the center cell. The agent receives a lower reward of 0.1 in the left-most state and a high reward of 1.0 upon visiting the right-most state. Say, the agent only has access to a dataset from a sub-optimal policy that always takes the left action to receive the 0.1 reward. Further, let's say we have access to a reference policy that demonstrates optimal behavior on the true MDP by always visiting the right-most state. However, it is unknown a priori that the reference policy is optimal. In such a case, typical offline RL methods can only recover the sub-optimal policy from the dataset as it is the best-covered policy in the data.
ARMOR can learn to recover the expert reference policy in this example by performing rollouts with the adversarially trained MDP model. From the realizability assumption (Assumption 1), we know that the version space of models contains the true model (i.e.,  $M^{\star} \in \mathcal{M}_{\alpha}$ ). The adversary can then choose a model from this version space where the reference policy  $\pi_{\mathrm{ref}}$  maximally outperforms the learner. In this toy example, the model selected by the adversary would be the one allowing the expert policy to reach the right-most state. Now, optimizing relative performance difference with respect to this model will ensure that the learner can recover the expert behavior, since the only way for the learner to stay competitive with the reference policy is to mimic the reference policy in the region outside data support. In other words, the reason why ARMOR has RPI to  $\pi_{\mathrm{ref}}$  is that
its adversarial model training procedure can augment the original offline data with new states and actions that would cover those generated by running the reference policy.
3.2 Theoretical Analysis
Now we make the above discussions formal and give theoretical guarantees on ARMOR's absolute performance and RPI property. To this end, we introduce a single-policy concentrability coefficient, which measures the distribution shift between a policy  $\pi$  and the data distribution  $\mu$ .
Definition 1 (Generalized Single-policy Concentrability). We define the generalized single-policy concentrability for policy  $\pi$ , model class  $\mathcal{M}$  and offline data distribution  $\mu$  as  $\mathfrak{C}_{\mathcal{M}}(\pi) := \sup_{M \in \mathcal{M}} \frac{\mathbb{E}_{d^{\pi}}[\mathcal{E}^{\star}(M)]}{\mathbb{E}_{\mu}[\mathcal{E}^{\star}(M)]}$ , where  $\mathcal{E}^{\star}(M) = D_{\mathrm{TV}}(P_M(\cdot | s, a), P_{M^{\star}}(\cdot | s, a))^{2} + (R_M(s, a) - R^{\star}(s, a))^{2} / V_{\max}^{2}$ .
Note that  $\mathfrak{C}_{\mathcal{M}}(\pi)$  is always upper bounded by the standard single-policy concentrability coefficient  $\|d^{\pi} / \mu\|_{\infty}$  (e.g., Jin et al., 2021; Rashidinejad et al., 2021; Xie et al., 2021b), but it can be smaller in general with model class  $\mathcal{M}$ . It can also be viewed as a model-based analog of the one in Xie et al. (2021a). A detailed discussion around  $\mathfrak{C}_{\mathcal{M}}(\pi)$  can be found in Uehara and Sun (2021).
First, we present the absolute performance guarantee of ARMOR, which holds for a well-tuned  $\alpha$ .
Theorem 2 (Absolute performance). Under Assumption 1, there is an absolute constant  $c$  such that for any  $\delta \in (0,1]$ , if we set  $\alpha = c \cdot (\log(|\mathcal{M}| / \delta))$  in Eq. (2), then for any reference policy  $\pi_{\mathrm{ref}}$  and comparator policy  $\pi^{\dagger} \in \Pi$ , with probability  $1 - \delta$ , the policy  $\widehat{\pi}$  learned by ARMOR in Eq. (1) satisfies that  $J(\pi^{\dagger}) - J(\widehat{\pi})$  is upper bounded by
$$
\mathcal {O} \left(\left(\sqrt {\mathfrak {C} _ {\mathcal {M}} (\pi^ {\dagger})} + \sqrt {\mathfrak {C} _ {\mathcal {M}} (\pi_ {\mathrm {r e f}})}\right) \frac {V _ {\max}}{1 - \gamma} \sqrt {\frac {\log (| \mathcal {M} | / \delta)}{n}}\right).
$$
Roughly speaking, Theorem 2 shows that  $\widehat{\pi}$  learned by ARMOR can compete with any policy  $\pi^{\dagger}$  with a large enough dataset, as long as the offline data  $\mu$  has good coverage on  $\pi^{\dagger}$  (good coverage over  $\pi_{\mathrm{ref}}$  can be automatically satisfied if we simply choose  $\pi_{\mathrm{ref}} = \mu$ , which yields  $\mathfrak{C}_{\mathcal{M}}(\pi_{\mathrm{ref}}) = 1$ ). Compared to the closest model-based offline RL work (Uehara and Sun, 2021), if we set  $\pi_{\mathrm{ref}} = \mu$  (data collection policy), Theorem 2 leads to almost the same guarantee as Uehara and Sun (2021, Theorem 1) up to constant factors.
In addition to absolute performance, below we show that, under Assumptions 1 and 2, ARMOR has the RPI property to  $\pi_{\mathrm{ref}}$ : it always improves over  $J(\pi_{\mathrm{ref}})$  for a wide range of parameter  $\alpha$ . Compared with the model-free ATAC algorithm in Cheng et al. (2022, Proposition 6), the threshold for  $\alpha$  in Theorem 3 does not depend on sample size  $N$  due to the model-based nature of ARMOR.
Theorem 3 (Robust strong policy improvement). Under Assumptions 1 and 2, there exists an absolute constant  $c$  such that for any  $\delta \in (0,1]$ , if: i)  $\alpha \geq c \cdot (\log(|\mathcal{M}| / \delta))$  in Eq. (2); ii)  $\pi_{\mathrm{ref}} \in \Pi$ , then with probability  $1 - \delta$ , the policy  $\widehat{\pi}$  learned by ARMOR in Eq. (1) satisfies  $J(\widehat{\pi}) \geq J(\pi_{\mathrm{ref}})$ .
The detailed proofs of Theorems 2 and 3, as well as the discussion on how to relax Assumptions 1 and 2 to the misspecified model and policy classes are deferred to Appendix A.
4 Practical Implementation
In this section, we present a scalable implementation of ARMOR (Algorithm 1) that approximately solves the two-player game in Eq. (1). We first describe the overall design principle and then the algorithmic details.
4.1 A Model-based Actor Critic Approach
For computational efficiency, we take a model-based actor critic approach and solve a regularized version of Eq. (1). We construct this regularized version by relaxing the constraint  $M \in \mathcal{M}_{\alpha}$
<sup>3</sup>Note that ARMOR does not depend on knowledge of the true reward function and similar arguments hold in the case of learned rewards as we illustrate in Appendix E.
Algorithm 1 ARMOR (Adversarial Model for Offline Reinforcement Learning)
Input: Batch data  $\mathcal{D}_{\mathrm{real}}$ , policy  $\pi$ , MDP model  $M$ , critics  $f_{1}, f_{2}$ , horizon  $H$ , constants  $\beta, \lambda \geq 0$ ,  $\tau \in [0,1]$ ,  $w \in [0,1]$
1: Initialize target networks  $\bar{f}_1\gets f_1,\bar{f}_2\gets f_2$  and  $\mathcal{D}_{\mathrm{model}} = \emptyset$
2: for  $k = 0, \dots, K - 1$  do
3: Sample minibatch  $\mathcal{D}_{\mathrm{real}}^{\mathrm{mini}}$  from dataset  $\mathcal{D}_{\mathrm{real}}$  and minibatch  $\mathcal{D}_{\mathrm{model}}^{\mathrm{mini}}$  from dataset  $\mathcal{D}_{\mathrm{model}}$ .
4: Construct transition tuples using model predictions
$$
\mathcal {D} _ {M} := \left\{\left(s, a, r _ {M}, s _ {M} ^ {\prime}\right): r _ {M} = R _ {M} (s, a), s _ {M} ^ {\prime} \sim P _ {M} (\cdot \mid s, a), (s, a) \in \mathcal {D} _ {\text {r e a l}} ^ {\text {m i n i}} \cup \mathcal {D} _ {\text {m o d e l}} ^ {\text {m i n i}} \right\}
$$
5: Update the adversary networks; for  $i = 1,2$
$$
l ^ {\text {a d v e r s a r y}} (f, M) := \mathcal {L} _ {\mathcal {D} _ {M}} (f, \pi , \pi_ {\text {r e f}}) + \beta \left(\mathcal {E} _ {\mathcal {D} _ {M}} ^ {w} (f, M, \pi) + \lambda \mathcal {E} _ {\mathcal {D} _ {\text {r e a l}} ^ {\min }} (M)\right) \tag {4}
$$
$$
M \leftarrow M - \eta_ {\text {f a s t}} \left(\nabla_ {M} l ^ {\text {a d v e r s a r y}} \left(f _ {1}, M\right) + \nabla_ {M} l ^ {\text {a d v e r s a r y}} \left(f _ {2}, M\right)\right)
$$
$$
f _ {i} \leftarrow \operatorname {P r o j} _ {\mathcal {F}} \left(f _ {i} - \eta_ {\text {f a s t}} \nabla_ {f _ {i}} l ^ {\text {a d v e r s a r y}} \left(f _ {i}, M\right)\right) \quad \text {a n d} \quad \bar {f} _ {i} \leftarrow (1 - \tau) \bar {f} _ {i} + \tau f _ {i}
$$
6: Update actor network with respect to the first critic network and the reference policy
$$
l ^ {\text {a c t o r}} (\pi) := - \mathcal {L} _ {\mathcal {D} _ {M}} \left(f _ {1}, \pi , \pi_ {\text {r e f}}\right) \tag {5}
$$
$$
\pi \leftarrow \operatorname {P r o j} _ {\Pi} \left(\pi - \eta_ {\text {s l o w}} \nabla_ {\pi} l ^ {\text {a c t o r}} (\pi)\right)
$$
7: If  $k\% H = 0$ , then reset model state:  $\bar{S}_{\pi} \gets \{s \in \mathcal{D}_{\mathrm{real}}^{\mathrm{mini}}\}$  and  $\bar{S}_{\pi_{\mathrm{ref}}} \gets \{s \in \mathcal{D}_{\mathrm{real}}^{\mathrm{mini}}\}$ .
8: Query the MDP model to expand  $\mathcal{D}_{\mathrm{model}}$  and update model state