SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hai Zhang Hang Yu Junqiao Zhao* Di Zhang
Chang Huang Hongtu Zhou Xiao Zhang Chen Ye
Department of Computer Science, Tongji University, Shanghai, China
MOE Key Lab of Embedded System and Service Computing, Tongji University, Shanghai, China
{zhanghai12138，2053881，zhaojunqiao}@tongji.edu.cn
Designing and deriving effective model-based reinforcement learning (MBRL) algorithms with a performance improvement guarantee is challenging, mainly attributed to the high coupling between model learning and policy optimization. Many prior methods that rely on return discrepancy to guide model learning ignore the impacts of model shift, which can lead to performance deterioration due to excessive model updates. Other methods use performance difference bound to explicitly consider model shift. However, these methods rely on a fixed threshold to constrain model shift, resulting in a heavy dependence on the threshold and a lack of adaptability during the training process. In this paper, we theoretically derive an optimization objective that can unify model shift and model bias and then formulate a fine-tuning process. This process adaptively adjusts the model updates to get a performance improvement guarantee while avoiding model overfitting. Based on these, we develop a straightforward algorithm USB-PO $^2$  (Unified model Shift and model Bias Policy Optimization). Empirical results show that USB-PO achieves state-of-the-art performance on several challenging benchmark tasks.
Nowadays, reinforcement learning (RL) has been gaining much traction in a wide variety of complicated decision-making tasks ranging from academia to industry [42, 9, 5, 13, 28]. Part of this is due to some remarkable model-free RL (MFRL) algorithms [40, 15, 11, 41, 18], which show desirable asymptotic performance. However, their applications are hindered by the bottleneck of sample efficiency. On the contrary, model-based RL (MBRL) algorithms, using a world model to generate the imaginary rollouts and then taking them for policy optimization [31, 20], have high sample efficiency while achieving similar asymptotic performance, thus becoming a compelling alternative in practical cases [39, 16, 52].
Typically, MBRL algorithms iterate between model learning and policy optimization. Hence the model quality is crucial for MBRL. Many prior methods [31, 20, 53, 38, 24] rely on return discrepancy to obtain model updates with a performance improvement guarantee. While having achieved comparable results, they only account for model bias in one iteration [20] but do not consider the impacts of model shift between two iterations [21], which can lead to performance deterioration due to excessive model updates. Although CMLO [21] explicitly considers model shift from the perspective of the performance difference bound, it only sets a fixed threshold to constrain the impacts of model shift and determines when the model should be updated accordingly. If this threshold is
$^{2}$ Code: https://github.com/betray12138/Unified-Model-Shift-and-Model-Bias-Policy-Optimization.git
set too low, the model bias of the following iteration will be large, which impairs the subsequent optimization process. If this threshold is set too high, the performance improvement can no longer be guaranteed. We remark that such an update is heavily dependent on the choice of this threshold and should be adjusted adaptively during the training process. Therefore, a smarter scheme is required to unify model shift and model bias, enabling adaptively adjusting their impacts to get a performance improvement guarantee.
In this paper, we theoretically derive an optimization objective that can unify model shift and model bias. Specifically, according to the performance difference bound, we propose to minimize the sum of two terms: the model shift term between the pre-update model and the post-update model and the model bias term of the post-update model, each of which is denoted by a second-order Wasserstein distance [47]. By minimizing this optimization objective after the model update via maximum likelihood estimation (MLE) [6, 20], we can tune the model to adaptively find appropriate updates to get a performance improvement guarantee.
Based on these, we develop a straightforward algorithm USB-PO (Unified model Shift and model Bias Policy Optimization). To the best of our knowledge, this is the first method that unifies model shift and model bias and adaptively fine-tunes the model updates during the training process. We evaluate USB-PO on several continuous control benchmark tasks. The results show that USB-PO has higher sample efficiency and better final performance than other state-of-the-art (SOTA) MBRL methods and yields promising asymptotic performance compared with the MFRL counterparts.
MBRL algorithms are promising candidates for real-world sequential decision-making problems due to their high sample efficiency. Existing studies can be divided into several categories [4, 45, 1, 34, 49, 8, 17] following their different usage of the model. Our work falls into the Dyna-style category [45, 44]. Specifically, after model learning, the model generates the imaginary rollouts into the replay buffer for subsequent policy optimization. Hence, both model learning and policy optimization have critical impacts on asymptotic performance.
Some previous algorithms focus on the policy optimization process. CMBAC [51] introduces conservatism to reduce overestimation of the action value function, and ME-TRPO [23] imposes constraints on the policy to get reliable updates within the trust region. Our work is oriented towards model learning, which is orthogonal to these algorithms. Hence, we are inclined to propose a generic algorithm similar to [24, 20, 21] that can be plugged into many SOTA MFRL algorithms [15, 29], rather than just proposing for a specific policy optimization algorithm.
A key issue in model learning is model bias, which refers to the error between the model and the real environment [20]. As the imaginary rollout horizon increases, the impacts of model bias accumulate rapidly, leading to compounding error and unreliable transitions. To mitigate this problem, several effective methods have been proposed. The ensemble model technique [6, 25, 37] and the dropout method [12] are employed to prevent model overfitting. The uncertainty estimation techniques are used to adjust the rollout length [20, 30] or the transition weight [19, 38]. Furthermore, The multi-step techniques [2, 48] are applied to prevent direct input of the imaginary states. We follow the previous work [20] to use the combination of the ensemble model technique with short model rollouts to mitigate the compounding error.
Performance improvement guarantee is a core concern in both MFRL and MBRL theoretic avenues. In MFRL, methods such as TRPO [41] and CPI [22] choose to optimize the performance difference bound, whilst most of the previous work in MBRL [31, 20, 53, 38, 24] choose to optimize the difference of expected return under the model and that of the real environment, which is termed return discrepancy. However, return discrepancy ignores model shift between two consecutive iterations compared to the performance difference bound under the MBRL setting, which can lead to performance deterioration due to excessive model updates. Although some recent methods have also employed performance difference bound to construct theoretical proofs, they still suffer from certain limitations. OPC [10] designs an algorithm to optimize on-policy model error, but it is similar to return discrepancy in nature. DPI [43] uses dual updates to improve sample efficiency but tries to restrict policy updates within the trust region, thus inhibiting exploration. CMLO [21] relies on a fixed threshold to constrain the impacts of model shift, resulting in a heavy dependence on the threshold and a lack of adaptability during the training process. Hence, we try to unify model shift
and model bias to form a novel optimization problem, adaptively fine-tuning the model updates to get a performance improvement guarantee. Still, some prior work [7, 36, 54] choose to consider regret bound, among which [54] also reduce the impacts of the model changing dramatically between successive iterations. Instead of unifying model shift and model bias, they choose to realize dual optimization by considering maximizing the expectation of the model value rather than that of the single model as a sub-process. Different from [54, 43, 27] that use dual optimization to train the policy, we devise an extra phase to fine-tune the model.
We consider a Markov Decision Process (MDP), defined by the tuple  $M = (\mathcal{S}, \mathcal{A}, p, r, \gamma, \rho_0)$ .  $\mathcal{S}$  and  $\mathcal{A}$  denote the state space and action space respectively, and  $\gamma \in (0,1)$  denotes the discount factor.  $p(s'|s, a)$  denotes the dynamic transition distribution and we denote  $p_{M^*}(s'|s, a)$  as that of the real environment.  $\rho_0(s)$  denotes the initial state distribution and  $r(s, a)$  denotes the reward function. RL aims to find the optimal policy  $\pi^*$  that maximizes the expected return under the real environment  $M^*$  denoted by the value function  $V_{M^*}^\pi$  as Eq.(1):
$$
\pi^ {*} = \arg \max  _ {\pi} V _ {M ^ {*}} ^ {\pi} = \mathbb {E} _ {a _ {t} \sim \pi (\cdot | s _ {t}), s _ {t + 1} \sim p _ {M ^ {*}} (\cdot | s _ {t}, a _ {t})} [ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, a _ {t}) | \pi , s _ {0} ], s _ {0} \sim \rho_ {0} (s) \tag {1}
$$
MBRL algorithms aim to learn the dynamic transition distribution model,  $p_{M}(s^{\prime}|s,a)$ , by using samples collected from interaction with the real environment via supervised learning. We denote the expected return under the model  $M$  of the policy  $\pi$  as  $V_{M}^{\pi}$  and denote that under the real environment of the policy  $\pi$  derived from the model  $M$  as  $V^{\pi|M}$ . Additionally, we assume that  $r(s,a)$  is unknown to the model  $M$  and the model will predict  $r_{M}$  as the reward function. Besides, we denote  $\mathcal{M}$  as a parameterized family of models and  $\Pi$  as a parameterized family of policies.
Let  $d_M^\pi(s, a)$  denote the normalized discounted visitation probability for  $(s, a)$  when starting at  $s_0 \sim \rho_0$  and following  $\pi$  under the model  $M$ . Let  $p_{M,t}^\pi(s)$  denote the probability of visiting  $s$  at timestep  $t$  given the policy  $\pi$  and the model  $M$ .
$$
d _ {M} ^ {\pi} (s, a) = (1 - \gamma) \sum_ {t = 0} ^ {\infty} \gamma^ {t} p _ {M, t} ^ {\pi} (s) \pi (a | s) \tag {2}
$$
We define the total variation distance (TVD) estimator as  $D_{TV}(\cdot ||\cdot)$  and the second-order Wasserstein distance estimator as  $W_{2}(\cdot ,\cdot)$ .
4 USB-PO framework
In this section, we demonstrate a detailed description of our proposed algorithmic framework. i.e., USB-PO. In Section 4.1, a meta-algorithm of the USB-PO framework is provided as a generic solution. In Section 4.2, we theoretically show how to unify model shift and model bias to get a performance improvement guarantee<sup>3</sup>. In Section 4.3, the practical algorithm is proposed to instantiate the USB-PO framework.
4.1 The overall algorithm
The general algorithmic framework of USB-PO is depicted in Algorithm 1, where the main difference compared to the existing MBRL algorithms is the two-phase model learning process, namely phase 1 and phase 2. Phase 1 uses traditional MLE loss to train the model, which may impair the performance by excessive model updates due to only considering the impacts of model bias. To mitigate this problem, we introduce phase 2 to further fine-tune the model updates, whose optimization objective is defined as Eq.(3).
$$
\underset {p _ {M _ {2}}} {\arg \min } \mathcal {J} _ {p h a s e 2} = \mathbb {E} _ {d _ {M _ {1}} ^ {\tau}} \left[ W _ {2} \left(p _ {M _ {1}}, p _ {M _ {2}}\right) + W _ {2} \left(p _ {M _ {2}}, p _ {M ^ {*}}\right) \right] \tag {3}
$$
<sup>3</sup>The detailed derivations of all the theorems in this section are presented in the appendix.
Algorithm 1 Meta-Algorithm of the USB-PO Framework
1: Initialize the policy  $\pi$  and the learned model;
2: Initialize the environment replay buffer  $\mathcal{D}$  and the model replay buffer  $\mathcal{D}_M$ ;
3: for each epoch do
4: Use  $\pi$  to interact with the real environment:  $\mathcal{D}\gets \mathcal{D}\cup \{(s,a,r,s')\}$
5: Backup the current learned model for future use and denote this backed-up model as  $p_{M_1}$ ;
6: Phase 1: use  $\mathcal{D}$  to train the learned model with the supervision of MLE and denote this updated model as  $p_{M_2}$ ;
7: Phase 2: use Eq.(3) as optimization objective to further fine-tune  $p_{M_2}$ ;
8: Use  $p_{M_2}$  to generate the imaginary rollouts:  $\mathcal{D}_M \gets \mathcal{D}_M \cup \{(s_M, a_M, r_M, s_M')\}$ ;
9: Use  $\mathcal{D} \cup \mathcal{D}_M$  to train the policy  $\pi$
Eq.(3) unifies the model shift term and the model bias term in the second-order Wasserstein distance form, namely  $W_{2}(p_{M_{1}}, p_{M_{2}})$  and  $W_{2}(p_{M_{2}}, p_{M^{*}})$ , thus achieving adaptive adjustment of their impacts during the fine-tuning process. As demonstrated in Section 4.2 and Section 5.4, this is not equivalent to the traditional methods of limiting the magnitude of model updates, but rather beneficial to get a performance improvement guarantee.
4.2 Theoretical proof
Definition 1 (Performance Difference Bound). Recalling that  $V^{\pi_i|M_i}$  denotes the expected return under the real environment of the policy  $\pi_i \in \Pi$  derived from the model  $M_i \in \mathcal{M}$ . The lower bound on the true return gap of  $\pi_1$  and  $\pi_2$  can be stated as,
$$
V ^ {\pi_ {2} \mid M _ {2}} - V ^ {\pi_ {1} \mid M _ {1}} \geq C \tag {4}
$$
By constantly increasing the value of  $C$ , the lower bound on the performance difference is guaranteed to be lifted, leading to performance improvement. Therefore, we try to take model shift and model bias into the formulation of  $C$  and maximize it to achieve our goal.
Theorem 1 (Performance Difference Bound Decomposition). Let  $M_{i} \in \mathcal{M}$  be the evaluated model and  $\pi_{i} \in \Pi$  be the policy derived from the model. The performance difference bound can be decomposed into three terms,
$$
V ^ {\pi_ {2} | M _ {2}} - V ^ {\pi_ {1} | M _ {1}} = \left(V ^ {\pi_ {2} | M _ {2}} - V _ {M _ {2}} ^ {\pi_ {2}}\right) - \left(V ^ {\pi_ {1} | M _ {1}} - V _ {M _ {1}} ^ {\pi_ {1}}\right) + \left(V _ {M _ {2}} ^ {\pi_ {2}} - V _ {M _ {1}} ^ {\pi_ {1}}\right) \tag {5}
$$
Obviously, compared to directly optimizing the return discrepancy of each iteration [20], the performance difference bound chooses to optimize the return discrepancy of two adjacent iterations, namely  $V^{\pi_2|M_2} - V_{M_2}^{\pi_2}$  and  $V^{\pi_1|M_1} - V_{M_1}^{\pi_1}$  respectively, and the expected return variation between these two iterations, namely  $V_{M_2}^{\pi_2} - V_{M_1}^{\pi_1}$ , demonstrating better rigorousness. For further discussion, we introduce the following theorem.
Theorem 2 (Return Bound). Let  $R_{\text{max}}$  denote the bound of the reward function,  $\epsilon_{\pi}$  denote  $\max_s D_{TV}(\pi_1||\pi_2)$  and  $\epsilon_{M_1}^{M_2}$  denote  $\mathbb{E}_{(s,a)\sim d_{M_1}^{\pi_1}}[D_{TV}(p_{M_1}||p_{M_2})]$ . For two arbitrary policies  $\pi_1,\pi_2\in \Pi$ , the expected return under two arbitrary models  $M_1,M_2\in \mathcal{M}$  can be bounded as,
$$
V _ {M _ {2}} ^ {\pi_ {2}} - V _ {M _ {1}} ^ {\pi_ {1}} \geq - 2 R _ {m a x} \left(\frac {\epsilon_ {\pi}}{(1 - \gamma) ^ {2}} + \frac {\gamma}{(1 - \gamma) ^ {2}} \epsilon_ {M _ {1}} ^ {M _ {2}}\right) \tag {6}
$$
By using Eq.(6), we can easily bound the decomposition terms of the performance difference bound in Eq.(5).
Theorem 3 (Decomposition TVD Bound). Let  $\epsilon_{M_i}^{\pi_i}$  denote  $\mathbb{E}_{(s,a)\sim d_{M_i}^{\pi_i}}[D_{TV}(p_{M_i}||p_{M^*})]$ . Let  $M_{i}\in \mathcal{M}$  be the evaluated model and  $\pi_i\in \Pi$  be the policy derived from the model. The decomposition terms can be bounded as,
$$
V ^ {\pi_ {2} \mid M _ {2}} - V ^ {\pi_ {1} \mid M _ {1}} \geq \frac {2 R _ {\text {m a x}} \gamma}{(1 - \gamma) ^ {2}} \left(\epsilon_ {M _ {1}} ^ {\pi_ {1}} - \epsilon_ {M _ {2}} ^ {\pi_ {2}} - \epsilon_ {M _ {1}} ^ {M _ {2}}\right) - \frac {2 R _ {\text {m a x}} \epsilon_ {\pi}}{(1 - \gamma) ^ {2}} \tag {7}
$$
Notably, the model shift term and the model bias term in TVD form, namely  $D_{TV}(p_{M_1}||p_{M_2})$  and  $D_{TV}(p_{M_2}||p_{M^*})$ , are already present in the right-hand side of the Eq.(7). However, due to the unknown term  $\pi_2$ , we can not sample from  $d_{M_2}^{\pi_2}$ . Thus, we need to make a further transformation about the Eq.(7).
Theorem 4 (Unified Model Shift and Model Bias Bound). Let  $\kappa$  denote the constant  $\frac{2R_{max}}{(1 - \gamma)^2}$  and  $\Delta$  denotes  $\mathbb{E}_{(s,a)\sim d_{M_1}^{\pi_1}}[D_{TV}(p_{M_2}||p_{M^*})] - \mathbb{E}_{(s,a)\sim d_{M_2}^{\pi_2}}[D_{TV}(p_{M_2}||p_{M^*})]$ . Let  $M_i\in \mathcal{M}$  be the evaluated model and  $\pi_i\in \Pi$  be the policy derived from the model. The unified model shift and model bias bound can be derived as,
$$
\begin{array}{l} V ^ {\pi_ {2} | M _ {2}} - V ^ {\pi_ {1} | M _ {1}} \\ \geq \kappa (\gamma \left(\mathbb {E} _ {(s, a) \sim d _ {M _ {1}} ^ {\pi_ {1}}} [ D _ {T V} \left(p _ {M _ {1}} \mid \mid p _ {M ^ {*}}\right) - D _ {T V} \left(p _ {M _ {1}} \mid \mid p _ {M _ {2}}\right) - D _ {T V} \left(p _ {M _ {2}} \mid \mid p _ {M _ {*}}\right) ] + \Delta\right) - \epsilon_ {\pi}) \tag {8} \\ \end{array}
$$
The  $\Delta$  term in Eq.(8) is still intractable. However, the fact that it covers  $d_{M_1}^{\pi_1}$  and  $d_{M_2}^{\pi_2}$  reminds us that we may explore its relationship with the three TVD terms lying in the expectation of  $d_{M_1}^{\pi_1}$ .
Theorem 5 (lΔl Upper Bound). Let  $M_{i}\in \mathcal{M}$  be the evaluated model and  $\pi_i\in \Pi$  be the policy derived from the model. The term  $\Delta$  can be upper bounded as:
$$
\left| \Delta \right| \leq \frac {2 \gamma}{1 - \gamma} \mathbb {E} _ {(s, a) \sim d _ {M _ {1}} ^ {\pi_ {1}}} \left[ D _ {T V} \left(p _ {M _ {1}} \mid \mid p _ {M _ {2}}\right) \max  _ {s, a} D _ {T V} \left(p _ {M _ {2}} \mid \mid p _ {M ^ {*}}\right) \right] + \frac {2 \epsilon_ {\pi}}{1 - \gamma} \max  _ {s, a} D _ {T V} \left(p _ {M _ {2}} \mid \mid p _ {M ^ {*}}\right) \tag {9}
$$
Under the online setting, it is assumed that the error caused by policy shift, namely  $\epsilon_{\pi}$ , compared to model bias has a relatively small scale [20, 53]. Therefore, we ignore the minor influence brought by the policy. Additionally, the terms lying in the expectation of  $d_{M_1}^{\pi_1}$  in Eq.(9) is the product of the model shift term and the model bias term in TVD form, both of which have the range [0, 1] [26], making the upper bound of  $|\Delta|$  become a higher-order term compared to each of them alone. Thus, the critical element to lift the lower bound in Eq.(8) is the terms lying in the expectation of  $d_{M_1}^{\pi_1}$ . By maximizing these terms, namely minimizing  $D_{TV}(p_{M_1}||p_{M_2}) + D_{TV}(p_{M_2}||p_{M^*})^4$ , we can achieve unifying model shift and model bias and adaptively fine-tuning the model updates to get a performance improvement guarantee.
To make it practically feasible, we make additional assumptions. Specifically, let  $(\mathcal{X},\Sigma)$  be a measurable space and  $\mathcal{F}$  is a set of functions mapping  $\mathcal{X}$  to  $\mathbb{R}$  that contains  $V_{M}^{\pi}$ . When  $V_{M}^{\pi}$  is  $L_{v}$ -Lipschitz with respect to a norm  $||\cdot ||$ , the integral probability metric [33] of two arbitrary dynamic transition function  $M,M^{\prime}\in \mathcal{M}$  defined on  $\mathcal{X}$  is as Eq.(10). Notice that if  $1\leq p\leq q$ ,  $W_{p}(p_{M},p_{M^{\prime}})\leq W_{q}(p_{M},p_{M^{\prime}})$  [32], and we can get  $W_{1}(p_{M},p_{M^{\prime}})\leq W_{2}(p_{M},p_{M^{\prime}})$ . Hence, Eq.(3) can be applied as the optimization objective to get a performance improvement guarantee.
$$
\sup  _ {f \in \mathcal {F}} \left| \mathbb {E} _ {s ^ {\prime} \sim p _ {M}} \left[ f \left(s ^ {\prime}\right) \right] - \mathbb {E} _ {s ^ {\prime} \sim p _ {M ^ {\prime}}} \left[ f \left(s ^ {\prime}\right) \right] \right| = \frac {R _ {\max }}{1 - \gamma} D _ {T V} \left(p _ {M} \| p _ {M ^ {\prime}}\right) = L _ {v} W _ {1} \left(p _ {M}, p _ {M ^ {\prime}}\right) \tag {10}
$$
4.3 Practical implementation
We now instantiate Algorithm 1 by demonstrating an explicit approach. To better clarify, we would like to state the four design decisions in detail: (1) how to parametrize the model, (2) how to estimate the model bias term and model shift term in phase 2, (3) how to use the model to generate rollouts and (4) how to use the model rollouts to optimize the policy  $\pi$ .
Predictive Model. We use a bootstrap ensemble of dynamic models  $\{p_M^1,\dots,p_M^B\}$ , whose elements are all probabilistic neural network, outputting the Gaussian distribution with diagonal covariance:  $p_{M}^{i}(s_{t + 1},r_{t + 1}|s_{t},a_{t}) = \mathcal{N}(\mu_{M}^{i}(s_{t},a_{t}),\Sigma_{M}^{i}(s_{t},a_{t}))$ . The probabilistic ensemble model can capture the aleatoric uncertainty arising from inherent stochasticities and the epistemic uncertainty corresponding to ambiguously determining the underlying system due to the lack of sufficient data [6]. In the field of MBRL, properly handling these uncertainties can achieve better asymptotical performance. Following the previous work [20], we select a model uniformly from the elite set to generate the transition at each step in the rollout process. In phase 1, the ensemble models are trained on shared but differently shuffled data, where the optimization objective is MLE [6, 20].
${}^{4}{D}_{TV}\left( {{p}_{{M}_{1}}\parallel {p}_{{M}^{ * }}}\right)$  is a constant for the optimization of  ${M}_{2}$
Model Shift and Model Bias Estimation. Recalling that the model shift term refers to  $W_{2}(p_{M_{1}}||p_{M_{2}})$  and the model bias term refers to  $W_{2}(p_{M_{2}}||p_{M^{*}})$ . For arbitrary two Gaussian distributions  $p \sim \mathcal{N}(\mu_1, \Sigma_1)$  and  $q \sim \mathcal{N}(\mu_2, \Sigma_2)$ , the second-order Wasserstein distance between them is derived as Eq.(11) [35].
$$
W _ {2} (p, q) = \sqrt {\left\| \mu_ {1} - \mu_ {2} \right\| _ {2} ^ {2} + \operatorname {t r a c e} \left(\Sigma_ {1} + \Sigma_ {2} - 2 \left(\Sigma_ {2} ^ {\frac {1}{2}} \Sigma_ {1} \Sigma_ {2} ^ {\frac {1}{2}}\right) ^ {\frac {1}{2}}\right)} \tag {11}
$$
Let  $k_{1}$  denote the index of the selected model from  $M_{1}$  ensemble and  $k_{2}$  denote that from  $M_{2}$  ensemble. For the model shift term, we approximate it as  $W_{2}(p_{M_{1}}^{k_{1}}, p_{M_{2}}^{k_{2}})$ . For the model bias term, we approximate it as  $\frac{1}{B - 1}\sum_{b = 1, b \neq k_{2}}^{B}W_{2}(p_{M_{2}}^{k_{2}}, p_{M_{2}}^{b})$ .
Model Rollout. Following the previous work [20], we use short branch rollouts to alleviate the impacts of compounding error.
Policy Optimization. Our work allows being plugged in many SOTA MFRL algorithms, e.g. SAC [15], TD3 [11], etc. Here we employ SAC as an example.
Our experiments are designed to investigate four primary questions: (1) Whether we need to propose an algorithm similar to USB-PO to adaptively adjust the impacts of model shift? (2) How well does USB-PO perform on reinforcement learning benchmark tasks compared to SOTA MBRL and MFRL algorithms? (3) How to understand USB-PO? (4) Does USB-PO have a similar performance on different learning rate settings in phase 2?
5.1 Necessity of USB-PO
We recall that CMLO [21] sets a fixed threshold to constrain the impacts of model shift, which we argue CMLO is heavily dependent on this threshold. To valid our statement, we devise an experiment that sets three different thresholds for CMLO on Walker2d environment in MuJoCo [46]. As Figure 1 shows, the performance corresponding to the other two thresholds (1.0
and 5.0) is severely affected. Therefore, setting a fixed threshold to constrain is inappropriate, and a smarter way like USB-PO should be applied to adaptively adjust the impacts of model shift.