SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuankun Jiang $^{1}$ , Nuowen Kan $^{1}$ , Chenglin Li $^{2}$ , Wenrui Dai $^{1}$ , Junni Zou $^{1}$ , Hongkai Xiong $^{2}$
$^{1}$ Department of Computer Science and Engineering,  $^{2}$ Department of Electronic Engineering* Shanghai Jiao Tong University
Meta-reinforcement learning (Meta-RL), though enabling a fast adaptation to learn new skills by exploiting the common structure shared among different tasks, suffers performance degradation in the sparse-reward setting. Current hindsight-based sample transfer approaches can alleviate this issue by transferring relabeled trajectories from other tasks to a new task so as to provide informative experience for the target reward function, but are unfortunately constrained with the unrealistic assumption that tasks differ only in reward functions. In this paper, we propose a doubly robust augmented transfer (DRaT) approach, aiming at addressing the more general sparse reward meta-RL scenario with both dynamics mismatches and varying reward functions across tasks. Specifically, we design a doubly robust augmented estimator for efficient value-function evaluation, which tackles dynamics mismatches with the optimal importance weight of transition distributions achieved by minimizing the theoretically derived upper bound of mean squared error (MSE) between the estimated values of transferred samples and their true values in the target task. Due to its intractability, we then propose an interval-based approximation to this optimal importance weight, which is guaranteed to cover the optimum with a constrained and sample-independent upper bound on the MSE approximation error. Based on our theoretical findings, we finally develop a DRaT algorithm for transferring informative samples across tasks during the training of meta-RL. We implement DRaT on an off-policy meta-RL baseline, and empirically show that it significantly outperforms other hindsight-based approaches on various sparse-reward MuJoCo locomotion tasks with varying dynamics and reward functions.
Reinforcement learning (RL) has achieved a remarkable success in a variety of sequential decision making tasks, such as intelligent gaming [1] and robotic control [2]. Agents trained by conventional RL methods aim at learning a single task, thus fail to adapt quickly to a new task with prior experience. In contrast, meta-reinforcement learning (meta-RL) focuses on learning to learn, i.e., learning how to adapt [3, 4]. In essence, it learns the underlying common structure from experience collected across a set of tasks, and then exploits this structure to fast adapt to similar new tasks with only a few trials.
Currently, one of the main challenges in meta-RL is the sparse-reward setting, which exists widely in real-world environments. When rewards become sparse, the agent receives only scarce information related to the task, bringing extreme difficulty to the meta-training and adaptation processes. To collect more useful experience, sample transfer approaches have been proposed. The key idea behind is to learn a new task by selectively leveraging samples generated from other tasks. Ideally, samples transferred from an original task to the new target task are required to be informative trajectories.
*Correspondence to: Chenglin Li <LCL1985@sjtu.edu.cn>, Wenrui Dai <daiwenrui@sjtu.edu.cn>, and Junni Zou <zoujunni@sjtu.edu.cn>.
Namely, the informative trajectories should present a higher and denser reward in the target task, though they might be unsuccessful attempts (i.e., with lower and sparser reward) in the original task.
Following the direction of sample transfer, hindsight experience replay (HER) and its variants [5, 6] have been developed as the typical methods in literature. Hindsight-based methods relabel a trajectory collected from the original task to a target task as informative trajectory, where rewards of this trajectory are re-computed using the target reward function to teach it how to achieve a different goal in the new task. However, HER algorithms are inherently designed for tasks that only differ in the reward function, while dynamics of all the different tasks are assumed to be identical. It is worth mentioning that in many real-world scenarios, tasks may differ not only in reward functions, but also in dynamics. For example, a robot is often trained under various working conditions, such as different load mass and joint damping, which may result in different distributions of dynamics. We are thus motivated to consider a more general meta-RL scenario where both dynamics and reward function can be different and varying across tasks, and study how to further cope with such a mismatch of dynamics distributions. This brings new challenges to existing meta-RL baselines, since consideration of only dynamics difference with the identical dense-reward setting fails to handle multi-goal tasks, while considering only the reward difference may misjudge values of transferred trajectories.
In off-policy RL, doubly robust (DR) estimators [7, 8] have emerged as an effective method to correct the distribution mismatch between a target policy and a behavior policy. They incorporate importance sampling with a fitted model of value function to alleviate the high variance issue of the original importance sampling-based estimator. In meta-RL, however, the discrepancy between the dynamics of the original and target tasks (as represented by the importance weight of dynamics) is practically inaccessible to the agent. This indicates that popular methods, such as directly clipping or removing large importance weight values [9, 10], proposed for providing a more accurate evaluation to enable efficient sample transfer and thus stable training, become infeasible in meta-RL.
In order to control the values of dynamics importance weights, we formulate the sample transfer as a mean square error (MSE) minimization problem and solve for the optimal dynamics importance weight. Since this optimal weight is intractable in practice, we propose an approximation strategy that constrains the estimated weight within an interval containing this optimal weight, and theoretically show that any estimate outside this interval would inevitably increase the MSE. Finally, we present a policy learning algorithm for sample transfer-based meta-RL, in which the proposed weight estimate is used for value-function evaluation. Our main contributions can be summarized as follows.
- Doubly Robust augmented Estimator. We design a doubly robust augmented estimator (DRaE), which for the first time accommodates to the more general and rational meta-RL setting and simultaneously allows varying dynamics and reward functions across different tasks. DRaE tackles the mismatch of dynamics distributions in meta-RL with a guaranteed optimum for the dynamics importance weight by minimizing MSE between the estimated and true values of the value function.
- Theoretically Robust Interval Approximation. We propose a tractable interval-based approximation for the optimal dynamics importance weight derived by our DRaE. This interval is deterministic to balance the variance and bias for a decreasing MSE and guaranteed to cover the optimum. We further verify that the proposed approximation is robust in theory with a constrained and sample-independent upper bound on the MSE approximation error from the optimum.
- Doubly Robust augmented Transfer Algorithm. We develop a doubly robust augmented transfer (DRaT) algorithm for transferring informative samples across tasks, based on the proposed DRaE. DRaT controls the importance weight under the guide of our interval-based approximation to minimize the MSE. It is noted that DRaT can be integrated into any value-based meta-RL baselines and implemented specifically on an off-policy meta-RL baseline, PEARL [4].
2.1 Meta-Reinforcement Learning (Meta-RL)
Meta-RL considers training on a task set  $\{\mathcal{T}_i\}$  to learn a set of meta-parameters, which can quickly adapt to solving a new (testing) task with only a limited number of samples. Each task  $\mathcal{T}_i$  then implies a Markov decision process (MDP) with both a distinct reward function  $r_i$  and a different transition probability distribution  $p_i(\cdot |s,a)$ . Such a meta-RL objective can be written as:  $\max_{\theta} \mathbb{E}_{P_{\mathcal{T}}} \left[ J_{\mathcal{T}}(\pi_{\theta_{\mathcal{T}_i}}) \right]$ , where  $P_{\mathcal{T}}$  denotes the sampling distribution over the training task set  $\mathcal{T}$ ,  $J_{\mathcal{T}}(\pi_{\theta_{\mathcal{T}_i}})$  is the expected
reward of policy  $\pi_{\theta \mathcal{T}_i}$  on the task  $\mathcal{T}_i$  , and the policy  $\pi_{\theta \mathcal{T}_i}$  is generated from  $\pi_{\theta}$  following a certain adaptation procedure given samples  $c_{i} = \{s_{t},a_{t},r_{t},s_{t + 1}\}$  from this task.
We use PEARL [4] specifically in this paper as the baseline meta-RL algorithm, which builds upon the soft actor-critic (SAC) [11]. Nevertheless, our proposed DRaT approach is compatible with and can be applied generally to any other meta-RL baselines. PEARL implements the adaptation by constructing an inference network  $q_{\phi}(z|c)$ , which takes in recently collected samples  $c$  of a task (also referred to as the context) and infers a latent context variable  $z$  that encodes salient information about this task. Then, a policy  $\pi_{\theta}(a|s,z)$  conditioned on  $z$  can adapt its behavior to this task. Both the inference network  $q_{\phi}(z|c)$  and critic  $Q_{\theta}(s,a,z)$  are trained with the following critic loss:
$$
\mathcal {L} _ {\text {c r i t i c}} = \frac {1}{2} \mathbb {E} _ {(s, a, r, s ^ {\prime}) \sim \mathcal {B}, z \sim q _ {\phi} (z | c)} \left[ Q _ {\theta} (s, a, z) - \left(r (s, a) + \gamma \bar {V} _ {\theta} \left(s ^ {\prime}, \bar {z}\right)\right) \right] ^ {2}, \tag {1}
$$
where  $\bar{V}_{\theta}$  is the target value network and the overline notation specifies no gradient backpropagation going through  $\bar{V}_{\theta}$  and  $\bar{z}$ . While the policy is trained using the same actor loss as in SAC [11].
2.2 Doubly Robust Estimator for Off-Policy Policy Evaluation
In off-policy policy evaluation, we expect to estimate the state value of a target policy  $\pi$  given data sampled from a behavior policy  $\mu$ , where distribution mismatch stems from varying policies. Importance sampling (IS) estimators can correct this mismatch by multiplying the importance weights of policies, thus providing an unbiased estimate:  $V^{IS}(s_t = s) = \left[\prod_{k = t}^{T - 1}\rho_\pi (k)\right]\cdot \left[\sum_{k = 0}^{T - t - 1}\gamma^k r(s_{t + k},a_{t + k})\right]$ , where  $\rho_{\pi}(k) = \frac{\pi(a_k|s_k)}{\mu(a_k|s_k)}$  denotes the importance weight between the target policy  $\pi$  and behavior policy  $\mu$  at the step  $k$ . Unfortunately, the IS estimator suffers from high variance due to possible high values of the policy importance weights, while the product of policy importance weights will also grow exponentially with the horizon [7]. Through incorporating a fitted value function into the IS estimator, this variance can be reduced by doubly robust (DR) estimators, which approximate the value-function at  $s_t$  recursively as:
$$
V ^ {D R} \left(s _ {t} = s\right) = \hat {V} (s) + \rho_ {\pi} (t) \left[ r \left(s _ {t}, a _ {t}\right) + \gamma V ^ {D R} \left(s _ {t + 1}\right) - \hat {Q} (s, a _ {t}) \right], \tag {2}
$$
where  $\hat{V}(s) = \mathbb{E}_{a \sim \pi(\cdot | s)}[\hat{Q}(s, a)]$  is computed by the fitted  $Q$ -function, i.e.,  $\hat{Q}$ . Here, "doubly robust" refers to that the estimated value is unbiased when either one of the  $\rho_{\pi}$  and  $\hat{Q}$  is an unbiased estimate.
3 Doubly Robust Augmented Estimator (DRaE) for Sample Transfer
In this section, we consider leveraging the sample transfer between tasks in a more general sparse-reward meta-RL setting, where both the reward functions and dynamics are varying across different tasks. Specifically, we formulate the sample transfer as a mean square error (MSE) minimization problem between the estimate value of samples transferred from other tasks and their true values in the target task. Other than the simple transfer of relabeled samples in hindsight-based methods without accommodating to the dynamics mismatch, solution to the proposed sample transfer problem can provide the true value estimate of transferred samples for the subsequent tasks in meta-training, which implicitly incorporates the differences of both reward functions and dynamics models.
We then show that a direct use of DR will bring additionally high variance incurred by environment dynamics, since the importance weight of dynamics might be large due to various dynamics. Though the unbiasedness of DR estimate is appealing, its high variance will cause a large MSE on the estimated value, which significantly disturbs the meta-training process. Since MSE can be decomposed into the sum of bias and variance-related terms, we thus propose to minimize a derived upper bound of MSE and determine the optimal dynamics importance weight, aiming at balancing the bias and variance. Finally, to overcome the intractability of solving this MSE minimization in practice, we propose instead an interval-based approximation for the optimal importance weight, by showing that any other values outside this interval will enlarge both the bias and variance, thus with a larger MSE.
3.1 Problem Formulation of Sample Transfer in Meta-RL
In meta-RL, a training task set  $\mathcal{T}$  can be characterized by tuples  $\{\mathcal{T}_i = \langle S, \mathcal{A}, p_i, r_i \rangle, i = 1, \dots, N\}$ , where  $\mathcal{S}$  and  $\mathcal{A}$  denote the common state and action spaces, respectively, while a task  $\mathcal{T}_i$  is distinct
in both the transition distribution  $p_i$  and reward function  $r_i$ . At each training iteration, informative trajectories that are sampled from other tasks but expected to achieve a higher sum of rewards for a certain task  $\mathcal{T}_j$  will be transferred to and leveraged by  $\mathcal{T}_j$ , which is the core idea behind hindsight-based methods. Here, we argue that their intuitive sum of rewards, however, cannot accurately indicate the true state value  $V_j$  of a trajectory that is sampled from other tasks and transferred to this target task, especially when dynamics mismatches arise across tasks. For a more accurate estimation, we therefore formulate the following sample transfer problem by minimizing the MSE between the true state value  $V_j$  and its estimate  $\hat{V}_j$ :
$$
\min  _ {\hat {V} _ {j}} \sum_ {\tau_ {i} \sim \mathcal {D} _ {j}} \sum_ {t} \mathbb {E} _ {\tau_ {i} | _ {t: T}} \left[ \left(V _ {j} \left(s _ {t}\right) - \hat {V} _ {j} \left(s _ {t}\right)\right) ^ {2} \mid s _ {t} = s \right], \tag {3}
$$
where  $\mathcal{D}_j = \{\tau_i = \{s_t, a_t, r_t, s_{t+1}\}\}_{t=0}^T\}_{i \neq j}$  is the dataset that contains informative trajectories sampled from other tasks and selected using any relabeling strategy  $S_I$ ,  $s_t = s$  is the state at step  $t$  from a trajectory  $\tau_i$ , and the expectation is taken w.r.t. the randomness of trajectory  $\tau_i$  from step  $t$  to the final step  $T$  sampled in the task  $\mathcal{T}_i$ . Further, the MSE objective in Eq. (3) can be decomposed as:
$$
\mathbb {E} _ {\tau_ {i} | t: T} \left[ \left(V _ {j} (s _ {t}) - \hat {V} _ {j} (s _ {t})\right) ^ {2} | s _ {t} \right] = \left(V _ {j} (s _ {t}) - \mathbb {E} _ {\tau_ {i} | t: T} \left[ \hat {V} _ {j} (s _ {t}) | s _ {t} \right]\right) ^ {2} + V a r (\hat {V} _ {j} (s _ {t})), \tag {4}
$$
where the first term on the RHS equals square of the bias of  $\hat{V}_j$  that is estimated from computation on the informative trajectories, while the second term denotes the variance of estimate  $\hat{V}_j$ .
3.2 Direct Use of Doubly Robust (DR) Estimator
DR estimators can provide an unbiased estimator for off-policy policy evaluation with existence of the distribution mismatch. For a trajectory that is sampled from task  $\mathcal{T}_i$  and transferred to a target task  $\mathcal{T}_j$ , the unbiased estimate for the state value can be determined by directly using the idea of DR:
$$
V _ {i j} ^ {D R} \left(s _ {t} = s\right) = V _ {\theta} \left(s, z _ {j}\right) + \rho_ {\pi} ^ {i j} (t) \left[ r _ {j} \left(s, a _ {t}\right) + \rho_ {d} ^ {i j} (t + 1) \gamma V _ {i j} ^ {D R} \left(s _ {t + 1}\right) - Q _ {\theta} \left(s, a _ {t}, z _ {j}\right) \right], \tag {5}
$$
where  $\rho_{\pi}^{ij}(t) = \frac{\pi_{\theta}(a_t|s_t,z_j)}{\pi_{\theta}(a_t|s_t,z_i)}$  is the importance weight of adapted policy,  $\rho_d^{ij}(t) = \frac{p_j(s_{t+1}|s_t,a_t)}{p_i(s_{t+1}|s_t,a_t)}$  denotes the importance weight of transition distributions incurred by dynamics discrepancies, and  $Q_{\theta}(s,a_t,z_j)$  denotes the  $Q$ -value function for task  $\mathcal{T}_j$  fitted by the critic network with parameter  $\theta$ , which is then used to compute the fitted state value function  $V_{\theta}(s,z_j)$ . In meta-RL, the adapted policy for each task can be explicitly known, thus allowing access to the true policy importance weight  $\rho_{\pi}^{ij}(t)$ . Referring to Appendix A.2, it is then easy to verify that  $V_{ij}^{DR}(s_t = s)$  still holds the doubly robust property. Namely,  $V_{ij}^{DR}(s_t = s)$  is unbiased when either the importance weight or value function is correctly estimated. Plugging the unbiasedness of  $V_{ij}^{DR}(s_t = s)$  to Eq. (4), the MSE of this DR estimator can thus be simplified to  $\mathbb{E}_{\tau_i|t:T}\left[\left(V_j(s_t) - V_{ij}^{DR}(s_t)\right)^2 |s_t\right] = Var\left(V_{ij}^{DR}(s_t)\right)$ , which is dominated by its variance that can be further determined with the following theorem.
Theorem 3.1 The variance of estimator  $V_{ij}^{DR}(s_t = s)$  can be recursively given by:  $\forall t = 1, \dots, T$
$$
\begin{array}{l} V a r _ {t} \left[ V _ {i j} ^ {D R} (s _ {t} = s) \right] = \mathbb {E} _ {t} \left[ (\rho_ {\pi} ^ {i j} (t)) ^ {2} V a r _ {t} \left[ r _ {j} (s _ {t}, a _ {t}) | a _ {t} \right] \mid s _ {t} \right] + V a r _ {t} \left[ \rho_ {\pi} ^ {i j} (t) \Delta (s _ {t}, a _ {t}) \mid s _ {t} \right] \\ + \mathbb {E} _ {t} \left[ \left(\gamma \rho_ {\pi} ^ {i j} (t) \left(\rho_ {d} ^ {i j} (t) V _ {i j} ^ {D R} \left(s _ {t + 1}\right) - \mathbb {E} _ {t + 1} \left[ V _ {j} \left(s _ {t + 1}\right) \right]\right) - \rho_ {\pi} ^ {i j} (t) \Delta \left(s _ {t}, a _ {t}\right) + V _ {\theta} \left(s _ {t}, z _ {j}\right)\right) ^ {2} \right] \\ - \mathbb {E} _ {t} \left[ \left(- \rho_ {\pi} ^ {i j} (t) \Delta \left(s _ {t}, a _ {t}\right) + V _ {\theta} \left(s _ {t}, z _ {j}\right)\right) ^ {2} \right], \tag {6} \\ \end{array}
$$
where  $Var_{t}\left[V_{ij}^{DR}(s_{T})\right] = 0, \Delta(s_{t}, a_{t}) = Q_{\theta}(s_{t}, a_{t}, z_{j}) - Q_{\pi}^{j}(s_{t}, a_{t})$ , and  $Q_{\pi}^{j}$  is the true  $Q$ -value.
On the RHS of Eq. (6), the first, second and fourth terms all take expectation over  $\pi_{\theta}(a_t|s_t,z_i)$  while the second variance term is also due to randomness of  $\pi_{\theta}(a_t|s_t,z_i)$ . The variance inside the first term stems from randomness of the rewards (which will become zero if the reward function is deterministic), while the expectation inside the third term considers randomness of the future. Theorem 3.1 implies that when the other variables are kept unchanged, a possibly larger value for the
estimate of dynamics importance weight  $\rho_d^{ij}$  will enlarge the third term in Eq. (6), which in turn results in a higher variance and consequently a larger MSE. As an empirical validation, in Fig. 1, we show the standard deviation (SD) of state value estimate in the Point-Robot environment by directly using the DR estimator in Eq. (5). It can be seen that the estimated value by DR oscillates significantly during the entire training process, which will eventually cause an unstable and unsuccessful meta-training.
3.3 DR Augmented Estimator (DRaE) with Optimal  $\hat{\rho}_d^{ij*}$  to Balance Bias and Variance
To overcome the high variance and large MSE issue suffered by the original unbiased DR estimator, we consider the MSE minimization for sample transfer as in Eq. (3), which is equivalent to jointly optimizing the bias and variance w.r.t. the dynamics importance weight  $\rho_d^{ij}$ . Since the actual transition probability distributions of different tasks are usually inaccessible, in practice, we will use the estimated value of dynamics importance weight  $\hat{\rho}_d^{ij}$  instead of its true value  $\rho_d^{ij}$ . However, by approximating  $\rho_d^{ij}$  with  $\hat{\rho}_d^{ij}$ , the original unbiased DR estimator  $V_{ij}^{DR}(s_t = s)$  in Eq. (5) becomes biased augmented estimator  $\tilde{V}_{ij}^{DR}(s_t = s)$ , with the bias given by:
Figure 1: SD of value estimated by direct DR estimator.
$$
\left. \right. \operatorname {B i a s} \left(\hat {\rho} _ {d} ^ {i j} (t)\right) = \left| \mathbb {E} _ {a _ {t} \sim \pi_ {i}} \mathbb {E} _ {s _ {t + 1} \sim p _ {i}} \left[ \gamma \rho_ {\pi} ^ {i j} (t) \left(\hat {\rho} _ {d} ^ {i j} (t) \tilde {V} _ {i j} ^ {D R} \left(s _ {t + 1}\right) - \rho_ {d} ^ {i j} (t) V _ {i j} ^ {D R} \left(s _ {t + 1}\right)\right)\right]\right|. \tag {7}
$$
Referring to Appendix A.5 for the detailed derivation, through bounding this  $\mathrm{Bias}(\hat{\rho}_d^{ij})$  and by further decomposing the MSE into the sum of bias and variance-related terms, we can obtain an upper bound for the MSE of the biased DR augmented estimator  $\tilde{V}_{ij}^{DR}(s_t = s)$ :
$$
\begin{array}{l} \operatorname {M S E} \left(\tilde {V} _ {i j} ^ {D R} \left(s _ {t} = s\right)\right) \leq \mathbb {E} _ {t} \left[ \gamma \rho_ {\pi} ^ {i j} (t) \left(\hat {\rho} _ {d} ^ {i j} (t) \tilde {V} _ {i j} ^ {D R} \left(s _ {t + 1}\right) - \rho_ {d} ^ {i j} (t) V _ {i j} ^ {D R} \left(s _ {t + 1}\right)\right) \right] ^ {2} + \left(\mathbb {E} _ {t} V _ {j} \left(s _ {t}\right)\right) ^ {2} + \mathbb {V} \left(\rho_ {\pi}\right) \\ + \mathbb {E} _ {t} \left[ \left(\rho_ {\pi} ^ {i j} (t) \hat {\rho} _ {d} ^ {i j} (t) \gamma \tilde {V} _ {i j} ^ {D R} \left(s _ {t + 1}\right) - \rho_ {\pi} ^ {i j} (t) \Delta \left(s _ {t}, a _ {t}\right) + \bar {V} _ {\theta} \left(s _ {t}, z _ {j}\right) - \rho_ {\pi} ^ {i j} (t) \gamma \mathbb {E} _ {t + 1} \left[ V _ {j} \left(s _ {t + 1}\right) \right]\right) ^ {2} \right], \tag {8} \\ \end{array}
$$
where  $\mathbb{V}(\rho_{\pi})$  denotes the terms on the RHS of Eq. (6) that contain  $\rho_{\pi}^{ij}$  but without  $\hat{\rho}_d^{ij}$ . We then minimize this upper bound w.r.t.  $\hat{\rho}_d^{ij}$  for the MSE reduction, which leads to a convex optimization problem. By letting the first-order derivative of the upper bound in Eq. (8) w.r.t.  $\hat{\rho}_d^{ij}$  equal to zero, we can thus obtain the optimal estimated value of dynamics importance weight, as:
$$
\left. \rho_ {d} ^ {i j *} (t) = \left(\gamma V _ {j} \left(s _ {t + 1}\right) - r _ {j} \left(s _ {t}, a _ {t}\right)\right) / \left(2 \gamma \tilde {V} _ {i j} ^ {D R} \left(s _ {t + 1}\right)\right). \right. \tag {9}
$$
3.4 Interval-Based Approximation for Optimal  $\rho_d^{ij*}$  of DRaE
Though the optimal  $\hat{\rho}_d^{ij*}$  of DRaE can effectively balance the bias and variance by minimizing the MSE, its computation is nontrivial in practice, since the true state value  $V_{j}(s_{t + 1})$  at a certain state is also infeasible to access. Here, we propose an interval-based approximation for the optimal  $\hat{\rho}_d^{ij*}$ , and show that it falls with an interval between  $\hat{\rho}_d^{var}$  that achieves the minimum variance in Eq. (6) and the true importance weight  $\rho_d^{ij}$ . We further verify that any other values of  $\hat{\rho}_d^{ij}$  outside this interval  $[\hat{\rho}_d^{var},\rho_d^{ij}]$  will simultaneously enlarge the bias and variance, and consequently result in a larger MSE.