SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1) Lower bound  $\hat{\rho}_d^{var}$ . The variance of DRaE  $\tilde{V}_{ij}^{DR}(s_t)$  can be determined by Eq. (6), with the true  $\rho_d^{ij}(t)$  estimated with  $\hat{\rho}_d^{ij}(t)$ . Through minimizing this variance w.r.t.  $\hat{\rho}_d^{ij}(t)$ , we formulate a convex optimization problem since the objective is a quadratic function of  $\hat{\rho}_d^{ij}(t)$ , with the optimal solution  $\hat{\rho}_d^{var}(t)$  obtained by letting the first-order derivative of this objective equal to zero:
$$
\hat {\rho} _ {d} ^ {v a r} (t) = \left(\rho_ {\pi} ^ {i j} (t) \gamma \mathbb {E} _ {t + 1} \left[ V _ {j} \left(s _ {t + 1}\right) \right] - V _ {j} \left(s _ {t}\right)\right) / \left(\gamma \rho_ {\pi} ^ {i j} (t) \tilde {V} _ {i j} ^ {D R} \left(s _ {t + 1}\right)\right). \tag {10}
$$
The distance from the optimal  $\hat{\rho}_d^{ij*}(t)$  to this lower bound  $\hat{\rho}_d^{var}(t)$  can then be determined as:
$$
\hat {\rho} _ {d} ^ {i j *} (t) - \hat {\rho} _ {d} ^ {v a r} (t) = \Big (2 V _ {j} (s _ {t}) - \rho_ {\pi} ^ {i j} (t) \gamma \mathbb {E} _ {t + 1} [ V _ {j} (s _ {t + 1}) ] - \rho_ {\pi} ^ {i j} (t) r (s _ {t}, a _ {t}) \Big) \Big / \Big (2 \gamma \rho_ {\pi} ^ {i j} (t) \tilde {V} _ {i j} ^ {D R} (s _ {t + 1}) \Big).
$$
which will be reduced by increasing the value of policy importance weight  $\rho_{\pi_i}^{ij}(t)$ . In meta-RL, informative trajectories for the target task  $\mathcal{T}_j$  are actually sampled from an original task  $\mathcal{T}_i$  using  $\pi_i$ , so the actions in these trajectories have a relatively larger sampling probability by  $\pi_i$  than  $\pi_j$ , resulting in a small  $\rho_{\pi}^{ij}(t)$  especially at the early stage of training. Then, as the training proceeds and with the increase of reward that policy  $\pi_j$  can achieve for task  $\mathcal{T}_j$ , these selected trajectories from task  $\mathcal{T}_i$  will be less informative as it is more likely for policy  $\pi_j$  to generate them also in the target task  $\mathcal{T}_j$ . This may lead to an increase of  $\rho_{\pi}^{ij}(t)$  along the training process, which in turn can consistently reduce the distance between  $\hat{\rho}_{d}^{ij*}(t)$  and  $\hat{\rho}_{d}^{var}(t)$ . In Section 5.3, we will empirically verify that  $\rho_{\pi}^{ij}(t)$  will generally increase from the initial small values that are much less than one to a stable value around one, which can lead to a tighter lower bound  $\hat{\rho}_{d}^{var}(t)$ . In addition, decreasing the estimated value of  $\hat{\rho}_{d}^{ij}(t)$  from the optimal  $\hat{\rho}_{d}^{ij*}(t)$  to the lower bound  $\hat{\rho}_{d}^{var}(t)$  will monotonically reduce the variance but increase the bias of DRaE, while a smaller estimate  $\hat{\rho}_{d}^{ij}(t) < \hat{\rho}_{d}^{var}(t)$  will cause a consistently larger MSE. Therefore, the optimal  $\hat{\rho}_{d}^{ij*}(t)$  is lower bounded by  $\hat{\rho}_{d}^{var}(t)$ .
2) Upper bound  $\rho_d^{ij}$ . Note that when  $\hat{\rho}_d^{ij}(t) = \rho_d^{ij}(t)$ , our DRaE  $\tilde{V}_{ij}^{DR}(s_t)$  becomes the unbiased DR estimator  $V_{ij}^{DR}(s_t)$ , while any deviation from the true dynamics importance weight  $\rho_d^{ij}(t)$  will result in a biased estimator. On the other hand, increasing the value of  $\hat{\rho}_d^{ij}(t)$  from  $\hat{\rho}_d^{var}(t)$  to  $\rho_d^{ij}(t)$  will consistently enlarge the variance. Therefore, a larger estimated value  $\hat{\rho}_d^{ij}(t) > \rho_d^{ij}(t)$  will enlarge both the bias and variance, which then indicates that the optimal  $\hat{\rho}_d^{ij*}(t)$  is upper bounded by  $\rho_d^{ij}(t)$ .
In conclusion, we show that the optimal dynamics importance weight  $\hat{\rho}_d^{ij*}(t)$  falls within the interval  $[\hat{\rho}_d^{var}(t),\rho_d^{ij}(t)]$ , while an estimated value of  $\hat{\rho}_d^{ij}(t)$  inside this interval can generally achieve a bias-variance tradeoff. In the following, we further derive a sample-independent upper bound to constrain the MSE difference between taking  $\hat{\rho}_d^{ij*}(t)$  and  $\hat{\rho}_d^{ij}(t)$  as the estimated value in DRaE, respectively.
Proposition 3.2 For an estimated dynamics importance weight  $\hat{\rho}_d^{ij}(t) \in [\hat{\rho}_d^{var}(t), \hat{\rho}_d^{ij}(t)]$ , the MSE difference between DRaE taking  $\hat{\rho}_d^{ij*}(t)$  and  $\hat{\rho}_d^{ij}(t)$  as estimated importance weight is bounded by:
$$
\begin{array}{l} \left| M S E \left(\tilde {V} _ {i j} ^ {D R}, \hat {\rho} _ {d} ^ {i j *} (t)\right) - M S E \left(\tilde {V} _ {i j} ^ {D R}, \hat {\rho} _ {d} ^ {i j} (t)\right) \right| \leq \frac {3}{4} \left| \mathbb {E} _ {a _ {t} \sim \pi_ {j}} \mathbb {E} _ {s _ {t + 1} \sim p _ {j}} \left[ \left(\frac {1}{\rho_ {d} ^ {i j} (t)} - 2\right) \gamma V _ {j} \left(s _ {t + 1}\right) \right] \right. \\ - \mathbb {E} _ {a _ {t} \sim \pi_ {j}} \left[ r _ {j} (s _ {t}, a _ {t}) \right] \Big | ^ {2} + \max \left(\mathbb {E} _ {t} \left[ \big (A \hat {\rho} _ {d} ^ {i j *} (t) - B \big) ^ {2} \right], \Big | V a r _ {t} (\tilde {V} ^ {D R}, \hat {\rho} _ {d} ^ {i j *} (t)) - V a r _ {t} (V ^ {D R}, \rho_ {d} ^ {i j} (t)) \Big |\right), \\ \end{array}
$$
where  $A = \gamma \rho_{\pi}^{ij}(t)\tilde{V}_{ij}^{DR}(s_{t + 1})$  and  $B = -\gamma \rho_{\pi}^{ij}(t)\mathbb{E}_{t + 1}[V_j(s_{t + 1})] - \rho_{\pi}^{ij}(t)\Delta (s_t,a_t) + V_\theta (s_t,z_j).$
It thus indicates that by setting an upper bound for  $\hat{\rho}_d^{ij}(t)$  smaller than the true  $\rho_d^{ij}(t)$ , we can further reduce the MSE difference between using  $\hat{\rho}_d^{ij}(t)$  and  $\hat{\rho}_d^{ij*}(t)$ . See Appendix A.8 for detailed proof.
4 Doubly Robust Augmented Transfer (DRaT) for Meta-RL
Based on our theoretical findings of the MSE reduction for sample transfer by DRaE, we formally present the doubly robust augmented transfer (DRaT) algorithm for meta-RL, and summarize its detailed implementation on PEARL [4] in Algorithm 1. At the data collection phase, for a task  $\mathcal{T}_i$ , we additionally store the distribution parameters of adapted policy  $\pi_i$  that collects data into buffer  $\mathcal{B}_i$ . Later at the training phase, we sample an informative trajectory set  $\mathcal{D}_j$  from all the replay buffers using strategy  $S_I$ , which selects trajectories with higher relabeled sum of rewards for task  $\mathcal{T}_j$ . Hence, any of the hindsight-based methods can be used to determine  $S_I$  here. While for environment where tasks share the same reward, we can simply choose the higher reward trajectories without relabeling.
We then compute the importance weights  $\rho_{\pi}^{ij}$  and  $\hat{\rho}_d^{ij}$  for each trajectory in  $\mathcal{D}_j$  between the target task  $\mathcal{T}_j$  and originally sampled task  $\mathcal{T}_i$  as in Line 10. We can obtain  $\pi_j$  for task  $\mathcal{T}_j$  following the same adaptation procedure as in PEARL, hence  $\rho_{\pi}^{ij}$  of actions in trajectories can be computed as the ratio of sampled probabilities  $\pi_j$  and  $\pi_i$ . While for the dynamics importance weight, a straightforward way to estimate  $\hat{\rho}_d^{ij}$  is to predict the distribution of transition  $(s,a,s')\in \tau_i\sim \mathcal{D}_j$  by  $p_{\psi}$  and compute it by  $\hat{\rho}_d^{ij} = p_\psi^j (\cdot |s,a) / p_\psi^i (\cdot |s,a)$ . However, the incorrect estimate of transition distribution in the original task  $\mathcal{T}_i$  may cause a small value of  $p_{\psi}^{i}(\cdot |s,a)$  and thus a large value of  $\hat{\rho}_d^{ij}$ , possibly frustrating
Algorithm 1 Doubly Robust augmented Transfer (DRaT) for Meta-RL
1:Require:Training tasks set  $\mathcal{T} = \{\mathcal{T}_i\}_{i = 1\dots N}$  from  $P_{\mathcal{T}}$  ,informative data selection strategy  $S_{I}$
2: Initialize replay buffers  $\mathcal{B}_i$  and context  $c_{i}$  for each training task  $\mathcal{T}_i$ , actor  $\pi_{\theta}$ , critic  $Q_{\theta}$ , and dynamics prediction network  $p_{\psi}^{i}(\cdot |s,a)$ .
3: while meta-training iteration do
4: for each training task  $\mathcal{T}_i$  do
5: Collect trajectories using policy  $\pi_i = \pi_\theta(a|s, z_i)$ ,  $z_i \sim q_\phi(z|c_i)$
6: Add trajectories and action selection probability of  $\pi_{i}$  to corresponding buffer  $\mathcal{B}_i$
7: for each task  $\mathcal{T}_j$  do
8: Sample informative trajectory set  $\mathcal{D}_j$  and corresponding policy  $\pi_i$  using strategy  $S_I$
9: Sample context  $c_{j}\sim \mathcal{B}_{j}$  and its latent variable  $z_{j}\sim q_{\phi}(\cdot |c_{j})$  , then  $\pi_j = \pi_\theta (\cdot |s,z_j)$
10: Compute importance weights  $\rho_{\pi}^{ij} = \frac{\pi_{\theta}(a|s,z_j)}{\pi_{\theta}(a|s,z_i)}$  and  $\hat{\rho}_d^{ij} = \max \left(\frac{p_\psi^j(\cdot|s,a)}{\mathcal{N}(\cdot,\sigma)},\hat{\rho}_d^l\right)$
11: Compute  $\tilde{V}^{DR}$  using Eq. (5) with  $\rho_{\pi}^{ij}$  and  $\hat{\rho}_d^{ij}(t)$
12: Compute critic loss in Eq. (1) using  $\hat{V}^{DR}$  as target value and update prediction network
13: Update the lower bound  $\hat{\rho}_d^l$  for clipping  $\hat{\rho}_d(t)$
the entire training process. Instead, we estimate  $\hat{\rho}_d^{ij}$  as guided by the interval-based approximation for optimal  $\hat{\rho}_d^{ij*}$  of DRaE in Section 3.4, which is constrained within an interval  $[\hat{\rho}_d^{var},\rho_d^{ij}]$ . We predict the distribution of transition  $(s,a,s')$  in task  $\mathcal{T}_j$  by prediction network  $p_{\psi}^j (\cdot |s,a)$  and in the original task  $\mathcal{T}_i$  by Gaussian distribution  $\mathcal{N}(s',\sigma)$ , which indicates a high probability of transition  $(s,a,s')\in \tau^i$  occurring in task  $\mathcal{T}_i$ . Hence, the maximum value of  $\hat{\rho}_d^{ij}$  computed by  $p_{\psi}^j (\cdot |s,a) / \mathcal{N}(\cdot ,\sigma)$  will automatically fall into interval  $[0,\rho_d^{ij})$  with a high probability, due to the large denominator resulted from the high belief we hold about the transition's occurrence in task  $\mathcal{T}_i$ . In addition, we also clip the small value of  $\hat{\rho}_d$  using  $\hat{\rho}_d^l$ , which is updated as the average value of  $\hat{\rho}_d^{var}$  at this epoch.
The importance weights  $\rho_{\pi}^{ij}$  and  $\hat{\rho}_d^{ij}$  are used to estimate the state value in  $\mathcal{D}_j$  by DRaE  $\tilde{V}^{DR}$  as in Eq. (5), which is used as the target value to compute critic loss in Eq. (1). Finally, the prediction network  $p_{\psi}$  is updated by optimizing the prediction error, and other networks are updated following the same procedure as in PEARL. It is noted that DRaT can also be implemented on other meta-RL baselines, such as MAML [12] and its variants, where the DRaE  $\tilde{V}^{DR}$  can be used to compute policy gradients for the meta-policy parameter optimization.
In this section, we assess performance of our DRaT approach following the same evaluation procedure as in [4]. For each benchmark environment, we construct a test task set that is disjoint with the training task set. At the end of each training epoch, we run the evaluation procedure, where trajectories of a fixed length are sampled by the adapted meta-policy from the updated context with continuously sampled transitions. Specifically, we conduct experiments to answer the following questions.
- Can DRaT achieve a better adaptation performance on meta-test tasks in the extreme challenging sparse-reward environment with varying reward functions and dynamics, as compared to hindsight relabeling methods that inherently considers only the difference of reward functions?
- Can DRaT effectively improve the adaptation performance on the standard meta-RL benchmarks where tasks share the identical dense reward function but only differ in distinct dynamics?
- Will the policy importance weight  $\rho_{\pi}^{ij}$  increase along the training process in consistency to the performance improvement of DRaT, which thus provides a tighter lower bound for  $\hat{\rho}_d^{ij}$ ? Will DRaE provide a better value estimate for meta-RL?
Environment setup. We compare our DRaT with other baseline algorithms on six robotic control tasks, which can be divided into two families according to their reward and dynamics setup. For the implementation detail of DRaT, please refer to Appendix B.1. These tasks are all simulated via MuJoCo [13], where we further generate various dynamics by randomly sampling the environment parameters, including body mass, body inertia, joint damping, and body component's friction. One
(a) Point-Robot-Params-Sparse
(b) Ant-Params-Sparse
Figure 2: Evaluation curves of average return: test-task performance vs. time steps.
(a) Walker-2D-Params
Figure 3: Evaluation curves of average return: test-task performance vs. time steps.
(b) Hopper-Params
(c) Humanoid-Params-Sparse formance vs. time steps.
(c) Point-Robot-Params
family contains the sparse-reward environments with varying reward functions and dynamics: 1) Point-Robot-Goal-Params: control the arm of a 3D robot to reach random goals in the 3D space; 2) Ant-Goal-Params-Sparse: navigate a quadruped robot to reach randomly generated goals along the perimeter of a semi-circle; 3) Humanoid-Goal-Params-Sparse: navigate a humanoid robot to reach randomly generated goals on a semi-circle. These environments are with the sparse-reward setting in the sense that the RL agent can receive a meaningful reward signal that indicates its distance from the goal only when it reaches areas near the goals. The other family includes environments that have varying dynamics and the identical dense reward function: 1) Hopper-Params: control a planar monopod robot to hop as fast as possible and avoid falling; 2) Walker-2D-Params: control a 2D bipedal robot to walk and perform the same task as in Hopper; 3) Point-Robot-Params: the same as Point-Robot-Goal-Params but with dense reward function. Please refer to Appendices B.2 and B.3 for the detail of varying dynamics and reward function settings, respectively.
5.1 Evaluation on Sparse-Reward Environments with Varying Rewards and Dynamics
To answer the first question, we conduct experiments on the first family of sparse-reward environments with varying reward functions and dynamics. We compare DRaT with four baselines: PEARL [4], Hindsight Task Relabeling (HTR) [14], Hindsight Foresight Relabeling (HFR) [15], and Approximate Inverse RL Relabeling (AIR) [6]. Specifically, PEARL is utilized as the baseline meta-RL algorithm for all the other algorithms. For a fair comparison, we keep all the hyper-parameters about meta-RL the same as those in PEARL's source code. In HTR, we implement the single episode relabeling strategy, where hindsight task goal is selected as a state that is reached in a sampled episode and networks in HTR are trained by transitions from that episode which is relabeled by the hindsight goal. HFR utilizes a utility function (foresight) to judge the informativeness of trajectories to a task, and trajectories with higher utilities in certain tasks are more likely to be transferred through the relabelling with reward function (hindsight) to this task. In AIR, a trajectory is relabeled to a new task, for which this trajectory beats the most previously sampled trajectories in terms of the total reward among a sampled candidate task set. We use the same relabeling strategy in both the AIR and our DRaT for selecting informative trajectories for each training task.
Evaluation results. We report the averaged return of trajectories sampled in the evaluation phase. From the evaluation return curves of the three tasks as shown in Figs. 2(a)-2(c), our DRaT significantly outperforms the other baseline algorithms, especially in the Ant and Humanoid environments that have higher dimensions of states. In comparison, PEARL struggles to train the Ant and Humanoid under such a challenging environment setting. PEARL-based HTR cannot even learn to move to goals for Ant in such a short period of time steps, while it achieves a similar performance to PEARL for Humanoid. PAERL-based HFR tends to outperform PEARL and other relabeling methods for both Ant and Humanoid, which is possibly because HFR considers relabeling w.r.t. performance of the adapted policy using trajectory instead of just considering the sum of relabeled rewards. We also observe that simply applying AIR into PEARL will even degrade the performance of PEARL. Providing a more accurate estimate of state value functions for transferred informative trajectories, our DRaT generally achieves a better asymptotic performance and can learn faster compared to the other baselines. Note also that in Fig. 2, the solid curve illustrates the mean return on all runs of each algorithm with different random seeds, while the shaded area shows the standard deviation.
(a) Point-Robot-Params
(c) Humanoid-Params-Sparse
Figure 4: Curves of  $\rho_{\pi}^{ij}$ : policy importance weight vs. training epochs.
(a) Point-Robot-Params
Figure 5: Curves of errors of value estimation: absolute difference vs. training epochs.
5.2 Evaluation on Dense-Reward Environments with Varying Dynamics Only
Then, to answer the second question, we conduct experiments on the second family of environments with varying dynamics and the identical dense reward function. We compare DRaT with four baselines: PEARL [4], Importance Sampling augmented Transfer (IST), Meta-Q-Learning (MQL) [16], and Hindsight Foresight Relabeling (HFR) [14]. Specifically, PEARL is utilized as the baseline meta-RL algorithm for DRaT, IST and HFR. In MQL, for a given transition and task, a fitted logistic classifier outputs probability of the transition being sampled from the task, from which the importance weight can be estimated. Transitions from other tasks are used to compute the meta-RL objective multiplied by this importance weight on a new task, hence the contribution of each transition to the update of meta-parameters is judged by similarity between the new task and its source task. In IST, we use the IS estimator instead of our DRaE, while keeping all the other settings the same as in DRaT.
Evaluation results. We report the averaged return of trajectories sampled in the evaluation phase at each epoch, and show the evaluation return curves of the three tasks, respectively, in Figs. 3(a)-3(c). For MQL, we focus specifically on its performance with transfer by the estimated importance weights. Hence, we report its evaluation returns of adapted meta-policy with the aid of transferred samples. It can be seen that with informative samples transferred by DRaT, we can improve the evaluation average return over the other baselines in all the benchmark environments. In addition, DRaT consistently accelerates the training process of Walker-2D-Params and Hopper-Params at the early stage, and also reaches the highest return in Point-Robot-Params. In addition, DRaT presents in general a smaller standard deviation.
5.3 Performance Evaluation of Proposed DRaE
We then experimentally estimate the value of  $\rho_{\pi}^{ij}$  to evaluate its trend during the training process of DRaT. We compute the averaged value of  $\rho_{\pi}^{ij}$  estimated for informative trajectories at each training epoch, and show curves of the mean value of Point-Robot-Params, Ant-Params-Sparse, and Humanoid-Params-Sparse in Figs. 4(a)-4(c), respectively, where the shaded area shows the standard deviation on several runs. The results show that value of  $\rho_{\pi}^{ij}$  increases with the improvement of performance on all the training tasks, since selected informative trajectories are more likely to be sampled on target task  $\mathcal{T}_j$ , which can generate a tighter lower bound as we state in Section 3.4.
At last, we evaluate the estimation error of our proposed DRaT compared to the value network  $V_{\theta}$  as used in PEARL. For a better illustration, we do not plot results of directly using DR estimator, since it may result in a huge variance and estimation error as we analyze in Section 3.2. We analyze the error by computing the absolute difference between the averaged estimated value and averaged true return at each training epoch, and show the error curves of Point-Robot-Params, Ant-Params-Sparse, and Humanoid-Params-Sparse in Figs. 5(a)-5(c), respectively. The results show that our DRaE provides a better value estimation during the training process. It is also observed that curves of DRaT and  $V_{\theta}$  present a similar trend, which is because the computation of DRaE depends on the value of  $V_{\theta}$ .
6 Computational Complexity Analysis
Additional computational complexity is brought by DRaT for training of dynamics prediction network for each task, computation of DR estimate, and computation of relabelling for informative trajectories. Here, we analytically show that this additional computational complexity is comparable to its baseline
PEARL with a linear scaling factor. Typically in an off-policy meta-RL algorithm, each epoch (i.e., meta-training iteration) is divided into sampling and training phases. In the sampling phase, computational cost stems mainly from the actions chosen by feed-forward computation of the policy network and inference networks. Since all the algorithms (i.e., HTR, HFR, AIR, DRaT) follow the same sampling process as PEARL, we only analyze the computational cost in the training phase.
PEARL: In the training phase, the computational cost contains the feed-forward and back-propagation computation of policy network, value network and inference network. Given  $K$  training iterations at each epoch, batch size  $N_{\mathcal{B}}$  of transitions from  $N$  training tasks, state space cardinality  $|\mathcal{S}|$  and action space cardinality  $|\mathcal{A}|$ , and assuming a constant computational cost of feed-forward and backpropagation computation  $c_{1}$ , the total computational cost of training phase is  $O(K \cdot N_{\mathcal{B}} \cdot N \cdot |\mathcal{S}| \cdot |\mathcal{A}| \cdot c_{1})$ .
DRaT: Besides the same training cost  $O(K \cdot N_{\mathcal{B}} \cdot N \cdot |\mathcal{S}| \cdot |\mathcal{A}| \cdot c_1)$  as PEARL, additional computational cost brought by DRaT includes training of dynamics prediction networks, computation of DR estimator, and computation of relabelling, as follows. 1) Considering building separate prediction networks for  $N$  training tasks and the cost of feed-forward and back-propagation computation  $c_1$ , using the same batch of transitions for training, the additional cost of training dynamics prediction networks is also  $O(K \cdot N_{\mathcal{B}} \cdot N \cdot |\mathcal{S}| \cdot |\mathcal{A}| \cdot c_1)$ . 2) Considering sampling informative trajectories with a maximum length of  $L$  for  $N$  training tasks and the cost  $c_2$  of DR estimation computation at each time step, the computational cost of DR estimation is  $O(K \cdot N \cdot L \cdot |\mathcal{S}| \cdot |\mathcal{A}| \cdot c_2)$ . 3) We use the approximate inverse RL relabeling (AIR), where we sample one trajectory from each training task for relabeling, leading to  $N$  candidate trajectories in total. Assuming that the cost of computing relabeled reward at each time step is  $c_3$ , the cost of relabeling is then  $O(K \cdot N \cdot L \cdot |\mathcal{S}| \cdot |\mathcal{A}| \cdot c_3)$ . Thus, the additional computational complexity of DRaT is dominated by  $O\left(K \cdot N \cdot \max\{N_{\mathcal{B}}, L\} \cdot |\mathcal{S}| \cdot |\mathcal{A}| \cdot \max\{c_1, c_2, c_3\}\right)$ , comparable to PEARL with a linear scaling factor  $\frac{\max\{N_{\mathcal{B}}, L\}}{N_{\mathcal{B}}} \cdot \frac{\max\{c_1, c_2, c_3\}}{c_1}$ .
Insights for complexity reduction. 1) To reduce the cost of training dynamics prediction network, we may consider training a meta-dynamics prediction network, which makes prediction for all the tasks with a single network, and thus eliminates the need of training a separate network for each task. 2) To reduce the cost of DR estimation, we may consider a similar solution in  $TD(n)$ , which makes a trade-off between  $TD(0)$  and Monte-Carlo estimation by using the  $n$ -step rollout and fitted network.