Doubly Robust Augmented Transfer for Meta-Reinforcement Learning
Authors: Yuankun Jiang
Nuowen Kan
Chenglin Li
Wenrui Dai
Junni Zou
Hongkai Xiong
Venue: NeurIPS 2023 poster (2023)
Keywords: Meta-reinforcement learning
doubly robust (DR)
sample transfer

Abstract: Meta-reinforcement learning (Meta-RL), though enabling a fast adaptation to learn new skills by exploiting the common structure shared among different tasks, suffers performance degradation in the sparse-reward setting. Current hindsight-based sample transfer approaches can alleviate this issue by transferring relabeled trajectories from other tasks to a new task so as to provide informative experience for the target reward function, but are unfortunately constrained with the unrealistic assumption that tasks differ only in reward functions. In this paper, we propose a doubly robust augmented transfer (DRaT) approach, aiming at addressing the more general sparse reward meta-RL scenario with both dynamics mismatches and varying reward functions across tasks. Specifically, we design a doubly robust augmented estimator for efficient value-function evaluation, which tackles dynamics mismatches with the optimal importance weight of transition distributions achieved by minimizing the theoretically derived upper bound of mean squared error (MSE) between the estimated values of transferred samples and their true values in the target task. Due to its intractability, we then propose an interval-based approximation to this optimal importance weight, which is guaranteed to cover the optimum with a constrained and sample-independent upper bound on the MSE approximation error. Based on our theoretical findings, we finally develop a DRaT algorithm for transferring informative samples across tasks during the training of meta-RL. We implement DRaT on an off-policy meta-RL baseline, and empirically show that it significantly outperforms other hindsight-based approaches on various sparse-reward MuJoCo locomotion tasks with varying dynamics and reward functions.

Doubly Robust Augmented Transfer for Meta-Reinforcement Learning
Yuankun Jiang', Nuowen Kan', Chenglin Li?, Wenrui Dai', Junni Zou', Hongkai Xiong?
DDparrmeet of Computer Science and Engineering, *Department of Electronic Engineering" Shanghai Jiao Tong University
Abstract
Meta-reinforrement learning (Meta-RL), though enabling fast adaptation learn new skills by exploiting the common structure shared among different tasks, suffers performance degradation in the sparse-reward setting. Current hindsight-baat baaeed sam- ple transfer approaches can alleviate this issue by transferring relabeled trajectories from other tasks a new task so as provide informative experience for the target reward function, but are unfortunately constrained with the unrealistic assumption that tasks differ only in reward functions. In this paper, we propose a doubly robust augmented transfer (DRaTT) approach, aiming at addressing the more general sparse reward meta-RI. scenario with both dynamics mismatches and varying reward func- tions across tasks. Specifically, we design doubly robust augmented estimator for efficient value-function evaluation, which tackles dynamics mismatches with the optimal importance weight of transition distributions achieved by minimizing the theoretically derived upper bound of mean squared error (MSE) between the estimated values of transferred samples and their true values in the target task. Due to its intractability, we then propose an interval- based approximation to this optimal importance weight, which is guaranteed to cover the optimum with constrained and sample-independent upper bound on the MSE approximation error. Based theoretical findings, we finally develop DRa'T algorithm for transferring informative samples across tasks during the training of meta-RL. We implement
DRaT on off- off-policy meta-RL. baseline, and empirically show that it significantly outperforms other hindsight-bass approaches on various sparse-reward MuJoCo locomotion tasks with varying dynamics and reward functions.
Introduction
Reinforcement learning (RL) has achieved remarkable success in variety of sequential decision making tasks, such as intelligent gaming 1 and robotic control 2 Agents trained by conventional RL methods aim at learning single task, thus fail to adapt quickly In contrast, meta-reinforcemennt learning (meta-RL) focuses on learning to learn, i.e., learning how to adapt 3 4. In essence, it learns the underlying common structure from experience collected across set of tasks, and then exploits this structure to fast adapt to similar new tasks with only a few trials.
task with prior experience.
Currently, one of the main challenges in meta-RL is the sparse- reward setting, which exists widely in real-world environments. When rewards become sparse, the agent receives only scarce information related to the task, bringing extreme difficulty to the meta-traint meta-training and adaptation processes. To collect more useful experience, sample transfer approaches have been proposed. The key idea behind is to learn new task by selectively leveraging samples generated from other tasks. Ideally, samples transferred from an original task to the new target task are required to be informative trajectories.
*Correspondence to: Chenglin Li <LCL19855 sju..cdu.cm> Wenrui Dai <daiwenrui @ sjtu.edu.en>, and Junni Zou <zoujunni @sjtu.edu.cns
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Namely, the informative trajectories should present a higher and denser reward in the target task, though they might be unsuccessful attempts (i.e., with lower and sparser reward) in the original task.
Following the direction of sample transfer, hindsight experience replay (HER) and its variants [5 have been developed the typical methods in literature. Hindsight-based methods relabel a trajectory collected from the original task to a target task as informative trajectory, where rewards of this trajectory are re-computed using the target reward function to teach it how achieve a different goal in the new task. However, HER algorithms are inherently designed for tasks that only differ in the reward function, while dynamics of all the different tasks are assumed to be identical. It is worth mentioning that in many real-world scenarios, tasks may differ not only in reward functions, but also in dynamics. For example, robot is often trained under various working conditions, such as different load mass and joint damping, which may result in different distributions of dynamics. We are thus motivated to consider a more general meta-RI. scenario where both dynamics and reward function can be different and varying across tasks, and study how to further cope with such mismatch of dynamics distributions. This brings new challenges to existing meta-RI. baselines, since consideration of only dynamics difference with the identical dense-reward setting fails to handle multi-goal tasks, while considering only the reward difference may misjudge values of transferred trajectories.
In off-policy RL, doubly robust (DR) estimators 7 g have emerged as an effective method to correct the distribution mismatch between a target policy and behavior policy. They incorporate importance sampling with a fitted model of value function to alleviate the high variance issue of the original importance sampling- paased estimator. In meta-RL, however, the discrepancy between the dynamics of the original and target tasks (as represented by the importance weight of dynamics) is practically inaccessible to the agent. This indicates that popular methods, such as directly clipping or removing large importance weight values (9] [0]. proposed for providing more accurate evaluation to enable efficient sample transfer and thus stable training. become infeasible in meta-RL.
In order to control the values of dynamics importance weights, we formulate the sample transfer as mean square error (MSE) minimization problem and solve for the optimal dynamics importance weight. Since this optimal weight is intractable in practice, we propose an approximation strategy that constrains the estimated weight within an interval containing this optimal weight, and theoretically show that any estimate outside this interval would inevitably increase the MSE. Finally, we present a policy learning algorithm for sample transfer- based meta-RI., in which the proposed weight estimate is used for value-function evaluation. Our main contributions can be summarized as follows.
Doubly Robust augmented Estimator. We design doubly robust augmented estimator (DRaE), which for the first time accommodates to the more general and rational meta-RL setting and simultaneously allows varying dynamics and reward functions across different tasks. DRaE tackles the mismatch of dynamics distributions in meta-RL. with a guaranteed optimum for the dynamics importance weight by minimizing MSE between the estimated and true values of the value function.
Theoretically Robust Interval Approximation. We propose tractable interval-based approxima- tion for the optimal dynamics importance weight derived by our DRaE. This interval is deterministic to balance the variance and bias for decreasing MSE and guaranteed to cover the optimum. We further verify that the proposed approximation is robust in theory with a constrained and sample- independent upper bound on the MSE approximation eтror from the optimum.
Doubly Robust augmented Transfer Algorithm. We develop a doubly robust augmented transfer (DRaT) algorithm for transferring informative samples across tasks, based on the proposed DRaE, DRaT controls the importance weight under the guide of our interval- based approximation to minimize the MSE. It is noted that DRaT can be integrated into any value-basee based meta-RL baselines and implemented specifically on an off-policy meta- RL baseline, PEARL (4.
2 Preliminary
2.1 Meta-Reiifforcement Learning (Meta-RL)
Meta-RL considers training on a task set (Ti) to learn a set of pertmeaar,,,ers, which can quickly adapt to solving a new (testing) task with only a limited number of samples. Each task Tú then implies a Markov decision process (MDP) with both a distinct reward function r; and a different transition probability distribution po(-|s, a). Such meta-RL objective can be written as: maxg Epy [J-(Te.) )] where Pr denotes the sampling distribution over the training task set T, T. (Te,, is the expected

---

reward of po on the task Ti. and the policy Rer; is generated from To following a certain adaptation procedure given samples = {st, at, Tt, St+1) from this task.
We use PEARL 4 specifically in this paper as the baseline meta-RL algorithm, which builds upon the soft actor-critic (SAC) [Л Nevertheless, our proposed DRaT approach is compatible with and can be applied generally to any other meta-RL baselines. PEARL implements the adaptation by constructing an inference network ( e), which takes in recently collected samples of a task (also referred as the context) and infers latent context variable that encodes salient information about this task. Then, a policy Te Te(als, 2) conditioned on 2 can adapt its behavior to this task. Both the inference network g (2(c) and critic Qo(s, a, 2) are trained with the following critic loss:
Ceritie - (.
[Qo(s, a, 2) - (r(s, + ())]=,
(I)
where Vo is the target value network and the overline notation specifies no gradient backpropagation going through Vo and E. While the policy is trained using the same actor loss as in SAC [11]
2.2 Doubly Robust Estimator for Off-Policy Policy Evaluation
In off- policy evaluation, expect to estimate the state value of a target policy given data sam- pled from a behavior policy j4, where distribution mismatch stems from varying polices. Importance sampling (IS) estimators correct this mismatch by multiplying the importance weights of policies,
thus providing an unbiased estimate: V s)
[TK= (x))
Ei= T-t-1 ( acth))
where Pr (k) = z(ek sx) denotes the importance weight between the target policy æ and behavior ples sp.
policy ye at the step k. Unfortunately, the IS estimator suffers from high variance due to possible high values of the policy importance weights, while the product of policy importance weights will also grow exponentially with the horizon 77]. Through incorporating a fitted value function into the IS estimator, this variance can be reduced by doubly robust (DR) estimators, which approximate the value-function at recursively as:
V VD 5) = ((s) + Px((t) [r(se, ar) + DV - Q(s oz)
(2)
where (s)) (+1)) [Q(s, a)] is computed by the fitted Q-function, i.e., Q. Here, "doubly robust" refers to that the estimated value is unbiased when either one of the p= and Q is an unbiased estimate.
3 Doubly Robust Augmented Estimator (DRaE) for Sample Transfer
In this section, consider leveraging the sample transfer between tasks in more general sparse- reward meta-RL setting, where both the reward functions and dynamics are varying across different tasks. Specifically, we formulate the sample transfer as mean square error (MSE) minimization problem between the estimate value of samples transferred from other tasks and their true values methods in the target task. Other than the simple transfer of relabeled samples in hindsight- without accommodating to the dynamics mismatch, solution to the proposed sample transfer problem can provide the true value estimate of transferred samples for the subsequent tasks in vetainraining, which implicitly incorporates the differences of both reward functions and dynamics models.
We then show that direct use of DR will bring additionally high variance incurred by environment dynamics, since the importance weight of dynamics might be large due to various dynamics. Though the unbiasedness of DR estimate is appealing, its high variance will cause large MSE on the esti- mated value, which significantly disturbs the meta-training process. Since MSE can be decomposed into the of bias and variance-related terms, we thus propose to minimize derived upper bound of MSE and determine the optimal dynamics importance weight, aiming at balancing the bias and variance. Finally, to overcome the intractability of solving this MSE minimization in practice, we propose instead an interval-based approximation for the optimal importance weight, by showing that any other values outside this interval will enlarge both the bias and variance, thus with a larger MSE.
3.1 Problem Formulation of Sample Transfer in Meta-RL
In LI, a training task set T can be characterized by tuples [T. ii where S and A denote the common state and action spaces, respectively, while task T, is distinct
3

---

in both the transition distribution p; and reward function r;. At each training iteration, informative trajectories that are sampled from other tasks but expected to achieve a higher sum of rewards for certain task T will be transferred to and leveraged by TΙ. which is the core idea behind hindsight- based methods. Here, we argue that their intuitive sum of rewards, however, cannot accurately indicate the true state value V, of trajectory that is sampled from other tasks and transferred to this target task, especially when dynamics mismatches arise across tasks. For a more accurate estimation, we therefore formulate the following sample transfer problem by minimizing the MSE between the true state value V and its estimate Vj:
min ΣΣ Braler
= _
(3)
DinDs
where D, {T+ = {st, at r't, S1+ [ lige is the dataset that contains informative trajectories sampled from other tasks and selected using any relabeling strategy S₁, St = is the state at step t from a trajectory Ti, and the expectation taken w.r.t the randomness of trajectory т; from step to the final step T sampled in the task Tr. Further, the MSE objective in Eq. (3) can be decomposed as:
E7 Hm (v)) - (s1)) se = (v)(st)) - Er (+r()) + Var (V)(st))
(4)
where the first term on the RHS equals square of the bias of VÚ that is estimated from computation on the informative trajectories, while the second term denotes the variance of estimate Vi.
3.2 Direct Use of Doubly Robust (DR) Estimator
DR estimators can provide unbiased estimator for off-policy policy evaluation with existence of the distribution mismatch. For a trajectory that is sampled from task T; and transferred to a target task Tj. the unbiased estimate for the state value can be determined by directly using the idea of DR:
VR s) Vo 2,) (t) (1)(s, + p3 (t + (DNVD(((SS))) Qo(s, a 23)]
(5)
where (t) = <a(a (x..a.) ma (ar (..2.) is the importance weight of adapted policy, (4) = (s
denotes the importance weight of transition distributions incurred by dynamics discrepancies, and Qa(s,at, 2z) denotes the Q-value function for task T; fitted by the critic network with parameter 0, which is then used to compute the fitted state value function (s, 2j). In meta-RI,, the adapted policy for each task can be explicitly known, thus allowing access the true policy importance weight p (t). Referring to Appendix A.2, it is then easy to verify that vi V.D s) still holds the doubly robust property. Namely, V,p s) is unbiased when either the importance weight or value function is correctly estimated. Plugging the unbiasedness of V.J V. s) to Eq. (4) the MSE of
this DR estimator can thus be simplified to Ey ler (VV (xx)) se = Var (s))). which is dominated by its variance that can be further determined with the following theorem.
Theorem 3.1 The variance of estimator V.D s) can be recursively given by: Vt = =1, 1..... T
Var, [V DR (s: = s)] = E. (vr, (r, a:) la. [sc] + Var, (t) az) 8t
+ E, (TP (p (0V))) (s441) (s441)
E44 1 (2)))))))())))1 (2) az) + Voloe, 23)
Er (- (t) Vo(Se, 2))²
(6)
where Var. [v) = 0, A(s a.) = Qo(sr, a, 2z) Q4 (St, az), and Qi is the true Q-value.
On the RHS of Eq. (6), the first, second and fourth terms all take expectation over (a 21). while the second variance term is also due to randomness of Te(a; 8t; 2;). The variance inside the first term stems from randomness of the rewards (which will become zero if the reward function deterministic). while the expectation inside the third term considers randomness of the future. Theorem 3.1 implies that when the other variables are kept unchanged, possibly larger value for the

---

estimate of dynamics importance weight P7 will enlarge the third term in Eq. (6). which in turn results higher variance and consequently a larger MSE. As an empirical validation, in Fig. L we show the standard deviation (SD) of state value estimate in the Point-Robot environment by directly using the DR estimator Eq. 5 can be seen that the estimated value by DR oscillates significantly during the entire training process, which will eventually cause an unstable and unsuccessful meta-training.
3.3 DR Augmented Estimator (DRaE) with Optimal Pd to Balance Bias and Variance
To overcome the high variance and large MSE issue suffered by the original unbiased DR estimator, we consider the MSE minimization for sample transfer as in Eq. ( which is equivalent to jointly optimizing the bias and variance w.r.t. the dynamies importance weight P Since the actual transition probability distributions of different tasks are usually inaccessible, in practice, we will use the
5D SD/ I Estimation)
Epoch
Figure 1: SD of value esti-
estimated value of dynamics importance weight Pi instead of its true value Pd However, by approximating Pa P with Pd the original
mated by direct DR estimator.
unbiased DR estimator ViD = s) in Eq. ( becomes biased, which is denoted as the DR augmented estimator
ij (sp = s). with the bias given by:
Bias (P) = E Eajsn, E. B. (v02 (p (t) - (DV (stta))
(7)
Referring Appendix A.5 for the detailed derivation, through bounding this Bias and by further decomposing the MSE into the sum of bias and variance- related terms, we can obtain an upper bound for the MSE of the biased DR augmented estimator V.Dn = s):
MSE ( s)) s Ee (1T10 ( (()V)(())))) + (s:)) + V(px) E. (n (8++1) 0 (t)A(s;, az) + Vo(s), (V 2
(8)
where V(pz) denotes the terms on the RHS of Eq. 6 that contain / but without P3 We then minimize this upper bound w.r.t. P for the MSE reduction, which leads to a convex optimization problem. By letting the first-order derivative of the upper bound in Eq. (8) w.r.t. pi equal to zero, we can thus obtain the optimal estimated value of dynamics importance weight, as:
* (1) = (7V3(*ut1) - (st( / DD (slt1)) (2++1D(())+1))
3.4 Interval-Based Approximation for Optimal P of DRaE
Though the optimal P of DRaF can effectively balance the bias and variance by minimizing the MSE, its computation is nontrivial practice, since the true state value (5(++1) at certain state also infeasible to access. Here, we propose an interval-based approximation for the optimal p and show that it falls with an interval between pver that achieves the minimum variance in Eq. (6) and the true importance weight p We further verify that any other values of p outside this interval Ipaer,
, Pd if will simultancously enlarge the bias and variance, and consequently result in a larger MSE.
1) Lower bound paar. The variance of DRaE V.PR(s) can be determined by Eq. with the true p (t) estimated with p (t). Through minimizing this variance w.r.t. p (t), we formulate a convex optimization problem since the objective is quadratic function of Pd (t), with the optimal solution pa (t) obtained by letting the first-order derivative of this objective equal to zero:
Pa (t) (V V,())) (1) (sr+1))
(10)
The distance from the optimal /3 (t) to this lower bound pg can then be determined as:
Pa (t)
3ij
Pa
qvar (t) = (2,)(s)) - of (1) (V5( p2)(() / (1) (s441))
5

---

which will be reduced by increasing the value of policy importance weight p (t). In meta-RL, informative trajectories for the target task Tj are actually sampled from an original task T; using Ty, so the actions in these trajectories have relatively larger sampling probability by n. than Tj. resulting in a small (²(() (t) especially at the early stage of training. Then, as the training proceeds and with the increase of reward that policy T can achieve for task T,, these selected trajectories from task T: will be less informative as it is more likely for policy xj to generate them also in the target task 7.. This may lead to an increase of p²((1) along the training process, which in turn can consistently reduce the distance between Pd (t) and PatT In Section 5.3 we will empirically verify that o (1) will generally increase from the initial small values that are much less than one to stable value around one, which can lead to a tighter lower bound / (t). In addition, decreasing
the estimated value of P (t) from the optimal P (t) to the lower bound (t) will monotonically reduce the variance but increase the bias of DRaF, while a smaller estimate Pd (t) < P (t) will cause a consistently larger MSE. Therefore, the optimal Pd (1) is lower bounded by (()r()).
2) Upper bound P/-- Note that when P (t) = P3 (t), our DRaE (s,) becomes the unbiased DR estimator (s²), while any deviation from the true dynamies importance weight P'd (t) will result in biased estimator. On the other hand, increasing the value of PJ/ (t) from P2 (t) to Pd (t) will consistently enlarge the variance. Therefore, larger estimated value Pid (t) > 032 (t) will enlarge both the bias and variance, which then indicates that the optimal p3 (1) upper bounded by pa (t)-
of
In conclusion, we show that the optimal dynamics importance weight p3" (t) falls within the interval [Pger (t), P [t)], while an estimated value of pa (t) inside this interval can generally achieve a bias- variance tradeoff. In the following, we further derive a
upper bound to constrain the MSE difference between taking 031" (1) and p (1) as the estimated value in DRaE, respectively.
Proposition 3.2 For an estimated dynamics importance weight (1) difference between DRaE taking p (t) and p3d (t) as estimated importance weight is bounded by:
(t) E [paer(t) Pt).
(t)), the MSE
E Extiop [ (0:(0)
(1/10) -2 [V360tti
MSS(VDD DR .2 MSE(DDR pd (t)) S 3
.P
EE
EE [[( + max (Ap (t) - "} Vari .p' (t)) - Varc(V DR .2(()))
where A = (1) (8(+1) and B = -Pxk (1)E -
(8), a.) + Vo(
It thus indicates that by setting an upper bound for PY (t) smaller than the true Pl (t), we can further reduce the MSE difference between using p3 (t) and p (t). See Appendix A.8 for detailed proof.
4 Doubly Robust Augmented Transfer (DRaT) for Meta-RL
Based on theoretical findings of the MSE reduction for sample transfer by DRaE, formally present the doubly robust augmented transfer (DRaT) algorithm for meta-RL,, and summarize its detailed implementation on PEARL [4] in Algorithm 1 At the data collection phase, for task T. we additionally store the distribution parameters of adapted policy n, that collects data into buffer B,. Later the training phase, we sample an informative trajectory set Dj from all the replay buffers using strategy S₁, which selects trajectories with higher relabeled sum of rewards for task Ty. Hence, any of the hindsight-basse baasi methods can be used determine Sj here. While for environment where tasks share the same reward, we can simply choose the higher reward trajectories without relabeling.
We then compute the importance weights p and pV for each trajectory in Dj between the target task T, and originally sampled task T: in Line 10. We can obtain ry for task T, following the same adaptation procedure in PEARL, hence p of actions in trajectories can be computed as the ratio of sampled probabilities Ty and ,,. While for the dynamics importance weight, a straightforward way to estimate P is to predict the distribution of transition (s, a, s') ~ Dj by Pv and compute by p = (-ss a)/pp (-1s, a). However, the incorrect estimate of transition distribution in the oij original task T: may cause small value of pt. (-Is, a) and thus large value of p possibly frustrating

---

Algorithm Doubly Robust augmented Transfer (DRaT) for Meta-RL
Training tasks set (T.) from PT, informative data selection strategy S 2: Initialize replay buffers I3 and context & for each training task Ti actor To, critic Qo. and dynamics prediction network p (-|s..).
Require:
3: while meta-training iteration do
4: for each training task T: do
5: Collect trajectories using policy म, Te (alss, (,)s,.2). 24), 2 ~ (s(2)))
6: Add trajectories and action selection probability of π. to corresponding buffer B; for each task 7, do
7:
8:
Sample informative trajectory set Dj and corresponding policy TH using strategy Sj Sample context e ~ By and its latent variable 2j ~ qo(-le)): then iT Te(-Is, 2z)
9:
p' (1s..)) malals.zz) and p = max Ni.a) Pla)
10:
Compute importance weights p
*,(a)s,2,)
Compute DDR using Eq. (5) with pl and Pa (t)
11:
12:
Compute critic loss in Eq. ( using VDD as target value and update prediction network
13:
Update the lower bound ø for clipping Pa(t)
the entire training process. Instead, we estimate P as guided by the interval-based approximation for optimal P/ of DRaE in Section 3.4 which is constrained within an interval [øger .p]]. We predict the distribution of transition (s, a, s') in task Ty by prediction network p (-1s, a) and in the original task T; by Gaussian distribution (Nss', ((ss',)), .)), which indicates high probability of transition (s, a, s') E 7¹ occurring task Tr. Hence, the maximum value of p computed by (+|s, a)/N(., will automatically fall into interval [0. P with high probability, due to the large denominator resulted from the high belief we hold about the transition's 's occurrence in task T. In addition, we also clip the small value of pa using Ø'a which is updated as the average value of puar at this epoch.
The importance weights p and P are used estimate the state value in Dj by DRaE VDD in Eq. (), which is used as the target value to compute critic loss in Eq. ① Finally, the prediction updated following
network pv is updated by optimizing the prediction error, and other networks the same procedure as in PEARL. It is noted that DRaT can also be implemented on other meta- RL baselines, such as MAML [12 and its variants, where the DRaE y DR can be used to compute policy gradients for the meta-policy parameter optimization.
Experiments
In this section, we assess performance of our DRaT approach following the same evaluation procedure in [4]. For each benchmark environment, we construct a task set that is disjoint with the training task set. At the end of each training epoch, we run the evaluation procedure, where trajectories of fixed length are sampled by the adapted meta meta-policy from the updated context with continuously sampled transitions. Specifically, we conduct experiments to answer the following questions.
Can DRaT achieve a better adaptation performance on meta-test tasks in the extreme challenging reward environment with varying reward functions and dynamics, compared hindsight relabeling methods that inherently considers only the difference of reward functions?
Can DRaT effectively improve the adaptation performance on the standard meta-RL RL benchmarks where tasks share the identical dense reward function but only differ in distinct dynamics?
Will the policy importance weight p increase along the training process in consistency to the performance improvement of DRaT, which thus provides a tighter lower bound for off? Will DRaE provide better value estimate for meta-RL?
Environment setup. We compare our DRaT with other baseline algorithms on six robotic control tasks, which can be divided into two families according to their reward and dynamics setup. For the implementation detail of DRaT, please refer to Appendix B. These tasks are all simulated via MuloCo [33, where we further generate various dynamics by randomly sampling the environment parameters, including body mass, body inertia, joint damping, and body component's 's friction. One
7

---

15.0
400
500
12.5
380
250
10.0
360
Averag
344
150
5.0
103
2.5
0.0
280
0.0
0.2
0.1
2.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
255
3.0
35
0.0
0.5
3.0
3.5
2.0
2.5
Emecsep
1e6
146
(a) Point-Robon Robot-Params-- Params-Sparse
(b) Ant-ParramsSppp Sparse
(c) Slumanoid-Paramm Params-Sparse
Figure 2: Evaluation curves of average return: test-task performance vs. time steps.
Average Return
Averap Retum
Average Return
300
200-
400-
150-
202
MG.
0.0
0.5
1.0
1.5
2.2
25
0.0
0.1
0.2
0.5
0.6
05
0.5
0.0
2.2
0.6
0.5
Emmsstps
346
Urmsstpa
146
limestops
305
(a) Walker-2D-Panee PD-Paamss
(b) Cropper-Parant
(c) Point-Robot- Params
Figure 3: Evaluation curves of average return: test-task performance vs. time steps.
family contains the sparse-reward environments with varying reward functions and dynamics: 1) Point-Robot--ovd Baatoo-GGol-PPraame control the arm of 3D robot to reach random goals in the 3D space; 2) SarGGoal-Paaas--ae the perimeter of semi-circle; 3) Humanoid- Goall ParrPPrrrmsssa SSprrs:- navigate humanoid robot to reach randomly generated goals on a semi-circle. These environments are with the sparse-reward setting in the sense that the RL agent can receive a meaningful reward signal that indicates its distance from the goal only when it reaches areas near the goals. The other family includes environments that have varying dynamics and the identical dense reward function: 1) Hopper-Paraae Purams: control a planar monopod robot to hop fast possible and avoid falling; 2) Balkkr--2D-Parumml control a 2D bipedal robot to walk and perform the same task as Hopper; 3) Point-Robot- SanntRRoot-Puunnn Parbom- the same as Soint-Robot-Gin Paramas but with dense reward function. Please refer to Appendices B.2 and B.3 for the detail of varying dynamics and reward function settings, respectively.
navigate quadruped robot to reach randomly generated goals along
5.1 Evaluation on Sparse- Rearre Environments with Varying Rewards and Dynamics
To answer the first question, we conduct experiments on the first family of sparse-reward environments with varying reward functions and dynamics. We compare DRaT with four baselines: PEARL 4. Hindsight Task Relabeling (HTR) [1]. Hindsight Foresight Relabeling (HFR) [15,, and Approximate Inverse RL Relabeling (AIR) (6. Specifically, PEARL is utilized as the baseline meta-RL algorithm for all the other algorithms. For fair comparison, we keep all the byper-parameter about meta-RL RL the same as those in PEARL's source code. In HTR, we implement the single episode relabeling strategy, where hindsight task goal is selected as state that is reached in sampled episode and networks HTR are trained by transitions from that episode which is relabeled by the hindsight goal. HFR utilizes utility function (foresight) to judge the informativeness of trajectories to task, and trajectories with higher utilities in certain tasks are more likely to be transferred through the relabelling with reward function (hindsight) to this task. In AIR, trajectory is relabeled to new task, for which this trajectory beats the most previously sampled trajectories terms of the total reward among sampled candidate task set. We use the same relabeling strategy in both the AIR and our DRaT for selecting informative trajectories for each training task.
Evaluation results. We report the averaged return of trajectories sampled in the evaluation phase. From the evaluation return curves of the three tasks as shown in Figs. 2(a) 2(c) our DRaTT significantly outperforms the other baseline algorithms, especially in the Ant and Humanoid environments that have higher dimensions of states. In comparison, PEARL struggles to train the Ant and Humanoid under such challenging environment setting. PEARL-based HTR cannot even learn to move to goals for Ant in such a short period of time steps, while it achieves a similar performance to PEARL for Humanoid. PAERL-based HFR tends to outperform PEARL and other relabeling methods for both Ant and Humanoid, which is possibly because HFR considers relabeling w.r.t. performance of the adapted policy using trajectory instead of just considering the sum of relabeled rewards. We also observe that simply applying AIR into PEARL will even degrade the performance of PEARL. Providing more accurate estimate of state value functions for transferred informative trajectories, our DRa'T generally achieves a better asymptotic performance and can learn faster compared to the other baselines. Note also that in Fig. 2 the solid curve illustrates the mean return all runs of each algorithm with different random seeds, while the shaded area shows the standard deviation.
8

---

A/tipprtance woihg
A/imprraacee wight
12
1.00
230
1.00
230
354
230
(a) Point-Robot-Paee Paramms
(b) Ant-Paramss Sparse
(c) Humanoid- Parnoiddd--aa-s---- Params--parse
Figure 4: Curves of pt
policy importance weight vs. training epochs.
Eroor of Extimation
Eroor of Extimation
Eroor of Extimation
13 ".
230
20.
24.5
209
Epoch
Epoch
Epoch
(a) Point-Ro Rae--Roooottaaaaee Params
(b) Ant-Params Params-Sparse (c) Humanoid Comanoid-Paramss Params-Sparse
Figure 5: Curves of errors of value estimation: absolute difference vs. training epochs.
5.2 Evaluation on Dewse-Reward Environments with Varying Dynamics Only
Then, to answer the second question, we conduct experiments on the second family of environments with varying dynamics and the identical dense reward function. We compare DRaTT with four baselines: PEARL 4] Importance Sampling augmented Transfer (IST), Meta-Q-Learning (MQL) [16,, and Hindsight Foresight Relabeling (HFR) [14), Specifically, PEARL is utilized as the baseline meta-RL algorithm for DRaT, IST and HFR. In MQL, for a given transition and task, a fitted logistic classifier outputs probability of the transition being sampled from the task, from which the importance weight can be estimated. Transitions from other tasks are used to compute the meta-RL RL objective multiplied by this importance weight update of meta-pane parameters is judged by similarity between the new task and its source task. In IST, use the IS estimator instead of our DRaF, while keeping all the other settings the same in DRa'T.
new task, hence the contribution of each transition to the
Evaluation results. We report the averaged return of trajectories sampled the evaluation phase at each epoch, and show the evaluation return curves of the three tasks, respectively, in Figs. 3(a] 3(c) For MQL, we focus specifically on its performance with transfer by the estimated importance weights. Hence, we report its evaluation returns of adapted meta-policy with the aid of transferred samples. can be seen that with informative samples transferred by DRaT, we can improve the evaluation average return over the other baselines in all the benchmark environments. In addition, DRaT consistently accelerates the training process Walker-2D+ D-Params and Hopper-Paramn at the early stage, and also reaches the highest return in Point-Robot Param-s. In addition, DRaT presents in general smaller standard deviation.
5.3 Performance Evaluation of Proposed DRaE
We then experimentally estimate the value of p evaluate its trend during the training process of DRa'T We compute the averaged value of / estimated for informative trajectories at each training epoch, and show curves of the mean value Point-Robot-Pareee Hlumanoid-Paramne Params-Sparse in Figs. 4(a) 4(c) respectively, where the shaded area shows the standard deviation on several runs. The results show that value of p² increases with the improvement of performance on all the training tasks, since selected informative trajectories are more likely to be sampled on target task T, which can generate a tighter lower bound as we state in Section 3.4
Ant-Params-SSpa
and
At last, we evaluate the estimation error of our proposed DRaT compared to the value network Ve as used in PEARL. For better illustration, we do not plot results of directly using DR estimator, since it may result in huge variance and estimation error as we analyze in Section 3.2 We analyze the error by computing the absolute difference between the averaged estimated value and averaged true return at each training epoch, and show the error curves of Point-Robot- Roint-Robot-Params,, Ant-Params-Speel Humanoid-Parte -aa---arrmmmmsaaaee better value estimation during the training process. It is also observed that curves of DRaT and Va present similar trend, which because the computation of DRaE depends on the value of Vo.
and Figs. 5(a) respectively. The results show that our DRaE provides
Computational Complexity Analysis
Additional computational complexity is brought by DRaT for training of dynamics prediction network for each task, computation of DR estimate, and computation of relabelling for informative trajectories. Here, we analytically show that this additional computational complexity is comparable to its bascline

---

PEARL with linear scaling factor. Typically in an off- off-policy meta-RL algorithm, each epoch (i.e.. meta- veeta-taiinnng iteration) is divided into sampling and training phases. In the sampling phase. computational cost stems mainly from the actions chosen by feed-forward computation of the policy network and inference networks. Since all the algorithms (i.e., HTR, HFR, AIR, DRaT) follow the same sampling process as PEARL, we only analyze the computational cost in the training phase.
PEARL: In the training phase, the computational cost contains the feed-forward and back bropppaopng computation of policy network, value network and inference network. Given K training iterations at each epoch, batch size Ng of transitions from N training tasks, state space cardinality |S] and action space cardinality |A|, and assuming a constant computational cost of feed-forward and back- propagation computation C1, the total computational cost of training phase
CKKKN--V --s---|s-|AA
|(|-cc))))) ) as PEARL. additional computational cost brought by DRaT includes training of dynamics prediction networks, computation of DR estimator, and computation of relabelling, as follows. 1) Considering building separate prediction networks for N training tasks and the cost of feed-forward and proppaaaaatiion computation ej using the same batch of transitions for training, the additional cost of training dynamics prediction networks is also O(K- NN- N -IS] |A| ci). 2) Considering sampling informative trajectories with a maximum length of L for N training tasks and the cost C2 of DR estimation computation at each time step, the computational cost of DR estimation is O(K N L- |S| . A c2). 3) We use the approximate inverse RL relabeling (AIR), where we sample one trajectory from each training task for relabeling, leading to N candidate trajectories in total. Assuming that the cost of computing relabeled
DRaT: Besides the same training cost O(K-N(KN))))
reward at each time step is 03, the cost of relabeling is then O(K- N L |S]
[A] ea). Thus, the
additional computational complexity of DRa'T is dominated by O K N max ( (NB, L. |S| IAI
maxfe1, c2. ca)) comparable to PEARL with a linear scaling factor max (Ne.L)
N1
Insights for complexity reduction. To reduce the cost of training dynamics prediction network, we may consider training a meta-dynamics prediction network, which makes prediction for all the tasks with a single network, and thus eliminates the need of training a separate network for each task. 2) To reduce the cost of DR estimation, we may consider similar solution in TD( [D),), which makes a trade-off between (0) and Monte-Carlo estimation by using the n-step rollout and fitted network.
7 Conclusion and Limitations
In this paper, we have proposed a doubly robust augmented estimator (DRaF) to tackle the mismatch of dynamics distributions in sample transfer under a more general sparse-reward meta-RI. setting, where dynamics and reward functions can both vary across different tasks. While DRaE was established with an optimal dynamics importance weight by minimizing the MSE between estimated and true values of the value function, we proposed a tractable interval-based approximation that guaranteed to cover the optimum. We further developed doubly robust augmented transfer (DRaT) algorithm for transferring informative samples across tasks in meta-RL. Experimental evaluation on several MuJoCo locomotion tasks demonstrated the effectiveness of our proposed DRaT algorithm.
The major limitation of our method that requires computation of the DR estimate and training of the dynamics prediction network for tasks, which may incur additional computational complexity. Besides, like other hindsight- methods, the reward function is required for relabeling and selection of informative trajectories, which might be inaccessible in some extreme scenarios.
8 Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grants 62250055, 61931023, T2122024, 62125109, 61932022, 61972256, 61971285, 61831018, 62120106007, 62320106003, and 6237 1288, in part by the Program of Shanghai Science and Technol- ogy Innovation Project under Grant 20511 100100, and in part by the Shanghai Rising-Star Program under Grant
10

---

References
[1] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6679, 2020.
[2] Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62):eabk2822, 2022.
[3] Yan Duan, John Schulman, Xi Chen, Peter Bartlett, llya Sutskever, and Pieter Abbeel. RI.:: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: /61 1.02779, 2016.
[4] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy reta-reinforeemmet learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
[5] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[6] Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. Advances neural information processing systems, 33:7754-7767, 2020.
[7] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning.
In International Conference on Machine Learning, pages 652-661. PMLR, 2016
[8] Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforce- ment learning. In International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
[9] Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray. Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 1), 2013.
[10] Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive blending for policy evaluation and learning. International Conference on Machine Learning, pages 6005-6014 4. PMLR, 2019.
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018
[12] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 11 1126-1135. PMLR, 2017.
[13] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1926-5033, 2012.
[14] Charles Packer, Pieter Abbeel, and Joseph E Gonzalez. Hindsight task relabelling: Experience replay for sparse reward meta-RL. Advances in Neural Information Processing Systems, 34:2466-2477, 2021.
[15] Michael Wan, Jian Peng, and Tanmay Gangwani. Hindsight foresight relabeling for meta- reinforcement learning. In International Conference on Learning Representations, 2022.
[16] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-learning, In International Conference on Learning Representations, 2020.


Reviews:
[{"review_id": "AxoMbly2NU", "reviewer": "Reviewer_L57L", "summary": "The paper introduces Doubly Robust Augmented Transfer (DRaT), a novel approach for dealing with sparse-reward scenarios in meta-reinforcement learning (Meta-RL). DRaT transfers informative trajectories from various tasks to a target task, effectively handling dynamics mismatches and different reward functions. The authors propose an interval-based approximation to the importance weight. The DRaT algorithm is applied to an off-policy Meta-RL baseline, demonstrating superior performance over other hindsight-based methods on various sparse-reward MuJoCo tasks with different dynamics and reward functions.", "questions": "1. Is a separate dynamics prediction network fitted for each possible task 'j' in the training set?\n2. Would it be possible to test the DRaT method using the MetaWorld benchmark, which is specifically designed for meta-reinforcement learning?\n3. Could you provide a detailed analysis of the time complexity of the DRaT method to assess its scalability and efficiency?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The paper presents a novel approach, Doubly Robust Augmented Transfer (DRaT), for dealing with sparse-reward scenarios in meta-reinforcement learning. The idea of transferring informative trajectories from various tasks to a target task is an interesting direction in the field of meta-reinforcement learning.\n\n**Quality**: DRaT effectively handles dynamics mismatches and different reward functions across tasks, for the MuJoCo environments it is tested on. The method outperforms other hindsight-based methods on various sparse-reward tasks. However, without a detailed analysis of the time complexity of the algorithm or a comparison with other state-of-the-art methods in terms of computational efficiency, it's hard to fully assess the quality of the method.\n\n**Clarity**: While the paper presents complex ideas, it does so in a clear and understandable manner. The authors have done a good job of explaining their method and its benefits. However, some sections could benefit from more detailed explanations or examples, particularly for readers who are not deeply familiar with the field.\n\n**Significance**: The significance of the paper is evident in its potential impact on the field of meta-reinforcement learning. DRaT addresses a key challenge in the field - dealing with sparse-reward scenarios. By demonstrating superior performance on various tasks, the paper shows that the method could be a valuable tool for researchers and practitioners in the field. However, the real-world applicability and scalability of the method would need to be further explored to fully understand its significance.", "weaknesses": "**More Extensive Evaluation**: While the paper demonstrates the effectiveness of DRaT on various sparse-reward tasks, a more extensive evaluation would strengthen the paper's claims. The current evaluation is somewhat limited and does not fully explore the method's performance across a wide range of scenarios. The authors could consider using the [MetaWorld](https://arxiv.org/pdf/1910.10897.pdf) benchmark, a comprehensive benchmark of environments specifically designed for meta-reinforcement learning. This would provide a more rigorous testing ground for the DRaT method.\n\n**Time Complexity Analysis**: The paper does not provide detailed information on the time complexity of DRaT. Information regarding the performance with respect to time in the experiments would be valuable in assessing the method's scalability and efficiency, particularly for large-scale problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1 - Providing Time Complexity Analysis of DRaT:** Thanks for this valuable suggestion. Due to the space limit, please refer to our General Response to the Common Concern of \"Providing Time Complexity Analysis of DRaT\" for the detail discussion on the time complexity analysis of DRaT, wall-clock time comparison of all the methods, and possible strategies for further reducing the computational cost of DRaT .\n\n****\n\n**Comment 2 - More Evaluation on MetaWorld Benchmark:** We reported in this paper an extensive\nevaluation of our DRaT on MuJoCo environments, which are the commonly\nadopted benchmark to evaluate meta-RL, such as in the meta-RL baselines\nlike PEARL \\[4\\], MAML \\[12\\], and PromP \\[19\\]. For a fair comparison\nwith PEARL, and with other comparison methods that are built upon PEARL\n(e.g., HTR, HFR, AIR), we thus also adopt MuJoCo as the benchmark for\nempirical evaluations. Please also note that the testing environments\nadopted in Section 5.1 are extremely challenging with both the\nsparse-reward and varying dynamics settings, where the baseline (PEARL)\nor comparison algorithm (HTR) may even fail to learn a useful policy.\nWhile our DRaT presents a significant performance gain, demonstrating\nits effectiveness in these challenging environments.\n\nFollowing the reviewer's advice, we have also tried to test our DRaT\nusing the MetaWorld benchmark. Due to the time constraint, we managed to\nconduct experiments on a pushing task in MetaWorld, which controls a\nsimulated Sawyer arm to push a block to a specified goal position. We\nalso modify the original sparse-reward Sawyer-Push to\nSawyer-Push-Params-Sparse by generating different dynamics for each\nsampled task. We compare DRaT with HFR and PEARL in Fig. 1 of the\nadditionally uploaded PDF, which is because they show better performance\nthan other baselines in MuJoCo, and also because HFR is the only one of\nall the comparison methods that is tested in MetaWorld. Our DRaT still\npresents a better performance than HFR and PEARL, while PEARL even fails\nto improve its policy.\n\n[4] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In ICML, 2019.\n\n[12] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.\n\n[19] J. Rothfuss, D. Lee, I. Clavera, T. Asfour, and P. Abbeel. Promp: Proximal meta-policy search. In ICLR, 2019.\n\n****\n\n**Comment 3 - Concern on Dynamics Prediction Network:** As in Line 2 of our\nproposed DRaT in Algorithm 1, a separate dynamics prediction network needs to\nbe fitted for each sampled training task. Please refer to our General Response to the Common Concern\nof \"Providing Time Complexity Analysis of DRaT\" for the additional complexity\nintroduced by doing so, where we also discussed about possible strategy\nthat can be used to reduce this part of complexity, i.e., by training a\nmeta-dynamics prediction network to make prediction for all the tasks\nwith a network, and thus eliminating the need of training a separate\ndynamics prediction network for each task."}, {"review_id": "N4hiIAUzux", "reviewer": "Reviewer_cabJ", "summary": "This paper focuses on the meta-reinforcement learning setting with sparse reward. Previous work with hindsight-based sample transfer approaches requires the assumption that tasks differ only in reward functions. This paper proposes a doubly robust augmented transfer (DRaT) approach that allows both dynamics mismatches and varying reward functions across tasks. They first theoretically derive an upper bound of mean squared error between the estimated values of transferred samples and their true values in the target task. Then they propose an interval-based approximation to empirically find the optimal importance weight. In the experiment part, they implement DRaT on top of an off-policy meta-RL method and show that this method outperforms hindsight-based approaches on various sparse-reward MuJoCo locomotion tasks.", "questions": "1.\tPlease provide the missing details in the experiment as I mentioned in the weaknesses.\n2.\tWhy use PEARL as the backbone? I am not familiar with the area of meta-RL with sparse reward, but I wonder if there is any later work that outputs PEARL in the meta-RL setting. If yes, maybe more baselines should be reported.\n", "limitations": "As discussed in the conclusion section, this method requires the reward function, which may not be available in some tasks. Since this is a common setting in existing methods for sparse reward meta-RL, it is still acceptable.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well-written and easy to follow. The motivation for using doubly robust estimators to solve the sparse reward problem in meta-RL is convincing.\n2.\tThe theoretical analysis of the doubly robust augmented estimator is detailed and well-motivated.\n3.\tThe empirical implementation of the proposed algorithm outperforms other baselines by a large margin.\n", "weaknesses": "1.\tThe details about the meta-RL setting in experiments are not provided. The authors mention that they generate various dynamics by randomly sampling the environment parameters, including body mass, body inertia, joint damping, and body component friction. However, I cannot find details about the range of those parameters.\n2.\tThe authors say that “One family contains the sparse-reward environments with varying reward functions and dynamics”. I cannot find details about how the reward functions are different in the settings.  Why does “control the arm of a 3D robot to reach random goals in the 3D space” has different reward functions? \n3.\tThe algorithm requires the reward function for relabeling and selecting informative trajectories, which is similar to previous hindsight-based methods.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1- Providing Detailed Experiment Settings:**  Thanks for this\nvaluable suggestion. We will provide in the final version more details\nabout the meta-RL setting in experiments, as follows.\n\nThe randomization of dynamics on all the environments in our experiments\nare implemented by generating different environment parameters through:\n$$\nparam_{ij} = \\beta_j*initparam_{i},\n$$\nwhere $\\beta_j = A^{x_j}, x_j\\sim Uniform(-B, B)$, $A$ and $B$ are the\nconstants which control the generation of $\\beta_j$ for each environment\nparameter of task $j$, and $initparam_{i}$ is the initial value of the\n$i$-th environment parameter. Overall, these randomly sampled\nenvironment parameters include the body mass, body inertia, joint\ndamping, and body component's friction, for which the values of constant\npair $(A,B)$ are listed in Table 2 of the additionally uploaded PDF.\nWith the initial values $initparam_{i}$ loaded directly from the\noriginal file of \"mujoco_py\", the randomized environment parameter\n$param_{ij}$ is then obtained and set on the MuJoCo simulation engine to\ngenerate various environment dynamics.\n\nPlease note that since the robot models in our testing environments are\ncomplicated with a large number of environment parameters, these initial\nvalues of $initparam_{i}$ are usually stored as high-dimensional\narrays in the \"mujoco_py\" file. Due to space limit, we only show the\ninitial values set for the Ant-Params-Sparse environments in Fig. 2 of\nthe additionally uploaded PDF as an example. For more details about\nsettings of these initial values, please refer to either the \"mujoco_py\"\nfile that is publicly available, or our source code that will be\nreleased shortly after the code optimization.\n\n****\n\n**Comment 2 - Using PEARL as Backbone:** We adopted PEARL as the off-policy\nmeta-RL backbone in this paper, mainly because it is a sample-efficient\nmeta-RL algorithm, which significantly outperforms the other baseline\nmeta-RL algorithms, such as MAML \\[12\\], PromP \\[19\\] and RL2 \\[3\\] on\nthe standard meta-RL testing benchmark. This also explains why the\nfollowing works that further tackle the sparse-reward challenge in\nmeta-RL are mostly built upon the PEARL backbone, such as the Hindsight\nTask Relabeling (HTR) \\[14\\] and Hindsight Foresight Relabeling (HFR)\n\\[15\\]. These methods have demonstrated a superiority over PEARL in the\nsparse-reward setting, and also were compared with our DRaT in the\nexperiments section.\n\n\n[3] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.\n\n[12] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.\n\n[14] C. Packer, P. Abbeel, and J.h E Gonzalez. Hindsight task relabelling: Experience replay for sparse reward meta-RL. In NeurIPS, 2021.\n\n[15] M. Wan, J. Peng, and T. Gangwani. Hindsight foresight relabeling for meta-reinforcement learning. In ICLR, 2022.\n\n[19] J. Rothfuss, D. Lee, I. Clavera, T. Asfour, and P. Abbeel. Promp: Proximal meta-policy search. In ICLR, 2019.\n\n****\n\n**Comment 3 - Details about Settings of Different Reward Functions:** Thanks\nfor pointing out this issue. In Section B.2 of the Supplementary\nMaterial, we had provided the reward function settings for the\nsparse-reward environments with varying rewards and dynamics, while the\nreward function settings for the dense-reward environments were stated\nas using the same implementation as in PEARL's open-sourced code. In the\nfollowing, we will provide further explanation to make the settings of\ndifferent reward functions clearer.\n\nThe reward function defined for the Point-Robot environments, which\ncontrol the arm of a 3D robot to reach a random goal position in the 3D\nspace, is given as: $$reward=\n\t\t\\begin{cases}\n\t\t\t-dist(robot,goal)+C  &     & if\\ dist(robot,goal)<D,\\\\\\\\\n\t     0  &     & otherwise.\\\\\\\\\n\t\t\\end{cases} $$ We set $C=0.0$, $D=+\\infty$ for the dense-reward\nfunction in Point-Robot-Params, where the reward signal stands for the\ndistance between the tip of robot arm and the goal position, which can\nalways be accessed. We set $C=1.0$ and $D=0.2$ for the sparse-reward\nfunction in Point-Robot-Params-Sparse, where the RL agent will get a\nmeaningful reward signal that indicates the distance to the goal only\nwhen the robot arm is close to its goal within a small range. In both\ncases, goal positions are randomly generated in the 3D space, and hence\nthe reward functions are also varying to indicate a distinct distance to\ndifferent goal positions.\n\n****\n\n**Comment 4 - Concern on Requiring Reward Function for Relabeling and\nSelecting Informative Trajectories:** Please note that the commonly used\nhindsight-based methods, such as HTR \\[14\\] and AIR \\[6\\], simply use\nthe reward function to relabel samples without accommodating to the\ndynamics mismatch across tasks. Our experiments in Section 5.1\ndemonstrated such an inefficiency of HTR and AIR on the extremely\nchallenging environments with both sparse-reward and varying dynamics\nsettings.\n\nDifferent from them, we design a doubly robust augmented estimator\n(DRaE) to accommodate to the more general and rational meta-RL setting\nwith both sparse-reward and varying dynamics across different tasks.\nDRaE can tackle the mismatch of dynamics distributions in meta-RL with a\nguaranteed optimum for the dynamics importance weight by minimizing MSE\nbetween the estimated and true values of the value function. Using DRaE\nfor a better value estimation of transferred samples, our proposed DRaT\nalgorithm demonstrated its superiority on several meta-RL testing\nbenchmarks, as verified by the experiments in Sections 5.1 and 5.2."}, {"review_id": "oCk6EfHMYE", "reviewer": "Reviewer_xMYP", "summary": "The paper identifies that some existing approaches ignore  varying dynamics in meta-RL and proposes a new method for addressing this. In particular, the paper focuses on minimizing the mean-squared error of value functions and showed that doubly robust estimators suffer from a high-variance problem. Consequently, the paper proposes the doubly robust augmented estimator to estimate the importance weight of the dynamics, that reduces variance at the cost of introducing bias. The paper further identifies the intervals for which the importance weight should lie in, in order to maintain low variance. Finally, the paper provides empirical analysis to demonstrate that the proposed approach yields smaller value prediction error than existing baseline, and demonstrated that the proposed method can outperform existing baselines in certain control environments.", "questions": "- The appendix mentions that the proofs really follow through by assuming that $\\gamma \\rho_{\\pi}^{ij}(t) \\tilde{V}_{ij}^{DR} (s)$ is not correlated to other expectation terms. I am wondering why this is a reasonable assumption---in particular, it is not immediately obvious that this term does not dominate the expectation terms.\n- How does different $\\sigma$ for approximating $N(s', \\sigma)$ affect the estimation? Is there a sensitivity analysis regarding this? My expectation is that very small $\\sigma$ would cause stability problem, while very high $\\sigma$ would prevent learning. Is there an informed way to select this?\n", "limitations": "The paper has listed few limitations of the proposed approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is well-written and easy to follow in general.\n- Visualizations that supports various claims regarding the behaviours of the estimators---the paper demonstrate the proposed estimator has lower error compared to existing approach, and that the policy importance sampling weights do increase to approximately $1$ as informative trajectories are sampled more frequently.\n- The paper demonstrated that the proposed method is theoretically sound (under some assumptions) and its empirical performance is comparable (if not better) to existing baselines.", "weaknesses": "## Major Comments\n- Figure 3 suggests that with same reward function but different dynamics, the current baseline is already fairly robust? Does this really suggest that in figure 2, the problem is stemming from sparse reward?\n\t- This may have stemmed from a confusion. For section 5.2, page 9, line 329, the paper mentions that \"..., while keeping all the other settings the same as in DRaT.\" Does that mean other approaches are now using DRaE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1 - Concern on Figures 2 and 3:**\n\nPlease note that though compared to Fig. 2, the current baselines (e.g.,\nPEARL \\[4\\] and HFR \\[15\\]) in Fig. 3 may have a smaller performance gap\nwith our DRaT, they still suffer from a larger standard deviation of\nperformance (as indicated by the larger shaded area in Figure 3). Such a\nlarger standard deviation of performance would still affect their\nrobustness. In comparison, our DRaT presents in general a smaller\nstandard deviation of performance in Fig. 3.\n\nIn comparison, the environments in Fig. 2 present an extremely\nchallenging scenario with both the sparse-reward and varying dynamics\nsettings. Specifically, the problem stemmed from sparse-reward is that\nthe robot can only get reward signals indicating its distance to the\ngoal position when it gets close to the goal position within a small\nrange. On the other hand, the dynamics of this robot may also vary by\nsetting different values of environment parameters, including the body\nmass, body inertia, joint damping, and body component's friction. As\nshown in Fig. 2, both the sparse-reward and varying dynamics settings\nbring challenges to the training of meta-RL algorithms.\n\n****\n\n**Comment 2 - Confusion on Using DRaE in Other Approaches:** There might be\nsome misunderstandings here. The original sentence in Line 329, Page 9\nis: \"In IST, we use the IS estimator instead of our DRaE, while keeping\nall the other settings the same as in DRaT.\"\n\nPlease note that this Importance Sampling augmented Transfer (IST)\nmethod is simply a variant of our Doubly Robust augmented Transfer\n(DRaT) method, with the same hyperparameter settings for network\ntraining and task relabeling as in DRaT, except for using the Importance\nSampling (IS) estimator instead of the proposed Doubly Robust augmented\nEstimator (DRaE). While for the other comparison approaches, they still\nused their original value networks for value estimation, and did not\nutilize our proposed DRaE.\n\n\n****\n\n**Comment 3 - Assumption on  $\\gamma \\rho^{ij}\\_{\\pi}(t) \\tilde{V}^{DR}\\_{ij}(s\\_{t+1}) $ Loosely\nCorrelated to Other Expectation Terms inside the Parentheses in Appendix\nA.7:** Thanks for pointing out this issue. In the following, we will\nbriefly explain why this assumption is made, which will be further\nincluded in the final version.\n\nThe original equation in Appendix A.7, after letting the first-order derivative of the objective function be zero, is given by:\n\n\\begin{align*}\n    &2\\mathbb E\\_t \\Bigg[ \\Big(\\gamma \\rho^{ij}\\_{\\pi}(t) \\Big)^2 \\tilde{V}^{DR}\\_{ij}(s\\_{t+1}) \\Big(\\hat{\\rho}^{ij}\\_d(t) \\tilde{V}^{DR}\\_{ij}(s_{t+1}) - {\\rho}^{ij}\\_d(t) V^{DR}\\_{ij}(s_{t+1}) \\Big)\\Bigg] \n    \\\\\\\\\n&+  2\\mathbb{E}\\_t \\Bigg[ \\gamma \\rho^{ij}\\_{\\pi}(t)   \\tilde{V}^{DR}\\_{ij}(s_{t+1}) \\Bigg( \\rho^{ij}\\_{\\pi}(t)  \\gamma \\left( \\hat{\\rho}^{ij}\\_d(t) \\tilde{V}^{DR}\\_{ij}(s\\_{t+1}) -\\mathbb{E}\\_{t+1}[V^j(s\\_{t+1})] \\right)  - \\rho^{ij}\\_{\\pi}(t)\\Delta(s\\_t,a\\_t) +\\bar{V}\\_{\\theta}(s\\_t,z\\_j)\\Bigg) \\Bigg] = 0.\n\\end{align*}  \nBy eliminating the constant of 2 and further merging the two expectations on the left-hand side into one expectation, we have:\n\\begin{align*}\n     \\mathbb E\\_t \\Bigg[ \\gamma \\rho^{ij}\\_{\\pi}(t) \\tilde{V}^{DR}\\_{ij}(s\\_{t+1}) & \\cdot \\Bigg(2\\gamma \\rho^{ij}\\_{\\pi}(t)\\hat{\\rho}^{ij}\\_d(t) \\tilde{V}^{DR}\\_{ij}(s\\_{t+1})\n    \\\\\\\\\n &- \\gamma \\rho^{ij}\\_{\\pi}(t){\\rho}^{ij}\\_d(t) V^{DR}\\_{ij}(s_{t+1}) -\\gamma \\rho^{ij}\\_{\\pi}(t)\\mathbb{E}\\_{t+1}[V^j(s\\_{t+1})]  - \\rho^{ij}\\_{\\pi}(t)\\Delta(s\\_t,a\\_t) +\\bar{V}\\_{\\theta}(s\\_t,z\\_j)\\Bigg)\\Bigg] =0. \n\\end{align*} \nNote that inside expectation on the left-hand side is the multiplication of $\\gamma \\rho^{ij}\\_{\\pi}(t) \\tilde{V}^{DR}\\_{ij}(s\\_{t+1}) $ with a summation enclosed by the parentheses, which can be rewritten as: \n\n\\begin{align*}\n&\\Bigg(2\\gamma \\rho^{ij}\\_{\\pi}(t)\\hat{\\rho}^{ij}\\_d(t) \\tilde{V}^{DR}_{ij}(s\\_{t+1}) -\\gamma \\rho^{ij}\\_{\\pi}(t){\\rho}^{ij}\\_d(t) V^{DR}\\_{ij}(s\\_{t+1}) - \\rho^{ij}\\_{\\pi}(t)\\Delta(s\\_t,a\\_t) +\\bar{V}\\_{\\theta}(s\\_t,z\\_j)\\Bigg)\n\\\\\\\\\n=& \\gamma \\rho^{ij}\\_{\\pi}(t) \\Bigg(2\\hat{\\rho}^{ij}\\_d(t) \\tilde{V}^{DR}\\_{ij}(s\\_{t+1}) - {\\rho}^{ij}\\_d(t) V^{DR}\\_{ij}(s\\_{t+1}) -\\mathbb{E}\\_{t+1}[V^j(s\\_{t+1})] \\Bigg)  - \\rho^{ij}\\_{\\pi}(t)\\Delta(s\\_t,a\\_t) +\\bar{V}\\_{\\theta}(s\\_t,z\\_j).\n\\end{align*}\n\nOn the right-hand side of this equation, the first three terms are closely correlated to $\\gamma \\rho^{ij}\\_{\\pi}(t) \\tilde{V}^{DR}\\_{ij}(s\\_{t+1}) $, while the last two terms are loosely correlated to it. Furthermore, given that $\\hat{\\rho}^{ij}\\_d(t)$ is inside an interval upper-bounded by its true value ${\\rho}^{ij}\\_d(t)$, the value of $\\hat{\\rho}^{ij}\\_d(t) \\tilde{V}^{DR}\\_{ij}(s_{t+1})$ is comparable to those of ${\\rho}^{ij}\\_d(t) V^{DR}\\_{ij}(s\\_{t+1})$ and $\\mathbb{E}\\_{t+1}[V^j(s\\_{t+1})] $. Thus, the first three terms will be compensated by each other, while value of the last two terms $ - \\rho^{ij}\\_{\\pi}(t)\\Delta(s\\_t,a\\_t) +\\bar{V}\\_{\\theta}(s\\_t,z\\_j)$ will dominate, which is loosely correlated to $\\gamma \\rho^{ij}\\_{\\pi}(t) \\tilde{V}^{DR}\\_{ij}(s\\_{t+1}) $.\n\n****\n\n**Comment 4 - Sensitivity Analysis of $\\sigma$:** Thanks for this\nvaluable suggestion. As will be included in the final version, we\nconduct experiments on the sensitivity analysis of $\\sigma$ for the\nAnt-Params-Sparse environments. By varying the values of $\\sigma$,\nwe plot the curves of returns in Fig. 3 of the additionally\nuploaded PDF. As expected, a smaller value of $\\sigma$ (e.g., when\n$\\sigma=0.1$) would lead to a performance degradation at the initial\ntraining stage, but this performance gap will be reduced as the training\nproceeds. On the other hand, a larger value of $\\sigma$ (e.g., when\n$\\sigma > 2$) may incur a very high value for the value estimation and\nthus prevent the learning, the training curves of which are thus not\nreported in Fig. 3 of the additionally uploaded PDF.\n\n"}, {"review_id": "NYKnQFC6np", "reviewer": "Reviewer_gRwA", "summary": "This study introduces the DRaT (Doubly Robust Augmented Transfer) algorithm, an advanced extension of hindsight-based transfer methods, which tackles not only reward mismatches but also discrepancies in transition dynamics. The authors provide a theoretical analysis to establish the optimality of their interval-based approximation, DRaE (Doubly Robust Augmented Estimator), in calculating the optimal importance sampling weight. Through empirical evaluation, it is demonstrated that DRaT surpasses conventional hindsight-based methods in performance, particularly in sparse-reward MuJoCo tasks that involve varying reward structures and transition dynamics.", "questions": "**Q1.** Results in Meta-RL are typically characterized by a high degree of variability, necessitating the aggregation of multiple runs for reliable evaluation. However, it appears that the number of seeds used for evaluation is not reported in the manuscript or the appendix. Could the authors clarify how many seeds were employed and consider including this information for a more transparent and rigorous evaluation?\n\n**Q2.** Given the computational overhead associated with the proposed method, as discussed earlier, are there any strategies or optimizations that the authors have considered to enhance its computational efficiency? Insights into potential avenues for reducing computational costs without compromising effectiveness would be valuable.\n\n**Q3.** Contrary to the checklist, I think the authors didn't submit the code or link to their codes. Any plans for releasing the code?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**S1.** Addressing a Critical Yet Underexplored Aspect of Transfer Learning\n\nThis study tackles an essential but often overlooked aspect of transfer, specifically in the context of varying transition dynamics. The authors have put forth commendable efforts in highlighting the significance of calculating the optimal dynamics importance weight, which is a critical component in transfer under changing transition dynamics.\n\n**S2.** Solid Theoretical Foundation and Novel Approximation Method\n\nThe work presents a robust theoretical underpinning for the proposed method, ensuring reliability and efficacy. The use of a tractable interval-based approximation is innovative and enhances the practicality of the approach. The proof confirming that the mean squared error (MSE) is guaranteed to encompass the optimum is both rigorous and convincing.\n\n**S3.** Thorough Empirical Evaluation in Diverse Scenarios\n\nThe authors have conducted a comprehensive empirical evaluation by testing their method in three environments with sparse rewards and three with varying state dynamics. Such thorough testing provides a more complete picture of the algorithm's capabilities. Impressively, the proposed method demonstrates substantial and consistent improvements over all baselines, attesting to its effectiveness and robustness.", "weaknesses": "**W1.** Computational Overhead\n\nA notable drawback of this study, which the authors have acknowledged in the limitations section, is the computational burden associated with calculating the doubly robust estimate and training the dynamics network. Given that practicality is an essential factor in the real-world application of algorithms, it is crucial to consider computational efficiency. It would be beneficial if the authors could include a comparison of the wall-clock time for each method, as this would provide valuable insights into the trade-offs between performance gains and computational costs.\n\n**Acknowledgment Following Rebuttal**\n\nI have reviewed the author's response. The response addressed my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1 - Computational Overhead Analysis and Wall-Clock Time\nComparison:**  Thanks for this valuable suggestion. Due to the space limit, please refer to our General Response to the Common Concern of \"Providing Time Complexity Analysis of DRaT\", for the detail discussion on the computational overhead analysis of our DRaT and wall-clock time comparison of all the methods.\n\n****\n\n**Comment 2 - Strategies to Further Reduce Complexity of DRaT:** Thanks\nfor this valuable suggestion. In the final version, we will provide some\ninsights into potential avenues for reducing computational costs of our\nproposed DRaT, as follows.\n\nHere, we discuss about some possible strategies for further reducing the\ncomputational cost of DRaT without compromising too much of the\neffectiveness. For example, to reduce the computational cost of training\ndynamics prediction network for each training task, we may consider\ntraining a meta-dynamics prediction network, which can make prediction\nfor all the tasks with a single network, and thus eliminate the need of\ntraining a separate dynamics prediction network for each task. To also\nreduce the computational cost of DR estimation, we may consider a\nsimilar solution in TD($n$), which makes a trade-off between TD($0$) and\nMonte-Carlo estimation by using the $n$-step rollout and fitted network.\n\n\n****\n\n**Comment 3 - Choice of Random Seeds:** Thanks for this valuable\nsuggestion. We will report in the final version that the results of each\nalgorithm are averaged across 3 random seeds, which is also a common\nsetting in the baseline meta-RL algorithms, such as PEARL \\[4\\] and\nPromP \\[19\\]. Please also note that the evaluation curves reported in\nthis paper are obtained by evaluating on a test task set that contains\ndozens of tasks, which already reduces the randomness of evaluation in\nthe meta-RL setting. To validate this, we further perform an experiment\ntrial with 6 random seeds in the Ant-Params-Sparse and Humanoid-Params-Sparse environments. We\nthen split the trail into two groups of 3 random seeds and average these\ntwo groups together. As shown in Fig. 4 of the additionally uploaded\nPDF, there is no significant difference of performance between these two\ngroups.\n\n\n****\n\n**Comment 4 - Plans for Releasing the Code:** We are now optimizing our\nsource code, and will shortly release it right after we finish the code optimization."}]

Author rebuttal:
**General Response:** We would like to thank all the reviewers for their helpful comments. Here, we will respond to the common concern on the time complexity of our proposed DRaT. For orther concerns, please see below our responses to each reviewer’s individual comments, where the newly added figures and tables can be found in our additionally uploaded PDF attached at the end of this General Response.

****

**Common Concern - Providing Time Complexity Analysis of DRaT:** As had been acknowledged in the limitations
section, additional computational complexity would be brought by our
DRaT for computation of DR estimate and training of dynamics prediction
network for each task. Here, **1)** we analytically show that this
additional complexity is comparable to its baseline PEARL's
computational complexity, with a linear scaling factor of
$\frac{\max(N_{\mathcal{B}}, L) }{N_{\mathcal{B}}} \cdot \frac{\max(c_1, c_2, c_3)}{c_1}$.
**2)** We then empirically verify this analysis by demonstrating that
the wall-clock time of DRaT is $1.3\times \sim 1.7\times$ that of PEARL,
and compare the wall-clock time of other state-of-the-art methods.
**3)** We also discuss about some possible strategies for further
enhancing DRaT's computational efficiency. These analyses will be
included in the final version.


**1) Complexity Analysis.**


Typically in an off-policy meta-RL algorithm, each epoch (i.e., meta-training iteration) is divided into the sampling and training phases. In the sampling phase, the computational cost stems mainly from the actions chosen by feed-forward computation of policy network and inference network. Since all the algorithms (i.e., HTR, HFR, AIR, DRaT) follow the same sampling process as PEARL, we only analyze the computational cost in the training phase.

**PEARL's complexity:** In the training phase, the computational cost contains mainly the feed-forward and back-propagation computation of policy network, value network and inference network. Given $K$ training iterations at each epoch, batch size $N_{\mathcal{B}}$ of transitions from $N$ training tasks, state space cardinality $\vert \mathcal{S} \vert$ and action space cardinality $\vert \mathcal{A} \vert$, and assuming a constant computational cost of feed-forward and back-propagation computation $c_1$, the total computational cost of training phase is $O(K \cdot N_{\mathcal{B}} \cdot N \cdot \vert \mathcal{S} \vert \cdot \vert \mathcal{A} \vert \cdot c_1 )$.

**DRaT's complexity:** Besides the same training cost $O(K \cdot N_{\mathcal{B}} \cdot N \cdot \vert \mathcal{S} \vert \cdot \vert \mathcal{A} \vert \cdot c_1 )$ as PEARL, additional computational cost brought by DRaT includes training of dynamics prediction networks, computation of DR estimator, and computation of relabelling.

- Considering building separate prediction networks for $N$ training tasks and the cost of feed-forward and back-propagation computation $c_1$, using the same batch of transitions for training, the additional cost of training dynamics prediction networks is also $O(K \cdot N_{\mathcal{B}} \cdot N \cdot \vert \mathcal{S} \vert \cdot \vert \mathcal{A} \vert \cdot c_1 )$.

- Considering sampling informative trajectories with a maximal length of $L$ for $N$ training tasks and the cost $c_2$ of DR estimation computation at each time step, the computational cost of DR estimation is $O(K \cdot N \cdot L \cdot  \vert \mathcal{S} \vert \cdot \vert \mathcal{A} \vert \cdot c_2)$.

- In this paper, we used the approximate inverse RL relabeling (AIR). We sample one trajectory from each training task for relabeling, leading to $N$ candidate trajectories in total. Assuming the cost of computing relabeled reward at each time step is $c_3$, the computational cost of relabeling is $O(K \cdot N \cdot L \cdot \vert \mathcal{S} \vert \cdot \vert \mathcal{A} \vert \cdot c_3)$.


Incorporating them together, we conclude that the additional computational complexity of DRaT is dominated by $O\Big(K \cdot N \cdot  \max ( N_{\mathcal{B}}, L ) \cdot \vert \mathcal{S} \vert \cdot \vert \mathcal{A} \vert \cdot \max ( c_1, c_2, c_3 ) \Big)$, which is comparable to PEARL with a linear scaling factor of $\frac{\max(N_{\mathcal{B}}, L) }{N_{\mathcal{B}}} \cdot \frac{\max(c_1, c_2, c_3)}{c_1}$.

**2\) Wall-Clock Time Comparison.**

To empirically verify the above complexity analysis, in Table 1 of the
additionally uploaded PDF, we also compare the wall-clock time for DRaT
and all the comparison methods, including PEARL, HTR, HFR, AIR. It can
be seen that for these robotic control tasks at different scales, DRaT
generally consumes $1.3\times \sim 1.7\times$ running time of PEARL,
which is also comparable to other baselines.

**3\) Strategies to Further Reduce Complexity.**

Here, we discuss about some possible strategies for further reducing the
computational cost of DRaT without compromising too much of the
effectiveness. For example, to reduce the computational cost of training
dynamics prediction network for each training task, we may consider
training a meta-dynamics prediction network, which can make prediction
for all the tasks with a single network, and thus eliminate the need of
training a separate dynamics prediction network for each task. To also
reduce the computational cost of DR estimation, we may consider a
similar solution in TD($n$), which makes a trade-off between TD($0$) and
Monte-Carlo estimation by using the $n$-step rollout and fitted network.

****

Meta-review: The review scores are 7, 7, 7, 6, with an average score of 6.75. The reviewers’ comments are generally positive. The authors have given a detailed response to the questions and comments, and the reviewers have positively responded to the comments. 

For the final version, please make sure that you include the comments and suggestion by the reviewers. 

Regards,

AC
Decision: Accept (poster)