Title: Doubly Robust Augmented Transfer for Meta-Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Meta-reinforcement learning

doubly robust (DR)

sample transfer
Abstract: Meta-reinforcement learning (Meta-RL), though enabling a fast adaptation to learn new skills by exploiting the common structure shared among different tasks, suffers performance degradation in the sparse-reward setting. Current hindsight-based sample transfer approaches can alleviate this issue by transferring relabeled trajectories from other tasks to a new task so as to provide informative experience for the target reward function, but are unfortunately constrained with the unrealistic assumption that tasks differ only in reward functions. In this paper, we propose a doubly robust augmented transfer (DRaT) approach, aiming at addressing the more general sparse reward meta-RL scenario with both dynamics mismatches and varying reward functions across tasks. Specifically, we design a doubly robust augmented estimator for efficient value-function evaluation, which tackles dynamics mismatches with the optimal importance weight of transition distributions achieved by minimizing the theoretically derived upper bound of mean squared error (MSE) between the estimated values of transferred samples and their true values in the target task. Due to its intractability, we then propose an interval-based approximation to this optimal importance weight, which is guaranteed to cover the optimum with a constrained and sample-independent upper bound on the MSE approximation error. Based on our theoretical findings, we finally develop a DRaT algorithm for transferring informative samples across tasks during the training of meta-RL. We implement DRaT on an off-policy meta-RL baseline, and empirically show that it significantly outperforms other hindsight-based approaches on various sparse-reward MuJoCo locomotion tasks with varying dynamics and reward functions.

Doubly Robust Augmented Transfer for Meta-Reinforcement Learning

Abstract

Meta-reinforcement learning (Meta-RL), though enabling a fast adaptation to learn new skills by exploiting the common structure shared among different tasks, suffers performance degradation in the sparse-reward setting. Current hindsight-based sample transfer approaches can alleviate this issue by transferring relabeled trajectories from other tasks to a new task so as to provide informative experience for the target reward function, but are unfortunately constrained with the unrealistic assumption that tasks differ only in reward functions. In this paper, we propose a doubly robust augmented transfer (DRaT) approach, aiming at addressing the more general sparse reward meta-RL scenario with both dynamics mismatches and varying reward functions across tasks. Specifically, we design a doubly robust augmented estimator for efficient value-function evaluation, which tackles dynamics mismatches with the optimal importance weight of transition distributions achieved by minimizing the theoretically derived upper bound of mean squared error (MSE) between the estimated values of transferred samples and their true values in the target task. Due to its intractability, we then propose an interval-based approximation to this optimal importance weight, which is guaranteed to cover the optimum with a constrained and sample-independent upper bound on the MSE approximation error. Based on our theoretical findings, we finally develop a DRaT algorithm for transferring informative samples across tasks during the training of meta-RL. We implement DRaT on an off-policy meta-RL baseline, and empirically show that it significantly outperforms other hindsight-based approaches on various sparse-reward MuJoCo locomotion tasks with varying dynamics and reward functions.

1 Introduction

Reinforcement learning (RL) has achieved a remarkable success in a variety of sequential decision making tasks, such as intelligent gaming [1] and robotic control [2]. Agents trained by conventional RL methods aim at learning a single task, thus fail to adapt quickly to a new task with prior experience. In contrast, meta-reinforcement learning (meta-RL) focuses on learning to learn , i.e., learning how to adapt [3, 4]. In essence, it learns the underlying common structure from experience collected across a set of tasks, and then exploits this structure to fast adapt to similar new tasks with only a few trials.

Currently, one of the main challenges in meta-RL is the sparse-reward setting, which exists widely in real-world environments. When rewards become sparse, the agent receives only scarce information related to the task, bringing extreme difficulty to the meta-training and adaptation processes. To collect more useful experience, sample transfer approaches have been proposed. The key idea behind is to learn a new task by selectively leveraging samples generated from other tasks. Ideally, samples transferred from an original task to the new target task are required to be informative trajectories.

Namely, the informative trajectories should present a higher and denser reward in the target task, though they might be unsuccessful attempts (i.e., with lower and sparser reward) in the original task.

Following the direction of sample transfer, hindsight experience replay (HER) and its variants [5, 6] have been developed as the typical methods in literature. Hindsight-based methods relabel a trajectory collected from the original task to a target task as informative trajectory, where rewards of this trajectory are re-computed using the target reward function to teach it how to achieve a different goal in the new task. However, HER algorithms are inherently designed for tasks that only differ in the reward function, while dynamics of all the different tasks are assumed to be identical. It is worth mentioning that in many real-world scenarios, tasks may differ not only in reward functions, but also in dynamics. For example, a robot is often trained under various working conditions, such as different load mass and joint damping, which may result in different distributions of dynamics. We are thus motivated to consider a more general meta-RL scenario where both dynamics and reward function can be different and varying across tasks, and study how to further cope with such a mismatch of dynamics distributions. This brings new challenges to existing meta-RL baselines, since consideration of only dynamics difference with the identical dense-reward setting fails to handle multi-goal tasks, while considering only the reward difference may misjudge values of transferred trajectories.

In off-policy RL, doubly robust (DR) estimators [7, 8] have emerged as an effective method to correct the distribution mismatch between a target policy and a behavior policy. They incorporate importance sampling with a fitted model of value function to alleviate the high variance issue of the original importance sampling-based estimator. In meta-RL, however, the discrepancy between the dynamics of the original and target tasks (as represented by the importance weight of dynamics) is practically inaccessible to the agent. This indicates that popular methods, such as directly clipping or removing large importance weight values [9, 10], proposed for providing a more accurate evaluation to enable efficient sample transfer and thus stable training, become infeasible in meta-RL.

In order to control the values of dynamics importance weights, we formulate the sample transfer as a mean square error (MSE) minimization problem and solve for the optimal dynamics importance weight. Since this optimal weight is intractable in practice, we propose an approximation strategy that constrains the estimated weight within an interval containing this optimal weight, and theoretically show that any estimate outside this interval would inevitably increase the MSE. Finally, we present a policy learning algorithm for sample transfer-based meta-RL, in which the proposed weight estimate is used for value-function evaluation. Our main contributions can be summarized as follows.

Doubly Robust augmented Estimator . We design a doubly robust augmented estimator (DRaE), which for the first time accommodates to the more general and rational meta-RL setting and simultaneously allows varying dynamics and reward functions across different tasks. DRaE tackles the mismatch of dynamics distributions in meta-RL with a guaranteed optimum for the dynamics importance weight by minimizing MSE between the estimated and true values of the value function.

Theoretically Robust Interval Approximation . We propose a tractable interval-based approximation for the optimal dynamics importance weight derived by our DRaE. This interval is deterministic to balance the variance and bias for a decreasing MSE and guaranteed to cover the optimum. We further verify that the proposed approximation is robust in theory with a constrained and sampleindependent upper bound on the MSE approximation error from the optimum.

Doubly Robust augmented Transfer Algorithm . We develop a doubly robust augmented transfer (DRaT) algorithm for transferring informative samples across tasks, based on the proposed DRaE. DRaT controls the importance weight under the guide of our interval-based approximation to minimize the MSE. It is noted that DRaT can be integrated into any value-based meta-RL baselines and implemented specifically on an off-policy meta-RL baseline, PEARL [4].

2 Preliminary

2.1 Meta-Reinforcement Learning (Meta-RL)

Meta-RL considers training on a task set {T i } to learn a set of meta-parameters, which can quickly adapt to solving a new (testing) task with only a limited number of samples. Each task T i then implies a Markov decision process (MDP) with both a distinct reward function r i and a different transition probability distribution p i ( ·| s, a ) . Such a meta-RL objective can be written as: max θ E P T [ J T ( π θ T i ) ] , where P T denotes the sampling distribution over the training task set T , J T ( π θ T i ) is the expected

reward of policy π θ T i on the task T i , and the policy π θ T i is generated from π θ following a certain adaptation procedure given samples c i = { s t , a t , r t , s t +1 } from this task.

We use PEARL [4] specifically in this paper as the baseline meta-RL algorithm, which builds upon the soft actor-critic (SAC) [11]. Nevertheless, our proposed DRaT approach is compatible with and can be applied generally to any other meta-RL baselines. PEARL implements the adaptation by constructing an inference network q ϕ ( z | c ) , which takes in recently collected samples c of a task (also referred to as the context) and infers a latent context variable z that encodes salient information about this task. Then, a policy π θ ( a | s, z ) conditioned on z can adapt its behavior to this task. Both the inference network q ϕ ( z | c ) and critic Q θ ( s, a, z ) are trained with the following critic loss:

where ¯ V θ is the target value network and the overline notation specifies no gradient backpropagation going through ¯ V θ and ¯ z . While the policy is trained using the same actor loss as in SAC [11].

2.2 Doubly Robust Estimator for Off-Policy Policy Evaluation

In off-policy policy evaluation, we expect to estimate the state value of a target policy π given data sampled from a behavior policy µ , where distribution mismatch stems from varying polices. Importance sampling (IS) estimators can correct this mismatch by multiplying the importance weights of policies, thus providing an unbiased estimate: V IS ( s t = s ) = [ ∏ T -1 k = t ρ π ( k ) ] · [ ∑ T -t -1 k =0 γ k r ( s t + k , a t + k ) ] , where ρ π ( k ) = π ( a k | s k ) µ ( a k | s k ) denotes the importance weight between the target policy π and behavior policy µ at the step k . Unfortunately, the IS estimator suffers from high variance due to possible high values of the policy importance weights, while the product of policy importance weights will also grow exponentially with the horizon [7]. Through incorporating a fitted value function into the IS estimator, this variance can be reduced by doubly robust (DR) estimators, which approximate the value-function at s t recursively as:

where ˆ V ( s ) = E a ∼ π ( ·| s ) [ ˆ Q ( s, a ) ] is computed by the fitted Q -function, i.e., ˆ Q . Here, 'doubly robust' refers to that the estimated value is unbiased when either one of the ρ π and ˆ Q is an unbiased estimate.

3 Doubly Robust Augmented Estimator (DRaE) for Sample Transfer

In this section, we consider leveraging the sample transfer between tasks in a more general sparsereward meta-RL setting, where both the reward functions and dynamics are varying across different tasks. Specifically, we formulate the sample transfer as a mean square error (MSE) minimization problem between the estimate value of samples transferred from other tasks and their true values in the target task. Other than the simple transfer of relabeled samples in hindsight-based methods without accommodating to the dynamics mismatch, solution to the proposed sample transfer problem can provide the true value estimate of transferred samples for the subsequent tasks in meta-training, which implicitly incorporates the differences of both reward functions and dynamics models.

We then show that a direct use of DR will bring additionally high variance incurred by environment dynamics, since the importance weight of dynamics might be large due to various dynamics. Though the unbiasedness of DR estimate is appealing, its high variance will cause a large MSE on the estimated value, which significantly disturbs the meta-training process. Since MSE can be decomposed into the sum of bias and variance-related terms, we thus propose to minimize a derived upper bound of MSE and determine the optimal dynamics importance weight, aiming at balancing the bias and variance. Finally, to overcome the intractability of solving this MSE minimization in practice, we propose instead an interval-based approximation for the optimal importance weight, by showing that any other values outside this interval will enlarge both the bias and variance, thus with a larger MSE.

3.1 Problem Formulation of Sample Transfer in Meta-RL

In meta-RL, a training task set T can be characterized by tuples {T i = ⟨S , A , p i , r i ⟩ , i = 1 , · · · , N } , where S and A denote the common state and action spaces, respectively, while a task T i is distinct

in both the transition distribution p i and reward function r i . At each training iteration, informative trajectories that are sampled from other tasks but expected to achieve a higher sum of rewards for a certain task T j will be transferred to and leveraged by T j , which is the core idea behind hindsightbased methods. Here, we argue that their intuitive sum of rewards, however, cannot accurately indicate the true state value V j of a trajectory that is sampled from other tasks and transferred to this target task, especially when dynamics mismatches arise across tasks. For a more accurate estimation, we therefore formulate the following sample transfer problem by minimizing the MSE between the true state value V j and its estimate ˆ V j :

̸

where D j = { τ i = { s t , a t , r t , s t +1 } ∣ ∣ T t =0 } i = j is the dataset that contains informative trajectories sampled from other tasks and selected using any relabeling strategy S I , s t = s is the state at step t from a trajectory τ i , and the expectation is taken w.r.t. the randomness of trajectory τ i from step t to the final step T sampled in the task T i . Further, the MSE objective in Eq. (3) can be decomposed as:

where the first term on the RHS equals square of the bias of ˆ V j that is estimated from computation on the informative trajectories, while the second term denotes the variance of estimate ˆ V j .

3.2 Direct Use of Doubly Robust (DR) Estimator

DR estimators can provide an unbiased estimator for off-policy policy evaluation with existence of the distribution mismatch. For a trajectory that is sampled from task T i and transferred to a target task T j , the unbiased estimate for the state value can be determined by directly using the idea of DR:

where ρ ij π ( t ) = π θ ( a t | s t ,z j ) π θ ( a t | s t ,z i ) is the importance weight of adapted policy, ρ ij d ( t ) = p j ( s t +1 | s t ,a t ) p i ( s t +1 | s t ,a t ) denotes the importance weight of transition distributions incurred by dynamics discrepancies, and Q θ ( s, a t , z j ) denotes the Q -value function for task T j fitted by the critic network with parameter θ , which is then used to compute the fitted state value function V θ ( s, z j ) . In meta-RL, the adapted policy for each task can be explicitly known, thus allowing access to the true policy importance weight ρ ij π ( t ) . Referring to Appendix A.2, it is then easy to verify that V DR ij ( s t = s ) still holds the doubly robust property. Namely, V DR ij ( s t = s ) is unbiased when either the importance weight or value function is correctly estimated. Plugging the unbiasedness of V DR ij ( s t = s ) to Eq. (4), the MSE of this DR estimator can thus be simplified to E τ i | t : T [ ( V j ( s t ) -V DR ij ( s t ) ) 2 ∣ ∣ s t ] = V ar ( V DR ij ( s t ) ) , which is dominated by its variance that can be further determined with the following theorem.

Theorem 3.1 The variance of estimator V DR ij ( s t = s ) can be recursively given by: ∀ t = 1 , · · · , T

where V ar t [ V DR ij ( s T ) ] = 0 , ∆( s t , a t ) = Q θ ( s t , a t , z j ) -Q j π ( s t , a t ) , and Q j π is the true Q -value.

On the RHS of Eq. (6), the first, second and fourth terms all take expectation over π θ ( a t | s t , z i ) , while the second variance term is also due to randomness of π θ ( a t | s t , z i ) . The variance inside the first term stems from randomness of the rewards (which will become zero if the reward function is deterministic), while the expectation inside the third term considers randomness of the future. Theorem 3.1 implies that when the other variables are kept unchanged, a possibly larger value for the

estimate of dynamics importance weight ρ ij d will enlarge the third term in Eq. (6), which in turn results in a higher variance and consequently a larger MSE. As an empirical validation, in Fig. 1, we show the standard deviation (SD) of state value estimate in the Point-Robot environment by directly using the DR estimator in Eq. (5). It can be seen that the estimated value by DR oscillates significantly during the entire training process, which will eventually cause an unstable and unsuccessful meta-training.

3.3 DR Augmented Estimator (DRaE) with Optimal ˆ ρ ij ∗ d to Balance Bias and Variance

To overcome the high variance and large MSE issue suffered by the original unbiased DR estimator, we consider the MSE minimization for sample transfer as in Eq. (3), which is equivalent to jointly optimizing the bias and variance w.r.t. the dynamics importance weight ρ ij d . Since the actual transition probability distributions of different tasks are usually inaccessible, in practice, we will use the estimated value of dynamics importance weight ˆ ρ ij d instead of its true value ρ ij d . However, by approximating ρ ij d with ˆ ρ ij d , the original

0

50

100

150

200

250

Epoch

10 3

10 4

10 5

10 6

SD (DR Estimation)

unbiased DR estimator V DR ij ( s t = s ) in Eq. (5) becomes biased, which is denoted as the DR augmented estimator ˜ V DR ij ( s t = s ) , with the bias given by:

Referring to Appendix A.5 for the detailed derivation, through bounding this Bias (ˆ ρ ij d ) and by further decomposing the MSE into the sum of bias and variance-related terms, we can obtain an upper bound for the MSE of the biased DR augmented estimator ˜ V DR ij ( s t = s ) :

where V ( ρ π ) denotes the terms on the RHS of Eq. (6) that contain ρ ij π but without ˆ ρ ij d . We then minimize this upper bound w.r.t. ˆ ρ ij d for the MSE reduction, which leads to a convex optimization problem. By letting the first-order derivative of the upper bound in Eq. (8) w.r.t. ˆ ρ ij d equal to zero, we can thus obtain the optimal estimated value of dynamics importance weight, as:

3.4 Interval-Based Approximation for Optimal ˆ ρ ij ∗ d of DRaE

Though the optimal ˆ ρ ij ∗ d of DRaE can effectively balance the bias and variance by minimizing the MSE, its computation is nontrivial in practice, since the true state value V j ( s t +1 ) at a certain state is also infeasible to access. Here, we propose an interval-based approximation for the optimal ˆ ρ ij ∗ d , and show that it falls with an interval between ˆ ρ var d that achieves the minimum variance in Eq. (6) and the true importance weight ρ ij d . We further verify that any other values of ˆ ρ ij d outside this interval [ ˆ ρ var d , ρ ij d ] will simultaneously enlarge the bias and variance, and consequently result in a larger MSE.

1) Lower bound ˆ ρ var d . The variance of DRaE ˜ V DR ij ( s t ) can be determined by Eq. (6), with the true ρ ij d ( t ) estimated with ˆ ρ ij d ( t ) . Through minimizing this variance w.r.t. ˆ ρ ij d ( t ) , we formulate a convex optimization problem since the objective is a quadratic function of ˆ ρ ij d ( t ) , with the optimal solution ˆ ρ var d ( t ) obtained by letting the first-order derivative of this objective equal to zero:

The distance from the optimal ˆ ρ ij ∗ d ( t ) to this lower bound ˆ ρ var d ( t ) can then be determined as:

which will be reduced by increasing the value of policy importance weight ρ ij π ( t ) . In meta-RL, informative trajectories for the target task T j are actually sampled from an original task T i using π i , so the actions in these trajectories have a relatively larger sampling probability by π i than π j , resulting in a small ρ ij π ( t ) especially at the early stage of training. Then, as the training proceeds and with the increase of reward that policy π j can achieve for task T j , these selected trajectories from task T i will be less informative as it is more likely for policy π j to generate them also in the target task T j . This may lead to an increase of ρ ij π ( t ) along the training process, which in turn can consistently reduce the distance between ˆ ρ ij ∗ d ( t ) and ˆ ρ var d ( t ) . In Section 5.3, we will empirically verify that ρ ij π ( t ) will generally increase from the initial small values that are much less than one to a stable value around one, which can lead to a tighter lower bound ˆ ρ var d ( t ) . In addition, decreasing the estimated value of ˆ ρ ij d ( t ) from the optimal ˆ ρ ij ∗ d ( t ) to the lower bound ˆ ρ var d ( t ) will monotonically reduce the variance but increase the bias of DRaE, while a smaller estimate ˆ ρ ij d ( t ) < ˆ ρ var d ( t ) will cause a consistently larger MSE. Therefore, the optimal ˆ ρ ij ∗ d ( t ) is lower bounded by ˆ ρ var d ( t ) .

2) Upper bound ρ ij d . Note that when ˆ ρ ij d ( t ) = ρ ij d ( t ) , our DRaE ˜ V DR ij ( s t ) becomes the unbiased DR estimator V DR ij ( s t ) , while any deviation from the true dynamics importance weight ρ ij d ( t ) will result in a biased estimator. On the other hand, increasing the value of ˆ ρ ij d ( t ) from ˆ ρ var d ( t ) to ρ ij d ( t ) will consistently enlarge the variance. Therefore, a larger estimated value ˆ ρ ij d ( t ) > ρ ij d ( t ) will enlarge both the bias and variance, which then indicates that the optimal ˆ ρ ij ∗ d ( t ) is upper bounded by ρ ij d ( t ) .

In conclusion, we show that the optimal dynamics importance weight ˆ ρ ij ∗ d ( t ) falls within the interval [ ˆ ρ var d ( t ) , ρ ij d ( t )] , while an estimated value of ˆ ρ ij d ( t ) inside this interval can generally achieve a biasvariance tradeoff. In the following, we further derive a sample-independent upper bound to constrain the MSE difference between taking ˆ ρ ij ∗ d ( t ) and ˆ ρ ij d ( t ) as the estimated value in DRaE, respectively.

Proposition 3.2 For an estimated dynamics importance weight ˆ ρ ij d ( t ) ∈ [ ˆ ρ var d ( t ) , ρ ij d ( t ) ] , the MSE difference between DRaE taking ˆ ρ ij ∗ d ( t ) and ˆ ρ ij d ( t ) as estimated importance weight is bounded by:

It thus indicates that by setting an upper bound for ˆ ρ ij d ( t ) smaller than the true ρ ij d ( t ) , we can further reduce the MSE difference between using ˆ ρ ij d ( t ) and ˆ ρ ij ∗ d ( t ) . See Appendix A.8 for detailed proof.

4 Doubly Robust Augmented Transfer (DRaT) for Meta-RL

Based on our theoretical findings of the MSE reduction for sample transfer by DRaE, we formally present the doubly robust augmented transfer (DRaT) algorithm for meta-RL, and summarize its detailed implementation on PEARL [4] in Algorithm 1. At the data collection phase, for a task T i , we additionally store the distribution parameters of adapted policy π i that collects data into buffer B i . Later at the training phase, we sample an informative trajectory set D j from all the replay buffers using strategy S I , which selects trajectories with higher relabeled sum of rewards for task T j . Hence, any of the hindsight-based methods can be used to determine S I here. While for environment where tasks share the same reward, we can simply choose the higher reward trajectories without relabeling.

We then compute the importance weights ρ ij π and ˆ ρ ij d for each trajectory in D j between the target task T j and originally sampled task T i as in Line 10. We can obtain π j for task T j following the same adaptation procedure as in PEARL, hence ρ ij π of actions in trajectories can be computed as the ratio of sampled probabilities π j and π i . While for the dynamics importance weight, a straightforward way to estimate ˆ ρ ij d is to predict the distribution of transition ( s, a, s ′ ) ∈ τ i ∼ D j by p ψ and compute it by ˆ ρ ij d = p j ψ ( ·| s, a ) /p i ψ ( ·| s, a ) . However, the incorrect estimate of transition distribution in the original task T i may cause a small value of p i ψ ( ·| s, a ) and thus a large value of ˆ ρ ij d , possibly frustrating

the entire training process. Instead, we estimate ˆ ρ ij d as guided by the interval-based approximation for optimal ˆ ρ ij ∗ d of DRaE in Section 3.4, which is constrained within an interval [ ˆ ρ var d , ρ ij d ] . We predict the distribution of transition ( s, a, s ′ ) in task T j by prediction network p j ψ ( ·| s, a ) and in the original task T i by Gaussian distribution N ( s ′ , σ ) , which indicates a high probability of transition ( s, a, s ′ ) ∈ τ i occurring in task T i . Hence, the maximum value of ˆ ρ ij d computed by p j ψ ( ·| s, a ) / N ( · , σ ) will automatically fall into interval [0 , ρ ij d ) with a high probability, due to the large denominator resulted from the high belief we hold about the transition's occurrence in task T i . In addition, we also clip the small value of ˆ ρ d using ˆ ρ l d , which is updated as the average value of ˆ ρ var d at this epoch.

The importance weights ρ ij π and ˆ ρ ij d are used to estimate the state value in D j by DRaE ˜ V DR as in Eq. (5), which is used as the target value to compute critic loss in Eq. (1). Finally, the prediction network p ψ is updated by optimizing the prediction error, and other networks are updated following the same procedure as in PEARL. It is noted that DRaT can also be implemented on other meta-RL baselines, such as MAML [12] and its variants, where the DRaE ˜ V DR can be used to compute policy gradients for the meta-policy parameter optimization.

5 Experiments

In this section, we assess performance of our DRaT approach following the same evaluation procedure as in [4]. For each benchmark environment, we construct a test task set that is disjoint with the training task set. At the end of each training epoch, we run the evaluation procedure, where trajectories of a fixed length are sampled by the adapted meta-policy from the updated context with continuously sampled transitions. Specifically, we conduct experiments to answer the following questions.

Can DRaT achieve a better adaptation performance on meta-test tasks in the extreme challenging sparse-reward environment with varying reward functions and dynamics, as compared to hindsight relabeling methods that inherently considers only the difference of reward functions?

Can DRaT effectively improve the adaptation performance on the standard meta-RL benchmarks where tasks share the identical dense reward function but only differ in distinct dynamics?

Will the policy importance weight ρ ij π increase along the training process in consistency to the performance improvement of DRaT, which thus provides a tighter lower bound for ˆ ρ ij d ? Will DRaE provide a better value estimate for meta-RL?

Environment setup. We compare our DRaT with other baseline algorithms on six robotic control tasks, which can be divided into two families according to their reward and dynamics setup. For the implementation detail of DRaT, please refer to Appendix B.1. These tasks are all simulated via MuJoCo [13], where we further generate various dynamics by randomly sampling the environment parameters, including body mass, body inertia, joint damping, and body component's friction. One

0.0

0.2

0.4

0.6

0.8

1.0

timesteps     1e6

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Average Return

PEARL

HTR

HFR

AIR

DRaT (Ours)

(a) Point-Robot-Params-Sparse

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

timesteps     1e6

280

300

320

340

360

380

400

420

Average Return

PEARL

HTR

HFR

AIR

DRaT (Ours)

(b) Ant-Params-Sparse

0.0

0.5

1.0

1.5

2.0

2.5

timesteps     1e6

0

50

100

150

200

250

300

350

Average Return

PEARL

HTR

HFR

AIR

DRaT (Ours)

(c) Humanoid-Params-Sparse

Figure 2: Evaluation curves of average return: test-task performance vs. time steps.

0.0

0.5

1.0

1.5

2.0

2.5

timesteps     1e6

0

200

400

600

800

Average Return

PEARL

IST

HFR

MQL

DRaT (Ours)

(a) Walker-2D-Params

0.0

0.1

0.2

0.3

0.4

0.5

0.6

timesteps     1e6

0

50

100

150

200

250

300

350

Average Return

PEARL

IST

HFR

MQL

DRaT (Ours)

(b) Hopper-Params

0.0

0.2

0.4

0.6

timesteps     1e6

14

12

10

8

6

4

2

0

Average Return

PEARL

IST

HFR

MQL

DRaT (Ours)

(c) Point-Robot-Params

family contains the sparse-reward environments with varying reward functions and dynamics: 1) Point-Robot-Goal-Params : control the arm of a 3D robot to reach random goals in the 3D space; 2) Ant-Goal-Params-Sparse : navigate a quadruped robot to reach randomly generated goals along the perimeter of a semi-circle; 3) Humanoid-Goal-Params-Sparse : navigate a humanoid robot to reach randomly generated goals on a semi-circle. These environments are with the sparse-reward setting in the sense that the RL agent can receive a meaningful reward signal that indicates its distance from the goal only when it reaches areas near the goals. The other family includes environments that have varying dynamics and the identical dense reward function: 1) Hopper-Params : control a planar monopod robot to hop as fast as possible and avoid falling; 2) Walker-2D-Params : control a 2D bipedal robot to walk and perform the same task as in Hopper; 3) Point-Robot-Params : the same as Point-Robot-Goal-Params but with dense reward function. Please refer to Appendices B.2 and B.3 for the detail of varying dynamics and reward function settings, respectively.

5.1 Evaluation on Sparse-Reward Environments with Varying Rewards and Dynamics

To answer the first question, we conduct experiments on the first family of sparse-reward environments with varying reward functions and dynamics. We compare DRaT with four baselines: PEARL [4], Hindsight Task Relabeling (HTR) [14], Hindsight Foresight Relabeling (HFR) [15], and Approximate Inverse RL Relabeling (AIR) [6]. Specifically, PEARL is utilized as the baseline meta-RL algorithm for all the other algorithms. For a fair comparison, we keep all the hyper-parameters about meta-RL the same as those in PEARL's source code. In HTR, we implement the single episode relabeling strategy, where hindsight task goal is selected as a state that is reached in a sampled episode and networks in HTR are trained by transitions from that episode which is relabeled by the hindsight goal. HFR utilizes a utility function (foresight) to judge the informativeness of trajectories to a task, and trajectories with higher utilities in certain tasks are more likely to be transferred through the relabelling with reward function (hindsight) to this task. In AIR, a trajectory is relabeled to a new task, for which this trajectory beats the most previously sampled trajectories in terms of the total reward among a sampled candidate task set. We use the same relabeling strategy in both the AIR and our DRaT for selecting informative trajectories for each training task.

Evaluation results. We report the averaged return of trajectories sampled in the evaluation phase. From the evaluation return curves of the three tasks as shown in Figs. 2(a)-2(c), our DRaT significantly outperforms the other baseline algorithms, especially in the Ant and Humanoid environments that have higher dimensions of states. In comparison, PEARL struggles to train the Ant and Humanoid under such a challenging environment setting. PEARL-based HTR cannot even learn to move to goals for Ant in such a short period of time steps, while it achieves a similar performance to PEARL for Humanoid. PAERL-based HFR tends to outperform PEARL and other relabeling methods for both Ant and Humanoid, which is possibly because HFR considers relabeling w.r.t. performance of the adapted policy using trajectory instead of just considering the sum of relabeled rewards. We also observe that simply applying AIR into PEARL will even degrade the performance of PEARL. Providing a more accurate estimate of state value functions for transferred informative trajectories, our DRaT generally achieves a better asymptotic performance and can learn faster compared to the other baselines. Note also that in Fig. 2, the solid curve illustrates the mean return on all runs of each algorithm with different random seeds, while the shaded area shows the standard deviation.

0

50

100

150

200

250

Epoch

0.5

1.0

1.5

2.0

2.5

(importance weight)

(a) Point-Robot-Params

0

50

100

150

200

250

Epoch

0.6

0.8

1.0

1.2

1.4

(importance weight)

(b) Ant-Params-Sparse

0

50

100

150

200

250

Epoch

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

(importance weight)

(c) Humanoid-Params-Sparse

Figure 4: Curves of

ρ

ij

π

: policy importance weight vs. training epochs.

0

100

200

300

400

Epoch

0

50

100

150

200

250

Error of Estimation

V

V DR

(a) Point-Robot-Params

0

100

200

300

400

Epoch

20

40

60

80

100

120

140

Error of Estimation

V

V DR

(b) Ant-Params-Sparse

0

50

100

150

200

250

300

350

400

Epoch

0

25

50

75

100

125

150

175

Error of Estimation

V

V DR

(c) Humanoid-Params-Sparse

Figure 5: Curves of errors of value estimation: absolute difference vs. training epochs.

5.2 Evaluation on Dense-Reward Environments with Varying Dynamics Only

Then, to answer the second question, we conduct experiments on the second family of environments with varying dynamics and the identical dense reward function. We compare DRaT with four baselines: PEARL [4], Importance Sampling augmented Transfer (IST), Meta-Q-Learning (MQL) [16], and Hindsight Foresight Relabeling (HFR) [14]. Specifically, PEARL is utilized as the baseline meta-RL algorithm for DRaT, IST and HFR. In MQL, for a given transition and task, a fitted logistic classifier outputs probability of the transition being sampled from the task, from which the importance weight can be estimated. Transitions from other tasks are used to compute the meta-RL objective multiplied by this importance weight on a new task, hence the contribution of each transition to the update of meta-parameters is judged by similarity between the new task and its source task. In IST, we use the IS estimator instead of our DRaE, while keeping all the other settings the same as in DRaT.

Evaluation results. We report the averaged return of trajectories sampled in the evaluation phase at each epoch, and show the evaluation return curves of the three tasks, respectively, in Figs. 3(a)3(c). For MQL, we focus specifically on its performance with transfer by the estimated importance weights. Hence, we report its evaluation returns of adapted meta-policy with the aid of transferred samples. It can be seen that with informative samples transferred by DRaT, we can improve the evaluation average return over the other baselines in all the benchmark environments. In addition, DRaT consistently accelerates the training process of Walker-2D-Params and Hopper-Params at the early stage, and also reaches the highest return in Point-Robot-Params. In addition, DRaT presents in general a smaller standard deviation.

5.3 Performance Evaluation of Proposed DRaE

We then experimentally estimate the value of ρ ij π to evaluate its trend during the training process of DRaT. We compute the averaged value of ρ ij π estimated for informative trajectories at each training epoch, and show curves of the mean value of Point-Robot-Params, Ant-Params-Sparse, and Humanoid-Params-Sparse in Figs. 4(a)-4(c), respectively, where the shaded area shows the standard deviation on several runs. The results show that value of ρ ij π increases with the improvement of performance on all the training tasks, since selected informative trajectories are more likely to be sampled on target task T j , which can generate a tighter lower bound as we state in Section 3.4.

At last, we evaluate the estimation error of our proposed DRaT compared to the value network V θ as used in PEARL. For a better illustration, we do not plot results of directly using DR estimator, since it may result in a huge variance and estimation error as we analyze in Section 3.2. We analyze the error by computing the absolute difference between the averaged estimated value and averaged true return at each training epoch, and show the error curves of Point-Robot-Params, Ant-Params-Sparse, and Humanoid-Params-Sparse in Figs. 5(a)-5(c), respectively. The results show that our DRaE provides a better value estimation during the training process. It is also observed that curves of DRaT and V θ present a similar trend, which is because the computation of DRaE depends on the value of V θ .

6 Computational Complexity Analysis

Additional computational complexity is brought by DRaT for training of dynamics prediction network for each task, computation of DR estimate, and computation of relabelling for informative trajectories. Here, we analytically show that this additional computational complexity is comparable to its baseline

PEARL with a linear scaling factor. Typically in an off-policy meta-RL algorithm, each epoch (i.e., meta-training iteration) is divided into sampling and training phases. In the sampling phase, computational cost stems mainly from the actions chosen by feed-forward computation of the policy network and inference networks. Since all the algorithms (i.e., HTR, HFR, AIR, DRaT) follow the same sampling process as PEARL, we only analyze the computational cost in the training phase.

PEARL: In the training phase, the computational cost contains the feed-forward and back-propagation computation of policy network, value network and inference network. Given K training iterations at each epoch, batch size N B of transitions from N training tasks, state space cardinality |S| and action space cardinality |A| , and assuming a constant computational cost of feed-forward and backpropagation computation c 1 , the total computational cost of training phase is O ( K · N B · N ·|S|·|A|· c 1 ) .

DRaT: Besides the same training cost O ( K · N B · N ·|S|·|A|· c 1 ) as PEARL, additional computational cost brought by DRaT includes training of dynamics prediction networks, computation of DR estimator, and computation of relabelling, as follows. 1) Considering building separate prediction networks for N training tasks and the cost of feed-forward and back-propagation computation c 1 , using the same batch of transitions for training, the additional cost of training dynamics prediction networks is also O ( K · N B · N · |S| · |A| · c 1 ) . 2) Considering sampling informative trajectories with a maximum length of L for N training tasks and the cost c 2 of DR estimation computation at each time step, the computational cost of DR estimation is O ( K · N · L · |S| · |A| · c 2 ) . 3) We use the approximate inverse RL relabeling (AIR), where we sample one trajectory from each training task for relabeling, leading to N candidate trajectories in total. Assuming that the cost of computing relabeled reward at each time step is c 3 , the cost of relabeling is then O ( K · N · L · |S| · |A| · c 3 ) . Thus, the additional computational complexity of DRaT is dominated by O ( K · N · max { N B , L } · |S| · |A| ·

Insights for complexity reduction. 1) To reduce the cost of training dynamics prediction network, we may consider training a meta-dynamics prediction network, which makes prediction for all the tasks with a single network, and thus eliminates the need of training a separate network for each task. 2) To reduce the cost of DR estimation, we may consider a similar solution in TD ( n ) , which makes a trade-off between TD (0) and Monte-Carlo estimation by using the n -step rollout and fitted network.

7 Conclusion and Limitations

In this paper, we have proposed a doubly robust augmented estimator (DRaE) to tackle the mismatch of dynamics distributions in sample transfer under a more general sparse-reward meta-RL setting, where dynamics and reward functions can both vary across different tasks. While DRaE was established with an optimal dynamics importance weight by minimizing the MSE between estimated and true values of the value function, we proposed a tractable interval-based approximation that guaranteed to cover the optimum. We further developed a doubly robust augmented transfer (DRaT) algorithm for transferring informative samples across tasks in meta-RL. Experimental evaluation on several MuJoCo locomotion tasks demonstrated the effectiveness of our proposed DRaT algorithm.

The major limitation of our method is that it requires computation of the DR estimate and training of the dynamics prediction network for tasks, which may incur additional computational complexity. Besides, like other hindsight-based methods, the reward function is required for relabeling and selection of informative trajectories, which might be inaccessible in some extreme scenarios.

References

Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6672-6679, 2020.

Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics , 7(62):eabk2822, 2022.

Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning , pages 5331-5340. PMLR, 2019.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.

Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. Advances in neural information processing systems , 33:7754-7767, 2020.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pages 652-661. PMLR, 2016.

Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning , pages 2139-2148. PMLR, 2016.

Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 14(11), 2013.

Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive blending for policy evaluation and learning. In International Conference on Machine Learning , pages 6005-6014. PMLR, 2019.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, 2012.

Charles Packer, Pieter Abbeel, and Joseph E Gonzalez. Hindsight task relabelling: Experience replay for sparse reward meta-RL. Advances in Neural Information Processing Systems , 34:2466-2477, 2021.

Michael Wan, Jian Peng, and Tanmay Gangwani. Hindsight foresight relabeling for metareinforcement learning. In International Conference on Learning Representations , 2022.

Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta-q-learning. In International Conference on Learning Representations , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study introduces the DRaT (Doubly Robust Augmented Transfer) algorithm, an advanced extension of hindsight-based transfer methods, which tackles not only reward mismatches but also discrepancies in transition dynamics. The authors provide a theoretical analysis to establish the optimality of their interval-based approximation, DRaE (Doubly Robust Augmented Estimator), in calculating the optimal importance sampling weight. Through empirical evaluation, it is demonstrated that DRaT surpasses conventional hindsight-based methods in performance, particularly in sparse-reward MuJoCo tasks that involve varying reward structures and transition dynamics.\n\nSTRENGTHS:\n**S1.** Addressing a Critical Yet Underexplored Aspect of Transfer Learning\n\nThis study tackles an essential but often overlooked aspect of transfer, specifically in the context of varying transition dynamics. The authors have put forth commendable efforts in highlighting the significance of calculating the optimal dynamics importance weight, which is a critical component in transfer under changing transition dynamics.\n\n**S2.** Solid Theoretical Foundation and Novel Approximation Method\n\nThe work presents a robust theoretical underpinning for the proposed method, ensuring reliability and efficacy. The use of a tractable interval-based approximation is innovative and enhances the practicality of the approach. The proof confirming that the mean squared error (MSE) is guaranteed to encompass the optimum is both rigorous and convincing.\n\n**S3.** Thorough Empirical Evaluation in Diverse Scenarios\n\nThe authors have conducted a comprehensive empirical evaluation by testing their method in three environments with sparse rewards and three with varying state dynamics. Such thorough testing provides a more complete picture of the algorithm's capabilities. Impressively, the proposed method demonstrates substantial and consistent improvements over all baselines, attesting to its effectiveness and robustness.\n\nWEAKNESSES:\n**W1.** Computational Overhead\n\nA notable drawback of this study, which the authors have acknowledged in the limitations section, is the computational burden associated with calculating the doubly robust estimate and training the dynamics network. Given that practicality is an essential factor in the real-world application of algorithms, it is crucial to consider computational efficiency. It would be beneficial if the authors could include a comparison of the wall-clock time for each method, as this would provide valuable insights into the trade-offs between performance gains and computational costs.\n\n**Acknowledgment Following Rebuttal**\n\nI have reviewed the author's response. The response addressed my concerns.\n\nQUESTIONS:\n**Q1.** Results in Meta-RL are typically characterized by a high degree of variability, necessitating the aggregation of multiple runs for reliable evaluation. However, it appears that the number of seeds used for evaluation is not reported in the manuscript or the appendix. Could the authors clarify how many seeds were employed and consider including this information for a more transparent and rigorous evaluation?\n\n**Q2.** Given the computational overhead associated with the proposed method, as discussed earlier, are there any strategies or optimizations that the authors have considered to enhance its computational efficiency? Insights into potential avenues for reducing computational costs without compromising effectiveness would be valuable.\n\n**Q3.** Contrary to the checklist, I think the authors didn't submit the code or link to their codes. Any plans for releasing the code?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies that some existing approaches ignore  varying dynamics in meta-RL and proposes a new method for addressing this. In particular, the paper focuses on minimizing the mean-squared error of value functions and showed that doubly robust estimators suffer from a high-variance problem. Consequently, the paper proposes the doubly robust augmented estimator to estimate the importance weight of the dynamics, that reduces variance at the cost of introducing bias. The paper further identifies the intervals for which the importance weight should lie in, in order to maintain low variance. Finally, the paper provides empirical analysis to demonstrate that the proposed approach yields smaller value prediction error than existing baseline, and demonstrated that the proposed method can outperform existing baselines in certain control environments.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow in general.\n- Visualizations that supports various claims regarding the behaviours of the estimators---the paper demonstrate the proposed estimator has lower error compared to existing approach, and that the policy importance sampling weights do increase to approximately $1$ as informative trajectories are sampled more frequently.\n- The paper demonstrated that the proposed method is theoretically sound (under some assumptions) and its empirical performance is comparable (if not better) to existing baselines.\n\nWEAKNESSES:\n## Major Comments\n- Figure 3 suggests that with same reward function but different dynamics, the current baseline is already fairly robust? Does this really suggest that in figure 2, the problem is stemming from sparse reward?\n\t- This may have stemmed from a confusion. For section 5.2, page 9, line 329, the paper mentions that \"..., while keeping all the other settings the same as in DRaT.\" Does that mean other approaches are now using DRaE?\n\nQUESTIONS:\n- The appendix mentions that the proofs really follow through by assuming that $\\gamma \\rho_{\\pi}^{ij}(t) \\tilde{V}_{ij}^{DR} (s)$ is not correlated to other expectation terms. I am wondering why this is a reasonable assumption---in particular, it is not immediately obvious that this term does not dominate the expectation terms.\n- How does different $\\sigma$ for approximating $N(s', \\sigma)$ affect the estimation? Is there a sensitivity analysis regarding this? My expectation is that very small $\\sigma$ would cause stability problem, while very high $\\sigma$ would prevent learning. Is there an informed way to select this?\n\nLIMITATIONS:\nThe paper has listed few limitations of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the meta-reinforcement learning setting with sparse reward. Previous work with hindsight-based sample transfer approaches requires the assumption that tasks differ only in reward functions. This paper proposes a doubly robust augmented transfer (DRaT) approach that allows both dynamics mismatches and varying reward functions across tasks. They first theoretically derive an upper bound of mean squared error between the estimated values of transferred samples and their true values in the target task. Then they propose an interval-based approximation to empirically find the optimal importance weight. In the experiment part, they implement DRaT on top of an off-policy meta-RL method and show that this method outperforms hindsight-based approaches on various sparse-reward MuJoCo locomotion tasks.\n\nSTRENGTHS:\n1.\tThis paper is well-written and easy to follow. The motivation for using doubly robust estimators to solve the sparse reward problem in meta-RL is convincing.\n2.\tThe theoretical analysis of the doubly robust augmented estimator is detailed and well-motivated.\n3.\tThe empirical implementation of the proposed algorithm outperforms other baselines by a large margin.\n\nWEAKNESSES:\n1.\tThe details about the meta-RL setting in experiments are not provided. The authors mention that they generate various dynamics by randomly sampling the environment parameters, including body mass, body inertia, joint damping, and body component friction. However, I cannot find details about the range of those parameters.\n2.\tThe authors say that “One family contains the sparse-reward environments with varying reward functions and dynamics”. I cannot find details about how the reward functions are different in the settings.  Why does “control the arm of a 3D robot to reach random goals in the 3D space” has different reward functions? \n3.\tThe algorithm requires the reward function for relabeling and selecting informative trajectories, which is similar to previous hindsight-based methods.\n\nQUESTIONS:\n1.\tPlease provide the missing details in the experiment as I mentioned in the weaknesses.\n2.\tWhy use PEARL as the backbone? I am not familiar with the area of meta-RL with sparse reward, but I wonder if there is any later work that outputs PEARL in the meta-RL setting. If yes, maybe more baselines should be reported.\n\nLIMITATIONS:\nAs discussed in the conclusion section, this method requires the reward function, which may not be available in some tasks. Since this is a common setting in existing methods for sparse reward meta-RL, it is still acceptable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Doubly Robust Augmented Transfer (DRaT), a novel approach for dealing with sparse-reward scenarios in meta-reinforcement learning (Meta-RL). DRaT transfers informative trajectories from various tasks to a target task, effectively handling dynamics mismatches and different reward functions. The authors propose an interval-based approximation to the importance weight. The DRaT algorithm is applied to an off-policy Meta-RL baseline, demonstrating superior performance over other hindsight-based methods on various sparse-reward MuJoCo tasks with different dynamics and reward functions.\n\nSTRENGTHS:\n**Originality**: The paper presents a novel approach, Doubly Robust Augmented Transfer (DRaT), for dealing with sparse-reward scenarios in meta-reinforcement learning. The idea of transferring informative trajectories from various tasks to a target task is an interesting direction in the field of meta-reinforcement learning.\n\n**Quality**: DRaT effectively handles dynamics mismatches and different reward functions across tasks, for the MuJoCo environments it is tested on. The method outperforms other hindsight-based methods on various sparse-reward tasks. However, without a detailed analysis of the time complexity of the algorithm or a comparison with other state-of-the-art methods in terms of computational efficiency, it's hard to fully assess the quality of the method.\n\n**Clarity**: While the paper presents complex ideas, it does so in a clear and understandable manner. The authors have done a good job of explaining their method and its benefits. However, some sections could benefit from more detailed explanations or examples, particularly for readers who are not deeply familiar with the field.\n\n**Significance**: The significance of the paper is evident in its potential impact on the field of meta-reinforcement learning. DRaT addresses a key challenge in the field - dealing with sparse-reward scenarios. By demonstrating superior performance on various tasks, the paper shows that the method could be a valuable tool for researchers and practitioners in the field. However, the real-world applicability and scalability of the method would need to be further explored to fully understand its significance.\n\nWEAKNESSES:\n**More Extensive Evaluation**: While the paper demonstrates the effectiveness of DRaT on various sparse-reward tasks, a more extensive evaluation would strengthen the paper's claims. The current evaluation is somewhat limited and does not fully explore the method's performance across a wide range of scenarios. The authors could consider using the [MetaWorld](https://arxiv.org/pdf/1910.10897.pdf) benchmark, a comprehensive benchmark of environments specifically designed for meta-reinforcement learning. This would provide a more rigorous testing ground for the DRaT method.\n\n**Time Complexity Analysis**: The paper does not provide detailed information on the time complexity of DRaT. Information regarding the performance with respect to time in the experiments would be valuable in assessing the method's scalability and efficiency, particularly for large-scale problems.\n\nQUESTIONS:\n1. Is a separate dynamics prediction network fitted for each possible task 'j' in the training set?\n2. Would it be possible to test the DRaT method using the MetaWorld benchmark, which is specifically designed for meta-reinforcement learning?\n3. Could you provide a detailed analysis of the time complexity of the DRaT method to assess its scalability and efficiency?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}