Offline Meta-Reinforcement Learning with Advantage Weighting
Authors: Eric Mitchell
Rafael Rafailov
Xue Bin Peng
Sergey Levine
Chelsea Finn
Venue:  (2021)
Keywords: offline
meta-reinforcement learning
meta-learning
reinforcement learning
maml

Abstract: This paper introduces the offline meta-reinforcement learning (offline meta-RL) problem setting and proposes an algorithm that performs well in this setting. Offline meta-RL is analogous to the widely successful supervised learning strategy of pre-training a model on a large batch of fixed, pre-collected data (possibly from various tasks) and fine-tuning the model to a new task with relatively little data. That is, in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks and adapt to a new task with a very small amount (less than 5 trajectories) of data from the new task. By nature of being offline, algorithms for offline meta-RL can utilize the largest possible pool of training data available and eliminate potentially unsafe or costly data collection during meta-training. This setting inherits the challenges of offline RL, but it differs significantly because offline RL does not generally consider a) transfer to new tasks or b) limited data from the test task, both of which we face in offline meta-RL. Targeting the offline meta-RL setting, we propose Meta-Actor Critic with Advantage Weighting (MACAW). MACAW is an optimization-based meta-learning algorithm that uses simple, supervised regression objectives for both the inner and outer loop of meta-training. On offline variants of common meta-RL benchmarks, we empirically find that this approach enables fully offline meta-reinforcement learning and achieves notable gains over prior methods.

Under review as a conference paper at ICLR 2021
OFFLINE META-REIINFORCMEENT LEARNING WITH ADVANTAGE WEIGHTING
Anonymous authors
Paper under double-blind review
ABSTRACT
This paper introduces the offline meta-reenforremeet learning (offline meta-RL) problem setting and proposes an algorithm that performs well in this setting. Offline meta-RL is analogous to the widely successful supervised learning strategy of pre- training a model on a large batch of fixed, pre-collected data (possibly from various tasks) and fine-tuning the model in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks and adapt new task with a very small amount (less than 5 trajectories) of data from the new task. By nature of being offlin,, algorithms for offline meta-RI. RI. can utilize the largest possible pool of training data available and eliminate potentially unsafe or costly data collection during meta-training. This setting inherits the challenges of offline RL, but it differs significantly because offline RL does not generally consider a) transfer to new tasks or b) limited data from the test task, both of which we face in offline meta-RL. RL. Targeting the offline meta- RL setting, we propose Meta-Actor Critic with Advantage Weighting (MACAW). MACAW is optimization-ban meta-learning algorithm that uses simple, supervised regression objectives for both the inner and outer loop of meta-training. On offlin variants of common meta-RL benchmarks, we empirically find that this approach enables fully offline meta-reinforcement learning and achieves notable gains over prior methods.
new task with relatively little data. That is,
INTRODUCTION
Meta-reinfforeement learning (meta-RL) has emerged as a promising strategy for tackling the high sample complexity of reinforcement learning algorithms, when the goal is to ultimately learn many tasks. Meta-RL algorithms exploit shared structure among tasks during meta-training amortizing the cost of learning across tasks and enabling rapid adaptation to new tasks during meta-testing from only small amount of experience. Yet unlike in supervised learning, where large amounts of pre-collected data can be pooled from many sources to train single model, existing meta-RL. algorithms assume the ability to collect millions of environment interactions online during meta-training. Developing offline meta-RL methods would enable such methods, principle, to leverage existing data from any source, making them easier scale to real-world problems where large amounts of data might be necessary to generalize broadly. To this end, we propose the offline meta-RL problem setting and a corresponding algorithm that uses only offline (or batch) experience from a set of training tasks to enable efficient transfer to new tasks without any further interaction with either the training or testing environments. See Figure for a comparison of offline meta-RL and standard meta-RL.
Because the offline setting does not allow additional data collection during training, it highlights the desirability of a consistent meta-RL algorithm. A meta-RL algorithm is consistent if, given enough diverse data on the test task, adaptation can find a good policy for the task regardless of the training task distribution. Such an algorithm would provide a) rapid adaptation to new tasks from the same distribution as the train tasks while b) allowing for improvement even for out of distribution test tasks. However, designing consistent meta-RI. algorithm in the offline setting difficult: the consistency requirement suggests we might aim to extend the model-agnostic meta-learning (MAML) algorithm (Finn et al. 2017a), since it directly corresponds to fine-tuning at meta-test time. However, existing MAML approaches use online policy gradients, and only value-based approaches have proven effective in the offline setting. Yet combining MAML with value-basce RL subroutines is not straightforward: the higher-order optimization in MAML-like methods demands stable and efficient gradient-deser descent updates, while TD backups are both somewhat unstable and require a large number of steps propagate reward information across long time horizons.

---

Under review as a conference paper at ICLR 2021
To address these challenges, might combine MAML with supervised, bootstrap-free RL subroutine, such as udvantage-weigee
regression (AWR) (Peters and Schaal 2007; (007.Pengg& et al.
2019), for both for the inner and outer loop of a gradient-based meta-learning algorithm, yielding "MAMI+AWR' algorithm. However, as we will discuss in SectionHand find empirically in Section 5| naively combining MAMI. and AWR in this way does not provide satisfactory performance because the AWR policy update is not sufficiently expressive. Motivated by prior work that studies the expressive power of MAML (Finn and Levine 2018), we increase the expressive power of the meta-learner by introducing a carefully chosen policy update in the inner loop. We theoretically prove that this change increases the richness of the policy's 's update and find empirically that this policy update dramatically improves adaptation performance and stability in some settings. We further observe that standard feedforward neural network architectures used in reinforcement learning are not well-suited to optimization- -based meta-learning and suggest an alternative that proves critical for good performance across four different environments. We call the resulting meta-RL. -L algorithm and architecture Meta-Actor Critic with Advantage Weighting, or MACAW.
Our main contributions are the offline meta-RL problem setting itself and MACAW, an offline rettrrriifforemmentt learning algorithm that possesses three key properties: sample efficiency, offline meta- meta-training training, and consistency at meta-test time. To our knowledge, MACAW is the first algorithm to successfully combine gradient-based meta-learning and off-policy value-based RL. Our evaluations include experiments on offline variants of standard continuous control meta-RI. benchmarks as well as settings specifically designed to test the robustness of an offline meta-learner when training tasks are scarce. In all of these settings, MACAW significantly outperforms fully offline variants ofte-offtheeart off- off-policy RL and meta-RL baselines.
2 PRELIMINARIES
In reinforcement learning, an agent interacts with a Markov Decision Process (MDP) to maximize its cumulative reward. An MDP a tuple (S, A... consisting of a state space S, an action space A. stochastic transition dynamics T S x A x S + [0, 1]. and reward function r. At each time step, the agent receives reward r; = r(s,, d₁ $t+1). 1). The agent's objective to maximize the expected return (i.e. discounted sum of rewards) R = Σ. 7'r,, where E [0. 1] is a discount factor. To extend this setting to meta-RL., we consider tasks drawn from a distribution T; p(T), where each task T = AS, Pt, r2) represents different MDP. Both the dynamics and reward function may vary across tasks. The tasks are generally assumed to exhibit some (unknown) shared structure. During meta-training the agent is presented with tasks sampled from p(T); meta-test time, agent's objective is to rapidly find a beigh-eeforming policy for a (potentially unseen) task T' ~ p(7). That is, with only small amount of experience on T', the agent should find policy that achieves high expected return on that task. During meta-training. the agent meta-learns parameters or update rules that enables such rapid adaptation at test-time.
aodel-agnostic meta-learning One class of algorithms for addressing the meta- RL problem (as well as meta-supervised learning) are variants of the MAML algorithm (Finn et al. 2017a). which involves bi-level optimization that aims achieve fast adaptation via few gradient updates. Specifically, MAML optimizes a set of initial policy parameters 0 such that a few gradient-descent steps from 0 leads to policy parameters that achieve good task performance. At each meta-training step, the inner loop adapts 0 to a task T by computing 0' = 0 a (oC( where L is the loss function for task T and a the step size (in general, 6' might be computed from multiple gradient steps. rather than just one as is written here). The outer loop updates the initial parameters as 0 レ 0 ,V,C'(0', where E a loss function for task T, which may or may not be the same as the inner-loop loss function LT, and / is the step size. MAML has been previously instantiated with policy gradient updates in the inner and outer loops (Finn et al., 201 7a Rothfuss et al. 2018)) which can only be applied to on-policy meta-RL settings; we address this shortcoming in this work.
Ad srrantagg-weighter regression. To develop an offline meta-RL algorithm, we build upon advantage-weighted regression (AWR) (Peng et al. 2019), simple offline RL method. The AWR policy objective is given by
log me (a|s) exp (+ (Rn(s, - -Vv(s))))VV(s)))
L. AWR (0,4p, sp, B)- E
where B = (s,, aj. s/,, ry} can be an arbitrary dataset of transition tuples sampled from some behavior policy, and R (s. a) is the return recorded in the dataset for performing action in state
2

---

Under review as a conference paper at ICLR 2021
Standard Meta-RA
Offline Meta-RL
Meta- teta-Trainnng
Meta-Testing Testing (do busel rettulbutten peior)
Meta- Pett-Training
Meta-Testin
Uxpertence
Vogerimace
Esperlesce
Action
Test
Oflliee Test Task Data
Train
Environment
Offine Train
Environments
Tosk Data
Figure 1: Comparing the standard meta-RL setting (left). which includes on- on-policy and off- off-policy meta-Re -LL, with offline meta-RI. (right). In standard meta-RI., new interactions are sampled from the environment during both meta-training and meta-testing, potentially storing experiences in a replay huffer (off-policy offtine meta-RI., batch of data is provided for each training task Tí. This data could he the result of prior skills learned, demonstrations, other means of data collection. The meta-leamer meta- meeaa-raining and can then learn a new test task when given a small buffer of data for that task.
RL). In
these static buffers of data for
Vip (s) is the learned value function for the behavior policy evaluated state and T > 0 is temperature parameter. The term Rs a) (s) represents the advantage of particular action. The objective can be interpreted advantages are assigned larger weights. The value function parameters ip are typically trained using simple regression onto Monte Carlo returns, and the policy parameters 1 are trained using LAWR, Next, we discuss the offline meta-RL problem and some of the challenges it poses.
weighted regression problem, where actions that lead to higher
3 THE OFFLINE META-RL PROBLEM
In the offline meta-RL. RL problem setting, we aim to leverage offline multi-task experience to enable fast adaptation to new downstream tasks. Each task Ti drawn from a task distribution p(T). In the offline setting, the meta- veta-training algorithm is not permitted to directly interact with the meta-training tasks Tú, but instead is provided with a fixed dataset of transition tuples B, {si, az.j. &'y. re,5] for each task. Each B, is populated with trajectories sampled from corresponding behavior policy μs. Each μ4 might be an expert policy, sub-optimal demonstrations, other RL agents, or some mixture thereof. Regardless of the behavior policies (4,, the objective of offline meta-RL. is to maximize return after adaptation on the test tasks. However, depending on the quality of the behavior policies, the maximum attainable return may vary. We observe such a phenomenon in offline data quality ablation experiment in Section 5
Sampling data from fixed dataset at both meta-training and meta-testing time, rather than from the learned policy itself, distinguishes offline meta-RL. from the standard meta-RL setting. This constraint is significant, because most algorithms for meta-RL require large amount of on-policy experience from the environment during meta-training; these algorithms are generally unable to fully make use of data collected by external sources. During meta-testing, (generally unseen) test task Tistt is drawn from pTT). and the meta-trained agent is presented with a new batch of experience D sampled from a distribution B The agent's 's objective to use this batch of data to find the tighest-performen policy for the test task. We consider the case where only B, is fixed during meta-training and Biest corresponds to sampling online trajectories to be the offline meta-RL problem. The case where both B, and Bee are fixed data buffers is called the fully offline meta-RL problem, which is especially applicable in situations when allowing online exploration might be difficult or dangerous. In the fully offline case, might also consider the setting where we perform additional online rollouts with our adapted policy and fine-tune with this online data after the initial offline adaptation step. We call this the fully offline meta-RI. problem with online fine-tuning. The experiments performed in this paper mostly correspond to the fully offline setting. In Appendix C.2 we also conduct an experiment in the setting of fully offline meta-RL with online fine-tuning.
Prior meta-RL methods require interaction with the MDP for each of the meta-training tasks (Finn) et al. 2017a), and though some prior methods build on off-policy RL algorithms (Rakelly al. 2019), these algorithms are known to perform poorly in the fully offline setting (Levine et al., 2000). Both of the offline meta-RL RL settings described above inherit the distributional difficulties of offline RL, which means that addressing this problem setting requires a new type of meta-RL RL method that is capable of meta-training from offline data.
4 MACAW: META ACTOR-CRITIC WITH ADVANTAGE WEIGHTING
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 MACAW Meta-Traiee Meta-Training
Algorithm 2 MACAW Meta-Testing
1: Input: Tasks (T.), offline buffers (D.)
1:
Input: Test task Ty, offline experience D. meta-policy we, meta-value function Va
2: Hyperparameters: learning rates a1, a2 71.
122, training iterations temperature T
3: Randomly initialize meta-parameterss 0, 6
Hyperparameters: learning rates 01, 7. adaptation iterations temperature T
4: for steps do
5: for task T; & { do
Initialize 0g- f- 0, @o f Φ.
6: Sample disjoint batches D. D. Dj
4: for ni steps do
7:
& + - 7/2 V Vo Dt)
5: @t+1 t Ø: -11 Vo, Cole,
8:
8
aj V
VoLa(t (0, o D.
8t41 +- 0: - a1 Vo, (0r, 0c+1,
9:
Φ + Φ 122 E, VoLv (d, D
10:
0 t 0 - 02 Ei O (0%, o4 DD)]
In addition to satisfying the demands of the offline setting, an ideal method for offline meta-RL. should not be limited to the distribution of tasks observed at training time. This is especially important in the offline meta-RL setting, in which the sampling of the training data is out of the control of the agent. In other words, it is critical that offline meta-RL algorithm be consistent, in the sense that given enough, sufficienlly diverse adaptation data at meta-test time, the algorithm can find a good solution to that task, regardless of the meta-training tasks.
To address the numerous challenges posed by offline meta-RI., we propose meta actor-critic with advantage weighting (MACAW). MACAW is offline meta-RL. algorithm that learns initializations respectively, that can rapidly adapt to a new task
á and 0 for a value function Vo and policy seen meta-test time via gradient descent. Both the value function and the policy objectives correspond to simple regression losses in both the inner and outer loop, leading to a stable and consistent inner-loop adaptation process and outer-loop meta- training signal. While these objectives build upon AWR, we show that the naive application an AWR update in the inner loop leads to unsatisfactory performance, motivating the enriched policy update that we describe in Section 4.1 In Sections and 4.3 we detail the full meta- veta-training procedure and an important architectural component of the policy and value networks.
CAWR
L LADV
π Ta
8)
A(s a)
action head
advantage head
4.1 INNER-LOOP MACAW PROCEDURE
body
The adaptation process for MACAW consists of a value function update followed by a policy update and can be found in lines 6-8 in Algo-
rithm [ Optimization-based meta-learning methods typically rely on truncated optimization for the adaptation process (Finn et al. 201 7a). to satisfy both computational and memory constraints (Wu et al.) 2018 Rajeswaran et al. 2019), and MACAW also uses truncated optimiza- tion. However, value-based algorithms that use bootstrapping, such as Q-learning can require many iterations for values to propagate. There- fore, we use a bootstrap- update for the value function that simply performs supervised regression onto Monte-Carlo returns.
Figure 2: MACAW policy ar- chitecture. Solid lines show the forward pass; dashed lines show gradient flow during the backward pass during adapta- tion only; the advantage head is not used in the outer loop policy update.
Given a batch of training data D collected for T, MACAW adapts the value function by taking one a few gradient steps the following supervised objective:
d 7n VoL D.,,
where
(v D) A Eg ((V RD(ss a)
(2)
and where (s, a) is the Monte Carlo return from the state s taking action a observed D.
After adapting the value function, we proceed to updating the policy. The AWR algorithm updates its policy by performing supervised regression onto actions weighted by the estimated advantage, where the advantage is given by the return minus the value: (s, a) Vg((s), (s). While it is tempting to use this same update rule here, we observe that this update does not provide the meta-learne with sufficient expressive power to be universal update procedure for the policy, using universality in the sense used by Finn and Levine (2018). For MAML- methods to approximate any learning procedure, the inner gradient must not discard information needed to infer the task (Finn and Levine
4

---

Under review as a conference paper at ICLR 2021
2018). The gradient of the AWR objective does not contain full information of both the regression weight and the regression target. That is, one cannot recover both the advantage weight and the action from the gradient. We formalize this problem in Theorem I in Appendix [A] To address this issue and make our meta-learner sufficiently expressive, the MACAW policy update performs both Dwvantage-weightee regression onto actions well as additional regression onto action advantages. This enriched policy update is only used during adaptation, and the predicted advantage is used only enrich the inner loop policy update during meeaa-raining:; during meta-test, this predicted advantage is discarded. We prove the universality of the enriched policy update in Theorem 2 in Appendix AA] We observe empirically the practical impact of the universality property with an ablation study presented in Figure[[(lett))
To make predictions for both the AWR loss and advantage regression, our policy architecture has two output heads corresponding to the predicted action given the state, no (-(s), and the predicted advantage given both state and action as. This architecture shown Figure Policy adaptation then proceeds follows:
0/+ an Vo. Dr
where ===LAW + XLADY
In our policy update, we show only one gradient step for conciseness of notation, but it can be easily extended to multiple gradient steps. The AWR loss is given in Equation [ and the advantage regression loss is given by:
CADY (0, (0,0',
E S
[AA(s) (Ro(s, Vor
(4)
Adapting with L. rather than LAWR addresses the expressiveness problems noted earlier. This adaptation process is done both in the inner loop of meta-training and during meta-test time, as outlined in Algorithm 2 MACAW is consistent at meta-test time because it executes a well-defined RL fine-tuning subroutine based on AWR during adaptation. Next, we describe the meta-training procedure for learning the meta- parammrrrsssess 0 and Φ the initializations of the policy and value function, respectively.
4.2 OUTER-LOOP MACAW PROCEDURE
To enable rapid adaptation at meta-test time, we meta-train set of initial parameters for both the value function and policy to optimize the AWR losses Ly and CAWR, respectively, after adaptation (L9-10 in Algorithm Db. We sample a batch of data D for the outer loop update that is disjoint from the adaptation data D in order to promote few-shot generalization rather than memorization of the adaptation data. The meta-learning procedure for the value function follows MAML, using the supervised Monte Carlo objective:
min ET, [Lv D -min [Cv(o- - 171 (o, D)), DT))
(5)
where Ly is defined in Equation This objective optimizes for a set of initial value function parameters such that one or few inner gradient steps lead to an accurate value estimator.
Unlike the inner loop, we optimize the initial policy parameters in the outer loop with standard addaanaago-eeighee regression objective, since expressiveness concerns only pertain the inner loop where only small number of gradient steps are taken. Hence, the meta-objective for our initial policy parameters is as follows:
min T..,, [LAWR (6', 0% DT) min E,, [CAWR -n VoL. ('). D.)] 6
(6)
where C, is defined in Equation 3 and LAWR is defined in Equation Note we use the adapted value
function for policy adaptation. The complete MACAW algorithm summarized in Algorithm I
4.3 MACAW ARCHITECTURE
MACAW's enriched policy update (Equation is motivated by the desire to make inner loop policy updates more expressive. In addition to augmenting the objective, we can also take an architectural approach to increasing gradient expressiveness. Recall that for an MLP, single step of gradient descent can only make a rank-1 update to each weight matrix. Finn and Levine (2018) show that this implies that MLPs must be impractically deep for MAML. to be able produce any learning
5

---

Under review as a conference paper at ICLR 2021
Cheetah-Direction
Cheetah-Velocity
100
-150
Peward
-200
400
-250
-300
10f
10a
10
100
Training Steps
Training Steps
Walker Params
Ant Direction
400
Reward
300
Roward
200
Offinna MT-FT
MACAW
200
Offinne PEARL
Meta BC
100
-20
10
10'
10a 10c
10a
10
10a
10c
Training
Training
Figure 3: Comparing MACAW with (i) an offiine variant of PEARL (Rakelly et al. 2019). state-of tooff-hh-ann art off-policy meta-RL. method, (ii) offline multi-task training fine tuming method based on AWR (Peng al.) 2019). and (iii) a meta-behavior cloning baseline. Shaded regions show one standard error of the reward of four seeds. MACAW is the only algorithm to consistently outperform the imitation learning baseline, and also learns with the fewest number of training steps in every environment (note the log x axis).
procedure. However, we can shorteut this rank-1 limitation with relatively simple change the layers of an MLP, which we call a weight transform layer. This layer maps latent code into the weight matrix and bias, which are then used to compute the layer's output just as in a typical fully-connected layer. This 'layer-wise linear hypernetwork' (Ha et al. 2016) doesn't 't change the class of functions computable by the layer on its input, but increases the expressivity of MAML's gradient. Because we update the latent code by gradient descent (which is mapped back into new weight matrix and bias in the forward pass) we can, in theory, acquire weight matrix updates of rank up to the dimensionality of the latent code. We use this strategy for all of the weights in both the value function network and the policy network. This architecture is similar to latent embedding optimization (LEO) (Rusu al. 2019), but the choice of using simple linear mapping functions allows us to apply weight transform layers to the entire network while still providing more expressive gradients. For more detailed explanation of this strategy, see Appendix B. Our experiments find that this layer significantly improves learning speed and stability.
5 EXPERIMENTS
The primary goal of our empirical evaluations is to test whether we can acquire priors from offline multi-task data that facilitate rapid transfer to new tasks. Our evaluation compares MACAW with three sensible approaches to this problem: a meta- imitation learning, multi-tast offline RL with fine-tuning, and offline variant of the state-of-the-art off-policy meta-RL method, PEARL (Rakelly eet al. 2019). Further, we analyze a) the importance of MACAW's enriched policy update (Equation 3) in various data quality regimes; b) the effect of the proposed weight transformation; and c) how each method's performance is affected when the sampling of the task space during training is very sparse. The first two settings highlight the differences between MACAW and the naive combination of MAML and AWR; the third setting represents realistic setting where fewer tasks are available during meta-traing traininniiing...... See Appendix Cfor additional experiments a) ablating the weight transform layer b) investigating the performance of MACAW and PEARL. when online fine-tuning is available and c) a richer task distribution.
For our experiments, we construct offline variants of the widely-used simulated continuous control benchmark problems introduced by Finn et al. (2017a); Rothfuss et al. (2018), including the half- cheetah with varying directions and varying velocities, the walker with varying physical parameters, and the ant with varying directions. If not noted otherwise, the offline data for each experiment is generated from the replay buffer of RL agent trained from scratch. This reflects practical scenario where an agent has previously learned a set of tasks via RL, stored its experiences, and now would like to quickly learn related task. Data collection information is available in Appendix D

---

Under review as a conference paper at ICLR 2021
Ablating MACAW's Enriched Policy Uodate
Ablating MACAW's Weight Transform
Test Porformance with Task Sampling
VACAN
Asmyptonic Reweord
-100
PEAAL
Reward
Rewerd
-100
-150
A
-150
-200
200
200
HALLAY WT Equall Weltt Noa Wilttqual Perurrs
Mediarn Data
-300
-250
MAA/L HUM
-250
100
200
1.00
200
20
10
(thousands)
Training Steps (thousends)
Number of Training Tasks
Figure Left: Ablating MACAW's ' enriched policy update when varying the quality of the inner loop adaptation data. Solid lines correspond to MACAW, dashed lines correspond to MACAW without the auxiliary policy loss (cquivalently, MAML+AWR with weight transforms). Both perform similarly with good quality adaptation data (orange), but the policy adaptation step without the auxiliary loss begins to fail as adaptation data is increasingly sub-optimal (blue and red). Bad, medium, and good data correspond to the first, middle, and last 500 trajectories from the lifetime replay buffer of the behavior policy for each task; see Appendix [D] for learning curves of the individual offline policies. Center: Ablating MACAW's s weight transform layer in the same experimental setting as the cheetah-velocity experiment in Figure 3 Without the additional expressiveness. learning is much slower and less stable. Right: Train task sparsity split performance of MACAW, Offline PEARL, and Offfine MT+fine tune. Each curve corresponds to the performance a method as the number of tasks available for training is varied. MACAW shows the most consistent performance when different numbers of tasks are used, performing well even when only three tasks are used for training.
Can we learn to adapt to new tasks quickly from purely offline data? Our first evaluation compares three approaches to the offline meta-RL problem setting, testing their ability to leverage the offline task datasets in order to quickly adapt to a new task. Specifically, we compare MACAW with i) offline PEARL (Rakelly al. 2019), ii) multi-task AWR (Peng et al. 2019), which uses 20 steps of Adam (Kingma and Ba 2015) to adapt to new task at meta-test time (Offline MT+FT) and iii) a meta-behavior cloning baseline. We choose PEARL and AWR because they achieve state-of-the-an performance in off-policy meta-RL. and offline RL, respectively, and are readily adaptable to the offline meta-RL problem. As in iakeakllyy et al. (219)) for each experiment, we sample a finite set of training tasks and held out test tasks upfront and keep these fixed throughout training. Figure shows the results. We find that MACAW is the only algorithm consistently outperform the meta- becaab cloning baseline. Multi-task AWR fine-tuning makes meaningful progress on the simpler cheetah problems, but it is unable to adapt well on the more challenging walker and ant problems. Offline PEARL shows initial progress on theetah-velocity and walker-params, but struggles to make steady progress on any of the problems. We attribute PEARL's failure to Q-function extrapolation error, a problem known to affect many off-policy RL algorithms (Fujimoto et al. 2019). as well generally unstable offline bootstrapping. MACAW's and AWR's 's value function is gootstrap-free and their policy updates maximize weighted maximum likelihood objective during training, which biases the policy toward safer actions (Peng et al. 2019), implicitly avoiding problems caused by extrapolation error. In contrast to Offline PEARL and multi-task AWR, MACAW trains efficiently and relatively stably on all problems, providing an effective approach to learning representations from multi-task offline data that can be effectively adapted to new tasks meta-test time.
How does MACAW's performance differ from MAML+AWR? MACAW has two key features distinguishing it from MAML+AWR: the enriched policy loss and weight transform layers. Here, we use the Chectah-Velocity setting to test the effects of both of these changes. We first ablate the enriched policy loss used in MACAW's inner loop update. This experiment compares MACAW and weeigt transform layers, which optimize Equation 3 and Equation [ in the policy inner-loop, respectively. To identify when policy update expressiveness is most crucial, we repeat this ablation study three times, meta-training and meta-testing with various qualities of inner-loop data, using good outer loop data for all experiments. Figure O (left) shows the results. MAML+AWR performs well when the offline adaptation data comes from a near-optimal policy. which is essentially a one-shot imitation setting (orange); however, when the offline adaptation data comes from a policy pre-convergence, the difference between MACAW and MAML+AWR becomes significant (blue and red). This result supports the intuition that policy update expressiveness is of greater importance when the adaptation data is more random, because in this case the adaptation data includes a weaker signal from which to infer the task (e.g. the task cannot be inferred by simply looking at the states visited). Because an agent is unable to collect further experience from the environment during offline adaptation, it is effectively data. An important property of a meta-RL algorithm is thus its robustness to sub-optimal behavior policies, a property that MACAW exhibits. Next, we ablate the weight transform layers, comparing
the mercy of the quality of the behavior policy that produced the
7

---

Under review as a conference paper at ICLR 2021
MAML+AWR+eniiche policy update with MACAW. Figure 4 (center) suggests that the weight transform layers significantly improve both learning speed and stability. The No WT-Equal Width variant removes the weight transform from each fully-connected layer, replacing it with regular fully-connected layer of equal width in the forward pass. The No WT-Equal Params variant replaces each of MACAW's s weight transform layers with regular fully-connected layer of greater width, to keep the total number of learnable parameters in the network roughly constant. In either case, we find that MACAW provides a significant improvement in learning speed, as well as stability when compared to the Equal Width variant. Figure 5 in the appendix shows that this result is consistent across problems.
How do algorithms perform with varying numbers of meta- meta-training tasks? Generally, we prefer an offline meta-RL algorithm that can generalize to new tasks when presented with only a small number of meta- veta-training tasks sampled from p(T). In this section, conduct experiment to evaluate the extent to which various algorithms rely on dense sampling of the space of tasks during training in order generalize well. We compare the test performance of MACAW, offline PEARL, and offline multi-task AWR + fine-tuning as we hold out an increasingly large percentage of the Cheetah-Velocity task space. The results are presented in Figure [ (right). Surprisingly, Offline PEARL completely fails to learn both when training tasks are plentiful and when they are scarce, but learns relatively effectively in the middle regime (5-20 tasks). In our experiments, we often observe instability in Offline PEARL's task inference and value function networks when training on too many offline tasks. On the other hand, with too few tasks, the task inference network simply learns a degenerate solution, providing no useful information for the value functions or policy to identify the task. The multi-task learning- + fine-tuning baseline exhibits a steadier degradation in performance as training tasks are removed, likely owing to its bootstrap-fiee learning procedure. Similarly to Offline PEARL, it is not able to learn a useful prior for fine-tuning when only presented with 3 tasks for training. However, MACAW finds solution of reasonable quality for any sampling of the task space, even for very dense or very sparse samplings of the training tasks. In practice, this property is desirable, because it allows the same algorithm to scale very large offline datasets while still producing useful adaptation behaviors for small datasets. Ultimately, MACAW effectively exploits the available data when meta-training tasks are plentiful and shows by far the greatest robustness when tasks are scarce, which we attribute to its 3GD-based adaptation procedure during both meta- veta-training and meta- testints
6 RELATED WORK
Meta-learning algorithms enable efficient learning of new tasks by learning elements of the learning process itself (Schmidhuber 1987; Bengio et al. 1992] Thrun and Pratt 1998; Finn 2018). We specifically consider the problem of reia-reenfor learning. Prior methods for meta-RL. can generally be categorized into two groups Contextual meta-RL methods condition a neural network on experience using a recurrent network (Wang et al. [2166] [Duan et al.) 2016; Fakoor et al. 2020), recursive network (Mishra et al., 2017), or stochastic inference network (Rakelly et al. 2019: Zintgraf et al. 2020; Humplik et al., 2019 Sæmundsson et al. 2018). Optimization-
meta-RL methods embed optimization procedure such as gradient descent into the meta-level optimization (Finn et al. 2017a| Nagabandi et al. 2019 Rothfuss et al. 2018) Zintgraf et al. 2019 al. 2019 Yang et al. [2019), potentially using learned loss function (Houthooft et al.] 2018) |echtle et al. 2019: Kirsch et al. 2020b(a). In prior works, the former class of approaches tend to reach higher asymptotic performance, while the latter class is typically more robust to out-of-distribution tasks, since the meta-test procedure corresponds to a well-formed optimization. Concurrent work by Dorfman and Tamar (2020) investigates the offline meta-RL setting, directly applying an existing meta-RL RL algorithm, VariBAD (Zintgraf et al. 2020). to the offline setting. The proposed method further assumes knowledge of the reward function for each task to relabel rewards and share data across tasks with shared dynamics. MACAW does not rely on this knowledge nor the assumption that some tasks share dynamics, but this technique could be readily combined with MACAW when these assumptions do hold.
Cupta et al.] 2018, Mendonca
Unlike these prior works, we aim to develop an optimization-baser meta-RL algorithm that can both learn from entirely offline data and produces monotonic learning procedure. Only a handful of previous model- frre meta-RL methods leverage off-policy data at all (Rakelly et al.] 2019 Mendoncal
et al. 2019). and none have considered the fully offline setting. Guided meta-policy search Mendoncal ct al. 2019) is optimization -aaiaatiiin-aaddddd... but is not applicable to the batch setting as it partially relies on
policy gradients. This only leaves PEARL (Rakelly et al. 2019) and its relatives (Fakoor et al.! 2020
8

---

Under review as a conference paper at ICLR 2021
which correspond to contextual meta-learning approach that is sensitive to the meta-training task distribution without fine-tuning (Fakoor et al., 2020) at test time. We also compare to PEARL, and find that, expected, it performs worse than in the off- aff-policy setting, since the fully offline setting is substantially more challenging than the off-policy setting that was designed for.
The proposed algorithm builds the idea of batch off- off-policy offline reinforcement learning (Fuji moto et al. 2019 Kumar et al. 2019b Wu et al., 2019 Levine et al.) 2020 Agarwal et al. 2020p. extending the problem setting the meta-learning setting. There are number of recent works that have demonstrated successful results with offline reinforcement learning and deep neural net-
works (Fujumoto et 2019; Daques et 2019 Kumar et al., 2019a Wu et al. 2019: Peng et al.) 2019 Agarwal et al., 2020). We specifically choose to build upon the advantage-wen weighted regression (AWR) algorithm (Peng et al., 2019). We find that AWR performs well without requiring dynamic programming, instead using Monte Carlo estimation to infer the value function. This property is ap- pealing, as difficult to combine truncated optimization-bn et al. 2017a) with TD learning, which requires a larger number of gradient steps to effectively back-up values.
meta-learners such as MAML (Finn)
7 CONCLUSION
In this work, we formulated the problem of offline meta-reinforcemeen learning and presented MACAW, a practical algorithm that achieves good performance on various continuous control tasks compared with other sfate-of-the-an meta-RL algorithms. We motivated the design of MACAW by the desire to build an offline meta-RL algorithm that is both sample-efficient (using value-based RL subroutines) and consistent (running full-fledged RL algorithm at test time). We hope that this work serves as the basis for future research in offline meta-RL, enabling more sample-efficient learning algorithms to make better use of purely observational data from previous tasks and adapt to new tasks more quickly.
We consider fully offline meta- teeta-training and meta-testing with and without online fine-tuning, showing that MACAW is effective both when collecting online data is totally infeasible as well as when some online data collection is possible meta-test time. However, an interesting direction for future work is to consider how we might enable online adaptation from purely offline meta-training while preserving the consistency property of MACAW. This would require an offline strategy for learning to explore, problem that has largely been considered in on-policy settings in the past (Gupta et al. 2018 Zintgraf et al. 2020) but also recently in offline settings (Dorfman and Tamar 2020).

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta-learning via learned loss. arXiv preprint arXiv:1906.05374, 2019.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, volume 2. Univ. of Texas, 1992.
Tristan Deleu, Tobias Würfl, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio. Torchmeta: Meta-Learning library for PyTorch, 2019. URL htt.://s//aaav...or//////1...... Available at: https://github.comm//
Ron Dorfman and Aviv Tamar. Offline meta reinforcement learning. arXiv preprint arXiv:2008.02598. 2020.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-learning. In International Conference on Learning Representations, 2020.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gra- dient descent can approximate any learning algorithm. International Conference on Learning Representations, 10 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning. 2017a
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. CoRL,
2017b. URL http:://axxii.orrtt
1709. 04905]
Chelsea Finn. Learning to Learn with Grudients. University California, Berkeley, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pages 1582-1591, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy
deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1052-2062. Long Beach, California, USA, 09-15 Jun 2019, PMLR. URL http::/pproceeiimg mlrr ppwss/vv777 wmm.:/llmmmmm....mmmmmmmmm html
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pages 5302-5311, 2018.
David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks, 2016.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.0122 100.01190.. 2018.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Pieter Abbecl. Evolved policy gradients. Advances in Neural Information Processing Systems, pages 5400-5409, 2018.
Jonathan Ho, and
Jan Humplik, Alexandre Galashov, Leonard Hasenelever, Pedro A Ortega, Yee whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint 100..099900442. 2019.
10

---

Under review as a conference paper at ICLR 2021
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy, batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: http://arxiv.orgg
6980
Louis Kirsch, Sjoerd van Steenkiste, and Juergen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In International Conference on Learning Repre- sentations, 2020a. URL
Louis Kirsch, Sjoerd van Steenkiste, and Jurgen Schmidhuber. Improving generalization in meta- reinforcement learning using learned objectives. ICLR, 2020b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019a.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing policy q-learning via bootstrapping error reduction. NeurIPS, 2019b. URL http::/aaaxx...rg/////////// 6906 00949
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through proba- bilistic program induction. Science, 133200666666)133133333333 December 2015. doi: 10.1126/science. aab3050. URL http::ttss//// org 1 wwwssscieecccccc aab3050
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta- meta-policy search. In Advances in Neural Information Processing Systems, pages 9653-9664, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. Meta-learning with temporal convolutions. arXiv: 2017.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abheel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. International Conference on Learning Representations (ICLR), 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement
learning with offline datasets, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightte regression:
Simple and scalable off-policy reinforcement learning, 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for op- erational space control. In Proceedings of the 24th International Conference on Machine Learning, ICML '07, page 745-750, New York, NY, USA, 2007. Association for Comput- ing Machinery, ISBN 978 9781 595937933. //doi.org/10.111 wtt.p////....1111111 1ew.111411122
doi:
012111222779912173990 URL https
1273590
Aravind Rajeswaran, Chelsea Finn. Sham M Kakade, and Sergey Levine. Mcta-lcarning with implicit gradients. In Advances in Neural Information Processing Systems, pages 113-124, 2019.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off policy meta-reinforeemenn learning via probabilistic context variables. In International Conference on Machine Learning, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint
2018.

---

Under review as a conference paper at ICLR 2021
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Confer- ence Learning Representations, 2019. URL htttps://apenrevee...e netmforufffiid B.JckhhAcK7
Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint 103.051803.0751, 2018
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The meta-metu-... hook.) Diploma thesis, Institul f. Informatik, Tech. Univ. Munich, 1:2, 1987.
Sebastian Thrun and Lorien Pratt. Learning learn: Introduction and overview. In Learning learn, pages 3-17. Springer, 1998.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. koray kavukcuoglu, and Daan Wierstra, Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 3630-3638, Cur- ran Associates, Inc., 2016. URL UtL/htttps:://prreeddingg neure
neurips wttpparipps.cpppppppeppee cctppppepppeee//////////
Jane X Wang. Zeb Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint 1rXiv:1611.05700 2016.
Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos,
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 2019.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization, arXiv preprint arXiv:1803.02000 2018.
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Jie Tan, and Chelsea Finn. Norml: No-reward meta learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiA- gent Systems, pages 323-331. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning. 2019.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning, In International Conference on Learning Representations, 2020.
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.
12

---

Under review as a conference paper at ICLR 2021
Appendix
A MACAW AUXILIARY LOSS AND UPDATE EXPRESSIVENESS
Finn and Levine (2018) lay out conditions under which the MAMI update procedure is universal, in the sense that it can approximate any function yx x*) arbitrarily well (given enough capacity), where x and y are the support set inputs and labels, respectively, and is the test input. Universality in this sense is an attractive property because it implies that the update is expressive enough to approximate any update procedure; a method that does not possess the universality property might be limited in its asymptotic post-adaptation performance because it cannot express (or closely approximate) the true optimal update procedure. In order for the MAML update procedure to be universal, several requirements of the network architecture, hyperparameters, and loss function must be satisfied. Most of these are not method method-specific in that they stipulate minimum network depth, activation functions, and non-zero learning rate for any neural network. However, the condition placed on the loss function require more careful treatment. The requirement is described in Definition Definition 1. A loss function is 'universal' if the gradient of the loss with respect to the prediction(s) an invertible function of the label(s) used to compute the loss.
We note that Definition T is necessary but not sufficient condition for an update procedure to be universal (see other conditions above and Finn and Levine (2018)). For the AWR loss function (copied below from Equation [ with minor changes), the labels are the ground truth action and the corresponding advantage Rs,, a) (s)
LAWR (s. a, 0,0 - log No (als) exp (R(s,a) (V(ss))
For simplicity and without loss of generality (see Finn and Levine (2018). Sections 4 & 5), we will consider the loss for only single sample, rather than averaged over batch.
In the remainder of this section, we first state in Theorem [ that the standard AWR policy loss function does not satisfy the condition for universality described in Definition | The proof is by simple counterexample. Next, we state in Theorem 2 that the MACAW auxiliary loss does satisfy the universality condition, enabling a universal update procedure given the other generic universality conditions are satisfied (note that the MACAW value function loss satisfies the condition in Definition] Cenatiionobeee because it uses L2 regression Finn and Levine (2018))
A.1 NON-UNIVERSALITY STANDARD AWR POLICY Loss FUNCTION
Intuitively, the AWR gradient does not satisfy the invertibility condition because it does not distinguish between small error in the predicted action that has a large corresponding advantage weight and large error in the predicted action (in the same direction) that has small corresponding advantage weight. The following theorem formalizes this statement.
Theorem 1. The AWR loss function LAWR is not universal according to Definition
The proof is by counterexample; we will show that there exist different sets of labels {ai, A (s. a1) and {az, A (s, a that produce the same gradient for some output of the model. First, rewriting Equation| with A(s, a) = (R a) - Vag (s)), we have
( A(s, a) T
CWRR (s, a, 0) log (als)) exp
Because our policy is parameterized Gaussian with fixed diagonal covariance TT again rewrite this loss as
|aa
ànll la-á,,112
AAss a) T
AWR (s. a, ā,) = log (2
+
exp
(8)
202
where å, is the mean of the Gaussian output by the policy and k dim(a). For the purpose of the simplicity of the counterexample, we assume the policy output á, is 0. The gradient of this loss with respect to the policy output is
1 0
A(s a) T
a L AWR
a,,a,0) =
exp
a
13

---

Under review as a conference paper at ICLR 2021
To demonstrate that the gradient operator applied to this loss function is not invertible, pick two distinct label values and show that they give the same gradient. We pick a = [I.....]]' A (s a = T and a2 [0.1,... 0.1]". A2(s. gives gradients 91 = 77 1 1]² and 92 = o²
log( 10)T. Inserting these values into Equation A.1 this
10e [0.1,.... 0.1]" = デ [1,....13
91- Thus
the gradient of the standard AWR loss does not possess sufficient information recover the labels uniquely and using this loss for policy adaptation does not produce universal policy update procedure. Next, we show how the auxiliary loss used in MACAW alleviates this problem.
A.2 UNIVERSALITY OF THE MACAW POLICY ADAPTATION LOss FUNCTION
In this section, we show that by adding an additional term to the AWR loss function, we acquire loss that satisfies the condition stated Definition which we state in Theorem [ Intuitively, the additional loss term allows the gradient to distinguish between the cases that were problematic for the AWR loss (large action error and small advantage weight vs small action error and large advantage weight).
Theorem 2. The MACAW policy loss function C, is universal according to Definition
i
The MACAW policy adaptation loss (given Equation 33 is the sum the AWR loss and an auxiliary advantage regression loss (the following is adapted from Equation [):
1
+ |a
a 11
AA(s T + A(A(s,a) A)²
(s, a. sy, A)
loog
exp
(2TO²)
20
where the predicted advantage output from the policy advantage head and is the advantage regression coefficient. The gradient of this loss with respect to the predicted advantage A is
!ADV V Ar a, A) 22(A A(s,
(9)
and the gradient of the loss with respect to å. is
A(s, T
gAWR Va, Cx, ay. A)
exp
(âp a)
(10)
SADV
We write the combined gradient g
In order to provide universal update procedure,
gawR
must be able to recover both the action label a and the advantage label A(s, a) from g. First, because JADV is an invertible function of Ass, a), we can directly extract the advantage label by re-arranging
Equation g
A(s, a) JADV -2X
-2A
Similarly, gawR an invertible function of a, so we can then extract the action label by re-arranging Equation 10]
(Ass,a)
gAWR 72 exp
åμ
a 1 oZ exp
(1I)
A(s.a)
Because we can compute A(s, a) from JADV. there are unknowns in the RHS of Equation 11 and we can compute a (here, σ, X, and T are known constants); it is thus the additional information provided by JADV that resolves the ambiguity that is problematic for the standard AWR policy loss gradient. We have now shown that both the action label and advantage label used in the MACAW policy adaptation loss are recoverable from its gradient, implying that the update procedure is universal under the conditions given by Finn and Levine (2018), which concludes the proof.
B WEIGHT TRANSFORM LAYERS
Here, describe in detail the 'weight transformation' layer that augments the expressiveness of the MAML update MACAW. First, we start with the observation past work (Finn et al. (017b) that adding "bias transformation to each layer improves the expressiveness of the MAMI. update.
14

---

Under review as a conference paper at ICLR 2021
To understand the bias transform, we compare with typical fully-connected layer, which has the forward pass
y = a o
where x is the previous layer's activations, b is the bias vector, W is the weight matrix, and y is this layer's activations. For bias transformation layer, the forward pass is
y = a (Wx + WDz)
where z and 8 are learnable parameters of the bias transformation. During adaptation, either only the vector z or both the vector z and the bias matrix W are adapted. The vector 2' has the same dimensionality the bias in the previous equation. This formulation does not increase the expressive power of the forward pass of the layer, but it does allow for a more expressive update of the 'bias vector' 4' (in the case of dim(z) dim(b) and W I, recover the standard fully-connected layer).
For weight transformation layer (used in MACAW), extend the idea of computing the bias from latent vector to the weight matrix itself. We now present the forward pass for a weight transformation layer layer with d input and d output dimensions and latent dimension c. First, we compute w W where W R(d The first d2 components of w are reshaped into the d x d weight matrix of the layer W*, and the last d components are used as the bias vector 6*. The forward pass is then the same as a regular fully-connected layer, but using the computed matrix and bias W and D" instead of a fixed matrix and bias vector; that is y = o(W"x b*). During adaptation, both the latent vector z and the transform matrix W are adapted. We note that adapting z enables the post-adaptation weight matrix used in the forward pass, W to differ from the pre-adaptation weight matrix W* by a matrix of rank up the dimension of z, whereas gradient descent with normal layers makes rank-1 updates to weight matrices. We hypothesize it is this added expressivity that makes the weight transform layer effective. A comparison of MACAW with and without weight transformation layers can be found in Figures center and 6
C ADDITIONAL EXPERIMENTS AND ABLATIONS
C.I WEIGHT TRANSFORM ABLATION STUDY
In addition the results shown in Figure 4 (center), we include ablation of the weight transform here for all tasks. F shows these results. We find that across environments, the weight transform plays a significant role increasing training speed, stability, and even final performance. On the relatively simple cheetah direction benchmark, it does not affect the quality of the final meta-trained agent, but does improve the speed and stability of training. On the other three (more difficult) tasks, we see much more noticeable affect in terms of both training stability as well as final performance.
Additionally, we investigate the effect of the weight transform in a few-shot image classification setting. We use the 20-way 1-shot Omniglot digit classification setup (Lake et al., (2015), specifically the train/val split used by (Vinyals al. 2016) as implemented by Deleu et al. (2019). We compare three MIP models, all with 4 hidden layers:
1. An MLP with weight transform layers of 128 hidden units and a latent layer dimension of 32 (4,866,048 parameters; Weight Transform in Figure 6).
2. An MLP without weight transform layers, with 128 hidden units (152,596 parameters;
No WT-Equal Width in Figure
3. An MLP without weight transform layers, with 1150 hidden units (4,896,720 parameters;
No WT-Equal Params in Figure 6
We find that the model with weight transform layers shows the best combination of fast convergence and good asymptotic performance compared with baselines with regular fully-connected layers. No WT-Equal Width has the same number of hidden units the weight transform model (128), which means the model has fewer parameters in total (because the weight transform layers include larger weight matrix). The No WT-Fqual Params baseline uses wider hidden layers equalize the number
15

---

Under review as a conference paper at ICLR 2021
Ablating the Weight Transformation MACAW
Cheetoh-Direction
Cheetah-Velocity
-50
900
-200
-150
400
MACAMI
-200
-250
Training Steps (Uhousaras)
T0 Painnnn Stops (thosandss)
Moller--Paaamme
Ant-Direction
400
400
Besunrd
300
200
100
175
Figure 5: Ablating the weight transformation in MACAW on the MuJoCo benchmark environments. All networks have the same number of hidden units. Although MACAW able to learn with regular fully-connected layers, the weight transformation significantly improves performance on all tasks that require adaptation to unseen tasks.
of parameters the entire model with the Weight Transform model. Somewhat surprisingly, the smaller baseline model (Equal Width) outperforms the larger baseline model (Equal Params).
When using MAML-style meta-learners, it is important to consider that adding parameters to the model affects the expressiveness of both the forward computation of the model and the updates computable with finite number steps of gradient descent.
Generally, increasing the number of parameters in the model should improve the model's ability to fit the training set (because the inner loop of MAML is more expressive), which we observe here. Increasing the expressiveness of the in- ner loop of MAML can also speed convergence, which we also observe in Figure 6 However, by simply adding neurons to typical MLP, the post-adaptation model tends to overfit the train- ing set more, as see in Figure 6 On the other hand, adding parameters through weight transformation layers increases expressiveness of the adaptation step by emabling weight updates with rank greater than 1 without changing the expressiveness of the forward computation of the model.
Omniglot Weight Transform Ablation
1.0
0.8
Accuracy
0.6
Train
Val
0.4
Weight Transform
0.2
No WT-Equal Params
No WT-Equal Width
60 Training Steps (thousands)
20
40
Figure 6: Faster convergence provided by the weight transform layer (orange) Omniglot 20- way 1-shot image classification (Lake et al. 2015)
C.2 ONLINE FINE-TUNING FOR OUT-OF-DISTEE DISTRRBUTION TEST TASKS
In some cases, it may be necessary or desirable to perform online fine-tuning after the initial offline adaptation step. This the fully offline meta-RL problem with online fine-tuning described in Section 3, where an algorithm is given a small amount of initial adaptation data from the test task, just as in the fully offline setting, and then able interact with the environment to collect additional training data and perform on-policy updates. This ability to continually improve with additional training after the initial offline adaptation step is what makes a consistent meta-reinforcemen learner advantagcous.
This hybrid setting (offline training with additional online fine-tuning) is known to be extremely challenging in traditional reinforcement learning. These difficulties are clearly documented by recent work (Nair et al. 2020). In short, this setting challenging in traditional RL because while offline pre- tre-training might produce a policy that performs well, online fine-tuning often leads to a significant drop in initial performance, which take very long time recover from (see Nair al. (0200).
16

---

Under review as a conference paper at ICLR 2021
Additional Env. Steps
Offline PEARL+FT
MACAW
Reward
Improvement
Reward
Improvement
-553,4 (21.2)
-323.1 (42.9)
20k
-565.0 (4.7)
(3.5)
-279.1 (16.8)
44.0 (14.5)
200k
-533.6 (19.8)
19.8 (3.7)
-272.0 (15.2)
51.1 (12.7)
Table 1: Absolute reward as well as improvement (in terms of reward) of Offline PEARL+FT and MACAW after 0, 20k, and 200k additional environment steps are gathered and used for online fine tuning. Standard errors of the mean over the 13 test tasks are reported in parentheses. Averages are taken over 10 rollouts of each policy. We find that MACAW achieves both better out-of-distribution performance before online training as well as faster improvement during online fine-tuning. Note that Offline PEARL+FT experiences steps, compared with the performance of the policy conditioned only on the initial batch of offline data. A similar effect has been reported in recent work in offline RL (Nair al. 2020
initial drop in average performance on the test task after 20k
In many cases, online fine-tuning can take a very long time to recover the performance of the offline- only policy, if it does so all. In offline meta-RL, we have similar challenge; an offline meta-RL. algorithm must not only meta-train for good performance on single batch of offline test data, but it must also learn set of parameters that enables fine-tuning to make productive updates to its policy and/or value function without completely destroying the meta-learned knowledge about the task distribution.
In this section, use hybrid setting described above to evaluate not only MACAW's consistency (its ability to continue to improve after an initial offline adaptation step), but its ability to continue to improve even when the fest task distribution differs from the train distribution. Because significant distribution shift means that some train tasks are irrelevant, or even detrimental to test compare
performance, this setting is very difficult. In order to make meaningful comparison, with an "Offline PEARL + fine-tuning" (Offline PEARL+FT) algorithm, which is also technically consistent (because it essentially performs the SAC algorithm on the test task after the initial task inference step). However, we hypothesize that MACAW will have an advantage over this Offline PEARL+FT algorithm because while both algorithms are consistent, MACAW explicitly trains for good fine-tunability with gradient descent, unlike task inference- bassdd meta-RI. algorithms.
The training procedure for Offline PEARL+FT is the same at the regular Offline PEARL training procedure. However, at test time, after receiving an initial small batch of offline data for task inference, we alternative between performing rollouts of the task con-iionditooned policy to collect additional data from the test task and perform gradient descent on the PEARL policy and value function objectives with this off-policy data. Similarly, for MACAW test time involves first using the small batch of offline test task data to take an initial gradient step the value and policy loss functions (Equs) 2 and 3. then alternating between rolling out the adapted policy and taking more steps of gradient descent the MACAW losses.
The specific experimental setup is as follows. We partition the individual tasks in the Cheetah- Vel problem such that training tasks correspond to target velocities in the range [0,2] and test tasks correspond to target velocities in the range [2.3]. After meta-training, for each test task, we provide the algorithm with a small batch of offline data for adaptation just in the fully offline setting. However, we allow the algorithm to then collect and train on up to 200k additional interactions from the environment. Both algorithms alternate between sampling a single trajectory (200 environment interactions) and performing 100 steps of gradient descent on the aggregate buffer of data for the test task, which contains both the initial offline batch of data as well as all online data collected so far. We evaluate both algorithms on their performance after 20k and 200k additional interactions with the environment. The results of this experiment are reported MACAW achieves both higher absolute reward on the OOD test tasks as well as faster relative improvement over the offline-only adapted policy compared to the Offline PEARL#FT baseline.
Table [ We observe that
C.3 METAWORLD ML45 BENCHMARK
As additional experiment, we test the training and generalization capabilities of MACAW on a much broader distribution of tasks, and where test tasks differ significantly from training tasks (e.g- picking up object opposed to opening window hammering nail). Recently,
17

---

Under review as a conference paper at ICLR 2021
Meta-World Benchmark
0.6
Suerbge Siizets Rate
0.3
0.1
MT "ine ture
0.0
MACAN
10
10a
10a
Training
Figure 7: Average success rates of MACAW, PEARL, and MT + fine-tuning (with 20 fine-tuning steps) on the 5 test tasks the Mcta- Woold ML45 suite of continuous control tasks. Dashed line shows final PEARL average success rate after 10m training steps.
Yu et al. (2019) proposed the Meta- World (Yu et al. 2019) suite of continuous control benchmark environments more realistic distribution of tasks for multi-task and meta-learning algorithms. This benchmark includes 45 meta-training tasks and 5 meta-testing tasks. The results of this experiment
summarized in Figure 7
find that all methods are able to make meaningful progress on the test tasks, with gradient-based methods (MACAW and MT fine tune) learning much more quickly than PEARL. MACAW does achieve quite high level of performance quite early on in training; however, it begins to overfit with further training. In the regime where periodic online evaluations are available for the purpose of early stopping, we could avoid this issue, in which case MACAW would slightly underperform the multi-task learning baseline. A possible reason for some inconsistency between the performance of each algorithm on Meta-Worke and the results reported in Figure Bss the difficult scaling of the benchmark. Rewards can vary by orders of
rewards in the current version of the magnitude, from negative values to values on the order of 100,000. This has been documented to adversely impact training performance even single-task RL and increase hyperparameter sensitivity (see https /tt/t/giii thub. com/ r www.wkgrooug. hwwpuuppmmttwwrr...o
issuuss//////ss 226). Because of the problems stemming from the current reward functions in Meta- World, the maintainers of bench- mark are updating them for the next version of the benchmark, which has not been released as of November 2020.
D EXPERIMENTAL SPT-UP AND DATA COLLECTION
D.1 OVERVIEW OF PROBLEM SETTINGS
The problems of interest include:
Direction Train simple cheetah run in one of two direction: forward and 1. backward. Thus, there are no held-out test tasks for this problem, making it more "proof of concept' than benchmark.
2. Half-Cheetah Velocity Train cheetah to run at a desired velocity, which fully parameter- izes each task. For our main experiment, values of the task parameters are sampled from a uniform interval of 40 velocities in the range [0, 3]. A subset of 5 target velocities is sampled randomly for evaluation. For ablation experiments
3. Ant-2D Direction Train simulated ant with articulated joints to run in random 2D direction. For our experiments, we sample 50 random directions uniformly, holding out for testing.
4. Walker-2D Params Train simulated agent move forward, where different tasks corre- spond to different randomized dynamics parameters rather than reward functions. For our experiments, we sample 50 random sets of dynamics parameters, holding out 5 for testing.
5. Meta- Woold ML44 Train simulated Sawyer robot to complete 45 different robotics ma- nipulation tasks (for training). additional tasks are included for testing, making 50 tasks in total. Tasks include opening window, hammering nail, pulling lever, picking & placing
18

---

Under review as a conference paper at ICLR 2021
Parameter
Standard Configuration
World
Optimizer
Adam
Meta batch size
4-10
16
Batch size
256
Embedding batch size
100-256
750
KL penalty
0.1
Hidden layers
Neurons per hidden layer
300
512
Latent space size
Policy learning rate
3e-4
Value function learning rate
3c-4
Context embedding learning rate
3e-4
Q-Function learning rate
3e-4
Reward scale
5.0
Recurrent
False
Table 2: Hyperparameters used for the PEARL experiments. For the MuJoCo tasks, generally used the same parameters as reported in (Rakelly et al. 2019), with some minor modifications. The different parameters used for the World ML45 environment are reported above.
an object. See Yu ct al. (2019) for more information. Our experiments use continuous space randomization for each task setup, unlike the experiments in (Yu et al. 2019), which sample from a fixed number of task states. This creates a much more challenging environment, as seen in the success rate curves above.
For the first 4 MuJoCo domains, each trajectory is 200 time steps (as in Rakelly et al. (019)) for World trajectories are 150 time steps long.
D.2 DATA COLLECTION
We adapt each task the offline setting by restricting the data sampling procedure to sample data only from a fixed offline buffer of data. For each task, we train a separate policy from scratch, using Soft Actor-Critic (Haarnoja et al. 2018) for all tasks except TD3 (Fujimoto et 2018) as proved more stable across the various Cheetah- save complete replay buffers from the entire lifetime of training for each task, which includes 5M steps for World, 2.5M steps for Cheetah-Velocity. 2.5M steps for Cheetah-Dir, 2M steps for Ant-Direction, and IM steps for Walker-Params. We use these buffers of trajectories, one per task for each problem, to sample data in both the inner and outer loop of the algorithm during training. See Figures and for the learning curves of the offline policies for each train and test task.
Ghcetah-Vellccie
for which we use
tasks. We
D.3 ABLATION EXPERIMENTS
For the data quality experiment, we compare the post-adaptation performance when MACAW is trained with 3 different sampling regimes for the Cheetah- Vel problem setting. Bad, medium, and good data quality mean that adaptation data (during both training and evaluation) is drawn from the first, middle, and last 500 trajectories from the offline replay buffers. For the task quantity experiment, we order the tasks by the target velocity in ascending order, giving equally spaced tasks with target i
velocities Øo- 0.075, 92 0.15, .... 039 3.0. For the 20 task experiment, for training and odd for testing. For the 10 task experiment, we move every other train task to the move every other remaining train task to the test set (e.g. tasks / = 4, 12, 20, ...), and for the 3 task experiment, we again move every other task the test set, that the train set only contains tasks 0, 16, and 32. Task selection was performed this way to ensure that even in sparse task environments, the train tasks provide coverage of most of the task space.
9: with
test (e.g. tasks 2, 6, 10. ...). For the 5 task experiment,
E IMPLEMENTATION DETAILS AND HYPERPARAMETERS
Peng et al. 2219)) note several strategies used to increase the stability of their devaataage-weigheee regression imple implementation. We normalize the advantage logits in the policy update step to have zero mean and unit standard deviation, as in Peng et al. 2019). Advantage weight logits are also clipped
19

---

Under review as a conference paper at ICLR 2021
Cheetah Direction Offline Palicy Reward
1000
Fanodde Damoard
800
600
400
200
2.0
Training Steps (millon)
Cheetah Velocity Offiine Policy Reward (Train Tasks)
Cheetah Velocity Offline Policy Reward (Test Tasks)
-100
-100
-200
Eprante Reaard
-100
-00
-300
400
-600
2.5
15
Training Sceas (milton)
Training Stass (millio))
Ant Direction Offiine Policy Reward (Train Tasks)
Ant Direction Offline Policy Reward (Test Tesks)
800
800
neword
Lonade Hewerd
400
1,25
1.50
1.75
2.00
0.25
1.50
1.75
Training Steps (millio))
(million)
Walker Parameters Offline Policy Reward (Train Tasks)
Wallear Paramaters Offline Policy Reward (Test Tasks)
700
000
600
Echoode Hewwerd
400
200
Training (million)
Training (million)
Figure 8: Learning curves for offline policies for the 4 different MuJoCo environments used in the experimental evaluations. Each curve corresponds to policy trained on a unique task. Various levels of smoothing are applied for the purpose of easier visualization.
20

---

Under review as a conference paper at ICLR 2021
MetaWorid 45 Offline Po Policy Reward (Train Tasks)
MetaWorid 45 Offline Policy Reward (Test Tasks)
250000
200000
200000
Epnade Rewwrrd
150000
150000
100000
100000
50000
Trein ng Steps (millton)
Training Steps (million)
MetaWorid 45 Offilne Policy Success Rate (Train Tasks)
MetaWorid 45 Offiine Policy Success Rate (Test Tasks)
0.0
Suctess Rate
0.6
0.4
0.2
0.2
0.0
0.0
Training Stops (million)
Training Stops (milloon)
Figure 9: Learning curves and success rates for all tasks in the MetaWorla 45 benchmark. Each curve corresponds to policy trained on a unique task. Various levels of smoothing are applied for the purpose of plotting.
Parameter
Standard Configuration
SotadWorld
Optimizer
Adam
Value learning rate
1c-4#
1e-6
Policy learning rate
1e-4
Value fine-tuning learning rate
le-4
1e-6
Policy fine-tuning learning rate
le-3
Train outer loop batch size
256
fine-tuning batch size
256
Number of hidden layers
Neurons per hidden layer
100
300
Task batch size
Max advantage clip
Table 3: Hyperparameters used for the multi-task task learning + fine tuning baseline. *For the Walker environment, the value learning rate was le-5 for stability.
21

---

Under review as a conference paper at ICLR 2021
Parameter
Standard Configuration
Meta-World
Optimizer
Adam
Auxiliary advantage loss coefficient
le-2
le-3
Outer value learning rate
le-5
le-6
Outer policy learning rate
le-4
Inner policy learning rate
1e-3 (learned)
le-2 (learned)
Inner value learning rate
1e-3 (learned)
le-4 (learned)
Train outer loop batch size Train adaptation batch size Eval adaptation batch size Number of adaptation steps Learning rate for learnable learning rate Number hidden layers
256
256
256
256
le-3
Neurons per hidden layer
100
300
Task batch size
10
Max advantage clip
20
AWR policy temperature
Table 4: Hyperparameters used for MACAW. The Standard Configuration is used for all experiments and all environments except for Meta- Meta-World (due to the extreme difference in magnitude of rewards in Meta- World, which has typical rewards 100-1000x larger than the other tasks). For the Meta Meta-World configuration, only parameters that differ from the standard configuration are listed.
to avoid exploding gradients and numerical overflow. To train the value function, squares regression onto Monte Carlo returns, rather than TD(A). Finally, our policy is parameterized by single Gaussian with fixed variance of 0.04; our policy network thus predicts only the mean of the Gaussian distribution.
simple least
In addition using weight transformation layers instead of regular fully-connected layers, also learn learning rates for each layer of our network by gradient descent. To speed up training, we compute loss using 'task minibatch' of 5 tasks at each step of optimization, rather than using all of the training tasks. Finally, specific to the RL setting, we sample experiences in contiguous chunks from the replay buffers during train-time adaptation and uniformly non-contigguuslyy from the replay buffers for outer-loop updates and test-time adaptation. For outer loop updates, we sample data selectively towards the end of the replay buffers.
E.1 HYPERPARAMETERS
Tables 2 3 and 4 describe the hyperparameterss used for each algorithm in our empirical evaluations. performed some manual tuning of hyperparameters for all algorithms, but found that the per- formance not significantly affected for environments other than Meta-World likely due to the difficult reward scaling in the current release of Meta-World.
22


Reviews:
[{"review_id": "xojCFeLv-bv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes the problem of fully offline meta-RL. Here, the idea is to leverage offline experience from multiple tasks to enable fast adaptation to new tasks. The paper distinguishes two settings of offline meta-RL, one where only the training data is collected offline and testing corresponds to sampling online trajectories, the other where both training and testing data are collected offline. The latter is termed as fully offline meta-RL and is the problem setting considered in this work.\nThe paper also proposes a method for the fully offline meta-RL problem based on the MAML method. They argue that a naive application of MAML with advantage weighted regression (a recently proposed approach to offline RL) is insufficient for this setting and propose to make the policy more expressive by including an advantage head that regresses the advantage conditioned on the state and action. The approach is evaluated on offline versions of 4 continuous control problems.\n\nStrong Points\n- The paper is well written and the problem setting is well explained.\n- A solution is proposed for the fully offline meta-RL problem. The modifications to the policy functions are backed by theory and is also empirically verified to be helpful in the experiments.\n- Extensive ablations on the various modifications to MAML+AWR confirm that the utility of the approach for the fully offline meta-RL problem.\n- The authors also explore settings of good/bad adaptation data showing the robustness of the proposed method to quality of offline adaptation data as compared with MAML+AWR \n\nWeak Points\n- While the offline meta-RL problem setting is well motivated, I am not certain that the “fully offline” setting is as interesting. Since the idea behind meta-RL methods is to adapt to new tasks quickly, it should be generally feasible to have a small number of online trajectories for adaptation.\n- The benchmarks used for evaluation are really toy benchmarks. I would have liked to see performance on MetaWorld benchmark. Interestingly, this evaluation is present in the Appendix where the approach seems to perform worse. Moreover, one of the continuous control benchmarks (cheetah direction) in the main paper doesn’t have a held-out test and is more a proof of concept.\n\nOverall, I find the ideas proposed quite interesting. But given my hesitation with the utility of the fully offline setting as well as the experiments, I am recommending a weak accept.\n\nQuestions for the authors:\n- Can you elaborate on why PEARL doesn’t work at all on even the Cheetah-direction evaluation which doesn’t have any held-out test set?\n- How many step of gradient descent are used in the inner loop for the MACAW method? How many steps of fine-tuning for the MT+FT baseline?\n- Why is the MetaWorld evaluation not part of the main paper? Can you please comment on the worse performance of MACAW on this more realistic benchmark for meta-RL?\n- It will also help if the authors can provide more motivation for the \"fully\" offline meta-RL setting.", "extra_scores": {}, "extra_text": {}}, {"review_id": "wbV0GHHu1V8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for \"fully\" offline meta-RL. Specifically, they assume there is no interaction with the environment at all neither during meta-train nor meta-test and this method only sees previously collected data at all times. Their method is built on top of AWR [1] in which policy updates are weighted by the advantage term.  Except for section 4.3 and the last paragraph of section 4.1 (see below), the paper is well-written and easy to follow.\n\nMy comments:\n- While this paper touches on a very interesting and practical problem in meta-rl and batch-rl, I didn't find their setting is very realistic with respect to batch and offline RL setup. In batch RL, even though we assume there is a fixed data for training ( no interaction with an environment whatsoever), the final performance will be measured by interaction with the environment (i.e. policy will be evaluated in an online setting). However, this paper assumes that there is no such interaction with the environment exists. The reason the original setup of batch-rl is important because it imposes lots of challenges such as extrapolation error, over-estimation, etc which are very critical in real-worlds and make batch-lr an important problem. However, in this paper, this is not the case. The question is how the setup in this paper is sensible and important at all?\n\n- I am not sure about the \"consistent\" definition in this work and it doesn't make sense to me. Per paper definition, if an algorithm can find a good solution to test tasks regardless of the meta-training task distribution, is called consistent. My understanding from this definition is training tasks are not important at all and test-tasks can have a very different distribution from training data and still work. If that is true, why we need training tasks in the first place? based on this definition, can't we just randomly initialize the algorithm and do a meta-test and get the same results? (it would be a good experiment to do as well)\n\n- For the experiment section, this paper uses the benchmark introduced by [2, 3] and it compares with PEARL. Did you follow PEARL setup for the experiments or ProMP/MAML?  PEARL samples a fixed set of tasks at the beginning of training while ProMP and MAML samples a set of tasks at every iteration. It is important because PEARL works in the former setups not the latter. In addition, you used the same parameters as PEARL for the experiments in this paper which is not fair as experiments (i.e. fixed dataset) in this paper are different from PEARL. PEARL might get better result with hyper-parameter tuning. \n\n- I am not convinced that why one should select L_\\pi in the inner loop instead of L_{AWR}? why not just use L_\\pi in both the inner and outer loop? Can you explain the motivation for this choice? Section 4.1 didn't make the case for this selection.\n\n- Experiments are done for only 1M steps. Why 1M for all algorithms as they are different? it is like supervised learning,  you can train for more gradient setups and it is likely results will change ( with hyper-paramaters tuning).  \n\n- Looking at Figure 6, MACAW doesn't show a good performance vs. others. Why this is the case? is there anything special about this experiment that MACAW doesn't work? \n\n- What is the difference between R(s,a) in eq.1 and R(s) eq 2? shouldn't  R(s) in eq 2 be R(s,a)?\n\nminor: Figure 7 and 8 don't have legends. \n\n[1] Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019 \n\n[2] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks\n\n[3] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6O-uSnwKIeZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new problem setting in meta reinforcement learning, namely metaRL. Here the agent is trained on a fixed offline dataset, which distinguishes it from most metaRL algorithms that interact with the environment during meta training. The authors propose a gradient-based meta learning method (MACAW) to approach this problem, which uses an actor-critic method combined with advantage weighting which is an offline RL method. In experiments they show that this outperforms the offline metaRL method PEARL, and combining multi-task offline RL with AWR in a naive way.\n\nOverall I like the paper and think the problem setting is very interesting, and I like the proposed method. I have to admit I was a little bit disappointed when I came to the experiments - they're fine, but not exciting. I also have a few concerns with the practicability of this method when it comes to real-world datasets, which is one of the main motivations for this work. For now I give a score of 5, but I'm open to increase this and look forward to the author's response! \n\nPros:\n- The paper is very well written, and easy to follow. Concepts are well explained and hypotheses clearly stated.\n- The proposed problem setting is timely and relevant for the metaRL community.  I like how explicitly the problem setting is described and how it is related to existing work and problem settings.\n- The proposed method MACAW is sound and all concepts and objectives explained well. Even though there are quite a few components, I think I would be able to re-implement the method just from the paper description.\n- In the experiments on 4 MuJoCo benchmarks the proposed method outperforms two baselines, Offline PEARL and Offline MT+FT.\n\nCons: \n- I feel like the experiment section is not as strong as it could be. All the ingredients are there though and I think with some extra work the authors can easily improve their paper.\n - Throughout the paper, you stress the importance of the algorithm being consistent. However, I feel like there's an experiment missing that shows that MACAW behaves like a consistent algorithm. What I would like to see is, if there is a significant shift between training and task distribution, can MACAW recover from a bad initialisation? How long does it take, and how does this compare to say continuing to train PEARL? Figure 4 (right) slightly hints at this but given that MACAW is still quite good at 3 tasks, and with 3 tasks you can somewhat cover the range of velocities, I think this is not enough. I'm not even entirely convinced by this \"consistency\" argument (especially not with empirical evidence). The agent only has a small dataset available at test time, so why care about consistency? Consistency is *only* useful if I can actually train longer than that - but in this case, I can also just continue training whatever other metaRL algorithm I pre-trained (like PEARL, etc). So what does consistency buy me? \n - The paper could also be considerably improved with a more realistic benchmark. A main motivation for the offline RL setting is that you can use real-world data and train a metaRL agent using this. So why not use something more realistic (even something like meta-world would already be cool)? Or maybe there exists an off-line dataset with real-world data where you don't actually execute the policy in the end (because you can't) but you can somehow compute how well it does (by comparing it with expert actions for example). \n- I wonder if considering the *fully* offline dataset makes the problem too easy. I can imagine that for many real-world applications where you want to deploy the agent on new tasks, it will have to gather the data to learn about the task by itself, instead of being given this in an offline fashion (since that might often not be possible!). The authors briefly discuss this at the end of the conclusion and cite concurrent work by Dorfman and Tamar (2020) who aim to do this. Would it be possible to extend MACAW to learn good exploration policies (doesn't even have to be online, there could be a separate exploration and exploitation phase)?\n\nOther questions: \n- Are the authors concerned that the updates for the value function (Eq 2) suffers from high variance due to the Monte Carlo returns? In the MuJoCo tasks considered in the experiments this might not be such a problem due to (a) dense rewards and (b) \"good\" offline data. But could this become a problem if those two things are not given? How would you deal with this?\n - Given that the training data is pretty good since it comes from policies that are pre-trained (using \"good data\"), would it make sense to add a simpler, supervised imitation learning algorithm as a baseline? Since we're in the *fully* offline metaRL setting this should be possible?\n- Would MACAW also work on the standard metaRL setting, where the agent is interacting with the environment during meta training and therefore responsible for collecting the data itself?\n\nSide comments:\n- Sec 5: I don't understand the second sentence; do you mean \"on\" this problem? Why is MACAW listed as a \"sensible\" approach? \n- Fig 4 left: The colours in the caption don't match the one in the figure.\n\n-------------------------------------------------------------------------------------------------------------------------\nUPDATE\n\nI have read the other reviews and the author's response. \n\nThank you for your thorough answer! It's great to see you've taken all feedback into account and updated the paper significantly. After looking through the changes in the paper I'm raising my score from 5 to 6. \n\nSome last comments:\n- Several other reviewers also raised concerns that the \"fully\" offline setting might be unrealistic. I saw you added a motivation for this in Sec 3, which makes me a little bit more convinced. It would still be great to add an experiment where MACAW is adapted online at test time (entirely without offline data) like in C.2 but on in-distribution tasks.\n- I understand how the reward function issues in ML45 could be the cause of the inconsistent results. Maybe ML1 would be a better choice at this point, or indeed waiting until v2 of the benchmark is released. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-r8xOGEU9DE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary:\n    - This paper makes two contributions:\n        - 1. Formalizing the offline meta-RL paradigm, where we meta-train on pre-collected (offline) data for several RL tasks and adapt to a new task with a small amount of data. Within offline meta-RL the experiments focus on the fully offline case, where the meta-test task is also offline. It could be online with the meta-train tasks remaining offline, but then we would have to meta-learn an exploration policy, which isn't done in this work.\n        - 2. Introducing MACAW: an algorithm for offline meta-RL that has the desirable property of being consistent (i.e. converges to a good policy if enough time and data for the meta-test task are given, regardless of meta-training). To do so they rely on MAML (which provides consistency) and AWR (a simple, popular offline RL algorithm) and add a couple of changes: some hyper-network like parameterization to add capacity and adding an extra objective in the policy update to enrich the inner loop.\n\n- Pros:\n    - 1. The paper proposes an important area of research\n    - 2. Most of the experiments are well executed, using good baselines and as well as providing understanding through ablations\n    - 3. MACAW is a nice simple algorithm with good guarantees.\n\n- Cons:\n    1. I think the offline meta-RL paradigm is not introduced correctly. \n        - In particular, the paper largely borrows the meta-RL formulation from the online setting where task=MDP. It then treats the collection of the batch data as an obvious after-thought once the task is defined. However, in the offline setting the policy that generated the batch of data is of critical importance and should be part of the task definition. \n        - For instance, it is not the same to receive an MDP and examples from the perfect policy (where you can take supervised examples of a given s) than from a random policy (where there is no signal) or an adversarial policy that tries to act as bad as possible. This also has consequences at the meta-level: if at meta-training time I only see examples of perfect policies I may learn to imitate them, then at meta-test time I see an in-distribution MDP but data coming a bad policy and imitating it is a bad idea. In the definition of 'task' described in the paper this is fine since task=MDP. However, we would expect this to work poorly since the policy is out-of-distribution with those since at meta-training.\n        - This effect of the quality of the data is used in the experiment of figure 4 left, which makes the experiment good (a Pro), but does not detract from having (IMO) the wrong formalization.\n        - This is a big minus since this formalization is one of the big contributions of the paper and it could affect further papers in that area. However, there is a chance I am wrong since the authors have spent months on this and I've spent only some hours reviewing the paper.\n\n    2. One of the two improvements over MAML+AWR, the weight transform, is not fully justified:\n        - From a conceptual point of view, it's not clear to me why this is the first time MAML has needed this change after being used in tens/hundreds of experiments. What is different on this task that hasn't been true in any other task in the past? I understood we're doing it to increase the representation capabilities of the gradient, but wouldn't this be useful for other meta-learning tasks? If so, why wasn't it used in the past? (specially since the bias-only version of the idea was already proposed by Finn et al. in 2017)\n        - From an experimental point of view I couldn't find the details of the ablated version on the main text appendix B, D or E. Therefore, I understood we're simply changing it to just a weight matrix of the same dimension. If that's the case one could argue that to make things comparable we should try increasing the width by a factor of $\\sqrt{c}$ (c defined as in the appendix) to have roughly the same number of parameters, as well as possibly the depth, since that would be a simpler change with a similar latency to the weight transform version.\n\n    3. [edited post-rebuttal to correct inaccuracy] The fact that there is a recent/concurrent paper is not ideal. However, I didn't weight it in my consideration.\n- Clarity: pretty high\n- Significance: somewhat high, except for similar concurrent NeurIPS work\n- Questions:\n    - Figure 4 left I didn't understand if the quality changed at meta-test only or both at meta-train and meta-test.\n    - Any conjecture on why PEARL performance goes down in cheetah-velocity?\n    - Doesn't AWR rely on the policy providing the data being somewhat good already? Otherwise the advantage function may be very different for the policy that generated the data vs. the optimal policy.\n- Details:\n    - Figure 3 has logarithmic x axis and figure 4 (left, center) have linear x axis. It may be better to keep them the same.\n    - \"An important property of a meta-RL algorithms is thus its robustness\" --> _offline_ meta-RL?\n    - In Related work Kirsch 2020b,a seems to be the same paper cited twice\n- Summary of review: because I believe the formalization has a big flaw and this paper is mainly about the formalization, I have to recommend rejection. I also have major concerns regarding the weight transform and the experiments that were done to prove its usefulness. The paper is otherwise good, interesting, well-written, and timely; I'm looking forward to the discussion and updating my review if my initial assessment was wrong.\n\n============\nUpdate after discussion with authors\n\nI had two main concerns:\n- The modification to MAML was unconvincing to me.\n- The offline meta-RL formulation should include behavior policy as part of the task definition.\n\nAfter a very detailed response from the authors, I am now happy with the response and extra experiments w.r.t. the MAML modification, but I still have concerns about the formulation. In particular, reading the final version I still think the policy giving the behavior data is treated as an after-thought and is instead assumed constant across all tasks. For instance, IMO figure 1 should contain multiple examples of the same \"RL task\" that are different \"offline RL tasks\"; i.e. learning to swim using guidance from a 3-year-old and learning to swim using guidance from Michael Phelps. This is one of the key differences, IMO, between meta-RL and offline meta-RL and given that this paper's main contribution is introducing offline meta-RL, I feel it really should be very clear about this point. It may be fine to first introduce the correct general version and then say something like \"it may be useful to assume each RL task is given by an expert of roughly the same characteristics\", i.e. we can assume behavior policy is constant across tasks. However, right now the original formulation directly borrows from regular meta-RL and I believe that may confuse future papers in offline meta-RL.\n\nI've increased my score from 4 to 5 since I'm now less concerned about the MAML improvement, but I cannot recommend acceptance given my concern about the formulation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for offline meta-RL, where we meta-train on pre-collected offline data for several RL tasks and adapt to a new task with a small amount of data. The paper assumes that there is no interaction with the environment either during meta-train or meta-test.  In this setting, motivated by the ide of leveraging offline experience from multiple tasks to enable fast adaptation to new tasks, the paper introduces MACAW, which combines the consistent MAML and the popular offline AWR, improving upon them by adding capacity through parameterization and adding an extra objective in the policy update. As a result, the MACAW proposed for the offline meta-RL has the desirable property of being consistent, i.e., converging to a good policy if enough time and data for the meta-test task are given, regardless of meta-training. 


Pros: 
+ Most of the experiments are well executed, using good baselines. Extensive ablations on the various modifications to MAML+AWR confirmed the utility of the approach for the fully offline meta-RL problem.
+ MACAW is a simple algorithm with theoretical guarantees; the modifications to the policy functions are backed by theory. 


Cons:
- The reviewers have concerns on the formulation of offline meta-RL. One major contribution of the paper is to introduce offline meta-RL. However the paper largely borrows the meta-RL formulation from the online setting where task=MDP. The reviewers think that directly borrowing from regular meta-RL as the formulation of offline meta-RL might be misleading. The reviewers suggest including behavior policy as part of the task definition for offline meta-RL formulation.

- Several reviewers raised concerns that the fully offline setting might be unrealistic. Although the author did add a motivation, the reviewers would be interested in seeing MACAW being adapted online at test time on in-distribution tasks. 

- Unfortunately, the authors accidentally revealed their names in one of the modified versions. 

Decision: Reject