Pre-training as Batch Meta Reinforcement Learning with tiMe 
Authors: Quan Vuong
Shuang Liu
Minghua Liu
Kamil Ciosek
Hao Su
Henrik Iskov Christensen
Venue:  (2020)
Keywords: Reinforcement Learning
Deep Reinforcement Learning
Meta Reinforcement Learning
Batch Reinforcement Learning
Transfer Learning

Abstract: Pre-training is transformative in supervised learning: a large network trained with large and existing datasets can be used as an initialization when learning a new task. Such initialization speeds up convergence and leads to higher performance. In this paper, we seek to understand what the formalization for pre-training from only existing and observational data in Reinforcement Learning (RL) is and whether it is possible. We formulate the setting as Batch Meta Reinforcement Learning. We identify MDP mis-identification to be a central challenge and motivate it with theoretical analysis. Combining ideas from Batch RL and Meta RL, we propose tiMe, which learns distillation of multiple value functions and MDP embeddings from only existing data. In challenging control tasks and without fine-tuning on unseen MDPs, tiMe is competitive with state-of-the-art model-free RL method trained with hundreds of thousands of environment interactions.

Under review as a conference paper at ICLR 2020
META REINFORCEMENT LEARNING FROM OBSERVA- TIONAL DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-training is transformative in supervised learning: large network trained with large and existing datasets can be used as an initialization when learning a new task. Such initialization speeds up convergence and leads higher performance. In this paper, we seek to understand what the formalization for pre-training from only existing and observational data in Reinforcement Learning (RL) and whether it is possible. We formulate the setting as Batch Meta Reinforcement Learning. We identify MDP mis-identification be central challenge and mo- tivate it with theoretical analysis. Combining ideas from Batch RI. and Meta RL, we propose tiMe, which learns distillation of multiple value functions and MDP embeddings from only existing data. In challenging control tasks and without additional exploration on unseen MDPs, tiMe is competitive with state-of-the-art model-free RL method trained with hundreds of thousands of interactions. This work demonstrates that Meta RL from observational data is possible and we hope it will gather additional interest from the community tackle this problem.
INTRODUCTION
Deep Reinforcement Learning algorithms still require millions of environment interactions to obtain reasonable performance, hindering their applications (Mnihetal 2015 Lillicrap al. 2016 Vuong
et al. 2018 Fujimoto et al. 2018b] Jaderberg et al.) 2018 Arulkumaran et al. 2019
Huang et al.
2019 This is due to the lack of good pre-training methods. In supervised learning. pre-trainee trained network significantly reduces sample complexity when learning new tasks (Zeiler & Fergus 2013
Devlin et al.] 2018 YYng et al., 2019). Meta Reinforcement Learning (RI. has been proposed as framework for pre- teeiirrin in RL (Wang et al.) 2016a; Duan et al. [206]] Finn al., (017)) However, such methods still require the collection of millions of interactions during meta-train, which means that they face the same sample complexity challenge standard RL algorithms. In this work, use the following definition of pre-training: the ability to use data from a set of tasks to improve performance on unseen, but related tasks.
In supervised learning, a key reason why pre-training is incredibly successful is that the dataset used for pre-training be collected from naturally occurring large-scale processes. This removes the need to manually collect data and allows for scalable data collection, resulting in massive datasets. For example, Mahajan et al. (2018) pre-trains using existing images and their corresponding hash- tags from Instagram to obtain state-of-the-an performance on ImageNet (Russakovsky et al. 2014).
In this paper, we seek to formalize pre-training in RL in a way that allows for scalable data collec- tion. The data used for pre- pre-training should be purely observational and the policies that are being optimized for should not need to interact with the environment during pre-training. To this end, we propose Batch Meta Reinforcement Learning (BMRL) as formalization of pre-training in RI. from only existing and observational data. During training, the learning algorithms only have access to a batch of existing data collected priori from a family of Markov Decision Process (MDP). During testing, the trained policies should perform well on unseen MDPs sampled from the family.
A related setting is Batch RL (Antos et al. 2007 Lazaric et al., 2008 Lange et al. 2012). which we emphasize assumes the existing data comes from a single MDP. To enable scalable data col- lection, this assumption must be relaxed: the existing data should come from family of related MDPs. Consider smart thermostats, whose goal is to maintain a specific temperature while mini- mizing electricity cost. Assuming Markovian dynamics, the interactions between thermostat and

---

Under review as a conference paper at ICLR 2020
its environment can be modelled as a MDP. Data generated by a thermostat operating in a single building can be used to train Batch RL algorithms. However, if we consider the data generated by the same thermostat operating in different buildings, much more data is available. While the inter- actions between the same thermostat model and different buildings correspond to different MDPs, these MDPs share regularities which can support generalization, such the physics of heat diffu- sion. In sesttction 6) we further discuss the relations between BMRL and other existing formulations.
The first challenge in BMRL the accurate inference of the unseen MDP identity. We show that existing algorithms which sample mini-batches from the existing data to perform Q-learning style updates converge to degenerate value function, a phenomena we term MDP mis disiidentifiiation. The second challenge is the interpolation of knowledge about seen MDPs to perform well on unseen MDPs. While Meta RI. algorithms can explicitly optimize for this objective thanks to the ability to interact with the environment, we must rely on the inherent generalizability of the trained networks. To mitigate these issues, we propose tiMe, which learns from existing data to distill multiple value functions and MDP embeddings. tiMe is flexible and scalable pipeline with inductive biases to encourage accurate MDP identity inference and rich supervision to maximize generalization. The pipeline consists of two phases. In the first phase, Batch RL algorithm is used to extract MDP- specific networks from MDP-specific data. The second phase distills the MDP-specific networks.
To summarize, our contributions are three folds: (1) Formulation of Meta RL from observational data as Batch Meta Reinforcement Learning (BMRL) (2) A simple saie approach which works well standard benchmarks in the BMRL setting (3) Most importantly, demonstration that Meta RL from only observational data is possible. We hope this work will direct the attention of the meta RL community towards this research direction.
2
PRELIMINARIES
2.1 BATCH REINFORCEMENT LEARNING
We model the environment as Markov Decision Process (MDP), uniquely defined 5 ele- ment tuple M, (S, Ti, R,,
with state space S. action space A, transition function T;, re- ward function RG and discount factor (Puterman 1994 Sutton & Barto 1998). At each dis- crete timestep. the agent is in a state 8, pick an action a, and arrives at the next state s' and re- ceives a reward The goal of the agent is to maximize the expected sum of discounted rewards ((x) = M, Et= 7 [& Cr x(,)) where t = (s0.i; 40.i, fo, 81.1; 01,5, 0 .i. is a tra- jectory generated by using ı interact with M. We will consider family of MDPs, defined formally in |subsection 2.3] We thus index each MDP in this family with i.
In Batch RL, policies are trained from scratch to solve single MDP M, using existing batch of N transition tuples B, = { Gr Fr,i, s(,,) I,..., N} without any further interaction with M,. At test time, we use the trained policies to interact with M, to obtain an empirical estimate of its performance J. Batch RL optimizes for the same objective as standard RL algorithms. However, during training, the learning algorithm only has access to B, and are not allowed interact with M,.
2.2
Q-LEARNING
Fujimoto et al. (2018a) identifies extrapolation error and value function divergence as the modes of failure when modern Q-learning algorithms are applied to the Batch RL setting. Concretely, deep Q- learning algorithms approximate the expected sum of discounted reward starting from a state-action So pair E r² Rise, de, s0)ss0 = s, ag = a] with a value estimate Q(s, a). The estimate can be learned by sampling transition tuples from the batch and applying the temporal difference update:
Q(s,a) + (1-a, ((s) + (r 7Q(s,
(r(+7((s ((5)))
(s') E arg max Q (s',a) (1)
aeA
The value function diverges if Q fails to accurately estimate the value of (8'). Fujimoto et al. (0118)) introduces Batch-Constraimed Q-Learning, constraining æ to select actions that are similar to actions in the batch to prevent inaceurate values estimation. Concretely, given s'. generator G outputs multiple candidate actions {am [mmm. A perturbation model g takes each state-candidate action pair input and generates small correction term {(s', am) for each candidate. The corrected
2

---

Under review as a conference paper at ICLR 2020
candidate action am + E(ss, am)) with the highest value estimated by learnt i is (s'):
iT(s)
arg max Q(s, am + E(s', am)) Our + (
am G(s', Zmm)m
3mm
(
Estimation error has also been previously studied and mitigated in model-free RL algorithms (Has- selt 2010 van Hasselt et al. 2015 Fujimoto et al. (0188)).
2.3
META REINFORCEMENT LEARNING
Meta RL optimizes for average return on family of MDPs and usually assume that the MDPs in this family share S, A, n. Each MDP is uniquely defined by tuple (T; RR). A distribution p(T, R,) defines a distribution over MDPs. During meta-train, we train a policy by sampling MDPs from this distribution and sampling trajectories from each sampled MDP, referred to as the meta-train MDPs. During meta-test, unseen MDPs are sampled from p TT Rs), referred to as the meta-test MDPs. The trained policy is used to interact with the meta-test MDPs to obtain estimate of its performance. The choice of whether to update parameters (Finn et al. 2017) ) or to keep them fixed during meta-test (Hochreiter et al. 2001 is left to the learning algorithms, both having demonstrated prior successes.
2.4 MDP IDENTITY INFERENCE WITH SET NEURAL NETWORK
A Meta RL policy needs to infer the meta-test MDP identity to pick actions with high return. Rakelly et al. (019) introduces PEARL, which uses set neural network (Qi et al. 2016; Zaheer al. 2017)
if as the MDP identity inference function. f takes as input a context set c {(s, ag, Tkk s } and
infers the identity of MDP in the form of distributed representation in continuous space. The
parameters of f is trained to minimize the error of the critic Q:
(Q(s, f(c)) (r V f()))))
where V is a learnt state value function. PEARL also adopts an amortized variational approach
Kingma & Welling 2013 Rezende al. 2014 Alemi al., 2016 Kingma & Welling 2019) to train a probabilistic which is interpreted as an approximation to the true posterior over the set of possible MDP identities given the context set.
3 BATCH META REINFORCEMENT LEARNING
Let K be the number of meta-train MDPs, N be the number of transition tuples available from each meta-train MDP, 0 be the parameter of the policy, we can formulate Batch Meta Reinforcement Learning (BMRL) an optimization problem:
8
arg max ((e = EM..DTT..R.
ETNTo.M M.
Σ
n² Ra at,i s)
(3)
where the learning algorithms only have access to the batch B during meta-train:
B B ,
B, = { Ge,s, Tt,t, s 1...,,
M4 ~ (T,,, R.)
We assume we know which MDP each transition in the batch was collected from. This assumption simplifies our setting and is used devise the algorithms. To maintain the flexibility of the for- malization, we do not impose restrictions on the controller that generates the batch. However, the performance of learning algorithms generally increases as the training data becomes more diverse.
MDP identity inference challenge To obtain high return on the unseen meta-test MDPs, the trained policies need to accurately infer their identities (Ritter et al., 2018 Gupta et al. 2018 Humplik et al. 2019). In BMRL, previously proposed solutions based on Q-learning style updates, where mini-batches are sampled from the batch to minimize TD eтror, converge to a degenerate solution. subsection 5.1 provides experimental result that demonstrates the phenomena. In finite MDP, this degenerate solution is the optimal value function of the MDP constructed by the relative frequencies of transitions contained in the batch. We formalize this statement with the following proposition.
3

---

Under review as a conference paper at ICLR 2020
83
Qg(s).a)
(*),
B
B2
Q Qivit sriit
P
Qs
Gs
initialize
K
first phuse: Batch RI.
sccond phase:
2
8j
a
distillation
E
f
learn MDP
embedding
((0,1)
(sy.)
meta-train
meta-test
(8):0g) {(ok, (ke Dhass
The above consist of two separate figures. (left) The training pipeline of tiMe in the simplest setting. (right) Architecture for the second phase when BCQ is used in the first phase.
Figure 1:
Let N(s, a, ) be the number of times the triple (s, a, s') appears B (with any reward). Performing Q-learning on finite S and A with all the Q-values Q(s, a) initialized to 0 and update rule ( where (s, a, r) is sampled uniformly at random from al each step I, will lead the Q-values to converge to the optimal Q-value of the MDP (S,A, R, 4) almost surely as long as O4 0. Er 04 =00 00, Ei-o oz 00, where
Proposition
Ls=rr if ''' NNss,,
0, (s Σ riss,,''.r)E otherwise. N(s,a,s)
SEES
R(s,a,s')
(8),
otherwise.
Σ ("),a,"") EE
Thus, performing Q-learning style update directly data sampled from the batch B fails to find good policy because the value function converges to the optimal value function of the wrong MDP. We refer this phenomena MDP mis- mis-identiffcatton. The proof is shown in subbsecctoo A.T.
Interpolation of seen MDPs to unseen MDPs challenge The trained policies need to generalize from the meta-train MDPs to unseen meta-test MDPs. Meta RL tackles this challenge by formulating an optimization problem that explicitly optimizes for the average return of the meta-trained policy after additional gradient steps in unseen MDPs (Finn al. 2017 Rothfuss al., 2018 Nichol ett al. 2018). This is possible thanks to the ability to interact with the environment during meta- train. However, in the meta-train phase of BMRL, the learning algorithms do not have access the environment. We must rely on the inherent generalizability of the trained networks to perform well on the unseen meta-test MDPs. The key challenge is therefore finding the right inductive biases in the architecture and training procedure encourage such generalization. The need to find the right inductive biases RL was highlighted by Botvinick et al. (2019); Zambaldi et al. (201)); Hessel et al. (2019). We note that previous works phrase the need to find inductive biases means to forgo generality for efficient learning. In our setting, these two goals need not be mutually exclusive.
LEARNING DISTILLATION OF VALUE FUNCTIONS AND MDP EMBEDDINGS
4.1 DESCRIPTION OF ARCHITECTURE AND TRAINING PROCEDURE
We propose a flexible and scalable pipeline for BMRL. Figure (left) provides an overview of the pipeline in the simplest setting. Meta-train comprises of two separate phases. The first phase consists of independently training value function Qi for each MDP-specifee batch B, using Batch RL algorithms. In the second phase, we distill the set of batch-specific value functions (Q; ) into super value function Qs through supervised learning (Hinton al.) 2015). Compared to the normal value function, a super value function takes not only a state-action pair as input, but also an inferred MDP identity, and outputs different values depending the inferred MDP identity.
The pipeline is flexible in that any Batch RL algorithms are applicable in the first phase. Figure (right) illustrates the architecture for the second phase given that the Batch RL algorithm used in
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1: tiMe training procedure when BCQ is used in the first phase
Input: batches {Bi).. Qs. Gs. Es. f, E, P parameterized
? Predict state-action value:
Q3 t Qs(5), (5.5).a..44 a,. M)
jointly by
(N, 1)
Qi G t-
(B:) Vi
9 Predict candidate action: <- Gs( 2, M)
2 Randomly choose B, out of [Bik
Sample transition (g) Sy,T,) from Bj
Obtain ground truth candidate action:
+ ()))
Sample context {(sk; aks sky Fk) hegs from B,
Predict correction factor: E Es(Ss, a)
Infer MDP identity:
M < f({(Sk, Ok, S/k rk))a)
L + 1| - s112 (f r)² + (Q²
Predict rr,r: 8 4- P(B(s),))),
(j(3), + all2 + || (3,,
0 + 1 VoL
the first phase is Batch Constrained Q (BCQ) Learning. As described in infsbsection 2.2 BCQ main- tains three separate components, a learnt value function Q, candidate action generator G and perturbation model ç. Therefore, the output of the first phase consists of sets {Q; hi {G),,. ). {57 J--
The second phase distills (Q (G h to Qs, Gs.Es respectively. The distillation of G and e is necessary to pick actions that lead to high return because each learnt value function Qi only provides reliable estimates for actions generated by G; and ç a consequence of the training procedure of BCQ. In addition to Qs, Gs. Es, the architecture consists of 3 other networks, f. P. E. takes imput a context {(sk, ag. Tk. 5k,) he and outputs a distributed representation of the MDP identity a fixed-dimension continuous space. The output of f is an input to Qs. Gs, Es. P. E and P predicts s', ry given state-action pair (sj; az)- P has low capacity while the other networks are relatively large. During the second phase, all networks are jointly trained end-to-end by the regression losses of predicting s', r and distilling {Qi J. ]G { Je. This is illustrated in details in Algorithm !
During meta-test, f is used to infer the identity of the meta-test MDP as a fixed-dimension contin- uous vector. The super functions Qs. Gs. Es are used to pick actions in the meta-test MDPs, using the same procedure as BCQ (subsection 2.2 The super functions also take as input the inferred MDP identity.
The key idea behind the approach is a simple stage-wise approach for the Meta RL from obser- vational data problem. In the second phase, we distill many policies into one for related tasks by jointly learning distillation of value functions and MDP embeddings. We therefore name the ap- proach tiMe. MDP embeddings refer to the ability to infer the identity of a MDP in the form of distributed representation in continuous space given a context.
4.2 BENEFITS OF THE PROPOSED PIPELINE
Inductive biases to encourage accurate MDP identification The first inductive bias is the rela- tionship between f and Qs. They collectively receive as input state-action pair (s); u,) and context {(sk, ak, Tk. s ha and regress to target value (s(s). a;). The target for each state-action pair can take on the values within the set {Q; (s), h. Similar state-action pairs can have very different re- gression targets if they correspond to different meta-train MDPs. The context the only information in the input to f and Qs that correlates with which Q"(sj, a,) out of the set (Q] a,)}: f and Qs should regress to. Thus, f and Qs must learn to interpret the context to predict the correct value for (sj, az). The second inductive bias is the auxiliary task of predicting 6/,, 3 A key design choice is that the network P which takes input E(s,, az) and f([(s), ag s(,)) and predicts s has low capacity. As such, the output of f must contain meaningful semantic information such that small network can use it to reconstruct the MDP. This is to prevent the degenerate scenario where if learns to copy its input as its output. To summarize, these two explicit biases in the architecture and training procedure encourage to accurately infer the MDP identity given the context.
Richness and stability of supervision Previous approaches update f to minimize the critic loss (subsection 2.4h. It is well-known that RL provides sparse training signal. This signal can also cause
5

---

Under review as a conference paper at ICLR 2020
-500
Velue
-1500
-2000
Batch SAC opisode return
optimal episodi
tiMe episode return
-3000
5000 12000 15000 20000 25000 30500 Number gradient steps
Figure 2: All three figures are from the 3 meta-train MDPs scenario. (left) Performance of Batch SAC in one meta-test MDP and the learnt value estimates of initial state-action pairs. The esti- mates do not diverge but are significantly higher than the actual returns, demonstrating the MDP mis-identification phenomena. In contrast, tiMe's performance is close to optimal. (middle) The behavior of the Batch SAC agent. The dark circle, red circle and dark crosses indicates the agent's starting locations, final location and the meta-train goal locations. The agent fails to find good actions during meta-test and navigates to location closest to the meta-train goals. (right) The behavior of an agent trained with tiMe. The crosses and circles indicate each meta-test MDP goal location and the agent's 's corresponding final location after evaluation the corresponding meta-test MDP. The agent trained with tiMe finds near-optimal actions in all 3 meta-test MDPs.
instability since the target values in the critic loss change over time, In contrast, our pipeline provides training signal for f that is both rich and stable. is rich because f is trained to infer representation of the MDP identity that can be used for multiple downstream tasks, such as predicting r. This encourages pereroa-purpose learnt representation and supports generalization. The training signal is also stable since the regression targets are fixed during the second phase of tiMe.
Scalability The pipeline is scalable in that arbitrary amount of purely observational data be used in the first phase so long as computational constraints permit. The data can also be het- erogeneous in the sense that they do not need contain only trajectories with high return. In the experimental section, we demonstrate the benefit of the approach when the data contains trajectories of varying qualities, some of which were generated by random policies. The extraction of the batch- specific networks, such as the batch-specific value functions {Q; J.. from the Mpe-specific batches can be trivially parallelized and scales gracefully as the number of meta-train MDPs increases.
EXPERIMENTAL RESULTS
Our experiments have main goals: (1) Demonstration of the MDP mis iitsideeniification phenomena and tiMe's ability to effectively mitigate it. (2) Demonstration of the scalability of the tiMe pipeline challenging continuous control tasks and generalization unseen MDPs.
In all experiments, the 4DP-specifi batch B; is the replay buffer when training Soft Actor Critic (SAC) (Haarnoja et al. (2018a(b) the MDP M, for a fixed number of environment interactions. Thus, the batch B; contains transitions with varying reward magnitude, some of which were gen- erated by random and poorly performing policies. While our problem formulation BMRL and pipeline tiMe allow for varying both the transition and reward functions within the family of MDPs, we consider the case changing reward function in the experiments and leave changing transition function future work. Thus, for the auxiliary prediction task in the second phase of the pipeline, P only predicts r and not
5.1 TOY EXPERIMENTS
This section illustrates MDP mis-identiffiction as the failure mode of existing Batch RL algorithms in BMRL. The toy setting allows for easy interpretability show that in the standard Batch RL setting, the Batch RL algorithm tested finds near-optimal policy. This means the failure of existing Batch RL algorithm in BMRL is not because of the previously identified extrapolation issue when learning from existing data (Fujimoto et al. 2018a
the trained agents' behavior. We also

---

Under review as a conference paper at ICLR 2020
Environment Description In this environment, the agent needs to navigate 2d-plane goal location. The agent is a point mass whose starting location is at the origin (0,0), 0). Each goal location is point on semi-circle centered at the origin with radius of 10 units. At each discrete timestep, the agent receives as input its current location (x. y), takes an action indicating the change in its position (Ar, Ay), transitions reward is the negative distance between the agent's current location and the goal location. The agent does not receive the goal location as input and | I 1, Ayy S Since the MDP transition function is fixed, each goal location uniquely defines MDP. The distribution over MDPs is defined by the distribution over goal locations, which corresponds to a distribution over reward functions.
new position (Ir + Ar., y + Ay) and receives a reward. The
Batch SAC We modify SAC to learn from the batch by initializing the replay buffer with existing transitions. Otherwise, training stays the same. We test Batch SAC on a simple setting where there is only one meta-train MDP and one meta-test MDP which share the same goal location. This is the standard Batch RL setting and a special case of BMRL. Batch SAC finds a near-optimal policy.
Three meta-train and meta-test MDPs This experiment has 3 meta-train MDPs with different goal locations. The goals divide the semi-circle into two segments of equal length. There are three meta- test MDPs whose goal locations coincides with the goal locations of the meta-train MDPs. This setting only tests the ability of the trained policies to correctly identify the meta-test MDPs and do not pose the challenge of generalization to unseen MDPs. Batch SAC was trained by combining the transitions from the 3 meta-train MDPs into a single replay buffer and sampling transitions from this buffer to perform gradient updates. Otherwise, training stays the same as SAC. [Figure 2 (left, middle) illustrates that Batch SAC fails to learn reasonable policy because of the MDP mis- identification phenomena.
Batch SAC with task inference function We also tried adding to the architecture of Batch SAC the probabilistic MDP identity inference function as described in subsection 2.4] This the equivalent of adapting PEARL (Rakelly et al.) 2019) to work in the BMRL setting. This approach fails to train a policy that performs well on all 3 meta-test MDPs. We note that the off- off-policy meta RL setting that the PEARL paper considers and the BMRL setting we consider are solving different problems. We do not argue that one easier than the other.
Performance of tiMe Since Batch SAC can extract the optimal value function out of the batch in the single meta-train MDP case, we use it as the Batch RL algorithm the first phase of the tiMe pipeline. The architecture in the second phase thus consists of E, P. ʃ and Qs- To pick an action, we randomly sample multiple actions and choose the action with the highest value as estimated by Qs. This method is termed random shooting (Chua et al.) 2018). As illustrated in [Figure 2 (left, right), tiMe can identify the identities of the three meta-test MDPs and pick near-optimal actions.
5.2 MUJOCO EXPERIMENTS
Environment Description This section illustrates the test of tiMe in challenging continuous control robotic locomotion tasks. Each task requires the application of control action to simulated robot so that it moves with a particular velocity in the direction of its initial heading. Formally, the MDP within each MDP family share 8, A, A,T, 7 and only differs in R where R is defined to be:
R(s, a,, 5') = alive bonus - alcurrent velocity target velocity| - 3||a||2
where a and B are positive constant. A one-to-one correspondence exists between MDP within the family and a target velocity. Defining family of MDP is equivalent to picking an interval of possible target velocity. This setting is instantiated on two types of simulated robots, hopper and halfcheetah, illustrated in Figurre 3 Experiments are performed inside the Mujoco simulator Todorov et al. 2012). The setting was first proposed by Finn et al. (017).
Zero-shot meta-test During testing, in contrast to prior works, we do not update the parameters of the trained networks, as is done in gradient-based meta RL, or allow for an initial exploratory phase where episode returns do not count towards the final meta-test performance, as is done in off-policy meta RL (Rakelly et al. 2019). This allows for testing the inherent generalizability of the trained networks without confounding factors. The meta-test MDPs are chosen such that they unseen during meta-train train, i.e. none of the transitions used during meta-train was sampled from any of the meta-test MDPs. At the beginning of each meta-test episode, the inferred MDP identity is initialized to zero vector. Subsequent transitions collected during the episode is used as the
7

---

Under review as a conference paper at ICLR 2020
Small Target Velocity
Medium Target Velocity
Large Target Velocity
1000
1000
800
800
800
400
400
400
SAC trained from scratch tiMe with fine tuning
200
SAC trained from scratch
SAC trained from scratch tiMe with fine tuning
with fine tuning
0
leb
1e6
lebi
Small Target Velocity
Medium Target Velocity
Large Target Velocity
-50
-50
-50
100
-150
-150
115
-00
SAC trained from scratch
SAC trained from scratch
SAC trained from scratch
-750
with
-250
tine
-250
t/Me with fine-tuning
-300
60000
-300
60000
-300
60000
Figure 3: The leftmost column illustrates hopper and halfcheetah. The remaining columns indi- cate the performance of SAC trained from scratch versus tiMe for different unseen MDPs during zero-shot meta-test. Because of these two properties, we highlight the difficult nature of obtain- ing high performance in this setting. The second forth columns correspond to small, medium, and large target velocities respectively. The first and second row indicates performance hopper and halfchectah respectively. The x-axis indicates SAC's number of environment interactions. The y-axis indicates the average episode return. The final performance of SAC is close- tose-tto-optiall in all plots in the sense that running SAC for many more timesteps will not increase its performance significantly.
context. The meta-test MDPs are also chosen to provide wide coverage over the support of the MDP distribution. This is to test whether our approach generalizes to a variety of meta-test MDPs, or simply overfit to a small region inside the support.
Meta-train conditions The target velocities of the meta-train MDPs divide the target velocity in- terval into equal segments. This removes the bias of sampling meta-train MDPs when evaluating performance. The target velocity intervals, episode length, and number of meta-train MDPs for hop- per and halfcheetah are [0.0, 2.35) and [0.0, 1.5]. 1000 and 200, 16 and 29 respectively. For hopper and halfcheetah, each meta-trait train MDP has million and sixty transitions respectively.
Performance analysis Figure 3 jillsstates tiMe's performance on unseen meta-test MDPs. tiMe is competitive with the stateoof-the-art model-free RL methods trained from scratch for million and sixty thousands environment interactions hopper and halfcheetah respectively. We perform experiments on halfcheetah with an episode length 200 because computational constraints. Pre- vious Meta RL works also use an episode length of 200 (Rakelly et al. 2019). The same network trained with tiMe also performs well in variety of different meta-test MDPs, demonstrating that does not over-fit fit to one particular meta-train MDP. We compare with SAC to demonstrate BMRL a promising research direction. We do not include other Meta RL algorithms they would require interacting with the environment during meta-train and thus, is not solving for the problem that BMRL poses. We tried removing Gss Es from the architecture Figure and picked action with Cross Entropy Method (Rubinstein & Kroese 2004). but that lead to poor performance because Qs over-estimates the values of actions not generated by Gs,Es.
baseline because
Limitations Our approach assumes that the transitions in the batch contain good enough transitions to learn a good policy in the batch RL setting. However, we note that the data in the batch contain generated by poorly performing policies. Also,
data of varying qualities, some of which approach has only been demonstrated to work on tasks where reset are not crucial for exploration needed for task inference, e.g. sparse reward setting. We leave this venue for future work.
8

---

Under review as a conference paper at ICLR 2020
6 RELATED WORKS
Supervised Learning and Imitation Learning The main differences between Batch (Meta) RL and supervised learning are: actions have long-term consequences and the actions in the batch are not assumed to be optimal. If they are optimal in the sense that they were collected from an expert, Batch RL reduces to Imitation Learning (Abbeel & Ng. 2004, Ho Ermon 2016). In fact, Fujimoto et al.
(2018)) demonstrates that Batch RL generalizes Imitation Learning discrete MDPs.
Meta RL Equation 3 is the same objective that existing Meta RL algorithms optimize for WWng et al. 2016b);[Fim) et al. (2017)). We could have formulated our experimental setting as a Partially Observable MDP, but we chose to formulate it as Batch Meta Reinforcement Learning to ensure consistency with literature that inspires this paper. The main difference between Meta RL and our formulation is access to the environment during training. Meta RL algorithms sample transitions from the environment during meta-train. We only have access to existing data during meta-train.
Context Inference Zintgraf et al. (201)) and Rakelly et al. (2019) propose learning inference mod- ules that infer the MDP identity. Their procedures sample transitions from the MDP during meta- train, which differs from our motivation of learning from only existing data. Killian et al. (2017) infers the MDP's "hidden parameters". inputs the parameters a learnt transition function to gener- ates synthetic data and train policy from the synthetic data. Such model -bsed approaches are still outperformed by the best model-free methods (Wang et al. (2019)), which our method is based on.
Batch RL Fujimoto et al. (2018a) and Agarwal et al. (2019) demonstrate that good policies can be learnt entirely from existing data in modern RI. benchmarks. Our work extends their approaches to train policies from data generated by family of MDPs. [i et al. (2004) selects transitions from the batch based on an importance measure. They assume that for state-action pair in the batch, their value under the optimal value function can be easily computed. We do not make such assumption.
Factored MDPs In discrete MDP, the number of possible states increases exponentialy in the num- ber of dimension. Kearns & Koller (2000) tackles this problem by assuming each dimension in the next state conditionally dependent on only a subset of the dimensions in the current state. In contrast, our method makes no such assumption and applies both discrete and continuous settings.
Joint MDP The family of MDPs can be seen as a joint MDP with additional information in the state which differentiates states between the different MDPs (Parisotto et al., (015). Sampling an initial state from the joint MDP is equivalent to sampling a MDP from the family of MDPs. However, without prior knowledge, it unclear how set the value of the additional information supports generalization from the meta-train MDPs to the meta-test MDPs. In fact, the additional information in our approach the transitions from the MDP and the network learns to infer MDP identity.
7 CONCLUSION
We propose new formalization of pre-training in RL as Batch Meta Reinforcement Learning (BMRL). BMRL differs from Batch RL in that the existing data comes from family of related MDPs and thus enables scalable data collection. BMRL also differs from Meta RL in that envi- ronment interaction happens during meta-train. We identified two main challenges BMRL: MDP identity inference and generalization to unseen MDPs. To tackle these challenges, we propose tiMe, a flexible and scalable training pipeline which jointly learn distillation of value functions and MDP embeddings. Experimentally. we demonstrate that tiMe obtains performance competitive with those obtained by state-of-the-an model-free RL. methods on unseen MDPs.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the Twenty-firs first International Conference on Machine Learning, ICML '04, pp- 1-, New York. NY, USA, 2004. ACM. ISBN 1-581 doi: 10.1145/1015300.0111 1015440. URL http::/ acm. org/o0gg/o..oooo.ooo
1ttp//00.114/11113..... wtt.../g1...11111111113333333330 1015433
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity in off-policy deep reinforcement leaming. 2019. URI. 1 https::/arxiv..rrtt/t////
907. 1090000443

---

Under review as a conference paper at ICLR 2020
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. CoRR, abs/1612.000400, 2016. URL httt:://axxiv.rrr/// abs 1612. 00410
András Antos, Rémi Munos, and Csaba Szepesvári. Fitted q-iteration in continuous action-space mdps. In Proceedings of the 20th International Conference on Neural Information Processing
Systems, NIPS'07, pp. 9-16, USA, 2007. Curran Associates Inc. ISBN 978-1-60560-352-00 URL http:://ddt acm org/ wttgg///aattttt. ion.
2981564
Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective, 2019. URL
01724 cite arxiv:1902.01724.
Mathew Botvinick, Sam Ritter, Jane Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Has- sabis. Reinforcement learning, fast and slow. Trends in Cognitive Sciences, 23, 04 2019. doi: 10.1016fj.ics.2019.222.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. CoRR, 0tos//805.12111 12, 2018.
URL httpthttt/aatiiiee org hwtp://b///11000 18111121111111
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http: org. hbsp://b////BB/ 04805
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI$22:: reinforceeet leanning via slow reinforcement learning. CoRR, 0abs/1611.027799 2016. URL
http:://axxi..org/ wtt..//rgg/bbbll.lll//. 02779
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leamning for fast adaptation of deep networks. CoRR, abs/1703.03 2017. URL http://ar.....orr/aaa///000
03400
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. CoRR, abs//1822.02900 2018a. Utt.:ttt..//arkk..ooo//////////2
Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing apncopiaaippaooooiitititioioooo emoor in actor- critic methods. CoRR, abs/1 802.09477, 2018h
URI. http://arxxv..orggbbb/////
09477
Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. CoRR, abs/1802.07245, 2018. URL
http:://rxiia org http//bs//1000 /ttp//1822.222 07245
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, 0////801101299
2018a. URL U http::/ttaxii.....t// http://axivv.o.. org. hatpg//bbs/88011 01290
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, abs/1812.05905, 2018b. URL http:://axiiii www.eorrgaaa. /bs/
1812.08 05905)
Hado V. Hasselt. Double q-learning- In J. D. Lafferty, C. K. I. Williams. J. Shawe-Taylor, R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 2613-2621. Curran Associates, Inc., 2010. URL http://papers mips.cc/paper"
Matteo Hessel, Hado van Hasselt, Joseph Modayil, and David Silver. On inductive biases in deep reinforcement learning, 2019. URL
net formmorrrmm
FJovf3RcFF
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http: /rxxvv. org/ 1503. 02531
10

---

Under review as a conference paper at ICLR 2020
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. CoRR, abs/ abs/1606.03476, 2016. /ttpat::p//iiiiii org /666 C3476
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In IN LECTURE NOTES ON COMP. SCL. 2130, PROC. INTL. CONE ON ARTI NEURAL NETWORKS (ICANN-2001, pp. 87-94. Springer, 2001.
Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching, 08 2019.
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. CoRR, abs//905, 905.06424, 2019.
URL http::/tp///ii.....//// org abs/1 06424
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio García Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Ko- ray Kavukcuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. CoRR, 18s/1807,01281, 2018. URL http://arxi: v.
011881
Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. 10 2000.
Taylor W Killian, Samuel Daulton, George Konidaris, and Finale Doshi-Velez. Robust and efficient transfer learning with hidden parameter markov decision processes. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6250- 6261. Curran Associates, Inc., 2017. URL http://aappessnnnssscccspappepee
ntw.
2205-robust-. anficci-eee wetpiiiiiinttttteeeeeeeee ttrnsferr
pdf
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. CoRR, abs/1906.02691, 2019. URL /ttt::://axivv wttp:/rr///////////oetos 02691
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforcement Learning: State of the Art, 01 2012. doi: 10. 10.10077978-3-44--776553-
Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of samples in batch rein- forcement learning. In ICML, pp. 544-551, 2008. URL Netpps://eol.org/10..
7390156.13900
Lihong Li, Vadim Bulitko, and Russell Greiner. Batch reinforcement learning with state impor- tance. In Jean-Frangee Boulicaut, Floriana Esposito, Fosca Giannotti, and Dino Pedreschi (eds.), Machine Learning: ECML 2004, pp. 566-568, Berlin, Heidelberg, 2004. Springer Berlin Heidel- berg. ISBN 978-3-540-30113 15-8.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinfrrmeeet learning. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2016. URL htttp://bblpp... uni-trier de/ db/ conf/iclr/ic
html #L #11111crapHPHETII..
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens der Maaten. Exploring the limits of weakly supervised pretraining. CoRR, abs/1805 htts/8805.0093.. 2018. http:
org/ abs 1805. 00932.00032 00932
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http://dx.. doi wtw.ddo....oo.oo.o 1038 naturel 4236

---

Under review as a conference paper at ICLR 2020
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. CoRR, abs/1803.02999, 2018. URL
http://sss///00 02999
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. CoRR, abs/151 11151.03444, 2015.
Martin L.. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John
Wiley Sons, Inc., New York, NY, USA, 1st edition, 1994. ISBN 0471619779.
Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. CoRR, abs/1612.00593,, 2016. URL http:
//arxiv.crg////a.ala..aoaa abs 6126 00593
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy mete-reinforreemnnt learning via probabilisticc context variables. CoRR, abs/1903.08254, 2019.
URL http::/ttpapiiiie wttp 08254
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. Eric P. Xing and Tony Jehara (eds.), Proceedings of the st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1278 1286, Bejing, China, 22 24 Jun 2014. PMLR. URL mlr press htt////////e/eeee htt.eennee....m...
http://proceeciing
Samuel Ritter, Jane Wang. Zeb Kurth-Nelson Siddhant Jayakumar, Charles Blundell, Razvan Pas- canu, and Matthew Botvinick. Been there, done that: Meta-learning with episodic recall, 05 2018.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. CoRR, abs//810.06684, 2018. URL
06784
Reuven Y. Rubinstein and Dirk Kroese. The Cross Entropy Method: A Unified Approach To Combinatorial Optimization, Monte-carlo Simulation (Information Science and Statisties). Springer-Verlag. Verlag, Berlin, Heidelberg, 2004. ISBN 038721240X.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei- Fei Li. Imagenet large scale visual recognition challenge. CoRR, abs/1409.0575, 2014. URL
http: / //trxiva.vv///a///a//...//aaaa.. org/ abs/ 409 0575
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press,
Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based
control. In IROS, pp. 5026-5033. IEEE, 2012. ISBN 978-1-4673-1737-5. URL http:
7/dblp. uni ttritrreee.. de/ http////coo////// irro/iio//oeo www.iiro201111. htt.00122.hmml1ooooooooooo
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. CoRR, Abs/1509., 2015. http:ht::/aaxx...rgggggoe
Quan Ho Vuong, Yiming Zhang, and Keith Ross. Supervised policy update. CoRR, abs/l 105.1 1706, 2018. httl:tttp/arrrt org. http//abs/1811.11111
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. CoRR, abs/ /bbs/661// 11111.055 1.05763, 2016a. URLhttp: arxiv. wtw.aaawwv....aaaaaaaaaa
abs////////// 6 61111116 05763
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning reinforcement learn. CoRR, abs//611..57700 2016b. htRt://t://ant
wwt./bs.///aeoooooooo..oo/... 11///11111111 05763
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shun- shi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking masel-based rein- forcement learning. CoRR, 0bss/1907.02077 2019. URL http: axxiv. wwtpxxi.org//ab/////
02657
12

---

Under review as a conference paper at ICLR 2020
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-23) 292999999,,92 1992.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, tbs//906.082377 2019, http::/// org/ abs /906/ 08237
Manzil Zaheer, Satwik Kottur. Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov. and Alexander J. Smola. Deep sets. CoRR, abs/1703.06114, 2017. URL http::/arxiv..rrrt
abs/ 06T14]
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep re- inforcement learning with relational inductive biases. In International Conference on Learning Representations, 2019. httppp/////pggrrrrrieeeeeee/reep
Matthew D. Zeiler and Rob Fergus. Visualizing and understandinng convolutional networks. CoRR. abs/1311.2901, 2013. URL
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7693-7702, Long Beach, California, USA, 09 15 Jun 2019. PMLR. URL httt:://prooceee/// ngs mlr (ptess/v97/zingggstee htm
A APPENDIX
A.I MDP MIS-IDENTIFICATION CONVERGENCE PROOF
Statement:
Performing Q-learning on finite S and A with all the Q-values Q(s, a) initialized to and update rule 1 where (s, a, 8'r) sampled uniformly random from B at each step will lead the Q-values converge the optimal Q-value of the MDP (S, A, R, 7) almost surely as long as a 2 0, Ei 04 = co, Er at < co, where
N(,,a,5)
Σ
N(s, (0,,,,")) >
7(,,a,8)
E"ES otherwise.
1s=e':
R(s,a,8))
a's,a.s')
0.
otherwise.
Proof. First note that for any (s, a) S x A such EprEs ((s, a, 8") = the initial Q(s, a) is already optimal and will never be updated; for all other (s, a) S A and any s' E S, have
T(s, 8')
Pr (20.0..6
(%6 = 81100 8, 8,00 a
R(s, a, s') = E (s0. ~8) (ro = s, au = a, s/ = s/
and with probability 1,
ag [Q(s. a) updated at round {) = 00
Of [Q(e, a) updated round < xc
Then convergence follows from the same argument for the convergence of Qeleaniig (Watkins & Dayan 1992)
13

---

Under review as a conference paper at ICLR 2020
B HYPER-PARAMETERS
The small, medium and large target velocity in hopper corresponds to 0.2, 1.0. The small, medium and large velocity halfcheetah corresponds 0.475, 1.075, 1.475. The learning rate is 3e4 and the Adam optimizer is used in all experiment. All neural networks used are feed-forward network. All experiments are performed on machines with up to CPU cores and 4 Nvidia GPU.
All experiments are performed in Python 3.7, mujoco-py 2.0.2.5 running top of mujoco 200. All neural network operations are in Pytorch 1.2.
In the toy experiment, Qs consists of 2 hidden layers, each of size 256. The inferred MDP size is 32. The context size is f consists of hidden layers of size 256. consists of hidden layers of size 256 and outputs 256 values. Same goes for P. Random shooting was performed with 100 random actions at each iterations.
In hopper, the size of the inferred MDP is 8. The context size is 1. / consists of 3 hidden layers, each of size 256. consists of 4 hidden layers, each of size 16, and outputs a vector of size 8. B consists of 2 hidden layers, each of size 4. Qs consists of 8 hidden layers, each of size 128. The training mini-batch size is 32. When BCQ is ran to extract the value function out of the batch, the same hyper-parameters //github. com/ s fuuj 1im/ BCQ, except the learning rate is lowered from 0.003 to 0.0003. The alpha and beta in the reward function definition 1.0 and 0.001. The diiveboouue is 1.0.
found in the official implementation
used https:
In halfcheetah, the size of the inferred MDP is 64. The context size is 1. f consists of 7 hidden layers, each of size 512. E consists of? hidden layers, each of size 512, and outputs vector of size 64. B consists of 1 hidden layers, each of size 64. Qs consists of hidden layers, each of size 512. The training mini-batch size is 64. Gg consists of 7 hidden layers, each of size 750. Es consists of 7 hidden layers, each of size 400. When BCQ is ran to extract the value function of the batch, unless otherwise mentioned, the same hyper-parameters. implementation are used https:: thhb......... com/ s fu jim / The learning rate is lowered from 0.003 to 0.0003. The perturbation model has 2 hidden layers, of size 400, 300. The critic also has 2 hidden layers, of size 400, 300. The alpha and bela in the reward function definition are 1.0 and 0.05. The alive_honus 0.0.
found in the official
In both hopper and halfcheetah, except for the super Q function loss, the terms in the loss L in Algorithm are scaled so that they have the same magnitude as the super Q function loss. Graphs for the mujoco experiments are generated by smoothing over the last 100 evaluation datapoints.
The performance on Mujoco was averaged over seeds 0 The hyper- pyrrmeeasameters for SAC are the same those found in the Pytorch public implementation https: qithub..bb.... com/ wwttcyr///////// rlkit, The standard deviations are averaged over 5000 timesteps during evaluation. This corre- sponds to 5 episodes in halfcheetah because there is no terminal state termination in halfcheetah and variable number of episodes in hopper because there is terminal state termination.
14


Reviews:
[{"review_id": "Hkl8hTCptr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the meta-RL problem in the off-policy, batch learning setting. Batch-RL is the setting in which a policy is learned entirely offline, that is, without interaction with the environment and given only trajectories collected by some policy. Compared to RL, Meta-RL involves the additional challenge of task-inference; the goal of Meta-RL is to train a policy that can generalize to a distribution of tasks (i.e. a distribution of MDPs), without actually being given a description of the task (unlike contextual multi-task policy learning). A simple approach for solving the meta-RL problem thus might be to first perform task inference by encoding data from some task into a task description, and then condition a contextual policy on this task description. \n\nThe authors state that the straight-forward application of such an idea in the Batch-RL setting fails due to an issue they term the MDP “misidentification” problem, wherein having multiple tasks in a single batch results in confusion between tasks. This issue really only arises in the setting where task inference is jointly learned with the multi-tasking contextual policy. Thus, they propose a stage-wise algorithm wherein first 1) the N tasks in some dataset of trajectories are learned by N separate policies, and then 2) these N separate policies are distilled into a single master policy, wherein the subpolicy enacted by the master policy is modulated by some sort of task description. The distillation procedure thus involves task description (i.e. mapping from task data to a task embedding) and supervised learning (i.e. imitation) of each sub policy when conditioned on respective task embedding.\n\n* Pros\n\t* Demonstrates the effectiveness of a straightforward stage-wise approach for off-policy meta-RL (albeit without comparison to alternatives)\n\t* The method seems to perform well on the tasks considered, approaching the level of performance of SOTA model-free algorithms.\n\t* The approach is general insofar as it could presumably be used with other batch-RL methods (for first stage of algorithm), or even non-batch RL methods. In this sense, the main contribution might be seen as an approach for distilling multiple policies into a single policy in a way that allows for interpolating between them.\n\t\n* Cons\n\t* Technical novelty: while they address a problem that has not yet seen much attention, their solution is combination existing solutions. I say this because I am not fully convinced that the more novel aspects of their distillation procedure (i.e. the auxiliary task) are absolutely necessary. I would be willing to change my stance on this, given evidence.\n\t\t* Ablations of i.e. the auxiliary task would help to clarify this. In all tasks considered, the transition function T does not change. Therefore, the function composite P(E(s_j, a_j)) should actually not learn any task-specific information when predicting s’_j. If the method works without the auxiliary loss, it is very similar to a stagewise version of PEARL, with Batch-RL. \n\t* Limited comparisons to relevant alternatives, simple baselines.\t Do not compare to anything other than SAC.\n\t* Only tasks considered are pointmass, hopper, and half-cheetah; other work (i.e. PEARL) has also been demonstrated on Ant in Mujoco. \n\t* Argues that it is more stable than variants that involve interaction with the environment / use critic loss for MDP identification, which is a somewhat unfair since in their case 1) they assume they have data that is good enough to learn a good policy in the batch-RL setting, and 2) the stability is by virtue of the fact that they do things in a stage-wise manner because batch-RL works, i.e. they are doing supervised learning without having to bootstrap.\n\t* Not sure if the use of term “pre-train” is appropriate, insofar as the test tasks are assumed to come from the same distribution as training tasks. It seems to be more about Batch-Meta-RL. \n\t* Not much attention given to attempts to solve the MDP mis-identification problem with simple solutions like giving side-information at training time (i.e. task-ID). \n\n* Would be helpful to see:\n\t* More baselines\n\t\t* A simple task classifier. This would basically do a nearest neighbor lookup over the training tasks (given a test task), but this might perform well under the reward function.\n\t\t* Comparison to something like PEARL, or some method that does involve interaction with the environment -> this would help shed light on whether interaction is necessary for task inference in the the tasks they consider, and if so, such methods would in some sense be oracles.\n\t* Ablations\n\t\t* Remove auxiliary forward prediction loss: if it works without auxiliary loss, this is very similar to a stagewise version of PEARL, with Batch-RL.\n\t\t* How much data per task is needed?\n\t* Harder tasks, where the method can’t approach SAC (as one would expect for sufficiently challenging tasks)\n\t* Experiment where they study zero-shot generalization by considering disjoint parts of task space\n\t\t* Claim zero-shot generalization but in this case they study tasks where resets are not crucial for exploration needed for task inference\n\t\t\n* Minor comments:\n\t* I found the discussion about inductive bias in RL at the end of section 3 (last few sentences of last paragraph) to be a bit vague.\n\nI've given a weak reject mainly because 1) auxiliary loss has not been experimentally shown to be crucial and therefore the technical novelty may be relatively limited, and 2) more comparisons are needed, to alternatives or other baselines.  I would be willing to change my decision on this, given supporting evidence.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg4ytqaYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies batch meta learning, i.e. the problem of using a fixed experience from past tasks to learn a policy which can quickly adapt to a new related task. The proposed method combines the techniques of Fujimoto et al. (2018) for stabilizing batch off-policy learning with ideas from Rakelly et al. (2019) for learning a set invariant task embedding using task-specific datasets of transitions. They learn task-specific Q-values which are then distilled into a new Q function which is conditioned on the task embedding instead of the task ID. The embedding is further shaped using a next-state prediction auxiliary loss. The algorithmic ideas feel a bit too incremental and the experimental evaluation could be stronger--I'd recommend trying the method on more complicated environments and including ablation studies.\n\nSpecific comments:\n1. I disagree that the cheetah and hopper environments are \"challenging\"--they're one of the simplest MuJoCo environments. \n2. The problem of adapting to run at a specific speed when the meta-learner observes the dense rewards is actually not a meta learning problem because the meta learner can uniquely identify the target speed from a single transition. This is because the current speed is part of the observation, and so given the value of the dense reward at this state, it is simple to calculate the target speed. Hence these environments are effectively the same as directly giving the agent the target speed as an input. Given this interpretation, I'm not sure what is \"meta\" about this environment. The problem then reduces to the question of whether the agent can generalize from the 16 or 29 training tasks. That this should be the case is not surprising considering the one-dimensional nature of the task space.\n3. It would also be useful to see some ablations. For example, is the auxiliary prediction task necessary? Would it be possible to side step the distillation process and directly learn Q_S from the buffer as done e.g. in Rakelly et al. (2019)? Could you show some data that the corrections from Fujimoto et al. (2018) are important in the batch setting?\n\n------------------------------------------------------------------------------------------------------------\nThanks for your comments. I still think this is too incremental, and my concerns regarding the environments and using the dense reward as a feature which identifies the task haven't changed and so I'm keeping my score as is.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BygNsRkAKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on a very interesting problem, that of pre-training deep RL solutions from observational data.   The specific angle selected for tackling this problem is through meta-learning, where a set of Q-functions / policies are learned during pre-training, and during testing the network identifies the training set MDP matching the data to extract a transferable solution.\n\nThe main strength of the paper is to draw attention to the issue of pre-training in RL, which is much less studied than in supervised learning, where it has been shown to have tremendous impact.   The paper also provides reasonable coverage of a large amount of related work.\n\nUnfortunately I really struggled (despite careful reading) to understand several aspects of the proposed methods.   The training of function f() is not clearly explained; is this done as per Sec.2.4?  What is the loss function for this?  Is it done end-to-end as per the pipeline in Fig.1 (right), so using a gradient propagated back from Q?   What is the purpose of Proposition 1?  The more interesting point seems to be that solutions can “converge to a degenerate solution”, but this is not formally defined (i.e. how do you assess degeneracy, and how is that information used?)  Furthermore Proposition 1 seems limited to discrete state/action spaces. Is this the case for TIME in general?  The results on Mujoco suggest not.  Regarding the second phase of the pipeline, it is briefly mentioned that “P has low capacity” (bottom of p.4), but this is not explained further.  Is this due to a generalization issue, or a computational issue?  Why would P be low capacity but not E?  How does this actually impact the implementation?\n\nAs a higher-level comment: is it really necessary (preferable) to infer the identity of a specific train MDP (using the function f)?  This is used as a premise in this work, but I am not convinced this is desirable (for good generalization) or scalable (in the case of several observed meta-train MDPs).   What is the advantage of proceeding in this way?  Much of the work on pre-training in supervised learning just exposes the learner to large amounts of observational data to pre-condition the solution.\n\nFinally, I have some concerns with the results as presented in the paper.  There are some details lacking, for example how specifically are the meta-test MDPs chosen for the Mujoco experiments?  How similar/different from the meta-train MDPs?  This is an issue because in Sec.5.1 the meta-test MDPs are chosen to coincide with meta-train MDPs.  So I am definitely interested in seeing how well the method actually generalizes to unseen MDPs, so need more detail on this part of the experiment.   I would also like to see a few additional naïve baselines.  First, what is the result if you do the pre-training as specified, and then at test time you randomly sample one of the pre-trained MDPs (rather than use the identification function).  Second, what is the result if you put all the meta-train data into a single batch, train a solution with SAC, then use this as a pre-trained solution (rather than the current “SAC trained from scratch”), allowing more training at test time.   Both these are useful sanity checks to verify the effectiveness of the proposed approach.\n\n============\nPost-rebuttal comments:\n\n1.  My question, as stated in the review is: \" how specifically are the meta-test MDPs chosen for the Mujoco experiments\".  I read that they are tested on unseen MDPs.  I want to know how those unseen MDPs are selected / specified, and again as per my review: \"How similar/different from the meta-train MDPs?\"\n\n2.  You should entertain the possibility that perhaps Sec.3 is not as clear as you think it is.  For example the MDP (S,A,\\hat{T}, \\hat{R}, \\gama) is not defined as \"the wrong MDP\" - which I assume is what you mean by degenerate solution?  \n\n3.  I would like to know what are the results from the 2 naive baselines I described, as good sanity check.\n\nIn general, the rebuttal is intended to be a conversation to clarify understanding of the work. It is insulting to the reviewer to say that they did not read the paper carefully when they indicate they did.  It is much more productive to assume that many of your other (future) readers might have the same need for clarification, and you should be thankful for help provided by the reviewers to achieve this.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers reached a unanimous consensus that the paper could not be accepted for publication in its current form. There were a number of concerns raised regarding (1) the clarity of the writing; (2) the comparisons, especially to prior work; (3) the details of the experimental setup.
Decision: Reject