Year: 2019

Abstract:
Pre-training is transformative in supervised learning: a large network trained with large and existing datasets can be used as an initialization when learning a new task. Such initialization speeds up convergence and leads to higher performance. In this paper, we seek to understand what the formalization for pre-training from only existing and observational data in Reinforcement Learning (RL) is and whether it is possible. We formulate the setting as Batch Meta Reinforcement Learning. We identify MDP mis-identification to be a central challenge and motivate it with theoretical analysis. Combining ideas from Batch RL and Meta RL, we propose tiMe, which learns distillation of multiple value functions and MDP embeddings from only existing data. In challenging control tasks and without additional exploration on unseen MDPs, tiMe is competitive with state-of-the-art model-free RL method trained with hundreds of thousands of interactions. This work demonstrates that Meta RL from observational data is possible and we hope it will gather additional interest from the community to tackle this problem.

heading: 1 INTRODUCTION
text: Deep Reinforcement Learning algorithms still require millions of environment interactions to obtain reasonable performance, hindering their applications (Mnih et al., 2015; Lillicrap et al., 2016; Vuong et al., 2018; Fujimoto et al., 2018b; Jaderberg et al., 2018; Arulkumaran et al., 2019; Huang et al., 2019). This is due to the lack of good pre-training methods. In supervised learning, a pre-trained network significantly reduces sample complexity when learning new tasks (Zeiler & Fergus, 2013; Devlin et al., 2018; Yang et al., 2019). Meta Reinforcement Learning (RL) has been proposed as a framework for pre-training in RL (Wang et al., 2016a; Duan et al., 2016; Finn et al., 2017). However, such methods still require the collection of millions of interactions during meta-train, which means that they face the same sample complexity challenge as standard RL algorithms. In this work, we use the following definition of pre-training: the ability to use data from a set of tasks to improve performance on unseen, but related tasks.
In supervised learning, a key reason why pre-training is incredibly successful is that the dataset used for pre-training can be collected from naturally occurring large-scale processes. This removes the need to manually collect data and allows for scalable data collection, resulting in massive datasets. For example, Mahajan et al. (2018) pre-trains using existing images and their corresponding hashtags from Instagram to obtain state-of-the-art performance on ImageNet (Russakovsky et al., 2014).
In this paper, we seek to formalize pre-training in RL in a way that allows for scalable data collection. The data used for pre-training should be purely observational and the policies that are being optimized for should not need to interact with the environment during pre-training. To this end, we propose Batch Meta Reinforcement Learning (BMRL) as a formalization of pre-training in RL from only existing and observational data. During training, the learning algorithms only have access to a batch of existing data collected a priori from a family of Markov Decision Process (MDP). During testing, the trained policies should perform well on unseen MDPs sampled from the family.
A related setting is Batch RL (Antos et al., 2007; Lazaric et al., 2008; Lange et al., 2012), which we emphasize assumes the existing data comes from a single MDP. To enable scalable data collection, this assumption must be relaxed: the existing data should come from a family of related MDPs. Consider smart thermostats, whose goal is to maintain a specific temperature while minimizing electricity cost. Assuming Markovian dynamics, the interactions between a thermostat and
its environment can be modelled as a MDP. Data generated by a thermostat operating in a single building can be used to train Batch RL algorithms. However, if we consider the data generated by the same thermostat operating in different buildings, much more data is available. While the interactions between the same thermostat model and different buildings correspond to different MDPs, these MDPs share regularities which can support generalization, such as the physics of heat diffusion. In section 6, we further discuss the relations between BMRL and other existing formulations.
The first challenge in BMRL is the accurate inference of the unseen MDP identity. We show that existing algorithms which sample mini-batches from the existing data to perform Q-learning style updates converge to a degenerate value function, a phenomena we term MDP mis-identification. The second challenge is the interpolation of knowledge about seen MDPs to perform well on unseen MDPs. While Meta RL algorithms can explicitly optimize for this objective thanks to the ability to interact with the environment, we must rely on the inherent generalizability of the trained networks. To mitigate these issues, we propose tiMe, which learns from existing data to distill multiple value functions and MDP embeddings. tiMe is a flexible and scalable pipeline with inductive biases to encourage accurate MDP identity inference and rich supervision to maximize generalization. The pipeline consists of two phases. In the first phase, Batch RL algorithm is used to extract MDPspecific networks from MDP-specific data. The second phase distills the MDP-specific networks.
To summarize, our contributions are three folds: (1) Formulation of Meta RL from observational data as Batch Meta Reinforcement Learning (BMRL) (2) A simple stage-wise approach which works well on standard benchmarks in the BMRL setting (3) Most importantly, demonstration that Meta RL from only observational data is possible. We hope this work will direct the attention of the meta RL community towards this research direction.

heading: 2 PRELIMINARIES

heading: 2.1 BATCH REINFORCEMENT LEARNING
text: We model the environment as a Markov Decision Process (MDP), uniquely defined as a 5 element tuple Mi = (S,A, Ti, Ri, γ) with state space S, action space A, transition function Ti, reward function Ri and discount factor γ (Puterman, 1994; Sutton & Barto, 1998). At each discrete timestep, the agent is in a state s, pick an action a, and arrives at the next state s′ and receives a reward r. The goal of the agent π is to maximize the expected sum of discounted rewards J(π) = Eτ∼π,Mi [ ∑∞ t=0 γ tRi(st,i, at,i, s ′ t,i)] where τ = (s0,i, a0,i, r0,i, s1,i, a1,i, r1,i, . . .) is a trajectory generated by using π to interact with Mi. We will consider a family of MDPs, defined formally in subsection 2.3. We thus index each MDP in this family with i.
In Batch RL, policies are trained from scratch to solve a single MDP Mi using existing batch of N transition tuples Bi = {(st,i, at,i, rt,i, s′t,i)|t = 1, . . . , N} without any further interaction with Mi. At test time, we use the trained policies to interact with Mi to obtain an empirical estimate of its performance J . Batch RL optimizes for the same objective as standard RL algorithms. However, during training, the learning algorithm only has access to Bi and are not allowed to interact withMi.

heading: 2.2 BATCH-CONSTRAINED Q-LEARNING
text: Fujimoto et al. (2018a) identifies extrapolation error and value function divergence as the modes of failure when modern Q-learning algorithms are applied to the Batch RL setting. Concretely, deep Qlearning algorithms approximate the expected sum of discounted reward starting from a state-action pair E[ ∑∞ t=0 γ tR(st, at, s ′ t)|s0 = s, a0 = a] with a value estimate Q(s, a). The estimate can be learned by sampling transition tuples from the batch and applying the temporal difference update:
Q(s, a)← (1− αt)Q(s, a) + αt(r + γQ(s′, π(s′))) π (s′) ∈ arg max a∈A Q (s′, a) (1)
The value function diverges if Q fails to accurately estimate the value of π (s′). Fujimoto et al. (2018a) introduces Batch-Constrained Q-Learning, constraining π to select actions that are similar to actions in the batch to prevent inaccurate values estimation. Concretely, given s′, a generator G outputs multiple candidate actions {am}m. A perturbation model ξ takes each state-candidate action pair as input and generates small correction term ξ(s′, am) for each candidate. The corrected
candidate action am + ξ(s′, am) with the highest value as estimated by a learnt Q is π (s′):
π(s′) = arg max am+ξ(s′,am) Q(s, am + ξ(s ′, am)) {am = G(s′, zm)}m zm ∼ N(0, 1)
Estimation error has also been previously studied and mitigated in model-free RL algorithms (Hasselt, 2010; van Hasselt et al., 2015; Fujimoto et al., 2018b).

heading: 2.3 META REINFORCEMENT LEARNING
text: Meta RL optimizes for average return on a family of MDPs and usually assume that the MDPs in this family share S,A, γ. Each MDP is uniquely defined by a tuple (Ti, Ri). A distribution p(Ti, Ri) defines a distribution over MDPs. During meta-train, we train a policy by sampling MDPs from this distribution and sampling trajectories from each sampled MDP, referred to as the meta-train MDPs. During meta-test, unseen MDPs are sampled from p(Ti, Ri), referred to as the meta-test MDPs. The trained policy is used to interact with the meta-test MDPs to obtain estimate of its performance. The choice of whether to update parameters (Finn et al., 2017) or to keep them fixed during meta-test (Hochreiter et al., 2001) is left to the learning algorithms, both having demonstrated prior successes.

heading: 2.4 MDP IDENTITY INFERENCE WITH SET NEURAL NETWORK
text: A Meta RL policy needs to infer the meta-test MDP identity to pick actions with high return. Rakelly et al. (2019) introduces PEARL, which uses a set neural network (Qi et al., 2016; Zaheer et al., 2017) f as the MDP identity inference function. f takes as input a context set c = {(sk, ak, rk, s′k)}k and infers the identity of a MDP in the form of distributed representation in continuous space. The parameters of f is trained to minimize the error of the critic Q:
(Q(s, a, f(c))− (r + V̄ (s′, f(c))))2 (2)
where V̄ is a learnt state value function. PEARL also adopts an amortized variational approach (Kingma & Welling, 2013; Rezende et al., 2014; Alemi et al., 2016; Kingma & Welling, 2019) to train a probabilistic f , which is interpreted as an approximation to the true posterior over the set of possible MDP identities given the context set.

heading: 3 BATCH META REINFORCEMENT LEARNING
text: Let K be the number of meta-train MDPs, N be the number of transition tuples available from each meta-train MDP, θ be the parameter of the policy, we can formulate Batch Meta Reinforcement Learning (BMRL) as an optimization problem:
arg max θ J(θ) = EMi∼p(Ti,Ri)
[ Eτ∼πθ,Mi [ ∞∑ t=0 γtRi(st,i, at,i, st′,i) ]] (3)
where the learning algorithms only have access to the batch B during meta-train:
B = ∪Ki=1Bi Bi = {(st,i, at,i, rt,i, s′t,i)|t = 1, . . . , N} Mi ∼ p(Ti, Ri)
We assume we know which MDP each transition in the batch was collected from. This assumption simplifies our setting and is used to devise the algorithms. To maintain the flexibility of the formalization, we do not impose restrictions on the controller that generates the batch. However, the performance of learning algorithms generally increases as the training data becomes more diverse.
MDP identity inference challenge To obtain high return on the unseen meta-test MDPs, the trained policies need to accurately infer their identities (Ritter et al., 2018; Gupta et al., 2018; Humplik et al., 2019). In BMRL, previously proposed solutions based on Q-learning style updates, where mini-batches are sampled from the batch to minimize TD error, converge to a degenerate solution. subsection 5.1 provides experimental result that demonstrates the phenomena. In finite MDP, this degenerate solution is the optimal value function of the MDP constructed by the relative frequencies of transitions contained in the batch. We can formalize this statement with the following proposition.
Proposition 1. Let N(s, a, s′) be the number of times the triple (s, a, s′) appears in B (with any reward). Performing Q-learning on finite S and A with all the Q-values Q(s, a) initialized to 0 and update rule (1) where (s, a, s′, r) is sampled uniformly at random from B at each step t, will lead the Q-values to converge to the optimal Q-value of the MDP (S,A, T̂ , R̂, γ) almost surely as long as αt ≥ 0, ∑∞ t=0 αt =∞, ∑∞ t=0 α 2 t <∞, where
T̂ (s, a, s′) =  1s=s′ , if ∑ s′′∈S N(s, a, s′′) = 0
N(s, a, s′)∑ s′′∈S N(s, a, s′′) , otherwise.
R̂(s, a, s′) =  0, if N(s, a, s′) = 0∑ r:(s,a,s′,r)∈B r
N(s, a, s′) , otherwise.
Thus, performing Q-learning style update directly on data sampled from the batch B fails to find a good policy because the value function converges to the optimal value function of the wrong MDP. We refer this phenomena as MDP mis-identification. The proof is shown in subsection A.1.
Interpolation of seen MDPs to unseen MDPs challenge The trained policies need to generalize from the meta-train MDPs to unseen meta-test MDPs. Meta RL tackles this challenge by formulating an optimization problem that explicitly optimizes for the average return of the meta-trained policy after additional gradient steps in unseen MDPs (Finn et al., 2017; Rothfuss et al., 2018; Nichol et al., 2018). This is possible thanks to the ability to interact with the environment during metatrain. However, in the meta-train phase of BMRL, the learning algorithms do not have access to the environment. We must rely on the inherent generalizability of the trained networks to perform well on the unseen meta-test MDPs. The key challenge is therefore finding the right inductive biases in the architecture and training procedure to encourage such generalization. The need to find the right inductive biases in RL was highlighted by Botvinick et al. (2019); Zambaldi et al. (2019); Hessel et al. (2019). We note that previous works phrase the need to find inductive biases as a means to forgo generality for efficient learning. In our setting, these two goals need not be mutually exclusive.

heading: 4 LEARNING DISTILLATION OF VALUE FUNCTIONS AND MDP EMBEDDINGS

heading: 4.1 DESCRIPTION OF ARCHITECTURE AND TRAINING PROCEDURE
text: We propose a flexible and scalable pipeline for BMRL. Figure 1 (left) provides an overview of the pipeline in the simplest setting. Meta-train comprises of two separate phases. The first phase consists of independently training a value function Q∗i for each MDP-specific batch Bi using Batch RL algorithms. In the second phase, we distill the set of batch-specific value functions {Q∗i }i into a super value function QS through supervised learning (Hinton et al., 2015). Compared to the normal value function, a super value function takes not only a state-action pair as input, but also an inferred MDP identity, and outputs different values depending on the inferred MDP identity.
The pipeline is flexible in that any Batch RL algorithms are applicable in the first phase. Figure 1 (right) illustrates the architecture for the second phase given that the Batch RL algorithm used in
Algorithm 1: tiMe training procedure when BCQ is used in the first phase
Input: batches {Bi}i, QS , GS , ξS , f, E, P parameterized jointly by θ
1 Q∗i , G ∗ i , ξ ∗ i ← BCQ(Bi) ∀i 2 Randomly choose Bj out of {Bi}i 3 Sample a transition (sj , aj , s′j , rj) from Bj 4 Sample context {(sk, ak, s′k, rk)}k 6=j from Bj 5 Infer MDP identity: M̂ ← f({(sk, ak, s′k, rk)}k) 6 Predict s′, r: ŝ′, r̂ ← P (E(sj , aj), M̂)
7 Predict state-action value: Q̂∗j ← QS(sj , aj , M̂) 8 z ∼ N(0, 1) 9 Predict candidate action: â← GS(sj , z, M̂)
10 Obtain ground truth candidate action: a← G∗j (sj , z) 11 Predict correction factor: ξ̂∗j ← ξS(sj , a) 12 L← ‖ŝ′ − s′‖22 + (r̂ − r)2 + (Q̂∗j −
Q∗j (sj , aj)) 2 + ‖â− a‖22 + ‖ξ̂∗j − ξj(sj , a)‖22
13 θ ← θ −∇θL
the first phase is Batch Constrained Q (BCQ) Learning. As described in subsection 2.2, BCQ maintains three separate components, a learnt value function Q, a candidate action generator G and a perturbation model ξ. Therefore, the output of the first phase consists of 3 sets {Q∗i }i, {G∗i }i, {ξ∗i }i. The second phase distills {Q∗i }i, {G∗i }i, {ξ∗i }i to QS , GS , ξS respectively. The distillation of G and ξ is necessary to pick actions that lead to high return because each learnt value function Q∗i only provides reliable estimates for actions generated by G∗i and ξ ∗ i , a consequence of the training procedure of BCQ. In addition to QS , GS , ξS , the architecture consists of 3 other networks, f, P,E. f takes as input a context {(sk, ak, rk, s′k)}k and outputs a distributed representation of the MDP identity in a fixed-dimension continuous space. The output of f is an input to QS , GS , ξS , P . E and P predicts s′j , rj given a state-action pair (sj , aj). P has low capacity while the other networks are relatively large. During the second phase, all networks are jointly trained end-to-end by the regression losses of predicting s′, r and distilling {Q∗i }i, {G∗i }i, {ξ∗i }i. This is illustrated in details in Algorithm 1.
During meta-test, f is used to infer the identity of the meta-test MDP as a fixed-dimension continuous vector. The super functions QS , GS , ξS are used to pick actions in the meta-test MDPs, using the same procedure as BCQ (subsection 2.2). The super functions also take as input the inferred MDP identity.
The key idea behind the approach is a simple stage-wise approach for the Meta RL from observational data problem. In the second phase, we distill many policies into one for related tasks by jointly learning distillation of value functions and MDP embeddings. We therefore name the approach tiMe. MDP embeddings refer to the ability to infer the identity of a MDP in the form of distributed representation in continuous space given a context.

heading: 4.2 BENEFITS OF THE PROPOSED PIPELINE
text: Inductive biases to encourage accurate MDP identification The first inductive bias is the relationship between f and QS . They collectively receive as input state-action pair (sj , aj) and context {(sk, ak, rk, s′k)}k and regress to target value Q∗j (sj , aj). The target for each state-action pair can take on the values within the set {Q∗i (sj , aj)}i. Similar state-action pairs can have very different regression targets if they correspond to different meta-train MDPs. The context is the only information in the input to f and QS that correlates with which Q∗(sj , aj) out of the set {Q∗i (sj , aj)}i f and QS should regress to. Thus, f and QS must learn to interpret the context to predict the correct value for (sj , aj). The second inductive bias is the auxiliary task of predicting s′j , rj . A key design choice is that the network P which takes as input E(sj , aj) and f({(sk, ak, rk, s′k)}k) and predicts s′j , rj has low capacity. As such, the output of f must contain meaningful semantic information such that a small network can use it to reconstruct the MDP. This is to prevent the degenerate scenario where f learns to copy its input as its output. To summarize, these two explicit biases in the architecture and training procedure encourage f to accurately infer the MDP identity given the context.
Richness and stability of supervision Previous approaches update f to minimize the critic loss (subsection 2.4). It is well-known that RL provides sparse training signal. This signal can also cause
instability since the target values in the critic loss change over time. In contrast, our pipeline provides training signal for f that is both rich and stable. It is rich because f is trained to infer a representation of the MDP identity that can be used for multiple downstream tasks, such as predicting s′, r. This encourages general-purpose learnt representation and supports generalization. The training signal is also stable since the regression targets are fixed during the second phase of tiMe.
Scalability The pipeline is scalable in that an arbitrary amount of purely observational data can be used in the first phase so long as computational constraints permit. The data can also be heterogeneous in the sense that they do not need to contain only trajectories with high return. In the experimental section, we demonstrate the benefit of the approach when the data contains trajectories of varying qualities, some of which were generated by random policies. The extraction of the batchspecific networks, such as the batch-specific value functions {Q∗i }i, from the MDP-specific batches can be trivially parallelized and scales gracefully as the number of meta-train MDPs increases.

heading: 5 EXPERIMENTAL RESULTS
text: Our experiments have two main goals: (1) Demonstration of the MDP mis-identification phenomena and tiMe’s ability to effectively mitigate it. (2) Demonstration of the scalability of the tiMe pipeline to challenging continuous control tasks and generalization to unseen MDPs.
In all experiments, the MDP-specific batch Bi is the replay buffer when training Soft Actor Critic (SAC) (Haarnoja et al. (2018a;b)) in the MDP Mi for a fixed number of environment interactions. Thus, the batch Bi contains transitions with varying reward magnitude, some of which were generated by random and poorly performing policies. While our problem formulation BMRL and the pipeline tiMe allow for varying both the transition and reward functions within the family of MDPs, we consider the case of changing reward function in the experiments and leave changing transition function to future work. Thus, for the auxiliary prediction task in the second phase of the pipeline, P only predicts r and not s′.

heading: 5.1 TOY EXPERIMENTS
text: This section illustrates MDP mis-identification as the failure mode of existing Batch RL algorithms in BMRL. The toy setting allows for easy interpretability of the trained agents’ behavior. We also show that in the standard Batch RL setting, the Batch RL algorithm tested finds a near-optimal policy. This means the failure of existing Batch RL algorithm in BMRL is not because of the previously identified extrapolation issue when learning from existing data (Fujimoto et al., 2018a).
Environment Description In this environment, the agent needs to navigate on a 2d-plane to a goal location. The agent is a point mass whose starting location is at the origin (0, 0). Each goal location is a point on a semi-circle centered at the origin with radius of 10 units. At each discrete timestep, the agent receives as input its current location (x, y), takes an action indicating the change in its position (∆x,∆y), transitions to a new position (x + ∆x, y + ∆y) and receives a reward. The reward is the negative distance between the agent’s current location and the goal location. The agent does not receive the goal location as input and |∆x| ≤ 1, |∆y| ≤ 1. Since the MDP transition function is fixed, each goal location uniquely defines a MDP. The distribution over MDPs is defined by the distribution over goal locations, which corresponds to a distribution over reward functions.
Batch SAC We modify SAC to learn from the batch by initializing the replay buffer with existing transitions. Otherwise, training stays the same. We test Batch SAC on a simple setting where there is only one meta-train MDP and one meta-test MDP which share the same goal location. This is the standard Batch RL setting and is a special case of BMRL. Batch SAC finds a near-optimal policy.
Three meta-train and meta-test MDPs This experiment has 3 meta-train MDPs with different goal locations. The goals divide the semi-circle into two segments of equal length. There are three metatest MDPs whose goal locations coincides with the goal locations of the meta-train MDPs. This setting only tests the ability of the trained policies to correctly identify the meta-test MDPs and do not pose the challenge of generalization to unseen MDPs. Batch SAC was trained by combining the transitions from the 3 meta-train MDPs into a single replay buffer and sampling transitions from this buffer to perform gradient updates. Otherwise, training stays the same as SAC. Figure 2 (left, middle) illustrates that Batch SAC fails to learn a reasonable policy because of the MDP misidentification phenomena.
Batch SAC with task inference function We also tried adding to the architecture of Batch SAC the probabilistic MDP identity inference function as described in subsection 2.4. This is the equivalent of adapting PEARL (Rakelly et al., 2019) to work in the BMRL setting. This approach fails to train a policy that performs well on all 3 meta-test MDPs. We note that the off-policy meta RL setting that the PEARL paper considers and the BMRL setting we consider are solving different problems. We do not argue that one is easier than the other.
Performance of tiMe Since Batch SAC can extract the optimal value function out of the batch in the single meta-train MDP case, we use it as the Batch RL algorithm in the first phase of the tiMe pipeline. The architecture in the second phase thus consists of E,P, f and QS . To pick an action, we randomly sample multiple actions and choose the action with the highest value as estimated by QS . This method is termed random shooting (Chua et al., 2018). As illustrated in Figure 2 (left, right), tiMe can identify the identities of the three meta-test MDPs and pick near-optimal actions.

heading: 5.2 MUJOCO EXPERIMENTS
text: Environment Description This section illustrates the test of tiMe in challenging continuous control robotic locomotion tasks. Each task requires the application of control action to a simulated robot so that it moves with a particular velocity in the direction of its initial heading. Formally, the MDP within each MDP family share S,A, T, γ and only differs in R where R is defined to be:
R(s, a, s′) = alive bonus− α|current velocity− target velocity| − β||a||2 where α and β are positive constant. A one-to-one correspondence exists between a MDP within the family and a target velocity. Defining a family of MDP is equivalent to picking an interval of possible target velocity. This setting is instantiated on two types of simulated robots, hopper and halfcheetah, illustrated in Figure 3. Experiments are performed inside the Mujoco simulator (Todorov et al., 2012). The setting was first proposed by Finn et al. (2017).
Zero-shot meta-test During testing, in contrast to prior works, we do not update the parameters of the trained networks, as is done in gradient-based meta RL, or allow for an initial exploratory phase where episode returns do not count towards the final meta-test performance, as is done in off-policy meta RL (Rakelly et al., 2019). This allows for testing the inherent generalizability of the trained networks without confounding factors. The meta-test MDPs are chosen such that they are unseen during meta-train, i.e. none of the transitions used during meta-train was sampled from any of the meta-test MDPs. At the beginning of each meta-test episode, the inferred MDP identity is initialized to a zero vector. Subsequent transitions collected during the episode is used as the
context. The meta-test MDPs are also chosen to provide wide coverage over the support of the MDP distribution. This is to test whether our approach generalizes to a variety of meta-test MDPs, or simply overfit to a small region inside the support.
Meta-train conditions The target velocities of the meta-train MDPs divide the target velocity interval into equal segments. This removes the bias of sampling meta-train MDPs when evaluating performance. The target velocity intervals, episode length, and number of meta-train MDPs for hopper and halfcheetah are [0.0, 2.35] and [0.0, 1.5], 1000 and 200, 16 and 29 respectively. For hopper and halfcheetah, each meta-train MDP has one million and sixty transitions respectively.
Performance analysis Figure 3 illustrates tiMe’s performance on unseen meta-test MDPs. tiMe is competitive with the state-of-the-art model-free RL methods trained from scratch for one million and sixty thousands environment interactions in hopper and halfcheetah respectively. We perform experiments on halfcheetah with an episode length 200 because of computational constraints. Previous Meta RL works also use an episode length of 200 (Rakelly et al., 2019). The same network trained with tiMe also performs well in a variety of different meta-test MDPs, demonstrating that it does not over-fit to one particular meta-train MDP. We compare with SAC to demonstrate BMRL is a promising research direction. We do not include other Meta RL algorithms as baseline because they would require interacting with the environment during meta-train and thus, is not solving for the problem that BMRL poses. We tried removing GS , ξS from the architecture in Figure 1 and picked action with Cross Entropy Method (Rubinstein & Kroese, 2004), but that lead to poor performance because QS over-estimates the values of actions not generated by GS , ξS .
Limitations Our approach assumes that the transitions in the batch contain good enough transitions to learn a good policy in the batch RL setting. However, we note that the data in the batch contain data of varying qualities, some of which were generated by poorly performing policies. Also, our approach has only been demonstrated to work on tasks where reset are not crucial for exploration needed for task inference, e.g. sparse reward setting. We leave this venue for future work.

heading: 6 RELATED WORKS
text: Supervised Learning and Imitation Learning The main differences between Batch (Meta) RL and supervised learning are: actions have long-term consequences and the actions in the batch are not assumed to be optimal. If they are optimal in the sense that they were collected from an expert, Batch RL reduces to Imitation Learning (Abbeel & Ng, 2004; Ho & Ermon, 2016). In fact, Fujimoto et al. (2018a) demonstrates that Batch RL generalizes Imitation Learning in discrete MDPs.
Meta RL Equation 3 is the same objective that existing Meta RL algorithms optimize for (Wang et al. (2016b); Finn et al. (2017)). We could have formulated our experimental setting as a Partially Observable MDP, but we chose to formulate it as Batch Meta Reinforcement Learning to ensure consistency with literature that inspires this paper. The main difference between Meta RL and our formulation is access to the environment during training. Meta RL algorithms sample transitions from the environment during meta-train. We only have access to existing data during meta-train.
Context Inference Zintgraf et al. (2019) and Rakelly et al. (2019) propose learning inference modules that infer the MDP identity. Their procedures sample transitions from the MDP during metatrain, which differs from our motivation of learning from only existing data. Killian et al. (2017) infers the MDP’s “hidden parameters”, inputs the parameters to a learnt transition function to generates synthetic data and train a policy from the synthetic data. Such model-based approaches are still outperformed by the best model-free methods (Wang et al. (2019)), which our method is based on.
Batch RL Fujimoto et al. (2018a) and Agarwal et al. (2019) demonstrate that good policies can be learnt entirely from existing data in modern RL benchmarks. Our work extends their approaches to train policies from data generated by a family of MDPs. Li et al. (2004) selects transitions from the batch based on an importance measure. They assume that for state-action pair in the batch, their value under the optimal value function can be easily computed. We do not make such assumption.
Factored MDPs In discrete MDP, the number of possible states increases exponentialy in the number of dimension. Kearns & Koller (2000) tackles this problem by assuming each dimension in the next state is conditionally dependent on only a subset of the dimensions in the current state. In contrast, our method makes no such assumption and applies to both discrete and continuous settings.
Joint MDP The family of MDPs can be seen as a joint MDP with additional information in the state which differentiates states between the different MDPs (Parisotto et al., 2015). Sampling an initial state from the joint MDP is equivalent to sampling a MDP from the family of MDPs. However, without prior knowledge, it is unclear how to set the value of the additional information to supports generalization from the meta-train MDPs to the meta-test MDPs. In fact, the additional information in our approach is the transitions from the MDP and the network learns to infer MDP identity.

heading: 7 CONCLUSION
text: We propose a new formalization of pre-training in RL as Batch Meta Reinforcement Learning (BMRL). BMRL differs from Batch RL in that the existing data comes from a family of related MDPs and thus enables scalable data collection. BMRL also differs from Meta RL in that no environment interaction happens during meta-train. We identified two main challenges in BMRL: MDP identity inference and generalization to unseen MDPs. To tackle these challenges, we propose tiMe, a flexible and scalable training pipeline which jointly learn distillation of value functions and MDP embeddings. Experimentally, we demonstrate that tiMe obtains performance competitive with those obtained by state-of-the-art model-free RL methods on unseen MDPs.

heading: A APPENDIX
text: A.1 MDP MIS-IDENTIFICATION CONVERGENCE PROOF
Statement: Performing Q-learning on finite S and A with all the Q-values Q(s, a) initialized to 0 and update rule (1) where (s, a, s′, r) is sampled uniformly at random from B at each step t, will lead the Q-values to converge to the optimal Q-value of the MDP (S,A, T̂ , R̂, γ) almost surely as long as αt ≥ 0, ∑∞ t=0 αt =∞, ∑∞ t=0 α 2 t <∞, where
T̂ (s, a, s′) =  N(s, a, s′)∑ s′′∈S N(s, a, s ′′) , if ∑ s′′∈S N(s, a, s′′) > 0
1s=s′ , otherwise.
R̂(s, a, s′) =  ∑ r:(s,a,s′,r)∈B r N(s, a, s′) , if N(s, a, s′) > 0
0, otherwise.
Proof. First note that for any (s, a) ∈ S × A such ∑s′′∈S N(s, a, s′′) = 0, the initial Q(s, a) is already optimal and will never be updated; for all other (s, a) ∈ S ×A and any s′ ∈ S, we have
T̂ (s, a, s′) = Pr (s0,a0,s′0,r0)∼B (s′0 = s ′|s0 = s, a0 = a) , R̂(s, a, s′) = E(s0,a0,s′0,r0)∼B [r0|s0 = s, a0 = a, s ′ 0 = s ′] ,
and with probability 1, ∞∑ t=0 αt1{Q(s, a) is updated at round t} =∞
∞∑ t=0 α2t1{Q(s, a) is updated at round t} <∞
Then convergence follows from the same argument for the convergence of Q-learning (Watkins & Dayan, 1992)

heading: B HYPER-PARAMETERS
text: The small, medium and large target velocity in hopper corresponds to 0.2, 1.1, 2.0. The small, medium and large velocity in halfcheetah corresponds to 0.475, 1.075, 1.475. The learning rate is 3e4 and the Adam optimizer is used in all experiment. All neural networks used are feed-forward network. All experiments are performed on machines with up to 48 CPU cores and 4 Nvidia GPU.
All experiments are performed in Python 3.7, mujoco-py 2.0.2.5 running on top of mujoco 200. All neural network operations are in Pytorch 1.2.
In the toy experiment, QS consists of 2 hidden layers, each of size 256. The inferred MDP size is 32. The context size is 1. f consists of 1 hidden layers of size 256. E consists of 1 hidden layers of size 256 and outputs 256 values. Same goes for P . Random shooting was performed with 100 random actions at each iterations.
In hopper, the size of the inferred MDP is 8. The context size is 1. f consists of 3 hidden layers, each of size 256. E consists of 4 hidden layers, each of size 16, and outputs a vector of size 8. B consists of 2 hidden layers, each of size 4. QS consists of 8 hidden layers, each of size 128. The training mini-batch size is 32. When BCQ is ran to extract the value function out of the batch, the same hyper-parameters as found in the official implementation are used https: //github.com/sfujim/BCQ, except the learning rate is lowered from 0.003 to 0.0003. The alpha and beta in the reward function definition are 1.0 and 0.001. The alive bonus is 1.0.
In halfcheetah, the size of the inferred MDP is 64. The context size is 1. f consists of 7 hidden layers, each of size 512. E consists of 7 hidden layers, each of size 512, and outputs a vector of size 64. B consists of 1 hidden layers, each of size 64. QS consists of 8 hidden layers, each of size 512. The training mini-batch size is 64. GS consists of 7 hidden layers, each of size 750. ξS consists of 7 hidden layers, each of size 400. When BCQ is ran to extract the value function out of the batch, unless otherwise mentioned, the same hyper-parameters as found in the official implementation are used https://github.com/sfujim/BCQ. The learning rate is lowered from 0.003 to 0.0003. The perturbation model has 2 hidden layers, of size 400, 300. The critic also has 2 hidden layers, of size 400, 300. The alpha and beta in the reward function definition are 1.0 and 0.05. The alive bonus is 0.0.
In both hopper and halfcheetah, except for the super Q function loss, the terms in the loss L in Algorithm 1 are scaled so that they have the same magnitude as the super Q function loss. Graphs for the mujoco experiments are generated by smoothing over the last 100 evaluation datapoints.
The performance on Mujoco was averaged over 5 seeds 0−4. The hyper-parameters for SAC are the same as those found in the Pytorch public implementation https://github.com/vitchyr/ rlkit. The standard deviations are averaged over 5000 timesteps during evaluation. This corresponds to 5 episodes in halfcheetah because there is no terminal state termination in halfcheetah and variable number of episodes in hopper because there is terminal state termination.

Review summaries:
The paper studies batch meta learning, i.e. the problem of using a fixed experience from past tasks to learn a policy which can quickly adapt to a new related task. The proposed method combines the techniques of Fujimoto et al. (2018) for stabilizing batch off-policy learning with ideas from Rakelly et al. (2019) for learning a set invariant task embedding using task-specific datasets of transitions. They learn task-specific Q-values which are then distilled into a new Q function which is conditioned on the task embedding instead of the task ID. The embedding is further shaped using a next-state prediction auxiliary loss. The algorithmic ideas feel a bit too incremental and the experimental evaluation could be stronger--I'd recommend trying the method on more complicated environments and including ablation studies.

This paper studies the meta-RL problem in the off-policy, batch learning setting. Batch-RL is the setting in which a policy is learned entirely offline, that is, without interaction with the environment and given only trajectories collected by some policy. Compared to RL, Meta-RL involves the additional challenge of task-inference; the goal of Meta-RL is to train a policy that can generalize to a distribution of tasks (i.e. a distribution of MDPs), without actually being given a description of the task (unlike contextual multi-task policy learning). A simple approach for solving the meta-RL problem thus might be to first perform task inference by encoding data from some task into a task description, and then condition a contextual policy on this task description.