Single Episode Policy Transfer in Reinforcement Learning
Authors: Jiachen Yang
Brenden Petersen
Hongyuan Zha
Daniel Faissol
Venue:  (2020)
Keywords: transfer learning
reinforcement learning

Abstract: Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy. This modular approach enables integration of state-of-the-art algorithms for variational inference or RL. Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot. In diverse experimental domains with a single episode test constraint, our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.

Published as a conference paper at ICLR 2020
SINGLE EPISODE POLICY TRANSFER IN REINFORCE- MENT LEARNING
Jiachen Yang
Brenden Petersen Lawrence Livermore National Laboratory, USA petersen33e11mg www.eeeneeennnneee.e...
Georgia Institute of Technology, USA jiachen. waachen.yagnaeee........ yangggatech......... edu
Hongyuan Zha
Daniel Faissol
Georgia Institute of Technology, USA zha@ec. gatech. edu
Lawrence Livermore National Laboratory, USA
Faissoll@linl,
ABSTRACT
Transfer and adaptation to new unknown environmental dynamics is key challenge for reinforcement learning (RL). An even greater challenge is performing near- optimally in a single attempt at test time, possibly without access to dense rewards, which not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in family of environments with related dynamics, we propose general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to universal control policy. This modular approach enables integration of state-of-the-an algorithms for variational inference or RI. Moreover, our approach does not require access to rewards at time, allowing
perform settings where existing adaptive approaches cannot. In diverse experimental domains with single episode test constraint, our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.
INTRODUCTION
One salient feature of human intelligence the ability perform well single attempt at new task instance, by recognizing critical characteristics of the instance and immediately executing appropriate behavior based on experience in similar instances. Artificial agents must do likewise in applications where success must be achieved in one attempt and failure is irreversible. This problem setting, single episode transfer, imposes a challenging constraint in which an agent experiences and is evaluated on on-only episode of test instance.
As motivating example, a key challenge in precision medicine is the uniqueness each patient's response therapeutics (Hodson 2016; Bordbar et al. 2015 Whirl-Carrillo al. 2012). Adaptive therapy is a promising approach that formulate lates a treatment strategy as a sequential decision-making problem (Zhang et al. [2017] West et al. 20188) Petersen et al.) 2019). However, heterogeneity among instances may require explicilly accounting for factors that underlie individual patient dynamics. For example, in the case of adaptive therapy for sepsis (Petersen al. 2019), predicting patient response prior to treatment is not possible, However, differences in patient responses can be observed via blood measurements very early after the onset of treatment (Cockrell and An 2018).
As first step to address single episode transfer in reinforcement learning (RL), we propose general algorithm for near-optimal test-time performance family of environments where differences in dynamics can be ascertained early during episode. Our key idea is to train an inference model and probe that together achieve rapid inference of latent variables-white family of similar dynamical deploy a universal policy conditioned on the estimated parameters for near-optimal control on the new instance. Our approach combines the advantages of robust transfer and adaptation saapattion-based transfer, as we learn a single universal policy that requires no further training during test, but which is adapted to the new environment by conditioning on unsupervised estimation of new latent dynamics.
account for variation in small fraction (e.g., 5%) of the test episode, then

---

Published as a conference paper at ICLR 2020
In contrast to methods that quickly adapt train policies via gradients during test but assume access to multiple test rollouts and/or dense rewards (Finn et al.) 2017] Killian et al. [017]] Rakelly er al. 2019), we explicitly optimize for performance in one test episode without accessing the reward function at test time. Hence our method applies to real-world settings in which rewards during test are highly delayed even completely inaccessible-e-g
reward that depends physiological factors that are accessible only in simulation and not from real patients. We also consider computation time crucial factor for real-time application, whereas some existing approaches require considerable computation during test (Killian et al., 2017). Our algorithm builds on variational inference and RL as submodules, which ensures practical compatibility with existing RL workflows.
Our main contribution is a simple general algorithm for single episode transfer in families of environments with varying dynamics, via rapid inference of latent variables and immediate execution of a universal policy. Our method attains significantly higher cumulative rewards, with orders of magnitude faster computation time during test, than the state-of-the-an model-based method (Killian al. 2017). on benchmark high-dimensional domains whose dynamics are discontinuous and continuous in latent parameters. We also show superior performance over optimization- meta-learning and favorable performance versus baselines for robust transfer.
2 SINGLE EPISODE TRANSFER IN RL: PROBLEM SETUP
Our goal is to train a model that performs close to optimal within a single episode of test instance with new unknown dynamics. We formalize the problem as a family (S, A. T. 7, where A, 7) are the state space, action space, reward function, and discount of an episodic Markov decision process (MDP). Each instance of the family is stationary MDP with transition function [ ss a) T. When a set Z of physical parameters determines transition dynamics (Konidaris and Doshi- Velez 2014, each T. has hidden parameter = E Z that is sampled once from a distribution P'z and held constant for that instance. For more general stochastic systems whose modes of behavior are not easily attributed to physical parameters, Z is induced by a generative latent variable model that indirectly associates each Tz to a latent variable learned from observed trajectory data. We refer to "latent variable" for both cases, with the clear ontological difference understood. Depending on application, Tg can be continuous discontinuous in We strictly enforce the challenging constraint that latent variables are never observed, in contrast to methods that use known values during training YYu et al. 2017), to ensure the framework applies to challenging cases without prior knowledge.
This formulation captures a diverse set of important problems. Latent space Z has physical meaning in systems where T. is continuous function of physical parameters (e.g., friction and stiffness) with unknown values. In contrast, a discrete set Z can induce qualitatively different dynamics, such as a 2D navigation task where E 1) decides if the same action moves in either cardinal direction or its opposite (Killian es al. 2017). Such drastic impact of latent variables may arise when a single drug is effective for some patients but causes serious side effects for others (Cockrell and An 2018).
Training phase. Our training approach is fully compatible with RL for episodic environments. We sample many instances, either via a simulator with controllable change of instances using off-policy batch data in which demarcation of instances- bust not values of latent variables- is known, and train for one or more episodes on each instance. While we focus on the case with known change of instances, the rare case of unknown demarcation can be approached either by preprocessing steps such as clustering trajectory data or using dynamic variant of our algorithm (Appendix (
Single test episode. In contrast to prior work that depend on the luxury of multiple experience rollouts for adaptation during test time (Doshi Velez and Konidaris 2016 Killian et al. 2017, Finn et al. 2017 Rakelly et al. 2019), we introduce the strict constraint that the trained model has access to to-and is evaluated on on-only one episode of a new test instance, This reflects the need to perform oear-optimally as soon as possible in critical applications such as precision medicine, where an episode for a new patient with new physiological dynamics is the entirety of hospitalization.
SINGLE EPISODE POLICY TRANSFER
We present Single Episode Policy Transfer (SEPT), high-level algorithm for single episode transfer between MDPs with different dynamics. The following sections discuss specific design choices in SEPT, all of which combined in synergy for near-optimal performance in a single test episode.
2

---

Published as a conference paper at ICLR 2020
3.1 POLICY TRANSFER THROUGH LATENT SPACE
Our best theories of natural and engineered systems involve physical constants and design parameters that enter into dynamical models. This physicalist viewpoint motivates partition for transfer learning in families of MDPs: 1. learn a representation of latent variables with an inference model that rapidly encodes vector of discriminative features for a new instance: 2. train universal policy [Ta s, 2) to perform near-optimally for dynamics corresponding to any latent variable in Z: 3. immediately deploy both the inference model and universal policy on a given test episode. To build on the generality of model-free RL, and for scalability to systems with complex dynamics, we do not expend computational effort to learn a model of (s'ss,, a), in contrast to model- approaches (Killian et al. 2017 Yao et al. 2018). Instead, we leverage expressive variational inference models to represent latent variables and provide uncertainty quantification.
In domains with ground truth hidden parameters, latent variable encoding is the most succinet representation of differences in dynamics between instances. As the encoding & is held constant for all episodes of an instance, universal policy [alss, z) can either adapt to all instances when 2 is finite, or interpolate between instances when T. is continuous in 2 (Schaul et al.) 2015). Estimating diseriminative encoding for new instance enables immediate deployment of π s, 2) the single test episode, bypassing the need for further fine-tuning. This is critical for applications where further training complex models on test instance is not permitted due safety concerns. In contrast, methods that do not explicitly estimate a latent representation of varied dynamics must use precious experiences in the test episode tune the trained policy (Finn et al. (017))
In the training phase, we generate an optimized" dataset D= { of short trajectories, where each 7² == (si, ai...., ST,, aTr,) is a sequence of early state-action pairs at the start of episodes of instance Ti E T (e.g. TN 5). We train a variational auto-encoder, comprising an approximate posterior inference model (=(TT) that produces a latent encoding E from and a parameterized generative model (T 2). The dimension chosen for may differ from the exact true dimension aid the choice of dimensionality reduction. Because dynamics of a large variety of natural systems are determined by independent parameters (e.g.. coefficient of contact friction and Reynolds number can vary independently), we consider disentangled latent representation where latent units capture the effects of independent generative parameters. To this end, we bring 3-VAE (Higgins et al.) (017)) into the context of families of dynamical systems, choosing an isotropic unit Gaussian as the prior and imposing the constraint DRL. (gs(2)>")]l((=)) L(i), d; r¹)
when it exists but is unknown; domain knowledge
6. The 3-VAE is trained by maximizing the variational lower bound for each 7³ across D:
max (o 2 C(v.d: == BDD ((02) + Bg (Lo [o v.@
This subsumes the VAE (Kingma and Welling 2014] as special case (3 1), and we refer to both as VAE in the following. Since latent variables only serve to differentiate among trajectories that arise from different transition functions, the meaning of latent variables is not affected by isometries and hence the value & by itself need not have any simple relation to a physically meaningful = even when one exists. Only the partition of latent space is important for training universal policy.
Earlier methods for a family of similar dynamics relied on Bayesian neural network (BNN) approxi- mations of the entire transition function St+1 1 (BBNN) (sz, az). which was either used to perform computationally expensive fictional rollouts during test time (Killian el al. (017) or used indirectly to further optimize a posterior over (Yao el al.) 2018). Our use of variational inference is more economical: the encoder go (z[T) can be used immediately to infer latent variables during test, while the decoder ( (2]z) plays crucial role for optimized probing in our algorithm (see Section 3.3).
In systems with ground truth hidden parameters, we desire two additional properties. The encoder should produce low-variance encodings, which we implement by minimizing the entropy of (2)r):
min (go(()))) = - Φ
1) log(2.r) 2
"(1 + log o3)
D
(2)
2
d=1
under a diagonal Gaussian parameterization, where o3 = Var(g)(2(7)) and dim(=) = D. We add
-II (q)(z))) as a regularizer to equation [
Second, we must capture the impact of on higher-
In the sense of machine teaching, as explained fully in Section 3.3
3

---

Published as a conference paper at ICLR 2020
order dynamics. While previous work neglects the order of transitions (sy, Qe, St+1) in trajectory (Rakelly et al.) 2019). we note that a single transition may be compatible with multiple instances whose differences manifest only at higher orders. In general, partitioning the latent space requires taking the ordering of a temporally-extent our encoder (o(=[7) using bidirectional LSTM as both temporal directions of (st, (t) pairs are
trajectory into account. Therefore, we parameterize
informative- and we use an LSTM decoder p (architecture in Appendix E.2) In contrast to embedding trajectories from single MDP for hierarchical learning (Co-Reyes et al. 2018). our purpose is to encode trajectories from different instances of transition dynamics for optimal control.
3.2 TRANSFER OF A UNIVERSAL POLICY
We train a single universal policy T(als,)) z) and deploy the same policy during test (without further optimization), for two reasons: robustness against imperfection in latent variable representation and significant improvement in scalability. Earlier methods trained multiple optimal policies {mī (a|s) ()))s) N
on training instances with a set {= of hidden parameters, then employed either behavioral cloning (Yao et al, 2018) or off-policy Q-learning (Amekkist el al. 2019) to train a final policy т [[a 8.)) using dataset (st, 2t; 0g x (a)s)))- However, this supervised training scheme may not be robust Yu et al. 2017 y: if ( 2) were trained only using instance-speciinn optimal statc-action pairs generated by x. (als) and posterior samples of & from an optimal inference model, it may not generalize well when faced with states and encodings that were not present during training. Moreover, it is computationally infeasible to train a collection {m; J Jifjji=-w -i=ih is thrown away during test-when faced with a large set of training instances from a continuous set 2. Instead, we interleave training of the VAE and single policy π [a|s, 2), benefiting from considerable computation savings at training time, and higher robustness due to larger effective sample count.
3.3 OPTIMIZED PROBING FOR ACCELERATED LATENT VARIABLE INFERENCE
To execute near-optimal control within single test episode, we first rapidly compute using short trajectory of initial experience. This is loosely analogous to the use of preliminary medical treatment to define subsequent prescriptions that better match a patient's unique physiological response. Our goal of rapid inference motivates two algorithmic design choices to optimize this initial phase. First, the trajectory used for inference by (=(T) must be optimized, in the sense of machine teaching (Zhu et 2018), as certain trajectories are more suitable than others for inferring latent variables that underlie system dynamics. If specific degrees of freedom are impacted the most by latent variables, an agent should probe exactly those dimensions to produce an informative trajectory for inference. Conversely, methods that deploy a single universal policy without an initial probing phase (Yao et al.) 2018) can fail in adversarial cases, such as when the initial placeholder 2 used in Te (a)ss,)) at the start of an instance causes failure to exercise dimensions of dynamics that are necessary for inference. Second, the VAE must be specifically trained on dataset DD of short trajectories consisting of initial steps of each training episode. We cannot expend a long trajectory for input to the encoder during test, to ensure enough remaining steps for control. Hence, single episode transfer motivates the machine teaching problem of learning to distinguish among dynamics: our algorithm must have learned both to generate and to a short initial trajectory estimate a representation of dynamics for control.
Our key idea of optimized probing for accelerated latent variable inference is to train a dedicated probe policy (a(a)s) to generate dataset D of short trajectories at the beginning of all training episodes, such that the VAE's performance D is optimized for faster exploration during standard RI. training (Xu et al. (2018), our probe and VAE are trained for the purpose of performing well on new test MDP. For ease of exposition, we discuss the case with access to a simulator, but our method easily allows use of off-policy batch data. We start each training episode using Ti for a probe phase lasting T, steps, record the probe trajectory Tp into D. train the VAE using minibatches from D, then use Tp with the encoder to generate 2 for use by no me(als s, 2) to complete the remainder of the episode (Algorithm [ At test time. SEPT only requires lines 5, 8, and 9 in Algorithm [T (training step in 9 removed; see (2.. The reward function for Tee defined as the VAE objective, approximated by the variational lower bound (T): Rp(T)= = C(v,, Φ; 7) < log (T). This feedback loop between the probe and VAE directly trains the
2 Orthogonal to training a meta-policy
II general, D is related to the replay huffer commonly used in off-policy RL. algorithms.
4

---

Published as a conference paper at ICLR 2020
Algorithm 1 Single Episode Policy Transfer:
training phase
1:
procedure SEPT-TRAIN
2: Initialize encoder Φ, decoder v5, probe policy s,, control policy 0, and trajectory buffer D for each instance T. with transition function sampled from T do each episode on instance Tz do
3:
4:
5:
Execute Te for I steps and store trajectory Ty into D
6:
Use variational lower bound ( as the reward to train xe by descending gradient 3
7:
Train VAE using minibatches from 2> for gradient ascent on (T) and descent on 2
8:
Estimate from Tр using encoder (2(=)))
9:
Exccute (a(a 2) with for remaining time steps and train with suitable RI. algorithm
10:
end for
11:
end for
12:
end procedure
probe to help the VAE's inference latent variables that distinguish different dynamics (Figure We provide detailed justification as follows. First we state result derived in Appendix AA]
Proposition Let (T) denote the distribution of trajectories induced by T(p. Then the gradient of the entropy H (p+(T)) is given by
V (oH(()()))
-Ep (pp()) V
log(T (gg(m((alss))((oooppp)
Noting that dataset DD follows distribution Pe and that the VAE is exactly trained to maximize the log probability of D. we use L(b,, 7) as a tractable lowerbound log (. Crucially, to generate op- timal probe trajectories for the VAE, we take minimum-entree
viewpoint and descend the gradient 3 This opposite of a maximum entropy viewpoint that encourages the policy to generate diverse trajectories (Co-Reyes et al. 2018). which would minimize P and produce an adversarial dataset for the hence, optimal probing is not equivalent to diverse exploration. The degenerate case of T.p learning to "stay still" for minimum entropy is precluded by any source of environmental stochasticity: trajectories from different instances will still differ, so degenerate trajectories result in low VAE performance. Finally observe that equation B is the defining equation of a simple policy gradient algorithm (Williams 1992) for training Te. with log PogP:. interpreted as the cumu- lative reward of trajectory generated by Ti.- This completes our justification for defining reward (T) == L(v,, T). We also show empirically in ablation experiments that this reward is more effective than choices that encourage high perturbation of state dimensions or high entropy (Section
The VAE objective function may not per- fectly evaluate probe trajectory generated by Te because the objective value increases due to VAE training regardless of Te. To give a more stable reward signal to m.. we use second VAE whose parameters slowly track the main VAE according to 2/ +- an) + (1 ④' for E [0, 1], and similarly for o' While analogous to target
Use ierratiinal bound
via gradiant
Maximize ze
n2
Ceneralashar
vajoctory
To
VAE 2
L.B.
TEp
Use Inleett input
Figure 1: Ti learns to generate an optimal dataset for the VAE, whose performance is the reward for T.. En coding by the VAE is given to control policy Ta.
networks in DQN (Mnih al. 2015), the difference is that our second VAE is used compute the reward for n...
RELATED WORK
Transfer learning in a family MDPs with different dynamics manifests in various formulations (Taylor and Stone 2009). Analysis of e-stationary MDPs and c-MDPs provide theoretical grounding by showing that an RL algorithm that learns an optimal policy in an MDP can also learn a near-optimal policy for multiple transition functions (Kalmár et al. 1998. Szita al. 2002).
5

---

Published as a conference paper at ICLR 2020
Markov decision process (HiP-MDP) formalism posits a space of hidden parameters that determine transition dynamics, and implements transfer by model-based policy training after inference of latent parameters (Doshi- Dolei-Velez Velez and Konidaris 2016 Konidaris and Doshi- 2014). Our work considers HiP-MDP as a widely applicable yet special case of a general viewpoint, in which the existence of hidden parameters is not assumed but rather is induced by a latent variable inference model. The key structural difference from POMDPs (Kaelbling et al. 1998) that given fixed latent values, each instance from the family is an MDP with no hidden states; hence, unlike in POMDPs, tracking history of observations provides no benefit. In contrast to multi-task learning (Caruana 1997). which uses the same tasks for training and test, and in contrast to parameterized-skit learning (Da Silva et al. 2012), where an agent learns from a collection of rewards with given task identities in one environment with fixed dynamics, our training and test MDPs have different dynamics and identities of instances are not given.
Imposing more structure, the
Prior latent variable based methods for transfer in RL depend on a multitude of optimal policies during training (Arnekvist et al.) 2019), or learn a surrogate transition model for model predictive control with real-time posterior updates during test (Perez et al. 2018). Our variational model-free approach does not incur either of these high computational costs. We encode trajectories to infer latent representation of differing dynamics, in contrast to state encodings in (Zhang et al. 2018). Rather than formulating variational inference in the space of optimal value functions (Tirinzoni el al.) 2018), we implement transfer through variational inference in a latent space that underlies dynamics. Previous work for transfer across dynamics with hidden parameters employ model-based RL with Gaussian process and Bayesian neural network (BNN) models of the transition function (Doshi Velez and Konidaris 2016 Killian et al. 2017). which require computationally expensive fictional rollouts to train policy from scratch during test time and poses difficulties for real-time test deployment. DPT uses a fully-traines BNN to further optimize latent variable during a single test episode, but faces scalability issues as it needs one optimal policy per training instance (Yao et al. (001)). In contrast, our method does not need transition function and can be deployed without optimization during test. Methods for robust transfer either require access to multiple rounds from the test MDP during training (Rajeswaran et al.) (2017). or require the distribution over hidden variables to be known or controllable (Paul et al.] (009)). While meta-learning (Finn et al. 2017 Rusu al. 2019)] Zintgraf et al., 2019 Rakelly et al.) 2019) in principle can take one gradient step during a single test episode, prior empirical evaluation were not made with this constraint enforced, and adaptation during test is impossible in settings without dense rewards.
EXPERIMENTAL SETUP
We conducted experiments on three benchmark domains with diverse challenges to evaluate the performance, speed of reward attainment, and computational time of SEPT versus five baselines in the single test e We evaluated four ablation and variants of SEPT to investigate the necessity of all algorithmic design choices. For each method on each domain, we conducted 20 independent training runs. For each trained model, we evaluate on M independent test instances, all starting with the same model; adaptations during the single test episode, if done by any method, are not preserved across the independent test instances. This means we evaluate on total of 20M independent test instances per method per domain. Hyperparameters were adjusted using a coarse coordinate search on validation performance. We used DDQN with prioritized replay (Van Hasselt et al. 20066] Schaul et al, 2016) as the base RL component of all methods for a fair evaluation of transfer performance; other RL algorithms can be readily substituted.
We use the same continuous state discrete action HiP-MDPs proposed by Killian et al. (2017) for benchmarking. Each isolated instance from each domain is solvable by RL, but it is highly challenging, if not impossible, for naïve RI. to perform optimally for all instances because significantly different dynamics require different optimal policies. In 2D navigation, dynamics discontinuous in 2 E {0, 1} as follows: location of barrier to goal region, flipped effect of actions (i.c., depending on z, the same action moves in either a cardinal direction or its opposite), and direction of a nonlinar wind. In Acrobot (Sutton et al. 1998), the agent applies [+1, -1) torques to swing a two-link pendulum above a certain height. Dynamics are determined by vector (m1, m2, 12) of masses and lengths, centered at 1.0. We use four unique instances in training and validation,
Domains.
*Code for all experiments is availahle at https::/ggtthbb.cmm///// htt.:////cmm01222213333///0/

---

Published as a conference paper at ICLR 2020
149
SUFT
EFoptetr
114
POPEA
323
Camuaitvee
Creck
100
SOFT 3MM EPOERONN dey Ouck
SEPT 3MM ENMMMMM Kag Drede
S0.00
S0e0
St.ep
(a) 2D navigation
(b) Acrobot
(c) 2D navigation
(d) Acrobot
(c) HIV
SWA 339
NAPT SEITAE
Yarkd
Cumu atte
28
402 450
80
1000
(f) 2D navigation
(g) Acrobot
(h) 2D navigation
ii Acrobot
() HIV
Figure 2: (a-e): Comparison against baselines. (a-b): Number of steps solve 2D navigation and Acrobot in single test episode; failure to solve is assigned a count of 50 in 2D nav and 200 in Acrobot. (c-e): Cumulative reward versus test episode step. BNN requires long computation time and showed low rewards on HIV, hence we report 3 seeds in Figure 4b (f-i): Ablation results. DynaSEPT out of range in (g), see Figure 4al Error bars show standard error of mean over all test instances over 20 training runs per method.
constructed by sampling As uniformly from { 0.3, 0..1, 0.3) and adding it to all components of 2. During test, we sample As from (-0.35, 0.2, 0.2, 0.35) evaluate both interpolation and extrapolation. In HIV, patient's state dynamics is modeled by differential equations with high sensitivity to 12 hidden variables and separate steady-state regions of health, such that different patients require unique treatment policies (Adams et al. (2004). Four actions determine binary activation of two drugs. We have M 10, 5, 5 for 2D navigation, Acrobot, and HIV, respectively.
Baselines. First, evaluated two simple baselines that establish approximate bounds on test performance of methods that train a single policy: as lower bound, Avg trains a single policy и (a]s)) on all instances sampled during training and runs directly on test instances; an upper bound in the limit of perfect function approximation for methods that use latent variables as input, Oracle 7 ( s, 2) receives the true hidden parameter during both training and test. Next adapted existing methods, detailed in Appendix E.T to single episode test evaluation: 1. we allow the model-based method BNN (Killian et al. 2017 to fine-tune a pre-trained BNN and train policy using BNN -egeeeneedt fictional episodes every 10 steps during the test episode; 2. we adapted the adversarial part of EPOpt (Rajeswaran et al. 2017), which we term EPOpt-adv, by training policy 7 (als) on instances with the lowest 10- (O-percentile performance; 3. we evaluate MAML as an archetype of meta-learning methods that require dense rewards or multiple rollouts (Finn et al. (0077). We allow MAML to use a trajectory of the same length as SEPT's probe trajectory for one gradient step during test. We used the same architecture for the RL module of methods (Appendix [..2). To our knowledge, these model-free baselines are evaluated single-episode transfer for the first time in this work.
Ablations. To investigate the benefit of our optimized probing method for accelerated in- ference, we designed an ablation called SEPT-NP, in which trajectories generated by the con- trol policy are used by the encoder for inference and stored into D to train the VAE. Second, investigated an alternative reward function for the probe, labeled TotalVar and defined as R(r) == 1/T, E D
18 St,al for probe trajectory т. In contrast to the minimum entropy viewpoint in Section 3.3 this reward encourages generation of trajectories that maximize total variation across all state space dimensions. Third, we tested the maximum entropy viewpoint on probe trajectory generation, labeled MaxEnt, by giving negative lowerbound the probe reward: (T) == L(b,, o; T). Last, we tested whether DynaSEPT, an extension that dynamically decides probe or execute control (Appendix Cp. has any benefit for stationary dynamies.
dim(S)
RESULTS AND DISCUSSION
2D navigation and Acrobot solved upon attaining terminal reward of 1000 and 10, respectively. SEPT outperforms all baselines in 2D navigation and takes significantly fewer number of steps to solve (Figures 2a and 2ch. While single instance of 2D navigation is easy for RI., handling multiple
7

---

Published as a conference paper at ICLR 2020
(a) 2D navigation
(b) Acrobot
(c) HIV
(d) 2D navigation
(c) Acrobot
(f) HIV
Figure 3: Cumulative reward on test episode for different Tp (a-c) and different dim(z) (d-f). 8M
independent test instances for each hyperparameter setting.
instances is highly non-trivial. EPOpt-ady and Avg almost never solve the test instance-wa set "steps to solve" to 50 for test episodes that were because interpolating between instance-specific optimal policies in policy parameter space is not meaningful for any task instance. MAML did not perform well despite having the advantage of being provided with rewards at test time, unlike SEPT. The gradient adaptation step was likely ineffective because the rewards are sparse and delayed. BNN requires significantly more steps than SEPT, and it uses four orders of magnitude longer computation time (Table D due training policy from seratch during the test episode. Training times of all algorithms except BNN are in the same order of magnitude (Table 3
In Acrobot and HIV, where dynamics are continuous in latent variables, interpolation within policy space can produce meaningful policies, so all baselines are feasible in principle. SEPT is statistically significantly faster than BNN and Avg, is within error bars of MAML, while EPOpt-adv outperforms the rest by a small margin (Figures 2b and 2d Figure 5 shows that SEPT is competitive in terms of percentage of solved instances. As the true values of latent variables for Acrobot test instances were interpolated and extrapolated from the training values, this shows that SEPT is robust to out-of- training dynamics. BNN requires more steps due to simultaneously learning and executing control during the test episode. On HIV, SEPT reaches significantly higher cumulative rewards than all methods. Oracle is within the margin of error of Avg. This may be due to insufficient examples of the nigh-dimenssonal ground truth hidden parameters. Due its long computational time, we run three seeds for BNN on HIV, shown in Figure 4b and find it was unable to adapt within one test episode.
Comparing directly to reported results in DPT Yao al. 2018), SEPT solves 2D Navigation least 33% (>10 steps) faster, and the cumulative reward of SEPT (mean and standard error) are above DPT's mean cumulative reward in Acrobot (Table D Together, these results show that methods that explicitly distinguish different dynamics (e.g., SEPT and BNN) can significantly outperform methods that implicitly interpolate in policy parameter space (e.g., Avg and EPOpt-ady) in settings where z has large discontinuous effect on dynamics, such as 2D navigation. When dynamics are continuous in latent variables (e.g., Acrobot and HIV), interpolation-base based methods fare better than BNN, which faces the difficulty of learning model of the entire family of dynamics. SEPT worked the best in the first case and is robust to the second case because it explicitly distinguishes dynamics and does not require learning full transition model. Moreover, SEPT does not require rewards at test time allowing it be useful on a broader class of problems than optimization-base meta-learning approaches like MAMI.. Appendix D contains training curves.
2g and 2] show that the probe phase is necessary solve 2D navigation quickly, while giving similar performance in Acrobot and significant improvement in HIV. SEPT significantly outperformed Total Var in 2D navigation and HIV, while Var gives slight improvement in Acrobot, showing that directly using VAE performance as the reward for probing in certain environments can be more effective than reward that deliberately encourages perturbation of state dimensions. The clear advantage of SEPT over MaxEnt in 2D navigation and HIV supports our hypothesis in Section 3.3 that the variational lowerbound, rather than its negation in the maximum entropy viewpoint, should used as the probe reward, while performance not significantly differentiated in Acrobot. SEPT outperforms DynaSEPT on all problems where dynamics are stationary during each instance. On the other hand, DynaSEPT is the better choice in a non-stationary variant of 2D navigation where the dynamics "switch" abruptly at t = 10 (Figure 4c)
Ablation results. Comparing to SEPT-NP, Figures 2f
Robustness. Figure 3 shows that SEPT is robust to varying the probe length Tр and dim(2). Even with certain suboptimal probe length and dim(2), it can outperform all baselines on 2D navigation in both steps-to-solve and final reward; it is within error bars of all baselines on Acrobot based on final
8

---

Published as a conference paper at ICLR 2020
cumulative reward; and final cumulative reward exceeds that of baselines in HIV. Increasing Tр means foregoing valuable steps of the control policy and increasing difficulty of trajectory reconstruction for the VAE in high dimensional state spaces; T is Hyper-parameet that should be validated for each application. Appendix shows the effect / on latent variable encodings.
7 CONCLUSION AND FUTURE DIRECTIONS
We propose general algorithm for single episode transfer among MDPs with different stationary dynamies, which is challenging goal with real-world significance that deserves increased effort from the transfer learning and RL community. Our method, Single Episode Policy Transfer (SEPT), trains a probe policy and an inference model to discover latent representation of dynamics using very few initial steps in single test episode, such that universal policy can execute optimal control without access to rewards at test time. Strong performance versus baselines in domains involving both continuous and discontinuous dependence of dynamics on latent variables show the promise of SEPT for problems where different dynamics can be distinguished via a short probing phase.
The dedicated probing phase may be improved by other objectives, in addition to performance of the inference model, to mitigate the risk and opportunity cost of probing. An open challenge single episode transfer in domains where differences in dynamics of different instances early during an episode, or where latent variables are fixed but dynamics are nonstationary. Further research on dynamic probing and control, sketched in DynaSEPT, is one path toward addressing this challenge. Our work is one step along a broader avenue of research on general transfer learning RL equipped with the realistic constraint single episode for adaptation and evaluation.
detectable
ACKNOWLEDGMENTS
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344, Lawrence Livermore National Security, LLC. LLNI-JRNIL-791194
REFERENCES
Adams, B. M., Banks, H. T., Kwon, H.-D., and Tran, H. T. (2004). Dynamic multidrug therapies for hiv: Optimal and sti control approaches. Mathematical biosciences and engineering, 1(2), 223-241.
Arnekvist, I., Kragic, D., and Stork, A. (2019). Vpe: Variational policy embedding for transfer reinforcement learning. In 2019 International Conference on Roboties and Automation (ICRA), pages 36 IEFE.
Bordbar, A., McCloskey, D., Zielinski, D. C., Sonnenschein, N., Jamshidi, N., and Palsson, B. O. (2015). Personalized whole-cell kinetic models of metabolism for discovery in genomics and pharmacodynamics. Cell systems, 1(4), 283-292.
Caruana, R. (1997). Multitask learning. Machine learning, 28(1), 41-75.
Co-Reyes, D., Liu, Y., Gupta, A., Eysenbuch, B., Abbeel, P., and Levine, S. (2018). Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In Proceedings of the 35th International Conference on Machine Learning, pages 1009-1018.
Cockrell, R. C. and An, G. (2018). Examining the controllability of sepsis using genetic algorithms on an agent-bar based model of systemic inflammation. PLOS Computational Biology, 14(2), 1-17.
Da Silva, B. C., Konidaris, G., and Barto, A. G. (2012). Learning parameterized skills. In Proceedings of the 29th International Coference International Conference on Machine Learning, pages 1443-1450 Omnipress.
Doshi- Velez, F. and Konidaris, G. (2016). Hidden parameter markov decision processes: semipara- metric regression approach for discovering latent task parametrizations. In IJCAI: proceedings of the conference, volume 2016, page 1432. NIH Public Access.

---

Published as a conference paper at ICLR 2020
Finn, C., Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1126-1135.
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, volume 3.
Hodson, (2016). Precision medicine. Nature, 537(7619), S49.
Kaelbling, L.. P., Littman, M. I.., and Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2), 99-134.
Kalmár, Z., Szepesvári, C., and Lörincz, A. (1998). Module-based reinforcement learning: Experi- ments with real robot. Autonomous Robots, 5(3-)) 273-295.
Kastrin, A., Ferk, P., and Leskošek, B. (2018). Predicting potential drug-drug interactions on topological and semantic similarity features using statistical learning. PloS one, 13(5), e0196865.
Killian, T. W. Daulton, S., Konidaris, G., and Velez, F. (2017). Robust and efficient transfer learning with hidden parameter markov decision processes. In Advances in Neural Information Processing Systems, pages 6250-6261.
Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In International Conference on Learning Representations.
Konidaris, G. and Doshi- F. (2014), Hidden parameter markov decision processes: an emerging paradigm for modeling families of related tasks. In the AAAI Fall Symposium on Knowledge, Skill, and Behavior Transfer in Autonomous Robots.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529.
Paul, S., Osborne, M. A., and Whiteson, S. (2019). Fingerprint policy optimisation for robust reinforcement learning. In International Conference Machine Learning, pages 5082-5091.
Perez, C. F., Such, F. P., and Karaletsos, T. (2018). Efficient transfer learning and online adaptation with latent variable models for continuous control. arXiv preprint arXiv:/8/2.00.
Petersen, B. K., Yang, J., Grathwohl, W. S., Cockrell, C., Santiago, C., An, G., and Faissol, D. M. (2019). Deep reinforcement learning and simulation as a path toward precision medicine. Journal of Computational Biology.
Rajeswaran, A., Ghotra, S., Ravindran, B., and Levine, S. (2017). Epopt:
Learning robust neural network policies using model ensembles. In International Conference on Learning Representations.
Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. (2019). Efficient off-policy meta- reinforcement learning via probabilistic context variables. In International Conference on Machine Learning, pages 5331 5331-5
Rusu, A. A., Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., and Hadsell, (2019). Meta-learning with latent embedding optimization. In International Conference Learning Representations (ICLR).
Schaul, T., Horgan, D... Gregor, K., and Silver, D. (2015). Universal value function approximators. In International Conference Machine Learning, pages 1312-1320.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized experience replay. In International Conference Learning Representations (ICLR), volume 2016.
Sutton, R. S., Barto, A. G., et al. (1998). Reinforcement learning: An introduction. MIT press.
Szita, I., Takács, B., and Lörincz, A. (2002). e-mdps: Learning in varying environments. Journal of Machine Learning Research, 3(Aug), 145-174.
10

---

Published as a conference paper at ICLR 2020
Taylor, M. E. and Stone, P. (2009). Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul), (633-1685.
Tirinzoni, A., Sanchez, R. R., and Restelli, M. (2018). Transfer of value functions via variational methods. In Advances in Neural Information Processing Systems, pages 6179-6189.
Van Hasselt, H., Guez, A., and Silver, D. (2016). Deep reinforcement learning with double q-learning. In Thirtieth AAAI Conference on Artificial Intelligence.
West, J., You, L.. Brown, J., Newton, P. K., and Anderson, A. R. A. (2018). Towards multi-drug adaptive therapy. bioRxiv.
Whirl-Carrillo, M., McDonagh, E. M., Hebert, J., Gong, L., Sangkuhl, K., Thorn, C., Altman, R. B.,
and Klein, T. E. (2012). Pharmacogenomies knowledge for personalized medicine. Clinical Pharmacology & Therapeutics, 92(4), 414-417.
Williams, R. J. (1992). Simple statistical gradient-following following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4), 229-256.
Xu, T., Liu, Q., Zhao, L., Xu, W., and Peng. (2018), Learning to explore with meta-policy gradient.
In Proceedings of the 35th International Conference Machine Learning, pages 5463-5472.
Yao, J., Killian, T., Konidaris, G., and Doshi- F. (2018). Direct policy transfer via hidden parameter markov decision processes, In LLARLA Workshop, FAIM, volume 2018.
Yu, W., Tan, J., Liu, C. K., and Turk, G. (2017). Preparing for the unknown: Learning a universal policy with online system identification. In Proceedings of Robotics: Science and Systems, Cambridge, Massachusetts.
Zhang, A.. Satija, H., and Pineau, J. (2018). Decoupling dynamics and reward for transfer learning. arXiv preprint 1804v:1804.110 10689.
Zhang, J., Cunningham, J. J., Brown, J. S., and Gatenby, R. A. (2017). Integrating evolutionary dynamics into treatment of metastatic castrate-resistant prostate cancer. Nature communications, 8(1), 1816.
Zhu, X., Singla. A., Zilles, S., and Rafferty, A. N. (2018). An overview of machine teaching. arXiv preprint arXiv:1801.0544
Zintgraf, L. M., Shiarlis, K., Kurin, V., Hofmann, K., and Whiteson, S. (2019). Fast context adaptation via meta-leamiin. In International Conference Machine Learning (ICML), volume 2019.

---

Published as a conference paper at ICLR 2020
A DERIVATIONS
Proposition 1. Let pT( denote the distribution of trajectories induced by Te. Then the gradient of the entropy H (P)( is given by
Tp
V H(p((T) = p [V
log (og(rp((4sss)((11pppp1))
(3)
Proof. Assuming regularity, the gradient of the entropy is
V.H(pp(FT) v f log
V /
log
Vi / PT(T) po(() log (T)) log P (T)
(r) ( V P (-)
For trajectory (8o, Ao. $1.-.., ; 86) generated by the probe policy mpp
t-1
(T) = p(00)
II
p(0 p am)pp((as))
==
Then
V (o) = V (logp(so) +
Σ log p(80(1118851 az) +
log n (asl8))
Since p(so) and p(8;41/8;; a;) do not depend on we get
V& (og = V
log Tp (ails))
Substituting this into the gradient of the entropy gives equation 3
B TESTING PHASE OF SEPT
Algorithm 2 Single Episode Policy Transfer: testing phase
procedure SEPT-TEST
2: Restore trained decoder vh, encoder é, probe policy y, and control policy 0
3: Run probe policy Te for Tp time steps and record trajectory Tp
4: Use Tp with decoder (2(2)T) to estimate &
5: Use with control policy Te To(ale (8, z) for the remaining duration of the test episode
6: end procedure
C DYNASEPT
In our problem formulation, it is not necessary compute å at every step of the test episode, each instance is a stationary MDP and change of instances known. However, removing the common assumption of stationarity leads to time-dependent transition functions T, (s') |s, a). which introduces (s'|s,
problematic cases. For example, a length T, probing phase would fail 2 leads to a switch in dynamics at time t Tp. such as when poorly understood drug-drus interactions lead to abrupt changes in dynamics during co-medication therapies (Kastrin et al.) (018).. Here we describe an alternative general algorithm for non-stationary dynamics, which call DynaSEPT. We train
12

---

Published as a conference paper at ICLR 2020
single policy Ta(alss 17) that dynamically decides whether to probe for better inference act to maximize the MDP reward Reav, based on a scalur-valued function 7: IR - [0, representing the degree of uncertainty in posterior inference, which is updated at every time step. The total reward is [T) = yy (T) + (1 n) Rem((T)) where т is a short sliding-windoss trajectory of length Tp. and Ty is the final state of The history-dependent term RT(r) is equivalent to delayed reward given for executing a sequence of probe actions. Following the same reasoning for SEPT, one choice for R2 is L(Ø, 7). Assuming the encoder outputs variance of of each latent dimension, one choice for is normalized standard deviation over all dimensions of the latent variable, i.e.
= b Σ (iim)) (gx) where iimax is running max of
Despite its novelty, we consider DynaSEPT only for rare nonstationary dynamics and merely as baseline in the predominant case of stationary dynamics, where SEPT is our primary contribution. DynaSEPT does not have any clear advantage over SEPT when each instance Tz a stationary MDP. DynaSEPT requires to start at 1.0, representing complete lack of knowledge about latent variables, and it still requires the choice of hyperparameter Tp. Only after Tp steps can it use the uncertainty of 4o (2[T) ( to adapt 17 and continue generate the sliding window trajectory to improve 2. By this time, SEPT has already generated an optimized sequence using Ti for the encoder to estimate ě. If trajectory of length T is sufficient for computing good estimate of latent variables, then SEPT is expected to outperform DynaSEPT.
200
Cumulative reward
Cumulative reward
Steps to solve
180
BNN
6.8
DynaSCPT
260
EPOpt-ady MAML
5-
140
120
Oracie
200
0.2
80
0.0
25
50
75
100
150 175 200
10
30
50
60
Step
Step
(a) Acrobot
(b) HIV
(c) 2D switch
Figure 4: (a) Ablations on Acrobot, including DynaSEPT with seeds; (b) BNN and MAML attained
orders of magnitude lower rewards than other baselines (3 seeds); (c) DynaSEPT performs well on nonstationary dynamics in 2D navigation.
D SUPPLEMENTARY EXPERIMENTAL RESULTS
D.1 2D AND ACROBOT
2D navigation and Acrobot have a definition of "solved". Table[ the number steps in test episode required solve the MDP. Average and standard deviation were computed across all test instances and across all independently trained models. If an episode was not solved, the maximum allowed number of steps was used (50 for 2D navigation and 200 for Acrobot). Table shows the mean and standard error of the cumulative reward over test episodes on Acrobot, The reported mean cumulative value for DPT in Acrobot is -27.7 Yao al. 2018)
Table Steps to solve 2D navigation and Acrobot
Table 2: Error bars on Acrobot
2D navigation
Acrobot
Acrobot
Average
49:10.5
114+4.2
Average
-22.2+2.3
Oracle
12:+0.3
88:+3.6
Oracle
-17.1+2.2
BNN
344:0.8
1694440
BNN
-50.6+4.7
EPOpt-adv
49.10.3
91:3.88
EPOpt-ady
-17.5-22.2
MAML
49-10.4
99-14.6
MAML
-20.142.6
SEPT
20:10.9
10044..4
SEPT
-23.143.1
13

---

Published as a conference paper at ICLR 2020
D.2
TIMING COMPARISON
Table 3: Total training times in seconds all experiment domains
2D navigation
Acrobot
HIV
Average
1.3e3++277
1.0e3++85
1.4
Oracle
1.6c3+113
1,1c3:1129
1.5e3:447
BNN
2.9e31244
9.0c443.0e3
4.3c41313
EPOpt-adv
1.1c3:444
1.1e3+110
1.9e3433
MAML
0.9
1.1e3+99
1.3e3#6.00
SEPT
1.9e3=70
2.3
2.8e3#11
Table 4: Test episode time in seconds on all experiment domains
2D navigation
Acrobot
HIV
Average
0.04
0.09:00.04
0.42:10.01
Oracle
0.02++0.04
0.09:10.04
0.45++0.02
BNN
2.6e3:9577
2.8c31966
1.4c3
EPOpt-adv
0.04+10.04
0.10::0.06
0.45+0.03
MAML
0.05+00.55
0.10+0.07
0.48+0001
SEPT
0.04+10.07
0.12+00.10
0.60++0.02
D.3 PERCENT SOLVED EPISODES
100
100
Percent solved
Percent solved
80
EPOst
EPOpt-adv
MAML
60
MAML
Average
Oracle
Oracie
20
40
25
50
100
125
150 200
Step
Step
(a) 2D navigation
(b) Acrobot
Figure 5: Percent of solved test instances as a function of time steps during the test episode. Percentage computed among 200 test instances for (a) 2D navigation and (b) 100 test instances for Acrobot.
2D navigation and Acrobot are considered solved upon reaching terminal reward of 1000 and 10, respectively. Figure Figure the percentage of all test episodes that solved as a function of time steps the test episode. Percentage measured from a total of 200 test episodes for 2D navigation and 100 test episodes for Acrobot.
14

---

Published as a conference paper at ICLR 2020
D.4 TRAINING CURVES
SE-T
Average return
1000
Average return
3.5
RIGHT
6:00
-100
Oriact
2.0
200
1.0
-200
0.0
0.2
0.4
0.6
0.8
1.0
-250
0.0
05
10 1,5 2.0 255 Episode
3.0 355 4.0
0.5
Episode
Episode
(a) 2D navigation
(b) Acrobot
(c) HIV
Figure 6: Average episodic return over training episodes. Only SEPT and Oracle converged in 2D navigation. All methods converged in Acrobot. All methods except MAML converged in HIV. BNN not shown as the implementation (Killian el al. 2017) does not record training progress.
Figure 6 shows training curves on all domains by all methods. None of the baselines, except for Oracle, converge in 2D navigation, because it is meaningless for Avg and EPOpt-adv interpolate between optimal policies for each instance, and MAML cannot adapt due to lack of informative rewards for almost the entire test episode. Hence these baselines cannot work for new unknown test episode, even in principle. We allowed the same number of training episodes for HIV as in Killian al. 2017), and all baselines except MAML show learning progress.
D.5 LATENT REPRESENTATION OF DYNAMICS
$=10
2.8
0.5
2.4
2.5
B=
$=50
$=100
$=500
-55
06
-25 06
25
Figure 7: Two-ddimesioonll encodings generated for four instances of Acrobot (represented by four ground-truth colors), for different values of B. We chose 3 1 for Acrobot.
There is tradeoff between reconstruction and disentanglement increases (Higgins et al. 2017) Increasing encourages greater similarity between the posterior and an isotropic Gaussian. Figure gives evidence that this comes cost of lower quality of separation in latent space.
15

---

Published as a conference paper at ICLR 2020
D.6 PROBE REWARD
30/0000
Figure 8: Probe policy reward curve in one training run in 2D navigation
E EXPERIMENTAL DETAILS
For 2D navigation, Acrobot, and HIV, total number of training episodes allowed for all methods are 10k, 4k, and 2.5k, respectively. We switch instances once every 10, 8 and 5 episodes, respectively. There are 2, and 5 unique training instances, and 2, 5, and validation instances, respectively, For each independent training run, we tested on 10, 5, and test instances, respectively.
E. ALGORITHM IMPLEMENTATION DETAILS
The simple baselines Average and Oracle can be immediately deployed in a single test episode after training. However, the other methods for transfer learning require modification work in the setting of single episode test, as they were not designed specifically for this highly constrained setting. We detail the necessary modifications below. We also describe the ablation SEPT-NP in more detail.
BNN. In Killian er al. (2017), a pre-trained BNN model was fine-tuned using the first test episode and then used to generate fictional episodes for training policy from seratch. More episodes on the same test instance were allowed to help improve model accuracy of the BNN. In the single test episode setting, all fine-tuning and policy training must conducted within the first test episode. We fine-tune the pre-trained BNN every 10 steps and allow the same total number of fictional episodes as reported in (Killian et al., 2017) for policy training. We measured the cumulative reward attained by the aolicy-while it is undergoing training wainigg-during the single real test episode.
EPOpt. EPOpt trains on the lowest e-percentile rollouts from instances sampled from a source distri- bution, then adapts the source distribution using observations from the target instance (Rajeswaran ler al.) 2017). Since we do not allow observation from the test instance, we only implemented the adversarial part of EPOpt. To run EPOpt with off off-policy DDQN, generated 100 rollouts per iteration and stored the lowest 10-percentile into the replay buffer, then executed the same number of minibatch training steps the number that regular DDQN would have done during rollouts.
MAML. While MAML uses many complete rollouts per gradient step (Finn et af., (017). the single episode test constraint mandates that can only use partial episode for adaptation during test, and hence the same must be done during meta-training. For both training and test, we allow MAML to take one gradient step for adaptation using a trajectory of the same length as the probe trajectory of SEPT, starting from the initial state of the episode. We implemented a first-order approximation that computes the meta-gradient at the post-update parameters but omits second derivatives. This was reported to have nearly equal performance as the full version, due to the use of ReL.U activations.
SEPT-NP. To (a|s, 2) begins with zero-vector for a at the start of training. When it has produced trajectory Tp of length Tp, we store T into D for training the VAE, and use Tp with the VAE to estimate z for the episode, Later training episodes begin with the rolling mean of all = estimated so far. For test, we give the final rolling mean of at the end of training as initial input to Ty(a|s, z).
E.2 ARCHITECTURE
Encoder. For all experiments, the encoder (2(T)) is bidirectional LSTM with 300 hidden units and tanh activation. Outputs are mean-pos meea--polled over time, then fully colnecco to two linear output layers of width dim(z), interpreted as the mean and log-variance of Gaussian over z.
16

---

Published as a conference paper at ICLR 2020
Decoder. For all experiments, the decoder pe (T(()))) is an LSTM with 256 hidden units and tanh activation. Given input [st, ag ag.2] 2] at LSTM time step t, the output is fully-connected to two linear output layers of width [S] + |.Al, and interpreted as the mean and log variance of Gaussian decoder for the next state-action pair (8++1; 0(+1).
Q network. For all experiments, the Q function is fully-connected neural network with two hidden layers width 256 and 512, ReLU activation, and a linear output layer of size |A]. For SEPT and Oracle, the input is the concatenation [st, z]. where estimated in the case of SEPT and the ground truth in for the Oracle. For all other methods, the input is only the state 8.
Probe policy network. For all experiments, (als)) is fully-connected neural network with hidden layers, ReLU activation, 32 nodes in all layers, and softmax in the output layer.
E.3 HYPERPARAMETERS
VAE learning rate was le-4 experiments. Size dataset 'D of probe trajectories was limited to 1000, with carliest trajectories discarded. 10 minibatches from D were used for each VAE training step. We used minibatch size 32, training step was done for every 10 environment steps, Cend 0.15, learning rate was 1e-3, gradient clip was 2.5, y = 0.99, and target network update rate was 5e-3. Exploration decayed according (++1 c every episode, where satisfies Cand c. Stan and N is the total number of episodes. Prioritized replay used the same parameters in (Killian et al. 2017).
for the VAE. Probe policy learning rate was 1c-3 for all experiments. DDQN
Table 5: Hyperparameters used by each method, where applicable
2D navigation
Acrobot
HIV
Тр
2
5
Instances
1000
500
500
Episodes per instance
10
8
5
VAE batch size
10
64
64
dim(2)
2
2
a Probe minibatches
1.0
0,005
1.0
1
10
DDQN Start
1.0
1.0
0.3
17


Reviews:
[{"review_id": "Hyl52JbTYS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper addresses the problem of transfer in RL. After an agent is given an opportunity to train from a distribution of environments, we want an agent to perform well on the test environment. This paper specifically focuses on the setting where the state space, action space, reward space, and discount factor are the same across all environments, while the transition dynamics may differ. An environment's transition dynamics is assumed to depend on a hidden parameter that is not observed by the agent, in contrast to some previous work which assumes observability.\n\nThe idea of the main algorithm is as follows. During the training phase, a series of environments is presented. The agent is aware of the demarcations between the environments, but not of the identity (i.e., hidden parameter) of the environments themselves. A \"probing\" policy is run for a specified number of time steps. From the trajectory generated, a VAE is used to estimate the hidden parameter governing the transition dynamics. For the remaining time in the environment instance, a master policy, conditioned on the estimated parameter, is trained on the reward (any usual RL algorithm suffices; DDQN is used in the experiments).\nThe novelties of the approach are the problem setting (no access to optimal policies, immediate deployment at test-time), the joint training of a probing policy and VAE model, and a universal policy conditioned on the estimated hidden parameters. The experiments were conducted on domains that were, individually, both continuous and discontinuous with respect to the hidden parameters. The paper concludes that the proposed method improves upon baselines in terms of speed of reward attainment and computation time. Ablations were run for some of the components of the proposed algorithm. \n \nDecision: Weak Reject\n \nThe two key reasons for the Weak Reject were the following. \n1. Statistically insignificant results (3 runs)\n2. Questions about baselines used (see question 6 below)\n\nThe proposed algorithm is promising and attempts to directly address limitations in previous literature. The paper thoroughly contextualizes and motivates the current approach in light of previous work in the literature and discusses possible advantages: simplicity and generality; the ability to deploy immediately at test-time instead of having to train another policy (from learning a universal policy); assuming that the hidden parameters are unobservable, which is more realistic; not assuming access to optimal value functions/policies; the algorithm is model-free, so one can avoid the computational and memory overhead (when compared to BNN) of learning a model and avoid compounding model error. I also appreciated the thorough discussion of the key algorithmic choices and their trade-offs in section 3. The presentation of ideas is clear throughout.\n \nHowever, weak experiments keep this paper at a weak reject. The paper claims that the proposed algorithm \"significantly outperforms\" the baselines (abstract); however, more runs are needed to substantiate these claims. Only 3 runs were made for the test episode, which is not enough to be able to make empirical claims with non-trivial confidence (see https://arxiv.org/abs/1709.06560).  As well, since baseline methods are adapted from the original papers, more thorough hyperparameter sweeps should be performed. I wasn’t sure what “coarse coordinate search” was (pg. 6), but in general it would be good to write down the hyperparameters swept over. \n\nSome additional questions about the experiments:\n1. Did you try oracle with lower dimensional hidden parameter embedding? (understanding more why oracle does possibly worse in HIV and Acrobot)\n2. There is the claim on page 2 that the proposed algorithm is \"orders of magnitude faster than best model-based method\". What is the best model-free method?\n3. Is the universal policy of SEPT optimized during test time?\n4. Did you manage to replicate the results of DPT in Yao (2018)? Were architectures the same?\n5. What is the significance of not requiring rewards at test-time?\n6. Why were BNN, MAML, and EPOpt the baselines chosen? What about the other works mentioned in the related work section, like Tirinzoni (2018) or Paul (2019)? Did you try using an LSTM as a baseline to directly learn the policy? One could imagine treating the existence of a hidden parameter as inducing a partial observability problem. \n7. What is a \"coarse coordinate search\" over hyperparameters? What hyperparameters were tried?\n8. Did you try regular DQN? DDQN w/o prioritized replay? \n9. Why were the cumulative reward curves not as significant for Acrobot and HIV as compared to 2D navigation?\n10. Why different probe length settings for 2D navigation vs. Acrobot and HIV? How did these hyperparameters affect the time to solve?\n11. Why do 2D-room and Acrobot have terminal rewards of 1000 and 10?\n\nI would be willing to raise the score if the key experimental problems I noted were addressed. \n \nMinor comments/questions that did not impact the score\n1. Should include the table of computational time in the main body if possible since computational efficiency is one of the core claims\n2. Move DDQN comment in 3.2 to the experiments section\n3. How does the probing policy approach relate to work in active perception? (https://link.springer.com/article/10.1007/s10514-017-9666-5)\n\n\nEDIT: After the comments below which addressed my concerns, I have raised the score for this paper from a weak reject to an accept. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgzZIPrcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a strategy for single-trajectory transfer of a reinforcement learned policy.  They follow a typical approach in the few-shot supervised-learning community, of assuming that the plausible set of solutions may be modelled as a much lower dimensional latent variable, and then try to quickly infer that latent variable at test time.  In this case, the latent variable is ‘Z’.\nAt test time, they first run an exploitative algorithm (they call this the probe) to rapidly infer the value of Z, and thereby the optimal policy which this indexes, before switching to running the policy alone.\n\nWeak reject.\n\nI think that this paper (if indeed novel) is interesting, and I do agree that few-trajectory transfer in RL is a potentially impactful area in which to be working.  I think that the magnitude of the contribution is medium->low, however (namely that they have an approach which is wall-clock efficient, for single trajectory transfer), however the precise details of this contribution/claim have not been adequately tested:\nWe do not see experiments which highlight the wall-clock or inference cost competitiveness of the approach, and we do not see experiments against other, existing, approaches for latent-variable based RL (mostly these involve model-based RL, however these would probably be easy to derive for many of the test cases).\nWe also don’t see any experiments which evaluate the performance of SEPT as the time spent optimising the probe policy is varied.\nAs an aside, in figure 2(e) it appears that the Oracle policy is significantly outperformed by SEPT, do you have any thoughts as to why this might be?\n\nTo improve the paper, I would propose a slight rewrite to focus only on the core claim (i.e. why this model outperforms anything else for wall clock competitive single episode RL policy transfer).  I’d also be interested to see how this approach scales as the size of the potential policy set increases (e.g. if we had to produce a policy which would generalise for for all ball sports).\nAny theoretical work would also be very welcome—one may be able to draw on existing work in the k-shot community to make a judgement as to what fraction of the trajectory is required before one should have a low entropy estimate of Z, and it would also be nice to know how the approach scales as the complexity of Z ramps up—i.e. as we need to support more tasks.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xwruy6tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of this paper is to learn a universal policy that is able to perform near-optimally on test tasks with transition dynamics that were never observed during training. This is achieved by using a \"probe policy\" to generate short trajectories that are then used to learn a latent encoding to categorise the transition dynamics of the current task. The universal policy is then conditions on both the state and this encoding so that the learned policy can perform well on tasks with different dynamics.\n\nOverall I really like this paper. There has been a big push in the RL community to start evaluating algorithms on test tasks that are different to trained tasks and this paper takes a good step in this direction.\n\nMy main concern with this paper, which I would appreciate some feedback from the authors on, is regarding the trajectory length of the probe policy:\n\n1. The method seems to rely heavily on the ability to learn an accurate latent encoding from a short trajectory which will may not be the case in many domains. It is easy to construct some domain where the dynamics behave identically initially but have some major differences deeper into the task at hand. I noted that the authors did mention this in their conclusion as an area for future research but I want to know if they have any high level ideas on how to rectify this because it seems that this will be a major limitation of the proposed algorithm in practice at least for certain domains.\n\n2. In the experiments I can see that in some cases a longer probe performed more poorly than a shorter probe (for example Fig3a for 2D navigation. My intuition tells me that a longer probe should outperform a shorter probe since it contains at least as much information. Can the authors explain why this occurs?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting paper that is concerned with single episode transfer to reinforcement learning problems with different dynamics models, assuming they are parameterised by a latent variable. Given some initial training tasks to learn about this parameter, and a new test task, they present an algorithm to probe and estimate the latent variable on the test task, whereafter the inferred latent variable  is used as input to a control policy.

There were several issues raised by the reviewers. Firstly, there were questions with the number of runs and the baseline implementations, which were all addressed in the rebuttals. Then, there were questions around the novelty and the main contribution being wall-clock time. These issues were also adequately addressed.

In light of this, I recommend acceptance of this paper.
Decision: Accept (Poster)