Keep Doing What Worked: Behavior Modelling Priors for Offline Reinforcement Learning
Authors: Noah Siegel
Jost Tobias Springenberg
Felix Berkenkamp
Abbas Abdolmaleki
Michael Neunert
Thomas Lampe
Roland Hafner
Nicolas Heess
Martin Riedmiller
Venue:  (2020)
Keywords: Reinforcement Learning
Off-policy
Multitask
Continuous Control

Abstract: Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior -- the advantage-weighted behavior model (ABM) -- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find  improvements on competitive baselines in a variety of RL tasks -- including standard continuous control benchmarks and multi-task learning for simulated and real-world robots. 

Published as a conference paper at ICLR 2020
KEEP DOING WHAT WORKED: BEHAVIOR MODELLING PRIORS FOR OFFLINE REIN- FORCEMENT LEARNING
Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, Martin Riedmiller
DeepMind
( s [siegelng [ @google.com
ABSTRACT
Off-poolic reinforcement learning algorithms promise be applicable in settings where only fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior - the sedvaattageweighhee behavior model (ABM)- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen extension of recent work batch-RL. that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks including standard continuous control benchmarks and multi- task learning for simulated and real-world robots. Videos are available at hhtpps //sites. http:///eao.////////////ooooooaa
INTRODUCTION
Batch reinforcement learning (RL) (Ernst et al. 2005 Lange et al. 2011) the problem of learning a policy from a fixed, previously recorded, dataset without the opportunity to collect new data through interaction with the environment. This is in contrast to the typical RL setting which alternates between policy improvement and environment interaction (to acquire data for policy evaluation). In many real world domains collecting new data is laborious and costly. both in terms of experimentation time and hardware availability but also in terms of the human labour involved in supervising experiments. This is especially evident in robotics applications (see e.g. |Riedmiller et al. 2018 Haarnoja et al. 2012288))) Kalashnikov et al. 2018 for recent examples learning robots). In these settings where gathering new data is expensive compared to the cost of learning, batch RL promises to be a powerful solution.
There exist a wide class of off-policy algorithms for reinforcement learning designed handle data generated by a behavior policy which might differ from क, the policy that we are interested in learning (see e.g. [Sutton & Barto (2018) for an introduction). One might thus expect solving batch RL to be straightforward application of these algorithms. Surprisingly. for batch RL in continuous control domains, however, FFujimooo al. (2018) found that policies obtained via the naive application of off-policy methods perform dramatically worse than the policy that was used to generate the data. This result highlights the key challenge batch RL: we need exhaustively exploit the information that is in the data but avoid drawing conclusions for which there is evidence (i.e. need to avoid over-valuing state-action sequences not present in the training data).
As we will show in this paper, the problems with existing methods in the batch learning setting further exacerbated when the provided data contains behavioral trajectories from different policies , NN which solve different tasks, the same task in different ways (and thus potentially execute conflicting actions) that are not necessarily aligned with the target task that π should accomplish. We empirically show that previously suggested adaptations for off-py learning

---

Published as a conference paper at ICLR 2020
(Fujimoto et al. 2018 Kumar et al. 2019) can be led astray by behavioral patterns in the data that consistent (i.e. policies that try to accomplish a different task or a subset of the goals for the target task) but not relevant for the task at hand. This situation is more damaging than learning from noisy or random data where the behavior policy is sub-optimal but is not predictable, i.e. the randomness is not correlated signal that will be picked up by the learning algorithm.
We propose to solve this problem by restricting our solutions to "stay close to the relevant data'. This done by: 1) learning prior that gives information about which candidate policies are potentially supported by the data (while ensuring that the prior focuses on relevant trajectories), 2) enforcing the policy improvement step to stay close to the learned prior policy. We propose a policy iteration algorithm in which the prior is learned to form an ad advantage-we waiihge-weighted model of the behavior data. This prior biases the RI. policy towards previously experienced actions that also have high chance of being successful in the current task. Our method enables stable learning from conflicting data sources and we show improvements on competitive baselines in a variety of RI. tasks including standard continuous control benchmarks and multi-task learning for simulated and real-world robots. We also find that utilizing an appropriate prior is sufficient to stabilize learning; demonstrating that the policy evaluation step is implicitly stabilized when a policy iteration algorithm is used long as care is taken to faithfully evaluate the value function within temporal difference calculations. This results in simpler algorithm than in previous work (Fujimoto et al. 2018 Kumar al. 2019).
2 BACKGROUND AND NOTATION
In the following we consider the problem of reinforcement learning, modeling the environment as a markov decision process (MDP) consisting of the continuous states E S, actions E A, and transition probability distribution p(s++1 8t, ax) describing the evolution of the system dynamics over time (c.g. probability of reaching 51+1 from state s; when executing action a,) - together with the stste-visitation distribution p(s). The goal of reinforcement learning is to find policy (als)) that maximizes the cumulative discounted return J(T) = E,
1== [ T (-Iss), St-1 p(-|st, ae), $1 ~ p(-)], for the reward function r(s, E R. We also define the state-action value function for taking action a, in state 8t, and thereafter following T: Q"
r'r(SE, at)la la) ~
E. ( 7r(ss, da) TT, 81+1 ~ p(-1s;, dz)], which we can relate the objective J via J(x) Ezmopt)) [. Fann(1a;)( (Q= (s, a))l, where x² the optimal policy. We parameterize the policy no (a(a)s) by 0 but we will omit this dependency where unambiguous. In some of the experiments we will also consider a setting where we learn about multiple tasks & E {l.... K}, each with their own reward function rs,ss a). We condition the policy and Q-function on the task index l (i.e. Q and r T[aas, k)), changing the objective to maximize the sum of returns across all tasks.
For the batch RL setting we assume that we given dataset D,, containing trajectory snippets (i.e. sub-trajectorics of length N) r = { ao), , (ST. }. with 7 E D,,. We assume access to the reward function for the task of interest and can evaluate it for all transitions in Dr (for example, may be some function of s). We further assume D, was filled, prior to training, by following a set of arbitrary N behavior policies Mi,... ,/N-- Note that these behavior policies may try accomplish the task we are interested in; or might indeed generate trajectories unrelated to the task at hand.
3 A LEARNED PRIOR FOR OFFLINE OFF-POLICY RL FROM IMPERFECT DATA
To stabilize off-policy RL from batch data, we want to restrict the learned policy to those parts of the state-action space supported by the batch. In practice this means that we need to approximately restrict the policy to the support of the empirical state-conditional action distribution. This prevents the policy from taking actions for which the Q-function cannot be trained and for which it might thus give erroneous, overly optimistic values (Fujimoto et al., 2018 Kumar et al. 2019). In this paper achieve this by adopting a policy iteration procedure- in which the policy constrained in the improvement step. As in standard policy iteration (Sutton & Barto 2018). the procedure consists of two alternating steps. First, starting with a given policy я. Te. iteration i (with Te corresponding to a randomly initialized policy distribution), we find an approximate action-value function QTi (s. a) ~ Q(s, a; Φ1). with parameters & (Section 3.1 (as with the policy we will drop the dependence on di and write Q (s, a) where unambiguous). Second, we optimize for T with respect Qt. subject to constraint that ensures closeness to the empirical state-conditional action
2

---

Published as a conference paper at ICLR 2020
distribution of the batch (Section 3.2). Iterating these steps, overall, optimizes J(x). We realize both policy evaluation and improvement via fixed number of gradient descent steps holding æ and Q fixed via the use of target networks (Mnih et al. 2015) We refer to Algorithm for details.
3.1 POLICY EVALUATION
To learn the task action-value function in each iteration we minimize the squared temporal difference error for a given reward - note that when performing offline RL from batch of data the reward might be computed post-hoc and does not necessarily correspond the reward optimized by the behuvior policies p1,"* ,PN. The result after iteration is given as
Φ; = arg min E & TWDD (7(s)) ar) + 7 (8++1) Q(e ori (r:)) ae (041)~~7 NT]
(1)
with (s) (-s)
We approximate the expectation required to calculate (s) with M samples from Tú i.e. V (s) R M [>j=1 =1 (s, (18,));0-1)) 0j ~ TT(-)s)]. As further discussed in the related work Section 4 the use of policy evaluation is different from the Q-learning approach pursued in Fujimoto et al. (2018;; actions and may be more susceptible to over- estimation of Q-values. We find that when enough samples are taken (we use M 20) and the policy appropriately regularized (see Section 3.2) learning is stable without additional modifications.
Kumar al. (2019), which requires maximum
3.2 PRIOR LEARNING AND POLICY IMPROVEMENT
In the policy improvement step we solve the following constrained optimization problem
11 TNDD
-]
Ti+1 arg max
(2)
s.t. R D.D
[||[(-|s)]| p-))))) 7] Se.
where D,, is the behavior data, the policy being learned, and Tproor the prior policy. This is similar to the policy improvement step in AAbollmaleki et al. (2018) but instead of enforcing closeness to the previous policy here the constraint is with respect to a separately learned "prior" policy, the behavior model. The role of prior in Equation iis is to keep the policy close to the regime of the actions found in D,.. We consider two different ways to express this idea by learning prior alongside the policy optimization.
For learning the prior, we first consider simply modeling the raw behavior data. This is similar the approach of BCQ and BEAR-QL (Fujimoto et al. [018 Kumar et al.) (2019). but
parametric behavior model and measure distance by KL; we refer to the related work for discussion. The behavior model can be learned by maximizing the log likelihood of the observed data
[r|
Oom arg max
E
Σ
log The (arlss)
(3)
Ove TND,
where 6mm are the parameters of the behavior model prior.
Regularizing towards the behavior model can help prevent the use of unobserved actions, but may also prevent the policy from improving over the behavior in D,,. In effect, the simple behavior prior in Equation (3 regularizes the new policy towards the empirical
conee-ioniiionnlll action distribution in Dp. This may be acceptable for datasets dominated by successful trajectories for the task of interest or when the unsuccessful trajectories are not predictable (i.c. they correspond to random behaviour). However, we here are interested in the case where D,, is collected from imperfect data and from multiple tasks. In this case, D,, will contain a diverse set of trajectories both (partially) successful and actively harmful for the target task. With this in mind, aalaaaatag-wiiihhee behavior model, Jahm- with which we can bias the RL policy to choose actions that are both supported by D,, and also good for the current task (i.e. keep doing actions that work).
consider second learned prior, the
3

---

Published as a conference paper at ICLR 2020
We can formulate this maximizing the following objective:
Σ log AB (ar)st)s (R(TEN)) ( - (vrc (6))
Onb = arg max E Opn Do
(4)
N-1
with R (TENN = -N-ipm (SN) + Σ (((
where f is an increasing, non-negative function, and the difference R (Fr.N) V= is akin to an n-step advantage function, but here calculated off ooflcy representing the "advantage" of the behavior snippet over the policy тς. This objective still tries to maximize the log likelihood of observed actions, and avoids taking actions not supported by data. However, by "advantage weighting" we focus the model on "good" actions while ignoring poor actions. We let -== 1+ (the unit step function with and otherwise) both for simplicty to keep the number of hyperparameters to minimum while keeping the prior broad and because it has an intuitive interpretation: such prior will start by covering the full data and, over time, filter out trajectories that would lead to worse performance than the current policy, until it eventually converges to the best trajectory snippets contained in the data. We note that Equation 4 similar to policy gradient, though samples here stem from the buffer and it will thus not necessarily converge to the optimal policy in itself; T0 will instead only cover the best trajectories in the data due to no importance weighting being performed for off solicol data. This bias is in fact desirable in batch-RL setting; want broad prior that only considers actions present in the data. We also note that we tried several different functions for including exponentiation, e.g. f(x)= exp(x), but found that choice of function did not make significant difference in our experiments.
f()) = for
Using either Te... Tm as priior Equation can be solved with variety of optimization schemes. We experimented with an EM-style optimization following the derivations for the MPO algorithm (Abdolmaleki et al. 2018), ,), as well as directly using the stochastic value gradient of Q". wrt. policy parameters (Heess et al. 2015).
It should be noted that, if the prior itself is already good enough to solve the task we can learn Q e.g. if the data stems from an expert has sufficiently high quality. In this case learning both QT and Tprior becomes independent of the RL policy improvement step; if we then set e = 0. skipping the policy improvement step, we obtain a further simplified algorithm consisting only of learning T0 and Tean (see Figure 9 for an ablation).
EM-style optimization We can optimize the objective from Equation 2 usiig a two-step procedure. Following (Abdolmaleki et al. 2018), we first notice that the optimal for Equation can be expressed as क (a]s) X Trrror (a]s) (a () exp((" (s.,///))) where 7) is a temperature that depends on the c used for the KL constraint and can be found automatically by convex optimization (Appendix B.I). Conveniently,
sample from this distribution by querying Q using samples from Tyrior- These samples can then be used to learn the parametric policy by minimizing the divergence K KL( T which is equivalent maximizing the weighted log likelihood
0.+1 = arg max 0
E E T~D,
exp (ep(""((sa)/))))) log [als(s))ssss ~T]]
(5)
which we optimize via gradient descent subject to an additional trust-region constraint on अe given as KT[TTo, (T))) Crrust to ensure conservative updates (Appendix B.I.
Stochastic value gradient optimization Alternatively, we can use Langrangian relaxation to turn Equation into objective amenable gradient descent. Inserting ne for and relaxing results in
(0.+1.7) arg max min E " T-DD
(anno((1s) [1 (s, a)] + n7(= (L]o (=)))]]
(6)
for 17 > 0 and which we can optimize by alternating gradient descent steps on 0 and 17 respectively, taking the stochastie gradient of the Q-value (Heess al. 2015) through the sampling of a (T(ss) via re-parameterization. See Appendix B.2 for a derivation of this gradient.
Full pseudocode for our approach can be found in Appendix Algorithm 1
4

---

Published as a conference paper at ICLR 2020
4 RELATED WORK
There exist a number of off-cy RL algorithms that have been developed since the inception of the RL paradigm (see e-g. [Sutton & Barto (2018) for an overview). Most relevant for our work, some of these have been studied in combination with function approximators (for estimating value functions and policies) with an eye on convergence properties the batch RL, setting. In particular, several papers have theoretically analyzed the accumulation of bootstrapping errors in approximate dynamic programming (Bertsekas & Tsitsiklis 1996 Munos] 2005) and approximate policy iteration (Farahmand et al., 2cherrec et al. 2015 for the latter of which there exist well known algorithms that are stable at least with linear function approximation (see e.g. Lagoudakis & Parr (2003)). Work RI. with non-linear function approximators has mainly considered the "online" or "growing batch" settings, where additional exploration data is collected (Ernst et al. [2005] Riedmiller 2005; Ormoneit & Sen 2002); though some success for batch RL in discrete domains has been reported (Agarwal et al.) 2019). For continuous action domains, however, off-policy algorithms that are commonly used with powerful function approximators fail in the fixed batch setting.
Prior work has identified the cause of these failures as extrapolation or bootstrapping errors (Fujimoto ct a 2018 Kumar et al.) 2019) which occur due to failure to accurately estimate Q-values, especially for state-action pairs not present in the fixed data set. Greedy exploitation of such misleading Q-values (e.g. due to a max operation) can then cause further propagation of such errors in the Bellman backup, and to inappropriate action choices during policy execution (leading to suboptimal behavior). In non-batch settings, new data gathered during exploration allows for the Q-function to be corrected. In the batch setting, however, this feedback loop is broken, and correction never occurs.
To mitigate these problems, previous algorithms based on Q-learning identified two potential solutions: 1) correcting for overly optimistic Q-values in the Bellman update, and 2) restricting the policy from taking actions unlikely to occur in the data. To address 1) prior work uses a Bellman backup operator in which the max operation is replaced by a generative model of actions (Fujimoto et al. 2018) which a learned policy is only allowed to minimally perturb; or via a maximum over actions sampled from a policy which is constrained to stay close to the data (Kumar al., 2019) (implemented through constraint on the distance to a model of the empirical data, measured either in terms of maximum mean discrepancy or relative entropy). To further penalize uncertainty in the Q-values this can be combined with Clipped Double-Q learning (Fujimoto et al., 2018) or an ensemble of Q-networks (Kumar et al., 2019). To address 2) prior work uses a similarly constrained max also during execution, by considering only actions sampled from the perturbed generative model (Fujimoto et al. 2018) or the constrained policy (Kumar et al. 2019), and choosing the best among them.
Our work is based on policy iteration scheme instead of Q-learning exchanging the max for an expectation. Thus we directly learn a parametric policy that we also use for execution. We estimate the Q-function part of the policy evaluation step with standard TD-0 backups. We find that for an appropriately constrained policy no special treatment of the backup operator is necessary, and that it is sufficient to simply estimating V (see Equation [T]. The only modification required is in the policy improvement step where we constrain the policy to remain close the adaptive prior in Equation As demonstrate in the empirical evaluation it is the particular nature of the adaptive prior which can adapt to the task hand (see Equation 4 - that makes this constraint work well. Additional measures account for uncertainty in the Q values could also be integrated into our policy evaluation step but we did not find it to be necessary for this work; we thus forego this in favor of our simpler procedure.
an adequate number of samples to approximate the expectation when
Our policy iteration scheme also bears similarity to previous works that use (relative) entropy regularized policy updates. These use constraints with respect to either a fixed (c.g. uniform) policy (c.g. [Harmoja et al., 2018a) or, in a trust-region like scheme, to the previous policy (c.g. Abdolmaleki et al. 2018). Other work has also focused on policy priors that are optimized to be different from the actual policy but so far mainly in the multi-task or transfer-learning setting (Teh et al. 2017 Galashoy ctt al. 2019 Tirumala et al.) 2019||p Jaques et al., 2017). The constrained updates are also related to trust-region optimization in action space, e.g. in TRPO / PPO (Schulman et al. 2015 2017) and
MPO (Abdolmaleki et al. 2018), ) which ensures stable learning by enforcing conservative updates; an idea that can be traced back to Kakade & Langford| (2002). Here take slightly different perspective: we enforce a trust region constraint not on the last policy in the policy optimization loop (conservative updates) but wrt. the advantage weighted behavior distribution.
5

---

Published as a conference paper at ICLR 2020
Cheetah (10K)
Hopper (1.kk
Que druper (10K)
(101)
8000
Average
4.00
600
8.00
400
2.00
200
200
Learming Step
le6
Learning Stop
leb
Learning Stop
1e6
Learning Stop
leb
AIM+NPO
AISM-SXC
8MM+MPO
MPD
SVG
prror
BEARKLI
BMiprior)
Figure Learning control suite tasks from fixed data. Standard off oof-po RL fails on Cheetah, Hopper, Quadruped. Methods that regularize to the data distribution succeed; for hopper the behaviour distribution is multimodal and standard behavior modlling prior broad (see (M(prior) here only ABM succeeds. Best trajectory in data marked with star.
Cheetah 12k)
Happer (240
Quadruped (2k)
Walleer (2k)
Average Rletum
750
500
400
500
250
200
250
Leaminng
1e5
Leamming
165
Learning
105
Learning
1e5
ABM MPO
BM MPO
AB M[pioo)
BEAR(KL)
BElprior]
Control suite when using only the first 2k episodes of low-quality data from each run; return for best behavior episode in the data was 718/159 011//43///55 435/728 (left to right). While plain RI. is able to learn on walker, learned priors improve performance and stabilize learning for cheetah and quadruped (with overall lower performance than when learning from good data).
Figure 2:
EXPERIMENTS
We experiment with continuous control tasks in two different settings. In first set of experiments we compare our algorithm to strong off- off-policy baselines on tasks from the DeepMind control suite (Tassa) (et al. 2018) give reference point as to how our algorithm performs common benchmarks. We then turn to the more challenging setting of learning multiple tasks involving manipulation of blocks using robot arm in simulation. These span tasks from reaching toward block to stacking one block on top of another. Finally, we experiment with analogous tasks on a real robot.
We use the same networks for all algorithms that compare, optimize parameters using Adam (Kingma & Bal 2013), and utilize proprioceptive features (c.g. joint positions velocities) together with task relevant information (mujoco state for the control suite, and position/velocity estimates of the blocks for the manipulation tasks). All algorithms were implemented in the same framework, including our reproduction of BCQ and BEAR, and differ only in their update rules. Note that for BEAR we use KL instead of the MMD as we found this to work well, see appendix. In the multi-task setting (Section 5. we learn task conditional policy Te (als, k) and Q-function Q7 (s, a, k) where k is one-hot encoding of the task identifier, that is provided as an additional network input. We refer the appendix for additional details.
5.1 CONTROL SUITE EXPERIMENTS
We start by performing experiments on four tasks from the DeepMind control suite: Cheetah, Hopper, Quadruped. To obtain data for the offline learning experiments we first generate a fixed dataset via standard learning run using MPO, storing all transitions generated; we repeat this with 5 seeds for each environment. We then separate this collected data into two sets: for experiments in the high data regime, we use the first 10,000 episodes generated from each seed. For experiments with low qualuullty data the first 2,000 episodes from each seed. The high data regime therefore has both more data and data from policies which are of higher quality on average. A plot showing the performance of the initial training seeds episodes is given in the appendix, Figure

---

Published as a conference paper at ICLR 2020
Reach
Grasp
StackLeave
Averag Ruturn
100
100
75
100
0
0.0
0.5
1.0
1,5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.
0.0
0.5
1.5
Learning Stop
1eb
Learnnng Stop
1e0
Learning Stop
1e6
Learning Stop
1e6
ABMi MPO
BM+MPO
MPO
ABM]prior]
BCQ BEAR
BM[orfor]
Figure 3: Learning curves for MPO with and without behavior extraction priors. We show 4 of the 7 tasks here. Five training runs are shown for each parameter setting, evaluation is average reward over 50 episodes. Behavioral modelling priors significantly improve performance across all tasks, and ABM further improves learning speed and final performance.
Dring to center
80 50
80
Average Retum
60
46
40
20
20
8.0
8.0
0.5
Leamning Stop
Learnnng Stap
APM-MPO
BM MPO
MPO
Figure 4: (Left) Learning curves for the tasks "bring to corner" and "bring to center". These tasks were learned using only data from the seven intial stacking tasks. The stacking dataset was rich enough to learn these new tasks fully offline with ABM. (Right) Simulated Sawyer environment.
For our offline learning experiments, we reload this data into a replay buffer The dataset is then fixed and no new transitions are added; the offline learner never receives any data that any of its current or previous policies have generated. We evaluate performance by concurrently testing the policy in the environment. The results of this evaluation are shown in Figure [ As can be observed, standard off off-policy RL algorithms (MPO SVG) can learn some tasks offline with enough data, but learning is unstable even on these relatively simple control suite tasks - confirming previous findings from Fujimoto et al. 2018 Kumar et al. 2019). In contrast the other methods learn stably in the high-data regime, with BCQ lagging behind in Hopper and Quadruped (sticking too close to the VAE prior actions, an effect already observed in (Kumar al. (019)). Remarkably our simple method of combining policy iteration loop with a behavior model prior (BM+MPO in the plot) performs well or better than the more complex baselines (BEAR and BCQ from the literature). Further improvement can be obtained using our advantage weighted behavior model even in some of these simple domains (ABM+SVG and ABM+MPO). Comparing the performance of the priors (B.Mprrior] vs ABM[prior], dotted lines) on Hopper we can understand the advantage that ABM has over B.: the BM prior performs well on simple tasks, but struggles when the data contains conflicting trajectories (as in Hopper, where some of the seeds learn sub-optima jumping) leading to too hard constraints on the RL policy. Interestingly, the ABM prior itself performs as well or better than the baseline methods for the control-suite domains. Furthermore, in additional experiments presented in the appendix we find that competitive performance can be achieved, in simple domains, when training only an ABM prior (effectively setting 0); providing even simpler method when does not squeezing out every last bit of performance. test on lower quality data, Figure ? shows similar trends, with method learning perform slightly better than the best trajectories in the data.
about
5.2 SIMULLTEDD ROBOT EXPERIMENTS
We experiment with a Sawyer robot arm simulated in Mujoco (Todorov et al. 2012) in a multi-task setting - as described above. The seven tasks are to manipulate blocks that are placed the workspace of the robot. They include: reaching for the green block (Reach), grasping any block (Grasp), lifting the green block (Lift), hovering the green block over the yellow block (Place Wide), hovering the green block over the center of the yellow block (Place Narrow), stacking the green block on top of
'Due to memory constraints, it can he prohibitive to store the full dataset in replay. To circumvent this problem run set of "restorers". which read data from disk and add it back into the replay in loop.
7

---

Published as a conference paper at ICLR 2020
230
Algerthm MPo Original Run
250
AB ABAMAADD) affine
300
200
Lift
Place Widerface Narrow
Stack
Stack/Lawve
Tasc
Figure 5: (Left) Original and offline learning on the real robot. The original training run takes over 200 hours to learn stack and leave (blue bar, right y-axis) bottle-necked by slow real-robot episode generation. Using the complete dataset of interactions in a batch-RL setting with the ABM prior, we are able to re-learn the final task in 12 hours purely from logged data (orange bar, right y-axis), while obtaining policy that achieves similar performance on all tasks (box plots, distribution of returns over 50 test episodes, orange vs blue, left y-axis). (Right) Real Sawyer environment.
yellow (Stack and Stavk/Leave i.. without gripper contact). To generate the data for this experiment again MPO here simultaneously learning all task-conditional policies for the full seven tasks. Data-was collected by randomly switching tasks after each episode (of 200 control steps) with random resets of the robot position every 20 episodes. As before, data from all executed tasks is collected in one big data-set annotating each trajectory snippet with all rewards (i.e. this is similar to the SAC-R setting from (Riedmiller al. 2018
During offline learning we then compare the performance of MPO and RL with a behavior modelling prior (BM+MPO and ABM+MPO). As shown Figure 3 behavioral modelling priors improve performance across all tasks over standard MPO which struggles in these more challenging tasks. This is likely due to the sequential nature the tasks: later tasks implicitly include earlier tasks but only smaller fraction of trajectories achieve success on stack and leave (and the actions needed for stack conflict, c.g., with lifting), this causes the BM prior to be overly broad (see plots in appendix). The ABM+MPO, with ABM in place, the RL policy learned using this prior still outperforms the prior, demonstrating that RL is still useful in this setting.
the other hand, achieves high performance across all tasks. Interestingly, even
As an additional experiment we test whether we can learn new tasks entirely from previously recorded data. Since our rewards are specified as functions of observations, tasks (bringing the green block to the center and bringing it to the corner) for the entire dataset- we then test the resulting policy in the simulator. As depicted in Figure 4 this is successful with ABM+MPO, demonstrating that we can learn tasks which were not originally executed in the dataset (as long as trajectory snippets that lead successful task execution are contained in the data).
compute rewards for two new
5.3 REAL ROBOT EXPERIMENTS
Finally, to validate that our approach is feasible solution to performing fast learning for real-robot experiments, we perform an experiment using a real Sawyer arm and the same set of seven tasks (implemented on the real robot) from Section 5.2 As before, data from all executed tasks is collected in one big data-set annotating each trajectory snippet with all rewards. The full buffer after about two weeks of real robot training used the data for offline learning: which we here only performed with ABM+MPO due to the costly evaluation. The goal is to re-learn all seven original tasks. Figure 5 shows the results of this experiment we ran an evaluation script on the robot, continuously testing the offline learned policy, and stopped when there was no improvement in average reward (as measured over a window of 50 episodes). As can be seen, ABM with MPO as the optimizer manages to reliably re-learn all seven tasks purely from the logged data in less than 12 hours. All tasks can jointly be learned with only small differences in convergence time - while during the initial training run the harder tasks, of course, took the most time to learn. This suggests that gathering large data-sets of experience from previous robot learning experiments, and then quickly extracting the skills of interest, might be viable strategy for making progress in roboties.
8

---

Published as a conference paper at ICLR 2020
6 CONCLUSION
In this work, we considered the problem of stable learning from logged experience with off-policy RL algorithms. Our approach consists of using learned prior that models the behavior distribution contained in the data (the advantage weighted behavior model) towards which the policy of an RL algorithm regularized. This allows to avoid drawing conclusions for which there is no evidence in the data. Our approach is robust to large amounts of sub-optimal data, and compares favourably to strong baselines on standard continuous control benchmarks. We further demonstrate that our approach can work in challenging robot manipulation domains - learning some tasks without ever seeing a single trajectory for them.
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv:1806.069: 2018.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity in off-policy deep reinforcement learning. CoRR, abs/ abs/1107..0m 200700444,, 2019.
Dimitri P. Bertsekas and John N. Tsitsiklis. Heuro-Dynamic Programming. Athena Scientific, 1996.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learn- ing. Journal of Machine Learning Research (JMLR), 2005. URL http:: acm. org/
citation wfm?id==0000000 14490 1088690
Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems (NeurIPS) 23. 2010.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900 2018.
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL-regularized RL. In International Conference on Learning Representations, 2019, URL //ttt:s:///ppnnevieew ne www.foorrmmiooooo S11.ddd-1111101011
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning (ICML). PMLR, 2018a. URL http://
proceedings mllr. hat..pprrsss////araaoooaa
html
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/ 812.05905, 2018b.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Frez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances Neural Information Processing Systems 28 (NeurIPS). 2015.
Natasha Jaques, Shixiang Gu. Dzmitry Bahdanau, José Miguel Hernández-Lobate Richard E. Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Proceedings of the 34th International Conference Machine Learning (ICML), 2017. BtRp: http/ttt////poceeiiggg.s.. mir /ttps///ww////uuuullaaaaaa html
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning (ICMI.), 2002.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scal- able deep reinforcement learning for vision- based robotic manipulation. CoRR, abs/ 808..006.. 2018.

---

Published as a conference paper at ICLR 2020
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015. URL htttp://axiv. org/ abs/1412.698
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 1906600944) 2019.
Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine Learning Research (JMLR), 2003.
S. Lange, T. Gabel, and M. Riedmiller. Batch Reinforcement Learning. In M. Wiering and M. van Otterlo (eds.), Reinforcement Learning: State of the Art. Springer, 2011.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Rémi Munos. Error bounds for approximate value iteration. In Proceedings of the Twentieth National Conference on Artificial Intelligence (AAAI, 2005.
Dirk Ormoneit and Saunak Sen. Kernel-based reinforcement learning. Machine Learning, 2002.
URL /BtEpp/://od..ccr//0..0////////
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning (ICML), 2014.
Martin Riedmiller. Neural fitted q iteration first experiences with data efficient neural reinforce- ment learning method. In Proceedings of the 16th European Conference on Machine Learning (ECML), 2005. httphttp:////t
doi org 10.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint arXiv: 10567, 2018
Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, and Matthieu Geist. Approximate modified policy iteration and its application to the game tetris. Journal of Machine Learning Research (JMLR), 2015, URL http: jmir. http://pppesselllle/eeeeeeeeeeee
html
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.063477 2017.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
html
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/180) 500600690, 2018.
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czameeki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. CoRR, 00s/1707.04175 2017. URL TtRp://t:://att
wrg.:brggbbbba.b 707. 04175
Dhruva Tirumala, Hyeonwoo Noh. Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Tch, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl. In arXiv: 1903.0748, 2019.
10

---

Published as a conference paper at ICLR 2020
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-50265555 IEEE, 2012.

---

Published as a conference paper at ICLR 2020
cony
cyme cherta tadk came
.00T FL crame copper fade rame sop
domatt sappe qua-iiuadddnn quadripe same
comTi pame
507
802
Figure 6: Learning curves for five seeds of training with MPO on the control suite tasks. Data generated during these training runs is later used for offline learning.
A
ALGORITHM
full algorithm listing for our procedure is given in Algorithm
Algorithm 1 RL with Behavior Extraction Priors
Input: N Nsteps number of learning steps, Ny: steps between target update, M number of action samples, KI. regularizziii parameter c, initial parameters for 0 17, and Φ
initialize N == 0, 0' = 0, 6'
while S NNscps do sample batch B of trajectories from replay buffer D sample M actions from policies to estimate expectations below I/ compute gradient for prior model
For BM prior:
60 Ooic ΣrEB E(sra.)er log Thn (ae(s))
For ABM prior:
8pp Ver Eres ((sraa)er 1 (R((T.)))) Ve (st)) log Temm (aclse)
I/ compute gradients for Q, and !)
8% + V Eres Enn EimreeEr (r(s), + (Vor(s() - Qs(s)
For MPO:
8 Vo ETEB Eoeer Eanipia (exp (Qo(()))))))) log [T(a[())]] + a(Cunu K KL [ (1=0))
8, - V. Eres EmEr ar(Eirust KT. KL ||=0])
8, +V Vnn(()) EnEB V V, + +1 log E Eawtaie exp (oxp(2wr) (s):a))/n))
For Stochastic Value Gradient:
8x +- -TEB Vo >mer En-oT (11s) [Qar (se, a)] + (e - n2 L
8n - TEB V, Σ Emer n/E K T. I |
8a f- // not used
I/ apply gradient updates
0, Oprior; 1), at á = aptimizer__pdate((( Opriors 77, at 0), 18n, 80 S,p Sc. 80l).
=i+1
// update target networks
if mod NTU 0 then
0' 0,
end if
end while
B DETAILS ON POLICY IMPROVEMENT
We here give additional details on the implementation of the policy improvement step in our algorithm. Depending on the policy optimizer used (MPO or SVG) different update rules are used to maximize the objective given in Equation 4 This is also outlined in Algorithm I We here describe the general form for both algorithms, using Terior to represent the prior which then will instantiated as either the ABM BM prior.
12

---

Published as a conference paper at ICLR 2020
B.1 MPO
For the EM-style optimization based on MPO we first notice that the optimal non-parametric policy that respects the KL constraint wrt. Tprior is
# #(ass) TTproor exp(("((x.33)/) log Z),
with 2 = constraint e. In practice estimate 2 based the M samples {a1..... am J ~ [alo that we draw for each state to perform the optimization of πT. That is we set = 1/as Σ}= exp( (x,() (x.a))/n). Using these samples we can then optimize for 7) in a way analogous to what is described in Abdol maleki et al. (2018). Specifically, we find that the objective for finding is
Tppior s) exp( (xpp(*"(s..a//))) (s,a)/,))da and where 17 is a temperature that depends on the desired
g(n) EErDD T7E + 1) Σ
log E. Eawnipi
exp (Qe (a.a/))))))
(8)
PLET
which can approximate based on a batch B of trajectories sampled from Dj (sampling M actions from Tprior for each state therein) and corresponding action samples where we used the samples {ai...., aMs } ~ Aprior (a|s) as
g(n)
T/E 11 Σ TEB
log
Σ exp (Q/)(e))))))
M
BLET
which readily be differentiated wrt. 7)- We then use Adam (with standard settings and learning rate 2e - 4) to take a gradient step in direction of V Vng(()) (we want to maximize (11)) for each batch. We start our optimization with 77 = 3 ensure stable optimization (i.e. to avoid large changes in the policy parameters in the beginning of optimization). Further, after each gradient step, we project to the positive numbers i.c. we set 77 max(1), 0.001). as 17 is required to be positive. We find that this procedure is capable of fulfilling the desired KL constraints well.
The same batch, and action samples, are then also used to take an optimization step for the policy parameters 6. In particular we find the parametric policy by minimizing the divergence K LL(() _ which equivalent
maximizing the weighted log likelihood of sampled actions (Equation 5). We only take M = 20 samples here, which is a relatively crude representation of the behavior model at state s; therefore, to prevent the policy from converging too quickly it can be useful to employ additional trust region constraint in this step that ensures slow convergence. We do this by adjusting the maximum likelihood objective Equation 5 to contain an additional KL regularization towards the previous policy, yielding the following optimization problem using samples (qI....,.a)) aas} ~ prioo (a)):
M
ΣΣΣ (xp(Q)(()0)))/)) log (o(a)(s)) + a<Cinus ([
max min 8 TEB 1=1
(10)
for > O. which is a Langrangian relaxation to the maximum likelihood problem under the additional constraint that ( || < <usst where a is the Langrange multiplier. This objective can be differentiated wrt. both 0 and and we simply take alternating gradient descent steps (one per batch for both f and a)) using Adam (Kingma & Ba 2015) (starting with a random Og and a = 1) and projecting back to the positive regime if it becomes negative; i.e. set max (a, 0.001).
B.2 SVG
To optimize the policy parameters 0 via the stochastic value gradient (Heess et al. 2015 (under a KL constraints) we can directly calculate the derivative of the Langrangian relaxation from Equation 6 In particular, again assuming that we have sampled a batch B, the gradient wrt. 0 can be obtained via the reparameterization trick (Kingma & Welling 2014 Rezende al. (014).. For this we first require that a sample from our policy Ta(a[s) can be obtained via deterministic function applied to a standard noise source. We first specify the policy class used in the paper to be that of Gaussian policies, parameterized Te(als) Ai(a (uo (s), (s)),, where ((aap, y,, Σ) denotes the pdf of standard Normal distribution and where we assume [4g is directly given as one output of the network whereas we parameterize the diagonal standard deviation as do(s) (s) so ((plus(h(s))) with (s)
13

---

Published as a conference paper at ICLR 2020
being output by the neural network. We can then obtain samples via the deterministic transformation f(s, 0) = (s) + (s)5, where Ç ~ (O, I) (I being the identity matrix). Using this definition can obtain the following expression for the value gradient:
5g Vo
IE TrDDp
B Ba-
(QT (s, a)] + n/( <- - KL[ Trioe (1s)]]
Σ
Σ
Va
( (s, + 17(5 (L|Ta(-|s)] prior
TEB
M
ΣΣΣ TEB
Σ
+ Ven(E - KL[T Trior (p)o(()))))
M
1=
(11)
where we use the Gaussian samples (EI....,Eas! ~ NNO, The gradient for the Langrangian multiplier is given as
6n
V 9
IE
Earro (1)) + n
- ( (-)))
(11)))
D.
(12)
Σ TEB EEET
V punl - KI. [1]] (-)))(()))))
where we dropped terms independent of 1) in the second line. Following dg to maximize the objective, and conversely moving in the opposite direction of 6, to minimize the objective wrt. 7) can then be performed by taking alternating gradient steps. We perform one step per batch for both 77 and 0 via Adam; starting from an initially random 0 and 1) I. As in the MPO procedure we ensure that 1 is positive by projecting it to the positive regime after each gradient step.
C DETAILS ON THE EXPERIMENTAL SETUP
C.1 HYPERPARAMETERS AND NETWORK ARCHITECTURE
Hyperparameters for MPO, SVG, and BCQ single-task experiments are shown in tables 1 and respectively. For multitask experiments, we modify the parameters shown in
C.2 DETAILS ON BCQ AND BEAR
To provide strong off- off-cylicy learning baselines we re-implemented BCQ (Fujimoto et al. 2018) and BEAR (Kumar et al. 2019) in the same framework that we used to implement our own al algorithm. As mentioned in the main paper we used the same network architecture for II algorithms. Algorithm specific hyperparameters where tuned via a coarse grid search on the control suite tasks; while following the advice from the original papers on good parameter ranges. To avoid bias our comparisons we did not utilize ensembles of Q-functions for any of the methods (e.g. we removed them from BEAR), we note that ensembling did not seem to have a major impact on performance (see appendix in (Kumar et al. 2019)). Parameters for all methods where optimized with Adam. Furthermore, to apply BEAR and BCQ the multi-task setting we employed the same conditioning of the policy and Q-function on one-hot task vector (which is used to select among multiple network "heads", yielding per task parameters, see description below).
For BCQ we used range of [0.25, 0.25] for the perturbative actions generated by the DDPG trained network, and chose a latent dimensionality of 64 for the VAE, see 3 the full hyperparameters.
For BEAR we used KL constraint rather than the maximum mean discrepancy. This ensures comparability with our method and we did not see any issues with instability when using a KL. To ensure good satisfaction of constraints
optimization for the Langragian multiplier required in BEAR that was also used for our method see description of SVG above. The hyperparameters for the BEAR training
used the exact
the control suite are given in Table 4
C.3
DETAILS ON THE ROBOT EXPERIMENT SETUP
The task setup for both the simulated and real robot experiments described in the following. A detailed description of the robot setup will be given in accompanying paper. We nonetheless give
14

---

Published as a conference paper at ICLR 2020
Hyperparameters
MPO
Policy net Prior net Number of actions sampled per state Q function net
256-256
256-256
20
256-256-256
0.1
5 x 10 10-3
EΣ Discount factor (?)
1 x 10-5
0.99
Adam learning rate
2 x 10-4
Replay buffer size
2 x 106
Target network update period
200
Batch size
512
Activation function
clu Yes
Layer norm on first layer
Tanh on Gaussian mean
No
Min variance
0.01
Max variance
unbounded
Table 1: Hyperparameters for MPO
Hyperparameters
SVG
Policy net Prior net
256-256
256-256
Q function net
256-256-256
é
0.2
Discount factor (7)
0.99
Adam learning rate
2 x 10
Replay buffer size
x 106
Target network update period
200
Batch size
512
Activation function
elu
Layer norm on first layer
Yes
Tanh Gaussian mean
No
Min variance
0.01
Max variance
unbounded
Table 2: Hyperparameters for SVG
a description here for completeness. We make no claim to have contributed these tasks specifically for this paper and merely them evaluation test-bed.
As the robot we utilize a Sawyer robotic arm mounted on a table and equipped with a Robotiq 2F-85 parallel gripper. A basket is positioned in front of the robot which contains three cubes (the proportions of cubes and basket sizes are consistent between simulation and reality). Three cameras on the basket track the cube using augmented reality tags. To model the tasks an MDP we provide both proprioceptive information from the robot sensors (joint positions, velocities and torques) and the tracked cube position, velocity (both in dimensions) and orientation to the policy and Q-function. Overall the observations provided to the robot are: Proprioception: Joint positions (7D, double), velocities (7D, double), torques (7D, double); wrist pose (7D, double), velocity (6D, double), force (6D, double); gripper finger angles (ID, int), velocity (ID, int), grasp flag (ID, binary) and the object features: Object pose (7D, double) averaged over all cameras observing the object, Relative pose between object and gripper (7D, double). In simulation the true object position and velocities / orientation are used instead of running a tracking algorithm.
We control both the robot and the gripper by commanding velocities in the 4 d-dimensional Cartesian space of the robot's endeffector (specifing three translational velocities plus the velocity for the wrists rotation) while the gripper control is 1-dimensional. The action limits [-0.07, 0.07] m/s
15

---

Published as a conference paper at ICLR 2020
Hyperparameters
BCQ
Encoder net
256-256
Latent size Decoder net Perturbation net
64
256-256
256-256
function net
256-256-256
Perturbation Scale Factor
0.25
Discount factor (7)
0.99
Adam learning rate
2 x 10-4
Replay buffer size
2 x 10
Batch size
512
Table 3: Hyperparameters for BCQ
Hyperparameters
BEAR
Policy net Prior net function net é for KL constraint action samples for BEAR-QL
256-256
256-256
256-256-256
0.2
20
Discount factor (?)
0.99
Adam learning rate
10-4
Replay buffer size
2 106
Target network update period
200
Batch size
512
Activation function
elu
Layer
first layer
Yes
Tanh on Gaussian mean
No
Min variance
0.01
Max variance
unbounded
Table 4: Hyperparameters for BEAR
for Cartesian actions. [-1, rad/s for wrist rotation and [-255, 255] for finger velocity (for units see gripper specifications). The control rate at which the actions are executed is 20 Hz. Episodes are terminated if the wrist force exceeds 20 N on any axis, encouraging gentle interaction of the robot with its environment.
For our experiment we use different task learn. The first tasks are auxiliary tasks for better exploration that help to learn the final task (Stack and Leave), i.e, stacking the green cube on top of the yellow cube. The rewards functions for all tasks are given as:
Reach: (oo(ddpossedttaaee pos presnrenn: ), 0.02, 0.15):
This function minimizes the distance d of the gripper the green cube.
2. Grasp:
Activate grasp sensor of gripper.
3. Lift: if height of green block is less than if is above 10 centimeters and (height
3cm)/(10cm 3cm) otherwise.
4. Place Wide: stol dd (d(pps poS + [0, 0, 0.05)), 0.01, 0.20) Move green cube to a position 5cm above the yellow cube.
5. Place Narrow: sdol(d(pos green pOSyellow + [0, 0, 0.05)). 0.00, 0.01): Like Place Wide but with more precision.
6. Stack: btol (pos green pos, poS 0.03) * btol (d_ (pos green pOS + 0.05, 0.01) * Grasp)
A binary reward for stacking green cube the yellow and deactivating the grasp
16

---

Published as a conference paper at ICLR 2020
Hyperparameters
Multitask
Policy net
200 > 300
Q function net
400 - 400
Encoder net (BCQ)
256 -> 256-256
Decoder net (BCQ)
256-256
Perturbation net (BCQ)
256 -> 100
Replay buffer size
4 x 10
Table 5: Network parameters for multitask experiments. "->" indicates the network branching into separate "heads" for each task, which are selected based on the one-hot task vector. This architecture similar the design presented in (Riedmiller al. 2018). Note that transitions are duplicated in replay for each task (so that transitions are sampled independently for each task).
7. STACK_ANDE LAEVVEG Y): stol(dz (poS endeffffttsttoo green) + 0.10, 0.03, 0.10) * Stack Like STACK(G, Y), but gets max reward if it moves the arm 10cm above the green cube.
Where d(a, b) denotes the euclidean distance between and b. We also define two tolerance functions with outputs scaled between and 1, i.e,
[1 1 tanh? aatan((vO.B) (o|)
iff |u| < else,
stol(w,c,r)
(13)
iff |o| <e
btol(v, c)
(14)
0 else.
D ADDITIONAL EXPERIMENTAN RESULTS
We present additional plots that show some aspects the developed algorithm more detail.
D.1 EXPANDED PLOTS FOR CONTROL SUITE
We provide expanded plots for MPO on the control suite in Figure These show in detail that the learned advantage weighted behavior model (ABM, in red in the left column) is far superior to the standard behavior model prior, leading less constrained RL policies.
D.2 EXPANDED PLOTS FOR ROBOT SIMULATION
Figure shows full results for the simulated robot stacking task, including all 7 intentions well as performance of the prior policies themselves during learning. The task set is structured: earlier tasks like reaching and lifting are necessary to perform most other tasks, so the simple behavioral model performs well on these. For the more difficult stacking tasks, however, the presence of conflicting data means the simple behavioral model doesn't achieve high reward, though still significantly improves performance of the regularized policy.
D.3 PERFORMANCE TABLES FOR CONTROL SUITE AND ROBOT SIMULATION
Table 6 shows final performance for all methods on control suite tasks. Table 7 shows final per- formance on simulated robotics tasks to make comparison between algorithms easier. The episode returns are averaged over the final 10% of episodes. ABM provides a performance boost over BM alone, particularly for difficult tasks such as block stacking and quadruped. The RL policy further improves performance on the difficult tasks. As noted in the main paper, one additional option, to further simplify the algorithm is to omit the policy improvement step, setting π++1 = Tprior (i.e. considering the case where 0) and, conversely learning the Q-values of the prior. In additional experiments we have found that this procedure roughly recovers the performance of the ABM prior when trained together with MPO (ABM+MPO); i.e. this is option for simpler algorithm to
17

---

Published as a conference paper at ICLR 2020
nomain_nams-cheee task name-run, BValustion-BC Palicy
domain name-cheetar task name-run, evaluation-Ri Palicy
800
800
600
roward
200
200
BM
MPO
MPD
domein_name-theg name=hopper, Lask namemhop. evaluation-BC Policy
domain_nnamentt nameehhgper, task namemhop, evaluation Policy
600
600
500
500
rewnrd
400
400
300
300
200
200
100
100
domain_names punduuuaam-uuddedd.....oooo task name-run, BV Palicy
domain_name naw.n-nmmmeeeeee quadruped, puddeguaduuedd task, name-run, evaluation-RL Policy
800
800
600
600
roward
400
400
200
200
domain_ nameewallerr, task namemrun, BB Palicy
domainn namemwalkee ket, task namemrun. evaluation Policy
Commain_namemmaalle
800
800
600
600
reword
400
400
200
200
Learning Step
Leaminng Step
Figure 7: Detailed plots for the control suite showing the performance of the respective learned prior (left column) and RL policy (right column) for all tasks (one per row).
18

---

Published as a conference paper at ICLR 2020
175
380
12
Agoltin
109
10
32
TM0--72 w61.833041 7a12
22
110h-2 3 200 www.saa-.......
TME--AL www.tor---...
Figure 8: All intentions for blocks stacking in simulation. Performance of executing the prior on the left, performance of the algorithm learned using this prior on the right.
19

---

Published as a conference paper at ICLR 2020
Tealrrecks wikk evid helly
Testo aed Mal.lattoe------
72
12
22
29
22
24
29
00
02
32
34
Todkesock and
22
5.
10
22
Tall- A2l 240h wator
Teaoo Aell a-g.rtooo Pullc
Figure 9: Learning the value function from the ABM prior policy block stacking in simulation. When the value function learned from the prior, the ABM prior no longer depends on the RI. policy, and can be learned its own. We can still perform MPO policy improvement using this value function and regularized to this new prior, shown on the right. Using a value function learned from the prior achieves similar performance, and may in fact perform slightly better more difficult tasks.
20

---

Published as a conference paper at ICLR 2020
Intention Wigorithm
ABM e MPO
peior
BM MPO
price
MPO
ABM + SVG
HM SVG
SVG
BCQ
BEAK
ABM (c 01
cheetah
907.4
558.7
891.8
466.3
496.2
920.7
924.1
726.1
914.9
843
S54.3
hopper
539.3
454.8
453.8
237.2
599.2
651.6
634.7
544 1
341.2
425
451.8
quadduped
786.6
733.4
710.8
398.8
463.6
$56.5
818.7
472.7
621.4
674
733.6
869.5
Table 6: Episode returns on control suite tasks when learning from 10k episodes for comparison
between algorithms.
Intention Allgrithhm
ABM MPO
prior
BM MPO
prior
MPO
BCQ
BEAR
Reach
182.3
180.8
178.3
172.9
178.3
172.6
176.9
Grasp
123.7
111.7
114.6
89.1
86.0
104.7
116.3
Lift
147.1
144.9
144.2
122.5
121.4
126.4
137.9
Place Wide
137.9
122.3
124.4
67.2
98.0
68.0
93.2
Place Narrow
126.3
106.1
111.3
46.7
63.3
58.3
64.1
Stack
109.9
78.8
86.9
20.1
17.5
39.3
67.8
Full Task
77.9
47.3
53.5
3.5
0.2
5.2
34.6
Table 7: Returns on each simulated robotics task - for comparison between algorithms.
implement at the cost of some performance loss (especially the most complicated domains). We included this setting as ABM (c 0, |r| = 2) the table noting that this case it is vital to choose short trajectory snippets in order to allow ppier pick the best action in each state.
21


Reviews:
[{"review_id": "S1gZXAlnKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel off-policy reinforcement learning algorithm based on a learned prior. Although it is just an extension of existing works, it does outperform the-state-of-the-art methods and can be used in real-world robots.The technique details are well introduced and analysed. The experimental results and comparison are sufficient. \n\nHowever, there are also some details require further explanation. For example, In Equation (10), the Langrange multiplier should be constrained to non-negative. But the authors have not taken it into consideration in the optimization process.\nBesides, there are also some typesetting problems in the paper, such as in Fig. 7.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HygcOQSTKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper present a novel approach to reinforcement learn from batched data that come from very different sources. To achieve this, they propose to learn a prior model and then constrain the RL policy to a trust-region that does not deviate from the domain where the current policy is close to. \n\nThe paper is clearly written.\n\nStrength: the paper is very novel and tries to solve a very challenging problem. The success of this approach shows the potential that collecting large-scale data without distinguishment is possible. With that, the data efficiency will be much improved! The experiment itself shows superior performance than other methods. This method also works for real robots.\n\nWeakness: There is no algorithm box so it can be hard to fully reproduce. The paper mentions stableness for the method however the analysis on this aspect is limited.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1eFTysoKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for “offline RL” (a.k.a “batch RL”), i.e. reinforcement learning from a given static dataset, with no option to perform on-policy data collection. Contrary to prior work (Fujimoto et al, Kumar et al, Agarwal et al) in this area which focuses on making Q-learning robust in the offline RL setting, the authors in this paper instead propose making the policy update step robust to the batch RL setting. They achieve this by constraining the policy that is being learned to stay close to a learned “prior policy” (using KL divergence as a distance metric). The authors provide two different ways of learning this prior policy  - the first one is a simple behavior model policy (which involves fitting a model using maximum likelihood on the entire dataset - effectively performing behavior cloning on the entire dataset), while the second is a more sophisticated “advantage behavior model”, which fits a model only on the “good” data (i.e. environment transitions with a positive advantage, where the advantage is estimated using the current policy). The authors extend the MPO and SVG algorithms to work with this constrained update step. \n\nThe authors provide experiments on the standard DM control suite tasks and some robotic tasks (both simulated and real world). For all experiments, the authors collect data by first running MPO in the standard RL setting (i.e. online data collection), and use the replay buffer from these successful RL runs to relearn a policy. The proposed method performs comparable to other methods in some settings (like all the experiments in the 2K episodes setting in Figure 2), and performs slightly better in some other settings (like Hopper and Quadruped in the 10K episodes setting). The gains are more significant in the non-standard robotics domains (Stack, Stack/Leave). The real world robotics results are not compared to any baseline (perhaps due to evaluation being expensive / time-consuming). However, I think the experimental results, especially Figures 7 and 8, point to a potential flaw in the proposed algorithm. In Figure 7 in the appendix, the learned behavior model (ABM) gets about the same performance as the proposed method (ABM+MPO) in pretty much all the four tasks. In one of the tasks (hopper), ABM+MPO does slightly better than ABM alone, but in this case plane MPO is better than ABM + MPO, indicating that this might just be a task on which MPO does well (for some unexplained reason). Similarly, in the robotics results in Figure 8, ABM alone does comparable to ABM+MPO in five of the seven tasks. \n\nThese experimental results imply to me that most of the performance actually comes from the learned prior (ABM), and whether performing MPO on top of it leads to any improvement or not is hard to predict^[1]. The ABM model itself is somewhat novel, and similar ideas under review at ICLR this year have shown that this alone is a useful algorithm for offline RL, see https://openreview.net/forum?id=H1gdF34FvS and https://openreview.net/forum?id=BJlnmgrFvS. However, the ABM model is not emphasized much in the paper. It is also unclear to my as to why the ABM/BM results were placed in the Appendix for Figure 3, while similar results for experiments in Figure 1 were provided using dotted lines.  \n\nFor the paper to be accepted for publication, I think it needs to make a stronger argument (experimentally, at least) about the proposed algorithm being superior to ABM. If this is not really the case, then I think it would require a substantial rewrite to emphasize that ABM is where most of the model performance comes from (as seen in concurrent work listed above). \n\n[1]: A slight caveat here: in the proposed Algorithm 1, the ABM model is learned online along with the policy, but I believe it could be learned without the policy - for example, by using MC returns as in https://openreview.net/forum?id=H1gdF34FvS. \n----------------------\n\nEdit: The author response has convinced me to bump my rating to a weak accept. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a novel stable RL algorithm for the batch off-policy setting, through the use of a learned prior.  Initially, reviewers had significant concerns about (1) reproducibility, (2) technical details, including the non-negativity of the lagrange multiplier, (3) a lack of separation between performance contributions of ABM and MPO, (4) baseline comparisons.  The authors satisfactorily clarified points (1)-(3) and the simulated baseline comparisons for (4) seem reasonable in light of how long the real robot experiments took, as reported by the authors.  Futhermore, the reviewers all agree on the contribution of the core ideas.  Thus, I recommend this paper for acceptance.
Decision: Accept (Poster)