Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
Authors: Tatsuya Matsushima
Hiroki Furuta
Yutaka Matsuo
Ofir Nachum
Shixiang Gu
Venue: ICLR 2021 Poster (2021)
Keywords: Reinforcement Learning
deployment-efficiency
offline RL
Model-based RL

Abstract: Most reinforcement learning (RL) algorithms assume online access to the environment, in which one may readily interleave updates to the policy with experience collection using that policy. However, in many real-world applications such as health, education, dialogue agents, and robotics, the cost or potential risk of deploying a new data-collection policy is high, to the point that it can become prohibitive to update the data-collection policy more than a few times during learning. With this view, we propose a novel concept of deployment efficiency, measuring the number of distinct data-collection policies that are used during policy learning. We observe that naïvely applying existing model-free offline RL algorithms recursively does not lead to a practical deployment-efficient and sample-efficient algorithm. We propose a novel model-based algorithm, Behavior-Regularized Model-ENsemble (BREMEN), that not only performs better than or comparably as the state-of-the-art dynamic-programming-based and concurrently-proposed model-based offline approaches on existing benchmarks, but can also effectively optimize a policy offline using 10-20 times fewer data than prior works. Furthermore, the recursive application of BREMEN achieves impressive deployment efficiency while maintaining the same or better sample efficiency, learning successful policies from scratch on simulated robotic environments with only 5-10 deployments, compared to typical values of hundreds to millions in standard RL baselines.

Published as a conference paper at ICLR 2021
DEPLOYMENT-EFRICIEN REINFORCEMENT LEARN- ING VIA MODEL-BASED OFFLINE OPTIMIZATION
Tatsuya Matsushima"
Hiroki Furuta"
Yutaka Matsuo
The University of Tokyo
(matsushima, furuta, mat.so) @weblaw.b................ w.t.ob...tutoooaa.....
Ofir Nachum Google Research [ofirnachum,
Shixiang Shane Gu
shanegu) @google.
ABSTRACT
Most reinforcement learning (RL) algorithms assume online access to the environ- ment, in which one may readily interleave updates to the policy with experience collection using that policy. However, in many real-world applications such as health, education, dialogue agents, and robotics, the cost or potential risk of deploy- ing new data-collection policy is high, to the point that it can become prohibitive to update the data-collection policy more than few times during learning. With this view, we propose a novel concept of deployment efficiency, measuring the number distinct data-collection policies that are used during policy learning. We observe that naïvely applying existing model-free offline RL algorithms re- cursively does not lead to a practical deployment--rficceett and sample-efficient algorithm. We propose novel model-based algorithm, Behavior Regullrized Mosel-ENsemble (BREMEN), that not only performs better than or comparably the state-of-the-ar waeraami--poggaamiiie based and concurrently-pron proopooed model-based offline approaches on existing benchmarks, but can also effectively op- timize policy offline using 10-20 times fewer data than prior works. Furthermore, the recursive application of BREMEN achieves impressive deployment efficiency while maintaining the same or better sample efficiency, learning successful poli- cies from scratch on simulated robotic environments with only 5-10 deployments, compared to typical values of hundreds millions in standard RL baselines.
INTRODUCTION
Reinforcement learning (RL) algorithms have recently demonstrated impressive success in learning behaviors for a variety of sequential decision-makin tasks (Barth-Maron Marton al., 2018: 8; Hessel et al., 2018; Nachum et al., 2019). Virtually all of these demonstrations have relied on online access to the environment, with the RL algorithms often interleaving each update to the policy with additional experience collection of that policy acting in the environment. However, in many real-world applications of RL, such as health (Murphy et al., 2001), education (Mandel et al., 2014), dialog agents (Jaques et al., 2019), and robotics (Gu et al.. 2017a; Kalashnikov et al., 2018), the deployment of a new data-collection policy may be associated with number of costs and risks. can learn tasks with small number of data collection policies, we substantially reduce them.
Based on this idea, we propose a novel measure of RL algorithm performance, namely deploymens efficiency, which counts the number of changes in the data-collection policy during learning, illus- trated in Figure 1. This concept may be seen in contrast to sample efficiency or data efficiency (Precup al., 2001; Degris al., 2012; Gu al., 201 7b; Haarnoja et al., 2018; Lillicrap al., 2016; Nachum et al., 2018), which measures the amount of environment interactions incurred during training, without regard to how many distinct policies were deployed perform those interactions. Even when the
"Equal contribution.
Codes and pre-trained models are available setps::/ggithe wtw.:hha..cmmm.ttuu.uuuo T BREMEN.

---

Published as a conference paper at ICLR 2021
Transitinss
Data Collection
Emt
Add Datasot
Deploy
Deploy
Deploy
Policy Update
Dutratt
BREMEN
TRPO, ME- TAPQ.
DOPG,
Offline RL
Ours
Online
# of Deployments (i)
5-10
100s-1000s
100Ks- 1Ms
High
Deployment Efficiency
Low
Figure I: Deployment efficiency is defined as the number of changes in the data-collection policy (1), which is vital for managing costs and risks of new policy deployment. Online RL algorithms lypically require many iterations of policy deployment and data collection, which leads extremely low deployment efficiency. In contrast, most pure offline algorithms consider updating policy from fixed datasct without additional deployment and often fail to learn from a randomly initialized data-collection policy. Interestingly, most state-of-the-art off-policy algorithms are still evaluated in heavily online settings. For example, SAC (Haarnoja al., 2018) 88 collects sample per policy update, amounting to 100,000 to million deployments for learning standard benchmark domains.
data efficiency is high, the deployment efficiency could be low, since many on-policy and off-policy algorithms alternate data collection with each policy update (Schulman et al., 2015; Lillicrap et al., 2016; Gu et al., 2016; Haarnoja et al., 2018). Such dependence is best illustrated in the recent works in offline RL (Fujimoto et al., 2019; Jaques et al., 2019; Kumar et al., 2019; Levine et al., 2020; Wu et al., 2019), where baseline off- off-policy algorithms exhibited poor performance when trained on a static dataset. These offline RL works, however, limit their study to single deployment, which is enough for achieving high performance with data collected from sub-optimal behavior policy, but often not from a random policy. In contrast to those prior works, we aim to learn successful policies from scratch in a manner that is both sample and deployment--efficcenn
high-freeqeney" policy deployments
Many existing model-free offline RL algorithms (Levine et al., 2020) are tuned and evaluated on massive datasets (e.g., one million transitions). In order to develop an algorithm that is both sample and deployment-efficienn work effectively on much smaller dataset sizes. We believe model-based RL. is better suited to this setting due to its higher demonstrated sample efficiency than model-free RL (Kurutach et al., 2018; Nagabandi al., 2018). Although the combination of model-based RL and offline or limited- deployment settings seems straight-forward, we find this naive approach leads to poor performance. This problem can be attributed to extrapolation errors (Fujimoto et al., 2019) similar to those observed in model-free methods. Specifically, the learned policy may choose sequences of actions which lead it to regions of the state space where the dynamics model cannot predict properly. due to poor coverage of the dataset. This can lead the policy exploit approximation errors of the dynamics model and be disastrous for learning. In model-free settings, similar data distribution shift problems are typically remedied by regularizing policy updates explicitly with a divergence from the observed data distribution (Jaques et al.. 2019; Kumar et al., 2019; Wu et al., 2019), which, however, can overly limit policies' expressivity (Sohn et al., 2020).
each iteration of the algorithm between successive deployments has to
In order to better approach these problems arising in limited deployment settings. we propose (BREMEN), which learns an ensemble of dynamies models in conjunction with policy using imaginary rollouts while implicitly regularizing the learned policy via appropriate parameter initialization and conservative trust-region learning updates. We evaluate BREMEN on standard offline RL benchmarks of nigh-dimensiona continuous control tasks, where only a single static dataset is used. In this fixed-batch setting, our experiments show that BREMEN can not only achieve performance competitive with state-of-the-art when using standard dataset sizes but also learn with 10-20 times smaller datasets, which previous methods are unable to attain. Enabled by such stable and sample-efficient offline learning, show that BREMEN can learn successful policies with only 5-10 deployments in the online setting, significantly outperforming existing off polico and offline RI. algorithms in deployment efficiency while keeping sample efficiency.
Behavior- Reauiirr-eeularizzed
2

---

Published as a conference paper at ICLR 2021
2 PRELIMINARIES
We consider a Markov Decision Process (MDP) setting, characterized by the tuple M = (S, p. where S is the state space, A the action space, p(s'|s, |s, a) is the transition prob- ability distribution or dynamics, r(s) is the reward function and Y (0, 1) is the discount factor. A policy π is function that determines the agent behavior, mapping from states to probability distribu- tions over actions. The goal is to obtain the optimal policy T², which maximizes the expectation of discounted sum of rewards. The transition probability p(s'|s, a) is usually unknown, and estimated with parameterized dynamics model fo (e.g. neural network) in model-based RL. For simplicity, we assume that the reward function r(s) is known, and the reward can be computed for any arbitrary state, but we may extend to the unknown setting and predict it using parameterized function.
On-policy vs Off-policy, Online vs Offline At a high-level, most RL algorithms alternate many times between collecting batch of transitions (deployments) and optimizing the policy (learning). If the algorithms discard data after each policy update, they are on-policy (Schulman et al., 2015; 2017), while if they accumulate data in a buffer D. i.e. experience replay (Lin, 1992). they are off-policy (Mnih et al.. 2015; Lillicrap et al., 2016; Gu et al., 2016; 2017b; Haarnoja et al., 2018; Fujimoto et al., 2019; Fakoor et al., 2019) because not all the data in buffer comes from the current policy. However, we consider all these algorithms to be online RL algorithms, since they involve many deployments during learning, ranging from hundreds to millions. On the other hand, in pure offline RL, one does not assume direct interaction and learns policy from only fixed dataset, which effectively corresponds to single deployment allowed for learning. Classically, interpolating these two extremes improve the policy through repetitions of collecting a large batch of transitions D = {(s, a, r'rr)} and performing many or full policy updates. While these semi-batch RI. also realize good deployment efficiency, they have not been extensively studied with neural network function approximators or in off-policy settings with experience replay for scalable sample-efficient learning. In our work, we aim to have both high deployment efficiency and sample efficiency by developing an algorithm that can solve the tasks with minimal policy deployments well as transition samples.
semi batch RL algorithms (Lange et al., 2012; Singh et al., 1995). which
3 DEPLOYMENT EFFICIENCY
Deploying a new policy for data collection can be associated with a number of costs and risks for many real-world applications like medicine, dialogue systems, or robotic control (Murphy et al., 2001; Mandel et al., 2014; Gu et al., 2017a; Kalashnikov et al., 2018; Nachum et al., 2019; Jaques et al., 2019). While there are abundant works on safety for RL (Chow et al., 2015; Eysenbach et al., 2018; Chow et al., 2018: 8; Ray et al., 2019; Chow et al., 2019), they often do not provide guarantees in practice when combined with neural networks and stochastic optimization. It is therefore necessary to verify each policy before deployment (e.g. measuring the variance of rewards or checking out-of-bounds actions). Due to such costs associated with each deployment, is desirable to minimize the number of distinct deployments needed during the learning process. Even ignoring safety considerations, frequent updates to a deployed policy can exacerbate communication bottlenecks in large-scale distributed RL systems, which are becoming more prevalent (Nair al., 2015; Espeholt al., 2018; 20. We additionally discuss on the importance of the deployment efficiency in real-world applications. See Appendix C.
In order to focus research on these practical bottlenecks, we propose a novel measure of RL algorithms, namely, deployment efficiency, which counts how many times the data-collection policy has been changed during improvement from random policy to solve the task. For example, if an RL algorithm operates by using its learned policy to collect transitions from the environment times, each time collecting batch of B new transitions, then the number of deployments is I, while the total number of samples collected I x B. The lower 7 is, the more deployment-efficee contrast, sample efficiency looks at I x B. Online RL algorithms, whether they are on-policy or off-policy, typically update the policy and acquire new transitions by deploying the newly updated policy at every iteration. This corresponds to performing hundreds to millions of deployments during learning on standard benchmarks (Haarnoja et al., 2018), which is severely deployment inefficient. On the other hand, offline RL literature only studies the case of I deployment. A deployment-efficien algorithm would stand in the middle of these two extremes and ideally learn a successful policy from scratch while deploying only few distinct policies, illustrated in Figure 1.
the algorithm is; in
3

---

Published as a conference paper at ICLR 2021
Recent deep RL literature seldom emphasizes deployment efficiency, with few exceptions in specific applications (Kalashnikov et al., 2018) where such a learning procedure is necessary. Deployment- inefficient algorithms will fail in scenarios where the deployment of each new policy is exorbitantly expensive, such as safety-critical robotics or user-facing products. Although current state-of-the-ar algorithms continuous control have substantially improved sample data efficiency, they have not optimized for deployment efficiency. For example, SAC (Haarnoja et al., 2018), an efficient model-free off-policy algorithm, performs half' million one million policy deployments during learning on MuJoCo (Todorov et al., 2012) benchmarks. ME-TRPO (Kurutach et al., 2018), podeel-basede algorithm, performs a much lower 100-300 policy deployments, although this still relatively high for practical settings." In our work, we demonstrate successful learning on standard benchmark environments with only 5-10 deployments.
4
REGULARIZED MODEL-ENSEMBLE
To achieve favorable combination of both high deployment and sample efficiency, we propose Sechavor-Regglariie Model-ENsemblle (BREMEN). BREMEN incorporates Dyna-style (Sutton, 1991; Kurutach et al., 2018) model-based RL, learning an ensemble of dynamics models in conjunc- tion with policy using imaginary rollouts and behavior regularization via conservative trust-region updates.
4.1 IMAGINARY ROLLOUT FROM MODEL ENSEMBLE
As in recent Dyna-style model-based RL methods (Kurutach et al., 2018; Wang et al., 2019), BRE- MEN uses an ensemble of K deterministic dynamics models fo = fer...... fox to alleviate the problem of model bias. Each model fo, is parameterized by o; and trained by the following objective, which minimizes mean squared error between the prediction of next state for (s,, at) and true next state 1 over a dataset D:
min
Σ
S1+1 - fo. (st, an)
(I)
D
2
(sL,0,,841)EDD
During training of a policy Ty. imagined trajectories states and actions are generated sequentially, using a dynamics model fo: that is randomly selected at each time step:
ag ~ (T)) $e11 = for (8e, as) where i i~ (1....K)
(2)
4.2 POLICY UPDATE WITH BEHAVIOR REGULARIZATION
In order to manage the discrepancy between the true dynamics and the learned model caused by the distribution shift in batch settings, propose to use iterative policy updates via trust-region constraint, re-initialized with a behavior-cloned policy after every deployment. Specifically, after each deployment,
given an updated dataset of experience transitions D. With this dataset, we approximate the true behavior policy Ty through behavior cloning (BC), utilizing a neural network मे parameterized by 9, where we implicitly assume fixed variance, common practice in BC (Rajeswaran et al., 2017):
min Σ (EL,0))D
2 |0, - n (x))1
After obtaining the estimated behavior policy, we initialize the target policy my Gaussian policy with mean from ity and standard deviation of 1. This BC initialization in conjunction with gradient descent based optimization may be seen as implicitly biasing the optimized ny to be close to the data-collection policy (Nagarajan & Kolter, 2019), and thus works as a remedy for the distribution shift problem (Ross al., 2011). 1). To further bias the learned policy to be close the data-collection
We examined the number of deployments by checking their original implementations, while the frequency of data collection is tunable hapar-parameter.
4

---

Published as a conference paper at ICLR 2021
Algorithm 1 BREMEN for Deployment-Efficiee Efficient RL
Input:
Empty dataset Dan. D, Initial parameters Φ = (6i,-.. Ox h. B, Number of policy optimization T,
Number of deployments I.
1:
Randomly initialize the target policy No.
for deployment = 1,...
3: Collect B transitions in the true environment using my and add them to dataset
Dau <- Dau {st, de, 8(41), D <- {st, (t, Pt, 8t413.
4: Train K dynamics models fe using Daul via Equation 1.
5: Train estimated behavior policy #3 using D by behavior cloning via Equation 3.
6: Re-initialize target policy TTo Nor,al(
policy optimization & 1, , do
8: Generate imaginary rollout via Equation 2.
9: Optimize target policy лa satisfying Equation 4 with the rollout.
policy, we opt to use a KL-based trust-region optimization (Schulman et al., 2015). Therefore, the optimization of BREMEN becomes
TT(a|() A (s. To (a|s)
+++1 arg max
E a Soc
s.t.
E
[DLL (m)(-I)]]Ta (+1s))] s 6, ne. Normal(fa,,
where 4 (s)a) is the advantage of Te, computed using model-based rollouts in the learned dynamics model and 8 is the maximum step size.
The combination of BC for initialization and finite iterative trust-region updates serves an implicit KL regularization. This is in contrast to many previous offline RL algorithms that augment the value function with a penalty of explicit KL divergence (Siegel al., 2020; Wu et al.. 2019) or maximum mean discrepancy (Kumar et al., 2019). Empirically, we found that our regularization technique outperforms the explicit KI. penalty (Section 5.3). Furthermore, we provide mathematical intuition explaining how our methods works as an implicit regularization of distributional shift in Appendix A.
By recursively performing offline procedure, BREMEN can be used for deployment-efficient learning as shown in Algorithm 1, starting from randomly initialized policy, collecting experience data, and performing offline policy updates.
5 EXPERIMENTS
In order to realize a deployment-efficient RI algorithm, the batch policy optimizer has to be stable and sample-efficient. We first evaluate BREMEN in the offline setting, where the algorithm learns the policy from a static dataset. Standard benchmarks of MuJoCo physics simulator shown in (Wu et al., 2019) and more recent datasets (Fu et al., 2020) are used in the evaluation, and we compared the asymptotic performance of BREMEN with other offline RL methods including the concurrent model-based approaches. We then tested the sample-efficiency of offline algorithms using smaller datasets. We lastly extend the experiment deploymeet--fffcieen settings, where the algorithms learn their policies from scratch via a limited number of deployments and perform some ablations to see how components in BREMEN affect performance. See Appendix for further details.
5.1 EVALUATING OFFLINE RI. PERFORMANCES
Standard Benchmarks We evaluate BREMEN on standard offline RL benchmarks following and identical protocol as in Wu et al. (2019): We first train online SAC to certain cumulative reward threshold, 4,000 in HalfCheetah, 1,000 in Ant, Hopper, and Walker2d, and collect offline datasets. We evaluate agents with the offline dataset of one million (IM) transitions, which is standard for BCQ and BRAC. Table (top) shows that BREMEN can achieve performance competitive with state-of-the-ee model-free offline RL algorithms when using the standard dataset size of 1M. We also test BREMEN with more recent benchmarks of D4RL (Fu et al., 2020) and compared the performance with the existing model-free and model-base methods. See Appendix D for the results.
5

---

Published as a conference paper at ICLR 2021
1,000,000 (1M) transitions
Method
Ant
HalfCheetah
Hopper
Walker2d
Dataset
1191
4126
1128
1376
BC
1321=1141
42811:12
1341+:111
1421=:17
BCQ
2021#:31
5783#272
1130+127
2133#7533
BRAC
2072:12855
7192=115
1422:190
2239=1124
BRAC (max Q
2369:1234
73201991
19161:33
2409=1210
BREMEN (Ours)
3328+-275
8055+1103
2058+885
2346+-300
ME-TRPO (offline)
1258+550
18044-924
518499
211+154
100,000 (100K) transitions
Method
Ant
HalfCheetah
Hopper
Walker2d
Dataset
1191
4066
1128
1376
BC
1330+881
4266+t-11
1322+1109
1426+t-47
BCQ
136
3915+-411
1129+:238
1187+196
BRAC
-157:1:883
2505:12501
1310.170
2162:11109
BRAC (max Q)
-226.1.87
2332.12422
1422.1
2164
BREMEN (Ours)
1633-1127
6095.1370
2191
2132.1301
ME-TRPO (offline)
974
2=:34
377
10
50,000 (50K) transitions
Method
Ant
HalfCheetah
Hopper
Walker2d
Dataset
1191
4138
1128
1376
BC
1270:t65
4230::49
1249:f6
1420=194
BCQ
1329+
1319=626
1178+235
1841
BRAC
2878=224
-597+73
1277=102
976=1207
BRAC (max
-843:1-779
-5901-56
1276:1:225
903-1:137
BREMEN (Ours)
1347:1283
58231146
1632.1.96
2280:1667
ME-TRPO (offline)
938-1:22
-731-95
1524#1
176-+-333
Table 1: Comparison of BREMEN to the existing offline methods on static datasets, Each cell shows the average cumulative reward and their standard deviation, where the number of samples is IM, 100K, and 50K, respectively. The maximum steps per episode is 1,000. BRAC applics primal form of KL value penalty, and BRAC (max Q) means its variant of sampling multiple actions and taking the maximum according to the learned Q function.
Evaluating Sample- Efficiecy We then evaluate the sample-efficiency by making much smaller datasets of 50k and 100k transitions (5~100 % of Wu et al. (2019)). Surprisingly, Table (middle and bottom) shows that BREMEN can also learn with smaller datasets, where BCQ and BRAC are unable to exceed even BC baseline. This is novel evaluation protocol we proposed, and our BREMEN's superior performance here exactly what enables recursive BREMEN in the next section be an effective algorithm in deploymeent-oonstrained settings.
5.2 EVALUATING DEPLOYMENT EFFICIENCY IN ONLINE RL BENCHMARKS
We compare BREMEN to ME- TREOO SAC, BCQ, and BRAC applied to limited deployment settings. To adapt offline methods (BCQ. BRAC) to this setting, we simply apply them recursive fashion; at each deployment iteration, we collect a batch of data with the most recent policy and then run the offline update with this dataset. As for SAC, we simply change the replay buffer to update only at specific deployment intervals. For the sake of comparison, we align the number of deployments and the amount of data collection at each deployment (either 100k or 200k) for all methods."
Figure 2 shows the results with 200k (top) and 100k (bottom) batched transitions per deployment. Regardless the environments and the batch size per update, BREMEN achieves remarkable performance while cxisting online and offline RL methods struggle to make any progress. As point of comparison, we also include results for online SAC and ME-TRPO without deployment-limit but using the same number of transitions. We additionally compare BREMEN the model-based offline RL methods with uncertainty- -based penalties, See Appendix for further details.
Following the motivation of deployment efficiency, obtaining successful policy under data-collection constraint conditions in the real application, we extensively evaluate our algorithm on more realistic robotics environments in OpenAl Gym. The experimental procedure is the same as above, while
RRecursive BCQ and BRAC also do behavioral cloning- doning-hased hased policy initialization after cach deployment. We evaluate the trade-off off between sample and deployment efficiency in Appendix

---

Published as a conference paper at ICLR 2021
BREMEN
Explest KL
ME-TRPO
SAC
BCQ
BRAC
BRACImex G)
ME-TRPD(oninee
SAC|entine
HarfChoetah
Happer
Walce
Butch-890,000
symment Ant
Desi corment HalfChoetain
Desteyment Hosperr
Dapleyment
Watcer2d
100,0000000
200
L00
Desteymmnt
Desloyment
Figure 2: Evaluation of BREMEN with the existing methods (ME-TRPO SAC, BCQ, BRAC) under deployment constraints (to 5-10 deployments with batch sizes of 200k 100k). The average cumulative rewards and their standard deviations with random seeds are shown. Vertical dotted lines represent where each policy deployment and data collection happen. BREMEN is able to learn successful policies with only 5-10 deployments, while the state-of-the- art off-policy (SAC), model-based (ME-TRPO) and recursively applied offline RL algorithms (BCQ, BRAC) often struggle to make any progress. For completeness, show (E-TRPOOonline) and SAC(online) which are their original optimal learning curves without deployment constraints, plotted with respect to samples normalized by the batch size. While SAC(online) substantially outperforms BREMEN in sample efficiency, it uses I deployment per sample, leading to 100k- 500k deployments required for learning. Interestingly, BREMEN achieves even better performance than the original ME- ME-TRROOnnliee), suggesting the effectiveness of implicit behavior regularization. For SAC and ME- TRPO under deployment-consttranee evaluation. their batch size between policy deployments differs substantially from their standard settings, and therefore we performed extensive hyper-parameter search on the relevant parameters such as the number of policy updates between deployments, as discussed in Appendix F.2.1.
BREMEN
ME-TRPO
SALionlmw)
Datche 2a0tt=25,0000 800 rewards
TetchReach Dersee
Handi Dhennee
-25
-50
-75
-100
Deployment
Depleyment
(a) FetchReach
(b) HandReach
(c) Results
Figure 3: Robotics environments and the results under deployment constraints (10 deployments with batch sizes of 25k). The performances are averaged over seeds. BREMEN seems the only method that shows both stable improvement and solving tasks without large degradation or sub-optimal convergence.
we limit the batch size at each deployment as only 25k. Figure 3 presents the reaching tasks with Fetch robot and 20-DoF shadow hand (Plappert et al., 2018), and the experimental results in both environments. Only BREMEN shows stable improvement and high performance, while other offline and online algorithms fail to learn. These results suggest that model- based method a desirable approach for satisfying practical requirements in robotics, i.e. sample and deployment efficiency.
5.3 ABLATION: EVALUATING EFFECTIVENESS OF IMPLICIT KL CONTROL
In this section, we present an experiment to better understand the effect of BREMEN's implicit regularization. Figure 4 shows the KL divergence of learned policies from the last deployed policy, We compare BREMEN variants BREMEN that use an explicit KI. penalty on value instead of BC initialization (conservative KL trust-region updates are still used). We find that the explicit KL without behavior initialization variants learn policies that move farther away from the last deployed policy than behavior initialized policies. This suggests that the implicit behavior regularization employed by BREMEN is more effective as conservative policy learning protocol. In addition, to assess the effect of repeated behavior cloning initialization,
also evaluate variant of BREMEN without behavior cloning re- re-initialization (grey). This variant works in easier environments (Ant. Halfcheetah), but does not show remarkable progress in more challenging ones with termination (Hopper, Walker2d).
7

---

Published as a conference paper at ICLR 2021
BREMEN
DREMENIAIC BC reiitaalizzt))
ME-TRPOKa-0)
Explict Kidam 1e-2)
2-1e4
1e-6
Ant
HaltChectah
Hopper
Walker2d
Cum Cumulitive rewaard
4900
2003
2300
Desteyment
Deployment HellCheeleh
Doployment Happper
Destoymont Walkerdd
Ari
20000
3500s
20000
Trees
1::00
2009
30300
Dapteyment
Deployment
Doployment
Dastoymant
Figure 4: We examine average cumulative rewards (top) and corresponding KL divergence between the last deployed policy and the target policy (bottom) with batch size 200K in limited deployment settings. The behavior initialized policy remains close the last deployed policy during improvement without explicit value penalty Da (Tell (fts).)) The explicit penalty is controlled by coefficient a.
This result empirically supports the need for repeated behavior initialization after each deployment. The results of further experiments are shown in Appendix G.
RELATED WORK
Deployment Efficiency and Offline RL Although we are not aware of any previous works which explicitly proposed the concept of deployment efficiency, its necessity in many real-world applications has been generally known. One may consider previously proposed semi- cemi-batch RL algorithms (Ernst et al., 2005; Lange et al., 2012: Singh et al., 1994; Roux, 2016) theoretical analysis of switching cost under the tabular PAC-MDP settings (Bai et al., 2019; Guo & Brunskill, 2015) as approaching this issue. More recently, related but distinct problem known offline RL has gained popularity (Levine et al., 2020; Wu et al., 2019; Agarwal et al., 2019; Kumar et al., 2020). These works consider an extreme version of 1 deployment, and typically collect the statie batch with a partially trained policy rather than a random policy. While offline RL. has shown promising results for a variety of real-world applications, such as robotics (Mandlekar al., 2019), dialogue systems (Jaques et al., 2019), or medical treatments (Gottesman et al., 2018). these algorithms struggle when learning a policy from scratch or when the dataset small. Nevertheless, common themes of many offline RL algorithms regularizing the learned policy to the behavior policy (Fujimoto et al., 2019; Kumar et al., 2019; Siegel et al., 2020; Wu et al., 2019) and utilizing ensembles to handle uncertainty (Kumar al., 2019; Wu et al., 2019) served as inspirations for the proposed our algorithm. A major difference of BREMEN from prior works is that the target policy not explicitly forced to stick close to the estimated behavior policy through the policy update except for the initial iteration. Rather, BREMEN employs a more implicit regularization by initializing the learned policy with a behavior cloned policy and then applying conservative trust-region updates. Another major difference is the application of model-based approaches to fully offline settings, which has not been extensively studied in prior works (Levine et al., 2020), except the two concurrent works (Kidambi al., 2020; Yu et al., 2020) that study pessimistic or uncertainty penalized MDPs with guarantees closely related Liu et al. (2019). By contrast, our work shows that a simple technique can already enable model- based offline algorithms to significantly outperform the prior model-free methods, and is, to the best of our knowledge, the first define and extensively evaluate deployment efficiency with recursive experiments.
Model-Based RL There are many types of model-based RL algorithms (Sutton, 1991; Deisenroth & Rasmussen, 2011; Heess et al., 2015). A simple algorithmic choice is Dyna-style (Sutton, 1991). which uses a parameterized model to estimate the true MDP transition function, stochastically mapping states and actions to next states. The dynamics model can then serve simulator of the environment during policy updates. Dyna-style algorithms often suffer from the distributional shift, also known model bias, which leads RL agents to exploit regions where the data is insufficient. and significant performance degradation. A variety of remedies have been proposed to relieve the issue of model bias, such the of multiple dynamics models as ensemble (Chua et al., 201 8; Kurutach
8

---

Published as a conference paper at ICLR 2021
al., 2018; Janner et al., 2019), meta-learing (Clavera et al., 2018), anergy-baseds regularizer (Boney et al., 2019), game-theoretic framework (Rajeswaran et al., 2020), and explicit penalty for unknown states (Kidambi et al., 2020; Yu et al., 2020). Notably, we have employed subset of these remedies model ensembles and trust-region updates (Kurutach et al., 2018) for BREMEN. Compared to prior works, our work is notable for using BC initialization in conjunction with trust-region updates to alleviate the distribution shift of the learned policy from the dataset used to train the dynamics model.
7 CONCLUSION
In this work, we introduced deployment efficiency, novel measure for RL performance that counts the number of changes in the data-collection policy during learning. To enhance deployment efficiency, Sabavior-Regularzee
we proposed a novel model -baed offline algorithm, MEN), combining -eneebles with trust region updates from model-based RL literature (Kuru- tach et al., 2), and policy initialization with behavior cloning from offline RI. literature (Fujimoto al., 2019; Wu al., 2019). Crucially, BREMEN can improve policies offline sumple-effccienlly even when the batch size is 10-20 times smaller than prior works, allowing BREMEN to achieve impressive results limited deployment settings, obtaining successful policies from scratch in only 5-10 deployments. Not only can this help alleviate costs and risks in real-world applications, but can also reduce the amount of communication required during distributed learning and could form the basis for communccation--ffii -ffiicient large-scat scale RL in contrast prior works (Nair et al., 2015; Espeholt et al., 8; 2019). Most critically, we show that under deployment efficiency constraints, most prior algorithms model-free model-based, online or offline -fill to achieve successful learning. One possible direction for future work to incorporate verification efficiency into consideration, since a stochastic multi-modal policy could collect more diverse transitions while it takes more trajectories to be verified for safety than a uni-modal policy. While we presented promising results on some realistic simulated environments, validating BREMEN on real robots another direction. We hope our work can gear the research community to value deployment efficiency as important criterion for RL algorithms, and to eventually achieve similar sample efficiency and asymptotic performance as the state-of-the-art algorithms like SAC (Haarnoja al., 2018) while having the deployment efficiency well-suited for safe and practical real- world reinforcement learning.
Model-ENt -Eeembbl (BRE-

---

Published as a conference paper at ICLR 2021
REFERENCES
Fabian Abel, Yashar Deldjoo, Mehdi Elahi, and Daniel Kohlsdorf. Recsys challenge 2017: Offline and online evaluation. ACM Conference on Recommender Systems, 2017.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. arXiv preprint arXiv:1907.04544 2019.
Christopher G. Atkeson, Benzun Wisely Babu, Nandan Banerjee, Dmitry Berenson, Christoper P. Bove, Xiongyi Cui, Mathew DeDonato, Ruixiang Du, Siyuan Feng, Perry Franklin, Michael Gennert, Joshua P. Graff, Peng He, Aaron Jaeger. Joohyung Kim, Kevin Knoedler, Lening Li, Chenggang Liu, Xianchao Long, Taskin Padir, Felipe Polido, G. G. Tighe, and X Xinjilefu. No falls, no resets: Reliable humanoid behavior in the darpa robotics challenge. In International Conference Humanoid Robots, 2015.
Yu Bai, Tengyang Xie, Nan Jiang, and Yu-Xiang Wang. Provably efficient q-learning with low switching cost. Advances in Neural Information Processing Systems, 2019.
Gabriel Barth-Maron, Matthew Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB,
Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. International Conference on Learning Representations, 2018.
Rinu Boney, Juho Kannala, and Alexander Ilin. Regularizing model- planning with energy- based models. In Conference on Robot Learning, 2019.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In Robotics: Science and Systems, 2020.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. In Advances in Neural Information Processing Systems, 2015.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A Iyapunov- based approach to safe reinforcement learning. In Advances in neural information processing systems, 2018.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint 1rXiv:1901.1003: 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta pollcy optimization. In Conference on Robot Learning, 2018.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv preprint 12844 2012.
Mare Deisenroth and Carl E Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, 2011.
Gabriel Dulac-Arood, Daniel Mankowitz, and Todd Hester. Challenges real-world reinforcement learning. arXiv preprini 1eXiv:11944 /2901, 2019.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 2005.
10

---

Published as a conference paper at ICLR 2021
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable distributed deep-rl with importance weighted actor-leamer architectures. In International Conference Machine Learning, 2018.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. SEED RL: Scalable and efficient deep-rl with accelerated central inference. arXiv preprint 1rXvv:1910.06591, 2019.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. International Conference on Learning Representations, 2018.
Rasool Fakoor, Pratik Chaudhari, and Alexander Smola. P3o: Policy-on policy-off policy opti- mization. arXiv preprint 1005:::900001550 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, 2019.
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, Jiayu Yao, Isaac Lage, Christopher Mosch, Li wei H. Lehman, Matthieu Komorowski, Matthieu Komorowski, Aldo Faisal, Leo Anthony Celi, David Sontag, and Finale Doshi- Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint vxXiv:/005. /2298, 2018.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning, 2016.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In International Conference Robotics and Automation, 2017a
Shixiang Gu, Timothy I. Lillicrap, Zoubin Ghahramani, Richard E. Turner, and Sergey Levine. Q-Prop: Sample-efficient policy gradient with an off off-policy critic. In International Conference Learning Representations, 2017b.
Zhaohan Guo and Emma Brunskill, Concurrent pac rl. In AAAI Conference on Artificial Intelligence, 2015.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, 2018.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, 2015.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.004453 2019.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Optt Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, 2018.
11

---

Published as a conference paper at ICLR 2021
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2014.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-Ensemble Trust-Region Policy Optimization. In International Conference Learning Representations, 2018.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 1992.
Yao Liu, Adith Swaminathan, Alckh Agarwal, and Emma Brunskill. Off- policy gradient with state distribution correction. arXiv preprint arXiv:1904.088 1904.09474.. 2019.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy eval- uation across representations with applications to educational games. In International Conference on Autonomous Agents and Multiagent Systems, 2014.
Ajay Mandlekar, Fabio Ramos, Byron Boots, Li Fei-Fei, Animesh Garg, and Dieter Fox. IRIS: Im- plicit reinforcement without interaction at scale for learning control from offline robot manipulation data. arXiv preprint arXiv:1911.05331,, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 2015.
Susan A Murphy, Mark van der Laan, James Robins, and Conduct Problems Prevention Re- search Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 2001.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hicrarchical reinforcement learning. In Advances in Neural Information Processing Systems, 2018.
Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. Multi-agent manipulation via locomotion using hierarchical sim2real. In Conference on Robot Learning. 2019.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural Network Dynamics for Model-Based Based Deep Reinforcement Learning with Model-Free Fine-Tuning International Conference on Robotics and Automation, 2018.
Vaishnavh Nagarajan and Zico Kolter. Generalization in deep networks: The role of distance from initialization. arXiv preprint /rXiv: 2019.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv: /50.::450002400 2015.
12

---

Published as a conference paper at ICLR 2021
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint 2xXx:20006.09559, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weiggted regressionn Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
Matthias Plappert, Marein Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learn- ing: Challenging roboties environments and request for research. arXiv preprint a8Xiv: 1002. 0944,9 2018.
Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal-differen learning with function approximation. In International Conference Machine Learning, 2001.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 779.. 10087, 2017.
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. arXiv preprint 20X4::200.007804, 2020.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 190.0010000000000 2019.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. International conference on artificial intelligence and statistics, 2011.
Nicolas Le Roux. Efficient iterative policy optimization. arXiv preprint arXiv:1612.08960 2016.
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In International Conference on Machine Learning, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707.06377 2017.
Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin A. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
Satinder P Singh, Tommi Jaakkola, and Michael I Jordan. Learning without state-estimation in partially observable markovian decision processes. Machine Learning Proceedings. Elsevier, 1994.
Satinder P. Singh, Tommi Jaakkola, and Michael Jordan. Reinforcement learning with soft state aggregation. In Advances in Neural Information Processing Systems, 1995.
Sungryull Sohn, Yinlam Chow, Jayden Ooi, Ofir Nachum, Honglak Lee, Ed Chi, and Craig Boutilier. BRPO: Batch residual policy optimization. arXiv preprint arXiv:2002.05522 2020.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 1991.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint 1rXiv:1907.02055 2019.
Yifan Wa, George Tucker, and Ofr Nachum. Behavior Regularized Offline Reinforcement Learning. arXiv preprint / 1361, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-bard baded offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.
13

---

Published as a conference paper at ICLR 2021
APPENDIX
A IMPLICIT KL CONTROL FROM A MATHEMATICAL PERSPECTIVE
We intuitively understand that behavior cloning initialization with trust-region updates works as a regularization of distributional shift, and this can be supported by theory. Following the notation of Janner et al. (2019), we denote the generalization error of dynamics model on the state distribution under the true behavior policy as ém = max, By-d, Drv (p( ( (8,, ax) )lpp( 4 [8,, (t)), where Drv represents the total variation distance between true dynamics and learned model Pú. We also (TT(|[T))
denote the distribution shift on the target policy as max, Drv(Tsl I 6.. bound relating the true returns v/]r] and the model returns 1)( on the target policy is given in Janner et al. (2019) as,
maz (cm + 2ex)
4 Wmaan Esr
||
+
(5)
(1 ?)?
(1-?)
This bound guarantees the improvement under the true returns as long as the improvement under the model returns increases by more than the slack in the bound due 6m: Cr (Janner al., 2019; Levine et al., 2020).
We may relate this bound to the specific learning employed by BREMEN, which includes dynamics model learning, behavior cloning policy initialization, and conservative KL-based trust-region policy updates. To do so, we consider an idealized version of BREMEN, where the expectations over states in equations Equation 1, 3, 4 are replaced with supremums and the dynamics model is set to have unit variance.
Proposition 1 (Policy and model error bound). Suppose we apply the idealized BREMEN on dataset D, and define Eg. Es in terms of the behavior cloning and dynamics model losses as,
Eg == sup Eer DX DC-leillaa nØ (s) /2] -H(T)(-1ss)
Co = sup S [1ls' fo a) 1)1/22 -H(p(-ls,))
where H denotes the Shannon entropy. If one then applies T KL-bused trust-region steps of step size 6 (Equation 4) using stochastic dynamics models with mean fo and standard deviation 1, then
Ex. -==V/e
1 j 1 1
de
da log +T
1 20 cm s V
dg log 2n,
where da and d, denotes the dimension of action and state space.
Proof.
We first consider
The behavior cloning objective in its supremum form is,
sup E EEw t (s) / H-TT(--ss)
EB
sED
da log 2m
sup E. Bo D
18) 1 log Tu, (a|s)] H((T((11s)
2
da log 2n.
sup D (-(-->>|| Te. (T. (-1s))
sED
2
We apply Pinsker's inequality to the true and estimated behavior policy to yield
1
+ da log 2n.
sup ( <
V
E
By the same Pinsker's inequality, we have, sup Drv (me, (-Is)]| (-18)) S v5/2.
Therefore, by triangle inequality, we have
2 + da log 2nT + TT 7//0- c..
1
sup ( s
4
14

---

Published as a conference paper at ICLR 2021
as desired.
We perform similarly for cm.
The model dynamics loss
sup DD ||s' fo(s, al1/2 H(((--(s,a)
EΦ =
5.a
dg log 2n
sup BB |- log po (og)o("' H(p(--1s,
8,0
dg DKL (p<k(,((ss,,)1pp <
sup
log 2
2
5.a
We apply Pinsker's inequality to the true dynamics and learned model to yield
de
S Sup <aupprrooof-ss.oolloo---- a)) - 225 log 2n,
Cm
as desired.
B TRADE-OFF BETWEEN SAMPLE AND DEPLOYMENT EFFICIENCY
An important aspect of deployment efficiency the trade-off between sample and deployment efficiency. To collect multiple data points per experiment and show this trade-off, we run recursive BREMEN with different batch sizes, and record how many samples are required cross different reward thresholds.
HalfCheetah (Reward 7,000 result) and other results from Figure 5 generally show that high deploy- ment efficiency lowers sample efficiency, confirming the inherent trade-off. However, in rare cases, e.g. Ant (Reward 5,000 result), could be possible to achieve both high deployment efficiency and high sample efficiency through the right choice of the batch size hyper-parameter.
Ant (BREMEN)
HalfCheetah (BREMEN)
+ 0000
25k
%000
10k
Number of Deployments
15.0
100k
23-
50k
200k
20
7000
200k
Number of
1000
4000
2000
3.0
2000
4000
5000
3000
4000
Secg
500g
6000
5000
7000
25. 2000
30go
14
10.0
12.5
15.0
Number of Samples (100k)
Number of Samples (100k)
Figure 5: From the view of both sample and deployment efficiency at certain cumulative reward threshold, we respectively represent the number of
evaluate BREMEN in Ant (left) and HalfCheetah (right). and samples and the number deployments. Each data point comes from running BREMEN with different reward thresholds and batch sizes. The numbers above the points (c.g. 1000, 2000, ...) represent the reward threshold. The results (especially, reward 7,000 threshold in HalfCheetah) generally show that high deployment efficiency lowers sample efficiency, confirming the inherent trade-off.
15

---

Published as a conference paper at ICLR 2021
C DISCUSSION: IMPORTANCE OF DEPLOYMENT EFFICIENCY IN WEALLWWOLD APPLICATIONS
Our notion of deployment--ffiiienne is necessitated by cost and safety constraints typical in many real world scenarios. Namely, a common approach to real-world applications (Cabi et al., 2020; Dulac- Arnold al., 2019; Kalashnikov et al., 2018) is the following iterative training and data-collection paradigm:
1. Aggregate past previous dataset from worker(s)
2. Update policy based on the collected data
3. Deploy the policy to the worker(s)
4. Monitor the policy works expected e.g. checking if it does not violate safety criterion (this safety verification step may alternatively happen before step 3)
5. Let the worker(s) collect experiences with the latest policy.
It is easy see that the number of deployments is critical bottleneck, as it involves both monitoring of the policy (Step 4) and communication to the workers (Step 3), and both of these steps can incur significant cost. Specifically, Step 4 requires evaluating the policy for safety, and often requires human monitors (Atkeson et al., 2015). As for Step 3, communication to workers can also be bottleneck, especially in highly-parallelized distributed RL systems (Nair al., 2015; Espeholt et al., 2018; 2019). Every policy deployment requires potentially expensive communication between different machines/processes, and this can be a bottleneck on the whole system.
As concrete example of the necessity of good deployment efficiency, consider optimization of personalization in web apps or recommender systems (Abel et al., 2017). Once a policy is learned on batch of past experiences, it is deployed both safety and communication concerns are relevant: Safety the new policy is typically ensured by initially deploying the policy small percentage of users; after monitoring the results for some length time (e.g. the newly deployed policy does not deteriorate user experiences), one can expand policy to web-hosting servers can be time
collection of web-hosting servers. In this scenario,
the target user set. As for communication, deploying intensive, especially in large-scale web applications where the policy must be deployed to a network of servers around the world. Thus, in this setting, is clear that online updating of the policy is infeasible due to both safety and communication constraints. Accordingly, the deployment-efficienng of any candidate RL algorithm is of tantamount importance.
The safe exploration might be mentioned as potential alternative to deployymen--effccinncyy While safe exploration can arguably tackle the first concern above (safety risks of the policy), does nothing to mitigate the latter (the engineering or communication costs associated with online deployment of a policy). Furthermore, this still ignores the fact that in many scenarios the ability exploration is not given. While some safe RI. algorithms can provide guarantees in tabular cases, these guarantees no longer hold when using function approximation with neural networks (Chow et al., 2018). In these cases, it can be much more difficult to perform "safe exploration" than it is to develop a deployment-eefficeen algorithm.
safe
16

---

Published as a conference paper at ICLR 2021
D EVALUATING OFFLINE PERFORMANCES ON D4RL DATASETS
We compare BREMEN to MOPO (Yu et al., 2020), concurrently proposed model-based offline methods penalized by model epistemic uncertainty, and state-of-the- model-free offline algorithms, namely, CQL (Kumar et al., 2020), BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019), AWR (Peng al., 2019) and BCQ (Fujimoto al., 2019), on the D4RL MuJoCo locomotion datasets (Fu et al., 2020), used as standard offline RL benchmarks (Kumar et al., 2020; Nair et al., 2020). They have several types offline data collected with different strategies. We choose hyper parameters of BREMEN in Section 5.1 and F.2.2. Table 2 shows BREMEN beats recent state-of-the-an algorithms with the highest normalized score (around 100 corresponds to an expert) several tasks, while none of the methods consistently achieves the best performance. This result suggests that the implicit regularization with the model-based method performs surprisingly well in offline settings despite of its simplicity.
Task Name
BC
BREMEN
MОPO
COL
BEAR
BRAC-v
AWR
BCQ
hallcheetah-random
2.1
36,9
31.9
35.4
25.1
31.2
2.5
2.2
walker2d-random
1.6
3.7
13.0
7.0
7.3
1.9
1.5
4.9
hopper-random
9.8
12.2
13.3
10.8
11.4
12.2
10.2
10.6
halfcheetah-mmddim
36.1
55.0
44.4
41.7
46.3
37.4
40.7
walker2d-medium
6.6
59.6
14.0
79.2
59.1
81.1
17.4
53.1
hopper-medium:
29.0
69.3
26.5
58.0
52.1
31.1
35.9
54.5
halfchhetaa-mmeiiu--erpaaa
38.4
47.2
54.0
46.2
38.6
47.7
40.3
38.2
walleer2d-meeium--rplayy
11.3
42.7
26.7
0.9
15.5
15.0
hopper-meddium-replay
11.8
24.1
92.5
48.6
33.7
0.6
28.4
33.1
half halfihheeta--mecduu--eeeerr
35.8
53.3
57.9
62.4
53.4
41.9
52.7
64.7
walker2ddmmedium-exper
6.4
55.2
55.0
98.7
40.1
81.6
53.8
57.5
hopper-medium-expeeee
111.9
64.6
51.7
111.0
96.3
110.9
Table 2: Evaluation on D4RL MuJoCo locomotion datasets. The normalized score of BREMEN are averaged over 4 random seeds. We refer the score of MOPO (Yu et al., 2020) and CQL (Kumar et al.. 2020) from their original papers. Other results cited from Fu et al. (2020). BREMEN achieves the best and competitive several domains, while of the algorithms heats all other methods.
E INCORPORATING PESSIMISTIC MODEL-BASED OFFLINE METHODS INTO BREMEN
The concurrent model-based offline RI. methods prescribe the use of uncertainty-basee penalties (Ki- dambi et al., 2020; Yu et al., 2020), which can be incorporated into BREMEN. We therefore augmented BREMEN with either hard (MOReL-like, green) seft (MOPO-like, orange) reward penalty according to model uncertainty. MOReL quantifies the uncertainty measuring the maximum discrepancy of the prediction across the ensembles of the models and receives constant negative reward (-5.0 in our experiments) if the discrepancy is larger than the threshold (we set 3.0). MOPO measures the uncertainty by the maximum standard deviation of the model ensembles and uses this as a reward penalty with a coefficient (0.1 in our experiments). Evaluations in Figure 6 reveal that the soft reward penalty has notable results in Hopper and Walker2d, where model uncertainty is more crucial due to the episode's s termination. Hard reward penalty seems overly pessimistic in depfoyteet-efffciient settings.
ME-TRPO
MDReL
MOPD
BREMEN
Beich 200.00000000 000 cumulatve rowardds
Ard
HallChwwlahh
Hopper
Walker2d
2603
2000
Deployment
Deployment
Deployment
Deployment
Comparison the pessimistic reward shaping incorporated into BREMEN. Soft reward penalty (MOPO-like, orange) performs environments where incomplete models appear to be fatal.
Figure 6:
17

---

Published as a conference paper at ICLR 2021
F DETAILS OF EXPERIMENTAL SETTINGS
F.1 IMPLEMENTATION DETAILS
For our baseline methods, we use the open-source implementations of SAC, BC, BCQ, and BRAC published in Wu et al. (2019). SAC and BRAC have (300, 300) Q-Network and (200, 200) policy network. BC has (200, 200) policy network, and BCQ has (300, 300) Q-Network, (300, 300) policy network, and (750, 750) conditional VAE. As for online ME-TRPO, we utilize the codebase of model-based RL benchmark (Wang et al., 2019). BREMEN and online ME-TRPO use policy consisting of two hidden layers with 200 units. The dynamics model also consists of two hidden layers with 1,024 units. We use Adam (Kingma & Ba, 2014) as the optimizer with the learning rate of 0.001 for the dynamics model, and 0.0005 for behavior cloning in BREMEN. Especially in BREMEN and online ME-TRPO, we adopt a linear feature value function to stabilize the training. BREMEN in deployment-efficien settings takes about two three hours per deployment NVIDIA TITAN V.
To leverage neural networks Dyna-style (Sutton, 1991) dynamics models, modify reward and termination function so that they are not dependent on the internal physics engine for calculation, following model-basd based benchmark codebase (Wang et al., 2019); see Table Note that the score of baselines (c.g.. BCQ. BRAC) is slightly different from Wu et al. (2019) due to this modification of the reward function. We re-run each algorithm our environments and got appropriate convergence.
The maximum length of one episode is 1,000 steps without any termination in Ant and HalfCheetah; however, termination function enabled in Hopper and Walker2d. The batch size of transitions for policy update is 50,000 in BREMEN and ME-TRPO, following Kurutach et al. (2018). The batch size of BC and BRAC is 256, and BCQ is 100, also following Wu et (2019).
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 7: Four standard MuloCo benchmark environments used in our experiments.
Environment
Reward function
Termination in rollouts
Ant
te 0.1 a 1/2
3.0 X (2t
0.37)
+1
False
HalfCheetah Hopper
0.1| a.
False
4. 0.001
True
Walker2d
4. 0.001
True
Table 3: Reward function and termination in rollouts in the experiments. We remove all contact information from observation of Ant, basically following Wang al. (2019).
18

---

Published as a conference paper at ICLR 2021
F.2 HYPER PARAMETERS
In this section, we describe the hyper-parameters in both deployment-efficient RL (Section F.2.1) and offline RL (Section F.2.2) settings. We all of experiments with five random seed, and the results are averaged.
F.2.1 DEPLOYMENT EFFICIENT RL
Table 4 shows the
of BREMEN. The rollout length is searched from {250, 500, 1000), and max step size is searched from (0.001, 0.01, 0.05, 0.1, 1.0). As for the discount factor and GAE A, we follow Wang et al. (2019).
Parameter
Ant
HalfCheetah
Hopper
Walker2d
Iteration per batch
2,000
2,000
6,000
2,000
Deployment
10
Total iteration
10,000
10,000
60,000
20,000
Rollouts length
250
250
1,000
1,000
Max step size
0,05
0.1
0.05
0.05
Discount factor
0,99
0.99
0.99
0.99
GAE
0.97
0.95
0,95
0.95
Stationary noise
0.1
0.1
0.1
0.1
Table 4: Hyper-parameters of BREMEN in deployment-erffcceet settings.
Number of Iterations for Policy Optimization To achieve high deployment efficiency, the number of iterations for policy optimization between deployments is one of the important hyper-parameters for fast convergence. In the existing methods (BCQ, BRAC, SAC), we search over three values: {10,000, 50,000, 100,000), and choose 10,000 in BCQ and BRAC, and 100,000 in SAC (Figure 8). For BREMEN, we also search over three values: (2,000, 4,000, 6,000). Figure 9 shows the results of iteration search, and we choose 2,000 in Ant, HalfCheetah, and Walker2d, and 6,000 Hopper.
10K
100K
Batch=200, .000 cumulative rewards
HallCheetah
Happer
Walker2d
2000
7500
3009
Deployment
Deployment
Deployment
Deployment
Figure Search number of iterations SAC policy optimization between deployments. The number transitions per one data-collection 200K.
4K
cumulative rewards
HalCChosthh
Hopper
Batch=200, 0000
3008
12:0
500
Deployment
Deployment
Deployment
Depleyment
Figure 9: Search on the number of iterations BREMEN policy optimization between deployments. The number of transitions per one data-collection is 200K.
19

---

Published as a conference paper at ICLR 2021
Stationary Noise in BREMEN To achieve effective exploration, the stochastic Gaussian policy a good choice. We found that adding stationary Gaussian noise to the policy in the imaginary trajectories and data collection led to the notable improvement. Stationary Gaussian policy is written as,
Gg tanh( +
<~ <~,
Another choice is learned Gaussian policy, which parameterizes not only pe but also dg. Learned gaussian policy is also written as,
at tanh((u(ss))) oo(st) O,,,
oo((s)
e~~(0,0")
We utilize the zero-mean Gaussian (0,,0²) and tune up in Figure 10 with HalfCheetah, comparing stationary and learned strategies. From this experiment, we found that the stationary noise, the scale of 0.1, consistently performs well, and therefore we used for all our experiments.
HalfCheetah
Batch= 200. ,000 Cumulative reward
6000
4000
Deployment
a= 0.05
d= 0.5
a= 0.5//eamed]
ME TRPO(anline)
0.1
g= 0.1 (le(aed)dd)
1.0/eammed)
Figure 10: Scarch on the Gaussian noise parameter HalfCheetah. The number of transitions per one data-collection is 200K.
Other Hyper-parameters in the Existing Methods As for online ME-TRPO, we collect 3,000 steps through online interaction with the environment per 25 iterations and split these transitions into a 2-to-1 ratio of training and validation dataset for learning dynamics models. In batch size 100,000 settings, collect 2,000 steps and split with 1-to-1 ratio. Totally, iterate 12,500 times policy optimization, which is equivalent to 500 deployments of the policy. Note that we carefully tune up the hyper-parameters of online ME-TRPO, and the performance is improved from Wang et al. (2019).
Table and Table shows the tunable hyper-parametten of BCQ and BRAC, respectively. We refer Wu al. (2019) to choose these values. In this work, BRAC applies primal form of KL value penalty, and BRAC (max Q) means sampling multiple actions and taking the maximum according to the learned function.
Parameter
Ant
HalfCheetah
Hopper
Walker2d
Policy learning rate Perturbation range +b
3e-05
3e-04
3e-06
3e-05
0.15
0.5
0.15
0.15
Table 5: Hyper-parametern of BCQ.
Parameter
Ant
HalfCheetah
Hopper 3e-5
Walker2d le-5
Policy learning rate
le-4
le-3
Divergence penalty
0.3
0.1
0.3
0.3
Table 6: Hyper-parameter of BRAC.
20

---

Published as a conference paper at ICLR 2021
F.2.2 OFFLINE RL
In the offline experiments, we apply the same haper-parameters as in the deployment-efficiett settings described above, except for the iteration per batch. Algorithm 2 is pseudocode for BREMEN in offline RL settings where policies are updated only with one fixed batch dataset. The number of iteration 7' is set to 6,250 in BREMEN, and 500,000 in BC, BCQ, and BRAC.
The datasets for 50k or 100k experiments are sliced from the beginning of the 1M batched datasets without shuffling, but we observed that the distribution of rewards in 50k or 100k is not different from 1M.
Algorithm 2 BREMEN Offline
Input: Offline dataset optimization T'
{8t; at, Pt,, Se41), Initial parameters f0r,--. pK 9, Number of policy
1: Train K dynamics models fo using via Equation 1.
Train estimated behavior policy its using D by behavior cloning Equation
3: Initialize target policy T6, = Normal(ft,.)
4: for policy optimization & 1, do
5: Generate imaginary rollout.
Optimize target policy ma satisfying Equation 4 with the rollout.
21

---

Published as a conference paper at ICLR 2021
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 PERFORMANCE ON THE DATASET WITH DIFFERENT NOISE
Following Wu al. (2019) and Kidambi al. (2020), we additionally compare BREMEN in offline settings to the other baselines (BC, BCQ, BRAC) with five datasets of different exploration noise. Each dataset has also one million transitions.
epsl: 40 % of the dataset is collected by data-collection policy (partially trained SAC policy)
Tb. 40 % of the dataset is collected by epsilon greedy policy with e = 0.1 to take a random action, and 20 % of dataset is collected by uniformly random policy.
eps3: Same epsl, 40 % of the dataset is collected by тT, 40 % is collected by epsilon greedy policy with = 0.3, and 20 % is collected by an uniformly random policy.
gaussianl: 40 % of dataset is collected by data-collection policy Ths 40 is collected by the policy with adding zero-mean Gaussian noise NN0, 0.1²) to each action sampled from Ty. and 20 % is collected by an uniformly random policy.
gaussian3: 40 of the dataset is collected by data-collection policy Th. 40 collected by the policy with zero-men mean Gaussian noise (0, 0,3²), and 20 % is collected by an uniformly random policy.
random: All of the dataset is collected by uniformly random policy.
Table 7 shows that BREMEN can also achieve performance competitive with state-of-the-ee model- free offline RL algorithm even with noisy datasets. The training curves of each experiment are shown in Appendix G.4.
G.2 COMPARISON AMONG DIFFERENT NUMBER OF ENSEMBLES
To deal with the distribution shift during policy optimization, also known model bias, we introduce the dynamics model ensembles. We validate the performance BREMEN with different number of dynamics models K. Figure 11 and Figure 12 show the performance of BREMEN with the different number of ensembles in deployment-efficcee and offline settings. Ensembles with more dynamics models resulted in better performance due to the mitigation of distributional shift except for K 10, and then we choose K 5.
K=11
K=3
K=S
K-10
Batch= Botch=200: 200 0000 cumulative rewards
Ant
HallChestah
Hopper
Walker2d
2000
2008
22
10K
Deployment
Deployment
Deployment
Deployment
Figure 11: Comparison of the number of dynamics models in deploymeent-fficient settings.
K=11
K=3
K
K=10
n=1 ,,000,,00000 000 cumulative rewards
Ant
HaliChestah
Hopper
Walker2d
200
130 300 4X0 SKE 600 1000 20E 300 MER 5W0 ear
3K0 2000
Iteration
AN0
500
1800
2000 3001
Figure 12: Comparison of the number of dynamics models offline settings.
22

---

Published as a conference paper at ICLR 2021
Noise: epsl, 1,000,000 (1M) transitions
Method
Ant
HalrCheetah
Hopper
Walker2d
Dataset
1077
2936
791
815
13811:11
3788:1740
266:1-486
1185+1155
BCQ
1937=1116
6046+-276
800:-659
479-:537
BRAC
2593-+155
7003+1188
1243+1162
3204+:103
BRAC (max Q
29071198
70701881
34881:38
337011177
BREMEN (Ours)
35191129
75851:422
2818:176
17101:429
ME- TRPO (offline)
15141:503
1009:1331
13011:654
128:113
Noise: eps3, 1,000,000 (1M) transitions
Method
Ant
HalfCheetah
Hopper
Walker2d
Dataset
936
2408
662
648
BC
13641:121
28771797
519:1532
10661176
BCQ
1938+-21
5789+188
1170:t:446
1018+1231
BRAC
2718+-90
6434+147
1224:171
2921
BRAC (max Q)
2913++87
6672+136
2103+:746
3079=110
BREMEN (Ours)
340911288
7632.104
2803
1586_1
ME-TRPO (offline)
1843
5504167
1308
354.3299
Noise: gaussian
1,000,000 (1M) transitions
Method
Ant
HalfCheetah
Hopper
Walker2d
Dataset
1072
3150
882
1070
BC
1279.180
4192:1189
31::16
113714777
BCQ BRAC
1958.176
585411498
47511416
60811416
2905+#81
7026+168
1456#111
3030+103
BRAC (max
2910=157
7026=168
1975+189
32421-97
BREMEN (Ours)
1612+165
79281:13
1999:1617
1402+290
ME-TRPO (offline)
1275--656
1275:1656
909-1631
171
Noise: gaussian3,
1,000,000 (IM) transitions
Method
Ant
HalfCheetah
Hopper
Walker2d
Dataset
1058
2872
781
981
BC
1300:1:34
4190:169
611:1467
12174:361
BCQ
1982-1-7
57811-543
11374:582
258-+286
BRAC
3084++180
3933-1-740
1432+-499
3253 118
BRAC (max Q
29161199
3997-2771
1417#266
3572:1153
BREMEN (Ours)
345
8124:1145
186711354
229911474
ME- TRPO (offline)
12371310
21411872
9731243
2191145
Noise: random,
1,000,000 (1M) transitions
Method
Ant
HalfCheetah
Hopper
Walker2d
Dataset
470
-285
34
2
BC
989110
-211
1061:62
10811100 110
BCQ
1222+=114
2887+2242
206+t7
228
BRAC
1057+-92
3449+-299
227+:30
29+-54
BRAC (max Q)
6834557
3418+171
2244:37
26:+50
BREMEN (Ours)
105111
3627-199
270.168
254.16
ME-TRPO (offline)
2221.1663
27011120
321.1.9
262.1 13
Table 7: Comparison of BREMEN the existing offline methods in offline settings, namely, BC, BCQ (Fujimoto al., 2019), and BRAC (Wu al., 2019). Each cell shows the average cumulative reward and their standard deviation with seeds. The maximum steps per episode 1,000. Five different types of exploration noise are introduced during the data collection, epsl, eps3, gaussian gaussian3, and random. BRAC applics primal form of KL value penalty, and BRAC (max Q) means sampling multiple actions and taking the maximum according the learned Q function.
23

---

Published as a conference paper at ICLR 2021
G.3 IMPLICIT KL CONTROL IN OFFLINE SETTINGS
Similar to Section 5.3, we present offline RL experiments to better understand the effect of implicit KL regularization. In contrast to the implicit KL regularization with Equation 4, the optimization of BREMEN with explicit KI. value penalty becomes
OK+1 = arg max
E SARNTHR for To (als) (A (s,a) (D
(6)
S.t. DRL (-)(-|))[ (-18))] 5,
where A (s, a) is the advantage of TØ computed using imaginary rollouts with the learned dynamics model and d' is the maximum step size. Note that BREMEN with explicit KL penalty does not utilize behavior cloning initialization.
We empirically conclude that the explicit constraint -ao (mo(())) (+())))))))))))))))))}){{))))( is unnecessary and just TRPO update with iaaiiooriiitializzatin as implicit regularization is sufficient in BREMEN algorithm. Figure 13 shows the KL divergence between learned policies and the last deployed policies (top row) and model errors measured by a mean squared error of predicted next state from the true state (second row). We find that behavior initialized policy with conservative KL trust-region updates well stuck to the last deployed policy during improvement without explicit KL penalty. The policy initialized with behavior cloning also tended suppress the increase of model error, which implies that behavior initialization alleviates the effect of the distribution shift. In Walker2d, the model error of BREMEN is relatively large, which may relate the poor performance with noisy datasets in Section G.I.
BREMEN
ME DE
Explicit KL(o=1e-5)
Ant
Hā HalfCheetah
Hopper
Walker2d
Currustive rewards
06.0
2000
2000
2000
was aoss viee seee
1900 2005 1000 aost Vore wre
1000 2005 splr 2000 wee wees
1000 2005 1000 a00n weee nees
Ant
HalfCheetah
Hopper
Walkerr2d
am00
30000
0000
10000
Dece
2000
seen
4000
20000
4000
50000
2000
10:00
1000 REST
3230 2000 ZEST vone Iteration
3300 2008
Iterstion Hopper
BERY sIce
2000 "MET BENY
HiltCheetahh
Walkord2
Phoee arror (N52)
8.000
6005
2.0
6420
8.000
8.000
Iteration
Theration
Iberation
lteration
Figure 13: Average cumulative rewards (top row) and corresponding KL divergence of learned policies from the last deployed policy (second row) and model errors (bottom row) in offline settings with 1M dataset (no noise). Behavior initialized policy (purple line) tends suppress the policy and model error during training better than no-initialization (red line) or explicit KI. penalty (green line).
24

---

Published as a conference paper at ICLR 2021
G.4 TRAINING CURVES FOR OFFLINE RL WITH DIFFERENT NOISES
In this section, we present training curves of our all experiments offline settings. Figure 14 shows the results in Section 5.1. Figure 15, 16, 17, 18, and 19 also show the results in Section G.I.
BREMEN
ME TRPO(attlin))
BC
BCQ
BRAC
BRAC(max Q2
HalfChectah
Hopper
Fim 1,0000000
2530
2000
3000
2500
1931
1500
2000
U830
Thene
WalkerZd
cumsuntivv recurrts
ne100 060
19:00
2000
10:04
Iteration
Iteration
Iteratinn
Hopper
cumuistive rewards
2000
3000
n-50 600
1000
2030
L200
10:00
-3000
-2000
Figure 14: Performance in Offline RI. experiments (Table 1). (top row) dataset size 1M, (second row) 100K, and (bottom row) 50K, respectively. Note that x-axis is the number of iterations with policy optimization in log-scale.
BREMEN
ME-TRPO(OMf ine)
BCQ
BRAC
BRAC(max
cumulative rewards
Ant
HallCheetah
Hopper
Walker2d
2009
epsl
1000
1:0
12
12
30
Figure 15: Performance in Offline RL experiments with e-greedy dataset noise 0.1. Dataset size is 1M.
BREMEN
ME-TRPO(oftlnn
BRAC
BRAC/max
cumulative rewards
Ant
HalfCheetah
Hopper
Walker2d
2040
eps3
1000
500
12
12
12
12
22
12
Figure 16: Performance in Offline RL experiments with e-greedy dataset noise = 0.3. Dataset size is 1M.
25

---

Published as a conference paper at ICLR 2021
BREMEN
ME-TRFOLoMing
BRACImax
gaussian1 cumulative rewards
Ant
HaltCheetah
Happer
Walker2d
SK
2203
10"
19
Iteraioo
Iteration
Iteration
Figure 17: Performance in Offline RL experiments with gaussian dataset noise NO 0.13). Dataset size is 1M.
BREMEN
ME-TRPOjoftlnn Ino)
BC
BCQ
BRAC
BRACImax
gaussian3 cumulativv rewards
Ant
HalfChectah
Hopper
Walker2d
2006
2003
19
Iteration
Figure 18: Performance in Offline RI. experiments with gaussian dataset noise No0 0.3²). Dataset size is 1M.
BREMEN
ME ME-TRFOLofting Inc)
BC
BCQ
BRAC
BRAC/max Q)
random cumulative rewards
Hopper
E00
Beration
Itraaion
Iteration
Iteration
Figure 19: Performance Offline RL experiments with completely random behaviors. Dataset size 1M.
26

---

Published as a conference paper at ICLR 2021
G.5 REPLOYMENT-ERRIIIEE RL EXPERIMENT WITH DIFFERENT REWARD FUNCTION
In addition to the main results in Section 5.2 (Figure 2), we also evaluate BREMEN in deployment- efficient setting with different reward function. We modified HalfCheetah environment into the one similar to cheetah-run task in Deep Mind Control Suite." The reward function defined as
00.1
(0 < 3t. S 10)
(#z 10),
and the termination is turned off. Figure 20 shows the performance of BREMEN and existing methods. BREMEN also shows better deployment efficiency than other existing offline methods and online ME-TRPO, except for SAC, which is the same trend as that of main results.
BREMEN
ME-TRPO
BCQ
BRAC(max Q)
SAC(online)
Explicit KL
BRAC
ME-TRPO(Online)
CheetahRun
CheetahRun
cumulative rewards rdrdd
Batch=100,000 ,000 cumulative rewards
Batch= tich=200 ,000
1000
1000
600
600
400
400
Deployment
Deployment
Figure 20: Performance in Deployment- Effficien RL experiments with different reward function of HalfCheetah.
cheetah.py cow/deemmm//emmmii http://wwwaaaaaaaaoocccc/////aaa te/
27


Reviews:
[{"review_id": "zB5-eiUBPyH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores an under-researched problem, that of minimizing the number of policy updates in an RL setting (or being “deployment efficient”). This is an important aspect of using RL agents in real “production” environments where there may be many reasons why updates are costly and limiting them is an important consideration in the choice of RL method (or whether to even use RL).\n\nThe paper shows that so-called \"off-policy” methods which, by their naming as such, it is implied that they should work in a sparse-deployments environment are, in fact, not suited (and often not evaluated) for this regime.\n\nBy introducing a set of simple and strait-forward steps to the update and deployment process, the paper shows performance that approaches the continuous-deployment performance of comparable un-constrained methods.\n\nThe main ingredients of the proposed method seem to be:\nModel based approach to support model-based offline exploration (an ensemble of models to prevent exploitation of model inaccuracies)\nRe-estimation of a model by Behaviour Cloning (BC) with data from last deployment (appears to have a large contribution though I don’t understand why)\nConservative off-line policy updates (constrained by KL to BC policy) using offline rollouts (with forward model ensemble)\n\nThe evaluation presented in the paper and extensive (13-page) appendix clearly show the advantage of the proposed method in the sparse-deployments regime and the overall competitive performance with regards to sample efficiency as well. Code was made available as supplementary material.\n\nI am unclear on why each training iteration should start with a policy learned from behavioural cloning of the last deployment’s data instead of the model that was deployed which would be available at that time. Figure 4 clearly shows BC to be the better approach In practice but I would appreciate some intuitive reasoning for this (unless this is standard practice).\n\nPerhaps my main concern with this paper, given the problem it addresses - that of a real deployment setting, is that it seems that some of the parameters that need to be tuned to achieve great performance require fitting to the specific task (i.e. deployments !). As far as I understand it, parameters such as the number of offline policy updates per deployment or the weight of the KL-divergence in the policy update step are crucial to good performance yet the paper does not explain how to choose them without engaging in the true environment. If that is correct than this seems to defeat the aims of the paper and question the overall methodology for deployment-efficiency.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nbL49zr2SGR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors  highlight the problem of iterated batch RL (and thus its deployment efficency) and propose an algorithm along\nwith novel evaluation schemes.\n\nThere are several things I like about this paper:\n\n- The topic is very important and underexplored, especially the deployment efficency w.r.t. the number of batch collections\n- Lots of relevant work is cited\n- The experiment make sense given the research question. Especially evaluations such as in Figure 2.\n- The results show significant improvements of the proposed method over state of the art\n- The method  is \"simple\", in a good way, and has not that much moving pieces, compared to other RL algorithms\n\nThe are a few things I think could be improved:\n\n- Because the paper considers \"repeated batch\" it should touch the subject of \"if I can collect x batches, what batches would tell me the most?\". I.e the algorithm as written right now is \"greedy\" in the sense that the policy will only act in the way it considers most optimal (and is not completely different from the behavior policy).  Batch exploration strategies would be highly interesting\nand as \"iterative batch RL\" is the domain of this paper, these need to be addressed.\n\n- Related:  While epistemic uncertainty is modeled (via Ensembles of dynamic models) it is not explicitly utilized.  Eq. (4) contains a \"safety mechanism\" via the trust region approach.  Why not include a risk criterion over the ensemble spread?\n\n- Why use only deterministic models?  Using something like [1,2] that enables modeling stochastic effects would improve the \nuniversality of the approach further, not being restricted to deterministic problems.\n\nI am a bit unsure about the novelty of Section 4  --  most pieces of the algorithm are already known or \"obvious\".  To me the \nbiggest novelty lies in identifying the \"repeated batch\" as  an important problem, that is underexplored and shows how \nclassical RL method underperform strongly here. From this  evaluations like shown in Figure 2 are novel and make a lot of sense. \n\n\n[1] Chua, Kurtland, et al. \"Deep reinforcement learning in a handful of trials using probabilistic dynamics models.\" Advances in Neural Information Processing\n\n[2] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" Advances in neural information processing systems. 2017.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hynMigrsjO4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper has been well written and is easy to understand. The premise of the paper is clear, and methods have been evaluated properly. \n\nSuggestions to improve the paper:\n- While the deployment efficiency metric has been clearly explained, the justification of this metric has not been elaborated on sufficiently. For the community to adopt a new metric, the paper needs to provide enough evidence to show that deploying policy is expensive, time consuming or requires manual curation. What are the steps involved in deployment? In what situations is deployment expensive? How common are they? Sample efficiency is a well established metric because it makes sense that we need to learn a good policy in as few real world interactions as possible. One can argue that safe exploration is essential for real world deployment. However, that is neither discussed nor compared with. \n- The lack of clarity in the deployment efficiency metric also appears in the evaluation. Why does a batch size of 100K make sense? Why limit it to 5 to 10 deployments? These numbers appear to be arbitrary. \n- How does one decide the trade off between sample efficiency and deployment efficiency? \n-  A clarification question: How is the 50k and 100k transition obtained for evaluating sample efficiency? Is it sampled randomly from the 1M dataset or is it sliced without shuffling?\n- The BREMEN algorithm borrows ideas from related prior works, and is reasonable. However, it is unclear why the algorithm is better suited to improve deployment efficiency compared to prior works. From the explanation provided in the paper, BREMEN is a good offline RL algorithm and is coincidentally better at deployment efficiency as well. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "929uFlWxmh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a new approach to learning control policies with improved data efficiency and fewer number of data collection sessions (with each session using a different policy). Further, the authors proposed a new concept of “deployment efficiency”, with a new “deployment” referring to using a new policy to interact with the real environment, for example, for data collection. The new approach belongs to the family of model-based online reinforcement learning algorithms and seems to primarily augment a prior approach, called ME-TRPO, by using a helper policy trained by behavior cloning data collected after the most recent deployment of learner policy. The experiment results validate that the proposed approach achieves better data efficiency and deployment efficiency compared to prior approaches.\n\n==============\n\nPositives\n\nThe experiments are very extensive and of high quality. A diverse set of tasks is used, along with a good range of prior approaches for comparison. The experiments are also well designed,   with one focusing on data efficiency in offline setting and the other one mimicking practical usage in online setting with limited number of deployments.\n\nThe experiment results are clear and convincing. Performance is the best in almost all tasks. \n\nIt is a great step towards wider application of RL algorithms in the real-world by recognizing practically important metric such as deployment efficiency.\n\nThe authors did a good job in introducing the varied set of related works and the general background, providing a clear organization and classification of prior approaches.\n\n==============\n\nNegatives\n\nThe authors could make a clear statement on the contributions of the paper. Several features of the proposed algorithm can be found in the prior approach of ME-TRPO, including all steps shown in Algorithm 1 except steps 2 & 5 & 6, i.e.: the use of an ensemble of learnt dynamics models to mitigate model bias and training instability, the use of imaginary rollout, the choice of TRPO for policy optimization, all are present in ME-TRPO. The clear innovation in this paper seems to be in using the behavior policy to bootstrap policy learning in a new iteration of deployment. The authors could make clear the approach is an extension of ME-TRPO, which at present is not in the way the authors proposed the algorithm.\n\nWhile I don’t dispute the merit of deployment efficiency, it can be better motivated. The first paragraph of Section 3 mentions “verify each policy before deployment”; it’d be good to show references of real world applications of this procedure. It’s conceivable in practical settings, there can be better ways to achieve the verification in a way that circumvents the cost of deployment (for example, limiting policy outputs to a safe range at runtime, instead of “checking out-of-bounds actions” as a way of pre-validation).\n\n==============\n\nRecommendation\n\nOverall the proposed approach clearly yields empirical advantages of prior approaches in both data efficiency and the newly proposed metric of deployment efficiency. The paper suffers from lack of algorithmic novelty, but the experimental efficiency outweighs. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Quality:
The algorithm is thoroughly evaluated and several interesting experiments are included in the appendix. 

# Clarity:
The paper is generally well written.

# Originality:
The proposed approach is a small but novel improvement over existing algorithms (to the best of the reviewers and my knowledge). The concept of "deployment-efficiency" is, in my opinion not novel, since it seems mostly a rebranding of what the MBRL community traditionally refers to as "data-efficiency" -- although I agree that deployment-efficiency is indeed a more accurate term.

# Significance of this work: 
The paper deal with a relevant and timely topic. However, the paper does not compare to the larger MBRL literature. Hence, it is difficult to gauge the significance of this work.

# Overall:
This manuscript offers a good contribution to the topic of model-based reinforcement learning algorithms.

# Minor comments: 
- I suggest removing the word "impressive" from the abstract. This is a subjective term, which should be avoided.
- In my personal opinion, it would be nice to include experiments with more state-of-the-art baselines such as PETS and POPLIN, for which code is available online. It is unclear to me how much the improvement in performance depends on the algorithm itself compared to just having larger batch sizes. From this perspective, Figure 5 in Appendix B is probably the most interesting insight of the manuscript, to me.
Decision: Accept (Poster)