AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
Authors: Ashvin Nair
Murtaza Dalal
Abhishek Gupta
Sergey Levine
Venue:  (2021)
Keywords: reinforcement learning

Abstract: Reinforcement learning provides an appealing formalism for learning control policies from experience. However, the classic active formulation of reinforcement learning necessitates a lengthy active exploration process for each behavior, making it difficult to apply in real-world settings. If we can instead allow reinforcement learning to effectively use previously collected data to aid the online learning process, where the data could be expert demonstrations or more generally any prior experience, we could make reinforcement learning a substantially more practical tool. While a number of recent methods have sought to learn offline from previously collected data, it remains exceptionally difficult to train a policy with offline data and improve it further with online reinforcement learning. In this paper we systematically analyze why this problem is so challenging, and propose an algorithm that combines sample-efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of reinforcement learning policies. We show that our method enables rapid learning of skills with a combination of prior demonstration data and online experience across a suite of difficult dexterous manipulation and benchmark tasks.

Under review as a conference paper at ICLR 2021
AWAC: ACCELERATING ONLINE REINFORCEMENT LEARNING WITH OFFLINE DATASETS
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning provides an appealing formalism for learning control policies from experience. However, the classic active formulation of reinforcement learning necessitates lengthy active exploration process for each behavior, making it difficult to apply in real-world settings. If we can instead allow reinforcement learning to effectively use previously collected data to aid the online learning process, where the data could be expert demonstrations or more generally any prior experience, we could make reinforcement learning substantially more practical tool. While a number of recent methods have sought to learn offline from previously collected data, it remains exceptionally difficult to train policy with offline data and improve further with online reinforcement learning. In this paper systematically analyze why this problem is so challenging, and propose algorithm that combines sample-efficient dynamic programming with maximum likelihood policy updates, providing simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of reinforcement learning policies. We show that our method enables rapid learning of skills with a combination of prior demonstration data and online experience across suite of difficult dexterous manipulation and benchmark tasks.
INTRODUCTION
Learning models that generalize effectively to complex open-world settings, from image recogni- tion (Krizhevsky al. (012) to natural language processing (Devlin et al. 2019). relies on large, high-capacity models and large, diverse, and representative datasets. Leveraging this recipe for reinforcement learning (RL. has the potential to yield real-world generalization for control appli- cations such as robotics. However, while deep RL. algorithms enable the use of large models, the use of large datasets for real-world RL has proven challenging. Most RL algorithms collect new data online every time new policy is learned, which limits size and diversity of the datasets for RL. In the same way that powerful models in computer vision and NLP are often pre-trained on large, general-purpont datasets and then fine-tuned on task-specific data, RL policies that generalize effectively to open-world settings will need to be able to incorporate large amounts of prior data effectively into the learning process, while still collecting additional data online for the task at hand.
For data-driven reinforcement learning, offline datasets consist of trajectories of states, actions and associated rewards. This data can potentially come from demonstrations for the desired task (Schaal 1997, Atkeson & Schaal 1997). suboptimal policies (Gao et al., 2018). demonstrations for related tasks (Zhou et al.] 2019). or even just random exploration in the environment. Depending the quality of the data that is provided, useful knowledge can be extracted about the dynamics of the world, about the task being solved, or both. Effective data-driven methods for deep reinforcement learning should be able to use this data to pre-train offline while improving with online ane-tuning.
Since this prior data can come from variety of sources, we would like to design an algorithm that does not utilize different types of data in any privileged way. For example, prior methods that incorporate demonstrations into RL directly aim to mimic these demonstrations (Nair et al. 2018). which is desirable when the demonstrations are known be optimal, but imposes strict requirements on the type of offline data, and can cause undesirable bias when the prior data is not optimal. While prior methods for fully offline RL provide mechanism for utilizing offline data (Fujimoto al. 2019 Kumar et al. 2019). as we will show in our experiments, such methods generally are not effective for fine-tuning with online data they are often too conservative. In effect, prior methods

---

Under review as a conference paper at ICLR 2021
require to choose: Do assume prior data is optimal not? Do only offline data, or only online data? To make feasible to learn policies for open-world settings, we need algorithms that learn successfully in any of these cases.
In this work, we study how to build RI. algorithms that are effective for pre-training from off- policy datasets, but also well suited to continuous improvement with online data collection. We systematically analyze the challenges with using standard off off-po RL algorithms (Haarnoja et al.) 2018, Kumar ct al. 2019, Abdolmaleki et al. 2018) for this problem, and introduce a simple actor critic algorithm that elegantly bridges data-driven pre-training from offline data and improvement with online data collection. Our method, which uses dynamic programming to train a critic but a supervised learning style update train a constrained actor, combines the best of supervised learning and actor-critic algorithms. Dynamic programming can leverage off off-policy data and enable sample-efficient learning. The simple supervised actor update implicitly enforces constraint that mitigates the effects of distribution shift when learning from offline data (Fujimoto et al., 2019 Kumar et al., 2019), while avoiding overly conservative updates.
We evaluate our algorithm on a wide variety of robotic control and benchmark tasks across three simulated domains: dexterous manipulation, tabletop manipulation, and MuloCo control tasks. Our algorithm, Advantage Weighted Actor Critic (AWAC), is able to quickly learn successful policies on difficult tasks with high action dimension and binary sparse rewards, significantly better than prior methods for off off-policy and offline reinforcement learning. Moreover, AWAC can utilize different types of prior data without any algorithmic changes: demonstrations, suboptimal data, or random exploration data. The contribution of this work is not just another RL algorithm, but a systematic study of what makes offline pre-training with online fine-tuning unique compared to the standard RL paradigm, which then directly motivates simple algorithm, AWAC, to address these challenges.
2 PRELIMINARIES
We consider the standard reinforcement learning notation, with states s, actions a, policy Taass) rewards r(s, a), and dynamics p(s'|s, a)). a). The discounted return is defined as R. = > Zi=t f'r(s; a.). for a discount factor 7 and horizon T which may be infinite. The objective of an RL agent is to maximize the expected discounted return J(rr Ep (r) [Ro] under the distribution induced by the J (Williams 1992). but this is often ineffective due to high variance of the estimator. Many algorithms attempt to reduce this variance by making use of the value function (s) pp)(r) [Re (s). action-value
policy. The optimal policy can be learned directly by policy gradient, estimating
function Q" (s, a) Ep, (+) [R [s, a). or advantage (s, a) Q a) - V> (s). The action-value function for a policy can be written recursively via the Bellman equation:
Q* a) - r(s, a) + "/,_pt"'ls.aa) (s)) = r(s, a) + -
((" a )']
(I)
Instead of estimating policy gradients directly, actor-critic algorithms maximize returns by alternating between two phases (Konda & Tsitsiklis 2000); policy evaluation and policy improvement. During the policy evaluation phase, the critic Q (s, a) is estimated for the current policy म. This can be accomplished by repeatedly applying the Bellman operator B, corresponding to the right-hand side of Equation I as defined below:
1dd Q(s, a) = r(s, a) + + sn)) [E>(a|s>) ' (s', a a')]]-
(2)
By iterating according to Q++1 = B Q*, Q* converges to QT (Sutton & Barto 1998). With function approximation, we cannot apply the Bellman operator exactly, and instead minimize the Bellman error with respect to Q-function parameters Φz:
Ø = arg min Ep[(Q,(s. a) - -)) r(s, a) + yE ar 1Qo, as).
(3)
During policy improvement, the actor π is typically updated based on the current estimate of QT. commonly used technique (Lillicrap et al., 2016 Fujimoto et al., 2018 Haarnoja et al. 2018)) is to update the actor wo, (als) via likelihood ratio or pathwise derivatives to optimize the following objective, such that the expected value of the Q-function Q" is maximized:
0₂ = arg max Eg-D
[a)y(a)s) [ [2 (s, a)]]
Actor-critic algorithms are widely used in deep RL (Miih et al. 2016 Lillicrap et al. 2016 Haarojja al. 2018 Fujimoto et al. 2018). With Q-function estimator, they can in principle utilize oof-po data when used with a replay buffer for storing prior transition tuples, which we will denote 3, to sample previous transitions, although we show that this by itself is insufficient for our problem setting.
2

---

Under review as a conference paper at ICLR 2021
1. Offine Learning
plals,)) 2. Online Fine-tuning
dipekky das
D ((0.0, (.n
nas
Upditt
Q
Update
B
β
x Qo
Figure 1: We study learning policies by offline learning on a prior dataset D and then fine-tuning with online interaction. The prior data could he obtained via prior runs of RI, expert demonstrations, any other source of transitions. Our method, advantage weighted actor critic (AWAC) is able to learn effectively from offline data and fine-tune in order to reach expert-level performance after collecting limited amount of interaction data. Videos and data are available at wisstt..google.coo www.://www...m/////ww/aaaaaaeaea annoyyyous
CHALLENGES IN OFFLINE RL WITH ONLINE FINE-TUNING
In this section, we study the unique challenges that exist when pre- pre-training using offline data, followed by fine-tuning with online data collection. We first describe the problem, and then analyze what makes this problem difficult for prior methods.
Problem definition. A static dataset of transitions, D = {(s s', rr,r) }. is provided to the algorithm the beginning of training. This dataset can be sampled from an arbitrary policy or mixture of policies, and may even be collected by a human expert. This definition is general and encompasses many scenarios, such as learning from demonstrations, random data, prior RL experiments, or even from multi-task data. Given the dataset D, our goal is to leverage D for pre- pre-training and use some online interaction to learn the optimal policy (als), with few interactions with the environment as possible (depicted in Fig D. This setting is representative of many real-world RL settings, where prior data is available and the aim is learn new skills efficiently. We first study existing algorithms empirically in this setting on the HalfCheetah-v2 Gym environment? The prior dataset consists of 15 demonstrations from an expert policy and 100 suboptimal trajectories sampled from behavioral clone of these demonstrations. All methods for the remainder of this paper incorporate the prior dataset, unless explicitly labeled "scratch".
3.1) Data Efficiency. One of the simplest ways to utilize prior data such as demonstrations for RL is to pre-train policy with imitation learning. and fine-tune with on-policy RL (Gupta al. 2019
Rajeswaran et al. 2018). This approach has two drawbacks: (1) prior data may not be optimal; (2) on-policy fine-tuning is data inefficient as it does not reuse the prior data in the RL stage. In our setting, data efficiency is vital. To this end, we require algorithms that are able to reuse arbitrary off- policy data during online RL for data eafa-efficciet -uning. We find that algorithms that use on on-policy fine-tuning (Rajeswaran et al. 2018 Gupta al. 2019)) or Monte- Schaal, [2077] Wang et al. 2018 Peng et al. 2019) are generally much less efficient than off- off-policy actor-critic algorithms.. which iterate between improving æ and estimating Q" via Bellman backups. This can be seen from the results in Figure I, where on-policy methods like DAPG (Rajeswaran) et al. 2018) and Monte-Carlo Carlo return methods like AWR (Peng al. (019) and MARWII. (Wang et al. 2018) are an order of magnitude slower than off-policy actor-critic methods. Actor-critic methods, shown in Figure plot 2, can in principle use off pol-po data. However, we will discuss next, naively applying these algorithms to our problem suffers from a different set of challenges.
return estimation (Peters &
3.2) Bootstrap Error in Offline Learning with Actor-Critic Methods. When standard off-policy actor-critic methods are applied to this problem setting, they perform poorly, as shown in the second plot in Figure 2 despite having a prior dataset in the replay buffer, these algorithms do not benefit significantly from offline training. We evaluate soft actor critic (Haarnoja et al., 2018), a state-of-the- art actor-crite cittrrcritic algorithm for continuous control. Note that "SAC-scratch." which does not receive the prior data, performs similarly to "SACID- prior,"" which does have access to the prior data, indicating that the off- off-policy RL algorithm is not actually able to make use of the off-policy data for pre-training. Moreover, even if the SAC is policy is pre-trained by behavior cloning, labeled "SACID-pretrain" " we still observe an initial decrease in performance, and performance similar to learning from scratch.
This challenge several prior works (Sutton & Barto] 1998; Kumar et al. 2019 Wu et al. 2020 Levine et al.) 2020
be attributed to off-policy bootstrapping error accumulation, observed in
'ee use this environment for analysis because it helps understand and accentuate the differences between different algorithms. More challenging environments like the ones shown in Fig are too hard to solve analyze variants of different methods.
3

---

Under review as a conference paper at ICLR 2021
Data Efficiency from Prior Data
2. Actor Critic Methods
3. Policy Construint Methods
4. Log Likelihood of
Average Returns
7500
3000
150K
200X
108K
150K
200K
100K
158K
200
401X.
Timcslog
Tiaesiops
Tinccllpp
Timesigg
DAING
AWAC (Dues)
SACTI-
DEAAR
Offire de3
AWX
MARWIL
SACII
SAC-eraach
BIAAK-loo loow
Omitze
Figure 2: Analysis of prior methods on HalfCheetah-v2 using offline RL with online fine-tuning. (1) On-policy methods (DAPG, AWR, MARWIL) learn relatively slowly, even with access to prior data. We present our method, AWAC, as an example how off-policy RI. methods can learn much faster. (2) Variants of soft actor- critic (SAC) with offline training (performed before timestep 0) and fine-tuning fine-tuning. We see a "dip" in the initial performance, even if the policy is pretrained with hehavioral cloming. (3) Offine RI. method BEAR (Kumar) et al. (219) on offline training and fine-tuning, including "loose" variant of BEAR with weakened constraint. Standard offline RL methods fine-tune slowly, while the "loose" BEAR variant experiences similar dip as SAC. (4) We show that the fit of the behavior models स. used by these offline methods degrades as new data is added to the buffer during fine-tuning, potentially explaining their poor fine-tuning performance.
2019). In actor-critic algorithms, the target value Q(s', a'), with a' ~ ग,, is used to Fujimoto et al.
update ((s, When a' is outside of the data distribution, Q a') will be inaccurate, leading to accumulation of error on static datasets.
Offline RL algorithms (Fujimoto et al. 2019 Kumar et al. 2019 Wu et al. 2020) propose to address this issue by explicitly adding constraints on the policy improvement update (Equation to avoid bootstrapping outoo-ddistributinn actions, leading to policy update of this form:
arg max E.-DD (Bro(a)s) (Q (s, a)]] s.t. D( n8) S.
(5)
Here, Te is the actor being updated, and (T(aas) represents the (potentially unknown) distribution from which all of the data seen so far (both offline data and online data) was generated. In the case of replay buffer, Ta corresponds to mixture distribution over all past policies. Typically, Ty is not known, especially for offline data, and must be estimated from the data itself. Many offline RL algorithms (Kumar et al.) 2019; Fujimoto et al. 2019 Siegel et al. 2020 explicitly fit parametric model to samples for the distribution Te via maximum likelihood estimation, where samples from Th are obtained simply by sampling uniformly from the data seen thus far: f.g = max*; Eggar [log a (a/s)). After estimating it., prior methods implement the constraint given in Equation in various ways, including penalties on the policy update (Kumar et al.) 2019 Wu et al.. 2020) or architecture choices for sampling actions for policy training (Fujimoto et al. 2019; Siegel ett al. 2020). As we will see next, the requirement for accurate estimation of म. makes these methods difficult to use with online fine-tuning.
3.3) Excessively Conservative Online Learning. While offline RL algorithms with constraints (Ku- mar et al.] 2019; Fujimoto et al.) 2019: Wu et al. 2020) perform well offline, they struggle to improve with fine-tuning, as shown the third plot in Figure 2 We see that the purely offline RL performance (at "OK" in Fig. 2 is much better than the standard off-policy methods shown in Section 3.2 However, with additional iterations of online fine-tuning, the performance increases very slowly (as seen from the slope of the BEAR curve in Fig 2 What causes this phenomenon?
This can be attributed to challenges in fitting an accurate behavior model data is collected online during fine-tuning. In the offline setting, behavior models must only be trained once via maximum likelihood, but in the online setting, the behavior model must be updated online to track incoming data. Training density models online (in the "streaming" setting) is a challenging research problem (Ramapuram et al., 2017), made more difficult by potentially complex multi-modal behavior distribution induced by the mixture of online and offfine data. To understand this, we plot the log likelihood of learned behavior models on the dataset during online and offline training for the HalfCheetah task. As we can see in the plot, the accuracy of the behavior models (log Ra on the y-axis) reduces during online fine-tuning, indicating that it is not fitting the new data well during online training. When the behavior models are inaccurate or unable to model new data well, constrained optimization becomes too conservative, resulting in limited improvement with fine-tuning. This analysis suggests that, in order address our problem setting, we require an off-policy RL algorithm that constrains the policy to prevent offline instability and error accumulation, but not so conservatively that it prevents online fine-tuning due to imperfect behavior modeling. Our proposed
4

---

Under review as a conference paper at ICLR 2021
algorithm, which discuss in the next section, accomplishes this by employing an implicit constraint, which does not require any explicit modeling of the behavior policy.
4 ADVANTAGE WEIGHTED ACTOR CRITIC: A SIMPLE ALGORITHM FOR FINE-TUNING FROM OFFLINE DATASETS
In this section, we will describe the advantage weighted actor-critic (AWAC) algorithm, which trains an off-policy critic and an actor with an implicil policy constraint. We will show AWAC mitigates the challenges outlined in Section 3 AWAC follows the design for actor-critic algorithms as described in Section 2 with a policy evaluation step to learn Q" and a policy improvement step to update m. AWAC uses off-policy temporal-Ce -aiioora-diiffeencee learning to estimate Q in the policy evaluation step, and a policy improvement update that is able to obtain the benefits of offline RI. algorithms at training from prior datasets, while avoiding the overly conservative behavior described in Section 3.3 We describe the policy improvement step AWAC below, and then summarize the entire algorith..
Policy improvement for AWAC proceeds by learning a policy that maximizes the value of the critic learned in the policy evaluation step via TD bootstrapping. If done naively, this can lead to the issues described in Section 3.3] but we can avoid the challenges of bootstrap error accumulation by restricting the policy distribution to stay close to the data observed thus far during the actor update, while maximizing the value of the critic. At iteration k AWAC therefore optimizes the policy to maximize the estimated Q-function Q (s, a) at every state, while constraining it to stay close to the actions observed in the data, similar to prior offline RL methods, though this constraint will be enforced differently. Note from the definition of the advantage in Section 2 that optimizing QT (s, is equivalent to optimizing A (s, a). We can therefore write this optimization as:
Th+1 arg max Bawn((1s] [A (s, a)] s.t. (I) c. elll
(6)
As we saw in Section 3.2
enforcing the constraint by incorporating an explicit learned behavior
model (Kumar et al.] 2019 Fujimoto et al.) 2019 Wu et al. 2020 (iigel et al. 2020) leads to poor fine-tuning performance. Instead, we enforce the constraint implicitly. without learning behavior model. We first derive the solution to the constrained optimization in Equation 6 to obtain a non- parametric closed form for the actor. This solution is then projected onto the parametric policy class without any explicit behavior model. The analytic solution to Equation can be obtained by enforcing the KKT conditions (Peters & Schaal 2007 Peters al. 2010 Peng al. 2019). The Lagrangian is:
C(T,, A) = (E)+ [AT (s. a)] + X(E DRL (T()sss)
and the closed form solution to this problem is x" (a]s) oc (als) exp (s, When using function approximators, such as deep neural networks as we do, we need to project the non- non-parametric solution into our policy space. For policy Te with parameters 0, this can be done by minimizing the KL divergence of my from the optimal non-parametric solution "² under the data distribution Pro (s):
arg min
E (agl()
[DKL (TT (-Is)][Te(- = arg min
I |- log To (-1s)]
(8)
Prg (s) ('(s))
Note that the parametric policy could be projected with either direction of KL divergence. Choosing the reverse KL results in explicit penalty methods (Wu et al. 2020) that rely on evaluating the density of learned behavior model. Instead, by using forward KL, we can compute the policy update by sampling directly from 3:
Oc+1 arg max
log To (als) exp
A AT (A
(8..a)
(l)s.)))))) ).
8,a~8
This actor update amounts to weighted maximum likelihood (i.e., supervised learning), where the targets are obtained by re- re-weighting the state-action pairs observed in the current dataset by the predicted advantages from the learned critic, withour explicitly learning any parametric behavior model, simply sampling (s. from the replay buffer B. See Appendix A.2 for more detailed derivation and Appendix A.3 for specific implementation details.
Avoiding explicit behavior modeling. Note that the update in Equation 9 completely avoids any modeling of the previously observed data 3 with a parametric model. By avoiding any explicit
5

---

Under review as a conference paper at ICLR 2021
learning of the behavior model AWAC is far less conservative than methods which fit model Rt explicitly, and better incorporates new data during online fine-tuning, as seen from our results in Section[6] This derivation is related to AWR (Peng et al. 2019), with the main difference that AWAC uses an off- off-policy Q-function Q" to estimate the advantage, which greatly improves efficiency and even final performance (see results in Section [.1). The update also resembles ABM-MPO, but ABM-MPO does require modeling the behavior policy which, as discussed in Section 3.3 can lead to poor fine-tuning. In Section 6.1 AWAC outperforms ABM-MPO on a range of challenging tasks.
During policy evaluation, we estimate the action-value (T a) for the current policy π,, as described in Section 2 We utilize a temporal difference learning scheme for policy evaluation (Haarnoja et al. 2018; Fujimoto et al. (0118), minimizing the Bellman error as described in Equation 2 This enables us to learn very efficiently from off- off-pooicy data. This is particularly important in problem setting to effectively use the offline dataset, and allows us to significantly outperform alternatives using Monte-Care evaluation or TD(A) to estimate returns (Peng et al. 2019)
Policy evaluation.
Algorithm summary. The full AWAC algorithm for of- fline RI. with online fine-tuning is summarized Algorithm
Algorithm 1 Advantage Weighted AC
1: Dataset D = {(s, a, r)))
I In practical implementation, we can parameterize the
2: Initialize buffer / D
actor and the critic by neural networks and perform SGD
3: Initialize To, Q
updates from Eqn. ald Eqn. 3 Specific details are pro- vided in Appendix A.3 AWAC ensures data efficiency with off- off-policy critic estimation via bootstrapping, and avoids offline bootstrap error with constrained actor update. By avoiding explicit modeling of the behavior policy, AWAC avoids overly conservative updates.
4: for iteration = 2, do
5: Sample batch (s. a, s', r) ~ 8
6: Update e according to Eqn.
7: Update according to Equ.
8:
if > num cileoooffffeseeeeeeee stteessssssssstee then
10: 8 Th...., TK ~ Pro (T)
9:
While AWAC is certainly quite related to several prior works,
note that there are key differences that make it particu- 11: end BU T
larly amenable to the problem setting we are considering offline RL with online fine-tuning, that none of the other
12: end for
methods are really able to tackle. As we show in our experimental analysis with direct comparisons to prior work, every one of the design decisions being made in this work are important for algo- rithm performance. As compared to AWR (Peng et al. 2019), AWAC uses TD bootstrapping for significantly more efficient and even asymptotically better performance. As compared to offlin RL techniques like ABM (Siegel et al. (020). MPO (Abdolmaleki et al., 2018). BEAR (Kumar et al.) 2019) or BCQ (Fujimoto et al. 2019) this work is able to avoid the need for any behavior modeling, thereby enabling the online fine-tuning part of the problem much better. As shown in Fig 3 when these seemingly ablations are made to AWAC, the algorithm performs significantly worse.
RELATED WORK
Off-policy RL algorithms are designed to reuse off-policy data during training, and have been studied
extensively (Konda & Tsitsiklis 2000; Degris et al. [2012] Mnih et al. [0016] Haarnoja al. 2018 Fujimoto et 20188 Bhhtnagar et al. 2009 Peters & Schaal 2008a Zhang et al. 2019 Wawrzynski 2009; Balduzzi & Ghifary) 2015). While standard off off-policy methods are able to benefit from including data seen during a training run, as we show in Section 3.2 they struggle when training from previously collected offline data from other policies, due to error accumulation with distribution shift (Fujimoto et al. 2019 Kumar et al. 2019). Offline RL methods aim to address this issue, often by constraining the actor updates to avoid excessive deviation from the data distribution (Lange cct al. 2012 Thomas & Brunskill 2016 Hallak et al. 2015 2016 Hallak & Mannor | Agarwal
et al. 2019: Kumar et al.[2019 Fujimoto et al.] 2019 Fakoor et al.) 2019: Nachum et al.) 2019]
Siegel et al.) 2020) Levine et al., 22200] Zhang et al., 2020p. One class of these methods utilize
importance sampling (Thomas & Brunskill| 2016 | et al. 2020, Nachum et al. 2019 Degris al. 2012 Jiang& & Li]20166 |Hallak& & Mannor 2017). Another class of methods perform onf reinforcement learning via dynamic programming. with an explicit constraint to prevent deviation
from the data distribution (Lange et al. 2012, Kumar et al. 2019 Fujimoto et al. 2019 Wu et al.
2020, Jaques et al. 2019). While these algorithms perform well in the purely offline settings, we show
Section 3.3 that such methods tend to be overly conservative, and therefore may not learn efficiently when fine-tuning with online data collection. In contrast, our algorithm AWAC is comparable to these algorithms for offline pre-training, but learns much more efficiently during subsequent fine-tuning.

---

Under review as a conference paper at ICLR 2021
pan-binary-vO
doer-bingr--00
relocate hinary-vi -0
1.0-
10-
1.0
0.6
0.6
Swcres Rate:
0.6
0.6
0.4
200K
4VIK
601K
500K
2008
d000 Timestops
KOOK
BCCK
3M Timestops
TM
Timeseegs
AWAC (Ours)
ABM
MARWIL
BEAR
UMAC
DAPG
SAC
SAC BC
Figure 3: Comparative evaluation on the dexterous manipulation tasks. These tasks are difficult due to their high action dimensionality and reward sparsity. We see that AWAC is able to Iearn these tasks with little online data collection required (100K samples R 16 minutes of equivalent real-world interaction time). Meanwhile, most prior methods are not able to solve the harder two tasks: door opening and object relocation.
Prior work has also considered the special case of learning from demonstration data. One class of algorithms initializes the policy via behavioral cloning from demonstrations, and then fine- tunetuues
with reinforcement learning (Peters & Schaal 2008b; Ljspeert et al.) [202 Theodorou al., 2010)
Kim et al. 2013) Rajeswaran et al. 2018: Gupta et al., 2019 Zhu et al. 2019p. Most such methods on-policy fine-tuning, which is less sample-effcient than off- policy methods that perform value function estimation. Other prior works have incorporated demonstration data into the replay buffer using off--poiccy RL methods (Večerík et al., 2017 Nair et al. 2017). We show in Section 3.2 that these strategies can result in large dip in performance during online fine-tuning, due to the inability to pre-train an effective value function from offline data. In contrast, our work shows that using supervised learning style policy updates can allow for better bootstrapping from demonstrations as compared to Večerík et al. (2017) and aair et al. (2017).
Our method builds on algorithms that implement a maximum likelihood objective for the actor, formulation of RL (Peters & Schaal 2007; Neumann & based on an Theodorou et al. 2010 Peters al., 2010 Peng al. 2019 Abdolmaleki al. 2018)
Peters 2008
Wang et al. 2018). Most closely related to our method in this respect are the algorithms proposed by Peng al. (2019) (AWR) and Siegel et al. (2020) (ABM). Unlike AWR, which estimates the
value function of the behavior policy, aa via Monte-Carlo estimation or TD- X, our algorithm estimates the Q-function of the current policy Q" via bootstrapping, enabling much more efficient learning, as shown in our experiments. Unlike ABM, our method does not require learning a separate function approximator to model the behavior policy TT. and instead directly samples the dataset. As we discussed in Section 3.3, modeling F. can be a major challenge for online fine-tuning. While these distinctions may seem somewhat subtle, they are important and we show in our experiments that they result in a large difference in algorithm performance. Finally, our work goes beyond the analysis in prior work, by studying the issues associated with pre-training and fine-tuning in Section 3 Concurrently to our work, WWan et al. (2020) proposed critic regularized regression for offline RL. which uses off- ofolcc Q-learning and an equivalent policy update. In contrast to this concurrent work, we specifically study the offline pretraining online fine-tuning problem, analyze why other methods are ineffective in this setting, and show that our approach achieves substantially better results.
6 EXPERIMENTAL EVALUATION
In our experiments, we first compare our method against prior methods in the offline training and fine-tuning setting. We show that we can learn difficult, high high-dimensional, sparse reward dexterous manipulation problems from human demonstrations and off- data. We then evaluate our method with suboptimal prior data generated by a random controller. Finally, we study why prior methods struggle in this setting by analyzing their performance on benchmark MuJoCo tasks, and conduct further experiments to understand where the difficulty lies (also shown in Section 3).
6.1) Comparative Evaluation Learning From Prior Data. We aim to study tasks representative of the difficulties of real-world robot learning, where offline learning and online fine-tuning most
7

---

Under review as a conference paper at ICLR 2021
relevant. We begin our analysis with set of challenging sparse reward dexterous manipulation tasks proposed by Rajeswaran et al. (2018). These tasks involve complex manipulation skills using a 28-DoF five-fingered hand in the MuloCo simulator (ooorovv et al. 2012) shown in Figure B in-hand rotation of a pen, opening a door by unlatching the handle, and picking up sphere and relocating it to target location. These environments exhibit many challenges: high dimensional action spaces, complex manipulation physies with many intermittent contacts, and randomized hand and object positions. The reward functions in these environments are binary 0-1 rewards for task
completion. Rajeswaran et al. (2018) provide 25 human demonstrations for each task, which are not fully optimal but do solve the task. Since this dataset is small, we generated another 500 trajectories of interaction data by constructing a behavioral cloned policy, and then sampling from this policy.
First, we compare our method on these dexterous manipulation tasks against prior methods for off off-policy learning, offline learning, and bootstrapping from demonstrations. Specific implementation details are discussed in Appendix [A.3] The results are shown in Fig. 3 Our method is able to leverage the prior data to quickly attain good performance, and the efficient off ofl-policy actor-critic component of our approach fine-tunes much more quickly than demonstration augmented policy gradient (DAPG), the method proposed by Rajessaaan et al. (2018). For example, our method solves the pen task in 120K timesteps, the equívalent of just 20 minutes of online interaction. While the baseline comparisons and ablations are able to make some amount of progress on the pen task, alternative off-policy RI. and offline RL algorithms are largely unable to solve the door and relocate task in the fram- considered. We find that the design decisions to use off-policy critic estimation allow AWAC to significantly outperform AWR (Peng et al. 2019) while the implicit behavior modeling allows AWAC to significantly outperform ABM (Siegel et al. 2020), although ABM does make some progress. Rajeswaran et al. (2018) show that DAPG can solve variants of these tasks with more well-shaped rewards, but still requires considerably more samples.
Additionally, we evaluated all methods on the Gym MuloCo locomotion benchmarks, similarly providing demonstrations as offline data. Due to space constraints, the results plots for these experiments included in Appendix [A.]] These tasks are substantially casier than the sparse reward manipulation tasks described above, and a number of prior methods also perform well. However, our method matches exceeds the best prior method in all cases, whereas other single prior method attains good performance on all of the tasks.
6.2) Tuni-TT from Random Policy Data. An advantage of using off- off-policy RL for reinforcement learning is that we can also incorporate suboptimal data, rather than demonstrations. In this experiment, we evaluate on a simulated tabletop pushing environment with Sawyer robot pictured in Fig 3 and described further in Appendix [A.4] To study
Learning From Random Dala
1.0
0.8
0.6
the potential to learn from suboptimal data, we use an off ofl-po dataset of 500 trajectories generated by random process. The task is to push an object target location in 40cm 20cm goal space. The results are shown in Figure We see that while many methods begin at the same initial performance, AWAC learns the fastest online and is actually able to make use of the offline dataset effectively.
0
20%
4GK
EBC
S0k
1000
Timestepa
AWACC
BRAC
SAC
ABM
Comparison of fine-tuning from initial dataset of suboptimal data on a Sawyer robot pushing task.
Figure 4:
DISCUSSION AND FUTURE WORK
We have discussed in detail the challenges existing RL methods face when fine-tuning from prior datasets, and proposed an algorithm, AWAC, that is effeective in this setting. The key insight in AWAC is that enforcing a policy update constraint implicitly on actor-critic methods results in a stable learning algorithm amenable for off-policy learning. With an informative action-value estimate, the policy is weighted towards high-advantage actions in the data, resulting in policy improvement without conservative updates. A direction of future work we plan to pursue is applying AWAC to solve difficult robotic tasks in the real world. More than just speeding up individual runs, incorporating prior data into the learning process enables continuously accumulating data by saving environment interactions of the robot- for instance, runs of RL with varying hyperparameters. We hope that this enables a wider array of robotic applications than previously possible.
Rajeswaran et al. 2018)) use combination of task completion factors the sparse reward. For instance, in the door task, the sparse reward as function of the door position was =001a>1.35 81 81..1.0 21..2-1.2 0.1|/d 1.57||2. We only use the success measure r 1a>1,4, which is substantially more difficult.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a Posteriori Policy Optimisation. International Conference on Learning Representations (ICLR), pp. 1-19, 2018.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An Optimistic Perspective on Offline Reinforcement Learning. In International Conference Machine Learning (ICML), 2019.
Christopher G Atkeson and Stefan Schaal. Robot Learning From Demonstration. In International Conference on Machine Learning (ICML), 1997.
David Balduzzi and Muhammad Ghifary. Compatible value gradients for reinforcement learning of continuous deep policies. CoRR, 0bs//5509000003 2015.
Shalabh Bhatnagar, Richard S. Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Autom., 45(1 1):2471-2482, 2009, doi: 00.1016f.automaatcc.200..0 200.000.08.0...8
Thomas Degris, Martha White, and Richard S. Sutton. Off-Policy Actor-Critic. In International Conference Machine Learning (ICML), 2012.
Jacob Devlin, Ming- Mei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of
Deep Bidirectional Transformers for Language Understanding. In Association for Compuational Linguistics (ACL), 2019.
Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3O: Policy-on Policy-off Policy Opti- mization. In Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning. 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. International Conference on Machine Learning (ICML), 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. In International Conference Machine Learning (ICML), 2019.
Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. CoRR, abs/1802.05313, 2018.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay Policy Learning: Solving Hong-Hoiizon Tasks via Imitation and Reinforcement Learning. In Conference Robot Learning (CoRL), 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. International Conference on Machine Learning, 2018.
Assaf Hallak and Shie Mannor. Consistent On-Line Off-Policy Evaluation. In International Confer- ence on Machine Learning (ICML), 2017.
Assaf Hallak, Francois Schnitzler, Timothy Mann, and Shie Mannor. Off-policy Model-based Learning under Unknown Factored Dynamics. International Conference on Machine Learning (ICML), 2015.
Assaf Hallak, Aviv Tamar, Rémi Munos, and Shie Mannor. Generalized Emphatic Temporal Dif- ference Learning: Bias- Variance Analysis. In Association for the Advancement of Artificial Intelligence (AAAI), 2016.
Auke Jan ljspeert, Jun Nakanishi, and Stefan Schaal. Learning Attractor Landscapes for Learning Motor Primitives. In Advances in Neural Information Processing Systems (NIPS), pp. 1547-1554, 2002. ISBN 1049-5258.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Ágata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, 006119977004566 2019.

---

Under review as a conference paper at ICLR 2021
Nan Jiang and Lihong Li. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. In International Conference on Machine Learning (ICML), 2016.
Beomjoon Kim, Amir-Massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from Limited
Demonstrations. In Advances in Neural Information Processing Systems (NIPS), 2013.
Vijay R Konda and John N Tsitsiklis. Actor-Critic Algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2000.
Alex Krizhevsky, Nya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems (NIPS), pp. 1097-1105, 2012.
Aviral Kumar. Justin Fu, George Tucker, and Sergey Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. In Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Marco Wiering and Martijn van Otterlo (eds.), Reinforcement Learning, volume 12 of Adaptation, Learn- ing, and Optimization, pp. 45-73. Springer, 2012. doi: 10. 10.1107/978-3-11 0-119997-8---------- 0-2--422224574 27645-31_2
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Technical report, 2020.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016. ISBN 0-7803-3213-X. doi: 10.1613/jair.301.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P Lillicrap, David Silver, and Koray Kavukeuoglu. Asynchronous Methods for Deep Reinforcement Learning. In International Conference on Machine Learning (ICML), 2016.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, Sergey Levine, Dian Chen, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining Self- Supervised Learning and Imitation for Vision-Based Rope Manipulation. In IEEE International Conference on Robotics and Automation (ICRA), 2017. ISBN 9781 7781509046331. doi: 10.1109/ ICRA.2017.79892477
Ashvin Nair, Bob Megrew, Marcin Andrychowicz. Wojeiech Zaremba, and Pieter Abbeel. Over- coming Exploration in Reinforcement Learning with Demonstrations. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
Gerhard Neumann and Jan Peters. Fitted Q-iteration by Advantage Weighted Regression. In Advances in Neural Information Processing Systems (NeurIPS), 2008.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- Weightedd Regression: Simple and Scalable Off-Policy Reinforcement Learning. 2019.
Jan Peters and Stefan Schaal. Reinforcement Learning by Reward-weighted Regression for Opera- tional Space Control. In International Conference Machine Learning, 2007.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputing, 7107-9):1180-111 2008a. doi:
10.1016/ 10.10166j. www.:./ewcm..00..... 11.02..
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21(4):682 697, 2008b. ISSN 08936080. doi: 10. www.:eeeee...002.....
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative Entropy Policy Search. In AAAI Conference Artificial Intelligence, pp. 1607-1612, 2010.
10

---

Under review as a conference paper at ICLR 2021
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Robotics: Science and Systems, 2018.
Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong Generative Modeling.
Neurocomputing, 2017.
Stefan Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems (NeurIPS), number 9, pp. 1040-1046, 1997. ISBN 1558604863, doi: 10. r0016//..obo0.204..0..
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. 1998.
Evangelos A Theodorou, Jonas Buchli, and Stefan Schaal. A Generalized Path Integral Control Approach to Reinforcement Learning- Journal of Muchine Learning Reseurch (JMLR), 11:3137- 1181,2010
Philip S. Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2139 2148. JMLR.org. 2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5026-50000000 5033, 2012. ISBN 9781467317375. doi:
Matej Večerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards. CoRR, abs/1707.0, 2017.
Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, and Tong Zhang. Exponentially Weighted Imitation Learning for Batched Historical Data. In Neural Information Processing Systems (NeurIPS), 2018.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando De Freitas. Critic Regularized Regression. 2020.
Pawel Wawrzynski. Real-time reinforcement learning by sequential actor-critics and experience replay. Neural Networks, 22(10):1844-1499 2009. doi: 10. 10.10166).nuunet220099 200.000000
Ronald J Williams. Simple Statistical Gradient-FFllooiing Algorithms for Connectionist Reinforce- ment Learning. Machine Learning, pp. 229-256, 1992.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior Regularized Offline Reinforcement Learning. 2020.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. GenDICE: Generalized Offline Estimation of Stationary Values. In International Conference on Learning Representations (ICLR), 2020.
Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Generalized off-policy actor-critic. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché Aauchéé-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 2001-2011. Curran Associates, Inc., 2019.
Allan Zhou, Eric Jang, Daniel Kappler, Alexander Herzog, Mohi Khansari, Paul Wohlhart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and reward. CoRR, Abs//996.00335 2019.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous Ma- nipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost. In Proceedings IEEE International Conference Robotics and Automation, volume 2019-May, pp. 3651-3657. Institute Electrical and Electronics Engineers Inc., 2019.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 GYM BENCHMARK RESULTS FROM PRIOR DATA
In this section, we provide a comparative evaluation on MuloCo benchmark tasks for analysis. These tasks are simpler, with dense rewards and relatively lower action and observation dimensionality. Thus, many prior methods can make good progress on these tasks. These experiments allow us to understand more precisely which design decisions are crucial. For each task, we collect 15 demonstration trajectories using a pre-trained expert on each task, and 100 trajectories of off-policy data by rolling out behavioral cloned policy trained on the demonstrations. The same data is made available to all methods. The results are presented in Figure AWAC is consistently the best or on par with the perf-perg method. No other single method consistently attains the best results- on HalfCheetah, SAC + BC and BRAC are competitive, while on Ant-v2 ABM is competitive with AWAC We summarize the results according to the challenges in Section 3
Data efficiency. The three methods that do not estimate QT are DAPG (Abdolmaleki al. 2018). AWR Peng et al., 2019). and MARWIL (Wang et al. 2018). Across all three tasks, we see that these methods are somewhat worse offline than the best performing offline methods, and exhibit steady but very slow improvement during fine-tuning In robotics, data efficiency is vital, so these algorithms not good candidates for practical real-world applications.
Bootstrap error in offline learning. For SAC (Haarnoja et al. 2018), across all three tasks, we see that the offline performance at epoch 0 is generally poor. Due to the data in the replay buffer, SAC with prior data does learn faster than from scratch, but AWAC faster to solve the tasks general. SAC with additional data in the replay buffer is similar to the approach proposed by Večerik et al. (2017). SAC+BC reproduces Nair al. (2018) but uses SAC instead of DDPG (Lillierap et al. 2016) as the underlying RL. algorithm. We find that these algorithms exhibit a characteristic dip at the start of learning. Although this dip is only present in the early part of the learning curve, a poor initial policy and lack of steady policy improvement can be a safety concern and a significant hindrance in real-world applications. Moreover, recall that in the more difficult dextrous manipulation tasks, these algorithms do not show any significant learning.
Conservative online learning. Finally, we consider conservative offline algorithms: ABM (Siegel eet al., 2020), BEAR (Kumar et al., 2019), and BRAC (Wu et al. 2020). We found that BRAC performs similarly to SAC for working hyperparameters. BEAR trains well offline Walker2d, BEAR significantly outperforms prior methods before online experience. However, online improvement is slow for BEAR and the final performance across all three tasks is much lower than AWAC. The closest in performance to our method is ABM, which is comparable on Ant-v2, but much slower other domains.
Ant and
HalfChoetah-v2
Ant- v2
Walker2d-v2
10000
6000
5000
Avenge Return
4000
2000
2000
1000
2000
-2000
-1000
0K
100K
200K
300K
400K
500K
OK
100K
200K
300K
400K
500K
0K
100K
200K
300K
400K
500K
Timesteps
Timesteps
Timesteps
(Oum)
ABM
MARWIL
DAPG
SAC
Figure 5: Comparison of our method and prior methods on standard MuJoCo benchmark tasks. These tasks are much casier than the dexterous manipulation tasks, and allow us to better inspect the performance of methods in the setting of offline pretraining followed by online fine-tuning. SAC+BC and BRAC perform on par with our method on the HalfCheetah task. and ABM performs on par with method on the Ant task, while our method outperforms all others on the Walker2D task. Our method matches exceeds the hest prior method in all cases, whereas no other single prior method attains good performance on of the tasks.
12

---

Under review as a conference paper at ICLR 2021
A.2 ALGORITHM DERIVATION DETAILS
The full optimization problem we solve, given the previous off-policy advantage estimate * and buffer distribution Tg, is given below:
#4+1 = arg max Bnwr(- (1x) a))
(10)
wEll
s.t. (TL((((-ss)) (KL(((((s)((=o((1s) S
(11)
J. (als))da=
(12)
Our derivation follows Peters et al. (0110) and Peng et al. (011). The analytic solution for the constrained optimization problem above can be obtained by enforcing the KKT conditions. The Lagrangian is:
[(m, a, = E a (-)) (1a) (ATH
(s.a)) a)] + A(e- DKL(T(()))l(<((ss)) a(1 m(als)da). (13)
Differentiating with respect to т gives:
ac A (s, a) Alog (als)) + A log (als))
(14)
Setting उā
8c to zero and solving for æ gives the closed form solution to this problem:
x (als) = Z(s) us (a(s))) exp () (s.
(15)
Next, we project the solution into the space of parametric policies. For a policy тя with parameters this can be done by minimizing the KL divergence of n. from the optimal non-parametric solution T under the data distribution Px, (s):
arg min
E Prg (8)
[KLL (T (-Is)][ = arg min Prg (8)
E
E [- log To (-1s)] (5*(1(8)
(16)
Note that in the projection step, the parametric policy could be projected with either direction of KL divergence. However, choosing the reverse KL direction has a key advantage: it allows us to optimize 0 as maximum likelihood problem with an expectation over data 5, B, rather than sampling actions from the policy that may be out of distribution for the Q function. In our experiments we show that this decision is vital for stable off-policy learning.
Furthermore, assume discrete policies with minimum probably density of T& 2 ag. Then the upper bound:
DKL(T [[TT) I Drv(TT =T)²
(17)
DTI.(Tol|TT)
(18)
holds by the Pinsker's inequality, where Drv denotes the total variation distance between distributions. Thus minimizing the reverse KL also bounds the forward KL. Note that we can control the minimum if desired by applying Laplace smoothing to the policy.
A.3 IMPLEMENTATION DETAILS
We implement the algorithm building on top of twin soft actor-critic (Haamoja al. 2018), which incorporates the twin Q- D-function architecture from twin delayed deep deterministic policy gradient (TD3) from from/Fujini Fujimoto al. (2018). All off-policy algorithm comparisons (SAC, BRAC, MPO, ABM, BEAR) are implemented from the same skeleton. The base hyperparameters are given in Table 2 The policy update replaced with:
exp (A
0x+1 arg max
E 8,00
log (als)
(19)
Z(s)
13

---

Under review as a conference paper at ICLR 2021
Similar to advantage weight regression (Peng al.) 2019) and other prior work (Neumann & Peters 2008; Wang et al. 2018 Siegel et al. 2020), we disregard the per-state normalizing constant (s)) Ja To a exp (3+ A (s, a)) da (=)) [A* (s, a)]. We did experiment with estimating this expectation per batch el- ement with K 10 samples, but found that this generally made performance worse, perhaps because errors in the estimation of (s)) caused more harm than the benefit the method derived from estimating this value. We report success rate results for variants of our method with and without Z (s) estimation in Table [
Use ((s)
Omit 2(s)
Env
pen
84%
98%
door
0%
95%
relocate 0%
54%
Table 1: Success rates after on- line fine-tuning (after 800K steps for pen, door and 4M steps for relocate) using AWAC with and without Z(s) weight. These re- sults show that although we can estimate (()), weighting by ((s) actually results in worse perfor- mance.
While prior work (Nuumann & Peters 2008 Wang et al. 2018 Peng et al. 2019) has generally ignored the omission of ((s) with- out any specitic justification, it is possible to bound this value both above and below using the Cauchy-Schwarz Schwarz and reverse Cauchy- Schwarz (Polya-Szego) inequalities, as follows. Let f(a) = π (alss) and g(a) exp(A(s A). Note f(a) for stochastic policies
Z(s) ʃa f(a)g(a)da S v.J. fla) g(a) C1. To apply Volya-Szego, let my and mg be the minimum of f and 9 respectively and My, M₁ be the
and g(a) > 0. By
maximum. Then (s) 2 2(1 mymy
M + M,M,, M,,,)--C = C2. We therefore have C1 S (()) < Cz.
though the bounds are generally not tight.
further, more intuitive argument for why omitting (s) may be harmless in practice comes from observing that this normalizing factor only affects the relative weight of different states in the training objective, not different actions. The state distribution in 3 already differs from the distribution over states that will be visited by TT, and therefore preserving this state distribution is likely to be of limited utility to downstream policy performance. Indeed, we would expect that sufficiently expressive policies would be less affected by small to moderate variability in the state weights. On the other hand, inaccurate estimates (s) may throw off the training objective by increasing variance, similar to the effect of degenerate importance weights.
The Lagrange multiplier À is treated as a hyperparameter in our method. In this work we use À 0.3 for the manipulation environments and X = 1.0 for the MuJoCo benchmark environments. One could adaptively learn À with a dual gradient descent procedure, but this would require access to Ty.
As rewards for the dextrous manipulation environments are non-positive, we clamp the Q value for these experiments be at most zero. We find this stabilizes training slightly.
A.4 ENVIRONMENT ENVIRONMENT-SPECCIIC DETAILS
We evaluate our method on three domains: dexterous manipulation environments, Sawyer manipu- lation environments, and MuJoCo benchmark environments. In the following sections we describe specific details.
A.4.1 DEXTEROUS MANIPULATION ENVIRONMENTS
These environments are modified from those proposed by Rajeswaran et al. 2018
The task is to spin a pen into a given orientation. The action dimension is 24 and the observation dimension is 45, Let the position and orientation of the pen be denoted by x, and T. respectively, and the desired position and orientation be denoted by dp and d, respectively. The reward function is r 1 -lz,-d,1go.ura 150.95 1. In Rajeswaran et al. (2018). the episode was terminated when the pen fell out of the hand; we did not include this early termination condition.
pen-binary-v0.
door-binary-v0. The task is to open door, which requires first twisting a latch. The action dimension 28 and the observation dimension is 39. Let d denote the angle of the door. The reward function is 12>14 - 1.
14

---

Under review as a conference paper at ICLR 2021
Alpper-parameter
Value
Training Batches Per Timestep
1
Exploration Noise
None (stochastic policy)
RL Batch Size
1024
Discount Factor
0.99
Reward Scaling
1
Replay Buffer Size
1000000
Number of pretraining steps
25000
Policy Hidden Sizes
[256, 256, 256, 256]
Policy Hidden Activation
ReLU
Policy Weight Decay
10
Policy Learning Rate
3 10
Q Hidden Sizes
[256, 256, 256, 256)
Hidden Activation
ReLU
Q Weight Decay
0
Q Learning Rate
3 x 10 4
Target Network
5 x 10-3
Table 2: Hyper-parameters used for RI. experiments.
relocate-binary-v0, The task is to relocate an object to goal location. The action dimension is 30 and the observation dimension is 39. Let xp denote the object position and dp denote the desired position. The reward is r = 1| 50.1
A.4.2 SAWYER MANIPULATION ENVIRONMENT
SawyerPush-vO. This environment is included in the Multiworld library. The task is to push a puck to goal position in 40em x 20em, and the reward function is the negative distance between the puck and goal position. When using this environment, we use hindsight experience replay for goal-conditioned reinforcement learning. The random dataset for prior data was collected by rolling out an Ornstein- Uhlenbeck process with f 0.15 and 0.3.
A.4.3 OFF-POLICY DATA PERFORMANCE
Env
Expert
BC (1)
BC (2)
The performances of the expert data, behavior cloning (BC) on the expert data (1), and BC on the combined eX data (2) are included in Table 3 For Gym benchmarks we report average return, and expert data is collected by a trained SAC policy. For dextrous manipulation tasks we report the success rate, and the expert data consists of human demonstrations provided by Rajeswaran et al. (0018)
cheetah
9962
2507
4524
walker
5062
2040
1701
5207
687
1704
pen
0.73
0.76
door relocate
0.10
0.00
0.02
0.01
A.5 BASELINE IMPLEMENTATION DETAILS
Table Performance of the pollcy data for each environment. BC (1) indicates BC on the expert data, while BC (2) indicates BC on the combined expert+BC data used off-policy data for pretraining.
We used public implementations of prior methods (DAPG, AWR) when available. We implemented the remaining algo- rithms in our framework, which also allows understand the effects of changing individual components of the method. In the section, describe the implementation details. The full overview of algorithms is given in
Figure 6
15

---

Under review as a conference paper at ICLR 2021
Name
Q
Policy Objective
#t?
Constraint
SAC
Q"
(xa(Fell
No
None
SAC + BC
Q"
Mixed
No
None
BCQ
Q"
DEL (xo(F)))))
Yes
Support (e
BEAR
Q
DEL (xo(Foll
Yes
Support (MMD)
AWR
Qa
DEL (Q]
No
Implicit
MPO
Q"
De (Q)
Yes"
Prior
ABM-MPO
Q"
DxI(Q]
Yes
Learned Prior
DAPG
No
None
BRAC
q"
DRI(FollQ)
Yes
Explicit KL penalty
AWAC (Ours)
Q"
(QI(Q1
No
Implicit
Figure 6: Comparison prior algorithms that can incorporate prior datasets. See section A.5 for specific implementation details. We argue that avoiding estimating ft (i.e., fg is "No") is important when learning with complex datasets that include experience from multiple policies, in the case of online fine-tuning, and maintaining constraint of some sort essential for offline training. At the same time, sample-efficient learning requires using Q" for the critic. Our algorithm is the only that fulfills all of these requirements.
Behavior Cloning (BC), This method learns a policy with supervised learning on demonstration data.
Soft Actor Critic (SAC). Using the soft actor critic algorithm from (Haarnoja et al. 2018), we follow the exact same procedure our method in order to incorporate prior data, initializing the policy with behavior cloning on demonstrations and adding all prior data to the replay buffer.
Behavior Regularized Actor Critic (BRAC). We implement BRAC as described in (Wu et al. 2020) by adding policy regularization log (als)) where Ti is behavior policy trained with supervised learning on the replay buffer. We add all prior data to the replay buffer before online training.
Advantage Weighted Regression (AWR). Using the advantage weighted regression algorithm from (Peng et al., 2019), we add all prior data to the replay buffer before online training. We use the implementation provided by Peng et al. (2019), with the key difference from our method being that AWR uses TD(A) on the replay buffer for policy evaluation.
Monotonic Advantage Re- Re-Weighted Imitation Learning (MARWIL.. Monotonic advantage re- weighted imitation learning was proposed by Wang et al. (018 for offline imitation learning. MARWIL was not demonstrated in online RL settings, but we evaluate it for offline pretraining followed by online fine-tuning as we do other offline algorithms. Although derived differently, MARWIL and AWR are similar algorithms and only differ in value estimation: MARWIL uses the on-policy single-path advantage estimate A(s, a) @T*(s,
(**(s) instead of TD(A) in AWR. Thus, we implement MARWIL by modifying the implementation of AWR.
Maximum Posteriori Policy Optimization (MPO). We evaluate the MPO algorithm presented by Abddllalekii et al. (2018). Due to a public implementation being unavailable, we modify our algorithm to be as close MPO possible. In particular, change the policy update in Advantage Weighted Actor Critic to be:
0, k argmax max E EEwDD log ne, (als) expl Q (s. (
(20)
0:
Note that in MPO, actions for the update are sampled from the policy and the Q-function is used instead of advantage for weights. We failed to offline or online improvement with this implemen- tation in most environments, so we omit this comparison favor of ABM.
Advantage-Weighted Behavior Model (ABM). We evaluate ABM, the method developed in Siegel 2020). As with MPO, we modify our method to implement ABM, as there no public
16

---

Under review as a conference paper at ICLR 2021
implementation of the method. ABM first trains advantage model (alm)
0 abm
arg
E ENDD
log Te (aol)s) (R()) (s))
(21)
0.
where f is an increasing non -egateete function, chosen to be = -+. In place of an advantage computed by empirical returns R(TE:N)- -s) we use the advantage estimate computed per transition by the Q value (s,, a) - V(s). This favorable for running ABM online, as computing R(TE:)) (s) is similar to AWR, which shows slow online improvement. We then use the policy update:
0: arg max Es-Daarmanlals) log no, (als) exp (,100)s.a)- -V
(22)
9.
Additionally, for this method, actions for the update are sampled from a behavior policy trained to match the replay buffer and the value function computed VT(s) (s, a) s.t. a
Demonstration Augmented Policy Gradient (DAPG). We directly utilize the code provided in (Rajeswaran al. 2018) to compare against our method. Since DAPG is on-policy method, only provide the demonstration data to the DAPG code to bootstrap the initial policy from.
Bootstrapping Error Accumulation Reduction (BEAR). We utilize the implementation of BEAR provided rikit We provide the demonstration and off-policy data to the method together. Since the original method only involved training offline, we modify the algorithm to include an online training phase. In general we found that the MMD constraint in the method was too conservative. As result, in order to obtain the results displayed in our paper, we swept the MMD threshold value and chose the one with the best final performance after offline training with offline fine-tuning.
17

---

Under review as a conference paper at ICLR 2021
A.6 EXTRA BASELINE COMPARISONS (CQL, ALGAEDICE)
In this section, we add comparisons to constrained Q-learning (CQL) (Kumar et al., 2020) and AlgaeDICE (Nachum et al. 2019). For CQL, we use the authors' implementation, modified for additionally online- finetuning instead of only offline training. For AlgaeDICE, we use the publicly available implementation, modified to load prior data and perform 25K pretraining steps before online RL. The results are presented in Figure 7
cux uuy_k-Aat-v2
env. www.jddddor--iaasaee bi
5010
800
100
AMACC
COL
1400
CalllalCheeeaal v2
CNV k--ean-inarr--it
Meaa
COL
CKT
14:00
20:00
14000
23:00
MI0X
30000
20000
30000
6000
X000
24-4
014
COL
0.1
CoAMertr Mon
AVFAC
016
0.4
610
500
Figure 7: Comparison of our method (AWAC) with CQL and AlgaeDICE. CQL and AWAC perform similarly offline, but CQL does improve when fine-tuning online. AlgaeDICE does not perform well for offline pretraining.
18

---

Under review as a conference paper at ICLR 2021
A.7 ONLINE TUNNIG FROM D4RL
In this experiment, we evaluate the performance of varied data quality (random, medium, medium- expert, and expert) datasets included in D4RL (Fu et al. 2020). dataset intended for offline RL. The results are obtained by first by training offline and then fine-tuning online on each setting for 500,000 additional steps. The performance of BEAR (Kumar al. 2019) is attached as reference. We attempted to fine-tune BEAR online using same protocol as AWAC but performance did not improve and often decreased; thus we report the offline performance. All performances are scaled to 100, where 0 the average returns of random policy and 100 is the average returns of an expert policy (obtained by training online with SAC), as is standard for D4RL.
The results are presented in Figure First, observe that AWAC (offline) is competitive with BEAR, commonly used offline RL algorithm. Then, AWAC is able to make progress in solving the tasks with online fine-tuning, even when initialized from random data "medium" quality data, as shown by the performance the best performing settings, AWAC (online) is significantly better than BEAR; it reasonable that AWAC excels in the lower-quality data regime because there is more for online improvement, while both offline RL methods often start at high performance when initialized from higher-quality data.
AWAC (online). In almost all settings, AWAC (online) is tied with BEAR. In four of the six lower quality (random medium) data
AWAC
AWAC
BEAR
(offline)
(online)
HalfCheetah
random
2.2
52.9
25.5
medium
37.4
41.1
38.6
medium-cxpert
36.8
41.0
51.7
expert
78.5
105.6
108.2
Hopper
random
9.6
62.8
9.5
medium
72.0
91.0
47.6
medium-exper expprt
80.9
111.9
4.0
expert
85.2
111.8
110.3
Walker2D
random
5.1
11.7
6.7
medium
30.1
79.1
33.2
medium-expert
42.7
78.3
10.8
expert
57.0
103.0
106.1
Figure 8: Comparison our method (AWAC) fine-tuning on varying data quality datasets D4RL [Fu 2020 AWAC is able to improve its offline performance by further tin-ttning online.
19


Reviews:
[{"review_id": "qhmsSRqSvt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nIn this paper, the authors intend to accelerate on-line reinforcement learning with off-line datasets. To achieve this goal, they propose an algorithm called advantage weighted actor-critic (AWAC), which uses an implicit constraint to reduce accumulated bootstrapping error when doing off-line training and reduce the conservation when doing on-line fine-tuning. The experiments show that the proposed method can learn difficult, high-dimensional, sparse reward dexterous manipulation problems from human demonstrations and off-policy data.\n\nPros:\n1. The organization is clear and related work is sufficient. This paper introduces the off-line training and on-line fine-tuning problem and its challenge by gradually summarize the existing studies, and with sufficient investigation of existing work, it is easy for readers to dive into the problem\n2. The proposed method is easy to implement. Thanks to the implicit formulation of the constraint, the proposed AWAC algorithm can sample directly from off-line datasets without fitting a parametric model. This strategy is especially advantageous when the dimensionality is high. Besides, since this algorithm does not need the parametric model any more, it is more friendly for users who are not familiar with the off-line dataset.\n3. The details of experimental settings are provided, and thus there should be no issues with the repeatability of the experiments. \n\nCons:\n1. It is doubtful for the contribution on online fine-tuning. The constraint is designed for off-line training because off-line training has the problem of bootstrapping error accumulation. There is no judgment or proof that online training also suffers from the bootstrapping problem. How can the authors claim their contribution on on-line learning with the constraint?\n2. The technical depth is not high. It seems the only contribution of this paper is to combine the constraint with an unparameterized strategy. But I would like to say that this is not a big problem, as many influential reinforcement learning algorithms are straightforward but have very significant improvement, e.g., DDQN and PPO.\n3. The writing needs polish. E.g., in Equation (4), it should be 'max' instead of 'argmax', and there exists non-English spelling in Section 3.1.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qFckh0XAqP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an improved actor-critic algorithm for learning from offline data while requiring minimum interactions during online learning. The approach implicitly updates a low-variance policy without the need to estimate the sampling policy. The empirical study shows that AWAC leads to a faster convergence rate compared to the competitors.\n\nOverall, the paper provides an incremental contribution over the existing works for off-policy RL problems. However, the proposed algorithm demonstrates some practical benefits in simulated scenarios and is analyzes from different perspectives. Hence, I vote for accepting, given the following concerns are addressed in the rebuttal period.\n\nPros:\n- Interesting adaption of algorithms to effectively use the offline data in online RL scenarios which is independent from the sampling policy\n- Ensuring the policy to stay close to the observed data without directly estimating the sampling policy\n- Nicely motivated with helpful examples\n- Clarifies differences with the related work\n\nComments:\n- The presentation of the paper could be improved. There are some inconsistencies in the text. At the beginning the approach is introduced to be based on dynamic programming and then turns into temporal difference learning, which is misleading. In addition, offline pre-training and online fine-tuning is not quite accurate as the algorithm 1 is using the mix of data in one phase which does not involve pre-training, or is it actually in two phases?\n\n- The paper seems to be an incremental improvement/combination of several recent works mentioned in the paper.\n\n- The experiments are all on simulated environments. It would be interesting to see how the approach performs on real-world applications, and with some other evaluation metric than convergence rate. \n\n- Demonstration data is not always as complete as mentioned in the paper. Expert demonstrations that are used for imitation learning, normally lack the reward values and in the form of (s,a) pairs only. \n\n\nMinor:\n-Typo in section 3.2: is policy is pre-trained\n- Plot 4 in Fig. 2 not addressed in section 3.3", "extra_scores": {}, "extra_text": {}}, {"review_id": "keh7ZefetV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies challenges of offline RL with online fine-tuning and proposes an off-policy actor-critic method to address these challenges. The proposed method uses a supervised learning style to update the model parameters and avoids the behavior model estimation.  Empirical results show that the proposed method provides rapid learning with prior demonstration data and online experience.\n\nI have a major concern about the novelty of this paper. The major component of the proposed AWAC method, updating the model parameter using supervised learning, is exactly the same as AWR (Peng et. al., 2019). One minor difference seems that AWAC uses an off-policy policy evaluation, but this contribution is very marginal. \n\nThis paper provides an analysis on challenges of combining offline RL with online improvement, which motivates this paper. However, most of the discussed challenges are quite well-known. For example, one major challenge is to estimate the behavior model in offline data. This paper does not discuss techniques/methods that address this challenge, like DualDICE (Nachum et al., 2019) and CQL (Kumaret al., 2020). A comparison to these methods is highly recommended. In addition, this paper may also include additional model-based offline RL methods, like MOPO.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "k1eo8zEGrn", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies shorting coming of existing off-policy methods when it comes to prior data and fine-tuning and shows that those existing methods can't effectively utilize previously collected data with online updates. To address this problem, they propose to constraint policy updates with respect to behavioral policy. Their proposed method is built mainly on the top of AWR [1]. \n\n- There is a significant similarity between the proposed method and AWR [1]. The main difference that I can see is AWR doesn't have a fine-tuning step but this paper does. Can authors list their differences with AWR? \n\n- CRR [3] is very similar to the proposed method in this paper as well, what are the differences? ( I disagree with the authors that CRR is a concurrent work with this paper)\n\n-  This paper claims that AWAC can utilize various types of prior data without any changes to their method and it is agnostic to the type of behavioral policy (e.g. random vs. expert). I don't see how this is the case. For example, if data were collected by a random policy, this method constrains the current policy to be a uniform one.\n\n- Experiments don't make a case for this method, i.e. AWAC is only better than others in relocate-binary-v0 and barely better in Walker2d-v2.  ABM almost gets the same performance.\n\n- Fine-Tuning with random policy data is not convincing either. I'd suggest running some experiments with D4RL [5] dataset like *-medium-expert, *-medium-replay, and *-random to build a better case for your proposed method.\n\n- One reason that existing RL methods can't fully utilize previously collected data with online updates is the distribution shift between the policy (a.k.a behavioral policy) that collected the data and the learned policy [2, 3]. Can the authors comment on this and discuss how the proposed method addresses the distribution shift problem that likely the root cause of this problem? A section in the paper about this will help to improve this paper.  \n \n- Related work section needs significant improvement. It only lists various papers without any useful discussion.  \n\nEven though I found this paper interesting, I'm still not convinced about the contribution of this paper. Given similarity and overlap with previous works [1,3], more experiments could be helpful to see how this method is better than others and why one should select this method vs. others.\n\nMinor comments:\n\nIt seems there are various problems with your bibtex, for example, \"Behavior Regularized Offline Reinforcement Learning\" wasn't published in iclr and author names are mixed with other information like in  \"A Generalized Path Integral Control Approach to Reinforcement Learning\" \"usc\" in author names and \"Web Services\" appeared with author names \"P3O: Policy-on Policy-off Policy Optimization.\"  \n\n[1] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. sep 2019. \n\n[2] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Technical report, 2020\n\n[3 Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3O: Policy-on Policy-off Policy Optimization. In Conference on Uncertainty in Artificial Intelligence (UAI), 2019. \n\n[4] Ziyu Wang, Alexander Novikov, Konrad Zołna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando De Freitas. Critic Regularized Regression. 2020.\n\n[5] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vurck0O34F0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes an approach to use offline data to accelerate the\nonline learning process of reinforcement learning.\nThe proposed approach, called AWAC, uses dynamic programming to train\na critic and supervised learning to trained a constrained actor.\nThe idea is to use a static dataset of experience tuples for\npre-training and use some online interactions to learn the optimal\npolicy for the current task.\nFor the policy improvement step they optimize the policy to maximize\nthe estimated Q-value function but constrained to remain close to the\nactions observed in the data, using the Kulback-Leibler divergence.\nThis constrain is incorporated in the optimization and solve using the\nLagrangian. \nThe proposed approach is very similar to AWR, however, instead of\nestimating the value function of the behavior policy with a Monte\nCarlo approach, they estimate the Q function of the current policy via\nbootstrapping. \nAWAC is tested in several MuJoCo simulator problems, three shown in\nthe main paper (in-hand rotation of a pen, opening a door by\nunlatching the handle, and picking up a sphere and relocating it in a\ntarget location), and others shown in the appendix.\n\nPositive:\n- Use previously collected data for online learning is certainly\nrelevant for reinforcement learning is domains such as robotics. The\nproposed approach shows that a slight change in how to evaluate the\nQ-function to estimate the advantage function can produce a\nsignificant difference\n\nNegative:\n- The paper follows very closely the description of AWR with only\na slight variation in how to evaluate the advantage function. \n- It is not clear how close the static dataset has to be to a \"good\"\npolicy and how it affects the learning process.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new method improving online reinforcement learning using offline datasets. Three reviewers suggested (borderline) acceptance and two did rejection. The main concerns of reviewers are (a) limited/incremental novelty (from all reviewers) and (b) limited experiments (from three reviewers). AC also agrees that the authors' response for novelty beyond the prior works, e.g., AWR (and CRR), is not convincing enough (although their goals/settings are different). AC also thinks that more discussion, analysis and results when offline datasets are poor (e.g., far from experts) are necessary to meet the high standard of ICLR (the authors provided some, but AC thinks it is not convincing enough). Hence, AC recommend rejection. 
Decision: Reject