AW-Opt: Learning Robotic Skills with Imitation andReinforcement at Scale
Authors: Yao Lu
Karol Hausman
Yevgen Chebotar
Mengyuan Yan
Eric Jang
Alexander Herzog
Ted Xiao
Alex Irpan
Mohi Khansari
Dmitry Kalashnikov
Sergey Levine
Venue: CoRL2021 Poster (2021)
Keywords: reinforcement learning
imitation learning

Abstract: Robotic skills can be learned via imitation learning (IL) using user-provided demonstrations, or via reinforcement learning (RL) using large amounts of autonomously collected experience. Both methods have complementary strengths and weaknesses: RL can reach a high level of performance, but requires exploration, which can be very time consuming and unsafe; IL does not require exploration, but only learns skills that are as good as the provided demonstrations. Can a single method combine the strengths of both approaches? A number of prior methods have aimed to address this question, proposing a variety of techniques that integrate elements of IL and RL. However, scaling up such methods to complex robotic skills that integrate diverse offline data and generalize meaningfully to real-world scenarios still presents a major challenge. In this paper, our aim is to test the scalability of prior IL + RL algorithms and devise a system based on detailed empirical experimentation that combines existing components in the most effective and scalable way. 
To that end, we present a series of experiments aimed at understanding the implications of each design decision, so as to develop a combined approach that can utilize demonstrations and heterogeneous prior data to attain the best performance on a range of real-world and realistic simulated robotic problems. Our complete method, which we call AW-Opt, combines elements of advantage-weighted regression and QT-Opt, providing a unified approach for integrating demonstrations and offline data for robotic manipulation.

AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
Yao Lu', Karol Hausman', Yevgen Chebotar', Mengyuan Yan', Eric Jang ,Alexander Herzog?, Ted Xiao', Alex Irpan', Mohi Khansari", Dmitry Kalashnikoy', Sergey Levine 1.3
Robotics at Google 2x, The Moonshot Factory UCC Berkeley
Robotic skills can be learned via imitation learning (IL) using user- provided demonstrations, or via reinforcement learning (RL) using large amounts of autonomously collected experience. Both methods have complementary strengths and weaknesses: RL can reach a high level of performance, but requires exploration, which can be very time consuming and unsafe; IL does not require exploration, but only learns skills that are as good as the provided demonstrations. Can a single method combine the strengths of both approaches? A number of prior methods have aimed to address this question, proposing a variety of tech- niques that integrate elements of IL and RL. However, scaling up such methods to complex robotic skills that integrate diverse offline data and generalize mean- ingfully to real-world scenarios still presents a major challenge. In this paper, our aim is to test the scalability of prior IL. RL algorithms and devise system based on detailed empirical experimentation that combines existing components the most effective and scalable way. To that end, we present series of experiments aimed at understanding the implications of each design decision, so as to develop a combined approach that can utilize demonstrations and heterogencous prior data to attain the best performance on a range of real-world and realistic simulated robotic problems. Our complete method, which call AW-Opt, combines ele- ments of eawaantage-weigheee regression [1, 2] and QT-Opt [3], providing a unified approach for integrating demonstrations and offline data for robotic manipulation. Please see htaps://wwop.ggihubb.u for more details.
Abstract:
Keywords: Imitation Learning, Reinforcement Learning, Robot Learning
Introduction
Learning methods for robotic control are conventionally divided into two groups: methods based on autonomous trial-and-error (reinforcement learning), and methods based on imitating user-provided demonstrations (imitation learning). These two approaches have complementary strengths and weaknesses. Reinforcement Learning (RL) enables robots to improve autonomously, but introduce significant challenges with exploration and stable learning. Imitation learning (IL) methods provide for more stable learning from expert demonstrations, but cannot improve from failures. This could potentially make covering the full distribution of data difficult.
Prior works have sought to combine elements of II and RL into a single algorithm (IL+RL), by pre-training RL policies with demonstrations [4], incorporating demonstration data into off-policy RL algorithms [5], and combining IL and RL loss functions into a single algorithm [6]. While such algorithms have been based on both policy gradients and value-based off-policy methods, the latter are particularly relevant in robotics settings due their ability to leverage both demonstrations and the robot's 's own prior experience. A number of such methods have shown promising results in simulation and in the real world [7]. but they often require extensive parameter tuning and, as we show in our experiments, are difficult to deploy at scale to solve multiple tasks in diverse scenarios.
In this paper, our goal is develop, through extensive experimentation, complete and scalable system for integrating II. and RL that enables learning robotic control policies from both demon- stration data and suboptimal experience. Our aim is not propose new principles for algorithms that combine IL and RL, but rather to determine how to best combine the components of existing methods enable effective large-scale robotic learning. We begin our investigation with two exist-
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

(6)
(6)
(4)
Offine Training Succees
Online Fineturing Sucoees
Samples (x1000))
Task3 Task5
20-
0.4
0.2
0.4
0.2
0.0
OF-Opt
WNAC
MWV Opt
0.0
OT-Opt
AMAC
AWV-pt
OP-Opt
AMACC
ANV Opt
(9)
(c)
Figure 1: We study algorithms that combine imitation and reinforcement learning in the context of high- dimensional visual manipulation tasks. Existing RL algorithms such as QT-Opt [3] can solve simple tasks from scratch with large amount of training data, but fail take advantage of high-quality demonstrations data provided up-front and fail to solve difficult tasks. Offiine RI. methods that combine imitation learning and RI,, such as Advantage- Weighted Actor-Critic (AWAC) [2], can learn good initial policies from demonstrations, but encounter difficulty when further fine-tuned with RL on large-scale robotic tasks, such as the diverse grasping scenarios in this figure. We present AW-Opt, an actor-critic algorithm that achieves stable learning in both offline imitation learning and RL fine- tine-tunng phases. We evaluate method on five different simulated (a.b) and real-world (c.d) visual grasping tasks. (a) Task 1: simulated indiscriminate grasping (b) Task 2: simulated instance grasping (c) Task 3: real indiscriminate grasping (d) Task 4 and Task 5: real instance grasping and grasping from a specific location (c) Running a real-world policy (f) Comparing success rates after initial offline imitation phase, final RL
phase, and the number of samples needed converge
ing methods: AWAC [2], which provides an integrated framework for combining IL and RL, and QT-Opt [3], which provides scalable system for robotic reinforcement learning. We observe that neither method by itself provides a complete solution to large-scale RL with demonstrations. We then systematically study the decisions in these methods, identify their shortcomings, and develop hybrid approach that we call AW-Opt. The individual design decisions that comprise AW-Opt may appear minor in isolation, but we show that combining these decisions significantly improves on the prior approaches, and scales effectively to learning high-capacity generalizable policies in multiple real and simulated environments shown in Fig. (a-d).
The main contributions of this work consist of a detailed analysis of the design decisions that in- fluence performance and scalability of IL+RL methods, leading to the development of AW-Opt, hybrid algorithm that combines insights from multiple prior approaches to achieve scalable and ef- fective robotic learning from demonstrations, suboptimal offline data, and online exploration. Not only can AW-Opt be successfully initialized from demonstrations (initial success in Fig. 1()), but also it responds well to RL fine-tuning (final success in Fig. (f)) while being sample-efficient.
In addition to the design of AW-Opt, we hope that the analysis, ablation experiments, and detailed evaluation of individual design decisions that present in this work would help to guide the de- velopment of effective and scalable robotic learning algorithms. Our experimental results evaluate method, as well as ablations, across range of simulated and real- domains, including two different real-world platforms and large datasets with hundreds of thousands of training episodes.
2 Related Work
Prior works have proposed a number of methods for learning robotic skills via imitation of ex- perts [8, 9, 10, 11, 12, 13), and via reinforcement learning [14, 15, 16, 17, 18). In the latter category, off-policy RI. methods are especially appealing for large-scale robot learning, since they allow for data reuse with large, diverse datasets [3, 19, 20]. Initially, methods that combined imitation learn- ing (IL) and reinforcement lcarning (RL) have tended to treat the two data sources differently, using IL demos only at initialization [4, 5, 21]. or using different losses for IL demos vs. RL episodes [6]. Other methods have used a different approach: combining both expert demos and suboptimal prior data into a single dataset used for off off-policy value-based learning [7]. Offline RL methods, such as CQL [22], QT- [3], and AWR [1] often use this approach, along with constrained optimization and regularization to encourage staying close to the data distribution [23, 24, 25, 26, 27J. Our method builds on these approaches, due to their effectiveness, simplicity, and easier data management. Re- gardless of whether the data consists of optimal demonstrations, suboptimal human providdd demos, or very suboptimal robot experience, all data can be treated the same way.

---

Our goal is to scale up IL+RL methods to large datasets with many objects and varied environments. Standard RL methods have proven difficult to scale up to the same level as modern methods in computer vision [28, 29] and natural language processing [30, 31, 32], where datasets are diverse and open-world, and models have millions or billions of parameters. Taking RI. to such settings requires large, distributed RI. systems, which often require careful design decisions in their underlying RL. methods. While standard methods such as actor-critic have been scaled up in this way [33, 34, 35, 361, we show in this work that scaling IL+RL algorithms requires certain amount of care, especially in robotic settings where collecting real-world experience is costly. Naively incorporating demonstrations into an RL system (QT-Opt in our experiments) does not by itself yield good results. Similarly, utilizzng prior IL+RL methods (such as AWAC [2] in our experiments) scales poorly. We provide a detailed empirical evaluation of various design decisions underlying IL+RL methods to develop a hybrid approach, which we call AW-Opt, that combines the scalability of QT--ppt with the ability to seamlessly incorporate suboptimal experience and bootstrap from demonstrations.
3 Preliminaries
Let (S,SA,, R, po, 7, T) define a Markov decision process (MDP), where S and A are state and action spaces, P: x A x S + J is state-transition probability function, R: S x A + IR reward function, Pa S - R. an initial state distribution, 7 discount factor, and T is the task horizon. We use T = (so: oo...., .r. 0r. to denote a trajectory of states and actions and [R(T) Ei R(s,,, ae) to denote the trajectory reward. Reinforcement learning (RL) methods find a policy π (aass) that maximizes the expected discounted reward over trajectories induced by the policy: E2 [R(T)] where so Pu, Se+1 ~ P(S st; ae) and a ~ x(a (s)).
In imitation learning, demonstrations are provided to facilitate learning an optimal policy, which can be produced through teleoperation [37, 38] or kinesthetic teaching [39, 12]. Given data set of demonstration state-action tuples (s, a) ~ Daemoo standard II. methods aim to directly approximate the expert's x" (a|s) via supervised learning. Another way of learning from prior data has been ex- plored in the field of offline RL [40]. where RL methods are used to learn from datasets of previously collected data. This data can include demonstrations, as well as other suboptimal trajectories. Such offline RL methods are particularly appealing for hybrid ILL methods, since they do not require any explicit distinction between demonstration data and suboptimal experience.
In this paper, we build on these approaches to develop a scalable algorithm that combines IL and RL. Our goal is to smoothly transition between two sources of supervision: the demonstrated be- haviors, provided as prior data, and RL exploration, collected autonomously. Like many offline RI. algorithms, our approach is based on Q-learning, which extracts the policy from Q-function that is trained satisfy the Bellman equation: Q" (st, az) = R(st, (+) + ymax, [ (8++111)))
To develop an effective IL-RL algorithm, we start by analyzing various design decisions in two re- cently proposed offline RL methods: QT-Opt [3] and Eavantage-Weighted Regression (AWR) [1], with its recent value-based instantiation AWAC [2]. QT-Opt is a distributed Q-learning framework that enables learning Q-functions in continuous action spaces by maximizing the Q-function using the cross-entropy method (CEM) [41], without an explicit actor model. However, initializing Q- function from demonstration data is particularly challenging as it often results in optr-optiii Q-values on unseen state-action pairs [42, 24, 22j. It has been shown that by introducing an explicit actor, one can mitigate this issue by limiting the actor to select actions that are within the (condi- tional) distribution of actions seen in the data [24, 1]. This property is also leveraged by AWAC (2), where the actor is initialized from demonstration data, and then further fine-tuned with RL rewards. The AWAC algorithm uses the following update to the actor:
[(s) exp (AA79(8100) log (a(ls))
0 + arg max Eg-D, (als)
(I)
where A(s, a) = Q(s, a) (a [Q(s, a)] is the advantage function, Z(s) is the normalizing partition function, A is a Lagrange multiplier, To is the learned policy and πe is the behavior policy, which represents the (unknown) distribution that produces the dataset. This corresponds to weighted maximum likelihood training, using samples from л (a
(a s) (obtained from the dataset).
In this paper, we analyze various design decisions in QT-Opt and AWAC, point out their shortcom- ings in the IL setting, and use these insights to construct an effective hybrid RL method that call AW-Opt.
3

---

4 Learning via Imitation and Reinforcement with AW-Opt
In order to derive AW-Opt, we first study the performance of QT-Opt [3] and AWAC [2] in the IL+LL setting, using a complex robotic manipulation task with high-resolution image observations as our working example. This setting requires algorithms that are more scalable than those typically employed in standard RL benchmarks. We will show that neither QT-Opt nor AWAC effectively handle this setting. We then progressively construct a new algorithm with elements of both ap- proaches, which we call AW-Opt, analyzing each design decision in turn. Although each individual design decision in AW-Opt may appear relatively simple, we will see that the specific combination of these design decisions outperforms both QT-Opt and AWAC by a very large margin, resulting in successful learning for tasks where the two prior methods are unable to make meaningful progress.
4.1 Example Task and Baseline Performance As a working example, we use a vision-based trash sorting task, where the robot must pick up specific type of trash object (e.g.. a com- postable objects) out of cluttered bins contain- ing diverse object types, including recyclables, landfill objects, etc., with only sparse rewards indicating success once the robot has lifted the correct object. We describe this task in detail in Section 5. We will use this environment to illustrate the importance of each component of our method, as we introduce them in this section. Then, in Section 5, we will evaluate our complete AW-Opt algorithm on wider range of robotic environments, including two tems and several high-fidelity simulation environments. Note that unless otherwise specified, all the plots in this paper is conducted with 3 experiments each. Each data point is averaged success rate evaluated with 700 episodes. The shaded region in the plot depicts the 90% confidence interval with the line being the mean.
0.075
Train domonstraiona
OT-Opt
succe
0.050
AWAC
Train
0.025
0.000
10000
20000
30000
40000
50000
steps
Figure 2: Baseline comparison for the example task.
Neither QT-Opt [3], nor AWAC [2] can solve the task.
robotic sys-
We begin our investigation with the two prior methods: QT-Opt [3], which can scale well to high- resolution images and diverse tasks, and AWAC [2], which is designed to integrate demonstrations and then fine-tune online. Both algorithms are provided with 300 demonstrations for offline pre- training, and then switch to on-policy collection after 10k gradient steps to collect and train with 200k additional on-policy episodes. In Fig. 2, we see that basic QT-Opt fails to make progress on both the pre- pre-training and online data. Q-learning methods are typically ineffective when provided with only successful trials [40]. since without both "positive" successful trajectories and "negative" failed trajectories, they cannot determine what not to do. Then, during the online phase, any RL method that is not pretrained properly is faced with a difficult exploration problem, and cannot make progress. QT-Opt is one of those RL algorithms. AWAC [2] does attain non-zero success rate from the demonstrations, but performance is still poor, and it does not scale successfully to the nigh-dimensional and complex task during online fine-tuning, Next, we will introduce a se- ries of modifications to AWAC that bring it closer to QT-Opt, while retaining the ability to utilize demonstrations, culminating in our full AW-Opt algorithm.
4.2 Positive Sample Filtering
One possible explanation for the poor perfor- mance of AWAC in Fig. 2 is that, with the rel- atively low success rate (around 4%) after pre- training, the addition of large amounts of addi- tional failed episodes during online exploration drowns out the initial successful demonstra- tions and, before the algorithm can learn an ef- fective Q- Q-function to weight the actor updates, results in updates that remove the promising pre-training initialization. To address this issue, we introduce two modifications: (a) We use prioritized buffer for the critic, where 50% of the data in each batch comes from successful (posi- tive) episodes that end with a reward of 1. Note that for tasks with non-binary rewards, we would need to introduce an additional hyperparameter for the "success" threshold, which we leave for future work. (b) We use positive filtering for the actor, applying the AWAC actor update Equa- tion I only on samples that pass the success filter (i.e., receive reward of 1). While modification
0.20
NMCC NWVC posttive fitertng
0.35
0.30
0.06
0.00
25
50 100 325
150
375
200
samples 011000)
Figure 3: Positive filtering prevents AWAC perfor- mance from collapsing during the offline phase, seen Fig. 2. However, AWAC + positive filtering does not improve during the on-policy phase.

---

a) ensures that the critic's training batch is balanced in terms of positive and negative samples, the modification in b) plays subtle but very important role for the actor. By filtering the data that is presented to the actor, this modification ensures that the actor's performance should not drop below behavioral cloning performance, making it independent of the potentially inaccurate critic during the early stages of training.
After applying the above adjustments, AWAC performance no longer collapses after switching to online fine-tuning, as shown in Fig. 3, but still fails to improve significantly, with a final success rate of 5%. We abbreviate this intermediate method as AWAC_P (AWAC + Positive filtering). To improve it further, we will discuss how to improve exploration performance in the next subsection.
4.3 Hybrid Actor-Critic Exploration
As Q-learning method, QT-Opt does not have an explicit actor. Since the task involves con- tinuous actions, QT-Opt uses the cross-entropy method (CEM) [41] to optimize the action us- ing its critic, which can be viewed as an implicit policy (CEM policy). Unlike QT-Opt, AWAC an actor-critic algorithm, which means it has access to both an explicit actor network and critic network. Exploration is performed by sampling actions from this actor. However, this can present an obstacle to learning in complex domains. First, the commonly used unimodal actor distribution may not capture the multimodal in- tricacies of the true Q-function, especially early on in training when the optimal action has not been determined yet. Second, the bulk of the learning must be done by the critic, since the actor doesn't ' reason about long-horizon rewards. Hence, the critic must first learn which actions are good, and then the actor must distill this knowledge into action distribution, which can introduce delays.
0.4
KNMC. P CANAC posttue fltering)
0.3
ANAC level random switcher
ANAC CEN oxpleration
ANNC ephoode level mendom saitcher
0.3
75
100 125
150
175
200
samples (=1000)
Figure 4: Comparisons of different hybrid actor-critic exploration strategies during the online fine-tuning phase. Using both the actor and critic for exploration significantly improves performance.
Here, we aim to address these issues by utilizing both the actor and the critic for exploration. Al- though we are building on AWAC, we can still use the implicit CEM policy from QT-Opt to select actions via the critic, bypassing the actor. We therefore compare five exploration strategies: (a) actor-only exploration (b) implicit critic policy (as in QT-Opt, where CEM is used to pick the best action according to the critic); (c) episode-level random switcher, which randomly picks a policy at the beginning of each episode (80% critic policy, 20% actor); and (d) step-level random switcher, which randomly picks policy at each time step (80% critic policy, 20% actor).
Fig. 4 shows the results of this comparison. The episode level random switcher outperforms the other candidates, achieving more than 10% success within 80k samples. Adopting the episode-level random switcher, we call this intermediate algorithm AWAC ELRS. Given the observed improve- ment in the early stage of exploration, we shift our focus to analyzing whether value propagation in the critic is sufficiently able to utilize the data during online fine-tuning.
4.4 Action Selection in the Bellman Update
In the previous section, we showed how using the critic directly for exploration, rather than the actor, can lead to significantly better per- formance. We speculate that this is due to the fact that such an exploration strategy provides a degree of "lookahead": the actor can only leam to take good actions after the Q-function has learned to assign such actions high values. This means that by taking actions according to the Q-function during exploration, we can "get out ahead" of the actor and explore more ef- ficiently. Can we apply similar logic dur- ing training, and construct target Q-values with some amount of CEM optimization over actions,
0.5
AMIKC_P P. CLRS (ANANC coisode level random switcher)
AMIAC, EUR max
0.6
AMDAC, P. CLRS ma10 Actortiian
AMIAC P. DURS max0 ActorCandidtte
0.2
00
30 75 100 150 175
200
Figure 5: Comparison of different target value cal- culations. Maximizing the critic values generally per- forms significantly better than the standard AWAC tar- get value, with the best version being maxQ + Actor- Candidate which uses the actor as an additional action candidate during the CEM optimization.
rather than simply sampling actions from the actor?
Based on this intuition, we hypothesize that, in much the same way that direct maximization over the Q-function provided better exploration, improving over the actor's action in the target value
5

---

calculation will also lead to faster learning. While previous offline RL techniques [24, 2] make sure to carefully calculate the target value avoid a potential use of out of distribution actions, we expect this to be less critical for our method, since the positive-filterng strategy in Section 4.2 already ensures that the actor is only trained on successful actions, significantly limiting the harm caused by an inaccurate Q-function. To evaluate this idea, we test several ways of selecting actions in the Bellman backup.
Algorithm 1 AW-Opt
In addition to the standard AWAC target value, which is taken as an expectation under the ac- tor, we test: (a) maxQ uses the same target
1: Dataset D (8t, az, r(st, at), 8141)
2: Initialize buffer B = D
3: Initialize parameter vectors Φ, 0,
value calculation as QT-Opt, optimizing the ac- tion in the target value calculation with CEM
4: for each iteration i tre do
5: if iieter >= T then
("maxQ"); (b) maxQ + ActorMean com- bines the actor with CEM by using the actor- predicted action as the initial mean for CEM; (c) max + CandrCand combines the ac- tor with CEM by using the actor-predicted ac- tion an additional candidate in each round of CEM. These target value calculations, particu-
6:
for each environment step do a, ~ =e(as(ss) or ar ~ TCRM(()((s))
7:
8:
8141 p(8t41(st, ai)
3 +- 3 U (st, at), st+1)
10:
Sample batch (8t, as, r(st, at), S(41) ~ B for each gradient step do
11:
12:
a" = arg maxa (Qo (0(1),)))
13:
O, < 0, - Xo (Ll(Qu (8t, as). r(s,
larly (a), bring the method significantly closer
Q0, (8++1, a')
to QT-Opt. In fact, (a) can be equivalently seen
14:
= T0 + (1 71)
Qp- Q-function learning method, aug- mented with an additional positive-filterd filltred ac- tor that is used for exploration. However, see in Fig. 5, the performance of this approach
15:
Adv
Qa(s,, a.) -
a:l
16:
Φi + O.- exp(eee)V)ALaaau (s))))) Imness
significantly better than both the AWAC-style
Bellman backup and the standard QT-Opt method, neither of which learn the task successfully. The "maxQ" strategy leads more than three-fold increase in performance after 200k samples over the previous best method. In addition, we notice that (b) is worse than (a). Our hypothesis for why "maxQ + ActorMean" is worse is the following: in actor-critic methods, the actor always lags be- hind the critic. Therefore, when the actor is not good enough, it may shift the initial distribution and lead the CEM in the wrong direction.
Fig. 5 demonstrates the comparison of these techniques. AWAC_P AWAC_P.ELRS + maxQ is significantly better than AWAC_P A.LRSS Two variants: AWAC_P ELRS maxQ ActorMean (using actor- predicted action as the initial CEM mean) and AWAC_P ELRS + maxQ + ActorCandidate (using actor-predicted action as an additional CEM candidate to compare with the best action selected by CEM) also improve over AWAC P_ELRS. As we notice that AWAC_P P_ELRS + maxQ + ActorCan- didate achieves the best performance, we turn AWAC.P.ELRS max + ActorCandidate into our final AW-Opt method, which is shown in Algorithm 1.
ELRS.
4.5 Ablations
After the step-by-step design of the AW-Opt al- gorithm, we run an additional ablation study by removing one feature at time to further identify the importance of each of the design choices we have made. We compare the fol- lowing four cases; (a) AW-Opt, (b) AW-Opt without positive-filterng,, (c) AW-Opt without ActorCandidate, (d) AW-Opt without hybrid actor-critic exploration.
0.6
positive
Adiionznneldttt
asploratios
022
0.0
(22000)
Figure 6: Ablations of each individual component of AW-Opt. Note that every ablation significantly de- grades performance, indicating the importance of each of the components.
As shown in Fig. 6, each of the introduced com- ponents plays a significant role in providing boost in performance. Among the proposed AW-Opt design decisions, positive filtering is particu- larly crucial for achieving good results in the online phase. Without applying positive filtering, the performance of AW-Opt collapses to 0% once on-policy training starts, confirming our hypothesis that resitive-filtering of the actor robustifies the actor updates against inaccurate Q-functions in the initial phases of training. However, each of the other components is also critical, with the best- performing ablation (the one that uses the standard AWAC target value calculation) still being about three times worse than full AW-Opt.

---

Experiments
In our experimental evaluation, we study the following questions: (1) Can AW-Opt learn from both expert demonstrations and suboptimal off-policy data? (2) Is AW-Opt viable RL fine-tuning algo- rithm that continues to improve after pre-training on offline data diverse real-world and simulated metrics important for robot learning, such real-world performance and sample efficiency? (4) Are the previously mentioned design choices critical for the success of the method on wide range of tasks?
settings? (3) Is AW-Opt competitive
5.1 Experimental Setup
We evaluate our method on simulated and real robotic manipula- tion tasks (Fig. and Fig. 7). We use two real robotic systems with parallel-jaw grippers and over-the-shoulder cameras: Kuka IIWA robots, and proprietary 7-DoF arm that we call "Pica" I. We use two simulated tasks and three real tasks in our experiments:
Task 1: (Sim) Indiscriminate Grasping with Kuka (Fig. (a)). Six random vrocedurally-generaa
objects are placed in a bin. If
the robot lifts any of the objects, the episode is successful. We uti- lize around 1000 off-policy simulation trials for this task.
Task 2: (Sim) Green Bowl Grasping with Pica (Fig. (b)).
green bowl is randomly placed on a table among other objects. If the robot lifts the bowl, the episode is considered success. We utilize 120 teleoperated simulated demonstrations for this task. Task 3: (Real) Indiscriminate Grasping with Kuka (Fig. (c)). This is the
version of Task 1. We follow the setup pro- posed in [3], and utilize off- off-policy data for this task for pre-tratee training. Task 4: (Real) Grasping from the Middle Bin with Pica (Fig. I (d)). Several types of trash are placed randomly in three bins. The episode is successful if the robot lifts any object from the middle bin. We use 1000 demonstrations for this task.
Task 5: (Real) Grasping Compostables from the Middle Bin with Pica (Fig. 1 (d)). This is a more difficult version of Task 4. While setup is equivalent, an episode is successful if compostable object was lifted. We utilize 300 demonstrations for this task.
Figure 7: Real robot setup.
In the pretraining phase, we utilize the prior data described above and train the network (the same network as QT-Opt [3]) for 5,000 to 10,000 gradient steps. Each algorithm then collects 20,000 to 200,000 episodes of on-policy data during further training. For Task 4 and Task 5, fine-tuning is done simulation, using RL-CCcleGAAN [43] to produce realistic images. This allows us to compare fine-tuning performance between QT-Opt, AWAC, and AW-Opt these tasks, using simulated evaluations. Separately, we conduct real-world evaluations, evaluating the offline pretraining stage and sim-to-real transfer of fine-tuned policies, since running each of the three algorithms online in the real world would be too time-consuming. This evaluation protocol provides evidence to support central claims: the simulated experiments confirms that method can effectively finetune from online data, while the real-world evaluation confirms that our approach can handle offline pre-training on real data effectively.
5.2 Learning from Demonstrations and Suboptimal Off-Policy Data
To address the question of whether AW-Opt can learn using different types of prior data, we prepare datasets from different sources: simulated off polico data (Task 1), simulated demonstrations (Task 2) and real demonstrations (Task 4 and Task 5). We compare the learning performance of QT-Opt, AWAC, and AW-Opt using these 3 different data sources in Fig. (left panels of each subplot). Both AWAC and AW-Opt can learn from both demonstrations and off-policy data, while QT-Opt cannot, since Q-learning requires both successful and failed trials to determine which actions better than others. Depending on the difficulty of the task, the success rates of the policies learned by AWAC and AW-Opt vary from 20% to 60% with the exception of the difficult Task 5, which requires on-policy finetuning.
Pica is pseudonym for this paper.
7

---

Task
Task
1.0
1.0
0.8
Train w/
0.8
Train w/
off-policy data
Ieleoperated demonstrations
success
0.6
0.6
0.4
0.4
Train w/
Train mn-policy dats
0.2
on-policy data
0.0
2
3
0
3
9
12
15
18
0.0
40
100
steps (x1000)
samples (x1000)
steps (x1000)
samples (x1000)
Task
Task
1.0
1.0
Train w/
0.8
Train w/ real demonstrations
Train on policy data
real demonstrations
success
0.6
0.6
Train w/ onipolcyy data
0.4
OT-Opt
0.2
0.2
ANAC
Opt
0
9
30 60
90 120
0.0
9 0
60 90 120 150 180
steps (x1000)
samples (x1000)
steps (x1000)
samples (x1000)
Figure 8: Offline pretraining followed by online finetuning on each of our simulated evaluation tasks. Each plot shows the offiine phase (left of the vertical line), followed by an online phase where each method uses its respective exploration strategy to collect more data. In most tasks, AW-Opt and AWAC learn a moderate level of success from offline data, while QT-Opt struggles due to the need for negatives. During online finetuning, QT-Opt generally performs better than AWAC, but AW-Opt outperforms both prior methods. both learning faster (e.g., for Task 1) and reaching a significantly higher final performance (Take 4 and Task 5).
5.3
Online Fine-tuning
As shown in Fig. (right panels of each subplot), during online fine-tuning, QT-Opt performs signif- icantly better than AWAC, while the latter struggles to effectively utilize online rollouts. However, with the modifications effectively during the online phase, while utilizing the offline data to bootstrap this fine-tuning pro- cess. While AW-Opt is more data-efficient than QT-Opt on Task I and Task 2, this difference is particularly pronounced on Task and Task 5, where the inability of QT-Opt to utilize the prior data significantly hampers its exploration.
AW-Opt, our method is able to attain the best of both worlds, fine-tuning
Table Task and Task real robot results.
5.4
Real-World Robotic Manipulation Results
Task 3
Task 3
Task 5 finetune
We evaluate the performance of policies trained with QT-Opt, AWAC, and AW-Opt after offline pre- training for Task 3 and after finetuning for Task 5. The fine-tuning stage is performed in simulation, as
positives only
positives negatives
QT-Opt
0%
61.23%
0%
AWAC
44.21%
0%
0%
AW-Opt
52.53%
57.84%
48.89%
discussed in Section 5.1. The goal of these experi-
ments is to validate that policies learned with AW-Opt can indeed handle offline demonstration data and generalize
real-world settings. The size and type of the datasets described in Appendix. For Task 3, we first train AWAC, QT-Opt, and AW-Opt with only positive data and then train with both positive and negative data. We evaluate each policy with 612 grasps, as shown in Table 1. When learning from the combined positive and negative data, AWAC performance drops to 0%. When QT-Opt learns from only positive data, it also fails to learn the task. Only AW-Opt is able to learn from both positive-only and mixed positive-negative data. For Task 5, we evaluate each policy with 180 grasps, as shown in Table 1. Only AW-Opt learns this task successfully, achieving a 48.8 18.89% success rate. These results support the conclusions drawn from our simulated experiments.
6 Conclusion
We presented AW-Opt, a scalable robotic RL algorithm that incorporates offline data and performs online finetuning. We show that each design decision in AW-Opt leads to significant gains in perfor- mance, and the final method can learn complex image-basee based tasks starting with either demonstra- tions or suboptimal offline data, even in settings where prior methods, such as QT-Opt or AWAC, either fail to leverage prior data fail to make progress during finetuning. Our evaluation does have several limitations. First, we only evaluate offline training in the real world, studying finetuning under simulated settings where extensive comparisons are feasible. Our experiments are also all concerned with sparse, binary reward tasks. Such tasks are common in robotics, and AW-Opt could be extended arbitrary rewards via threshold for positive filtering, a promising direction for future work. Nonetheless, our results suggest that AW-Opt can be a powerful tool for scaling up IL+RL methods. Aside from motivating each design decision, we also hope that the detailed evaluations of each component will aid the design of even more effective and scalable RL methods in the future.
8

---

Acknowledgments
We would like to give special thanks to Dao Tran, Conrad Villondo, Clayton Tan for collecting demonstration data as well as Julian Ibarz and Kanishka Rao for valuable discussions.
References
[1] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regres- sion: Simple and scalable off off-policy reinforcement learning. CoRR, abs/1910.00177, 2019. URL http:/ html #abs-1910-00177
uni-trier.des ditee//de//////// wtw.d////////aaleeeeeeee journals//orr//oooooooooo 9900
[2] A. Nair, M. Dalal, A. Gupta, and Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09333 2020.
[3] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly,
M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: 10293, 2018
[4] S. Schaal et al. Learning from demonstration. Advances in neural information processing systems, pages 1040-1046, 1997.
[5] J. Kober, B. Mohler, and J. Peters. Imitation and reinforcement learning for motor primitives with perceptual coupling. From motor learning to interaction learning robots, pages 209-225, Springer, 2010.
[6] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 6292-6299. IEEE, 2018.
[7] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv: arXiv: /77770887777.... 2017.
[8] N. Ratliff, J. A. Bagnell, and S. Srinivasa. Imitation learning for locomotion and manipula- tion. In 2007 7th TEEE-RAS International Conference on Humanoid Robots, pages 392-397, 2007. doi: 10..1109/CHH..1000000 2007.4813788
[9] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and F. Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligett Robotics and Applications, 3(4):362-369, 2019.
[10] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
[11] Pastor, H. Hoffmann, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. In 2009 IEEE International Conference on Robotics and Automation, pages 763-768. IEEE, 2009.
[12] S. Khansari- and A. Billard. Learning stable non-linear dynamical systems with gaussian mixture models. IEEE Transaction on Robotics, 27(5):943-95 2011.
[13] K. Mülling, J. Kober, O. Kroemer, and J. Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal Robotics Research. 32(3):263- 279, 2013.
[14] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05000 2018.
[15] P. Lillicrap, J. J. Hunt, A. Pritzel, Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 150900297,, 2015.

---

[16] E. Theodorou, J. Buchli, and S. Schaal. Reinforcement learning of motor skills in high dimen- sions: path integral approach. In IEEE International Conference on Robotics and Automa- lion (ICRA), 2010.
[17] Peters and S. Schaal. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21 (4):682-697, 2008.
[18] J. Peters and S. Schaal. space control. Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
Reinforcement learning by reward- weighted regression for operational
[19] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv: (OXiv:20040022111 12, 2021.
[20] M. Khansari, D. Kappler, J. Luo, J. Bingham, and M. Kalakrishnan. Action image repre- sentation: Learning scalable deep grasping policies with zero real world data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3597-3603, 2020.
[21] X. Zhang and H. Ma. Pretraining deep actor-critic reinforcement learning algorithms with expert demonstrations. arXiv preprint arXiv: 2018.
[22] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. CoRR, abs/2006.04779, 2020.
[23] S. Fujimoto, D. Meger, and D. Precup. Off- Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[24] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
[25] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning, 2020. URL https::t////.. http:://ppppeeeeeeeee net/ forum? i d=BJg 9hTNKPH.
[26] C.-A. Cheng, X. Yan, and B. Boots. Trajectory-wise control variates for variance reduction in policy gradient methods. In Conference on Robot Learning, pages 1379-1394. PMLR, 2020.
[27] G. Kahn, A. Villaflor, P. Abbeel, and S. Levine. Composable action-conditioned predictors: Flexible off-policy learning for robot navigation. In Conference on Robot Learning, pages 806-816. PMLR, 2018.
[28] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 15:1097 2012.
[29] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In NIPS, 2017.
[31] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.1ee 101.089,0. 10683, 2019.
[32] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint ar05v:2005. 14165, 2020.
[33] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
10

---

[34] E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg. J. Gonzalez, M. Jordan, and I. Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference Machine Learning, pages 3053-3062. PMLR, 2018.
[35] OpenAl, C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz. S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, Sutskever, J. Tang, Wolski, and S. Zhang. Dota with large scale deep reinforcement learning. 2019.
URL https: /aaxivivvvv..////i org//brg//.... 1912. 0668C.
[36] O. Vinyals, I. Babuschkin, W. M. Czamecki, M. Mathieu, A. Dudzik, Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782)350-354 2019.
[37] Pastor, Hoffmann, T. Asfour, and Schaal. Leaning and generalization of motor skills by learning from demonstration. In ICRA, pages 763-768, 2009.
[38] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality lelcoperation. ICRA, pages 1-8. IEEE, 2018. ISBN 978- 1188--1336-30811111
URL http:://t///t uni teriittrer..o /d/ conf/icra/icra22 8 html 4 #2hangMJLCCCA1..
[39] S. Calinon, F. Guenter, and A. Billard. On learning, representing and generalizing a task in a humanoid robot. IEEE Trans. on Systems, Man and Cybernetics, Part B, 37(2), 2007.
[40] Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[41] R. Y. Rubinstein and D. P. Kroese. The Cross Entropy Method: A Unified Approuch To Combi- natorial Optimization, Monte- Carlo Simulation (Information Science and Statistics). Springer- Verlag, Berlin, Heidelberg. 2004. ISBN 038721240X.
[42] R. Laroche, P. Trichelair, and R. T. d. Combes. Safe policy improvement with baseline boot- strapping. arXiv preprint 1512:1712.069 2017.
[43] A. Irpan, C. Harris, J. Ibarz, K. Rao, M. Khansari, and S. Levine RI-cyclegan: Improving deep-rl roboties with simulation-to-real. arXiv preprint arXiv:2006.0900 2020.


Reviews:
[{"review_id": "T48yUVT7YLi", "reviewer": "Reviewer_qeFD", "summary": "This paper, on the one hand, investigates experimentally why two existing RL methods have difficulties solving a bin picking task with a sparse reward for pick-up success.\nOn the other hand, the authors compare performance improvements by introducing small changes in three different aspects of the RL problem:\n1) Data balancing for training the Q function and the policy.\n2) Exploration with a learned actor policy vs optimizing the learned Q function. \n3) Different ways of calculating the Bellman update.\n\nIn the experiments, the influence of these changes both in terms of asymptotic performance and the capability of the system to include demonstration data is investigated.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- I appreciate the research question of figuring out why existing methods have difficulties in incorporating demonstrations for later fine tuning.\n- Interesting ablation studies/comparisons. \n- Good video!\n\nWeaknesses:\n- No real methodological advancement. Proposed approach consists of only small tweaks to existing methods.\n- Results don't allow a clear conclusion (QT-opt can also incorporate negative demonstrations).\n- The choice of QT-opt and AWAC seem rather arbitrary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am not sure what to think about this paper.\nThe gained insights are not really surprising.\nIt is well-known that RL is difficult due to the exploration problem of collecting meaningful data, the credit assignment problem and the learning problem itself (whether it is Q functions or policies, even given optimal (but imbalanced) data).\nWhat I have learned from the paper is that small modifications can make a large difference without changing the underlying learning methods significantly. But this also implies that the problem is not yet well understood yet and I am unsure if the findings of the paper are applicable to other domains/tasks as well.\nNevertheless, the approach of investigating the data balance problem, exploration and Bellman update is interesting and relevant.\nI would recommend ICRA instead of CoRL.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- \"Then, during the online phase, QT-Opt is faced with a difficult exploration problem\". Why QT-opt particularly?\n- The fact that the episode-level random switcher is the best in Fig. 4 seems to be caused by the fact that it introduces the most useful randomness for exploration compared to CEM exploration only? How has the 80% 20% ratio been chosen?\n- In Fig. 5 why is the grey curve better than the yellow one given the fact that the blue one outperforms all? Maybe a clarification how actor candidate differs from actor mean would be useful. (particularly I don't full get what it is meant with \"maxQ + ActorCandidate combines the actor with CEM by using the actor-predicted action as an additional candidate in each round of CEM.\").\n- Why not include non successful demonstrations for QT-opt (as done in Tab. 1)? It seems arbitrary to not do that for most of the experiments and indeed in sec. 5.4 QT-opt performs best on task 3.\n- Line 13 in Algo 1 does not make sense (typo, I think)."}}, {"review_id": "JjQrsFlb6c", "reviewer": "Reviewer_3em5", "summary": "In this paper, the authors have tried to combine the advantage of imitation learning (IL) and reinforcement learning (RL) into a single framework. Mainly authors have focused on Advantage-Weighted Actor-Critic (AWAC), which learns good initial policy from demonstration, and QT-Opt, which can learn a task from scratch with a large amount of training data, to develop AW-Opt. To make AW-Opt fair well with IL plus RL, authors have modified data sampling, exploration strategy, and action selection in the Bellman equation. Authors have shown that these modifications helped robotic manipulators to solve 5 different robotic tasks, consist of both simulator and real-world, when provided with initial demonstration and then finetuned using RL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Figured out the limitation and advantages of each technique and combined them well to figure out technique which works fairly well for IL and RL setup\n* Provide ablation study to show the importance of each strategy\n* Carried out experiment both in simulation and real world robotic platform.\n\nWaekness\n* It felt to me that method was overly optimized for the task in consideration. I would love to know how the method works for wide range of IL  + RL task, ranging from learning policy for solving video games to complex manipulation (like opening doors, cabinets, etc) \n* No good explanation for choosing some of the hyper parameters, like 20-80 % selection in Hybrid Actor-Critic Exploration", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am providing the weak accept based out of the following +ve and -ve points\n\n### +ve points\n* The paper proposes new way of performing IL and RL based out of vision inputs in sample efficient manner. \n* They have shown that the method they are proposing, is able to solve 3 robotic task in real world based out of demonstration collected for those tasks\n* Detailed ablation study is provide which shows that each of the strategy is contributing significant amount to the overall success\n\n### -ve points\n* seems method is overly optimized for task in consideration\n* explanation for some of the hyper parameters are missing", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "One of the major concerns I have is that in this paper authors are proposing a new algorithm for solving IL + RL problem, but has only shown result based out of the one vision-based robotic grasping task with some amount of variation in what needed to be grasped. However, it would be interesting to know how this algorithm works for some complex manipulation tasks like opening microwave, cabinet, door or push stacking objects. "}}, {"review_id": "-W9_hOo9IQa", "reviewer": "Reviewer_qTLQ", "summary": "This paper proposes a hybrid method AW-Opt, built on two previous methods AWAC and Qt-Opt, for scalable robotic manipulation with imitation learning and reinforcement learning.\nThe resulting method is driven by empirically analyzing the design choices in both baseline methods and developing modifications that address the issues in the original components. Specifically, the proposed changes are balanced data for training both the critic and actor, improved exploration by switching between the actor and CEM policy, and bellman update target using maximum Q value optimized by CEM with \"elites\" from the actor. \n\nWith extensive experiments in simulation and real-world, the paper demonstrates that the proposed method significantly improves over the baselines AWAC and Qt-Opt in challenging tasks where offline learning and online finetuning are needed for achieving high performance. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The presentation of the paper is clear\n2. The modifications proposed in this paper are well motivated and they are effective as shown by extensive experiments\n3. The complete algorithm archives strong empirical performance\n\nWeaknesses/Suggestions\n1. It missed some details that are important for interpreting the assessing the experimental results: \n\n  a. Most figures showed the success rate but did not give details on how many trials the rate was based on.  \n  b. All learning curves were shown without mentioning what the line and shaded region represent. Is it the mean and standard deviation? If so, how many runs/random seeds were they based on?\n\n2. Can the authors please clarify the following design choices in the paper?  \n  a. The “switcher” exploration in Section 4.3 used 80% critic and 20% actor. I wonder if this choice is backed up by some experiment. How sensitive is the result to this choice of ratio?  \n  b. The offline datasets for the five tasks contain different positive/negative ratios. I wonder if this is a somewhat arbitrary choice (in terms of that the extreme case of 100% negative is bad but other ratios are fine). How sensitive is the result to this choice of ratio when constructing the dataset?  \n  c. The loss function shown in the appendix has weights for subactions. It would be great if the values of the weights are given.\n\n3. The legend in Figure 6 does not quite match the text description. “AW-Opt - MaxQ” in the figure becomes “AW-Opt without ActorCandidate” in the text. It seems to me that they are not the same thing: we recover the AWAC target value without the MaxQ but “without ActorCandidate” can be interpreted as AW-Opt + MaxQ. It confuses me as to which one the paper meant to refer to. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I base my recommendation mainly on the strong experimental results and the potential impact of the proposed ideas and insights. In the robotics domain, simple but effective methods often have great impacts. This paper falls into this category. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please refer to the items discussed in the weaknesses/suggestions section. My main concern is in the missing details of the learning curves.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are mixed. On the one hand, all reviewers appreciate the study on understanding why two well-known existing algorithms struggle to incorporate demonstrations well, and on proposing specific improvements to these algorithms accordingly. On the other hand, two reviewers question whether the conclusions would apply to other tasks, beyond the grasping task studied in this paper, and Reviewer qeFD notes a number of other minor potential weaknesses. Authors should address these issues in a rebuttal.

-------

Following the reviews, the authors have provided further experiments and have addressed the issues raised by the reviewers. All three reviewers now recommend acceptance of the paper. For the new navigation experiments (Figure 10) in the final paper, I encourage the authors to perform these over more than one random seed, as has been done for the manipulation experiments.
Decision: Accept (Poster)