Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation
Authors: I-Chun Arthur Liu
Shagun Uppal
Gaurav S. Sukhatme
Joseph J Lim
Peter Englert
Youngwoon Lee
Venue: CoRL2021 Poster (2021)
Keywords: Reinforcement Learning
Motion Planning
Robot Manipulation

Abstract: Learning complex manipulation tasks in realistic, obstructed environments is a challenging problem due to hard exploration in the presence of obstacles and high-dimensional visual observations. Prior work tackles the exploration problem by integrating motion planning and reinforcement learning. However, the motion planner augmented policy requires access to state information, which is often not available in the real-world settings. To this end, we propose to distill a state-based motion planner augmented policy to a visual control policy via (1) visual behavioral cloning to remove the motion planner dependency along with its jittery motion, and (2) vision-based reinforcement learning with the guidance of the smoothed trajectories from the behavioral cloning agent. We evaluate our method on three manipulation tasks in obstructed environments and compare it against various reinforcement learning and imitation learning baselines. The results demonstrate that our framework is highly sample-efficient and outperforms the state-of-the-art algorithms. Moreover, coupled with domain randomization, our policy is capable of zero-shot transfer to unseen environment settings with distractors.

Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation
I-Chun Arthur Liu2*
Shagun Uppal!*
Gaurav S. Sukhatme Joseph J. Lim
Peter Englert? Youngwoon Lee
CCognitive Learning for Vision and Robotics Lab
Robotic Embedded Systems Laboratory
University of Southern California
Abstract: Learning complex manipulation tasks in realistic, obstructed environ- ments is a challenging problem due to hard exploration the presence of obstacles and high-dimensional visual observations. Prior work tackles the exploration prob- lem by integrating motion planning and reinforcement learning. However, the motion planner augmented policy requires access to state information, which is often not available in the real-world settings. To this end, we propose to distill state-based motion planner augmented policy to a visual control policy via (1) visual behavioral cloning to remove the motion planner dependency along with its jittery motion, and (2) vision-based reinforcement learning with the guidance of the smoothed trajectories from the behavioral cloning agent. We evaluate our method three manipulation tasks in obstructed environments and compare it against various reinforcement learning and imitation learning baselines. The re- sults demonstrate that our framework highly sample-efficient and outperforms the state-of-the-un algorithms. Moreover, coupled with domain randomization, our policy capable of zero-shot transfer to unseen environment settings with distractors. Code and videos are available at https //ccll//all com/mopa pd.
Keywords:
Visual Policy Distillation, Motion Planning, Reinforcement Learning
Introduction
Solving complex manipulation tasks in obstructed environments is challenging problem in deep reinforcement learning (RL) since it requires precise object interactions as well as collision-fres movement across obstacles. To tackle this problem, prior works [1-3] have proposed to combine the strengths of motion planning (MP) and RI. safe collision-free maneuvers of MP and sophisticated contact-rich interactions of RI,, demonstrating promising results. However, MP requires access to the geometric state of an environment for collision checking, which is often not available in the real world, and is also computationally expensive for a real-time control. To deploy such agents in realistic settings, we need to resolve the dependency on the state information and costly computation of MP, such that the agent can perform task in the visual domain.
To this end, we propose a two-step distillation framework, motion planner augmented policy distilla- tion (MoPA-PD), that transfers the state-based motion planner augmented RL policy (MoPA-RL RL [1) into a visual control policy, thereby removing the motion planner and the dependency on the state information. Concretely, our framework consists of two stages: (1) visual behavioral cloning (BC [4]) with trajectories collected using the MoPA-RL. policy and (2) aasond RL training with the guid- ance of smoothed trajectories from the BC policy. The first step, visual BC, removes the dependency on the motion planner and the resulting visual BC policy generates smoother behaviors compared to the motion planner's ss jittery behaviors. Then, in the second step, we further improve the visual policy
"qqual contribution
1 "Gaurav Sukhatme holds concurrent appointments as a Professor at USC and as an Amazon Scholar. This paper describes work performed at USC and is not associated with Amazon.
1 AI Advisor NAVER Al Lab
Work done during internship at NAVER AT Lab
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Pre-training
Stage Visual Behavioral Cloning
Stage Il: Vision-based RL Training
State-based MoPA-RL Agent
Visual RL Agent
Smoothed Trajectories
MoPA-RL Trajectoris Dap
Export Replay Buffer R,
State based Action
State based Critic
State based Cribic Kd
Visual BC Agent
Motion Planner
E
Visin based Actor leg)
Envronment
(nt)
Agent Replay Buffer
Weight Initialization
lect Rollouts
Training Data
Our two-step framework distills the state-based MoPA-RI. agent into visual control policy. In stage I, we learn a visual BC policy using MoPA-RL trajectories, thereby ensuring smooth trajectories for the expert replay buffer Re, while removing the motion planner. In stage 2, we learn BC trajectory-guided vision-based RL agent with an asymmetric actor-critic algorithm using both expert replay buffer R. and agent replay buffer Ry To improve sample efficiency, initialize the actor and the critic of the vision-based agent with the weights of the BC policy and the MoPA-RL critic, respectively.
Figure 1:
using asymmetric actor-critic RL [5] directly from the image observations to overcome sub-optimality the MoPA-RL policy through self-exploration.
For efficient vision-based based RL training, our method leverages the experience of the BC policy and the state-based critic from MoPA-RL by initializing the critic network with the MoPA-RL RL critic and the actor network through BC pre-tras pre-training; BC-trajectory guided RL training using separate expert replay buffer; tuning the entropy coefficient to encourage exploitation (i.e., maximizing reward) over exploration (i.e., maximizing entropy).
In summary, contributions
follows:
We propose novel framework to learn visual control policy in cluttered scenarios by distilling state-space policy that uses a motion planner into an image-space policy by removing the motion planner dependency.
Our asymmetric visual policy leaning ensures high sample-efficiency using weight initialization followed by BC trajectory-gums guidedd RL with entropy coefficient tuning.
The distilled visual policy is capable of zero-shot domain transfer to unseen environments with visual domain randomization during training.
Our method outperforms the state-of-the-art methods in terms of success rate, sample efficiency, and path length on three manipulation tasks in obstructed environments.
Related Work
The objective of our work is to learn complex manipulation tasks in realistic and obstructed envi- ronments. Motion planner augmented RI methods [1-3) have shown promising results in solving complex tasks in obstructed environments by combining MP and RL. However, these methods cannot be deployed in real-world settings due to their dependency on state information. To remove this dependency, state-based policies must be transferred to the visual domain.
typical approach to distill state state-based policies into vision-based policies behavioral cloning (BC [4]). However, BC often suffers when encountering states unseen during training [6]. Therefore, prior works have proposed various methods to improve BC agents using offline RL [7] and online RL [8]. However, these works do not involve the distillation of motion planners.
Several recent efforts have been made towards distilling MP algorithms into neural motion planners using learning-basee based methods [9-13]. Most of these efforts can be broadly categorized into imitation

---

learning (IL) and RL paradigms. For IL, supervised learning on MP trajectories has been used to learn neural network-basee planners [10-14]. However. the performance of such supervised learning approaches are limited by the collected dataset and demonstrator's performance.
To discover better policies with additional interactions, various off-policy RI methods have been studied for superseding motion planners with neural network policies [13, 15, 16]. These approaches utilize expert trajectories stored in the replay buffer for guided exploration, which leads to better policies than experts [17-20]. However, these works learn either in unobstructed environments [17- 20] or on simpler tasks not involving object manipulation [13]. They also assume fully observable environments and learn policies in state space. In this work, we focus on complex manipulation tasks in obstructed environments using visual observations.
Moreover, most offline RI. approaches [21, 22] in sparse reward scenarios are coupled with hindsight experience replay (HER [23].. They empirically show that not using HER significantly hurts the agent's performance [13, 5, 16, 17]. However, HER is designed to efficiently train goal-conditioned policies for multi-goal RL, which makes unsuitable for tasks with multiple sequential steps without explicitly conditioning on goals. In contrast, our method successfully learns composite manipulation tasks in cluttered environments without using HER for tackling sparse rewards.
Method
Our goal is solve complex manipulation tasks in obstructed environments with visual inputs. While MP-based methods can efficiently solve such tasks, they are restricted to learning state- base-bbsed policy which is difficult to transfer real environments. Moreover, sampling- based motion planning incurs sigmificant computational costs, making it hard to integrate them into real-time controllers. Thus, we propose a method that distills an MP-augmented state-based agent into a visual policy, removing the MP and state dependency. We formally define our problem and introduce the augmeneddt RL. in Section 3.1. Then, we describe our two-step visual distillation approach in Section 3.2.
3.1 Preliminaries
Problem formulation We formulate problem as Markov Decision Process (MDP) M defined by a tuple (S, O, A, A, P, R, 7) that consists of a state space S, partial observation space O (visual inputs corresponding
states), action space A. transition function P : S x A S + R, reward function Rt initial state distribution Po, and discount factor The agent is represented as a policy T (arlor), which takes an action a under a visual observation on and receives reward r, = [[(s,, az), with the state transitioning to 8141- The goal of the agent is to maximize the expected discounted sum of rewards E (ss,a)= ETD >rr,. where T is the episode horizon.
Motion Planner-Augmented RL (MoPA-RL) To tackle manipulation tasks in cluttered envi- ronments, Yamada et al. [1] proposed to incorporate MP and RL via an MP-augmented MDP M = (8, A, P, R, Pa: 7) that consists of the state space S, enlarged action space A augmenting the direct action space with the MP action space A, augmeeted transition function P S x AxS + R. augmented reward function R(s, ä), initial state distribution Po. and discount factor ?.
MoPA-RL [1] learns policy no (ā, (se) on the augmented MDP M with an off-policy RL algorithm, SAC [24]. Precisely, given a state 8t, the policy predicts an action âz, which is defined as a robot joint displacement Aq, If the action lies within the direct action space A, it is directly executed by the controller as the agent performs sophisticated and contact-rich manipulations. However, when the actions are larger in magnitude (i.e. à E A), the probability of collisions in the presence of obstacles increases. Therefore, a sampling based motion planner, RRT- URT-Connect [25], is called to realize such actions with large joint displacements by computing collision-fre free paths. This allows the agent to efficiently explore obstructed environments while avoiding collision [1].
'M [1] defines the direct action space A = 1-Aguep. Agaup 14, where Agup is the maximum | - Aqua AqMe)* where AQMP is the motion planner action limit with Ague Agaep In MoPA-RI, actions in the direct action space a E A are directly applied as joint torgue while other large actions à E A = Á \ A invoke motion planning. We refer the readers to Yamada et al. [1] for more details.
joint displacement, and the enlarged MP-augmented action space as A =
3

---

3.2 Motion Planner Augmented Policy Distillation
Despite the advantages of MoPA-RL, its applicability to the real world is limited due to the high computational cost of MP and dependency on fully observable environment states. To this end, we propose visual distillation method, motion planner augmented policy distillation (MoPA- PD), that learns an mmaagebbased control policy (without MP) from a state-based MP-augmented policy by leveraging its rollouts and the learned state-based critic guidance. Concretely, our proposed method consists of two stages, as illustrated in Figure 1. Given a state-based MoPA-RL agent, we first use BC train vision-based actor with trajectories collected from the MoPA-RL policy (Section 3.2. 1), and then we further improve the vision-based agent via BC sereector-guuided asymmetrie actor-critic RL [5] with the visual BC actor and MoPA-R -RL critic (Section 3.2.2).
3.2.1 Stage 1: Visual Behavioral Cloning and Trajectory Smoothing
The MoPA-RL policy is learned with access to complete information about the environment states and also utilizes MP for executing large action steps without collisions. In this paper, we aim to utilize demonstrations from this planner-based policy and distill into a visual control policy, thereby deducting expensive MP computations and training the actor in visual domain. With the learned MoPA-RL. policy, we first collect multiple transitions d, low-level (direct) action space and store them into the MoPA-RL demonstration dataset Dmp {d1, d2, d3...., where d, (8,, O2, Oj, Ta, 11+1). Then, we train a visual BC actor Te (a](or) using observation-actinn pairs (o,, a.) from the dataset Dmp by minimizing the mean squared error.
Distilling the MoPA-RL trajectories using BC not only enables the BC actor to work directly on visual inputs but also reduces jerky motion planning behaviors of MoPA-RL RL,, which occur due to motion planner's priority on obstacle avoidance over trajectory smoothness [26]. By removing unnecessary. jerky movements through visual BC, the resulting trajectories become smoother and even shorter, and thus help learning consistent and smooth motions.
However, the BC actor often fails when it encounters states not seen during training due to covariate shift [6]. To improve the robustness of the policy, we further train the visual BC actor using RL with additional environment interactions. In other words, the BC actor can be a good starting policy for visual RL training and the trajectories collected from the BC actor can effectively guide exploration.
3.2.2 Stage 2: Vision-based RL with Asymmetric Actor-Critic
In the second stage, we further train the visual policy using RI. directly on image observations to enhance the robustness of the policy and overcome sub-optimality in the planner-based policy. For efficient RL. training, we adopt an asymmetric actor-critic architecture [5], where the actor acts based environment images and robot joint states while the critic learns from environment states. This architecture is motivated by learning transferable visual policies, which do not require state information during inference but benefit from them while learning in simulation. Our training procedure comprises of the following components:
Weight initialization of actor and critic networks Initializing actor and critic networks with suitable weights can accelerate RL training by providing more optimal rollouts from the actor and informative learning signals from the critic compared to the randomly initialized actor and critic [27, 28]. Especially when learning from pixels, suitable initialization can significantly improve sample efficiency [20, 29]. To this end, we propose to leverage the state-based agent's experience by utilizing its weights for initializing our asymmetric visual agent's networks, thereby bringing their initial distributions closer. Thus, the visual actor network (a(a lo:) is initialized with the weights of the visual BC actor learned in Section 3.2. I. We also initialize the critic (u(st, at) with the critic of MoPA-RL. Note that the action spaces of two critic networks are different, but initialization is still feasible since the action space of MoPA-RL is superset of the direct action space.
BC trajectory-guided asymmetric RL After initializing the actor and critic networks, we collect agent rollouts into the agent replay buffer R... As an RI. algorithm, we adopt Soft Actor-Critic (SAC [24]), an off- off-policy model-free RL algorithm for continuous control. We optimize our asym-

---

(a) Sawyer Push
(b) Sawyer Lift
(c) Sawyer Assembly
Manipulation tasks in obstructed environments from Yamada et al. [1]. (a) Sawyer Push: Push the red cube in the box to the green goal. (b) Sawyer Lift: Lift out the can inside the box. (c) Sawyer Assembly: Insert the table leg in the hole in the table top.
Figure 2:
metric visual policy To by maximizing the following objective:
J(mo) Σ (s. (Q (st, ar) a'l (To (+)):
(I)
where the temperature parameter a balances between exploration and exploitation using entropy H.
RL training with BC-initialized policy often quickly deviates from the original policy. To prevent this problem and guide exploration during RL, we update our RL agent not only with agent trajectories but also with BC policy (i.e. expert) trajectories [18]. Thus, we collect smoothed trajectories from our visual BC policy, (8;, 0s; 0g ri, 1;+1), ), in an expert replay buffer R.. Then, for each RL training iteration, we sample 1:3 transitions from Re and R.,, respectively. A separate expert replay buffer ensures guided exploration from the expert and also circumvents catastrophic forgetting after weight initialization. Note that we use the BC trajectories R. instead of the MoPA-RL data 2mp because jittery motion planning paths the MoPA-RL data make RL training difficult and sub-optimal.
Entropy coefficient tuning The performance of SAC is known be sensitive [30]. Since our asymmetric agent already starts with the well-trained BC actor and MoPA-RL critic, we initialize values lower than the final a obtained in MoPA-RL. This is to ensure that with prior knowledge in the state-based agent, the visual agent focuses more on maximizing rewards over exploration. We describe the hyperparameter choice and implementation details in Section and Section B.
In summary, we propose a two-step visual distillation method for a state-based MoPA-RL agent using visual BC followed by BC trajectory-guidet vision-based RI. remove the dependency MP and environment states. For efficient RL training, our method leverages weight initialization of the actor and critic, the expert replay buffer of BC smoothed trajectories, and entropy coefficient tuning.
Experiments
In this paper, we propose to distill a motion planner augmented policy into a visual control policy for complex manipulation tasks in obstructed environments. In our experiments, aim to answer following questions: (1) Does IL efficiently learn policies for obstructed environments image space? Moreover, is naively combining IL with RL sufficient for solving complex tasks? (2) Is distillation better than directly learning visual policy using MP? (3) Does our approach, MoPA-PD, efficiently learn a visual policy using prior state-based exploration knowledge? (4) Is our visual policy capable of domain transfer and robust to unseen distractors?
4.1 Environments
We evaluate our approach on three obstructed environments (see Figure/ 2) from Yamada et al. [1], simulated using the MuJoCo physics engine [31]. We use a 32x32 image visual observation.
5

---

Sawyer Push
Sawyer Lift
Sawyer Assembly
ASR f
AEL L
ADR f
ASR t
AEL
ADR f
ASR
AEL
ADR
MoPA-RL
98.2
111.0
52.7
95.0
109.2
52.7
99.8
63.3
82.4
BC- Visuallll
99.4
118.0
46.9
62.0
108.8
34,6
97.0
115.1
50.4
Asym. SAC
0.0
250.0
0.0
0.0
250.0
0.2
0.0
250.0
3.4
MoPA-Asym. SAC
0.0
250.0
0.0
0.0
250.0
0.0
0.0
250.0
0.0
CoL
0.0
250.0
0.0
29.8
173.3
9.7
0.0
250.0
0.0
CoL w/ BC smoothing
0.0
250.0
250.0
0.0
250.0
0.0
Ours w/o BC smoothing
100.0
34.0
108.7
99.4
43.6
100.7
0.0
250.0
0.0
Ours
100.0
32.0
110.8
99.0
42.0
101.7
100.0
61.7
84.5
Table I: Average success rate (ASR), episode length (AEL), and discounted return (ADR) of our method and baselines averaged over five seeds. Each method is evaluated after 3M environment steps. MoPA-RL [1] is the expert agent trained with state-based policy. All baselines below horizontal line are trained with asymmetric actor-critic for fair comparison. Maximum horizon is 250 for all tasks.
Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside the box, needs to reach an object placed inside the box and push it to the goal location within the box.
Sawyer Lift: The Sawyer arm must reach an object placed inside a box, grasp. and lift outside the box, avoiding collisions. The Sawyer arm is always initialized outside the box.
Sawyer Assembly: The Sawyer arm needs to assemble the fourth leg (already attached to the gripper) of table into the vacant hole while avoiding collisions to other three legs. This environment is built upon the IKEA furniture assembly environment [32].
For Sawyer Push and Sawyer Assembly, the agent receives reward proportional the distance between the end-effector and object, or the object and goal position when the distance is less than e. We use e = 0.1 for Sawyer Push and 0.3 for Sawyer Assembly while the distance between initial and goal state is around 1. 1.2. An episode is considered successful when the distance between the cube and goal (Sawyer Push) or the peg-head and goal (Sawyer Assembly) is less than 0.05. For Sawyer Lift, we use the reward function similar to Fan et al. [33], where reward is defined for all three intermediate stages of the task: reach, grasp and lift. A bonus reward signal of 150 received upon successful task completion in all the environments. We refer readers to Section D for more details.
4.2 Baselines
We compare our method with the following baselines to evaluate the merits our approach:
MoPA-RL [1]: A MP-augmented policy, for learning large displacement actions with MP and smaller actions with RI. This policy also serves as our state-based expert agent.
Visual Behavioral Cloning (BC-Visual): A behavioral cloning policy trained the image- action pairs collected from MoPA-RL. trajectories.
Asym. SAC [5]: asymmetric actor-critic method, trained using SAC where the critic is learned in the state space and the actor is learned in the image space.
MoPA Asym. SAC: MoPA-RL policy [1] learned using the asymmetric framework. Note that this method still uses motion planner with augmented action space. This method is direct attempt at learning a visual policy using a MP-augmented framework [1].
CoL [29]: A policy learned using the Sycle-of-Leante Learning framework, which is the state-of-the-ar
algorithm for learning from demonstrations (Lff)).
[29]: A policy similar to CoL [29], with BC for trajectory smoothing. Ours (w/o BC Smoothing): A policy learned using our approach described in Section 4 without BC trajectory smoothing, i.e., we directly use MoPA-RL RL trajectories in R..
CoL (w BC Smoothing)
4.3
Evaluation
We compare our approach against baselines on the following evaluation metries averaged over five random seeds and 100 unseen episodes per seed:

---

Average Discounted Rowards
Average Discounted Rewards
Average Discounted Reward re
60-
66-
44
22
0.75
2:25
3000
6.00
2.25
0.00
0.75
1,50
Environment (3M)
Environment steps (3M)
Environment (3M)
(a) Sawyer Push
(b) Sawyer Lift
(c) Sawyer Assembly
Figure 3: Learning curves of our method compared baselines. All methods are trained for 3M environment steps. For the methods that require MoPA-RL, we train MoPA-RL for 1M steps and then train the methods for 2M steps (total 3M) for fair comparison. Our method solves all the tasks with the highest average discounted return. Our method w/o BC smoothing can solve two tasks with slower convergence compared Ours, but fails on Sawyer Assembly.
Average Success Rate (ASR) the average number of successful episodes.
Average Episode Length (AEL) is the average length of successful episodes.
Average Discounted Return (ADR) is the average discounted sum of rewards E/s R'[s dz), with 7' being the episode horizon. An episode completed in more time steps has a lower discounted reward, due exponentially discounted reward with y 0.99.
4.4 Results
Comparisons with baselines Figure 3 illustrates the learning curves of method and all other baselines using discounted rewards against the number of training steps. As per the trend, our method far more sample-efficient and outperforms all the baselines. Asym. SAC [5] fails directly learn visual policy for Sawyer Lift and Sawyer Assembly in obstructed environments, where exploration is hard. Moreover, MoPA Asym. SAC, which direct attempt at learning a visual policy while using MP, does not successfully learn to solve the tasks. CoL [29] which has a straightforward combination of RL and BC objective for actor optimization receives partial rewards for some tasks at around 1.5-2M steps, much slower than our method's 's convergence.
As reported in Table BC achieves decent performance for Sawyer Push and Assembly, but not for the Sawyer Lift task. However, it achieves high AEL and low ADR for all tasks, showing its inefficiency to solve the task fast. This is because BC is bound perform as good as the expert demonstrations (here from MoPA-RL) at its best. In contrast, our method optimizes trajectories beyond expert signals and achieves significantly lower AEL and higher ADR compared baselines.
Ablation on BC trajectory smoothing Our method without BC smoothing has higher variance across seeds and converges slower than our method for two tasks (see Figure 3). For Sawyer Assembly, it does not learn to solve the task at all. In short, BC smoothing of the motion planner trajectories is important for our method to work across all environments. This is because the motion planner based trajectories are usually jittery and non-smooth. Using behavioral cloning refines each transition, thereby making the entire trajectories much smoother and help in efficiently training the RI. agent.
Ablation on weight initialization We compare our method with and without actor-critic initializa- tion in appendix, Figure 7 and do not observe any episode success until 1 1.2M environment steps for the latter. This shows the importance of our proposed weight initialization for learning in the visual domain. We further elaborate the ablation results in Section A.2.
Ablation on entropy coefficient tuning We examine the effect of different values of a on the trade-off between entropy maximization and reward maximization for the SAC objective as shown in Section A. Compared to higher values, smaller values of improve the sample efficiency during visual policy learning. Since we utilize prior knowledge from the state-based agent, we use smaller alpha to exploit the previously acquired knowledge instead of exploring the entire state space again.
7

---

(a) Domain randomization
(b) Original environment
(c) Scenario 1
(d) Scenario
Illustration of environments for domain transfer. (a) Domain randomized environments are used for training; For testing, use (b) original environment without randomization, (c) Scenario I with distractors such as unseen cubes (blue and magenta), walls, floor, and furniture, and (d) Scenario similar to Scenario along with variation in size of the table and texture of the box and table.
Figure 4:
Sawyer Push AEL 4
Sawyer Lift
Sawyer Assembly
ASR
ADR
ASR
AEL 4
ADR
ASR f
AEL + ADR t
Original env
99.7
39.1
103.1
99.3
37.3
106.5
100.0
49.7
93.4
Scenario Scenario 99.3
100.0
40.4
102.2
96.7
38.3
103
100.0
47.9
94.8
43.4
99.04
97.3
37.0
104.7
100.0
47.7
95.0
Table 2: Evaluation metrics for domain transfer unseen environments, illustrated in Figure 4.
4.5 Policy Transfer to Different Domains
In this experiment, we learn policy with domain randomization (DR) verify its domain transfer capabilities. DR is a promising approach for modelling transferable policies using a multitude of variations in simulation during training [34, 35], which makes the policy invariant to changes that insignificant for the task. For this work, we randomize the simulation environment using different textures, colors, and lighting conditions for training shown in Figure 4 and Figure 10. We then test our DR policy on three unseen scenarios, as illustrated in Figure 4 comprising of (1) the original environment without any randomization; (2) Scenario with realistic background distractions (e.g., furniture, walls, and floor) and unseen distractor cubes (blue and magenta) near the target cube (red); and (3) Scenario with additional changes in the texture of the table and box, and different size of the table. Our policy attains more than 96% ASR in all three unseen scenarios for all the tasks (see Table 2). This illustrates the robustness of our proposed framework learning transferable visual policies, robust to unseen distractors in a sample-efficient manner.
Conclusion
In this paper, introduce two-step distillation method for learning manipulation tasks in obstructed environments in the visual domain. In step one, we use a MP-augmented RL policy as the state-based expert and subsequently learn visual BC agent from it, removing the motion planner dependency. In step two, we learn a vbsion-based agent using an asymmetric actor-critic framework. This step is further expedited via proper weight initialization, BC trajectory-guided RL training, and entropy coefficient tuning, making our method highly sample-efficient Our visual policy combined with domain randomization demonstrates successful zero-shot transfer to unseen environments with new visual domains and distractors. Beyond zero-shot transfer, fine-tuning the policy in the real world by learning vision-basee based critic or applying simulation-to. to-reell techniques [36] is our definite future work to realize simulation-to- ro-ioll-tt--rlll-- transfer.
Acknowledgments
This research is supported by the Annenberg Fellowship from USC, NAVER AT Lab, and NSF NRI-2024768. We thank our colleagues from the CLVR lab and RESL. for the valuable discussions that considerably assisted the research.
8

---

References
[1] J. Yamada, Y. Lee, G. Salhotra, K. Pertsch, M. Pflueger, G. S. Sukhatme, J. J. Lim, and P. En- glert. Motion planner augmented reinforcement learning for robot manipulation in obstructed environments. In Conference on Robot Learning, 2020.
[2] Xia, C. Li, Martin-Martin, O. Litany, A. Toshev, and Savarese. Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation. In IEEE International Conference on Robotics and Automation, 2021.
[3] G. Matheron, N. Perrin, and O. Sigaud. Pbcs: Efficient exploration and exploitation using synergy between reinforcement learning and motion planning. In International Conference on Artificial Neural Networks, pages 295-307. Springer, 2020.
[4] D. A. Pomerleau. Alvinn: An autonomous land vehicle a neural network. In Advances in Neural Information Processing Systems, pages 305-313, 1989.
[5] L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-based robot learning. In Robotics: Science and Systems, 2018.
[6] S. Ross, G. J. Gordon, and Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2011.
[7] A. X. Lee, C. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg, A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, C. Fantacci, J. F. Chen, A. Raju, R. Jeong, M. Neunert, A. Laurens, Saliceti, F. Casarini, M. Riedmiller, R. Hadsell, and F. Nori. Beyond pick- and-place: Tackling robotic stacking of diverse shapes. In Conference on Robot Learning. 2021.
[8] R. Jeong, Y. Aytar, D. Khosid, Y. Zhou, J. Kay, T. Lampe, K. Bousmalis, and F. Nori. Self- supervised sim-to-real adaptation for visual robotic manipulation. In IEEE International Conference Robotics and Automation, 2020.
[9] A. H. Qureshi and M. C. Yip. Deeply informed neural sampling for robot motion planning. IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 6582-6588, 2018.
[10] A. H. Qureshi, M. J. Bency, and M. C. Yip. Motion planning networks. In IEEE International Conference on Robotics and Automation, pages 2118-2124, 2019.
[11] M. J. Bency. Towards Neural Network Embeddings of Optimal Motion Planners. PhD thesis, UC San Diego, 2018
[12] M. Pfeiffer, M. Schaeuble, J. Nieto, R. Siegwart, and C. Cadena. From perception to decision: data-driven approach to end-to-end motion planning for autonomous ground robots. IEEE International Conference Robotics and Automation, pages 1527-1533, 2017.
[13] T. Jurgenson and A. Tamar. Harnessing reinforcement learning for neural motion planning. In Robotics: Science and Systems, 2019.
[14] R. Strudel, R. Garcia, J. Carpentier, J. Laumond, I. Laptev, and C. Schmid. Learning obstacle representations for neural motion planning. In Conference on Robot Learning, 2020.
[15] H. Ha, J. Xu, and S. Song. Learning a decentralized multi-arm motion planner. In Conference on Robot Learning. 2020.
[16] S. Luo, H. Kasaei, and L. Schomaker. Self-imitation learning by planning. In IEEE International Conference on Robotics and Automation, 2021.
[17] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration reinforcement learning with demonstrations. IEEE International Conference on Robotics and Automation, pages 6292-6299, 2018

---

[18] T. Hester, M. Vecerík, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Amold, Agapiou, J. Z. Leibo, and A. Gruslys. Deep q-learning from demonstrations. In Association for the Advancement of Artificial Intelligence, 2018.
[19] M. Vecerík, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. ArXiv, abs/1707.08817, 2017.
[20] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learn- ing Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Robotics: Science and Systems, 2018.
[21] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. ArXiv, abs/2006.0955 2020.
[22] S. Endrawis, G. Leibovich, G. Jacob, G. Novik, and A. Tamar. Efficient self-supervised data collection for offline robot learning. In IEEE International Conference on Robotics and Automation, 2021.
[23] M. Andrychowicz. D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, 2017.
[24] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference Machine Learning, 2018.
[25] J. Kuffner and S. M. LaValle. Rit-connect: An efficient approach to single-query path planning. In IEEE International Conference on Robotics and Automation, 2000.
[26] Z. Zhu, E. Schmerling, and M. Pavone. A convex optimization approach to smooth trajectories for motion planning with like robots. IEEE Conference on Decision and Control, 2015.
[27] G. Cruz, Y. Du, and M. E. Taylor. Pre-training neural networks with human demonstrations for deep reinforcement learning, ArXiv, abs/1709. 2017.
[28] C. Anderson, M. Lee, and D. L. Elliott. Faster reinforcement learning after pretraining deep networks to predict state dynamics. International Joint Conference on Neural Networks, 2015.
[29] V. G. Goecks, G. Gremillion, V. Lawhern, J. Valasek, and N. R. Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in sparse reward environments. International Conference on Autonomous Agents and Mulli-Agent Systems, page 465-473, 2020.
[30] Y. Wang and T. Ni. Meta-sac: Auto-tune the entropy temperature of soft actor-critic via metagradient. In 7th ICML Workshop on Automated Machine Learning, 2020.
[31] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 6026-5033, 2012.
[32] Y. Lee, E. S. Hu, and J. Lim. IKEA furniture assembly environment for long-horizon complex manipulation tasks. In IEEE International Conference on Robotics and Automation, 2021.
[33] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Creus-Costa, S. Savarese, and L. Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning, pages 767-782, 2018
[34] J. Tobbin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. IEFF/RSJ International Conference on Intelligent Robots and Systems, pages 23-30, 2017.
[35] M. Sheckells, G. Garimella, S. Mishra, and M. Kobilarov. Using data-driven domain random- ization to transfer robust control policies to mobile robots. IEEE International Conference on Robotics and Automation, pages 3224-3230, 2019.
[36] G. Zhang, Zhong, Y. Lee, and J. J. Lim. Policy transfer across visual and dynamies domain gaps via iterative grounding. In Robotics: Science and Systems, 1.21.
10


Reviews:
[{"review_id": "z253Pwm3zph", "reviewer": "Reviewer_acdh", "summary": "This paper proposes a technique for transferring a learned policy from privileged state information to images. The method can complete cluttered manipulation tasks after about 1 million training steps (a few days of real robot time), whereas appropriate baselines are unable to make any progress. This method combines two key ideas from prior work: the MoPA-RL framework which can solve cluttered manipulation tasks with privileged state information, and asymmetric actor-critic, which allows for efficient transfer to a visual policy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-An elegant combination of existing ideas\n\n-Compelling results on difficult manipulation tasks\n\n-Addresses an important problem -- efficiently learning a visuomotor policy for complex manipulation tasks\n\nWeaknesses:\n\n-No real robot experiment. Although the pandemic made this difficult, without such an experiment the suggestions/claims about sim2real transfer are not supported. There are many things other than textures, colors, and lighting which vary between simulations and the real world, such as physical properties, geometric differences, controller differences, etc. So the results for evaluating transfer are not convincing.\n\n-It seems the actor still requires privileged state information, even during distillation. This is potentially a big limitation, because further training on a real robot is not possible without training a visual critic from scratch. To address this, perhaps one could train a visual critic during this whole process (using both step 1 and step 2 data, via simulation rendering + D.R.)? Then at the end you could conceivably fine-tune in the real world with efficiency boosts from all the pretraining.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "With the addition of a real robot result quantifying success on a task using the proposed method, I would change to a strong accept. This paper combines existing methods elegantly and has strong results. However, a real robot experiment is missing, and I do not see any important insights or theoretical advances. There are a few components of the method which I believe limit its usefulness, which I discuss further in the Issues section.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "-Since the RRT is going to produce different paths for the same/similar start & goal based on the random seed, it seems like you could run into issues with BC based on this, especially if the policy network outputs a single Gaussian over the action vector. Does the method have a way to compensate for this?\n\n-Why does BC cause smoothing, and seemingly do something smarter than averaging over actions (which could cause issues in some environments)? Why isn’t multimodality of actions an issue? Are there cases where it is? Answering some of these questions would strengthen this paper.\n\n-I find the description of the MoPA vs direct action spaces as “different spaces” misleading. They have different domains, but they aren’t different spaces. A reader might think you’re adding additional dimensions to the action vector space, rather than expanding the domain of the action vector. \n\n-I also do not entirely agree that the different action spaces “lead to different transition function … and reward function” (line 117). For the actions within the direct action subset, the transition and reward are the same, inputs and outputs shouldn’t change. In fact, that’s the reason why transferring the critic works, because the transition/reward (and hence Q-value) functions are unchanged. This is a minor point that could be clarified in the text.\n\nQuestions:\n\n-For table 1, how many environment steps of training were used? I could not find that number. Perhaps put it in section 4.3. I also wonder, more generally, how much success in MoPA pretraining is needed? Does it still work if you stop pretraining before the task is “mostly solved”?\n\n-The discussion of HER was not sufficient. I did not follow the explanation of why it’s not applicable. From a quick read of the HER paper, it seems like it should be applicable.\n\nMinor comments:\n\n-On line 261, consider removing “extremely”, I would not consider one million time steps extremely sample efficient. Perhaps replace with “far more sample-efficient than ...”\n\n-Table 1: in the Sawyer Lift experiment, you bolded “Ours” but “Ours (w/o BC smoothing)“ had a better score, I think that should be bolded instead.\n\n-Table 1&2: consider adding Up and Down arrows to indicate which metrics should be high and which should be low. That would save the reader from having to go back and re-read what the acronyms mean.\n\n-Consider running more seeds, 3 seems like too small of a number to have much confidence in.\n\n-There is a comma missing on line 61. I would have written it “in terms of success rate, sample efficiency, and average path length”\n\n-Section A.3 is not particularly helpful because it’s hard to see visually which is smoother. Perhaps show only one trajectory, or split the graphs up more. You could also try connecting the points with lines. More importantly, what is being graphed? Is it end-effector position? That’s not explicitly state anywhere, I just guessed.\n"}}, {"review_id": "yTzujyAeB6y", "reviewer": "Reviewer_M4xK", "summary": "This paper proposes a learning approach for manipulation tasks in obstructed environments. \nIt has a hybrid aspect, where a sampling-based (thus model/state-based) motion planning algorithm is combined with a model-free reinforcement learning formulation. \nAs an extension of a prior work, here the focus is on obstructed environments, for which RL methods are prone to fail, whereas sampling-based motion planners are effective. \nHowever, switching to a sampling-based motion planner during execution or learning is costly and also using them in the real-world is harder due to their dependency on full state knowledge. \nWith this motivation, a two-stage learning approach is proposed. \nInitially a policy is learned based on the state-space representation using motion planning with RL.\nThen, an asymmetric actor-critic framework is used to train a policy that is now observation-based, i.e. visual perception data. \nThe initially learned state-based policy is imitated by behavioral cloning to provide (smoothed) expert data for training the observation-based policy.\nThe results demonstrate the efficiency of the proposed framework on three manipulation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper showcases the strengths of combining model-based methods with model-free approaches. In that regard, the motivation of the proposed approach is clearly explained w.r.t. the target scenarios, and the execution of the framework is technically sound and justified by several manipulation tasks.\n\nThe only slight weakness is the domain randomization experiments, as the distractors seem to be weak, and there isn't much analysis on the performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a novel approach combining state-based motion planners with model-free reinforcement learning methods for tackling challenging robot manipulation tasks. Even though it is an incremental update on prior work, the extension is well motivated and justified by demonstrating its performance on a range of scenarios. Overall, the paper is well structured and written. It can be further improved by a critical discussion on the limitations of the proposed approach and a deeper analysis on the domain randomization experiments. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "My comment on domain randomization mentioned above (Strengths & Weaknesses) could be an improvement point.\n\nAnother recommendation is to include some limitation and discussion of the proposed approach. Such a critical evaluation of the overall architecture, where it fails, for which tasks it might be applicable (or not), what are lessons learned, etc. would be valuable for the community. Especially, for example the dependency of the success of the whole framework to the 'weight initialization' is highly interesting, which might be further elaborated."}}, {"review_id": "TptbDABRIx7", "reviewer": "Reviewer_rTHv", "summary": "This paper presents an approach for distilling motion-planner augmented policies that operate in state space into visual control policies that operate without augmented actions from a motion planner. The first stage of this approach is to use Motion Planner Augmented Reinforcement Learning (MoPa-RL) to train an actor and critic model that operates in state space and has augmented actions from a motion planner to learn a policy to complete a task. The second stage of this approach uses behavioral cloning to train an actor network that operates in visual space and without a motion planner by using demonstrations from the expert MoPa-RL agent. The third stage trains an asymmetric actor-critic model to learn a visual motor policy and state-based critic that do not use the augmented action space by using demonstrations from the behavioral cloning agent and weight initializations from the MoPa-RL expert agent. The authors conduct experiments in 3 object manipulation domains (object pushing, object lifting, and object assembly) that involve obstacles and visual obstructions, and compare their full method to ablated versions of their method as well as other state-of-the-art RL and learning from demonstration approaches. This paper also demonstrates the robustness of their approach to domain randomization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper presents a novel and interesting approach for distilling a motion-planner augmented policy that operates in state space into a visual control policy that does not rely on motion planning at all. The method is also extremely clear and well-presented, and the reviewer commends the authors for their clarity. Overall, this method is a very good contribution for enabling motion planners to be useful for learning visual motor policies without requiring a motion planner or hand-crafted state space to be used at inference time. Overall this paper and method deserve praise and could provide a good contribution to the field, and the reviewer really enjoyed reading the paper.\n- This paper includes various ablations of their method and plenty of comparisons to other state-of-the-art approaches for this robot policy learning task, and report quantitative results on average success rate, episode length, and the discounted returns.\n-The figures in the paper are clear and useful for understanding the method, robot domains, and results.\n\nWeaknesses:\n- This proposed approached requires several stages of environment interactions that are unaccounted for the in quantitative results, which questions how sample efficient this method really is. The first stage requires MoPA-RL to interact and learn from the environment. The second stages requires behavioral cloning to occur (to get a smoother policy), and then they also collect more samples using the behaviorally cloned agent before doing the final stage of learning with the asymmetric actor, which further requires environment interactions. However, it appears none of the results reported in this paper appear to include how long these previous stages up to the last one take, or at least it is not clear from the graphs at which part each stage occurs. The authors should be explicit in their graphs about how many samples each of the stages require if they wish to make claims that their overall method is more sample efficient than other approaches. This is especially important since the authors emphasize how important the BC step is for good results (re: Sawyer assembly task)\n- The authors claim in the related work that \"However, using HER is not applicable for composite tasks involving goals associated with multiple sequential steps\", but this statement needs to be justified since it is not clear why this would be true, since rewards can just be relabeled based on whether they achieve a subgoal (defined by composite task) or not. \n- The authors should clarify what the state space was used in the experiments, and they should also clarify the size of the images used for the observations.\n- In section 3.3.1, the authors claim to store transitions from MoPA-RL into a demonstration dataset for later stages. Does this include transitions that include motion-planner augmented actions?\n- The authors claim they conduct their experiments in a sparse reward setting, but in the the tasks they use a dense reward around an epsilon ball of the goal proportional to the distance, they should specify how large epsilon is. \n- The authors do not specify what the gamma value is, which is important for the sparse reward setting because if it is not less than 1, then there is no useful distinction between the ASR and ADR metric.\n- The ASR metric for MoPA-RL against the proposed method across all three tasks is extremely similar (at max, a ~5% improvement in the lift task, and only a ~1% improvement in the other two tasks). Although AEL and ADR are improved, the ASR is the most interesting metric when considering whether these methods can operate in tasks with obstacles where vision is important for completing it. This makes the results much less convincing: the authors should choose a task where MoPA-RL is not able to reliably complete the task to demonstrate the importance of this distilling approach (although the reviewer recognizes it is useful regardless of success rate since it removes motion planning from the loop).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a novel and interesting approach for distilling motion-planner augmented policies into visual motor policies, and is very easy to read and is overall enjoyable. The reviewer really enjoys this paper and the contribution it offers, but the the experimental validation of the sample and computational efficient of this method due to the MoPA-RL expert and BC stages is not included and brings into question the efficacy of this method. There are also other important experimental details that should be included. Overall this paper has the potential to be a good contribution to the field (and is very accessible and easy to read), but requires further experimental validation for each of the stages and needs other experimental details to be included before publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The authors need to explicitly validate the sample complexity for each of the stages of the proposed approach, and they also need to include the experimental details mentioned in the \"Weaknesses\" previously mentioned. If these are addressed, then this paper stands to present a good contribution and deserve publication."}}]

Author rebuttal:

Meta-review: All three reviewers were generally positive about this paper. They agree that the method is novel and interesting, albeit somewhat incremental and an "elegant combination of existing ideas”. Reviewers comment that the paper is particularly well written. However, reviewers do raise some important points that should be addressed in a rebuttal. For example, further details on the sample and computational efficiency are important for a full evaluation, and some insights into the limitations and failure cases would be helpful. I would like to see the authors address the comment from Reviewer acdh about whether the actor still requires privileged state information during distillation, and how this relates to implementation on a real robot. Reviewers comment on the lack of real-world experiments, particularly given that sim-to-real transfer features prominently in this work, which is an important limitation. However, I do not necessarily expect authors to provide real-world results for the paper to be accepted. Overall, this is a good paper but I encourage authors to address the reviewers' criticisms and major queries in a rebuttal, in order to maintain good reviews.

------

Following the reviews, the authors have addressed a number of issues raised by the reviewers, and new experiments with a visual critic have been provided. There is still concern about the applicability of the method to real-world tasks, given the lack of real-world experiments and the sub-optimal results for the visual critic. However, the reviewers still found the overall idea appealing and with potential for interesting future work, and so all three reviewers recommended to accept the paper.
Decision: Accept (Poster)