Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation
Authors: Charles Sun
Jędrzej Orbik
Coline Manon Devin
Brian H Yang
Abhishek Gupta
Glen Berseth
Sergey Levine
Venue: CoRL2021 Poster (2021)
Keywords: mobile manipulation
reinforcement learning
Reset-Free

Abstract: In this paper, we study how robots can autonomously learn skills that require a combination of navigation and grasping. Learning robotic skills in the real world remains challenging without large scale data collection and supervision. Our aim is to devise a robotic reinforcement learning system for learning navigation and manipulation together, in an autonomous way without human intervention, enabling continual learning under realistic assumptions. Specifically, our system, ReLMM, can learn continuously on a real-world platform without any environment instrumentation, without human intervention, and without access to privileged information, such as maps, objects positions, or a global view of the environment. Our method employs a modularized policy with components for manipulation and navigation, where uncertainty over the manipulation success drives exploration for the navigation controller, and the manipulation module provides rewards for navigation. We evaluate our method on a room cleanup task, where the robot must navigate to and pick up items of scattered on the floor. After a grasp curriculum training phase, ReLMM can learn navigation and grasping together fully automatically, in around 40 hours of real-world training.

Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation
Charles Sun", Jędrzej Orbik", Coline Devin, Brian Yang,
Abhishek Gupta, Glen Berseth, Sergey Levine
(charfessuunjeeezzeejor......... howpreicoooim.bbbarananaaaaee briaahyagg.dbiiuuuggghrr.. weaprehhsseeeeieeeeeeeeeeeeeeeee @berkeley.edu
Berkeley Al Research
Abstract: We study how robots can autonomously learn skills that require a com- bination of navigation and grasping. While reinforcement learning in principle provides for automated robotic skill learning, practice reinforcement learning in the real world is challenging and often requires extensive instrumentation and supervision. Our aim is to devise robotic reinforcement learning system for learning navigation and manipulation together, in autonomous way without hu- man intervention, enabling continual learning under realistic assumptions. Our proposed system, ReLMM, can learn continuously out any environment instrumentation, without human intervention, and without access to privileged information, such maps, objects positions, global view of the environment. Our method employs a modularized policy with components for manipulation and navigation, where manipulation policy uncertainty drives exploration for the navigation controller, and the manipulation module provides rewards for navigation. We evaluate our method on room cleanup task, where the robot must navigate to and pick up items scattered on the floor. After grasp curriculum training phase, ReLMM can learn navigation and grasping together fully automatically in around 40 hours of autonomous real-world training.
real-world platform with-
Keywords: Mobile Manipulation, Reinforcement Learning, Reset-Free
1 Introduction
Learning-base approaches have the potential to bring robots into open-world environments with end-to-end vision-based policies that can perform tasks without external instrumenta- tion, AR- ARttagging, or expensive sensors. Such learning systems can be particularly beneficial for mobile manipulators, which perform tasks while navigating through open-world environ- ments that are not practical to instrument or map out in ad- vance. Unfortunately. in practice, most real-world reinforce- ment learning (RL) systems require careful environment in- strumentation and human supervision or demonstrations dur- ing the training process to ensure that the robot performs ef- fective exploration and resets between trials, making them dif- ficult to apply to the kinds of open-wer open-world domains where we might want mobile manipulators to operate. Such systems might require specially installed infrastructure that provides explicit re- sets [1, 2, 3, 4], or the presence of a person providing resets and monitoring the learning pro- cess [5, 6], and cannot simply be dropped into natural environment and continue learning.
Figure ReLMM enables learn- ing mobile manipulation skills au- tonomously in the real world, us- ing only on-board sensing.
To address this issue and make possible for mobile manipulators to learn with RI. directly in the real world, we propose system for learning mobile manipulation skills without instrumentation, demonstrations, or robot learn autonomously in settings such as homes and offices, such that anyone could simply
reset mechanisms. We aim to produce system that enables
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Navigation
Grasp Detection
Grasp
Pseudo-reset
obs
Figure 2: Method overview. ReLMM partitions the mobile manipulator into a navigation policy and grasping policy. Both policies are rewarded when an object is grasped. We use an ensemble of cecion-conditioned grasp success prediction functions estimate the success of potential grasp better and use the uncertainty as an exploration bonus. If grasp success is likely, a grasp action is sampled from the grasp predictors and executed. If the grasp is successful during training, the robot executes a pseudo-reset by placing the collected object back down in random location.
place the robot down, start the learning process, and return to a trained robot. several constraints that shape our method: (1) the robot must learn entirely from its own sensors, both to select actions and to compute rewards; (2) the entire learning process must be efficient enough for wolld training: (3) the robot must be able continually gathering data scale without human effort. A system that meets these requirements would not only be able to learn skills in open-world settings, but could also continue to improve throughout its lifetime: when learning can be performed practically without instrumentation. there is no reason to stop the learning process at deployment time, and the robot keep getting better and better at its given task perpetually. Our aim is not to propose the best possible system for solving any particular task. Rather, we aim to show how to create a real- reinforcement learning system that enables learning mobile manipulation skills entirely from real-world interaction, with minimal human intervention.
This goal dictates
Our contribution a system for autonomously training a mobile manipulation robot that satisfies the above constraints, which we call Reinforcement Learning for Mobile Manipulation (ReLMM). We apply ReLMM to the task of collecting items scattered across a room. Our system learns directly from on-board ego-centric camera observations and uses proprioceptive grasp sensing to assign itself rewards. To ensure the learning process is sample efficient and to facilitate exploration, we split the robot's s controller into separate navigation and grasping neural network policy modules that choose when to act based on their predicted value and are continuously trained together for the same objective: successfully grasping objects in the environment. Separating the policies enables the use of uncertainty-based exploration for the grasping module, which uses an ensemble of Q- functions to explore grasp actions efficiently. Lastly, to reduce the need for humans to provide interventions in the form of resets, we develop an autonomous resetting behavior where the robot re- arranggge the environment as it learns, the agent continually "practice." Together, these components plus a brief grasping curriculum enable system that can operate in real-world environments, learning how to navigate and grasp from its own collected experience, and mastering room cleanup tasks with about 40-60 hours of autonomous interaction. Videos are available at https: //st:e//sttt/teeee google. com.ee..coo/ee www/viiiiiieeeeeeeeeeeeo http/vvie/eeeeeeeeeeeeeeeeeeeeee ewmmelmm
to continually create new arrangements of objects for
2 Related Work
Robotic mobile manipulation tasks pose a number of unique challenges [7]. Many prior methods have addressed these challenges by requiring human effort for instrumentation and state estima- tion [8, 9, 10, 11], hand-coded controllers [12], or demonstrations [10, 13, 14). Several methods also require external instrumentation such as top-down camera views, oracle knowledge of object pose, or precomputed navigation maps [15, 16, 17]. textcolorMobile manipulation has also seen bene- fits from combining learning and planning for more efficient exploration in simulation [18, 19, 20], which allow for safe and effective behavior in the real world if object or goal locations provided. While these choices are pragmatic, they do not address the problem of learning continually unin- strumented real-world settings and require simulated world to train in. In contrast, proposed system is aimed at enabling reinforcement learning directly on the real hardware that is maximally autonomous, and does not require external instrumentation.
Hierarchical reinforcement learning has been shown to learn interactive navigation and object re- arrangement tasks in simulation operating from first-person view [21, 22], but require millions of

---

timesteps to train and have not been demonstrated in the real world. Our system uses a similar hier- archical structure these, but targets accessible real world learning by leveraging policy uncertainty estimates and curriculum learning for realistic sample efficiency.
While RL-bassed methods allow agents to improve via interaction, enabling robots that learn outside of instrumented laboratory settings is difficult. Such settings lack episodic structure and well-shaped reward functions [23, 24, 25, 26, 27] that are important for success [28, 29]. Large-scale uninterrupted deployment and training in the real world has been studied in several prior works [30, 31, 32, 33, 34, 35, 36]. While many of these works train robots in the real world without human interventions, they focus on navigation without manipulation. Large-scale real-world learning has also been successful for robotic grasping in laboratory settings. By letting robots learn from their own experience, prior approaches have shown that robots can learn to grasp from images [37, 38, 39, 40] or point clouds [41, 42]. These methods show real world learning can lead to robust robotic behavior, but they do not tackle challenging mobile manipulation tasks.
Gupta al. show an approach to training a grasping policy on a mobile manipulator, but unlike our work, do not attempt to learn to navigate [43]. Wang et al. train a mobile manipulator with RL in simulation, but require known object pose, dense rewards, and episodic resets single initial state [16]. In contrast, our approach learns entirely from a sparse reward and onbaord sensors. Past approaches to learning without episodic resets have focused on stationary manipulation or simu- lation [44, 45, 2]. In this work we explicitly focus on how mobile manipulation robots can learn without external sources of resets or state estimation in the real world and lay out a set of design decisions that makes this process practical one for acquiring robot skills.
3 Preliminaries
In this work, we use reinforcement learning (RL) as a general purpose algorithm for learning robotic behaviors. Reinforcement learning has the advantage of being able to operate autonomously collected data, and enables the robot to improve through trial and error. To this end, the mobile manipulation task is formulated as a partially observed Markov decision processes (POMDP) with an observation space of first person RGB images, state space S, action space AA, reward function r(st, de), transition dynamics P(s |s,, az), observation probability P(o, (s)) a discount factor y and an initial state distribution p(aa)- The goal of reinforcement learning is to learn a control policy (axlor) that can determine which actions to take in each observation such that the expected sum of 12, 7 7r((s at)],
rewards is maximized. This objective can be written as J(r) = E Ba as for standard RL problem.
4 ReLMM: RL For Mobile Manipulation
We develop the ReLMM system to enable train- ing mobile manipulation robots with RL. While we specifically apply it to a room cleaning task, in principle ReLMM could be used to learn other mobile manipulation tasks as well. Each
Algorithm 1 'TrainGrasp(G*..... ...., G.M, DDy N, st)
1: for t= 0, .... N steps do
2: Get grasp observation
component of ReLMM is chosen in order to maximise the autonomy of learning while re- taining the sample efficiency needed to train in the real world. The specific task that we study
3: Sample 0g Ty (-1ō). // see Equation 2
4: Perform grasp 0g. receiving T, 0 or 1.
5: Store (ō, 0gg rg) in Dg,
6: Update G ..., GM on DD
in our experiments involves training a robot to quickly navigate around in room with ob- stacles, physically pick up many objects, and place them in a basket mounted on the robot, as shown in Figure and 3.
7:
st's =True, Drop object held. 8: elseif r, 1 and st =False, return ro
9: return 0
Our system provides for efficient autonomous learning by decomposing the policy into grasping and navigation policies, using an ensemble of grasping models to explore based on uncertainty, automat- ically rearranging the environment after successful grasps, and using a curriculum to bootstrap and stabilize the concurrent training of both policies. Our final system can learn room cleaning skills in number of different room configurations in ~ 40 hours directly in the real world.
3

---

4.1 Grasping Policy Training
As noted previously, we decomposed the control problem into grasping and navigation. This gives the manipulation policy two objectives: given an image observation, accurately model the robots chance of success should attempt grasp and, if so, select an appropriate action maximize suc- cess, We obtain the former by training an ensemble of grasping policies and using their uncertainty to efficiently explore grasping- For choosing how to grasp, the policy must learn with sufficiently low sample complexity so as to make real-world training feasible. To reduce the complexity of the exploration problem we formulate grasping as a discrete single-step top-down action. The grasp policy "y (a)lo)) is parameterized with the action dg discretized in the x-y plane, and the observa- tion 0g corresponding to a RGB image from the robot's camera. Such single-step action selection formulations are amenable to more efficient training than more complex multi-step tasks [46, 47J.
Framed in this way, the grasping task corresponds a contextual aunti-armed bandit problem. Specifically, we train grasping policies that, given an image, predict the likelihood of grasp success for each action. We use soft-max over the action values to sample actions in proportion to their exponentiated probability of success. To create the ensemble, we train M = 6 independent grasp policies, G CM that are each by minimizing the cross-entropy loss on the same dataset:
Cg = -,,,,,,,,)))-,,,1- log G" (og, (g) - -11 (1- log(1 (og, ag))].
(I)
Here, Tg is when the robot successfully grasps an object, which is determined by presenting the gripper to the onboard camera. The grasping exploration policy is formed by constructing Boltz- mann distribution from optimistic estimates of grasp success, where the mean estimate from the ensemble, ( ag]], is modified by adding multiple of the ensemble variance oG (og. ag)), which we expect to be larger for actions where success is more uncertain:
G(og:ay) ag) aBE(G' (og. a,)] + Bo (o)g.a)))
(2)
with (g(gg)og) exp(E(og. (g)l ag)). The expectation and standard deviation are taken over the en- semble, and a, 3 2 O are hyperparameters. Other equivalent multi-step off-policy or contextual bandit algorithms can be used to train the grasping policy. However, we show in Section 6 that they are not as sample efficient for the mobile manipulation task in this work. Algorithm 1 describes the grasp training process in further detail.
4.2 Navigation Policy Training
For navigation, the policy must be able to control the mobile base to approach objects in a way that the current grasping policy can succeed. The navigation policy n (a, (o) outputs the action a, that controls the forward and turn velocities of the mobile robot base.
At every time step, the agent has to decide whether to perform a grasp or not by balancing the opportunity of receiving reward, the chance to collect novel data for the grasping policies, and the cost of wasting a timestep if there is no object within reach. This balancing act is done by reusing the same uncertainty measure described in Equation when choosing whether to attempt a grasp. The probability of whether to attempt grasp is
P P[grasplo| max G(o,ag). a
Under this design, the navigation policy nn (a,lo) continues to experience observations and output navigation actions, and at every step the choice of whether perform a grasp or not is made by sampling from grasp success probability of the current grasp model P(grasplo) of defined in Eqn 3. When the model decides attempting grasp is worth the risk, the robot executes grasp and evalu- ates the outcome to provide the navigation agent reward. From the perspective of the navigation policy, the choice of whether to grasp or not is part of the inherent dynamics of the environment. We compare two possible rewards rn for the the navigation policy. The first option is directly opti- mizing for the task by rewarding the navigation when a grasp is successful (i.e. where the current grasp ensemble has high performance): Ta (o) = (o) - 1. The second option is to reward the navigation for reaching states that the current grasp ensemble will choose to grasp at, which is a function of its mean and uncertainty: (o) (P(grasp/o) - 1) that we use to relabel states without successful grasps during SAC policy update. As this reward function only depends on the grasp ensemble and not on actual grasp success, this reward is computed at every policy update step and

---

in Figure 5 we show how it can improve sample efficiency. The RL objective for navigation is max,, (a..(a..lon) EE=n (+-(( We train the navigation policy for this objective using soft actor critic (SAC) [48, 49].
4.3 Training with Autonomous Pseudo-Resets
While the training schemes described above allow ReLMM to learn efficiently, both the contextual bandit grasping formulation and the navigation training setup requires an
Algorithm 2 ReLMM (with Stationary Curriculum)
1: Hyperparameters: M, Na. N Ngnap: relabel
Init: function estimators nu. G ..... G
episodic training setup, where the environ-
ment is reset between trials, for example by replacing the objects into the world at ran- dom positions. To enable the robot to learn mobile manipulation skills without manually
3: Replay buffers D,, (). D2 = 0
4: TrainGrasp(G'...
M gg. No. True)
5: for t 0, ..... T steps do
6: Get navigation observation or
provided resets, we construct an automated pseudo-reset system that allows method to learn autonomously without human inter- vention. After successful grasp, the envi- ronment would ideally be reset by relocating the object new, randomly selected loca-
Sample a, TT (-1o)) and perform On
8: if uniform() < PPlgrasplos 0r then
rg -TrainGrasp((G ,.., GM Nrusspr
10:
else ry =0
11:
if relabel then
12: Navigation reward rn = P(grasplor] -
tion. In stationary bin grasping setups, this can be automated simply by dropping the ob-
13:
else rn -
14:
Get next navigation observation 0,-1
ject back into the bin. However, for mo- bile manipulator, dropping the object back where it was grasped would promote policies
15:
Store (ot, (g,0n::n (0t,an: Tn, oc+1) in D..-
16:
Update n, with D, using SAC.
17:
Pseudo-reset
that fail to search for new objects, and sim-
18:
end for
ply remain in the same location. To force the policies to learn to grasp in varied situations, commanding random navigation actions while the robot is holding the object, placing down the ob- ject in this new location, and then navigating randomly away. This ensures the robot will not always be near an object during training and must instead learn to seek objects out, We outline the overall algorithm used for training in Algorithm
we perform an automated random pseudo-reset, by
4.4 Training Curricula
The separate grasping and navigation policies lend themselves naturally to curricular training, where the grasping policy, which needs to have some successes to provide rewards to the navigation policy, can be prioritized at the beginning of the learning process. We propose two types of curricula, which both break the potential rchicken-and-egee training problem of providing poor reward signals for navigation from an untrained grasping model.
Stationary curriculum. The simplest curriculum is to place single object in front of the robot and run Algorithm with st True for No 2000 steps. After each successful grasp, the robot places the object down randomly. If the object is knocked into an area the robot can not reach (because the base is stationary), which happens about 5% of the time, a human observer must push the object back into the graspable area. This curriculum is very time efficient, but does require occasional human intervention.
Autonomous curriculum. For fully autonomous learning, we develop training curriculum that favors collecting grasping data early on by performing high number of grasps at the beginning of training, according to hyperparameters Nstart, N Natop. and Nmar- The robot attempts N graasp Nstart grasps after every navigation step. If the robot succeeds at grasping an object will practice with this object until N Ngrasp = stop unsuccessful grasps. This initial automatic grasping curricula ends when a total of N grasps are complete. More details on the grasping curriculum algorithm are in Appendix ??
5

---

ao UT (TA
Real Traininng Perfarmance
mannanggo time
Figure 3: Top Left: Learning curves for training the real robot showing the number of objects grasping in the last 15 min of operation, which is ~ 250 actions. This metric are divided by the rooms' surface areas to make them comparable. Top mid-left: The simulated environment, which we run with and without obstacles. Right & Bottom: Snapshots from the evaluation tasks without obstacles in a 4m room mid-mii--igh,, no obst, bottom-left, diverse), with obstacles obst in 9m3 room (top-right), and with obstacles and rugs obst+rugs a 10m" room (bottom-right). Videos are available at
com/vies, http//elm//hhhoe home
Robotic Mobile Manipulator: System Overview
Our choice of robotic system reflects the need for robust robotic platform that can operate au- tonomously for long periods of time, and is unlikely to cause damage to itself and its surroundings. Ensuring safety during autonomous operation is itself a significant research challenge, which outside of the scope of this work. Therefore, we utilize a small-scale low-cost mobile manipulation platform based on the LocoBot design [43, 50]. shown in Figure (left), which consists of iClebo Kobuki mobile base and a WidowX200 5-DoF arm. The robot sensors include an Intel RealSense D435 camera at the top of the robot, as well as bump sensors on the base. We use an onboard Intel NUC to command the robot, and connect wirelessly to a server for data processing and training. Random exploration is generally safe with the LoCoBot as it is small, light, and will automatically stop the arm's 's motors if they encounter resistance. We also use the depth camera output to automat- ically avoid collisions, as described in ??. The robot learns a real-world office space, with varied lighting conditions, distractors, and surface textures. Our experiments utilize small objects that the robot can feasibly pick up, which consist of socks and toys sampling of objects one may want a robot to pick up off the floor.
To control the robot, we separately command the iClebo mobile base and the WidowX200 robotic arm with the corresponding navigation and grasping policies. The navigation policy stops the robot during grasp. For grasping, we use a directed end-effector control space. Assuming the floor flat, the vertical position for grasping is always chosen to be just above the floor, and the learning algorithm chooses a point in X space in front of the robot to perform a grasp at. This chosen position then dictates where move the gripper using inverse kinematics. Details on the learned network architectures are in Appendix ??.
Experimental Results
Our experiments aim evaluate our autonomous reinforcement learning system in number of real-world environments, as well as to provide ablation experiments and analysis in simulation. In particular, we aim study the following questions: (1) Can ReLMM Iearn autonomously in the real world? (2) How does the control hierarchy affect learning performance? (3) How does ReLMM compare other policy designs and prior methods? Our goal is to study real-world reinforcement learning, rather than necessarily provide the best possible solution to the room clearing task, and hence we design our system to be general, with only the reward determining the task. Experiment details For all real-world experiments, the entire training procedure is performed in the real world on the LoCoBot platform, with about 25 to 50 hours of training depending on the environment. The training is autonomous, with the exception of the stationary curriculum (if used), and battery changes every hours, during which time we may replace objects that become stuck near corners, or walls.

---

As our task is non-episodic, the policies are evaluated at the end of training. This is done by scat- tering the objects, executing the policy (without pseudo-resets) for 15 minutes, and counting the number of objects collected in that time. We use four real environments in our evaluation (no obsta- cles, with obstacles, with diverse objects, and with obstacles and rugs and simulated environments, as shown in Figure 3. Each room has a different size and number of objects, so we report the percent of objects collected in each 15 minute block of training, as shown in Figure 3. In simulation, we plot the evaluation performance using 250 timesteps instead of 15 minutes.
We compare our approach to several prior methods and baselines. The baselines include: Rand all, a policy that selects navigation and manipulation actions randomly, as a lower baseline; Rand nav, a method similar to Gupta et al. [43] that collects grasping data in the same way as our method, but has a random navigation controller. These baselines disentangle the benefits we get from learning both the grasping and the navigation. We also include hand-coded controller (Scripted) specifically engineered for this task that locates objects by thresholding the image pixels and grasps at their centroids, which provides a strong hand-des baseline. For more detail on the implementation of these baselines, see Appendix 6.
Real Robot Evaluation Our real-world experiments evaluate how well ReLMM can learn a mobile grasping policy au- tonomously in a variety of rooms. We conducted separate experiments for each of the rooms shown in Figure 3, which differ in terms of size, furniture, layout, and objects. ReLMM can train using both the stationary and autonomous grasping curricula, with the autonomous curricu- lum requiring less human effort, at the cost of increased training time. The result are summarized in Table with the training time given in ??. Examples of the learned behaviors are also shown in ??, and are il- lustrated in more detail in the supplemen- tary video.
Eny
no obst
obst
diverse
obst+
ReLMM StatCurr ReIMM-AutoCur Scripted
88 - 2 93 1
63 18
78 1 6
77 8 8
75
75+4 4
75 + 4
88 +
56
+
65 + 9
Rand nav [43]
52 12
38
10
22
20
=7
Rand all
12
6 2
2
2
5 4
Table 1:
Percentage of objects that the robot collects dur- ing eval in each environment (shown in Figure ??) (higher is better). Each method is trained once per env, and evaluated 3 times. The numbers mean and stddev of the evaluations. ReLMM outperforms the baselines by learning both grasping and navigation jointly, without requiring environment instru- mentation. Due to ReL.MM-AttCCur 's slower learning, we only evaluate it in no obst with diverse. In ?? we provide the training time in each env.
To provide context for these results, we compare ReLMM to the previously described baselines. The
Evaluation success rate obst+rugs room Rand all and Rand nav [43] baselines perform poorly, since neither method learns a directed nav- igation behavior, making it difficult to effectively navigate the room to collect objects, though Rand nav [43] is effective at picking up the objects if it stumbles upon them. This illustrates the importance of an intelligent navigational strategy in these set- tings. The Scripted baseline, which was specifi- cally engineered for this task, performs reasonably in most settings, but still falls short of the pol- icy learned by ReLMM in all of the environments, and crucially cannot improve from these results au- tonomously. This is particularly important in the di- verse and obst+rugs environments where it is diffi- cult find pixel threshold that works for all objects and all backgrounds, ReLMM can automat- ically learn how to identify objects to grasp via interaction. In Figure 4, we plot the evaluation performance of our BeLMM-StatCur agent learning in obst+rugs for a few checkpoints, showing that our method is improving from its experience and is still improving after 50 hours of training. This is important in open-world world settings, where our approach would enable mobile manipulators to improve perpetually over the lifetime of their deployment.
objects collected %
80-
60
40
20
20 40
training
Figure 4: Different checkpoints RcLMM in the obst+rugs room. Performance improves steadily with more training. and indeed is still in- creasing even at our final checkpoint, suggesting that lifelong training enables continual improve- ment for ReLMM.
Simulation Analysis To study questions (2) and (3), we perform detailed ablation analysis in simulation. As discussed in ??, we find that discretized grasping policy learns with the samples compared continuous one. In Figure 5, we show the performance of various ablations of our
7

---

Statlonary Curriculum Hyperparameter
140 Aytonomous Curriculum Hyperparameter
Simulation Task Performance
objects collectad to
odyjects collected %%
60
60
80
40
sbx
obyects
20
20
18 P0a 2000000000 and
20
50355330
Gougle
10000
2000C
30C03
10000
20000
30000
10000 20300 trai training staps
30000
training staps
training steps
Figure 5: Performance in the simulated environment without obstacles. Each plot is the mean and stddev over 3 random seeds. Left: Ablation of different number of grasp "pretraining" samples for the stationary curriculum (Na) and navigation reward relabeling (described in subsection 4.2) in simulation. Increased pre- training of the grasping policy improves overall mobile manipulation performance. Center: Analysis to find the best parameters for the autonomous curriculum. With the proper settings [Nstart]Naaton ( the tonomous curriculum can be almost as efficient as the stationary, without requiring as much manual effort. Right: Ablation of ReLMM, all use the stationary curriculum and relabeling except for Ours (AutoCurr). We find that the uncertainty bonus and joint training are critical components of our system.
system in the simulated environment with the stationary curriculum. First, we ablate the policy decomposition by training single policy with reinforcement learning using grasp success as the reward. The joint action space poses a much larger exploration problem, and the policy is unable to make headway the task in a reasonable number of samples. This indicates that the hicrarchical decomposition used by ReLMM significantly improves training performance. Next we see that freezing the grasp policy after the stationary phase (Pretrain Online) and only training the navigation is much worse than training both together. This illustrates the interplay between the two policies, as the navigation is dependent on the grasping for obtaining rewards. As shown in the plot, training the grasp policy without the uncertainty bonus (i.e. 3 = 0) leads to significantly poorer performance for ReLMM, as it is less incentivized to explore. Finally, we see that the autonomous curriculum with reward relabeling can get similar final performance as the stationary curriculum with less human intervention at the cost of longer training time.
Lastly, we examine how to get the best performance in terms of sample efficiency by performing an ablation the choices of curriculum minimize the samples needed. This is especially important as ReL.M trains two policies concurrently, which can often be unstable. As can be seen in Figure (left) strong final navigation and grasping policy can be learned with just 500 stationary grasps while applying relabeling to the navigation rewards. Next, we expand on the curriculum results on the real robot in order to tune the autonomous curriculum stationary curriculum. In Figure 5 (middle) we can see that the best final performance is achieved by increasing the frequency of grasps early on in training and adding a bonus using the grasp model uncertainty.
be nearly as sample efficient as the
Discussion and Future Work
We presented ReLMM, system for autonomously learning mobile manipulation skills in the real world, without instrumentation, and with minimal human intervention. Our real-world experiments, conducted in four separate rooms, show that ReLMM can train continuously for several days (40-60 hours) with only occasional interventions, and that the resulting policies are effective at cleaning up the room. Furthermore, our experiments show that ReLMM continues to improve with more training, suggesting that it provides an effective approach for lifelong learning for robotic systems deployed in open-world settings. Our simulated ablation studies further provide support for the de- sign decisions in ReLMM, indicating that the hierarchical decomposition of navigation and grasping greatly improves learning performance, autonomous practicing with a suitable exploration bonus enhances learning speed, and our automated curriculum can provide effective performance when leamning from scratch. Even when using the prrovidd curriculum, the grasp pretraining phase does not need to be long: only 1000 attempted grasps were needed to get reasonable room- cleaning performance, although more pretraining can improve performance further. Extending this system to more complex manipulation tasks would be exciting direction for future work. We hope that this system for practical RL on mobile manipulators will make mobile manipulation more accessible to outside lab spaces and to users who need maximally autonomous learning algorithms.
8

---

Acknowledgments
We thank the anonymous reviewers for their helpful feedback in revising our manuscript. This research was supported by the Office of Naval Research, ARL DCIST CRA W911NF-17-2-0181, the National Science Foundation, and Schmidt Futures.
References
[1] Y. Chebotar, M. Kalakrishnan, A. Yahya, Li, Schaal, and Levine. Path integral guided policy search. In 2017 IEEE international conference on robolics and automation (ICRA), pages 3381-3388. TEEE, 2017.
[2] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The in- gredients of real world robotic reinforcement learning. In International Conference on Learn- ing Representations, 2019.
[3] A. Nagahandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112, 2019.
[4] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Ex- periential learning of intuitive physics. Advances in neural information processing systems, 2016.
pages
[5] Y. Chebotar, K. Hausman, M. Zhang, G. Sukhatme, S. Schaal, and Levine. Combining model-based and model-free updates for trajectory-centric reinforcement learning. pages 703- 711, 2017.
[6] A. Ghadirzadeh, A. Maki, D. Kragic, and M. Björkman. Deep predictive policy training using reinforcement learning. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2351 1-2358. IEEE, 2017.
[7] T. Nishida, Y. Takemura, Y. Fuchikawa, S. Kurogi, S. Ito, M. Obata, N. Hiratsuka, H. Miya- gawa, Watanabe, Koga, et al. Development of outdoor service robots. 2006 SICE- ICASE International Joint Conference, pages 2052-2057. IEEE. 2006.
[8] M. Stilman and J. J. Kuffner. Navigation among movable obstacles: Real-time reasoning in complex environments. International Journal of Humanoid Robotics, 2(04))479-503. 2005.
[9] Stulp, A. Fedrizzi, and M. Beetz. Learning and performing place-based mobile manipula- tion. In 2009 IEEE 8th International Conference Development and Learning, pages 1-7, June 2009.
[10] Stulp. A. Fedrizzi, Zacharias, M. Tenorth, J. Bandouch, and M. Beetz. Combining anal- ysis, imitation, and experience-based learning to acquire concept of reachability in robot mobile manipulation. In 2009 9th IEEE-RAS International Conference on Humanoid Robots, pages 161-167, Dec. 2009.
[11] L. P. Kaelbling and T. Lozano-Pérez. Unifying perception, estimation and action for mobile manipulation via belief space planning. In 2012 IEEE International Conference Robotics and Automation, pages 2952-2959. IEEE, 2012.
[12] P. Lehner, S. Brunner, A. Domel, H. Gmeiner, S. Riedel, B. VVodemaaer, and A. Wedler. Mobile manipulation for planetary exploration. 2018 IEEE Aerospace Conference, pages 1- 11, 2018, doi: 100.100///00020 8396776...
[13] M. R. Walter, Y. Friedman, M. Antone, and S. Teller. One-shot visual appearance learning for mobile manipulation. Int. J. Rob. Res., 31(4):554-566 Apr. 2012.
[14] T. Welschehold, C. Dornhege, and W. Burgard. Learning mobile manipulation actions from human demonstrations. In 20/7 JEEF/RSJ International Conference Intelligent Robots and Systems (IROS), pages 12001,,,, Sept. 2017.

---

[15] B. S. Iapata-Impata, V. Shah, H. Singh, and R. W. Platt Autotrans: an autonomous open world transportation system. CoRR, abs/1810.033000, 2018. URL http:://axiiy org/abg/ 1810. 1810.03400.
[16] C. Wang. Q. Zhang, Q. Tian, S. Li, X. Wang. D. Lane, Y. R. Petillot, and S. Wang. Learning mobile manipulation through deep reinforcement learning. Sensors, 20(3):939, 2020. doi: 10.3390//200309939 URL https: /doi. org/10 htt//1..33900 0000/3300000000
[17] Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser. Spatial Action Maps for Mobile Manipulation. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020. 16607/... 2020.XVI.035
[18] J. Kindle, F. Furrer, T. Novkovic, J. J. Chung, R. Siegwart, and J. Nieto. Whole-body body control of a mobile manipulator using end-to-end reinforcement learning. arXiv preprint arXin:2003.02637, 2020.
[19] M. Mittal, D. Hoeller, F. Farshidian, M. Hutter, and A. Garg. Articulated object interaction in unknown scenes with whole-body mobile manipulation. arXiv preprint arXiv:2103.10534 2021.
[20] D. Honerkamp, T. Welschehold, and A. Valada. Learning kinematic feasibility for mobile manipulation through deep reinforcement learning. IEEE Robotics and Automation Letters, 6: 0289-6296 2021.
[21] C. Li, F. Xia, R. Martún-Martín, and S. Savarese. Hrl4in: Hierarchical reinforcement learning for interactive navigation with mobile manipulators. In Conference on Robot Learning, pages 603-616. PMLR, 2020.
[22] Xia, C. Li, R. Martín-Martín, O. Litany, A. Toshev, and S. Savarese, ReLMoGen: Leverag- ing motion generation in reinforcement learning for mobile manipulation. Aug. 2020.
[23] M. Sridharan and P. Stone. Structure- based color learning on a mobile robot under changing illumination. Autonomous Robots, 23(3):161-188) 2007.
[24] T. Krajník, J. P. Fentanes, M. Hanheide, and T. Duckett. Persistent localization and life-long mapping in changing environments using the frequency map enhancement. In 2016 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 2016.
IEEE,
[25] J. Biswas and M. M. Veloso. Localization and navigation of the cobots over long-term deploy- ments. I. J. Robotics Res., 32714);167- 2013. 461:10..1777007333911333322 URL nttps://doi. org/10.1r 1177 1177/0277336441366882
[26]- S. Rosenthal, M. Veloso, and A. K. Dey. Learning accuracy and availability of humans who help mobile robots. In Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
[27] J. Pineau, M. Montemerlo, M. Pollack, N. Roy, and S. Thrun. Towards robotic assistants in nursing homes: Challenges and results. Robotics and autonomous systems, 42(3-4):271-281, 2003.
[28] Lesort, V. Lomonaco, Stoian, Maltoni, Filliat, and N. Díaz-Rodríguez. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information Fusion, 58:52-68, 2020.
[29] J. D. Co-Reyes, S. Sanjeev, G. Berseth, A. Gupta, and S. Levine. Ecological reinforcement learning. CoRR, abs/2006. 12478, 2020. URL https: //ap::////xvvvv org/ abs/ab///// /200/ 12478.
[30] S. Thrun, D. Fox, W. Burgard, and F. Dellaert. Robust monte carlo localization for mobile robots. Artificial intelligence, 128(1-2):99-14 1411, 2001.
[31] D. Fox, W. Burgard, Dellaert, and S. Thrun. Monte carlo localization: Efficient position es- timation for mobile robots. In J. Hendler and D. Subramanian, editors, Proceedings of the Six- teenth National Conference on Artificial Intelligence and Eleventh Conference on Innovative Applications of Artificial Intelligence, July 18-22, 1999, Orlando, Florida, USA, pages 343- 349. AAAI Press/ / The MIT Press, 1999. URL http: //www. aaai, aaaai.orgae www/orrrgrgr/......... AAAII/ 1999/aaai99 -001999-555 050. 050.php.
10

---

[32] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 5129-5136 IEEE, 2018
[33] A. Scherer, S. Singh, L. Chamberlain, and M. Elgersma. Flying fast and low among obstacles: Methodology and experiments. 1. J. Robotics Res., 27(5):549-574, 2008. doi: 101177700783644900099944 URL https: org/10. 1177/02276664988999999
[34] Bruce, N. Sünderhauf, P. Mirowski, R. Hadsell, and M. Milford. Learning deployable nav- igation policies at kilometer scale from single traversal. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, vol- ume 87 of Proceedings of Machine Learning Research, pages 346-361. PMLR, 2018. URL html.
http:://rroceedings. mlr. wtt.a/eeresee//eeceeeel....
[35] A. Loquercio, A. I. Maqueda, C. R. delnBBlacoo, and D. Scaramuzza. Dronet: Learning to fly by driving. IEEE Robotics Autom. Lett., 3(8); 2018, 4o1::0.119//RRA221122766666 URL https: :/tps::///oi.. org/ 10. 1109/LRA 2018 2795643.
[36] G. Kahn, P. Abbeel, and S. Levine. Badgr: An autonomous self-supervised learning-based navigation system. IEEE Robotics and Automation Letters, 6(2):1312-139,, 2021.
[37] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Reseurch, 34(4-5):05--24, 2015.
[38] L. Pinto and A. Gupta. Supersizing self-supervision: Learning grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406 8406-3413. IEEE, 2016.
[39] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, (7(4-5):421-446 2018.
[40] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, Robot Learning. volume 87 of Proceed- ings of Machine Learning Research, pages 651-673. PMIR, 29-31 Oct 2018. URI. http: //proceedings..o mlr wwwpppeeeess. /kal walasssiikvvill.. html.
editors, Proceedings of The 2nd Conference
[41] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, 36(1 (6(13-14)1145-1444 1473, 2017.
[42] J. Mahler, J. Liang. S. Niyaz, M. Laskey, R. Doan, X. Liu, J. Aparicio, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In Proceedings of Robotics: Science and Systems, Cambridge, Massachusetts, July 2017. doi: Ni..1..5567RRSS. 2015.
[43] A. Gupta, A. Murali, D. Gandhi, and L. Pinto. Robot learning in homes: Improving gener- alization and reducing dataset bias. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 9112-9122. Curran Associates Inc., Red Hook, NY, USA, 2018
[44] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine. Leave no trace: Learning to reset for safe and au- tonomous reinforcement learning. 6th International Conference on Learning Representations, ICLR 2018- Conference Track Proceedings, 2018
[45] W. Han, S. Levine, and P. Abbeel. Learning compound multi-step controllers under unknown dynamics. 2015 IEEE/RSI International Conference on Intelligent Robots and Systems 100.1100/RRO.2211...777
(IROS), pages 6435-6442, 2015,
[46] C. Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan. Is q-learning provably efficient? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- netr, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL. https: https//////oeee/s..eeo.e. 4Ba11000994444422551996666621113 Paper pdf.
cc/paper/2018/filee
11

---

[47] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to Manipulate Deformable Objects without Demonstrations. Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020. doi: 1oi::00.160770 15607/RSSS.0000.XVI.
[48] T. Haarmjja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[49] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning walk via deep reinforcement learning. In Robotics: Science and Systems, 2019.
[50] A. Murali, T. Chen, K. V. Alwala, D. Gandhi, L. Pinto. S. Gupta, and A. Gupta. Pyrobot: An open-source robotics framework for research and benchmarking. CoRR, abs/1906.08236. 2019. URL http: / arxii. org/
08236.
12


Reviews:
[{"review_id": "bc-mF2EkpRZ", "reviewer": "Reviewer_m31K", "summary": "The main contribution of the paper is a robotic reinforcement learning system that learns mobile manipulation in an autonomous way in the real world, without human intervention or instrumented laboratory settings. To be more specific, the paper presents 1) a modular navigation and grasping policies with onboard sensing observations that make use of uncertainty-based exploration, 2) pseudo reset mechanism to minimize necessary human intervention, and 3) curriculum learning with two variants of grasp pre-training.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a solid robotic system that supports 40 hours of real-world reinforcement learning training with minimal human intervention, which is quite impressive. The solution also has some level of algorithmic novelty that includes uncertainty-based exploration, control hierarchy of navigation and grasping, and curriculum learning. The authors extensively evaluate their solution against several baselines in a few different task setups (with and without obstacles, with and without rugs, diverse objects, varying lighting conditions, etc) and achieve favorable results compared to baselines. The authors also conduct several ablation studies in simulation for different action space and curriculum methods, and demonstrate the validity of their algorithmic design choices. \n\nOne main weakness of this work seems to be the narrow applicability of the proposed method. Although it’s mentioned the ReLMM could be used to learn other mobile manipulation tasks other than room cleaning tasks, it’s doubtful whether this is actually the case. For instance, both the pseudo-reset mechanism and overall method (Algorithm 1 and 2) seem to only work with the room-cleaning task specifically. The paper would be a lot stronger if it shows ReLMM can also be applied to a different family of tasks, e.g. pick-and-place tasks that require robot base movement. The second main weakness is the mixed results in Sec 6.3 and Appendix C and the lack of clarity in the analysis. It’s claimed that the navigation reward relabeling helps sample efficiency. Yet in Figure 4 Left, the results are very mixed, without proper analysis or hypothesis for why this is the case. Same for autonomous curriculum; It was claimed that with proper parameter tuning, autonomous curriculum can learn almost as fast as stationary curriculum (Figure 4 Center), but the opposite result is shown in Appendix C.2 and Figure 8. In general, Sec 6.3 and Appendix C has room for improvement.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While this paper stands as a strong system paper with real robot experiments and produces favorable results over baselines in the task of interest, there still remain questions about the generality of the proposed method and the validity of all the algorithmic design choices. A thorough response to the listed issues below will help inform the final recommendation.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Main issues to be addressed\n* Better showcase the generality of the proposed method, (e.g. how can the system and the algorithm be easily adapted to new types of mobile manipulation tasks)\n* Improve writing Sec 6.3 and Appendix C: clarify whether navigation reward relabeling helps and whether autonomous curriculum can be as efficient as stationary curriculum\n* Real world experiment of autonomous curriculum with the proper settings: it would be great to demonstrate autonomous curriculum with proper settings (e.g. the best parameters you found in simulation) in the real world across all four environments (currently only one environment is included in Table 1)\n* Add related work for RL for mobile manipulation tasks in simulation that only relies on onboard sensors, e.g. [1]. Note that [1] does not “require external instrumentation in the form of top-down camera views, oracle knowledge of object pose, or precomputed navigation maps”. The top-down occupancy map used in [1] is generated from onboard LiDAR signals.'\n\nMiscellaneous issues & questions\n* Algorithm 1: it should be G^M, instead of G^N?\n* For the navigation reward, why -1 is needed from the grasping reward?\n* It’s understood that the grasping problem is formulated as a contextual bandit problem (with episode length = 1). What is the episode length for the navigation problem? Is it also 1, infinite, or the steps between two pseudo resets? \n* Fig 3 Left and Fig 4 are very dense and hard to read.\n\n\n[1] Xia, Fei, et al. \"Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation.\" arXiv preprint arXiv:2008.07792 (2020).\n"}}, {"review_id": "HBtnw7JcNHm", "reviewer": "Reviewer_rFXj", "summary": "This paper proposes a novel reinforcement learning framework for a mobile robot to autonomously learn to perform manipulation tasks with on-board sensors only.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths and Weaknesses\nThe proposed framework requires no extra instrumentation and few human interventions. The navigation policy could be learnt to support the object manipulation tasks through the influence of grasping rewards. Experimental results show the advantage of learning both grasping and navigation policies in real world tasks.\n\nThe grasping actions are discretized to simplify the learning of grasping policy, which is quite reasonable for an initial study. But I am also very curious about applying this framework to more complicated manipulation tasks that require continuous action spaces in future studies.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed learning framework enable the mobile robot to achieve better performance on real world tasks than other existing methods while being mostly autonomous during the learning process. It could effectively save human effort and other equipment usage, which may be significant for mobile robots operating in a large-scale environment. Hence I think that it serves as a good reference for relevant research in the future.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "1. Some typos: Page2, Figure 2 caption, line 4, ‘succes’ should be ‘success’; Page 7, line 307, ‘making is difficult’ probably should be ‘making it difficult’; Appendix A.2, line 516, ‘is pass’ should be ‘is passed’.\n2. According to section 4.4, it’s said that in stationary curriculum, grasping policy training (Algorithm 1) is performed multiple steps when the robot base keeps being stationary, and does not mention about the navigation part, hence I initially thought that the stationary curriculum uses a stationary robot in comparison to the mobile robot used in the autonomous curriculum. But the experimental results shown in Figure 3 suggest that stationary curriculum also trains a mobile robot. So now I guess that the stationary curriculum also follows Algorithm 2 except changing the last argument from 0 to 1. Is my understanding correct? And could you please also clarify it in the paper?  \n"}}, {"review_id": "DiznO62rG92", "reviewer": "Reviewer_W8H3", "summary": "The paper present a reinforcement learning method for training a grasping and navigation policy for a mobile manipulation robot, that learns to clean up the floor. The exploration strategy for the navigation leverages the uncertainty over the grasp success, as to explore states that will increase the probability for grasping success. The policy was trained with a physical robot in a real setting, with several training choices, like pre-training the grasping policy with automatic reset of the learning episodes and an autonomous curriculum.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found the idea of the automatic resets a nice design decision for the problem of learning mobile manipulation. \nA major weakness is the final behavior of the robot that seems to be pushing objects as obstacles instead of stopping and grasping them. I find the idea of proposing to solve mobile manipulation in an end to end fashion is rather greedy. Moreover, the task- oriented policies are not a new idea in robot learning, and the combination of grasp success to a subsequent policy has been previously proposed, but perhaps not in the exact same setting of this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I have three main concerns that drive my recommendation.\n1. Since top-down grasping has been generally solved, and in particular the method described resembles quite much the DexNet method, I don’t see why the authors overly analyze this part. Moreover, The objects used are very trivial, and do not correspond to objects found in a real home environment. The grasping policy is also rather simplified in an x,y position, that makes the learning rather easy. \n2. Not sufficient comparisons with baselines. The authors do not refer or compare with current works on mobile manipulation, that show these benefit of coupling learning with planning, The authors also do motivate why model-free RL is a suitable method for solving mobile manipulation tasks.\n3. It is not clear how the robot learns to avoid obstacles, as in the provided video it seems like the robot is sometimes pushing away objects that are fallen in the ground and should have been grasped.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "UPDATE: after the rebuttal I raised my score to weak reject.\n\n- The related works are missing references to recent papers on the topic of learning mobile manipulation, like: \n\n https://arxiv.org/abs/2103.10534\n\n https://arxiv.org/pdf/2006.04271.pdf\n\n https://arxiv.org/pdf/2101.05325.pdf\n\n https://arxiv.org/pdf/2003.02637.pdf\n\n- The preliminary section is wrong. The POMDP definition you provide is wrongly stated. The definition of the reward containing observations is wrong. There is also an abuse of notation. Moreover, in the subsequent methodology, you do not provide an explanation as to how you handle partial observability. Therefore, this preliminary section is not connected to the described methodology. \n\n- The grasping policy is resembling the method of DexNet. Why is it important to learn these features for top-down grasping, and you do not rely on well-known and well-performing methods for top-down grasping? Moreover, you are over-simplifying the grasping problem to a simple 2D position on the thrown objects on the ground, hence computing 2D grasps, while I would argue that 6D grasps are more important for real world problems. Do you think that if you were to train a more complex grasping policy for 4D or 6D grasps, the behavior of the subsequent navigation policy would be the same? How would the higher uncertainty in orientation estimation affect the whole learning process? In algorithm 1, what is N, and pt? Please denote all notation. Finally, what is the observation? Is it RGB, depth, or point cloud? Are you employing some additional object detector to decide on the amount of objects? How is the high-level decision of the number of objects to be grasped being decided? Is this parameter already known?\n\n- It was a bit hard for me to follow section 4.2, in the part describing the relabeling that you mention. Moreover, please state all used notation, e.g. what is $r_{n-std}$, what is n? What is weird for the navigation task, is that there is no reference to an obstacle avoidance objective. How can your navigation policy handle collisions, what are the sensorial data employed that will allow the agent to understand the difference between obstacles and objects to be grasped? In the video, it seems like the robot sometimes overrides objects on the ground. Is this a good behavior? Would this strategy be transferable to realistic mobile manipulation platforms? \nOn another note, you state that SAC is data efficient. Why is an algorithm that needs millions of samples to converge data efficient? This is a false statement. \n\n- The experimental comparison is not fair. You should have compared your method to other approaches in literature that consider interactive navigation, like https://arxiv.org/pdf/2101.05325.pdf and https://arxiv.org/pdf/1910.11432.pdf\nThe comparison of only a random navigation strategy is not aligned with the claimed objective of showcasing advances in learning of mobile manipulation. \nMoreover, I would expect an ablation on different grasping strategies, rather than a simple comparison to a scripted strategy. The plots of Fig, 3 are difficult to interpret, and I am not sure why the curves are interrupted. Also, you do not state how many times you have conducted these experiments, and what is the uncertainty, as the mean behavior is very noisy, and it doesn't look like the algorithm converged to a good behavior, e.g., in Fig. 3top, both blue and green curves are oscillating, and interrupted. There should be sufficient explanation on the presented results, and I would suggest to report success rates at least on 5 experiments with different random seeds. Additionally, the results of Fig. 4 are not allowing to derive sufficient conclusions on the algorithmic decisions. \n\nFinally, I would recommend that the authors do not make strong claims in their conclusions that are not justified by the results, and the absence of comparisons with state-of-the-art works. \n \n\n\n"}}, {"review_id": "AfhWr0-iGn2", "reviewer": "Reviewer_aRW7", "summary": "This paper presents a deep reinforcement learning approach for training an agent to both navigate and grasp objects in the real world given only an RGB video feed. \n\nThe key idea is to decompose grasping and navigation into two separate modules, with manipulation being trained to estimate the uncertainty of grasp success with rewards given for successful grasps, and navigation being trained to navigate in such a way as to minimize this uncertainty. \n\nTwo forms of curriculum are also suggested: (1) the robot is stationary and only does grasping actions, with some human intervention to ensure objects are present, and (2) the robot both navigates and grasps, but performs grasping actions more frequently than navigation actions. To continuously learn in the real world without human intervention, a pseudo-reset mechanism is employed in which the robot randomly moves and releases a grasped object and moves to a random location. Compared to prior work, this overall approach allows for training directly in real world without any input information except an RGB observation stream, which can be done in the span of dozens of hours. \n\nEvaluation for training in the real world shows the approach to be able to achieve high success rates. Several baselines that do not use different grasping and navigation modules or a curriculum are also evaluated, and attain low success rate. Evaluation in simulation also presents ablation results that justify the need for the various components of the system. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The problem of training both navigation and manipulation actions using only raw sensory inputs and a reward function is useful to address, with deep RL only having been explored in this setting in a few prior works\n* The paper is clearly written and has few ambiguities, and is easy to follow\n* The claim that this approach requires less knowledge about the environment and less human intervention is accurate and significant \n* The key new ideas are well suited to the task, and are mostly novel. In particular, using the grasp uncertainty to reward good navigation is a good idea. See caveat below.\n* Experiments are partially well done, with real world results presenting impressive success rates and simulated results presenting more detailed justification of the need for parts of the approach. However, see caveats below.\n* Compared to prior work, demonstrating learning results entirely from real world experience is impressive and indicates the approach's usefulness.\n\nWeaknesses:\n* While the related work surveyed is fairly comprehensive, the work \"HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators\" is not cited. This is problematic, as this work has a similar concept of decomposing the policy into manipulation and navigation components, and training primarily from raw sensor inputs. Its manipulation and navigation policies can arguably be change to this paper's action space and the curriculums suggested could be applied to its approach, meaning that there is significant overlap conceptually. The This work does use additional inputs to the agent such as door locations, but this is not strictly required. \n* Additional citations for hierarchical RL approaches would also make sense, given that they also decompose the skills needed to learn (even if not specifically for the task of mobile manipulation). \n* Unlike prior work on deep RL for mobile manipulation, this approach does not allow for learning arbitrary mobile manipulations skills, and is instead specialized to navigation and grasping \n* The approach is evaluated in a simplified setting of having simple ball-like objects scattered on the floor, in a space of limited size that does not require much navigation. This makes it so both the perception and manipulation aspects are significantly simplified compared to prior tasks in deep RL for mobile manipulation. It is unclear whether this approach can scale to more complex settings.\n* The evaluation in the real world is not presented very clearly, with Figure 3 having only partial learning curves and Table 1 having missing entries. This is especially problematic for Figure 3, since the shown curves are quite chaotic. Additionally, variance values are presented in Table 1, and not in Figure 3. These things combined make me worry that the results are partially cherry-picked.\n* The baselines being compared to are very simple, with HRL4IN being a much more suitable algorithm to compare against.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend rejection, despite this presenting ideas with good originality, technical quality, and potential impact. This is primarily due to issues I presented with respect to the experiments section, and due to weak baselines that do not include an approach similar to HRL4IN, which is also not cited and not compared to (which reflects my evaluation of clarity). However, I hope the author response and revisions could be sufficient to address these issues, in which case I would likely recommend acceptance.\n\nUpdate after responses:\nI have updated my recommendation to weak accept from weak reject - see my reply to the authors in comments below.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "* Cite HRL4IN and compare your approach to their in text. Ideally, also compare your approach to a policy similar to theirs experimentally\n* Explain the experimental protocols in greater detail, \n* Provide more results for the real world experiments, in particular by presenting full results for Figure 3 and Table 1\n\nUpdate after responses:\nAs stated in my comments below, I believe the writing needs to be updated to more strongly reflect the limitations of the contribution in terms of lack of human intervention and the degree to which the system design as opposed to task design allows for efficient learning. "}}]

Author rebuttal:

Meta-review: This paper presents a real robotic system that is trained to learn navigation and grasping skills via reinforcement learning. The system is capable of learning clean-up actions based solely on onboard sensing capabilities over 40 hours of training. The components of training curricula and pseudo-resets have been appreciated by the reviewers.

Although clear and easy to follow in general, as pointed out by the reviewers, the paper presents technical inaccuracies and the methodology should be clarified.

A common issue raised by the reviewers is the deficiencies in the evaluation which seem to be set more like a semi-toy-problem that seem to  suit the method. In particular, comparison with appropriate baselines and a lack of more realistic, general tasks have been missed.

The reviewers made several comments regarding a lack of related work and appropriate references, including related work in learning mobile manipulation and hierarchical RL in mobile manipulation (HRL4IN). 

Post-rebuttal ====================

Thank you for answering our questions and requests, including the addition of a baseline (HRL4IN) and for clarifying HRL4IN sample inefficiency. Overall, this paper has been well received and the reviewers have been positive about the author's response..


Decision: Accept (Poster)