Continuously Improving Mobile Manipulation with Autonomous Real-World RL

Authors: russell  mendonca
emmanuel  panov
bernadette  bucher
jiuguang  wang
deepak  pathak
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 09:37:05

Abstract: We present a fully autonomous real-world RL framework for mobile manipulation that can learn policies without extensive instrumentation or human supervision. This is enabled by 1) task-relevant autonomy, which guides exploration towards object interactions and prevents stagnation near goal states, 2) efficient policy learning by leveraging basic task knowledge in behavior priors, and 3) formulating generic rewards that combine human-interpretable semantic information with low-level, fine-grained observations. We demonstrate that our approach allows Spot robots to continually improve their performance on a set of four challenging mobile manipulation tasks, obtaining an average success rate of 80% across tasks, a 3-4 times improvement over existing approaches. Videos can be found at https://continual-mobile-manip.github.io/.

Main paper: 
 introduction: 
 How do we build generalist systems capable of executing a wide array of tasks across diverse environments, with minimal human involvement? While visuomotor policies trained with reinforcement learning (RL) have demonstrated significant potential to bring robots into open-world environments, they often first require training in simulation [1, 2, 3, 4, 5, 6] . However, it is challenging to build simulations that capture the unbounded diversity of real-life tasks, especially involving complex manipulation. What if learning instead occurs through direct engagement with the real world, without extensive environment instrumentation or human supervision?Prior work on real-world RL for learning new skills has been shown for locomotion [7, 8] , and in manipulation for pick-place [9, 10, 11, 12] or dexterous in-hand tasks [13, 14, 15] in stationary setups. Consider a complex, high-dimensional system like a legged mobile manipulator learning in open spaces. The feasible space of exploration is much larger than in constrained tabletop setups. Autonomous operation of such a complex, high-dimensional robots often does not result in data that has useful learning signal. For example, we would like to avoid the robot simply waving its arm in the air without interacting with objects. Furthermore, even after making some progress on the task, the robot should not stagnate near goal states. While prior work has explored using goal cycles [16, 13, 17] to help maintain state diversity, this has not been shown for mobile systems. Such systems also need to learn more complex skills, involving constrained manipulation of larger objects and moving beyond pick and place, making sample-efficient learning critical. Finally, reward supervision using current RL approaches often requires physical instrumentation using specialized sensors [18, 19] or humans in the loop [20, 21, 22, 23] , which is difficult to scale to different tasks.Our approach tackles each of these issues of autonomy, efficient policy learning, and reward specification. We enable higher-quality data collection by guiding exploration toward object interactions using off-the-shelf visual models. This leads the robot to search for, navigate to, and grasp objects before learning how to manipulate them. We preserve state diversity to prevent robot stagnation by extending the approach of goal-cycles to mobile tasks and with multi-robot systems. For sample efficient policy learning, we combine RL with behavior priors that contain basic task knowledge. These priors can be planners with a simplified incomplete model, or procedurally generated motions. For rewards without instrumentation or human involvement, we combine semantic information from detection and segmentation models with low-level depth observations for object state estimation.The main contribution of this work is a general approach for continuously learning mobile manipulation skills directly in the real world with autonomous RL. The main components of our approach involve: (1) task-relevant autonomy for collecting data with useful learning signals, (2) efficient control by integrating priors with learning policies, and (3) flexible reward specification combining high-level visual-text semantics with low-level depth observations. Our approach enables a Spot robot to continually improve in performance on a set of 4 challenging mobile manipulation tasks, including moving a chair to a goal with the table in the corner or center of the playpen, picking up and vertically balancing a long-handled dustpan, and sweeping a paper bag to a target region. Our experiments show that our approach gets an average success rate of about 80% across tasks, a 4× improvement over using either RL or the behavior prior individually with our task-relevant autonomy component. 
related work: 
 Autonomous Real-World RL: Previous work for real-world RL mostly involves either manipulation for table-top pick-place settings [9, 10, 8] , in-hand dexterous manipulation [15, 13, 14] or locomotion behavior [24, 25, 8] . Approaches for automated resets needed for continual practice include instrumented environments [9, 10] , forward-backward policies [26] , graph structure of subtasks that serve as resets for one another [13, 14] , or pre-trained, reliable reset policies [7] . For mobile manipulation, real-world RL has been limited to pick and place tasks [11, 12] . In our work, we extend the RL framework to learn challenging manipulation skills such as sweeping and moving chairs for a mobile system. Autonomous mobile systems should leverage the ability of the robot to move around to extend the effective reach of the robot and attempt manipulation tasks with large objects that are not possible on a table-top setup. For efficient learning on these complex tasks, we leverage behavior priors, which have some basic task knowledge. Moreover, task specification is a big challenge [27] for real-world learning. Current approaches often require physical instrumentation using specialized sensors [18, 19] or humans in the loop [20, 21, 22, 23] , which is difficult to scale to different tasks. There has been some work on completely self-supervised learning systems with some extensions to robotics [28, 29] , but these approaches are challenging to deploy on complex tasks due to intractability, underspecification, and misalignment. We extend the approach of using language goals and combining these with large-scale visual models [30] , conditioned on open-vocabulary prediction [31, 32, 33] , to obtain object states, which can be used to compute reward. Mobile Manipulation In the 2015 DARPA Robotics Challenge Finals, mobile manipulation solutions primarily relied on pre-built object models and task-specific engineering to enable mobile manipulation [34] . More recent work modularizing tasks into skill primitives and interacting with those primitives using flexible planners, including large language models, has enabled more generalization outside of pre-coded tasks [35, 36, 11, 37] . Imitation learning approaches to mobile manipulation enable joint reasoning over manipulation and navigation actions and generalize across broad sets of tasks [38, 39, 40, 41, 42] . However, imitation learning requires an expensive collection of expert trajectories. In contrast, RL methods can learn from experience without requiring extra human labor for each new task. Decomposing the action space over which the RL policy operates enables more tractable and efficient learning of long-horizon mobile manipulation skills [43, 44, 45, 46] . In our work, we move beyond tasks that involve picking and placing to instead learn skills that require coordination between the legs and arms, e.g., moving chairs or sweeping. Switch task goal G T 16: end while We design our approach to allow robots to autonomously practice and efficiently learn new skills without task demonstrations or simulation modeling, and with minimal human involvement. The overview of the approach we use is presented in Alg.1. Our approach has three components, as depicted in Fig 2 : task-relevant autonomy, efficient control using behavior priors, and flexible reward specification. The first ensures the data collected is likely to have learning signal, the second utilizes signal from data to collect even better data to quickly improve the controller, and the third describes how to define learning signal for tasks. This allows learning difficult manipulation tasks, including tool use and constrained manipulation of large and heavy objects. Next, we describe each of these components in further detail. In addition to providing data with better signal to the learning process, priors can reliably make reasonable progress on a task. This is because they often generalize well, for example, an SE(2) planner will make reasonable progress in moving a robot between any two points in the x-y plane, even when it performs constrained manipulation. We would need to sample many times from the prior to distill this information purely via the data buffer. Hence, a more direct approach is to utilize the prior along with the policy for control. We do this by sequentially executing the prior, followed by the policy. That is, trajectories collected in this manner take the form:EQUATIONThus, the prior structures the policy's initial state distribution, making learning easier. The data collected by the prior is added to the data buffer, allowing the policy to learn from these transitions. 
task-relevant autonomy: 
 Auto-Grasp/Auto-Nav: For safe autonomous operation, we first create a map by walking the robot around the environment. This map is used by the robot to avoid collisions during its autonomous learning process. To ensure data collected involves object interaction, every episode begins with the robot estimating, moving to, and/or grasping the object of interest for the task. The object state is estimated using detection and segmentation models along with depth observations, as described in section 3.3. The robot then navigates towards the object position using RRT* to plan in SE(2) space using the collision map, and optionally deploys the grasping skill from the Boston Dynamics Spot SDK depending on the task. This grasp is generated via a geometric algorithm that fits a grasp location with a geometric model of the gripper, scores different possible grasps, and picks the best one. We do not constrain the grasp type, or on which portion of the object the grasp is performed. This allows the robot to keep practicing regardless of which position or orientation the object might end up in as a result of continual interaction.Goal-Cycles: To prevent robot stagnation near goal states, we set up 'goal-cycles' within tasks, which serve as automated task resets. We show the different goal states used in each of the 4 tasks we consider in Fig. 3 . In the case of the chair moving tasks (Fig. 3 : a-d), the robot alternates between goals that are far apart in the x-y plane, and for the dustpan stand-up task (Fig. 3 e, f ), the robot needs to pickup the fallen dustpan and vertically orient and balance it. For the sweeping task (Fig. 3 : g-h), we use a multi-robot setup for the goal cycle, where one robot holds the broom and needs to sweep the paper bag into the target region (denoted by the blue box), while the other needs to pick up the bag and drop it back into the region where it can be swept. Since we only need learning for the sweeping skill, the robot that picks up the bag runs the previously described auto-grasp procedure.Incorporating Priors: We enable efficient learning by leveraging behavior priors that utilize basic knowledge of the task. This removes the burden from the learning algorithm from having to rediscover this knowledge and instead focus on learning additional behavior needed to solve the task. For example, an RRT* planner with a simplified 2D model can help an agent move between two points in the x-y plane while avoiding obstacles. Starting with this prior, using RL can help the robot learn to recover from collisions and deal with dynamic constraints not represented in the model. Concretely, the prior is a function P (.) that takes in an observation o t and produces an action a t , similar to a policy π(a t |o t ). We can deploy the prior and the policy in the following ways:1.Trajectories are collected independently using either the prior {P (a 0 |o 0 ), . . . , P (a T |o T )} or the policy {π(a 0 |o 0 ), . . . , π(a T |o T )}. Instead of learning entirely from scratch, we incorporate the (potentially) suboptimal data from the prior into the robot's data buffer to bootstrap learning. Intuitively, the prior is likely to see a higher reward than a completely randomly initialized policy, especially for sparse reward tasks. We make no assumptions on the optimality of the prior, and bootstrap learning via incorporating its data. In practice, we first collect trajectories using the prior, to initialize the data buffer for training the online RL policy π(.).In certain cases, the prior might not be robust enough to deploy directly but nonetheless provide reasonable bounds on what actions should be executed. For example, for sweeping an object, the robot's base should roughly be in the vicinity of the trash being swept, but this does not prescribe what exact actions to take. Such a prior can be used residually, where a policy adjusts the actions of the prior at every time step before being executed. These trajectories take the form:EQUATIONThe RL objective is learn parameters θ of a policy π θ to maximize the expected discounted sum of rewards R(s t , a t ):EQUATIONwhere p 0 is the initial state distribution, P is the transition function and γ is the discount factor. For sample efficient learning that effectively incorporates prior data, we use the state-of-the-art model-free RL algorithm RLPD [47] . RLPD is an off-policy method based on Soft-Actor Critic (SAC) [48] , which samples from a mixture of data sources for online learning. Like REDQ [49] , RLPD uses a large ensemble of critics and in-target minimization over a random subset of the ensemble to mitigate over-estimation common in TD-Learning. Since our observations consist of raw images, we incorporate the image augmentations added by DrQ [50] to the base RL algorithm.For flexible reward supervision, we combine semantic high-level information from vision and language models with low-level depth observations. Each task is defined by a desired configuration of some object of interest, so we derive a reward function by comparing the estimated state of the object at a given time to this desired state (see Section 4 for task-specific details). To estimate the state of the object, we start by using an open-vocabulary detection model Detic [51] to obtain the bounding box corresponding to the object of interest. We then obtain the corresponding object mask by conditioning a segmentation model, Segment-Anything [30] , on the bounding box. Finally, using depth observations and the calibrated camera system for either the egocentric or fixed third-person cameras, we get a point cloud. Although this estimation is noisy, we find it sufficient to enable learning effective control policies via real-world RL. This system is flexible enough to handle different objects of interest, such as the chair, long handled dustpan for vertical orientation, or the paper bag for sweeping. Full details on the prompts, detection and segmentation models, and reward functions for each task in the supplemental materials.The results we show for continual improvement during training, as well as the evaluation of the final policies report success rate. Success is defined for an episode in the following manner:• Chair tasks: Max reward in episode is above 1, implying the chair is very close to its target.• Dustpan Standup: Episode ends with a reward of 10 (indicating the dustpan is standing up).• Sweeping: Episode ends with a reward of 10 (paper bag is swept into the goal region). Set yaw hand rotation ω to either +0.5 or -0.5 6:for t = 1 to episode len do 7:Set vertical hand action z to be either +0.2 or -0. Execute A on the robot, record observations, add to D 12: end for 13: return Prior data buffer D For the chair moving tasks we use RRT* for planning a path in SE(2) space with a simplified model that only has 2D occupancy of the top surface of the table, and is not aware of the chair, or robot-chair or chair-table interactions. This generates a set of way-points for the target position of the center of mass of the robot in SE(2) space, in global coordinates. We use coordinate transforms to convert these targets to be in the robot's body frame in order to use the same action space as the reactive RL policy. We are able to perform this computation since we know the robot's body position in global coordinates. Specifically, we have W body = W global * T -1 , where W f denotes the way-point with respect to frame f and T is the matrix transform of the robot body center of mass with respect to the global coordinates. For sweeping, the prior is simply to stay within 0.5m of the last detected location of the paper bag. For dustpan standup we use a simple procedural function to generate trajectories to create a prior dataset, which we detail in Algorithm 2 We use the GraphNav functionality provided in the SpotSDK by Boston Dynamics for Spot robots for generating a map of the playpen. This involves walking the robot around with some fiducials (we use 5) in the arena. This needs to be performed only once, and is used to obtain a reference frame to localize the robot, which is useful to record body pose information and also to implement safety checks to make sure the robot is not executing actions that collide with the playpen railings. While Spot has inbuilt collision avoidance we implement an additional safety layer using the map to clip unsafe actions that would move the robot too close to the playpen railings. For navigation we use RRT* to plan in SE(2) space given the obstacles, using the collision map of the playpen as shown in Fig. 8 . The red region denotes the estimate of the robot's position in the x-y plane, with the blue marking denoting its heading.We use a workstation with a single A5000 GPU to run RLPD online, which requires about 20GB GPU memory, mostly owing to all the image inputs that need to be processed. The detection and segmentation models are run on cloud compute on a single A100 GPU. The fixed third person camera images from the realsense are streamed to a local laptop. Communication between the laptop, workstation and cloud server is facilitated via GRPC servers, and the main program script is run on the workstation, which also controls the robot. Commands are issued to the robot over wifi using the SpotSDK provided by Boston Dynamics.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 
experimental setup: 
 For our experiments, we run continual autonomous RL using the Spot robot and arm system in a playpen of about 6×5 meters, enclosed with metal railings for safety. The playpen is mapped before autonomous operation to ensure the robot stays within bounds and doesn't collide with the railings. The navigation aspect of task autonomy involves searching for objects of interest. Since the main focus of this work is on learning complex manipulation skills, we do not use learning for the search problem; instead, we rely on a fixed camera in the scene. In addition to this, we also use the 5 egocentric body cameras of the Spot while searching for objects. The chair-moving task requires the robot to grasp a chair and move it between goal locations. We consider two variants, chairtablecorner(Fig. 3 a-b ) and chairtablemiddle(Fig. 3 c-d ). The latter is more challenging since collisions between the chair and table base are much more frequent and the robot has to operate in a much tighter space. The dustpan standup task involves lifting up the long handle of a dust-pan (Fig. 3-e ), and then vertically balancing it so that it can stay upright on its base (Fig. 3-f ). Sweeping involves two robots, where one of the robots holds a broom in its gripper and needs to use it to sweep a paper bag into a goal region (Fig. 3-g ). The other robot does not use learning, instead using the auto-grasp procedure to reset the paper bag by picking it up and dropping it close to the initial position(Fig. 3-h ). For each task, we specify success criteria for task completion, which corresponds to reaching the goal states in Fig. 3 . We list the choice of the prior, its combination with the policy, the state measurements used for reward, and reward sparsity in Table 1 .The observation space for RL policy training for all tasks consists of three 128X128 RGB image sources: the fixed, third-person camera and two egocentric cameras on the front of the robot. Additionally, we use the body position, hand position, and target goal. The action space for the chair and sweeping tasks is 5 dimensional, with base (x, y, θ) control and (x, y) control for the hand relative to the base. The dustpan stand-up task is 3 dimensional, consisting of (z, yaw, gripper) commands for the hand, where the gripper open action terminates the episode. We use the same network architectures for image processing, critic functions, policy, etc., for all comparisons. Please see supplementary materials for more details on the full reward functions, success criteria, procedural functions for priors, hyper-parameters for learning, and network details. 
results: 
 Our real-world experiments test whether autonomous real-world RL can enable robots to continuously improve mobile manipulation skills for performing various tasks. Specifically, we seek to answer the following questions: 1) Can a real robot learn to perform tasks that require both manipulation and mobility in an efficient manner? 2) Does performance continually improve as the robot collects more data? 3) How does the approach of structured exploration using priors along with RL, compare to solely using the prior, or using only RL? 4) How does the policy learned via autonomous training perform when evaluated in test settings?Task-relevant Autonomy: Running the robot without auto-grasp or goal-cycles, with the full action space comprising base and arm movement to any position in the playpen does not lead to any meaningful change in task progress even over long periods of time. Further, such operation is unsafe since the robot arm can get stuck in the enclosure railings, or strike the wall in an outstretched configuration. Hence, all the experiments we conduct, including those for baselines, utilize the task-relevant autonomy component so that the robot can make some progress on the task. only prior. Note that we use our task-relevant autonomy approach with all methods. We see that our approach continuously improves with experience across tasks, learning much faster than RL without priors, and attaining significantly higher performance than just using the prior.Continual Improvement via Practice: Given our task autonomy procedure, how effective is our proposed approach of combining real world RL with behavior priors, as opposed to using either only the prior or RL? From Fig. 4 , we see that our approach learns significantly faster than using only RL, and attains much superior performance than the prior, for each of the tasks. On the especially challenging sweeping task which involves tool use of the broom with a deformable paper bag, using only the prior or only RL leads to almost no progress, while our method is able to learn the task. Each robot training run takes around 8-15 hours, with the variation in time owing to different goal reset strategies across tasks and variance in how often the robot retries grasping objects for task-relevant autonomy. Hence, for fair comparisons across methods, we use the number of real-world samples collected to measure efficiency. The system also needs to be robust to many different factors in order to learn these tasks. The training area is exposed to sunlight, and the robot keeps collecting data and learning throughout the day with varying degrees of illumination. Object starting positions and grasps can vary widely, which affects the resulting object dynamics when practicing the task. RL without Prior: For some tasks, using RL without the prior does improve in performance, but at a much slower rate than our method. Without the prior, RL often spends samples exploring parts of the state that are far from the goal. To illustrate this, we plot the average reward over each trajectory for the chair tasks (Fig. 5 ). The reward for this task is of the form -x + e -x , where x is the distance of the chair to the goal position of the chair. The negative mean reward for RL without the prior implies that the distance x to the goal is quite large, meaning that the robot is often far from the goal. On the other hand, since our method executes the prior and policy sequentially for the chair task, our policy always starts out reasonably close to the goal, and can thus can pick up on high reward signal more often, leading to faster learning. We observe a similar pattern for the sweeping task, where using only RL leads the robot to wander around the playpen, greatly decreasing the likelihood of interacting with the paper bag and obtaining high reward.Prior without RL: While the behavior priors are effective at bootstrapping learning, they are not sufficient on their own. This is because they do not adapt or learn from experience, and so keep repeating the same mistakes without improvement over time. We illustrate a qualitative failure example of the behavior prior for the chair moving task in Fig. 6 , where the robot following the RRT* planner runs into a collision state due to the simplified model being used. In contrast, our approach adapts the policy based on its experience to improve its performance, avoiding such collisions. For some tasks like sweeping the behavior prior is much simpler, only providing a constraint not to move too far away from the paper bag, which does not specify how the robot should sweep.Figure 6 : Left: The prior (RRT* with incomplete model) gets stuck in a collision with the table and is unable to recover as the planner does not have a model of chair-table interaction dynamics. Right: Our approach effectively recovers from collisions to complete the task.Final Policy Evaluation: We evaluate the final policies obtained after autonomous, continual practice and find that our approach obtains an average success rate of 80% across tasks from Table 2 . For comparisons between our method and using only RL, we evaluate models obtained with the same number of real world samples. For evaluation, we use the deterministic policy instead of sampling from the stochastic distribution, which is used during training. The success rate of the final policy evaluated on different tasks. For evaluation, we use the deterministic policy instead of sampling from the stochastic distribution like in training. Our approach gets an average success rate of 80%, about 4× improvement over using only the prior or only RL.Further, we set the initial state of the objects to be close to the opposite goal in the goal cycle. For instance, in the sweeping task, we initialize the paper bag roughly in the location shown in Fig. 3 -h. This is different from training, where the paper bag could end up in any location, and success is continually evaluated. We note that on the particularly hard task of sweeping, none of the other methods are successful, while our approach gets 80% success.Prior Data Quality: The behavior prior helps our approach in two ways, by structuring exploration for online learning, and also by providing higher quality data than random search, containing higher reward. To test the quality of the data obtained by the prior, we run offline RL on the dataset collected by the prior. This utilizes the reward of transitions to learn a policy, without any online rollouts.From Table 2 , we see that on the chair and sweeping tasks, the behavior prior data quality is much worse, with an average success rate of 13%. The case of dustpan standup is notable since offline RL performs on par with our method, getting about 60% success. While the numerical performance is similar, there is a considerable qualitative difference in the behavior learned. Our approach learns strategies that are very different from the behavior prior, through exploration. This involves raising the robot's arm and dropping the dustpan, such that it lands upright. On the other hand, offline RL sticks close to the successful examples from the behavior prior generations. 
discussion and limitations: 
 We have presented an approach for continuously learning new mobile manipulation skills. This is enabled using task-relevant autonomy, efficient real-world control using behavior priors, and flexible reward definition. The current approach uses learning primarily for acquiring low-level manipulation skills after objects are grasped. Using automated procedures for navigation and search making use of a fixed third-person camera is a current limitation. This can be addressed by adding learning for the higher-level search problem too, which would allow the robot to rely just on its egocentric observations. This would allow learning in more unstructured, open-ended environments. For each task, there is an object of interest, the state of which is used to compute the reward. We specify the object using a text prompt, which is used by the detection model to obtain a bounding box. This is then used to condition the Segment Anything [30] model to obtain a 2D object mask, as shown in Fig. 7 . For text-based detection we use either Grounding-Dino [53] or Detic [51] .For Grounding-Dino, we append the task-specific prompt to the list of class names in COCO [54] (to avoid cases of false positive detection), and we use Detic with objects365 vocabulary class names. The task-specific text prompts we use are 'chair' for the chair tasks, 'red broom' for the dustpan standup task, and 'box.bag.poster.signboard.envelope.tag.clipboard.street sign' for the sweeping task. The object of interest in the sweeping task is a paper bag being swept and we use many different possible matching text descriptions since it is detected as different classes due to its deformable nature. We list the detection model and the confidence threshold for a detection to be accepted for each task in Table 5 .Once we obtain object masks, we can obtain the corresponding object point-cloud using depth observations. Some detections are rejected based on estimated position, eg: if there is a detection of an object outside the playpen. This filtering is essential since the robot often picks up on known infeasible objects, eg: the box in the middle of the playpen, or some chairs outside the railings. Chair-moving tasks: For this task, we compute reward at every timestep of the episode. Given the estimated chair point cloud using the detection-segmentation system along with depth observations, we estimate the center of mass x t and the yaw rotation w t . Given the goal position g and orientation g w (extracted from the goal image), we compute position x diff and yaw difference w diff norms. Then the reward is given by :r position = -x diff + e (-xdiff) + e (-10•xdiff)r ori = e (-wdiff) + e (-10•wdiff)Total Reward = r position + r oriIn this task, it is difficult to provide reward when the robot is interacting with the dustpan, since the detection model fails to pick up on the dustpan from the third person or egocentric image observations. We can measure reward at the end of the episode (when the robot has released its grasp) to detect the dustpan and estimate the center of the handle x T , and provide a large bonus if the height of the handle (z component of x T ) is above a set threshold. To prioritize faster task completion, we use an alive penalty of -0. Sweeping: Similar to the chair task, we compute reward at every timestep of the episode. We estimate the point cloud of the paper bag, let its center of mass be denoted by x t . The target region is a rectangle, denoted by G r . Let d(x, G r ) denote the distance from position x to the closest corresponding point on the rectangle given by G r . Then the reward is given by:r distance = -0.2 • d(x t , G r ) + e (-10•xdiff) r progress = 10 • max(0, d(x t-1 , G r ) -d(x t , G r )) r bonus = 10, if d(x t , G r ) = 0 0, else Total Reward = r distance + r progress + r bonus C. 
 
 Appendix:  


Decision: The reviewers agree that the system and demo performance is great.  There is also general concern that the system may require a large amount of task-specific engineering.  One reviewer originally felt that the technical contributions were not described with enough clarity and detail, but was happy with the detailed response.  On balance, the feeling was that this paper provides a useful example of RL for manipulation that may inform and inspire future work.

Reviews: review: title: title: Interesting, highly-engineered RL method for mobile manipulation
paper_summary: paper_summary: This paper addresses the problem of performing reinforcement learning (RL) in real-world mobile manipulation settings. The authors propose a learning approach that uses reward supervision via pre-trained models such as Detic and SAM. They also use priors relevant to the mobile manipulation tasks to speed up learning and show efficiency and performance benefits for four autonomous real-world robot learning tasks.
main_review: main_review: - The authors propose an approach that, in theory, does not require human supervision for resets. This focus on autonomous learning in the real world without human oversight is an important and relevant topic for further research in the field.
- The paper is clearly written and easy to follow. The figures and tables are self-explanatory, and the authors do a good job explaining the robot experiments and setup.
However, there are several limitations to the approach:
- A significant amount of engineering appears to have been employed for the tasks where prior knowledge of task parameters, such as how to place the dustpan, is required.
- The priors and the experiment design do much of the heavy lifting. The approach seems heavily scripted for specific scenarios, which raises questions about its ability to generalize to other tasks or environments.
- Missing citations on RL for mobile manipulation:  
[52] Honerkamp, D., Welschehold, T., & Valada, A. (2023). N2M2: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environments. IEEE Transactions on Robotics, 39(5), 3601-3619.  
[53] Jauhri, S., Peters, J., & Chalvatzaki, G. (2022). Robot learning of mobile manipulation with reachability behavior priors. IEEE Robotics and Automation Letters, 7(3), 8399-8406.  
[54] Arm, P., Mittal, M., Kolvenbach, H., & Hutter, M. (2024). Pedipulate: Enabling manipulation skills using a quadruped robot’s leg. In 41st IEEE Conference on Robotics and Automation (ICRA 2024).
questions: questions: - The authors proposed three methods for incorporating priors into their approach: separate, sequential, and residual. A justification for their effectiveness in different contexts would be valuable. Why do particular priors work better in each environment, and how could this be predicted for future experiments on different tasks?
- Why does the performance of RL-only degrade in Fig 5?
- The authors can comment on how they could extend this approach to larger dimensional action spaces, where exploration would be more challenging and time-consuming.
limitations: limitations: 3
review_summary: review_summary: While the authors have shown useful real-world RL results for mobile manipulation, the overall approach isn't particularly novel. In its current form the paper also does not include an evaluation of why some behavior priors work better than others. However, extensive real-world experiments make me lean slightly toward acceptance.
title: title: Initial Review of Continuously Improving Mobile Manipulation with Autonomous Real-World RL Paper
paper_summary: paper_summary: The authors propose a framework to train RL agents for mobile manipulation tasks directly on the real system. To achieve this, they propose 3 components: (1) task-relevant autonomy component that is tailored to enable task-specific data collection; (2) efficient control component that combines a behavior prior with a learning policy for online training and (3) flexible supervision component that integrates off-the-shelf detection/segmentation models for reward estimation. The authors use the framework to train and evaluate RL models on several tasks that involve mobile manipulation with quadruped robot in a simplified real-world environment. Their results show that the different components of their approach improve the sample-efficiency and the performance of the learned policy when compared with baselines that rely only on priors or only on RL training from scratch.
main_review: main_review: The paper addresses a relevant and open problem in RL – autonomous training and improvement in the real world. While there is a limited novelty in the components of the proposed framework, their combination and the application domain (robotic tasks, robotic system), to the best of my knowledge, is novel. Regarding the technical details, the methodology is sound and the descriptions provided in the main text and the supplementary material give enough information to understand the methodology and the experimental setup. 

Strengths:

-	The framework enables real-world training that can learn meaningful policy in a reasonable time
-	The experiments show that the baselines (just priors or just RL) are inefficient or cannot learn a suitable policy, and the combination of components proposed by the authors significantly improves the results
-	The application area (the robotic tasks and the mobile manipulation with quadruped robot) is relatively unexplored by related work

Weaknesses:

-	Limited novelty – it combines existing components in a very general framework 
-	Many decisions are task-specific, which makes it hard to apply this methodology easily on new tasks
questions: questions: -	There are three modes for combining the policy and the prior, is it necessary for a human to decide on the mode, or can it be automated in any way? For the tasks you used, did you decide based on the nature of the task, or did you check different modes and decide based on the empirical results? 
-	You are using off-the-shelf models for detecting the environment state /calculating the reward and you use dense reward in most cases. How often do the off-the-shelf models fail on average in detecting the objects of interest and what happens in this case for calculating the reward in these timesteps?
-	How many runs (seeds) per approach did you do to calculate the mean reward/success rate, and what does the shaded area represent – please specify this for the figures.
limitations: limitations: 3
review_summary: review_summary: I recommend accepting the paper due to its application focus on real systems and the attempt for improving real-world RL, as well as because I think that future work in this field can be informed by the way the different components are integrated or can be used to improve the real-world LR training process.
title: title: Review for Submission 548
paper_summary: paper_summary: This paper presents a system for allowing for online learning in mobile manipulation without human supervision. The authors present several components in their approach, including task-relevant autonomy for collecting data, efficient control by integrating priors, and flexible reward specification. The authors find that combining their learning approach with a prior leads to the best performance when utilizing a Spot robot to perform various tasks.
main_review: main_review: Overall, while this paper does have a nice robotics demo, the main method is missing key detail and it is unclear the exact novelty in the proposed approach.


Strengths
+ This paper studies an important problem, continuous learning for new mobile manipulation skills.
+ The paper contains impressive robot demos.



Weaknesses
- To my knowledge, this work utilizes out-of-the-box RLPD and utilizes a prior to help guide the samples obtained online to better support the reinforcement learning. As these priors are hand-designed, can you comment on the key novelties of this technique compared to prior work and why this approach is better than prior work? For example, robots with intrinsic motivation strategies also have some prior to explore certain regions of the world. Could you also clarify the design of the prior for each task?
- The method section could be much improved with additional details about the priors, learning, and reward specifications.
- The mention of multi-robot early in the paper is misleading.
- The figures could be improved with better annotations. It is unclear what readers should focus on in Figure 3.
- The authors mention that this framework is for continuously learning new mobile manipulation skills. In the results section, it is unclear if the framework is attempting to learn a multi-task policy capable of many skills or if each task policy is separate. If the former, does the ordering of tasks being learned matter?
questions: questions: - Please address the weaknesses noted above.
- Could you comment on the feasibility of this approach? It seems 8-15 hours of training time for each task is relatively large.
limitations: limitations: 2
review_summary: review_summary: The paper is missing details about the method and it is unclear what the novelties of this work are compared to those prior.
title: title: Promising approach to a difficult problem in RL-based robotics
paper_summary: paper_summary: The paper presents an approach to learning manipulation skills in the real world using a mechanism to collect useful training data, using task-specific prior knowledge to aid in training the controller, and flexible automated reward generation. Experiments presented in the paper show the effectiveness of the proposed approach in training a Spot robot to perform 4 real-world manipulation tasks.
main_review: main_review: **Quality, clarity, originality and significance**:

Two aspects of this paper appeal to me the most: (i) The leveraging of task-specific priors that help the robot navigate the search space effectively and helps it to focus on learning the actual manipulation skill (rather than learn how to reach target objects and learn the complete skill from scratch); and (ii) the mechanism for reward specification by utilizing off-the-shelf vision and language models. Training robots to learn manipulation skills with RL directly in the real world is challenging and this paper proposes a novel solution to this hard problem.

**Strengths**

1. The paper addresses a difficult problem in RL (training directly in the real world) and provides experimental evidence to show the effectiveness of the proposed approach.

2. Suitable experiments are performed in the real world involving manipulation skills that are not trivial to learn.

3. The paper is written well and is organized.

**Weaknesses**

1. The caption of Fig. 1, and the keywords prominently mention "Continual learning". For example, the caption of Fig. 1 states: "Continual Autonomous Learning: We enable a legged mobile manipulator to learn a variety of tasks ...". This gives the impression that the robot learns all 4 tasks with the same model with continual/lifelong learning, which I don't think is true.

2. All the manipulation tasks involve handling relatively large objects. It is not completely clear that the same level of performance can be maintained when handling smaller objects (e.g. spoon, an apple, etc.). At least one task with the manipulation of smaller objects would have further strengthened the paper. 

3. Some experiments to show generalizability to different object variants (e.g. different chairs, brooms, etc.), lighting conditions, etc. would have been very nice.
questions: questions: 1. Line 114: Please mention and cite the "geometric algorithm" that generates grasps.

2. Table 1: What is the reason for selecting different policy modes (sequential, separate, residual) for the different tasks? This should be mentioned in the paper.

3. Line 235: Each training run takes a long time (8-15 hours), making it very difficult to train several models or train in parallel (since the physical space cannot be simultaneously used by multiple robots). Can you comment on how your approach can be made more efficient in terms of training time?

4. Line 274: Please explain what is meant by "opposite goal".

5. Line 287: Which offline RL algorithm did you use?

6. How many independent trials were used to compute the reported results?
limitations: limitations: 3
review_summary: review_summary: The paper addresses a difficult problem in RL-based robotics and proposes an intuitive and effective solution. Multiple real-world experiments show that the proposed approach is effective and the work is presented in an organized and coherent manner. Although further experiments on generalizability could further strengthen the paper, I feel that the current results will benefit the robot learning community.