Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper present a reinforcement learning method for training a grasping and navigation policy for a mobile manipulation robot, that learns to clean up the floor. The exploration strategy for the navigation leverages the uncertainty over the grasp success, as to explore states that will increase the probability for grasping success. The policy was trained with a physical robot in a real setting, with several training choices, like pre-training the grasping policy with automatic reset of the learning episodes and an autonomous curriculum.
strengths_and_weaknesses: I found the idea of the automatic resets a nice design decision for the problem of learning mobile manipulation. 
A major weakness is the final behavior of the robot that seems to be pushing objects as obstacles instead of stopping and grasping them. I find the idea of proposing to solve mobile manipulation in an end to end fashion is rather greedy. Moreover, the task- oriented policies are not a new idea in robot learning, and the combination of grasp success to a subsequent policy has been previously proposed, but perhaps not in the exact same setting of this paper.
summary_of_recommendation: I have three main concerns that drive my recommendation.
1. Since top-down grasping has been generally solved, and in particular the method described resembles quite much the DexNet method, I don’t see why the authors overly analyze this part. Moreover, The objects used are very trivial, and do not correspond to objects found in a real home environment. The grasping policy is also rather simplified in an x,y position, that makes the learning rather easy. 
2. Not sufficient comparisons with baselines. The authors do not refer or compare with current works on mobile manipulation, that show these benefit of coupling learning with planning, The authors also do motivate why model-free RL is a suitable method for solving mobile manipulation tasks.
3. It is not clear how the robot learns to avoid obstacles, as in the provided video it seems like the robot is sometimes pushing away objects that are fallen in the ground and should have been grasped.
issues: UPDATE: after the rebuttal I raised my score to weak reject.

- The related works are missing references to recent papers on the topic of learning mobile manipulation, like: 

 https://arxiv.org/abs/2103.10534

 https://arxiv.org/pdf/2006.04271.pdf

 https://arxiv.org/pdf/2101.05325.pdf

 https://arxiv.org/pdf/2003.02637.pdf

- The preliminary section is wrong. The POMDP definition you provide is wrongly stated. The definition of the reward containing observations is wrong. There is also an abuse of notation. Moreover, in the subsequent methodology, you do not provide an explanation as to how you handle partial observability. Therefore, this preliminary section is not connected to the described methodology. 

- The grasping policy is resembling the method of DexNet. Why is it important to learn these features for top-down grasping, and you do not rely on well-known and well-performing methods for top-down grasping? Moreover, you are over-simplifying the grasping problem to a simple 2D position on the thrown objects on the ground, hence computing 2D grasps, while I would argue that 6D grasps are more important for real world problems. Do you think that if you were to train a more complex grasping policy for 4D or 6D grasps, the behavior of the subsequent navigation policy would be the same? How would the higher uncertainty in orientation estimation affect the whole learning process? In algorithm 1, what is N, and pt? Please denote all notation. Finally, what is the observation? Is it RGB, depth, or point cloud? Are you employing some additional object detector to decide on the amount of objects? How is the high-level decision of the number of objects to be grasped being decided? Is this parameter already known?

- It was a bit hard for me to follow section 4.2, in the part describing the relabeling that you mention. Moreover, please state all used notation, e.g. what is $r_{n-std}$, what is n? What is weird for the navigation task, is that there is no reference to an obstacle avoidance objective. How can your navigation policy handle collisions, what are the sensorial data employed that will allow the agent to understand the difference between obstacles and objects to be grasped? In the video, it seems like the robot sometimes overrides objects on the ground. Is this a good behavior? Would this strategy be transferable to realistic mobile manipulation platforms? 
On another note, you state that SAC is data efficient. Why is an algorithm that needs millions of samples to converge data efficient? This is a false statement. 

- The experimental comparison is not fair. You should have compared your method to other approaches in literature that consider interactive navigation, like https://arxiv.org/pdf/2101.05325.pdf and https://arxiv.org/pdf/1910.11432.pdf
The comparison of only a random navigation strategy is not aligned with the claimed objective of showcasing advances in learning of mobile manipulation. 
Moreover, I would expect an ablation on different grasping strategies, rather than a simple comparison to a scripted strategy. The plots of Fig, 3 are difficult to interpret, and I am not sure why the curves are interrupted. Also, you do not state how many times you have conducted these experiments, and what is the uncertainty, as the mean behavior is very noisy, and it doesn't look like the algorithm converged to a good behavior, e.g., in Fig. 3top, both blue and green curves are oscillating, and interrupted. There should be sufficient explanation on the presented results, and I would suggest to report success rates at least on 5 experiments with different random seeds. Additionally, the results of Fig. 4 are not allowing to derive sufficient conclusions on the algorithmic decisions. 

Finally, I would recommend that the authors do not make strong claims in their conclusions that are not justified by the results, and the absence of comparisons with state-of-the-art works.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contribution of the paper is a robotic reinforcement learning system that learns mobile manipulation in an autonomous way in the real world, without human intervention or instrumented laboratory settings. To be more specific, the paper presents 1) a modular navigation and grasping policies with onboard sensing observations that make use of uncertainty-based exploration, 2) pseudo reset mechanism to minimize necessary human intervention, and 3) curriculum learning with two variants of grasp pre-training.

strengths_and_weaknesses: The paper presents a solid robotic system that supports 40 hours of real-world reinforcement learning training with minimal human intervention, which is quite impressive. The solution also has some level of algorithmic novelty that includes uncertainty-based exploration, control hierarchy of navigation and grasping, and curriculum learning. The authors extensively evaluate their solution against several baselines in a few different task setups (with and without obstacles, with and without rugs, diverse objects, varying lighting conditions, etc) and achieve favorable results compared to baselines. The authors also conduct several ablation studies in simulation for different action space and curriculum methods, and demonstrate the validity of their algorithmic design choices. 

One main weakness of this work seems to be the narrow applicability of the proposed method. Although it’s mentioned the ReLMM could be used to learn other mobile manipulation tasks other than room cleaning tasks, it’s doubtful whether this is actually the case. For instance, both the pseudo-reset mechanism and overall method (Algorithm 1 and 2) seem to only work with the room-cleaning task specifically. The paper would be a lot stronger if it shows ReLMM can also be applied to a different family of tasks, e.g. pick-and-place tasks that require robot base movement. The second main weakness is the mixed results in Sec 6.3 and Appendix C and the lack of clarity in the analysis. It’s claimed that the navigation reward relabeling helps sample efficiency. Yet in Figure 4 Left, the results are very mixed, without proper analysis or hypothesis for why this is the case. Same for autonomous curriculum; It was claimed that with proper parameter tuning, autonomous curriculum can learn almost as fast as stationary curriculum (Figure 4 Center), but the opposite result is shown in Appendix C.2 and Figure 8. In general, Sec 6.3 and Appendix C has room for improvement.

summary_of_recommendation: While this paper stands as a strong system paper with real robot experiments and produces favorable results over baselines in the task of interest, there still remain questions about the generality of the proposed method and the validity of all the algorithmic design choices. A thorough response to the listed issues below will help inform the final recommendation.

issues: Main issues to be addressed
* Better showcase the generality of the proposed method, (e.g. how can the system and the algorithm be easily adapted to new types of mobile manipulation tasks)
* Improve writing Sec 6.3 and Appendix C: clarify whether navigation reward relabeling helps and whether autonomous curriculum can be as efficient as stationary curriculum
* Real world experiment of autonomous curriculum with the proper settings: it would be great to demonstrate autonomous curriculum with proper settings (e.g. the best parameters you found in simulation) in the real world across all four environments (currently only one environment is included in Table 1)
* Add related work for RL for mobile manipulation tasks in simulation that only relies on onboard sensors, e.g. [1]. Note that [1] does not “require external instrumentation in the form of top-down camera views, oracle knowledge of object pose, or precomputed navigation maps”. The top-down occupancy map used in [1] is generated from onboard LiDAR signals.'

Miscellaneous issues & questions
* Algorithm 1: it should be G^M, instead of G^N?
* For the navigation reward, why -1 is needed from the grasping reward?
* It’s understood that the grasping problem is formulated as a contextual bandit problem (with episode length = 1). What is the episode length for the navigation problem? Is it also 1, infinite, or the steps between two pseudo resets? 
* Fig 3 Left and Fig 4 are very dense and hard to read.


[1] Xia, Fei, et al. "Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation." arXiv preprint arXiv:2008.07792 (2020).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a novel reinforcement learning framework for a mobile robot to autonomously learn to perform manipulation tasks with on-board sensors only.  
strengths_and_weaknesses: Strengths and Weaknesses
The proposed framework requires no extra instrumentation and few human interventions. The navigation policy could be learnt to support the object manipulation tasks through the influence of grasping rewards. Experimental results show the advantage of learning both grasping and navigation policies in real world tasks.

The grasping actions are discretized to simplify the learning of grasping policy, which is quite reasonable for an initial study. But I am also very curious about applying this framework to more complicated manipulation tasks that require continuous action spaces in future studies.

summary_of_recommendation: The proposed learning framework enable the mobile robot to achieve better performance on real world tasks than other existing methods while being mostly autonomous during the learning process. It could effectively save human effort and other equipment usage, which may be significant for mobile robots operating in a large-scale environment. Hence I think that it serves as a good reference for relevant research in the future.
issues: 1. Some typos: Page2, Figure 2 caption, line 4, ‘succes’ should be ‘success’; Page 7, line 307, ‘making is difficult’ probably should be ‘making it difficult’; Appendix A.2, line 516, ‘is pass’ should be ‘is passed’.
2. According to section 4.4, it’s said that in stationary curriculum, grasping policy training (Algorithm 1) is performed multiple steps when the robot base keeps being stationary, and does not mention about the navigation part, hence I initially thought that the stationary curriculum uses a stationary robot in comparison to the mobile robot used in the autonomous curriculum. But the experimental results shown in Figure 3 suggest that stationary curriculum also trains a mobile robot. So now I guess that the stationary curriculum also follows Algorithm 2 except changing the last argument from 0 to 1. Is my understanding correct? And could you please also clarify it in the paper?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a deep reinforcement learning approach for training an agent to both navigate and grasp objects in the real world given only an RGB video feed. 

The key idea is to decompose grasping and navigation into two separate modules, with manipulation being trained to estimate the uncertainty of grasp success with rewards given for successful grasps, and navigation being trained to navigate in such a way as to minimize this uncertainty. 

Two forms of curriculum are also suggested: (1) the robot is stationary and only does grasping actions, with some human intervention to ensure objects are present, and (2) the robot both navigates and grasps, but performs grasping actions more frequently than navigation actions. To continuously learn in the real world without human intervention, a pseudo-reset mechanism is employed in which the robot randomly moves and releases a grasped object and moves to a random location. Compared to prior work, this overall approach allows for training directly in real world without any input information except an RGB observation stream, which can be done in the span of dozens of hours. 

Evaluation for training in the real world shows the approach to be able to achieve high success rates. Several baselines that do not use different grasping and navigation modules or a curriculum are also evaluated, and attain low success rate. Evaluation in simulation also presents ablation results that justify the need for the various components of the system. 
strengths_and_weaknesses: Strengths:
* The problem of training both navigation and manipulation actions using only raw sensory inputs and a reward function is useful to address, with deep RL only having been explored in this setting in a few prior works
* The paper is clearly written and has few ambiguities, and is easy to follow
* The claim that this approach requires less knowledge about the environment and less human intervention is accurate and significant 
* The key new ideas are well suited to the task, and are mostly novel. In particular, using the grasp uncertainty to reward good navigation is a good idea. See caveat below.
* Experiments are partially well done, with real world results presenting impressive success rates and simulated results presenting more detailed justification of the need for parts of the approach. However, see caveats below.
* Compared to prior work, demonstrating learning results entirely from real world experience is impressive and indicates the approach's usefulness.

Weaknesses:
* While the related work surveyed is fairly comprehensive, the work "HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators" is not cited. This is problematic, as this work has a similar concept of decomposing the policy into manipulation and navigation components, and training primarily from raw sensor inputs. Its manipulation and navigation policies can arguably be change to this paper's action space and the curriculums suggested could be applied to its approach, meaning that there is significant overlap conceptually. The This work does use additional inputs to the agent such as door locations, but this is not strictly required. 
* Additional citations for hierarchical RL approaches would also make sense, given that they also decompose the skills needed to learn (even if not specifically for the task of mobile manipulation). 
* Unlike prior work on deep RL for mobile manipulation, this approach does not allow for learning arbitrary mobile manipulations skills, and is instead specialized to navigation and grasping 
* The approach is evaluated in a simplified setting of having simple ball-like objects scattered on the floor, in a space of limited size that does not require much navigation. This makes it so both the perception and manipulation aspects are significantly simplified compared to prior tasks in deep RL for mobile manipulation. It is unclear whether this approach can scale to more complex settings.
* The evaluation in the real world is not presented very clearly, with Figure 3 having only partial learning curves and Table 1 having missing entries. This is especially problematic for Figure 3, since the shown curves are quite chaotic. Additionally, variance values are presented in Table 1, and not in Figure 3. These things combined make me worry that the results are partially cherry-picked.
* The baselines being compared to are very simple, with HRL4IN being a much more suitable algorithm to compare against.
summary_of_recommendation: I recommend rejection, despite this presenting ideas with good originality, technical quality, and potential impact. This is primarily due to issues I presented with respect to the experiments section, and due to weak baselines that do not include an approach similar to HRL4IN, which is also not cited and not compared to (which reflects my evaluation of clarity). However, I hope the author response and revisions could be sufficient to address these issues, in which case I would likely recommend acceptance.

Update after responses:
I have updated my recommendation to weak accept from weak reject - see my reply to the authors in comments below.
issues: * Cite HRL4IN and compare your approach to their in text. Ideally, also compare your approach to a policy similar to theirs experimentally
* Explain the experimental protocols in greater detail, 
* Provide more results for the real world experiments, in particular by presenting full results for Figure 3 and Table 1

Update after responses:
As stated in my comments below, I believe the writing needs to be updated to more strongly reflect the limitations of the contribution in terms of lack of human intervention and the degree to which the system design as opposed to task design allows for efficient learning.
review_title: null

Meta-review: This paper presents a real robotic system that is trained to learn navigation and grasping skills via reinforcement learning. The system is capable of learning clean-up actions based solely on onboard sensing capabilities over 40 hours of training. The components of training curricula and pseudo-resets have been appreciated by the reviewers.

Although clear and easy to follow in general, as pointed out by the reviewers, the paper presents technical inaccuracies and the methodology should be clarified.

A common issue raised by the reviewers is the deficiencies in the evaluation which seem to be set more like a semi-toy-problem that seem to  suit the method. In particular, comparison with appropriate baselines and a lack of more realistic, general tasks have been missed.

The reviewers made several comments regarding a lack of related work and appropriate references, including related work in learning mobile manipulation and hierarchical RL in mobile manipulation (HRL4IN). 

Post-rebuttal ====================

Thank you for answering our questions and requests, including the addition of a baseline (HRL4IN) and for clarifying HRL4IN sample inefficiency. Overall, this paper has been well received and the reviewers have been positive about the author's response..


Decision: Accept (Poster)