Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a novel learning framework for quadruped to learn coordinated dynamic manipulation and locomotion tasks. Instead of training two separate policies for manipulation and locomotion tasks, the proposed approach learns a single unified policy for arm and leg joints motion. The authors use advantage mixing to reduce the coupling between manipulation and locomotion tasks, thereby improving the learning efficiency. Moreover, the authors propose a novel Regularized Online Adaptation algorithm (similar to RMA) to improve the sim-to-real transfer performance. The proposed framework was in simulation and real-world experimentations through several tests. The results show improved learning efficiency, operation capability, and success rates over baseline approaches.
strengths_and_weaknesses: ### Strength

- It uses advantage mixing to improve the learning efficiency of coupled tasks without compromising on traditional decoupled loco-manipulation approaches.
- This paper shows that coordinating arm manipulation and leg locomotion could improve the robot’s operation space and overall stability.
- The Regularized Online Adaptation algorithm appears to provide good sim-to-real performance.
- The authors clearly explained their simulation and experimental setups and results.
- The trained neural-network policy can realize faster motion when compared to online optimal control (MPC-IK).

### Weakness:

- The dimension of the inputs of the unified policy network is large, which would potentially lead to a very large neural network. While this is a common approach in many end-to-end learning for robotics, it may lead to sampling inefficiency and extended training time. How long does it take to train a policy in simulation?
- The policy outputs joint position commands and then uses PD controllers to track the target positions. However, the joint tracking performance in simulation and robot hardware could be significantly large due to model mismatch, potentially introducing a large sim-to-real gap. How did you account for the PD tracking errors in experiments?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, the paper is well written and the results are interesting. While the robot does not move as smooth as SOTA robots in experiment, it is understandable considering the relatively cheap robotic arm being used. The authors presented a comprehensive framework to learn coordinated loco-manipulation policy with an ability to bridge the sim-to-real transfer gap. While some additional details (such as training time, etc.) can be provided, the work presented would provide an excellent example of the advantages of coordinated loco-manipulation policies in legged robots. I highly recommend to accept the paper to CoRL.
issues: - Some additional details regarding learning efficiency can be provided.
- How did the difference in PD tracking in simulation and hardware being addressed?
- There are some minor typos. Please carefully proofread the paper.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a novel method for whole body control of a legged manipulator using sim-to-real reinforcement learning. The method is based on a teacher-student framework similar to [1], where an encoder uses privileged information in simulation to predict latent environment extrinsics and an adaptation module learns to mimic it from a history of sensor measurements. The output of the adaptation module is fed into a whole body control policy, and the entire system is trained end-to-end. In contrast to [1], the encoder output is regularized to overcome the realizability gap, and the system is trained in a single phase. The authors also propose advantage mixing to provide a natural curriculum for policy learning. Extensive simulation and hardware experiments demonstrate that the learned controller can successfully coordinate the quadrupedal base and manipulator to perform a variety of complex tasks.

[1] A. Kumar, et. al., RMA: Rapid Motor Adaptation for Legged Robots, RSS 2021.

strengths_and_weaknesses: **Strengths**

Comprehensive Experimental Evaluation

The real-world robot experiments are an impressive demonstration of learning based control working on complex tasks that require coordination between large degrees of freedom as well as generating dynamic motions. The simulated experiments and baseline comparisons clearly demonstrate the benefits of every of the proposed training methodology.

Training 

The paper provides novel contributions to the training strategy. Regularizing the encoder output to overcome the realizability gap allows training the encoder and adaptation module in a single phase, which helps reduce the overall training time. Furthermore, the idea of advantage mixing to exploit the causal structure of the locomotion and manipulation is a simple yet effective technique for avoiding local minima.

**Weaknesses**

1. It would be helpful if the authors discuss more details of the low-cost hardware setup and how it can be re-created by other researchers. 

2. The limitations section does not provide any insights into scenarios where the current method will fail and how it can be improved. In particular, the authors should discuss the limits of sim-to-real when it comes to both locomotion and manipulation (especially cases that involve object interaction that can be hard to simulate), and how the current methodology can be potentially adapted to them. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: My recommendation for strong accept is based on the impressive real-world results, a thorough evaluation of different components and interesting improvements in the training pipeline. In the future it would be interesting to see how such approaches can be used to autonomously perform more complicated and fine-grained manipulation tasks.  

issues: The paper draft can be significantly improved from a proof-reading pass to improve several grammatical errors. A few examples,

Line 107 - “rely on” instead of “relies on”
Line 127 - “start” instead of “starts”

The authors should fix these in the final version and address the comments in the weaknesses section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a method for learning a coordinated policy to control a quadruped with mounted arm. There are two key algorithmic components. The first is advantage mixing, wherein the locomotion and manipulation policies are first trained independently, then gradually integrated together with shared reward terms. The second key algorithmic component is Regularized Online Adaptation, a modification of RMA where the environment encoder with privileged information is updated throughout training to more closely match the adaptation module, which has no privileged information. The resulting whole-body controller rolls and pitches the body to help the arm reach, as well as successfully trotting and reaching. This is demonstrated on hardware in three control scenarios.
strengths_and_weaknesses: Strengths:
-  Very nice results. It's awesome to see the torso tilt to stretch the arm farther, especially in the video.
-  Techniques are relatively simple, but they seem to make a big difference in the learned policy.
Thorough ablation studies.
-  I like the idea of Regularized Online Adaptation. I wasn't convinced at first, but after thinking about it, here is my reasoning: Suppose there is some privileged information the encoder is encoding that the adaptation module simply cannot infer from the information it has. The policy may come to rely on that information, but the adaptation module cannot provide it. By training the encoder to match the adaptation module, you ensure the encoder isn't passing on any information the adaptation module cannot infer.

Weaknesses:
-  Section 2.2 (Regularized Online Adaptation for Sim-to-Real Transfer) could be clearer. In particular it would benefit from a brief summary of RMA and a more thorough explanation of the differences between Regularized Online Adaptation and RMA.
-  Line 234 says "The failed trials of our method are largely due to the mismatch between the actual cup position and the AR tag position." I understand that vision is not the focus of this work, but as this comparison with MPC+IK is an important result, this visual estimation issue should be resolved. As one simple option, why not put an April tag on each side of the cup and average the coordinates?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: There are a couple places where the paper should be more clear, but overall it is convincing. The problem addressed is an interesting one: yes, robots with both arms and legs should ideally coordinate their movements, as humans do. The two non-standard algorithmic components (advantage mixing and regularized online adaptation) make sense and are well-investigated with ablations and comparison methods. And the results look good: the robot twists its torso to help the arm reach.
issues: -  Section 2.2 should be clarified. In particular it would benefit from a brief summary of RMA and a more thorough explanation of the differences between Regularized Online Adaptation and RMA.
-  From line 93, I understood that the spherical coordinate system for the end effector was relative to the robot's torso, which would mean the legs could not help the arm reach. From the supplementary material I realize that the coordinate system is actually independent of the torso's height, roll, and pitch. This should be stated in the main paper.

Typos:
-  Line 2, "picking in [cluttered] environments."
-  Line 63, "pre-designed controllers that [can't] be changed."
-  Line 73, "([roll], pitch, and base angular velocities)"
-  Line 139, "[Lagrangian] multiplier"
-  Line 148, "open-loop [replay] of human demonstrations"
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors present a lowcost quadrupped robot with a manipulator. They then introduce a controller that, given commands for the end-effector and base, learns a single policy to control both arm and locomotion to track this command.
strengths_and_weaknesses: Strengths
- the advantage mixing approach seems simple and sensible for this domain
- the regularization during the teaching process seems to work well
- the experiments and videos demonstrate good behavior and the usefullness of the approach

Weaknesses
- the paper does not make fully clear from the beginning that the task is to follow given commands. E.g. l76 p^cmd, v^cmd are just appearing as given. It would be beneficial for clarity to describe that the task is to follow given commands earlier on.
- the paper proposes to jointly train teacher and student instead of in two phases, but it does not evaluate the impact of this. It would be good to compare to the "standard" approach of training in two phases.
- generally the experiments could benefit from a few more additions, comparisons, see list of issues below

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall a good paper. Though additional comparison and a bit more clarity in certain parts would be very beneficial.

Post rebuttal: the authors have addressed my main concerns, with the changes discussed in the process I think the paper is ready to be accepted and I am happy to raise my score
issues: - l65: what does low-cost mean in comparison to the 100k for ANYmal?
- l141: "This optimization process is known to converge." I believe convergence is only guarantueed under certain assumptions.
- l166: Why not give the "separate" baseline access to the same observations, i.e. both base and arm state?
- Minor: there are a number of spelling/grammar mistakes that could be cleared up 
- l200: This section should make clear what is OOD. At the moment the reference to table 3 is hidden in the caption.
- l239: TTC is never explained in the text
- Table 7: would be beneficial to not just have 1 difficult point
- l 251: grouping 17 references into a single parantheses seems a bit much
- Sec. 5: obstacle avoidance? As I understand it, the current policy is blind?
review_title: null

Meta-review: metareview: This paper presents a reinforcement learning framework for learning mobile manipulation using a quadruped with an arm. The main technical contributions are 1) a curriculum learning with only one parameter (Advantage Mixing) and 2) regularized online adaptation for sim-to-real transfer. The trained policy is evaluated extensively in both simulation and hardware experiments.

The paper is well written and the contributions are clear. The reviewers were very positive and raised only some presentation issues, which the authors successfully addressed in the rebuttal. They also conducted additional visual tracking experiments with better target localization accuracy to further reinforce the efficacy of their method. 
confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)