Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper describes the framework to produce walking gaits from on board proprioceptive sensors of an off-the-shelf small humanoid robot, by using Distributional Maximum a posteriori Policy Optimization (DMPO) and processing of visual data directly to policy generation. 

The paper highlights that the reinforcement learning training is executed on the robot in real time with visual input data processed through frames with 3-block Resnet networks for the policy and critic networks. 

The experiments show that the robot learns to walk while avoiding collision with the walls, and can benefit from sharing the learning experience with another robot. 

strengths_and_weaknesses: Strengths:
The inclusion of proprioceptive sensing provides another venue for the learning of gaits and navigation for humanoid robots.
The policy training was realized in real time, so the robot had to be provided with a standing up strategy that allowed it to keep training.
It is stated how the vision system improves on the ground truth pose when walls are added, providing evidence of the importance of the vision system.

Weaknesses:
The paper lacks a theoretical formulation to support the generality of their learning method. Could the experiments be scaled for more complex bipedal robots?
There is little detail about the weights of the reward function or the reinforcement learning action space. 

summary_of_recommendation: The paper requires to be written with a proper formulation of the reinforcement learning algorithm that can be generalized to more complex or different bipedal systems. Also, there are many missing details on the construction and training of the neural networks and their action on the robot. 

issues: More mathematical background is required for the formulation and justification of the proposed reinforcement learning method. The idea of using a vision system for the walking gait is relevant, as well as a discussion about the scalability of the method.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a case study of learning locomotion skills on a small table-top size robot, Darwin OP3. The key achievements include 1) deploying DMPO, a data-efficient off-policy deep RL algorithm, on real robots, 2) training a policy from two robots in different geographic locations, and 3) training a visual policy out of real-world experience. As far as I know, these have not been achieved for real-world robot learning. The authors have demonstrated that the proposed on-robot learning system can train an effective locomotion policy on a small humanoid robot.
strengths_and_weaknesses: + The paper demonstrates a practical case study of learning bipedal locomotion from real-world experience.
+ The paper achieves some state-of-the-art results, such as learning bipedal locomotion, multi-robot learning, or learning visual policies from real-world experience.
- The techniques in the paper are not too novel (because this is a case study).

summary_of_recommendation: This is a case study paper that investigates the scenario of deploying deep RL algorithms to on-robot learning environments. In summary, the presented experiments are quite interesting and can serve as a good data point for other researchers. However, we have to admit that the paper does not contain much technical novelty.

The most impressive result is visual navigation. Although it is hard to say that the robot learns a precise visual navigation skill: rather, the robot seems to estimate the distance to the goal from RGB images and tries to avoid the collision. But I can imagine that visual inputs would also suffer from the sim-to-real gap. It will be great if the authors can highlight the sim-to-real gap a little bit more by comparing simulation-generated images and real observations. It might be pretty obvious, but better to be explicit about it.

Another interesting contribution is to learn a locomotion skill based on proprioceptive observation. I am a little bit surprised (in both good and bad ways) that it works for real bipedal locomotion. It might be a little bit better if the authors can discuss this issue more. Why can we assume non-slip conditions? I guess locomotion of many small bipedal robots accompanies foot slips. Would it be possible to measure the accuracy of this reward function design by comparing it against the groundtruth mocap data?

I also want to see more discussion about the resetting. Human interventions are also needed when the robot is out of bound (happens very often when the policy is matured) or stuck at the corner. Did you also conduct manual resets in this case? I would suggest the authors describe their approaches. 

Simulation results are not presented in the supplemental video. It might be good to include a couple of simulation results as well.

issues: I made most of the suggestions above: please refer to the comments.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a system to train bipedal robot policies directly on hardware, with only on robot sensors, including an RGB camera. Result demonstrates that it is possible to train bipedal policies from scratch on hardware, including learning behaviors such as wall avoidance.
strengths_and_weaknesses: Pro:

1. Learning bipedal locomotion policies on hardware, with RGB camera as sensors.

Con:

1. A manually defined reset controller is still needed. To truly "learn from scratch", it feels like this reset controller should also be learned. And it is not clear how this reset controller should be constructed for larger legged robot. For example, generating a stand up controller for a large quadruped is not trivial (e.h, Learning agile and dynamic motor skills for legged robots, by Hwangbo et al), not to mention a large biped.

2. It feels like using simulation to tune some of the hyper-parameters is still necessary. (Please correct me if I am wrong).
summary_of_recommendation: While this paper presents an approach to learn a policy directly on real bipedal robots (and maybe the first one to do so), one could raise the question whether "learning from scratch" is actually true.

First, a reset mechanism (in this case, a manually constructed controller)  is still needed. And given the complexity of the policy structure, it seems like a simulation is still needed to tune hyperparameters before deploying it on the robot, so the claim that "largely lift the assumption of the availability of a simulation model" is not completely true. This raises the question, since we already need a simulation anyway, why not warm start from a policy trained in simulation instead of wasting additional efforts by training from scratch?

That being said, it is still interesting to see a policy can be learned on hardware directly within reasonable amount of time. Allow training on hardware is definitely valuable (wether it is training from scratch or warm starting), and this paper presents a system to do that.  And using a simple RGB camera is enough for learning wall avoidance is also a nice contribution.
issues: 1. It will be interesting to know how much hyperparameter tuning is necessary in simulation before deploying the system on the real robot. e.g, the reward weights, the history length of the observation. It is not clear to me that I can arbitrary choose these parameters (within reasonble ranges) and the system will still work well on the real robot (or simulation). The supplementary material provides ablation on the resnet module, which seems to indicate tuning in simulation is a necessary first step.
review_title: null

Meta-review: The paper is borderline. Following the authors' responses, two of the three reviewers are positive. The AC is also positive, in part due to the focus on experiments with real hardware, which is encouraged at CoRL. The AC supports the majority recommendation.
Decision: Accept (Poster)