Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an experimental setup for learning locomotion controllers for  legged robots, with an emphasis on robotic quadrupeds. The system consists of a simulation pipeline on which multiple robot simulations can be run in parallel, along with a policy optimization algorithm that combines data from all simulations. The proposed simulation environment is shared by multiple instances of simulated robots. The environment is divided into tiles with different terrains, which pose challenges of varying difficulties to a walking robot. This setup is used to design a curriculum where robots are assigned a tile, a desired heading and desired velocity, and are promoted/demoted to harder/easier environments depending on whether they exit the tile at the desired heading. The paper shows how this setup can be used to train walking controllers with online policy search (PPO) on a single GPU (with 48gb of VRAM)
strengths_and_weaknesses: The presentation and the engineering work done to produce the results in this paper are very itneresting. The proposed approach of large scale data parallelism on a single GPU is definitely relevant for robotics research.

It is not clear whether the technique for reset handling described in section 2.2.2 is doing anything different from existing implementations of PPO. Reset handling as described in this section is already present in other implementations of PPO (e.g. in openai baselines and the spinningup repositories), which use the standard gym interface.

Since the paper makes an emphasis on training on a single GPU, it would be very useful to have an evaluation in terms of available GPU RAM given that not many practitioners have access to the GPU used in this work. 


summary_of_recommendation: This is a neat demonstration of how existing algorithms for policy search can benefit from scaling up data collection in simulation. This paper describes the engineering decision taken to enable training on a single GPU. 
issues: Some references to related work on curriculum learning for locomotion [1] and [2], and on adapting PPO to parallel simulation [3].

[1] Paired Open-Ended Trailblazer (POET):
[2] ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills
[3] Accelerated Methods for Deep Reinforcement Learning
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper demonstrates that it is possible to use a GPU-based physics simulator to run thousands of simulations in parallel, which can greatly accelerate the DRL training of quadruped locomotion. Batch size hyper-parameters are tuned to adapt to this setting, and a curriculum of "different levels" with gradually increasing terrain difficulty is designed. The learned policy can also be deployed on a real robot, showing its robustness.
strengths_and_weaknesses: The paper's main strength is the quality of results -- as the authors claimed, it was not previously possible to train quadruped locomotion on a single workstation within minutes. The presentation of the material is also pretty clear.

There are two apparent weaknesses for papers like this. 1. lack of technical novelty and 2. lack of open-source implementation. The reviewer would especially like to emphasize the importance of 2 -- CoRL has long been advocating for open-sourcing; if the authors do want to publish their findings at CoRL, they should seriously consider packaging and releasing their implementation to help real advance of the community.

Related work (GPU for massively parallel DRL, with open-source code: https://graphics.stanford.edu/projects/bps3D/)
summary_of_recommendation: The results of the paper look impressive. The main reason for "weak" accept is that without open-sourced code, it is difficult to validate how fast the implementation really is.
issues: As discussed, code release will make this paper more consistent with the spirit of CoRL.

Question:
Is it possible to reset simulation to previously visited states, rather than always close to the starting pose? Seems like if so we can worry less about number of consecutive steps being too small?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper explores learning locomotion policies for quadruped robots using massive parallel simulation. The main technical contributions of the paper is an adapted setting of the PPO algorithm for massive parallel simulations, as well as a curriculum learning scheme for training the policies to tackle complex environments. They also demonstrated that the resulting control policy can be successfully deployed to the real robot.

strengths_and_weaknesses: Strengths:
The main strength of the paper is that it demonstrates the possibility of leveraging large-scale simulations to learn legged locomotion for a real robot in a very short amount of time. The modifications to the standard PPO setup in response to the issues arising from the massive parallelization is reasonable and supported by their experiments. The proposed curriculum scheme also makes sense.


Weaknesses:
1. Some details in the paper are not clear. For example, the paper mentioned that the episode length for each simulation is short (<5s) due to the large number of robots being simulated. At the end of the episode, is the robot reset to the initial state? The paper also mentioned that a random push force is applied every 10s to the robot. How is this 10s computed since it will span multiple episodes?
2. In [10], the GPU-based simulation was also used to learn several robotic tasks in a short amount of time, similar to the presented work. It’s not very clear what’s the main difference between the two works in terms of large-scale training. For example, if one takes the learning algorithm from [10], applies the curriculum learning scheme and reward function in this work, what would the result be like?
3. The proposed method converged to the same locomotion gait (trotting) for all three robots tested. This is good for real robot transfer given the stability of trotting gaits, meanwhile the lack of diverse behaviors in the robots is a bit unsatisfying. It would be nice if more locomotion forms can be tested such as bipedal walking.

summary_of_recommendation: The paper presents an interesting direction in learning robot locomotion controllers with massively parallelized simulations. The result is solid in general with real robot validations and can inspire further research in this direction. The several technical details introduced in the paper to help improve training performance is also reasonable and effective. However, there are a few issues with the paper such as missing details, limited discussion with relevant work, and limited diversity in the learned behaviors. It would be a good contribution to CoRL if those can be addressed.
issues: As mentioned in the weakness part, I think there are three issues I’d like to see more discussions/experiments:

1. How is the perturbation applied to the robot during training? Is the robot reset if the episode ends due to time-out?
2. Add discussion on how the work differs from [10] and ideally experiments to show that the delta in the learning algorithm makes a significant difference.
3. It would be great to have more diverse behaviors emerge from this nice framework, for example biped locomotion or manipulation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Main Contributions
================
A new training setup for model-free reinforcement learning that allows training policies for quadruped robots very efficiently.
An ablation study that investigates the impact of batch size and number of robots simulated in parallel on final performance.

Summary
========
The paper demonstrates a learning strategy for quadruped locomotion that is efficiently implemented entirely on the GPU and allows massive parallelization during training time. 
Combined with a game-inspired curriculum learning, the policies successfully learn to navigate complex terrain and also transfer to a real robot. 
In the experimental part, the paper presents an ablation study of batch size and number of robots on the final performance and training time required. 



strengths_and_weaknesses: Strengths
=========
- Nice video!
- The paper is well written. 
- This training methodology would be very useful if made open-source, as it might be generalized to other robotic platforms as well. 

Weaknesses
===========
- The paper contains very limited scientific novelty. Previous works [1, 2] have already demonstrated the feasibility of training a navigation policy for a quadruped robot in simulation and deployment in the real world. 
- The game-inspired curriculum is the main novelty of this work, but is not compared to the previous approach used for training (particle filter approach proposed in [1]).


[1] Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V. and Hutter, M., 2020. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47).

[2] Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V. and Hutter, M., 2019. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26).
summary_of_recommendation: The usability of this work is highly dependent on the fact if the code will be available open-source. Due to the very limited scientific novelty (i.e. the game-inspired curriculum), the main purpose of this paper would be to publicize a new highly parallelizable training framework. 
issues: The paper would be stronger if the game-inspired curriculum would be compared to the previous, particle-filter based approach of curriculum learning.
review_title: null

Meta-review: This paper addresses the acceleration issue on the DRL training for quadruped locomotion, and presents a training set-up by using massive parallelism on a single GPU workstation, and a novel game-inspired curriculum well suited for parallel training.  

I agree with reviewers that this paper is well organized and the experimental results is quite promising, while the scientific novelty seems to be limited, reviewers also concerns on the lack of open-source details.  

Reviewers also pointed out several other concerns or deficiencies about the paper. I agree with these comments which might be valuable in improving the quality of the paper, mainly including:

1. Comparison or discussion to some related works, both on quadruped robot locomotion and the game-inspired curriculum learning.
2. More expeirmental details to make clear some points reviewers raised, and more locomotion forms like bipedal walking tested if possible to make the paper more solid.
3. Other minor points as reviewers mentioned.

More added: 
I would like to thank authors for your carefully and detailed responses to comments raised by reviewers.
Although there are still concerns on the limited scientific novelty, the scores of this paper was improved after rebuttle. 
I thus recommend this paper to be accepted.

Decision: Accept (Poster)