Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Summary: This paper proposes a hierarchical approach to enable visual locomotion from onboard cameras. The high-level planner predicts the desired base pose, velocity and the target for the swing leg at a low frequency which is executed by an MPC running at a high frequency and uses a simplified robot dynamics model. The results are demonstrated in a wide variety of simulated terrains and also on real hardware for the stepping stone tasks.


strengths_and_weaknesses: Comments:
1. The idea of decoupling foot-step planner and leg controller is quite natural and elegant, and seems quite promising from the results presented in the paper.
2. The heuristic controller used in Section 4.3.2 is not very clear and the failure mode analysis of not very convincing. Instead, the authors should use the heuristic method of foot placement as proposed in Section IV-B of [14], which takes the com speed into account before deciding the foot placement.
3. One limitation of the proposed method is that there is a separate policy trained for each of these seemingly similar tasks. It would be good to know the benefits of the proposed method compared to [14], which uses the same heuristic across their experiments and only uses onboard cameras as well.
4. The paper should at least discuss (and ideally compare) to [A] below which uses the same decoupling of a high-level planner and a low-level controller.
5. It is quite interesting to see a simple visual processing pipeline that is quite effective in bridging the sim to real gap. It would be interesting to understand the limitations of this.
6. The paper should talk about the major failure cases and the limitations of their overall method since the reported accuracies even in the simulation are not 100%.
7. The paper is very well written and easy to follow and understand. The figures are great.

[A] Tsounis, Vassilios, et al. "Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning." IEEE Robotics and Automation Letters 5.2 (2020): 3699-3706.
summary_of_recommendation: The paper executes the idea high-level learned visual planner coupled with a low-level MPC-based controller quite well and my current inclination is to accept the paper if the heuristic baseline controller is modified as suggested above.
issues: As mentioned earlier, I would like to see a discussion on the following points: 
1. One limitation of the proposed method is that there is a separate policy trained for each of these seemingly similar tasks. It would be good to know the benefits of the proposed method compared to [14], which uses the same heuristic across their experiments and only uses onboard cameras as well. 
2. The paper should at least discuss (and ideally compare) to [A] below which uses the same decoupling of a high-level planner and a low-level controller. 
3. It is quite interesting to see a simple visual processing pipeline that is quite effective in bridging the sim to real gap. It would be interesting to understand the limitations of this. 
4. The paper should talk about the major failure cases and the limitations of their overall method since the reported accuracies even in the simulation are not 100%.

[A] Tsounis, Vassilios, et al. "Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning." IEEE Robotics and Automation Letters 5.2 (2020): 3699-3706.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a reinforcement learning based **visual policy** for quadrupedal robots to walk on **structured rough terrain** with discrete footholds such as stepping stones and stairs with varying step lengths and step heights. The learning based visual policy takes in two depth images (one each attached near the front and rear of the robot) and provides high-level commands such as desired center-of-mass positions and velocities and swing foot positions to a low-level MPC controller. The policy is validated in a wide range of scenarios, both, in simulation and hardware on the Unitree Laikago quadruped and compared against a heuristic policy and an end-to-end trained policy. 
strengths_and_weaknesses: The primary strength of this paper is that it does a good job of experimentally validating their policies on hardware, and also comparing against relevant baselines. 

As also suggested in the paper, the main idea behind using a learning based approach is the avoidance of building an explicit elevation map and planning footsteps/base motions using that map. Instead, the visual policy outputs the plan for the footholds and base motions directly from the depth camera images and robot states. However, this is not very new and several works have previously looked at combining high-level visual inputs to low-level model based controllers like MPC. 

I see the following limitations:

1. Related Works: There are key references missing from the related works and which might help the authors to further improve on their research as well. See end of this section for the list of additional references and below for a summary:

  (a) Learning based approaches:

  [R1] DeepGait - This paper introduces a learning based approach for legged robots to navigate on discrete terrain using the local height map data as the visual input and outputs desired base positions, velocities, foot positions and gait timing variable, which are then fed to a low-level controller. I believe this is very relevant and similar to the proposed method (except for the visual input representation). 

  [R2] This paper presents a learning based approach for visual navigation in cluttered environment that uses a VAE to build a predictive model of the depth images. This is then used to train a model-based RL policy that outputs high-level commands for a low-level model-based controller.

[R3] is also very relevant to this current manuscript, where a highlevel CNN policy is used along with a low-level MPC controller.

  (b) Control theoretic approaches [R4]-[R5]: There's quite a few works on using control theoretic (and with non-visual inputs) for legged robots to walk on discrete terrain like stepping stones. I believe the authors can use some ideas from these works to improve on the foot tracking performance without significant domain randomization of the dynamics parameters.

2. Dynamic capabilities of the locomotion controller:

The learned policies only consider motions along a straight line. Are there any limitations on the turning capabilities of the policy (especially on the terrain with the stepping stones)? The authors should consider stating the limitations of the locomotion controller. 

3. MPC as a limitation to learn more complex behaviors: As stated in the manuscript, the current policy only considers gaits of a particular contact schedule (trotting/static walk/etc). In this regard, it seems like the MPC places additional constraints and limitations on the achievable robot behaviors. It would be interesting to see if this limitation can be lifted if desired joint trajectories can be learned from visual inputs. The authors do consider an end-to-end approach, however, it is unclear why this has poor performance.

4. Details on the policy implementation:
It would be good to have some more info on whether the current policy can be run on-board or requires compute on an off-board workstation. 

5. CoM x-y position estimation: 
It would be good to have some details on how the x-y positions are estimated on the robot since this is one of the inputs to the vision policy. 

6. Requirement of 2 cameras: Although this is a good engineering solution, I believe that this simplifies the learning problem quite a bit. 


[R1] Tsounis, Vassilios, et al. "Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning." IEEE Robotics and Automation Letters 5.2 (2020): 3699-3706.

[R2] Hoeller, David, et al. "Learning a state representation and navigation in cluttered and dynamic environments." IEEE Robotics and Automation Letters 6.3 (2021): 5081-5088.

[R3] Villarreal, Octavio, et al. "MPC-based controller with terrain insight for dynamic legged locomotion." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.

[R4] Grandia, Ruben, et al. "Multi-layered safety for legged robots via control barrier functions and model predictive control." arXiv preprint arXiv:2011.00032 (2020).

[R5] Nguyen, Quan, et al. "Dynamic Walking on Randomly-Varying Discrete Terrain with One-step Preview." Robotics: Science and Systems. Vol. 2. No. 3. 2017.

summary_of_recommendation: While the paper presents some good experimental results, there are several works that look into visual locomotion for legged robots in challenging terrain with discrete footholds. The use of two cameras simplifies the learning problem and avoids having to create an explicit map or use RNNs/LSTMs. 
issues: It would be good to address the comments under the listed 'Weakness' of the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper provides a hierarchical control that combines a high-level vision-based planner and a low-level MPC controller. The planner is an RL policy that takes the processed raw image data to decide the target CoM pose trajectory and the target feet locations. The main contributions include:
1. Use raw image data as the policy input instead of a processed height map.
2. Learn not only the foot location but also the target CoM trajectory.
strengths_and_weaknesses: The paper is written clearly with enough discussion, explanation. I do appreciate the detailed appendix that provides useful information to elaborate the content in the main paper. I think the control method described in the paper is reasonable and the simulation results are good. I support the argument in the paper that learn both CoM pose and foot location is better than learn foot location only. 

For the weak points, the paper may miss one case in the discussion. If we two actions: CoM pose and foot location and two controllers: learned and heuristic, we can come up with four combinations: 1. learned both CoM pose and foot location; 2. learned CoM pose, heuristic foot location; 3. heuristic CoM pose, learned foot location; 4. heuristic both CoM pose and foot location. The paper's method is 1. and the baselines are 4 and most of the literature is 3. However, citation [18] is case 2 where learn only and CoM pose and can pass all the tasks in the current papers. I would suggest discussing [18] in 4.3.2 Heuristic-based foot placement. It would support the last sentence that landing positions should compatible with the robot's CoM speed. 

Other weakness points are:
1. Prefer more hardware experiments, like stair climbing. The current experiments are easy. The simple QP stand control + Raibert's heuristic foot placement can complete the tasks. The gap avoidance just needs to find the nearest feasible ground to adjust the target landing positions. I have tested this on A1. 
2. Prefer elaborating more about the action space. The MPC requires a trajectory of CoM and foot locations, not just an instant one. The action space in line 115 looks like a single state of CoM and foot locations. So just to be clear, does the policy output q_d(t=1) or {q_d(t=1), q_d(t=2), ...q_d(t=H))? So does the foot location. 
summary_of_recommendation: The paper provides a clean hierarchical control strategy to include raw images in the legged locomotion control. Learning both CoM and foot location is useful. 
issues: 1. Please discuss citation [18]. Even if it uses heuristic foot placement, but performance-wise, it looks no different than the current method. 
2. Please elaborate more about the action space. The way it is present in line 115 looks like a vector with dimension 20, but MPC requires a sequence of target states as input.
3. Sec 2 second paragraph, should you cite DeepGait?
4. Line 112, there is no Appendix A.5. Should be A.4
5. Appendix line 444, why it is SQP, not a QP? My understanding is SQP is a nonconvex problem, but the formulation looks the same as MIT chhetah3 convex-MPC formulation.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a hierarchical vision-based controller for quadruped locomotion on a variety of random terrains. The high-level component is trained using DRL and specifies the walking pattern for the legs and the robot's base. The low-level component separately controls the swing legs and the stance legs to realize the movement asked by the high-level controller. The system is successfully tested on a robot using zero-shot sim2real transfer.
strengths_and_weaknesses: Strengths:
- The system design is straightforward.
- The observation and action spaces are defined in a compact way.
- The approach is tested in a wide range of randomized environments with good results. Comparision to the baselines and the ablation study look convincing as well.
- The paper is well-written. I couldn't find any typos.

Weaknesses:
- The training details are only explained in the appendix. Even there the paper does not report any learning curves or the number of simulation budget for the training.
- It seems that the approach only supports the gait that is encoded in the gait scheduler.
- It would have been nice to see how the model works using other SOTA RL algorithms such as SAC & TD3.
summary_of_recommendation: I liked the paper since it is well-written, the approach is clear, and the results are looking good. I am a bit suspecious that the learning contribution is minor, but that should be OK cosidering the challenges of deploying such a system on an actual robot. I am happy with the paper as it is, but below you can find my suggestions for improving it.
issues: - Explain the training process details in the main paper body.
- The policy archicture is only briefly explained in the appendix without any accompanying text. For example, I'd like to know if the vision inputs are processed using CNN networks. 
- If possible, test the approach with other SOTA RL approaches such as SAC & TD3.
review_title: null

Meta-review: The author propose a deep vision based policy to enable a quadrupedal robot to walk on various terrains.  In particular, a RL policy is proposed that takes visual information as input and outputs commands to a low-level MPC controller. Simulation results as well as experimental validation is presented.  The reviewers agree that the paper is well written and presentation is good.  A comparision with other methods would be a good addition to further strengthen the contributions of the paper and establish why the proposed hierarchical policy is better.  Reviewers also point out several related papers that should be part of the literature review and potentially used for comparison.  Please also discuss failure cases.

I am recommending an accept as a poster.
Decision: Accept (Poster)