Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to incorporate the proprioceptive and visual information together for quadrupedal locomotion. The authors introduce a new model architecture named LocoTransformer that consists of separate modality encoders for proprioceptive and visual inputs, the output of which is fed through a shared Transformer encoder to predict actions and values.

Through experiments, the authors demonstrate that the robot, with the help of both the proprioceptive and visual inputs, can walk through different sizes of obstacles and even moving obstacles. They have also transferred the learned policy from simulation to a real robot by running it indoors and in the wild with unseen obstacles and terrain.
main_review: [Strength]

This paper tackles an important question of how to incorporate visual information in learning policies for quadrupedal locomotion, where most existing learning-based control of quadruped robots in the published works only considered proprioceptive information, and the robots are essentially "blind."

The use of visual information can allow the robots to be less conservative and plan their actions for a longer time horizon, as has been evident from the authors' comparison with a state-only baseline that only considers proprioceptive inputs.

This paper has extensive experiments and in-depth analysis in simulation, which provides a good reference for the readers to understand the benefits and limitations of different design choices.

The real-world demo from sim-to-real transfer also provides concrete empirical evidence on the practical use of the proposed method.


[Weakness]

While I like the direction this paper is going, I'm not entirely convinced that the real-world experiments in the paper fully demonstrate the necessity of visual information. For example, in [1, 2], the authors have shown working demos on terrains seemingly much more challenging than this paper. [1, 2] also showed examples of stair climbing, a task where vision is supposed to be extremely helpful: a blind robot may have to make a few failed trials before it knows the height of a stair. You could also imagine the benefit of vision in cases that require more precise footstep planning (e.g., https://youtu.be/k7s1sr4JdlI?t=176). The paper will be much stronger by including some more concrete comparisons with the current state-of-the-art learning approaches on what can be made possible via vision while previous blinds robots struggle.

While I agree that vision is important for robots to make long-term plans and autonomously traverse around obstacles, I'm not sure whether this paper's approach is better than more classic robotic pipelines. For example, instead of treating the depth image as a 2D grid and processing it using CNN, one could use the depth camera to build a 3D map of the surrounding environment and blend in the explicit notion of what's traversable and what's not. You can then plan the trajectory based on the perception results. This seems to be how Boston Dynamics' Spot uses the visual information (https://www.youtube.com/watch?v=Ve9kWX_KXus) and has shown great generalization ability in real-world scenarios — showing examples of how this paper's way of using visual information is better than classic pipelines may be essential to claim improvements.

Continuing my previous point, it would be better if the authors could include more discussion on the current state of quadrupedal locomotion both in academia and industry, where Boston Dynamics' Spot is seemingly better in terms of generalization and robustness than any of the reinforcement learning-based approaches. The authors may also shed light on under which scenarios we should choose RL-trained robots over Spot.

How does the method work in the real world if there are moving obstacles, e.g., humans and other animals?

How well does the method work compared with the built-in controller of the robot?

The paper may need a few passes of proofreading where the current manuscript includes a lot of typos, just to name a few:
Section 1, contribution bulletin points: We the propose LocoTransformer, ...
Section 3: a MDP --> an MDP
Section 6: The visual inputs also inputs the locomotion ...


[1] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter, "Learning Quadrupedal Locomotion over Challenging Terrain"
[2] Ashish Kumar, Zipeng Fu, Deepak Pathak, Jitendra Malik, "RMA: Rapid Motor Adaptation for Legged Robots"

=====================

[Post Rebuttal]

I thank the authors for the detailed feedback and additional experiments, which addressed most of my concerns. Great job! I have also read the reviews from other reviewers and decided to raise my score to 8: accept, good paper.
summary_of_the_review: I like the direction this paper is going: combining visual and proprioceptive information to train RL agents for quadrupedal locomotion. I also love that the authors include real-world demos of the learned policy on a physical robot. However, my main concern is that, although there are extensive evaluations in the simulation, the current set of real-world examples may not be sufficient to show the benefit of visual inputs. Examples like climbing stairs or scenarios that require more precise footstep planning would make the paper much stronger.

I'm generally excited about the progress in this direction, thus I'm currently leaning towards the acceptance side, but I hope the authors can address the issues mentioned above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a transformer based architecture that combines both visual (depth) and proprioceptive inputs (i.e. IMU and joint angles) to solve visual locomotion tasks. The authors demonstrated that their approach can solve challenging visual navigation tasks and locomotions task on uneven terrains. The proposed method out perform proprioceptive only, visual only, and HRL baselines. The sim trained policy has been demonstrated on the real A1 hardware.
main_review: The main strengths of the paper:

(1) Proposed a novel transformer based architecture that can train visual-locomotion policies end-to-end, and demonstrated good navigation/obstacle avoidance/uneven terrain walking results in the simulation.

(2)  Zero-shot real world transfer to a A1 robot and demonstrates walking + navigation behavior in various environments.

The main weakness of the paper:

Not enough baselines to compare with. As the authors cited, there are many approaches to tackle visual locomotion + navigation problem besides end to end training. For example in the hierarchical approach one can combine: learned/optimization based navigation + pre-trained or hand tune walking (i.e. MPC) motions.  So in total even the hierarchical approach can have four different combinations to compare with. Yet I saw non of them here. I would say the authors should include at least one or two such baselines to compare with, and document the performances and cons and pros. 



summary_of_the_review: The authors proposed to use transformer architecture to solve visual locomotion + navigation tasks. The proposed approach is compared with a few end-to-end trained baselines including HRL and has demonstrated advantages. The authors also deployed the trained policy successfully to the real robot.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a novel architecture for quadrupedal locomotion that fuses proprioceptive and visual information with a transformer-based model to enable an agent to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. The method is extensively evaluated in simulation and on a sim to real transfer tasks. The method is shown to both achieve higher reward  but also better capacity to generalise in the context of sim to real. Overall, the paper is well written and the provided evaluation is conducted fairly and well.
main_review: ## Strengths and Weaknesses

### Things I liked about this paper
- **a powerful framework**: fusing visual and proprioceptive data for quadrupedal locomotion using transformer architectures is an interesting and also valuable approach that works well and sets an excellent opportunity for future work.
- **a well written paper**: overall the paper is well written and all sections are broadly very clearly described.
- **useful insight**: I like the provided key insight that proprioceptive states offer contact measurements for immediate reaction while visual sensory observations can help with longer-term planning.

### Things that can be improved
- **number of seeds is not great**: The adopted model-free approach is known to have very unstable learning process of the dynamics function which ideally requires 10 or more seeds to provide a solid results. Using only 5 seeds is not great. More details below.
- **prose is not perfect**: There are some minor details and clarifications that may help further improve clarity.

Using 5 random seeds for a model free approach is rather small as a number. Ideally, the evaluation should be done on 10 or more seeds. In fact, I suspect that some of the results such as the moving obstacles from Table 3, would change if the approach was evaluated on more runs. Nevertheless, the provided training curves seem to have fairly small variance as illustrated in Figure 4 which makes me more inclined to agree that 5 seeds are sufficient to report on accurate results. In addition, I would expect that the variation in the learnt dynamics to primarily affect the performance of the learnt agent on the physical quadruped system.  However, this does not seem to be the case in the reported results, which is great as long as all 5 seeds were used to extract those results. It is great that the paper considers 15 runs per seed but I wonder if the results were acquired through cherry picking best n seeds. This is a detail that is not currently mentioned in the paper but would certainly improve clarity if it did.

There are a few additional minor comments. Currently, the distance measurement reported in meters is mentioned only in the text and not in the tables. Stating this there too would make it much clearer. Similarly, what exactly does the collision happened represent. Are these total number of collisions over 1000 steps? 'the number of time steps where collision happens between the robot and obstacles
over the course of an episode' states the explanation seems a bit overly complicated. Why not just 'the number of collisions with obstacles per 1k step long episode' or something along those lines?

There is a typo in the contribution 'we the propose' should be 'we propose'. Another typo is '... whereas it for our method either plateaus' should be 'whereas for our method it either plateaus'
summary_of_the_review: Overall this paper is written well and has a sound idea that is supported well by an extensive evaluation. There are some minor details that may further improve the quality of the paper but I see this as a strong submission which I can recommend for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach to legged locomotion which leverages a Transformer-based model and is trained via end-to-end reinforcement learning. It provides extensive experimental evaluation of the approach in terms of performance and safety metrics, both in simulation and using real-world experiments. The code is expected to be open-sourced.
main_review: This paper is very clear in its exposition, providing a detailed diagram of the method, clearly labeled inputs and outputs, and relevant implementation details. The experiments appear sound, with a number of baselines and ablations provided. I want to emphasize the value of real-world experiments in this space, as the 'sim-to-real' gap can be significant and invalidate otherwise good looking results. Scientifically, the paper proposes an architecture that is novel and can serve as a broader proof point that vision-based locomotion can be competitive and robust when trained with a sufficiently expressive model.

Strengths:
+ The paper tackles an important research problem: vision-guided locomotion and has made significant progress along this direction.
+ Novel network architectures (such as Transformers) are under-explored in the legged robots community. This paper demonstrates that incorporating such architecture indeed makes a difference in performance.
+ The proposed method is validated on a real robot. The evaluations are comprehensive and the conclusions are convincing.

Weakness:
- The technical novelty is lean. Neither of the key components of the paper are novel: RL for locomotion and Transformers. Although this can be considered as a weakness, it is not a deal breaker given that the combination of these two and the application to legged robots are novel and potentially influential.
- The results are mostly obstacle avoidance on the flat ground, where legs are not essential. Most of the experiments can be done with a wheeled robot. To show the true value of this paper, more challenging terrain needs to be considered and tested on, such as stairs, stepstones, tall grasses, rocks, etc. In these terrains, both vision and legs are critical. From the accompanying video, it is a bit disappointing that the robot only learns steering for obstacle avoidance, but does not learn foot clearance or foothold location on different types of terrains. The robot always drags the foot, even on pebbles (5:25s in the video) where higher foot clearance is clearly a preferred choice. Since this paper trains end-to-end, I would expect that these behaviors would emerge automatically if trained in relevant environments. Showing these behaviors (foot clearance, foothold location, change of gait pattern) in addition to obstacle avoidance, would significantly strengthen the paper.

Additional questions:
1) In addition to domain randomization, does the paper apply other techniques for sim-to-real transfer? For example, I would imagine that there will be a large sim-to-real gap in vision. The depth images from Intel realsense can be noisy and with holes, especially in outdoor environments. Do these sim-to-real gaps in vision cause any problems when deploying the policy on the robot?
2) How much tuning is needed to learn natural (deployable) locomotion gaits? In the video, the learned locomotion gait is quite reasonable and deployable on the robot. The paper explicitly mentioned that it did not use trajectory generators. Does it purely rely on reward shaping? If so, how much tuning is needed? And what are the most important terms in the reward function that encourage the emergence of natural gaits?

summary_of_the_review: Good paper on relevance and experimental evidence, on a topic that is very much of interest to the robot learning community today.
Novelty limited due to combination of known techniques.
EDIT: bumped confidence to a 5 based on comments and rebuttal. This is a solid contribution.
review_title: null

Meta-review: The paper addresses vision-based and proprioception-based policies for learning quadrupedal locomotion, using simulation and real-robot experiments with the A1 robot dog. The reviewers agree on the significance of the algorithmic, simulation, and real-world results. Given that there are also real-robot evaluations, and an interesting sim-to-real transfer, the paper appears to be an important acceptance to ICLR.
Decision: Accept (Spotlight)