Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an imitation learning algorithm, named WASABI, that enables quadruped robots to learn from rough and partial expert demonstration using generative adversarial learning. Instead of using task-specific reward functions, as with many robotics learning approaches, the proposed approach can infer rewards from demonstration using GAN. In particular, the authors incorporate task-agnostic regularization terms and adversarial rewards to enable the robot to learn stable and agile locomotion behaviors. Moreover, this approach can learn only from partial and physically incompatible demonstrations (e.g., a human operator holds and moves the robot's trunk in a specific fashion). The authors also showed simulation and experimental results of a quadruped robot learning four challenging tasks. The papers show that WASABI outperforms the least-square GAN (LSGAN) in most of the tasks (3 out of 4) in terms of motion similarity (measured by DTW distance). 
strengths_and_weaknesses: **Strength:**

- Able to learn from limited demonstration without defining task-specific reward functions, which often need to be designed and tuned by hand.
- Replacing LSGAN loss with Wasserstein loss improves the efficiency in discriminating the reference and the generated motions.
- Able to prevent mode collapse using gradient penalty term in the loss function.
- Demonstrating the performance of the proposed approach in hardware experiments

**Weakness:**

- The final reward function also includes a regularization reward, which seems to be specific to the robot. How much does this term contribute to the total reward?
- The observation space does not include the position/orientation of the robot base (it only includes linear and angular velocities). Will the demonstration speed of the human operator would affect the final learned locomotion?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, this work presented in the paper is well developed. In robot learning (especially when it involves hardware), the reward design or shaping is often heuristic and requires manual tuning for different tasks. WASABI, which uses generative adversarial learning, provides a principled means to design rewards for robot learning. The demonstration of hardware shows the promising potential of the proposed approach. 
issues: Please explain the reasoning behind the choice of observation speed.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed an adversarial imitation learning method to learn agile motor skills of a quadrupedal robot from partial demonstrations.
The main contributions of the paper are following two points:
1. The authors introduced the Wasserstein loss to the adversarial reward learning.
2. The authors showed objective and reward functions to acquire the agile motor behaviors from the rough and partial demonstrations.

strengths_and_weaknesses: Strength:
* The authors successfully demonstrated that the proposed method, Wasserstein Adversarial Behavior Imitation (WASABI), could learn the agile behaviors from the handcrafted partial state information.
* The authors successfully showed that the agile behaviors of not only the simulated but also real quadrupedal robot could be generated with WASABI by transferring the learned policies.

Weakness:
* Although the experimental results are fascinating, the paper seems difficult to read. Especially, readers may need to read the original paper of adversarial motion priors (AMP) [Peng et. al., TOG, 2021] to understand the learning process of the adversarial reward learning and policy training. To improve this, for example, I recommend that the authors make a new section to explain the original AMP method as a preliminary in the paper.
* The ablation study of the proposed reward function Eq. (6) is missing.

Minor comments:
* Since the backflip behavior of the real quadrupedal robot is impressive, it would be better if the authors show snapshots of the backflip on the paper like Figure 1 (left). Figure 1 (right) is hard to see.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The contribution of this paper is sufficient: strength in both originality of the approach and thoroughness of results. The experiments are performed in not only simulation but also real environments. However, the clarity of the presentation is insufficient. The paper should be self-contained.
issues: * I suggest the authors to improve the paper structure so that, ideally, the paper becomes understandable without reading the original AMP paper.
* I suggest the authors to perform the ablation study for the proposed reward function Eq. (6) to show that both $r^{T}$ and $r^{R}$ are important.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the authors proposed a GAN-based framework to train agile locomotion policies using partial demonstrations. Similar to AMP (Adversarial Motion Prior) work, the authors use Wasserstein-GAN to discriminate between the expert demonstrations and the learned policies, to generate the reward signal for reinforcement learning. Different from previous works, the demonstrations in this work are not from experts (dogs for examples). Instead, only rough base movement demonstrations are provided for the learner policy to figure out dynamically feasible motions that can realize the base trajectories. The authors has demonstrated successful deployment of the learned policies on the hardware with zero-shot sim-to-real transfer. 

strengths_and_weaknesses: The strengths of the paper:

1, The authors show that it is possible to only provide base trajectories as demonstrations to train imitation policies for quadruped robots. This opens many new potentials for quadruped robots, since expert demonstrations are often hard to obtain. 

2, There is a good ablation study and analysis on the different GAN losses (least square vs Wasserstein)

3, The learned policy can transfer to the real hardware on various agile tasks including back flipping. 

The weaknesses of the paper:

1, the delta between AMP and this work is not quite large, especially since AMP has also been tested on hardware recently. The main differences seem to be (1) whether or not to include the joint angles as part of the discriminator observations for reward generation and (2) use of WGAN vs LSGAN. 

2, Not enough baselines to compare with. I think it will be great to compare AMP with the proposed method on the selected tasks. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: See the strengths and weakness section. 
issues: Here are some additional comments, in no particular order:

1, There are also other AMP extension works on hardware to refer to, such as this one: “Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions”

2, Running mean and variance for the discriminator: Are they learnable parameters of a gaussian output?

3, Tabel 2 bottom line, why only two columns?

4, Action space for the policy is better to be included in  the main text. Also, please list the joint PD gains in the supplementary text.

5, Have the authors tried to use different observation spaces for the control policy than the discriminator? For example, base linear velocity is often not needed for the control policy to function properly. And in this way the policy may transfer to the real hardware without motion capture. 

6, Why is there no SOLOSTANDUP task on the real robot? Does it transfer or not?

7, Figure 4A and 4B are not quite clear, maybe due to the color coding.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Authors present a novel generative adversarial method named WASABI for inferring reward functions from partial and potentially physically incompatible demonstrations. WASABI simultaneously trains Wasserstein GAN to distinguish between real and generated observations, and policy which uses output of the discriminator as a reward. This approach allows to use partial demonstrations which don't contain full information about the state. Authors experimentally compare performance of WGAN and LSGAN for reward inference and experimentally validate the proposed method in simulation and on a quadruped robot training complex skills such as a backflip. 

strengths_and_weaknesses: The paper proposes an extension of Generative adversarial imitation learning method (GAIL) method. Wasserstein GAN formulation allows to use the discriminator output as a reward function directly only with simple normalization. This method allows to train a policy for complex behaviour from partially observed expert demonstrations. The paper is  well written and enjoyable to read. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: It would be interesting to see ablation on how much expert data needed to train a discriminator 
issues: It seems important to mention how much expert data was used to train discriminator for each of the tasks
review_title: null

Decision: Accept (Oral)