Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the imitation learning (IL) problem, including learning from observation (LfO) and learning from demonstration (LfD), is studied. The paper proposes a new solution to IL which include three key aspects: (1) a new ranking loss is proposed for IL, which is capable to learn whenever the rankings of the trajectories are given or not; (2) a mixup-like data augmentation strategy is introduced to smooth the loss landscape; (3) the IL process is formulated into Stackelberg game, allowing the studied of the update frequencies of reward and policy functions. The proposed method is tested under benchmark continuous control tasks, showing performance improvement over existing IL approaches.


strengths_and_weaknesses: Strengths:

1. The paper proposes several interesting ideas for IL. In my view, the most significant ones are two-fold. One is the importance of smoothing the loss landscape, which can be achieved by the proposed loss and the mixup-like data augmentation. Another is the Stackelberg game formulation. The experiment of task changes in Section 5.3 is very interesting to me.

2. The overall experimental performance is impressive. 

Weaknesses:
1. The proposed approach is somehow complicated. There are several components included, making it hard to identify their contributions to the performance improvement. Even though a number of ablation study results are reported, they only show that none of the components is useless. Their precise contributions, as well as underlying mechanisms, remain unclear.

2. It still remains unclear to me why the proposed ranking loss could outperform other classical ranking losses. As discussed in the paper, the loss is somehow like the one proposed in [1]. While in [1], since the algorithm is Q-learning, the choice of the reward function seems to have a less impact on the optimization objective. 

3. I think conducting deeper studies following the direction in Section 5.3 would make the paper more interesting. The results in Section 5.3 seem to indicate that the PAL training is robust to changes in reward functions, while the RAL robust to changes in dynamics. I think these results show that one of the major advantages of formulating IL into Stackelberg games is the convenience of dealing with task changes. 

Overall, I think the paper is more on proposing several useful techniques for IL without exploring the underlying mechanism deeply, which can be of good practical usefulness. 

[1] SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards, ICLR'20.
questions: 1. Can you provide some explanations on why the proposed ranking loss is better than classical ranking losses, under the IL setting?
limitations: The limitations and negative societal impact are properly discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers the standard imitation learning problem by treating imitation as a two-player ranking-based game between a policy and a reward. A novel ranking loss is proposed which has the ability to learn from both expert demonstrations and preferences while keeping the learned reward bounded. The authors introduce two optimization methods based on Stackelberg games with different leaders. Experiments show that the two proposed algorithms work well in LfO settings and outperforms the baselines. In addition, by incorporating additional offline annotated rankings, the proposed method can work in more complex environments where other baselines fail. Finally, the pros and cons of the proposed two algorithms are also discussed.
strengths_and_weaknesses: **Strengths**

The idea of treating imitation as a two-player ranking-based game is novel, which encompasses both a kind of IRL methods and ranking-based methods. The proposed ranking loss learns a smoother reward function (Figure 2) compare with the IRL methods. The experimental results are strong, showing the efficacy of the proposed methods and the benefit of the additional annotated rankings. The comparison of the two proposed method is interesting. The writing of the paper is generally good. 

**Weaknesses**

- The ranking loss Eq (3) is novel, but it may lack some intuition. The loss of GAIL is based on occupancy measure matching, and the loss of AIRL makes it possible to learn the ground-truth reward under some assumptions, and the loss of TREX is based on Luce-Shepard rule. However, the proposed loss lack some theoretical background and I would appreciate if the authors can provide additional intuitions behind this loss. 

- Line 215: Here the behavior generating trajectories in the ranking dataset is assumed to be accessible. How do you generate the ranking dataset? Also, if you use auto magically generated rankings, does it mean that the demonstrations are not fully optimal? Will this cause some problem when combining the automatically generated rankings with the vanilla rankings, e.g., the learned policy may behave better than the low-ranked trajectories in the demonstrations, leading to some noisy ranking data?

- In Section 5.2, the results in Door-v0 show that RANK-PAL (pref) is the only method that can solve the problem. Can you provide some insight why the additional annotated rankings are better than the automatically generated rankings, and why the additional annotated rankings can reduce the exploration requirements of LfO (line 315) that much?

- In Section 5.2, the authors only show the performance of RANK-PAL. How does RANK-RAL work in these scenarios?

- For the writing:
  - Line 21: [Leaning from expert data (imitation learning) alone]: "data" here is a bit unclear here since actions are also data.
  - Line 25: [assumes no environment interaction]: The definition of "environment interaction" is unclear. I think you mean the environment interaction while learning the reward function, since DREX needs to collect interaction data while collecting noisy trajectories. In addition, are you suggesting no environment interaction is a bad thing here? Why is this a bad thing?
  - Line 57: [suboptimal rankings]: This is vague. What do you mean by saying rankings are "suboptimal"?
  - Line 135: a right parenthesis is missed.
  - Line 182: What is $\epsilon_r$?
questions: The questions are mainly discussed in the Weaknesses. Here are the selected questions that I think are most important:

- The intuition behind the proposed ranking loss (Eq 3).

- How to generate the ranking dataset in the setting of reward loss w/ automatically generated rankings? Will this cause some problem when combining the automatically generated rankings with the vanilla rankings?

- Can you provide some insights on the reason why the additional annotated is so useful?
limitations: The limitations and potential negative societal impact mentioned in the paper are adequately addressed. One suggestion: the authors can consider how the framework will work in the setting of learning from suboptimal demonstrations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a ranking-game setting for imitation learning by using a ranking loss for the reward agent, and incorporating automatically generated rankings data augmentation as well as offline annotated rankings. The proposed method show improved performance and data efficiencies compare to state-of-the-art baselines on several benchmark tasks.
strengths_and_weaknesses: Strengths:
1. The relationship between the proposed setting with the existing IRL frameworks is well described.
1. The author gives a performance bound for the proposed ranking loss at equilibrium.
2. The experiments for the proposed method without offline preference show clear sample efficiency improvement compared to other methods.
3. The experiments for using offline preference give a significant performance improvement. While the improvement is expected but is still impressive.
4. The ablation study is thorough.

Weaknesses:
1. I am not quite convinced about the automatically generated rankings. Why would a linear combination of trajectories give a linear combined ranking?
2. The other limitations are described at the end of the paper, which I feel is acceptable.

questions: May the author give some more explanation on the automatically generated rankings. Why would a linear combination of trajectories give a linear combined ranking?
limitations: The major limitations are described at the end of the paper, which I feel is acceptable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to model imitation learning as a two-player general sum game, where one player, the "reward player", aims to rank a set of behaviors correctly, where the "policy player" aims to maximize the reward the "reward player" uses for ranking. This formulation unifies multiple prior methods for LfD and LfO. Then, the authors propose a novel way to solve this two player game as a Stackelberg game which leads to a novel imitation learning algorithm that outperforms prior methods in a series of simulated MuJoCo experiments. The proposed algorithm learns from demonstrations and preferences simultaneously which allows it to combine the benefit of both.
strengths_and_weaknesses: **Strengths**
- The two-player game framework is a conceptually elegant way of describing and unifying existing methods for IL. It is neat how the choice of different ranking loss functions yields different algorithms.
- The proposed algorithm is quite natural, and it is analyzed theoretically and empirically.
- The empirical evaluation is particularly thorough. The authors compare to various natural baselines on different tasks. They provide ablations of various components of their algorithm, providing the reader with a clear understanding of the performance of their algorithm in MuJoCo environments.

**Weaknesses**
- The paper does not provide a good intuition for the proposed ranking loss. After reading Sec. 4.2 I understood the choice that was made, but did not have a good understanding of why this is the right choice among many possible ranking losses.
- Parts of the paper could be written more clearly. For example, in Sec. 4.2 it would be natural to split it up in subsections discussing the ranking loss, the theoretical properties, and then the ways to generate the ranking dataset.
- I found it hard to understand the theoretical results following Theorem 4.1 without reading the Appendix.
- If I understand correctly, the method assumes all preferences that are provided to be noise-free, which limits the applicability of the method significantly.
- The experimental evaluation is limited to MuJoCo locomotion tasks, which can sometimes be too simple to draw general conclusions.

questions: What is the intuitive motivation for choosing the proposed ranking loss?

Do you consider any of the theoretical results apart from Thm. 4.1 a key contribution of your paper?

Does you assume all comparisons in the dataset are noise-free?

Could the proposed approach be extended to active learning by modifying what the reward agent does (i.e., it also chooses queries to make to a human)?


**Minor suggestions:**

- Capitalization:
   - "bradley-terry" -> "Bradley-Terry" (Table 1)
   - "jacobian" -> "Jacobian" (line 253)
   - In the references: "stackelberg" -> "Stackelberg", "Iq-learn" -> "IQ-learn", "soft-q" -> "soft-Q", etc.

- Also, make sure named paragraphs are consistently capitalized. For example "Ablation of method components" (line 350), but "Limitations and Negative Societal Impacts" (line 363)


limitations: If the paper indeed assumes noise-free preferences, this should be made clearer when setting up the problem. It would also be good to include as a dimension in Table 1, to provide a fair comparison.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors create a new method for learning a policy through IRL. The method contains two different agents: (i) a policy agent; and (ii) a reward agent. The authors perform a set of experiments with the OpenAI Gym + MuJoCo environments and show that they achieve better results than prior work.
strengths_and_weaknesses: - Strengths:
    - The authors create a consistent work with a good explanation of why they created the method, and how the method works (theoretically and practically).
    - They perform extensive ablations on their method and how each part of the method reacts when deviating from the proposed method in the main material.
    - The work is well presented, with good writing and a couple of minor misspellings, nothing major that would damage the reading comprehension and experience.

- Weaknesses
    - I think that the major weakness of this work is the lack of experimentation with sub-optimal expert samples. With each year, we have more IL methods being developed and one of the main difficulties for a lot of them is to be resilient to sub-optimal samples. I think this should have been addressed in this work. Yes, the method is very efficient but what is the trade-off that controls this efficiency.
    - Even though RANK achieves good results with only one trajectory, I think there should be a study on how well it performs with more samples. We cannot forget that RANK has access to the environment and online samples have a higher cost than offline samples to collect and learn. I think this work would benefit from drawing this distinction and showing how much it gains when using different numbers of trajectories. Even more so when we think that there are other methods that were not used in these baselines (such as MobILE [1] and ILPO [2]) that already use a lower number of trajectories (10) while not having this iterative nature of RANK. Moreover, using a single trajectory was a limitation that RANK imposed to itself.
    - Table 2 only shows the Performance metric, and does not display the expert rewards, which hurts the readability of the results.
    - Another major concern that I have is the efficiency study. The authors say that their method achieves higher reward thresholds in 2M timesteps with only one trajectory (that can vary in sample size). However, this is not entirely true. If we consider that offline samples are more efficient to learn, we could argue that BCO, and all future work based on it, could be more efficient by using 10 trajectories and only needing a fraction of the timesteps in this experiment, for example.
    - Finally, regarding the experiments, 5 trajectories seem a lower number of trajectories to have an accurate statistical result. OPOLO used 10 times more samples to draw its average and standard deviation, while GaIFO used 100 trajectories for each experiment.
    - The authors’ overuse of math notations hurts the readability of their work quite extensively. Most of the time the iterator changed without proper explanation and it takes the reader quite some time to understand what the authors are iterating over.  I think that rewriting the equations and standardizing the notation would benefit this work.
    - If the policy agents interact with the environment to reduce the reward function, is it really IL? Why training an imitation learning agent if, in the end, we will have an agent trained by the reward function of the environment (which theoretically is the most optimal thing we could have in an MDP problem)?
    
    [1] Kidambi, Rahul, Jonathan Chang, and Wen Sun. "MobILE: Model-Based Imitation Learning From Observation Alone." *Advances in Neural Information Processing Systems*
     34 (2021): 28598-28611.
    
    [2] Edwards, Ashley, et al. "Imitating latent policies from observation." *International conference on machine learning*. PMLR, 2019.
questions: - How much time does it take to run each experiment in each of these methods? I think that a reason that the authors used so few trajectories in their experiments could be a hint that even though RANK is sample efficient, it does not seem to be time efficient. Clarification on time efficiency would be very important here.
limitations: - I think the authors addressed some of the concerns I had during my review in their limitation section. The ones they have not addressed are pointed out in the weakness section of this review.
review_title: null

Meta-review: metareview: I went through the paper, reviews and responses. This is a borderline paper. The negative to neutral reviews are more detailed and convincing.
confidence: Certain
Decision: Reject