Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper addresses the problem of learning multimodal reward functions from interaction with a human teacher. The main contribution is the use of rankings, rather than just pairwise comparisons, to pose more informative queries to a teacher. Additionally, the paper presents an information-gain method for selecting optimal queries to pose to the teacher.
strengths_and_weaknesses: Strengths:
* Writing is clear and concise
* Ability to model conflicting feedback from teachers
* Use of human-provided data in the evaluation
* Information gain-based approach to query selection results in improved performance over baseline

Weaknesses:
* Multimodal approach is not well justified. More discussion is needed regarding the cases where a multimodal approach is preferable over learning multiple, unimodal models. 
* Evaluation focuses mainly on testing the querying approach, rather than demonstrating the benefit of a multimodal approach.
summary_of_recommendation: This paper addresses the problem of learning multimodal reward functions from interaction with a human teacher. The main contribution is the use of rankings, rather than just pairwise comparisons, to pose more informative queries to a teacher. Additionally, the paper presents an information-gain method for selecting optimal queries to pose to the teacher.

The experiments described in this paper make use of data provide by real users in a user study, and the results indicate that the information gain method produces better results in a simulated lunar lander task and in a sorting task on a Fetch robot. 

Overall, the goal of learning a multimodal model is an interesting technical challenge. However, the motivation for it is not yet clear. What is the benefit of learning a single model that is a mixture of multiple teachers' conflicting feedback? Rather than learning multiple models that reflect these preferences separately? The evaluation also focuses on the benefit of using an information gain approach to selecting queries, but does not provide a compelling justification for why a multimodal approach is ideal in the first place.

The implications of using a multimodal model should also be addressed. What makes multiple experts' reward weights compatible enough that they can be represented using a single model? And at what point *should* they be represented using multiple models? How would this approach scale to M > 2 modes?

Overall, this paper addresses a very interesting technical problem by modeling teachers' conflicting preferences in a single model. While the evaluation provides support for the proposed querying method, it does not yet fully justify why/when a multimodal approach should be used.
issues: * What is the benefit of learning a single model that is a mixture of multiple teachers' conflicting feedback? Rather than learning multiple models that reflect these preferences separately? 
* What makes multiple experts' reward weights compatible enough that they can be represented using a single model? And at what point *should* they be represented using multiple models? 
* How would this approach scale to M > 2 modes?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of learning from multi-model ranking feedback from humans. Given that different people may have different preferences over a learning agent’s behavior, it is important to explicitly model this effect to capture the underlying data distribution. This paper also proposes an active learning algorithm that proposes ranking queries to maximize information gain.
strengths_and_weaknesses: To effectively and efficiently learn from human feedback is an important problem in robot learning. Learning from multiple humans adds additional challenges. This work proposes to explicitly learn multi-modal ranking models when we know there are different modes in the data. Experiments in simulation and with real human users demonstrate the effectiveness of the proposed active learning method. 

One major weakness of the proposed method, as the authors also acknowledge, is that it requires knowing the number of different modes, which is unlikely to be true for many real applications. It is desirable to see results from simulation experiments to show how robust or sensitive the proposed algorithm is to a noisy estimation of the number of modes. 

summary_of_recommendation: The problem setting is important, the proposed algorithm is novel and the experiments are convincing. However, the limitations of the algorithm is not well-tested. I recommend accepting the paper but will not argue against rejecting it. 
issues: See weaknesses. More ablation experiments are needed.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Paper proposes to learn multimodal reward functions where the datasets comprises trajectories generated with different intentions or goals. The proposed active learning method leverages rankings that are provided by the experts to better distinguish between the different modes. Theoretical justifications for the approach is given and the method worked well in the LunarLander and Fetch Robot experiments with faster convergence and lower MSE scores. 
strengths_and_weaknesses: The paper is well-organized and clearly conveys the method and results. Using rankings to improve multiple-intention learning is new to my knowledge and is relevant to the robotics community, especially those working in LbD and/or IRL. The problem setting is realistic since datasets may not have goals/intentions explicitly labelled. The experiments and user study appear well-executed and show the approach is effective compared to volume removal and random query selection methods. 

Although I'm positive about the paper, it does miss out on related work on multiple intention IRL [Babes-Vroman, 2011; Ramponi et al, 2020] where a similar mixture model was proposed. The key difference is that rankings are used in the learning process but these prior works should be discussed and compared against.  

In Eqn (3), the term $Pr (x^{(t)} | \Theta, Q^{(t)})$ should be expanded and clarified. Since the queries depend on the previous rounds due to the inference, it's unclear whether (3) remains valid under the active inference setting? Would the estimated posterior be biased? 

### References
- Monica Babes-Vroman, Vukosi Marivate, Kaushik Subramanian, and Michael L. Littman. Apprenticeship Learning About Multiple Intentions. In Proceedings of the 28th International Conference on Machine Learning, ICML ’11 
- Giorgia Ramponi, Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, and Marcello Restelli. Truly Batch Model-Free Inverse Reinforcement Learning about Multiple Intentions. In International Conference on Artiﬁcial Intelligence and Statistics, 2020

summary_of_recommendation: Overall, the paper makes good technical contributions to the relevant problem of learning from a trajectory dataset with multiple intentions. I vote for acceptance. 
issues: Please see above regarding related work on MI-IRL and potential bias.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a method for multi-modal reward learning by formulating the problem as learning the parameters of a continuous mixture distribution based on ranked query responses by humans.

strengths_and_weaknesses: **Paper Strengths**
Writing: This paper is very well written, with good flow throughout the introduction and a clear explanation of the method. Furthermore, the experiments section is formatted well; the jumps from one subsection to the next made sense and the order in which results were presented makes the experiments feel intuitive.

Technical Contribution: The method is novel AFAIK, and it is a significant technical contribution with sensible design choices along the way. 

Experiments: The experiments are comprehensive as they measure MSE, log likelihood, and perform real-world human studies on both a simulated environment and on a real robot. Furthermore, they justify the necessity of the design choices made for the method.

Results: The authors’ method shows decent to solid improvements over baselines across all of the experiments, thus demonstrating the superiority of the method.

**Paper Weaknesses**
Experiments: Although the simulated human feedback experiments demonstrate the need for multi-modal instead of unimodal reward models, results on user studies would still be elucidating. 

summary_of_recommendation: The paper is a solid accept; experiments are robust and the method is a solid contribution. 

issues: See weaknesses section.
review_title: null

Meta-review: **Update after rebuttal**
I thank the authors for their thorough response, revising the paper based on feedback and clarifying some misunderstandings. My concerns have been addressed. I recommend accept

**initial meta-review**
**summary**
This work addresses the problem of learning reward functions from demonstrations that may have been generated from varying reward functions. It proposes an approach to leverage rankings over demonstrations to learn multi-modal reward functions. Furthermore, an active querying framework is introduced that aims to reduce the amount of demonstrations required.

**Strengths**

Learning reward functions from multiple different demonstrations is an important problem in the robotics community

A novel methodology of learning from rankings of demonstrations is presented, together with active query-ing approach that addresses scalability concerns

The approach is evaluated in simulated settings, as well as on demonstrations collected on real fetch robot.

**Weaknesses**

I agree with the reviewer asking for more motivation on why it’s important to learn multi-modal reward functions.  Additionally, I would expect additional baselines. For instance, a simple baseline could have been to just pick the top ranked demonstration and learn a uni-modal reward from that. This would have told us whether there is any benefit from learning multi-modal reward functions. Your current comparison (learning unimodal rewards from conflicting demonstrations vs learning multi-modal reward from conflicting demos) does not really answer the question of “are multi-modal rewards needed”. Without this additional baseline the impact of the proposed approach cannot really be measured.

The related work section should more clearly create connections between what has existed before, and how this work is extended. For instance currently the manuscript does not very clearly state which type of approach towards learning reward function weights is used, and what work your method is based on. Only the linear reward function assumption is stated, however various IRL styles use this assumption. After some searching, it seems like for instance this could be a good reference to clearly state as setting some foundations of the work in this manuscript: https://arxiv.org/pdf/2006.14091.pdf) . This lack of clearly stating what overall IRL approach is used makes it a) harder to understand the overall approach quickly b) harder to appreciate the progress made. Finally some other relevant related work should be included as pointed out by one of the reviewers.

The mixture of reward functions probably makes the issue of ambiguous reward functions worse? If yes it would be great to see some discussion on this topic, as it is typically a major concern of this type of IRL approach.

Decision: Accept (Oral)