Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper formulates a method for active reward learning from user feedback, where the agent’s goal is to learn the task reward function, while also having control over what type of feedback to query from the user. Four types of feedback are considered: demonstrations, preferences between two trajectories, trajectory corrections, and binary good/bad labels. The paper formulates the choice for feedback queries as an optimization to maximize information gain in the distribution over reward, which is estimated via a discrete set of weight hypotheses.
Results are presented using a simulated oracle with access to a large set of demonstrations and their rewards to provide feedback using the available modalities. The paper shows very interesting results, where INQUIRE tends to prefer demonstrations initially, and later switches to preferences and other types of feedback, thereby outperforming a learning from demonstration while using less expensive type of supervision.

strengths_and_weaknesses: Strengths:
- The paper looks at an interesting, relevant and under-studied problem.
- The method is very clean and well motivated.
- Paper is well written with clear problem formulation, method description, notation etc.
- Extensive experiments show effectiveness of the method on an appropriate set of simulated tasks.

Weaknesses:
- It seems that the formulation assumes a reasonably low-dimensional reward, query, and answer space. I am not sure how well this method would scale to more complex high-dimensional tasks, such as manipulation.
- Did not test on a real-robot problem.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a well-formulated method to a relevant and well-motivated problem. However, the experimental domains were somewhat simple which casts doubt on the applicability to more complex robotics tasks.
issues: Questions:
- It seems a little unintuitive that in the unweighted case the agent often selects preference queries over demonstrations and corrections. Why are they considered more informative?
- If the feedback provider was able to observe the behavior of the policy in practice over time, like a human bystander would, could they provide more informative demonstrations that target the current failure modes?
- Would it make sense to compare to a baseline that emulates an online imitation learning algorithm such as DAgger? For example, by rolling in a policy that optimizes the current most likely weights, and then requesting a correction from some random starting state. It seems like one of the obvious approaches to building an online learning system in cases where trajectory-level supervision is possible.

Suggestion:
- Add some visualizations that better explain the specific tasks and give an intuitive understanding of complexity?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper tackles interactive querying across multiple human interaction modalities for reward learning. The INQUIRE approach selects queries to maximize a cost-weighted information gain about reward feature weights across demonstration, preference, correction, and binary feedback modalities. The paper compares their approach with baselines that use a single human interaction modality or a fixed pattern of interactions in simulated environments with an optimal oracle as a human proxy. The INQUIRE method is able to often outperform baseline methods, especially when the initial state is held static.

strengths_and_weaknesses: Strengths:
- The writing and figures are clear and concise.
- The paper presents a clear description with matrix formulas of how the information gain across querying modalities is efficiently computed.
- The paper presents clear evaluation metrics and visualizations (# of queries and cost of queries against task performance and feature learning error; querying modality by step).
- The approach appears competitive with baselines in simulated environments.

Weakness:
- A weakness is the lack of real robot experiments and a lack of real human user studies—real humans can be significantly different from a Boltzmann-rational model and exhibit multimodal reward preferences. Further, the INQUIRE method performs best relative to baselines in - the more unrealistic static initial state settings.
- It is unclear if INQUIRE is significantly better than baseline approaches.
- It is assumed that “good” trajectories can be randomly sampled from the environment.
- The corrections and binary human response models may be unrealistic (see below).
- See other issues below.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper tackles the interesting problem of deciding not only which queries to make within a human interaction modality but also which modality to query. The paper solves this problem by elegantly extending past information-gain techniques to make cost-weighted comparisons between modalities. However, without analysis of statistical significance of simulation AUC results, real robot experiments, or human user studies it is difficult to assess how effective the INQUIRE approach is. Further, inconsistencies in the human response model for corrections and binary responses call into question the information gain computation used for these interaction modalities. The work could be significantly improved by (1) clarifying the response model for computing correction and binary response information gain, (2) conducting statistical analysis of the benefits of the approach, and (3) running experiments on real robots and/or with real human teachers.

issues: - Statistical significance of the AUC metric against baselines should be clearly analyzed.

- It is unclear how corrections differ from demonstrations in terms of information—for both settings, the choice space is the full trajectory space regardless of query. So for the queries to have different value, the human response model would need to be different. Meanwhile, looking at Eqs. 7 and 55, the distribution p(c|w) does not seem to sum to 1, making it hard to understand what the intended response model is. Further, the response model probability p(c|w) for corrections in Eq. 7 doesn’t appear to depend on the similarity between the corrected trajectory and the queried trajectory, which is inconsistent with how the oracle selects trajectories (by maximizing a ratio between the value of a trajectory and it’s dissimilarity with the query). This response model seems very inconsistent with how trajectories would be selected by a human (a corrected trajectory one state away from the query is much more likely to be selected than a completely unrelated trajectory as a correction). The authors should clarify the human response model for computing information gain for corrections, and how this model causes it to differ for the information gain.

- The binary response model seems to assign a very low probability to c=1 when there are multiple high-value trajectories in T. This is inconsistent with the oracle model, which responds with c=1 when a trajectory is in the top 75%, and could artificially deflate information gains.

- Why is gradient ascent used to update the posterior Omega? Gradient ascent could bias the posterior to place undue weight on its modes. Past works approximating information gain / volume removal with posterior sampling have used Markov chain methods to approximate posterior sampling [1].

- The authors should clarify if demonstrations and corrections need to be drawn from the randomly sampled pool T. A user making sufficiently fine corrections / demonstrations would usually want to construct a trajectory that doesn’t randomly lie within the sampled pool.

- Why is the simulated oracle deterministic rather than Boltzmann-rational like the response model for computing information gain?


[1] https://escholarship.org/content/qt88k894w7/qt88k894w7.pdf
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a method for optimizing the information gained from interaction with a human teacher over several possible types of interaction that have been studied in prior works. The key contribution is an algorithm that uses greedy selection to determine the interaction type. This step is incorporated into an algorithm that builds on previous work in interactive robot learning to update the learner’s estimation of feature weights. An empirical evaluation of the algorithm shows how interaction types vary over the learning process, in different types of problem, and with non-uniform weighting on interaction types. Finally the manuscript demonstrates that  utilizing multi-type interactions generally outperforms a single interaction type methods.
strengths_and_weaknesses: #### Strengths
- The paper integrates several well-founded interaction methods into a uniform framework.
- The empirical results of the interaction selection (figure 1) and its accompanying discussion are clear and support the central idea that combining interactions should lead to overall better performance by the learner.
- In comparisons to learners using single types of interactions, the paper reports multiple performance measures (alignment with ground truth weight vectors and task reward).
- Result show the algorithm presented generally outperforms comparison points in 3 of 4 tasks and has best mean learning performance of the 6 frameworks evaluated.

#### Weaknesses
- As discussed in the limitations section, an example of the algorithm implemented with a human user would be valuable. The paper does not discuss how the different interaction type requests would be communicated to the user. As this paper does seem to take this step closer to implementation ‘in the wild’, it would be helpful to address this.
- The comparison experiments utilize at least 1 demonstration to initiate the learning algorithm. However, there is little discussion of what assumptions are made about the demonstrations. Prior work has shown that suboptimal (A. Colome and C. Torras, “Dual reps: A generalization of relative entropy policy search exploiting bad experiences,” Transactions on Robotics, vol. 33, no. 4, pp. 978–985, 2017.), failed demonstrations (K. Shiarlis, J. Messias, and S. Whiteson, “Inverse reinforcement learning from failure,” 2016; D. H. Grollman and A. G. Billard, “Robot learning from failed demonstrations,” Int. Journal on Social Robotics, vol. 4, no. 4, pp. 331–342, 2012.), and even intentionally bad demonstrations (A. Kalinowska, A. Prabhakar, K. Fitzsimons, and T.D. Murphey, “Ergodic imitation: Learning from what to do and what not to do,” ICRA, pp. 3648–3654, 2021.). Would the performance of the algorithm still hold if the demonstrations were suboptimal or even negative demonstrations?

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents an integration of several interactive learning methods. The framework could easily be built upon by others in the robotics community to incorporate other interaction modalities and improve learning through HRI.
issues: No major or minor issues other than the weaknesses outline above.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: A lot of works studied robots learning from various forms of human feedback, e.g., comparison, demonstrations, corrections, etc. Additionally, a lot of effort has been put into how to actively query humans for a specific feedback type. This paper tries to unify different feedback types so that the robot can optimize for both what type of feedback to acquire and what query (of that feedback type) to make.

The paper is well-written and mostly easy to follow. Multiple experiments have been conducted in simulated domains with simulated oracles. Baseline methods from the literature has been used.
strengths_and_weaknesses: Strengths:
- The paper is trying to tackle a problem that has been discussed in the community for a relatively long time now.
- The framework allows more feedback types than any other paper in this domain (except [15]).

Weaknesses:
- The paper is trying to solve this famous problem using the most obvious approach: let the system designer decide a cost value for each feedback type. However, the difficulty of the problem is already because it is difficult to come up with those cost values. In that sense, it is not really solving the problem. If we knew cost values, this approach would already be what we would use, because it is already pointed out by earlier work (e.g., [12] or "Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences" by Biyik et al.)
- Comparisons with baseline methods are useful and needed, but it does not really say much: the baselines are already expected to perform worse than the proposed approach because they do not have the same "action space" as the proposed approach. For example, one of the baselines is allowed to only make preference queries whereas the proposed approach can ask for demonstrations, corrections, and binary feedback in addition to preferences.
- Equations 12 and 13 are confusing. Worse than that, I am concerned the paper makes a mathematical mistake while obtaining and using the set \Omega. Please see my detailed comments below.




Below is my detailed review of the paper.


A lot of works studied robots learning from various forms of human feedback, e.g., comparison, demonstrations, corrections, etc. Additionally, a lot of effort has been put into how to actively query humans for a specific feedback type. This paper tries to unify different feedback types so that the robot can optimize for both what type of feedback to acquire and what query (of that feedback type) to make. I am very familiar with the literature in this domain, and I know this has been an open problem for some time, which many researchers have been discussing. So I appreciate the authors' effort for trying to address this problem and the contribution.

However, the paper is trying to tackle this problem through the most obvious / straightforward way: "Let the system designer decide the cost for each particular feedback type, and then maximize informativeness under some cost constraint, or maximize the difference". I think the difficulty of the problem is not in optimizing over different feedback types or learning from various forms of feedback. The difficulty is in coming up with the cost values. Hence, this paper is not really solving the problem. This is my most major concern about the paper.

Below are my other comments and criticisms:
1- This is just formatting, but is the paper scanned from hard-copy papers? For some reason, I am not able to select texts in the PDF and some portions of the PDF have lower quality. This makes reviewing on a computer a little painful (and I don't want to print the paper for obvious reasons). Please upload the LaTeX output next time.
2- In line 22 and in various other places, the paper uses the term "binary feedback" to refer to a feedback type where a human says "successful" or "unsuccessful" for a trajectory. However, "binary feedback" is a more general term than that. For example comparisons (or "preferences" as used by the paper) are also binary. I suggest using another term instead of "binary feedback". Similarly, preferences do not have to be binary: they can be, for example, rankings. But I know the term preference is usually used for pairwise comparisons, so this is okay.
3- The second paragraph of the Introduction does not mention the mental or physical effort required from the teacher while talking about the differences between the feedback types. But this is really one of the main differences here. This is later discussed, but it should be mentioned here, too, because this is the first place where differences are discussed.
4- The third paragraph says previous works typically do not work with multiple feedback types. However, [12] and the paper by Biyik et al. that I mentioned earlier handle preferences and demonstrations together.
5- I do not understand what "adapting to a state" means in the last sentence of Introduction.
6- Section 2 mentions volume removal and information gain based active querying techniques, which have really been the two most dominant techniques (even though recent works, e.g., [1], showed info gain is strictly better). But it would be nice to mention other alternatives too, e.g., max regret optimization. See "Active preference learning using maximum regret" by Wilde et al.
7- Relatedly, in general, the paper is missing many important citations, e.g.,:
7a) "Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations" by Brown et al., which also leverages both demonstrations and rankings.
7b) "ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapes" by Li et al., which combines preferences with ordinal labels.
7c) "APReL: A Library for Active Preference-based Reward Learning Algorithms" by Biyik et al., which introduces a software library that combines multiple different preference feedback types and demonstrations.
7d) "Unified learning from demonstrations, corrections, and preferences during physical human-robot interaction" by Mehta and Losey. The name should say it all.
8- Line 88-98 introduce an implicit assumption that the initial state is fixed during a querying iteration, so the robot cannot choose which state the demonstration should start at. This is okay, but this should be clearly and explicitly stated.
9- If the initial state is fixed, then there is no active querying in demonstrations and corrections. Is this true? If yes, then "Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences" by Biyik et al. already proved that active feedback should be collected after non-active feedback. Why is this knowledge not utilized in selecting feedback types?
10- It seems demonstrations and corrections are mathematically equivalent, as they are non-active. Is this correct?
11- In Corr. row of Table 1 and in line 94, I believe "$q$" must be replaced with "$t$".
12- Equation 3 reduces to Bayesian IRL, so that paper by Ramachandran and Amir should also be cited.
13- The paper says $\beta$ was set to 20. This is meaningful only if there is a magnitude constraint on $\omega$. Otherwise, one can scale \omega arbitrarily and $\beta$ becomes meaningless. Please clarify any constraints on the magnitude of $\omega$.
14- The paper says $E$ is a matrix, but it is not. It is a tensor. Relatedly, what does the transpose of a tensor mean in Equation 6? Is it being transposed over the first two dimensions? Please clarify.
15- Lines 115 through 126 (and equations 4-9) are just an implementation trick, not a novelty. Although I appreciate the fact that it is shared, I think it best fits to the Appendix.
16- To calculate information gain, the framework uses a discrete set of trajectories of size $T$. For demonstrations (and corrections), this means as $T$ increases, the information gain value (which is approximated) will increase (as can be seen from Equation 9). So the information gain estimates are not unbiased, they are under-estimated. This should be discussed.
17- In Equation 10, what's the motivation for using $1/\log(\lambda_i)$ instead of a simple additive or multiplicative term? Is this somehow more interpretable or intuitive? Please elaborate.
18- Lines 136 and 137 are concerning. Do they mean the framework does not sample the set $\Omega$ from the posterior over $\omega$? The way the paper currently reads tells me that there is a bunch of samples, and then gradient ascent is applied to each sample until they converge (before the next query). What if they all go to the same local maximum (which is unlikely not to happen)? However, if the set $\Omega$ is obtained in this way, then the samples in it are not really samples from the posterior. In turn, this makes the set useless for information gain computation, because its derivation is based on the fact that $\Omega$ are the samples from the posterior.
19- The paper should cite OpenAI Gym's paper for LunarLander. I am not familiar with the Pizza Arrangement domain, but if it is from a literature, that should also be cited.
20- Experiments do not really say much, because of multiple reasons: (1) the baselines are already expected to perform worse than the proposed approach because they do not have the same "action space" as the proposed approach. For example, one of the baselines is allowed to only make preference queries whereas the proposed approach can ask for demonstrations, corrections, and binary feedback in addition to preferences. (2) The simulated humans are oracles. (3) The system has not been deployed on a real robotic system.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The problem is interesting, relevant and important. But the proposed framework is not addressing the main difficulty of the problem, it is introducing an obvious framework for combining different feedback types. Besides, the literature review in the paper is very limited. Finally, experiments are not realistic (e.g., oracle humans and simulated systems). All other concerns are relatively minor.
issues: Please see my comments in the "Strengths and Weaknesses" section. I moved everything there due to character limits.
review_title: null

Meta-review: metareview: Summary:

In this paper, we propose a method for estimating which robot is the target of human attention in a situation where a multi-agent robot and a human are interacting by using the information quantity Transfer Entropy. The proposed method enables active estimation, replacing conventional methods for eye detection.

Strength:

- The paper is well written with precise problem formulation, method description, notation, etc.
- The paper presents a clear description with matrix formulas of how the information gain across querying modalities is efficiently computed.
- Results show the algorithm presented generally outperforms comparison points in 3 of 4 tasks and has the best mean learning performance of the six frameworks evaluated.

Weakness:	

- A weakness is the lack of actual robot experiments and a lack of real human user studies—real humans can be significantly different from a Boltzmann-rational model and exhibit multimodal reward preferences.
- The paper does not discuss how the different interaction type requests would be communicated to the user.
- The comparison experiments utilize at least one demonstration to initiate the learning algorithm. However, there is little discussion of what assumptions are made about the demonstrations.
- It is unclear if INQUIRE is significantly better than baseline approaches.
- Some visualizations would be required to give a better explain the specific tasks and an intuitive understanding of the complexity.

confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)