Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper poses a problem for “explicable policy search,” which is defined as the problem of developing a planner (i.e., via reinforcement learning) that produces behavior that are aligned with the expectations of a human observer. The paper develops an approach that infers the human’s latent objective function through preference-based RL (though a Behavior Cloning objective is described first). The paper evaluates the approach in a computational environment with synthetic humans showing promising results. The paper then conducts a human-subject experiment showing that human’s subjectively rate the proposed approach higher than baselines.
strengths_and_weaknesses: trengths:
+ The paper’s writing is generally clear with some grammatical issues. The figures were helpful.
+ The topic is timely, as researchers express increasing interest in explainable artificial intelligence in general. Policy explicability is an under-studied topic.
+ The problem formulation is clear, and the approach is well-reasoned.
+ The computational results (Table 1) show competitive returns and superior explicability scores.
+ The visualization in Figure 2 helps provide insight into the behavior of the planners.
+ The paper conducts a human-subject experiment to evaluate the approach, showing that it outperforms baselines with statistically significant results. The study was IRB-approved.

Weaknesses:
-Line 85 states that the surrogate reward function encodes the “necessary information.” Is that true? Is the information learned truly “necessary” or are components of the surrogate reward only approximately correct, superfluous, and/or incongruent with the truly-necessary information? What would the sufficient information be?
-In this paper, the term “explicable” in regards to planning is defined as a plan that aligns with the human’s expectations for what a plan should look like. This is but one form of explicability, or, colloquially, explainability. For example, the paper seems to miss prior work on value-aligned reinforcement learning, such as:

Nahian, M.S.A., Frazier, S., Harrison, B. and Riedl, M., 2021. Training value-aligned reinforcement learning agents using a normative prior. arXiv preprint arXiv:2104.09469.

Brown, D.S., Schneider, J., Dragan, A. and Niekum, S., 2021, July. Value alignment verification. In International Conference on Machine Learning (pp. 1105-1115). PMLR.

The value-aligned RL work seems to be highly similar in idea: creating RL agents that align with human expectations. The paper is also missing prior work on interpretable reinforcement learning, such as: 

Silva, A., Gombolay, M., Killian, T., Jimenez, I. and Son, S.H., 2020, June. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International Conference on Artificial Intelligence and Statistics (pp. 1855-1865). PMLR.

Paleja, R., Niu, Y., Silva, A., Ritchie, C., Choi, S. and Gombolay, M., 2022. Learning interpretable, high-performing policies for continuous control problems. arXiv preprint arXiv:2202.02352.

While these other approaches are not “explicable” in the sense that they do not align with a human’s expectations, they are interpretable, i.e. directly “explainable.”

It would have been helpful to include the Brown et al. (2021) method as a baseline and argue why explicable behavior is better than interpretable behavior (relatively to Silva et al., 2020), considering the position of Rudin (2019).

Rudin, C., 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), pp.206-215.

Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L. and Zhong, C., 2022. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16, pp.1-85.

-The approach described in Equation 1 seems quite similar to the mechanisms explored in InfoGAIL (Li et al., 2017). In their paper, Equation 4 shows a discriminator loss for a generative-adversarial approach to essentially Behavior Cloning/Dagger, an entropy bonus, a mutual information term, and the reward function. If we throw out the entropy bonus and mutual information term (which is used for aligning with heterogeneous humans), then we basically recover what we have in Equation 2 of this paper: one term for the expected return (or reward function) and one for aligning with human behavior (behavior cloning).

Li, Y., Song, J. and Ermon, S., 2017. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems, 30.

- Following up on the previous point, it is fair to note that replacing the divergence term in Equation 2 and arriving at Equation 6 is literally a different formulation (e.g., replacing imitation learning with a preference-based RL, exponential distribution-type reward function inference setup) than in InfoGAIL. However, the improvement seems incremental to replace the KL divergence with U_H, and it would have been helpful to acknowledge this connection.

- Further, the inclusion of entropy (as described in Lines 205-207 further recreates the setup from InfoGAIL without attribution. 

- The equation in referenced from [7] goes back at least a decade further to Lucas et al. (2008) and was leveraged again by Brown et al. (2019 and 2020) for preference-aligned RL.

Lucas, C., Griffiths, T., Xu, F. and Fawcett, C., 2008. A rational model of preference learning and choice prediction by children. Advances in neural information processing systems, 21.

Brown, D., Goo, W., Nagarajan, P. and Niekum, S., 2019, May. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning (pp. 783-792). PMLR.

Brown, D.S., Goo, W. and Niekum, S., 2020, May. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning (pp. 330-359). PMLR.

- The design of the Likert scales and analysis could be improved. For example, results of test for the ANOVA assumptions (e.g., normality and homoscedasticity) would be helpful. A useful reference would be Schrum et al. (2020).

Schrum, M.L., Johnson, M., Ghuy, M. and Gombolay, M.C., 2020, March. Four years in review: Statistical practices of likert scales in human-robot interaction studies. In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (pp. 43-52).

Writing:
-There are two uses of the word “it” in Line 4 that may refer to different subjects. Please try to avoid using pronouns with ambiguous antecedents.
-“behavior, due” does not need a comma
-Line 16, “they” is ambiguous

questions: -Line 14, are the human’s expectations truly “hidden” or just partially observable, latent random variable?
-How is proposition 1 fundamentally different than claims of Maximum Entropy Inverse Reinforcement Learning for learning a unique reward function? How does Proposition 1 eliminate the reward ambiguity problem?
-Could the authors provide more information about why EPS did not conform to the human’s behavior in D3?

limitations: - The authors explicitly have a section on limitations and future work, which is helpful. 
- The paper states that the research is “fundamental in nature without notable negative societal impacts.” I appreciate the response, but the response does not seem to be in-line with the expectations for NeurIPS. At one point, GANs might have seemed to be fundamental research, but Deep Fake videos offer the world a tool for disinformation. I would recommend spending additional time pondering whether explicability in planning – as defined here – has benefits and potential harms to society and whether the benefits outweigh the harms. For the record, this reviewer believes the research is ethical. The reviewer is only asking for a more full attempt to fulfill the expectations of NeurIPS.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors propose Explicable Policy Search (EPS), a method that aligns the behavior of an RL agent with the one expected by a human observer. More precisely, the goal of the method is to learn a policy that reconciles between maximizing the long-term return and minimizing the deviation from the expected behavior. This is achieved by considering a linear combination of two obectives, namely the cumulative reward and a policy explicability score. Since the human's belied about the domain dynamics is hidden to the agent and cannot be queried, the authors demonstrate that it is possible to use a surrogate reward function that encodes the information needed for EPS.

strengths_and_weaknesses: #### Strengths

Overall, the paper describes a nice and interesting work.
I found the paper very well written and easy to follow. The content is presented in a clear and effective way. 

The method is sound, simple and elegant.

The paper provides a nice formalism to extend explicable planning to a RL setting with continuos state and action spaces.


#### Weaknesses


The paper is interesting but in the current version the experimental evaluation is a bit lacking. Experiments are mainly performed on synthetic tasks, which are good for a preliminary assessment of the soundness and feasibility of the method, but should be complemented with more analyses. The study with human subjects is interesting but it is only performed on one toy task.

Moreover, the evaluation assesses the deviation from the human's expected behavior in Table 1 based on the explicability score. However, it looks obvious that EPS achieves a better explicability score compared to the other approaches as it is designed to optimize the explicability score. I understand that it is interesting that the averaged return is still high compared to the baselines, but it would be more interesting to perform a human evaluation of the explicability of the approaches.

Overall, I think the paper would benefit from more experiments on real-world data, but even more ablation studies would be interesting. For instance, the appendix includes an experiment on the choice of $\lambda$ for the synthetic domains. I think these experiment could be in the main paper and it would be nice if the paper could include a similar experiment for the autonomous driving task too. Also, more experiments aimed at evaluating the role of the environment entropy (as described in lines 210-215) would be interesting.
questions: * The paper mentions that the actions space in the authonomous driving domain consists of 5 actions, but the actions are only reported in the Appendix. I think it would be helpful to the reader if these actions where listed in the main paper in order to better understand the task.
limitations: The authors properly described the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes an approach for incorporating human preferences into policy search by shaping the 'original' reward function with a 'preference-based' reward function for the human (other than learning the user belief via a respective policy and transition model). The authors learn the human preference-based reward based on available techniques. The approach is evaluated on environments with cont. state and action spaces (navigation & autonomous driving) based on both synthetic human preferences as well as a user study. The results show that the approach works better than pure RL algorithms as well as a preference-based approach.
strengths_and_weaknesses: Strengths: The topic of the paper is relevant and up-to-date, focussing on including user feedback into RL while taking into accont possible misjudgments of the end-users when giving preferences. The taken approach is sensible, shaping the agent's reward function with the user-dependent reward function. The emprical evaluation comprises a user study, which is very helpful for interactive ML.

Weaknesses: While the evaluation is insightful, it does not compare to more recent (and also already referenced) work on interactive RL, e.g. [24] of the paper. Also, it would significantly help the presentation to include the algorithm (available in the suppl. material) into the main paper. Lastly, I am wondering if such combination of agent- and user- reward function is helpful for the general advancement of RL. It feels like the human should remain in control when configuring the system (which seems to be the poinf of explainable, interpretable, safe RL research), but possibly convinced to change her/his perception. So wouldn't a better way be to incrementally suggest improvements to a policy learned via preference-based RL? Maybe this is equal to the direction mentioned in the future work section (pointing to more complex combinations of both reward functions), but it might also be necessary part to generate value.

Besides, a relevant related work (more into the direction of preference-based RL) is also: Reddy, S., Dragan, A., Levine, S., Legg, S. and Leike, J., 2020, November. Learning human objectives by evaluating hypothetical behavior. In International Conference on Machine Learning (pp. 8020-8029). PMLR.
questions: * How does the approach related to multi-objective RL? E.g. Yang, R., Sun, X. and Narasimhan, K., 2019. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in Neural Information Processing Systems, 32.

* How would the system perform against more recent preference-based RL approaches, improving on the used baseline? Could one (with confidence) state that even with close to optimal approximation of the user reward function, the systems would benefit from the combination with the "original" reward? 
limitations: The central limitation which is discussed deals with: The argumentation for learning a user-specific reward function for shaping states that it is unnecessary to learn the user-dependent transition function. Is this always the case? What is the underlying assumption for this? The discussion is surely userful. 

It would also be good to discuss the effects of such a straightforward combination of both reward functions. What kind of real world scenarios would this cover? Under which assumptions can a system disregard user preferences (e.g. when safety is incorprated via preferences)?

#### Update after author response & discussion
Again, I thank the reviewers for their answers and changes to the paper. After reading the answers, comments of the other reviewers and especially the changes to the paper, I am wiling to increase my score to above the acceptance threshold. Main concerns have been addressed and approriate changes are in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops both a formulation and solution to the problem of what the authors term Explicable Policy Search (EPS), i.e. how to compute policies that maximize a tradeoff between task reward and how expected (or "explicable") a policy is to a human teammate, in order to improve human-robot teaming. The authors formalize this as an objective that combines expected cumulative reward with an "explicability score" for the robot's policy, which is defined as the KL divergence between the robot's policy and the policy that would be expected under the human's model of the environment and the reward function they impute to the robot.

Naively, solving this problem would require access to or inference of the human's model of the environment, and the policy they expect the robot to perform. Instead, the authors show that this latent information can be encoded into an auxiliary reward function, which can be learned from human pairwise comparisons about whether some robot trajectories are more expected than others. Solving the modified MDP with these auxiliary rewards via standard RL methods yields policies that achieve high reward while being "explicable". The authors demonstrate this on a continuous RL test domain using simulated human data where each "human" has a different belief about the domain dynamics, as well as a self-driving car domain with real human feedback. The EPS algorithm outperformed non-"explicable" baselines in terms of the explicability score while still achieving a high amount of cumulative reward, and generated self-driving car trajectories that were more preferred by human raters.
strengths_and_weaknesses: This paper builds upon prior work on "explicable" and legible planning by introducing a clear and intuitive formulation of "planning in a way that is expected by humans" in the context of stochastic domains where autonomous agents compute policies (distributions over actions given states) and not just plans (ordered sequences of actions). Given that most real-world domains are in fact stochastic, this extension of explicable planning is undoubtedly a useful and significant one.

The mathematical insight used to solve this problem -- showing that human expectations can be encoded as auxiliary rewards -- is also sound, original, clearly explained. The resulting EPS algorithm is well-evaluated on the whole, with clear qualitative demonstrations of how EPS-computed policies lead to more expected behavior given human assumptions about the domain, a user study, and average quantitative performance that is higher in terms of explicability score than a number of baselines (though I have questions about the statistical significance and effect size).

The main concern I have about this paper is surrounding the term "explicable", and how "explicability" is formalized as a (negative) KL-divergence, even though negative KL is better understood as something like "expectedness". I understand that this is similar to how the term "explicable" has been used in prior work in planning, but I initially found it confusing and poorly chosen, because "explicable" evokes the term "explainable", and it's not at all obvious how minimizing KL-divergence (or difference from expected behavior in general) makes behavior more explainable.

Upon further thought however, I do think there *is* an intuitive sense in which the term "explicable" means something like "minimize KL divergence", but it requires a bit of justification / explanation: Typically, when we have a (probabilistic) model of some observations or behavior, we say that the observations are "well-explained" by the model when the observations are assigned high-probability by the model (i.e. their log-likelihood is high). This sort of language is especially apt when the model in question is causal model of the world, and so assigning high probability to the observations corresponds to having found a good causal explanation of the data. In contrast, when the observations have low probability, we might say that they are "inexplicable" with respect to the model.

In the context of EPS, the model in question is the human's intuitive causal model of how the world works $\mathcal{T}_A^H$, and how an approximately rational agent would act in that world to achieve its goals. And so, if a trajectory $\tau$ pursued by an agent has high-probability under the human's model, it is "well-explained" under the model, hence "explicable". Otherwise it is "inexplicable". Hence, the log-likelihood of an agent's trajectory $\tau$ under the human's model $p_A^H(\tau)$ can be treated as an explicability metric.

Now as the authors point out, we don't just want to evaluate specific agent trajectories or plans, but agent policies $\pi_A$, which (when combined with the true environment dynamics) are distributions over trajectories. This motivates taking an expectation over this distribution over how explicable each trajectory is:
  - If the explicability of a single trajectory is defined as $\log p_A^H(\tau)$, then expectation is the negative cross-entropy $E_{p_A(\tau)}[\log p_A^H(\tau)]$ of the human's model over trajectories, relative to the true model.
  - If the explicability of a single trajectory is instead defined as the log odds ratio $\log \frac{p_A^H(\tau)}{p_A(\tau)}$, then the resulting expectation is the negative KL divergence $-D_{KL}(p_A(\tau)||p_A^H(\tau))$.

Either of these gives us a way to evaluate how explicable a policy is relative to the human's model. It's interesting to consider the differences -- and I'd be curious what the authors think about using cross-entropy instead -- but I think the more important point is that this sort of justification for using the term "explicable" is missing from the paper as currently written. I would highly encourage the authors to reframe how their explicability score is introduced and defined, in order to incorporate something like the above. Otherwise, the conceptual gap between "expected by the human" and "explicable" is not an obvious one for readers to bridge. In case helpful, [1] is a paper that connects the quantity of log-likelihood to the concept of "inexplicability", which is formalized alongside other measures of human surprise.

Apart from the above, I only have a number of other minor questions and suggestions, which I'll list in the next section.

[1] Zhi-Xuan, T., Gothoskar, N., Pollok, F., Gutfreund, D., Tenenbaum, J. B. & Mansinghka, V. K.  (2022). [Solving the Baby Intuitions Benchmark with a Hierarchically Bayesian Theory of Mind.](https://social-intelligence-human-ai.github.io/docs/camready_13.pdf) Social Intelligence in Humans and Robots, RSS 2022 Workshop.
questions: 1. As highlighted above, why call formalize "explicability" as negative KL-divergence, when that more directly captures "expectedness under the human model"? It'd be great if the authors could do more to explain this.

2. Per the discussion above, have the authors considered using negative cross-entropy $E_{p_A(\tau)}[\log p_A^H(\tau)]$ as an explicability score, instead of negative KL-divergence? It's not obvious to me that the incentive to maximize environment entropy or policy entropy (Lines 208-215) is always desired, since this means the agent's behavior will be more unpredicatable!

3. On Line 130, setting the true domain dynamics $\mathcal{T}_A$ to be unknown seems unnecessarily strong a requirement. Surely explicable policy search can be applied in the case where the domain dynamics known as well -- you could just use model-based RL / value iteration instead of model-free methods like SAC.

4. In equation 6, why approximate the original objective by discounting the surrogate reward $u_H$, along with the other entropy terms? It seems like you could still treat them as additional non-discounted rewards, as long as you're optimizing over finite horizons.

5. Also, the phrase "ignoring the influence of the discount factor on the surrogate reward function and entropy term" on Lines 197-198 was confusing to me -- aren't you doing the opposite, by applying discounting to the surrogate reward?

6. Are the standard deviations shown in Table 1 taken over the full set of 100 rollouts (i.e. the sample standard deviation)? Or is that the standard error of the mean? I ask because if it's the latter, then the higher explicability of EPS relative to SAC wouldn't be statistically significant. (No doubt, average-case behavior isn't everything, but is still important).

7. Relatedly, I'm surprised that SAC does so well in terms of explicability score on average. Is there a reason why this is the case? Is it just because in most case, the most optimal behavior happens to be mostly explicable?
limitations: Authors address a number of limitations which are relevant to extending EPS to real settings where human beliefs about the world may change. Another limitation comes to mind is that EPS is currently restricted to the case where agents have a fixed goal known to the human. How to extend EPS to the case where the agent's goal is not predetermined, and the human may have uncertainty with respect to the agent's goal. Can some combination of explicable and legible planning be used here, and how?

As for potential negative societal impact, the authors claim their work in fundamental, but in my mind it is fairly close to being applied (in e.g. human robot collaboration contexts). I would suggest adding a few words about potential negative impacts from failing to adequately learn and encode the human model through the surrogate reward, or the downsides of trading off too much performance or safety for explicability (which seems to arise in the autonomous car driving scenario).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a way to make policy search "explicable" to humans, by eliciting their preferences over trajectories and combining a conventional "engineered" reward function with those trajectory preferences. The method is validated in empirical experiments as well as a user study in a small autonomous driving example.

strengths_and_weaknesses: Strengths:
- Appears to be a novel niche in a relevant problem.
- Inclusion of a user study is a nice addition to this kind of paper.

Weaknesses:
- The initial motivation of the approach is weak and confusing. The positioning against related work in RL could also be better. It wasn't clear until the end of the paper that this might be a valid (if rather narrow) problem and that the approach might work. 
- The mathematical derivation seems needlessly complex and seems to have a typo or mistake (see below). The technical contributions do not appear very significant.

Details:
- Motivation: The "explicable" solution in the motivating example (Fig 1a). is actually inexplicable to me. Since the proposed approach attempts to plan w.rt. wind, I assume the wind speed is constant (non-constant wind gusts are almost unpredictable and hence unplannable). Having worked with planning for drones, if the wind is constant, but less than the control authority of the drone (here drawn as a quadrotor), it would deviate some depending on the type/parameters of the controller,  strength of wind, and length of the trajectory. However, without knowing those variables, either of the drawn scenarios could happen. In addition, the "expected" dashed line is not straight either but curves in the opposite direction? This example could use a rework as the supposedly explicable and optimal solutions do not appear more explicable or optimal respectively.

- As you appear to use the preference estimation machinery from [25,7], you might want to compare your work against those in the related work. Other RL works have also combined conventional rewards with a KL-divergence objective on trajectories (e.g. variants of Guided Policy Search). Here you are essentially learning this distribution by human preferences learned elicited as in [25,7]?

- The mathematical derivation in 4.1 seems needlessly complicated, you already defined u_H on line 172 to be identical to the log of the human trajectory distribution, so Prop 1 and its proof seems redundant (c.f. e.g. line 172 vs. 188-189).

- Eq. 6 does not appear to be a reformulation of Eq. 2 as claimed. As the authors note they suddenly made the KL-divergence term use the RL discount factor, but more importantly, if you convert the KL-divergence terms from (2) into entropies, shouldn't they be outside the expectation over p_A? 

- Eliciting reward functions from human observers appears like a slow process, and the empirical benchmarks are all very small toy examples. 

- The autonomous driving user study is a nice inclusion that perhaps should have been given more attention earlier on. 

Minor: 
- l46: calling it "business" seems informal
- l53: "we assume that the human is nosily rational at generating her expectation of the agent to accommodate her computational limitation" - noisily rational? I am not familiar with this particular model of rationality, it might be good to explain what this means.


questions: 1) Is the actual contribution here the more efficient way of estimating the human beliefs over dynamics + policy by adding them together into a "surrogate reward function" directly learned by preference learning over trajectories, or the concept itself of using a linear combination learned preferences from [25,7] with a regular (given) RL reward function, or both? Please clarify positioning vs. earlier work in RL.

2) Shouldn't the entropies in Eq. 6 be outside the expectation over p_A? Is this just a typo?

3) Estimating the entropy over the true dynamics seems potentially very difficult in the general case for more complex and higher-dimensional state distributions. It is just briefly mentioned that this is done by learning the true dynamics via a NN with a Gaussian head. For more complex state distributions than narrow trajectory distributions, I wonder how reliable this would be. Any error here seems like it could also affect tuning of lambda.

4) How to set lambda? The first part of the objective is a reward, the second part is a messy approximation to a KL divergence. It is not obvious to me that even if you elicit trajectory preferences from people that making a good trade off here in the end won't still require human input.

limitations: Sufficiently adequate (pending questions above).
review_title: null

Meta-review: metareview: I thank the authors for their submission and active participation in the discussions. This paper studies the problem of devising an RL planner that produces bahviour consistent with human observer preferences. Reviewers remarked that the paper studies a timely problem [VqJ9,gaQq,5GPj], containing clear writing [VqJ9,PBHQ] and useful visualizations [VqJ9], and provides insightful human evaluations [VqJ9,gaQq,5GPj], and that the method is sound and elegant [PBHQ,TX5X]. During AC/reviewer discussion, reviewers TX5X, gaQq and VqJ9 agree that the author response has addressed the main concerns. I am slightly discounting the negative score by reviewer PBHQ as I don't find their suggestion to include more experiments and real world data very concrete or actionable. Thus, I overall see support for accepting the paper and am therefore recommending acceptance while encouraging the authors to further improve their paper based on the reviewer feedback.

confidence: Certain
Decision: Accept