Explicable Policy Search
Authors: Ze Gong
Yu Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Explainable Decision Making
Human-Aware AI

Abstract: Human teammates often form conscious and subconscious expectations of each other during interaction. Teaming success is contingent on whether such expectations can be met. Similarly, for an intelligent agent to operate beside a human, it must consider the human’s expectation of its behavior. Disregarding such expectations can lead to the loss of trust and degraded team performance. A key challenge here is that the human’s expectation may not align with the agent’s optimal behavior, e.g., due to the human’s partial or inaccurate understanding of the task domain. Prior work on explicable planning described the ability of agents to respect their human teammate’s expectations by trading off task performance for more expected or “explicable” behaviors. In this paper, we introduce Explicable Policy Search (EPS) to significantly extend such an ability to stochastic domains in a reinforcement learning (RL) setting with continuous state and action spaces. Furthermore, in contrast to the traditional RL methods, EPS must at the same time infer the human’s hidden expectations. Such inferences require information about the human’s belief about the domain dynamics and her reward model but directly querying them is impractical. We demonstrate that such information can be necessarily and sufficiently encoded by a surrogate reward function for EPS, which can be learned based on the human’s feedback on the agent’s behavior. The surrogate reward function is then used to reshape the agent’s reward function, which is shown to be equivalent to searching for an explicable policy. We evaluate EPS in a set of navigation domains with synthetic human models and in an autonomous driving domain with a user study. The results suggest that our method can generate explicable behaviors that reconcile task performance with human expectations intelligently and has real-world relevance in human-agent teaming domains.

Explicable Policy Search
Ze Gong Arizona State University Tempe, AZ 85281 zgong1 @asu. edu
Yu Zhang
Arizona State University Tempe, AZ 85281
yzhan4420asu, edu
Abstract
Human teammates often form conscious and subconscious expectations of each other during interaction. Teaming success is contingent whether such expec- tations can be met. Similarly, for an intelligent agent to operate beside a human, it must consider the human's expectation of its behavior. Disregarding such ex- pectations can lead to the loss trust and degraded team performance. A key challenge here is that the human's expectation may align with the agent's s optimal behavior, e.g., due to the human's ' partial or inaccurate understanding of the task domain. Prior work explicable planning described the ability of agents to respect their human teammate's expectations by trading off task performance for more expected "explicable" behaviors. this paper, we introduce Explicable Policy Search (EPS) to significantly extend such an ability to stochastic domains in a reinforcement learning (RL) setting with continuous state and action spaces. Furthermore, in contrast to the traditional RL methods, EPS must at the same time infer the human's 's hidden expectations. Such inferences require information about the human's 's belief about the domain dynamics and her reward model but directly querying them impractical. We demonstrate that such information can be necessarily and sufficiently encoded by surrogate reward function for EPS, which can be learned based on the human's feedback on the agent's behavior. The surrogate reward function is then used reshape the agent's reward function, which is shown to be equivalent to searching for an explicable policy. We evaluate EPS set of navigation domains with synthetic human models and in autonomous driving domain with a user study. The results suggest that our method can generate explicable behaviors that reconcile task performance with human expectations intelligently and has world relevance in human-agent teaming domains.
Introduction
Intelligent agents are quickly becoming a part of our daily lives in variety of domains, including smart home, autonomous driving, education, and so on. In such domains, the agents are expected to perform in human inhabited environments and even collaborate closely with them. Members in such a collaborative setting often form conscious and subconscious expectations of each other and the success of collaboration depends whether such expectations can be met. A key challenge here is that the human's 's expectation may not align with the agent's optimal behavior. Hence, agents choosing their optimal behaviors without considering the human observers or collaborators could be seen as unexpected, leading to degraded team performance and loss of trust [19 6
Consider drone navigation scenario in Fig. La where a drone is tasked to navigate from the starting position (as shown) to a destination. On a good day, the drone would be expected to navigate (almost) straight to the goal under mild air perturbation (thus possibly introducing a slightly curved trajectory as illustrated red). When there is heavy wind that changes the domain dynamics (which the human observer is unaware of), it may become impossible for the drone navigate usual. Two
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Agent Behavior Generation
Expectation Generation
TA
MA
π
M
Optimal
Expected
Dyplcablle
(a) Motivating scenario.
(b) Problem setting of EPS.
The agent learns its behavior with a given engineered reward model, YA, in the task environment under the true domain dynamics TA. The human generates her expectation of the agent's behavior T based on TAA that captures her belief or understanding of the true domain dynamics, and her reward model ти. which may differ from Ta and r. respectively. Such differences may be due to personal preferences in rHz biases and misunderstandings about the domain dynamics, etc. In the traditional RL setting, the agent computes the optimal policy æ under its models, which could be different from its expectation formed by the human based on her models. Shaded nodes are known to the learning agent and unshaded nodes are unknown.
Figure 1:
alternatives illustrated where the drone navigates in zigzag (green) and a curved pathway (black), respectively. The curved pathway is more optimal since makes fewer sharp turns but the former is likely to be perceived as more expected to the human observer, given its similarity to the expected behavior. We note that searching for the explicable behavior goes beyond choosing from or combining the agent's optimal behavior and the human's 's expected behavior. Hence, achieving such capability requires a more fundamental treatment. In the experiments, we show that our method can generate novel behaviors that are unseen in the training scenarios (where human feedback is solicited).
The problem setting of explicable policy search (EPS) is illustrated Fig. T6 which fundamentally model reconciliation problem (7]. In such a setting, considering the learning problem only from the perspective of the agent could sabotage the teaming performance when the agent's behavior differs from human's 's expectation, while considering only from the human's 's perspective may be undesirable either. Our method considers any feedback the human may provide to the agent based on her expectation of its behavior to help the agent improve its behavior. Hence, our method has variety of applications where machines must be tuned to individual users, such as an autonomous vehicle that learns improve its driving behavior based on its owner's feedback but at the same time must abide by standardized operating requirements to ensure safety, comfort, etc. In our problem, we assume that the human is nosily rational at generating her expectation of the agent accommodate her computational limitations. Noisy rationality is one type of bounded rationality (45) due to a limited computational capability, resulting in sub-optimal decisions. While noisy rationality may not perfectly describe the human's 's computational model [43 due to various cognitive biases, modeling humans as such has been a common practice and a reasonable assumption in prior work [34] 2). Furthermore, similar to prior work on explicable planning [48] 25). we assume that the human is a sole observer of the agent. Extending to an collaborative setting requires additional machinery and will be discussed in future work, with steps already taken in a classical planning setting [7].
In this paper, we formulate explicable policy search (EPS) in model reconciliation setting. The goal is to learn a policy that reconciles maximizing the long-term return (based on the engineered reward model ra) with maximizing the expectedness of the agent's behavior from the observer's perspective, which is quantified by an explicability score. We formulate the problem as a simple linear combination of two objectives, which already yields substantial benefits compared to several baseline methods in our evaluation. The challenge in our problem formulation mainly lies in the fact that the human's expectation is hidden (but partially observable via human feedback) and must be learned. A straightforward solution involves learning both the human's 's belief about the domain dynamics (T and her reward model (r )) from human feedback, which is possible but impractical. Instead, we show that the information needed from them for EPS can be necessarily and sufficiently encoded by a surrogate reward function, which is much easier to learn. Such a reward function can then be incorporated into policy search process to recover an explicable policy. The final learning objective of EPS after the modification bears some similarity to maximum entropy reinforcement learning [20 21 but is derived under completely different motivation.

---

For evaluation, we design a set of navigation domains where the human's 's feedback on her expectation of the agent's behavior is synthetically generated based on the true domain dynamics modified with various "misunderstaandngss.. We compare our method with three baselines that are selected to repre- sent the traditional RI. methods to demonstrate the key advantages of EPS under model differences. Furthermore, to show its real-world relevance, we conduct a human study in an autonomous driving domain [29] where we design driving scenarios to elicit existing human biases about the vehicle's s domain dynamics that could be dangerous to ignore during learning. The results show that our method can intelligently generate explicable behaviors that are safe and preferred over those of the baselines.
The contribution of this paper is three-fold. First, we introduce and formulate explicable policy search (EPS), which significantly extends explicable planning to stochastic domains in a reinforcement learning setting with continuous state and action spaces. Second, we propose practical solution for this challenging problem by introducing a surrogate reward function learned from human feedback data, which encodes the necessary and sufficient information for explicable policy search. Third, we evaluate our learning method extensively with simulations and human subjects.
2 Related Work
The problem of generating communicative actions or behaviors (a.k.a. explainable behaviors) has been well studied as subarea of explainable AL. Explainable behavior generation has a subtle but important difference from interpretable behavior generation [54 38 39). While the former aims at generating behaviors that are inherently and implicitly interpretable humans, the latter is focused on generating behaviors under interpretable model representations so that their rationales can be explicitly conveyed. In this regard, a behavior generated by the latter may not be interpretable by itself but an explainable behavior will be. In domains where explicit communication should be used sparingly, such as in human-agent teaming, or high-stake domains where "surprises" should be minimized, explainable behaviors would be more desirable. Various terms have been introduced for different aspects of explainable behavior generation such as legibility, explicability, transparency, etc. For a review of their similarities and differences, refer to 77]. A closely related term to our work, legibility, for legible motion planning [12]. is about generating trajectories that better reveal the underlying goal of the agent. Explicable planning [4] 25 16 47), which is the focus of our work, differs from legible motion planning in that it aims at generating plans that better align with the human's expectation given the goal. The key characterization of explicable planning methods revolves around the idea of model reconciliation where an agent makes decisions based on two different models instead of one, with focus on the differences in domain dynamics models X 6 77.
While there exists prior work (33 5 on training value-aligned learning agents that learn to respect different reward models, they do not consider the differences in the domain dynamics. Prior methods on explicable planning addressed the problem in a classical planning setting (e.g.. PDDL (T5)) under deterministic domains. It remains a challenge to extend the problem formulation to stochastic domains with continuous state and action spaces. Also, differing from the prior work on explicable planning, we consider that the user's reward model may also differ from the agent's, but assume the differences do not introduce a discrepancy in the perception of the agent's 's "goal" (which the user knows). Considering multiple candidate goals is the problem setting of legible motion planning [12]
As shown in Fig. 1b estimating the human's 's expectation requires learning both the human's 's belief about the domain dynamics and her reward model based human feedback. Existing work on inverse reinforcement learning (IRL) I 36 49 and reward learning 11 41 [3] can learn the reward model from human data with the assumption that the human has access an accurate model of the domain dynamics. It was shown that the human's 's inaccurate belief about the domain dynamics may skew the human's 's feedback and lead to learning the opposite preferences with respect to the human's 'ss true reward model (17). A similar issue may occur in policy learning for various methods that are devised use human feedback, which include reward shaping, policy shaping [18] and interactive
10 27 When the human has an inaccurate belief about the domain dynamics, as we will show, it can lead to convergeee during learning or high variances in the learned behaviors. While it may be possible to learn the human's 's belief about the domain dynamics separately [37], the belief and human's reward model are generally tightly coupled in the human's feedback and should be jointly learned. Joint learning is possible 23 17 but challenging due to its high dimensionality, which makes it impractical for real-world domains. similar vein, InfoGAll.. R00 uses latent code to discover salient semantic features to accommodate demonstrations from different experts. Even
RL [24 32
3

---

though the aim there is quite different from ours, one may consider the differences in the domain dynamies as captured by the latent code. However, their work is focused on the "discovery" while we focus on "reconciling" the differences (without explicit modeling them). When considering only differences in the domain dynamics, our problem may be viewed as form of off-dynamics learning [14] that addresses transfer learning from source to target domain. However, we do not have direct access to the target domain in EPS, which is hidden in the human's 's mind.
3 Problem Formulation
In this paper, we formulate any task domain as a Markov Decision Process (MDP). An MDP is represented by tuple M A, T ;r;;;7) P)7), where S is the set of states, A the set of actions, T(s)s,a) a) the transition function, r the reward function, p(.) the initial state distribution, and ? the discount factor. For the problem setting as shown in Fig. [b Assuming the two MDPs share the same S, A. and y. one is the agent's model MA with the true domain dynamics Ta and the engineered reward function TA and the other is the human's model of expectation M with her belief about the domain dynamics T and the human's 's reward function rи- For explicable policy search, TA, TH and rH are unknown. Note that our method can be easily applied to model-based learning setting when TA is known.
need to consider two MDPs.
Definition 1. Explicable Policy Search (EPS) is the problem of searching for a policy via learning
to maximize two objectives: the expected cumulative reward, and a policy explicability score between the agent's 's policy under MA and the expected policy under the human's 's model M
In this paper, we consider a linearly weighted sum of the two objectives:
arg max TT π.'
Σ ('ra((s, oz)
+ X E(T, T M
policy explicability score
cunulative reward
where x and n" denote the agent's policy and human's expected policy, respectively. We combine the two objectives linearly via a reconciliation factor À to be consistent with the literature [48] 25 and simplify the technical development.
Next, we motivate the design of our explicability score. Intuitively, a data generation model can better explain the data when it assigns the data with a high probability. Similarly, under the assumption that the human generates her expectations based on MA a behavior (i.e., trajectory) T is likely to be generated by the human's model (or expected by the human) when it is associated with a high probability in the model. contrast, when the trajectory has a low probability, we could say that it is "inexplicable" with respect to the model. As result, the probability of the trajectory in the human's 's model (M ), denoted by p (T), may be considered as our objective at first thought and its expectation over the distribution of trajectories as the explicability score (i.e., Er, PAA (r)). However, using pl (7) only ignores an important aspect of explicability under the model reconciliation setting considered since it involves two models, which requires it to be contrastive. More intuitively, for a policy to be the most explicable, trajectory with given probability in the human's model M A should also appear with the same probability in M Appearing more or less likely in MA should result a reduced explicability score. Hence, evaluating the explicability of a policy under such a requirement becomes evaluating the closeness of the two distributions. This observation naturally equates the choice for our explicability score to the negative KL-divergence of the two distributions of trajectories under the agent and human's 's models, respectively.
Hence, we introduce the policy explicability score for stochastic environments, which differs funda- mentally from the explicability scores defined in the prior work for deterministic domains. In [4] 25). the explicability score has been considered as a similarity metric between the agent's plan and the expected plan. Policy explicability score extends such a metric to consider plan distributions. They are orthogonal aspects of explicability that are equally important. Using the newly established policy explicability score, agent that learns to maximize it would learn to replicate (as closely as possible) what the human expects the agent to behave. When the two distributions match exactly, the policy explicability score would be O, exerting no influence on the final policy as desired. Eq. (T) becomes:
Σ (rralss + - (PA(T) (7)),
πt = arg max ET,TA
(2)
4

---

where PT(T) and (T) denote the distributions of trajectories under MA and M respectively.
Remark: Depending whether the agent's state and action are observable, the distribution of trajectories may be computed differently. When both the state and action are observable, the human would be able contrast both the agent's action and resulting state with her expectation; otherwise, only the observable part needs to be considered. In the following discussion, we assume that both the state and action are observable. For explicable policy search, the implication here is that the agent optimizing Eq. (2) has the incentive to avoid parts of the state space where either its policy differs from the human's expectation or the domain dynamics models differ.
4 Explicable Policy Search (EPS)
To present our learning method, we start by expressing the probability distribution of the agent's s trajectories and human's s expectation. We parameterize the agent's 's policy using 0. The probability of realizing the agent's 's trajectory I with Ty (and similarly for the human's 's expectation) is:
pA(T) ((0)) II TA (St+1 18t, ae mo) Pd (T) p(so) II TA (8t+1 (141119;0))7 (acls)). (3)
Given these two distributions, we can now derive our solution for EPS. Since both objectives in Eq. 2 are expressed as expectations over the same distribution of trajectories (i.e., (T)),), we can combine them after rewriting the policy explicability score in Eq. ( as follows:
DKL (Pal)))())
-Epa Σ log TA(s44155) a.) + log (arls)) - log TH (st+1lse, ar) - log of (aelse)
+ +C,
(4)
where C is a constant. The main challenge solving the optimization problem in Eq. ( now lies in the fact that neither the human's policy ли nor her belief about the domain dynamics T are known.
4.1 Surrogate Reward Function
Eq. can be merged into Eq. (2) such that the log terms essentially reshape the engineered reward function. In such a case, a straightforward approach is to learn the human's belief about the domain dynamics and her expected policy separately based human feedback, while maintaining estimates of the true domain dynamics and the current agent's policy. While possible, it is inefficient and unnecessary. Instead, we propose to use surrogate reward function. The goal is learn such function u that retains the necessary and sufficient information about the human's 's belief and the expected policy for policy search, i.e., u = log T ($1+1 8t, a.) log o (ar(s:). Next, we reveal how such a goal can be achieved by manipulating preference-bene based formulation.
Intuitively, we learn reward function that alone explain the human's 's expectation of the trajectories-a a reward function that introduces the same distribution of the expected trajectories. At the same time, we must take care constrain the complexity of human feedhack data to make it practical. A popular approach based on simple human feedback is preference-based RI. [46] 10. For EPS, instead of soliciting human preferred trajectories learn the human's 's reward model, we can present pairs of trajectories and ask humans to comment on which one is more expected. Then, we can fit reward function that prefers generating more expected trajectories. More specifically, correlate the distribution of the expected trajectories with the surrogate reward function as follows:
P (T) x exp Σ un (8c: (BL.
(5)
Proposition 1. There exists unique reward function uy such that the trajectory distribution under the softmax human preference model with described above matches with the human' 's expected trajectory distribution given Eq. (3).
and 5 ). we see that the When we compare mathematically the two equations of p (7) (i.c., Eqs. only way to match the distributions is by satisfying:
Σ uh (St, ai) Σ log ( (sc+118t. + Σ log x (ag(s.) + Ct.
5

---

One straightforward way for this is to set uy (st, ai) = log To (st+1|st, ai) + log (ogls)). In the other direction, since the trajectories may be of various lengths general (including the length of 1 as special cases), we can conclude further that:
un (St: de) = log T (s141 Se, ae) + log A (acls:) + Ct.
Plugging this into any trajectory of length greater than we can conclude that Ci = 0. To learn un we learn to maximize the likelihood of the human data. This is similar to MaxEnt IRL [49] such that the reward function is learned exhibit no preferences beyond matching the human's 's feedback data.
This means that H in such a case becomes equivalent to log T and log тH for representing the distribution of expected trajectories, which implies that the information needed from 7 and тH for optimizing Eq. (2) can be substituted with ug. To learn the unique surrogate reward function and avoid the non-identifaability issue [40), potentially, requires the reward function to be globally optimized over all trajectory pairs, which is not feasible. In our implementation, we simply normalize the learned rewards. Now, we rewrite Eq. 2 based on the above result:
Σ o
( (ra(s), az) + X (se, ae) + HT, [81t1/8e, ar) + Hap (arlaaels)) [aclse))] (6)
6- = arg max E X0
Note that Eq. (6) is an approximation of the original objective in Eq. 2 by applying discounting to the surrogate reward function and entropy terms. This allows us to them to reshape the reward function. We can view this objective function as two parts. The first two terms, (st, az) - Au (St, az), requires the agent maximize rewards from two sources: the engineered reward and the surrogate reward learned from human feedback. These two reward functions are weighted according the reconciliation parameter as expected. The second part of this objective function is HT. [st-1 [st. az] - Hap [z). which are two entropy terms. The first term is for the agent's 's domain dynamics and the second term is for the target policy.
The second entropy term for the target policy (referred to as "policy entropy") is well studied in the maximum entropy (MaxEnt) RL framework. (20 21],, However, note that this term in our work is derived for a completely different reason. In MaxEnt RL, maximizing H., encourages the agent to explore during learning and increase robustness. Given fixed surrogate reward function, this term in EPS likewise encourages stochasticity in the agent's ' policy to increase robustness, A more distinguishing feature of explicable policy search lies in the first term. Maximizing HT, encourages the agent to prefer parts of the environment where there is more stochasticity, which we refer to as "environment entropy". The incentive here is to reduce the influence on the policy explicability score due to differences in the agent's 's policy and the expected policy. Intuitively, at more stochastic parts of the environment where action choices for the agent matter less, the agent's trajectory is more likely to match with the human's 's expectation even when the agent's policy and the expected policy differ.
Expectation-based Preferences: For learning uy, we apply a preference-basee learning frame- work. We request humans to provide their feedback on which segment in a pair of segments } extracted from the agent's trajectories is more expected. To consider noise in human feedback, the human's expectation preference is formulated follows, similar to that in 10
exp (st at) exp Σ un at) exp Σ uH (s7
il0¹ >
0²1
We learn uy to minimize the cross entropy between our prediction of the expectation preferences and feedback data. In order to efficiently solicit the human's 's expectation preference for the agent's ss behavior, we leverage uncertainty- based sampling [10 28 to select traces for the human in an active learning fashion. To estimate TA, we assume it follows a Gaussian distributions and model it using two-headed neural network that outputs its mean and logarithm of standard deviation given the state and action. The agent interacts with the enviromment and collects transition data for learning. TA is estimated using data-driven method by minimizing the L2 one-step prediction loss 26). We use Soft Actor Critie (SAC) 21 for policy learning but other policy search methods are also applicable. To alleviate the issue of non-stationary prediction of expectation preferences during learning, we relabel the data samples every time uy is updated [27]. The algorithm for EPS is in the appendix.

---

Evaluation
We evaluate our method on set of continuous navigation domains with synthetic human models and simulated autonomous driving domain with human subject study. The study is IRB approved and all protocols have been followed. The synthetic experiments are used to validate the effectiveness of our method for searching for explicable policies. The user study is to 1) confirm that our belief's about domain dynamics can be inaccurate or biased, which can affect our judgements and lead to severe consequences if ignored, and 2) show that our method can effectively address such hidden issues by achieving an intelligent reconciliation between task performance and human's expectation.
5.1 Synthetic Navigation Domains
We conduct experiments on four navigation domains with continuous state and action spaces, as illustrated in Fig. 2 More detailed descriptions of our domains are in the appendix. For all the domains, the agent starts from the upper left corner and aims to navigate to the goal area (depicted in green). We introduce various human inaccurate beliefs about domain dynamics for these four domains" For Domain 1 (D1) and Domain 2 (D2) (i.e., the first two rows in Fig. 2). the human believes that the agent is more likely to fall into the pit when it is close by; the truth is everywhere is the same. For Domain 3 (D3) and Domain 4 (D4) (i.c.. the third and fourth row in Fig. 2 the human's belief is that the agent could easily slip on ice. Morcover, the human believes that the agent readily handle sandy roads. The truth is that the agent is proficient on ice but can easily get stuck on sand in D3. For D4, the agent is in addition capable of handling sand albeit being more costly. The human's 's expectation preferences for the agent's behaviors is generated synthetically with respect to the human's 's model.
5.1.1 Baselines
We compared method three baselines illustrate its advantages compared to the traditional RL methods that do not consider model differences: Soft Actor Critic (SAC) [21], Deep RI. from Human Preferences (DRLHP) 110, and Policy Shaping (PS) [8] SAC optimizes policy with respect the engineered reward function and true domain dynamics without considering the human's 's expectation. DRLHP uses human expectation-base function and applies it to guide policy search. It ignores the engineered reward function. For SAC and DRLHP, we could not use reward signals from both sources due to their different formats. The closest competitor to EPS is policy shaping. Policy shaping learns two policies using reward signals from the environment and human expectation preferences, respectively. A combination policy is then obtained by multiplying them together. It also seeks to combine different information sources that are however assumed to consistent. That is why has difficulty handling situations where the human's 's expectation and agent's optimal behavior conflict (see D3 in Fig. 2
preferences for the agent's behavior to estimate reward
5.1.2 Results and Discussion
Table shows the average return and policy ex- plicability score of EPS compared to the base- lines over 100 rollouts, with the sample stan- dard deviations. The policy explicability score computed using Eq. (4) based on the synthetic human models. For EPS, the reconciliation pa- rameter À is tuned manually to show the differ- behaviors compared the baselines in Fig. 2. with À E [2.0, 2.8 For all domains, EPS achieved the highest policy explicability score and followed the best performer for average re- turn. Note that SAC also achieved good explicability scores in the domains since the synthetic human models did not penalize the optimal behaviors much as they could (i.e., the human beliefs were not
(a)
(b) D2
(c) D3 & D4
Figure 3: Visualization of the learned surrogate reward functions for D1-D4. The darker the lower reward value and the brighter the higher.
For simplicity, modified the human's 's belief about the domain dynamics only while keeping the reward functions the same for the two models. This should not impact the evaluation since the surrogate reward function can encode information for both for policy search.
The spectrum of policies generated difference values of À is included in the appendix.
7

---

DI:
D2:
D3:
D4:
(a) Human
(b) EPS
SA)
(d) DRLHP
(c) PS
Figure 2: Comparison of different learning methods with human's 's expectation (left). The dark grey area represents the walls. The brown area is the pit (with -100 penalty) and the green area the goal (with +1 +10 reward). The blue area represents icy roads and the yellow area represents sandy roads.
Avg. (Std)
Avg. Explizahility (Std)
Domain
IPS
SAC
DRLHP
PS
EPS
SAC
DRLIIP
PS
95.53 (1.88)
97.46 (1.53)
74.51 (29.92)
95.91 (3.55)
94.87 (1.95)
92.57 (1.48)
73.01 (28.75)
89.17 (4.82)
D2
94.07 (2.43)
95.65 (1.98)
23.28 (43:71)
95.55 (3.17)
92.10 (2.44)
91.00 (3.15)
20.94 (43,97)
90.66 (3 94)
(1.16)
94.72
(3.32)
-50.79
-181.46 (63.87)
D4
92.37 (1.80)
93.27 (2.03)
26.52 (49.02)
90.11 (04.1
92.17 (1.86)
90.77 (2.44)
26.49 (49.03)
90.1 (16.63)
Table 1: Comparison of EPS to baselines using averaged return and explicability score.
modeled be much stochastic than the ground truth). PS performed well most tasks while being considerably worse than others in D3. Due to the conflict between the human's 's expectation and task priorities in D3, the method failed achieve meaningful combination. DRLHP performed poorly on all the domains because preference-basee domains with sparse or delayed rewards, which makes credit assignment challenging and results in large variations. Since EPS also applies a preference-bart reward function, we demonstrate this using the learned function. As shown in Fig. 3 this function has significant amount of uncertainty so is not ideal for guiding policy search. Although it adds variability into policy learning, serves very well an auxiliary objective by informing the agent where the human expects it to perform.
learning methods have difficulty dealing with
learning method to learn the surrogate
We show sampled trajectories for EPS, the baselines, and the human's expectation computed using the synthetic human models in Fig. 2 In DI and D2, SAC agent always chooses the shortest path while EPS agent takes detour that bypasses the pit and hence chooses the path passing through the top since the agent can get stuck on sandy roads in D3 and the top path more efficient in D4. It is worth noting that EPS agent makes different decisions on these two domains. It selects the top path in D3 when the sandy road is difficult to navigate even though against the human's expectation. This is because if the agent conforms to the human's expectation, it would result in huge loss to the engineered reward. However, when it can better navigate on sand in D4, it chooses to respect the human's 's expectation be more explicable, at the cost of lower task performance. Policy shaping agent is also successful in D4, but gets stuck in D3, because simply multiplying two different policies could result in a new policy that is uninformative, irrespective of the À used (e.g., when we have P1 = (0.1, 0.9), P2 = (0.9, 0.1). the resulting policy from multiplying them would always p = (0.5, 0.5)), which can lead to poor behaviors. In summary, we show that EPS can successfully generate effective behaviors that achieve an intelligent reconciliation between
explicable. In D3 and D4, SAC agent

---

the human's 's expectation and task priorities. It can generate novel reconciled behaviors (D1 and D2), stick the task priorities (D3), or follow the human's 's expectation (D4), as appropriately.
5.2 Autonomous Driving Domain
We used a simulated autonomous driving domain (29] to evaluate our method with human subjects and demonstrate its real-world relevance, as illustrated in Fig. The state space is featured by the position and velocity of the ego-vehicle and nearby vehicles. The action space consists of five discrete actions. Initially, the autonomous driving vehicle (green) is running on the highway with a car in front of it (blue) on the same lane and with the same speed. The task of the autonomous driving agent is to handle situations when the front car slows down quickly and abruptly. One common type of cognitive bias is the availubility bias, which reflects humans' tendency to overestimate the likelihood of events with greater availability in memory. Such biases can occur driving since we regularly drive under familiar conditions. To design an experiment where such biases are present, we considered scenarios with regular driving condition and sleety condition where the vehicle's domain dynamies became more stochastic due to slippery roads. With the availability bias, however, humans likely to make the same decisions under these similar but in actuality different conditions, leading to safety risks.
The user study consists of two phases: training and testing. At the beginning of the training phase, we requested the participates to provide the importance ratings for several factors governing the autonomous driving behavior [9] in a 5-point Likert scale, such as average speed (3.42), distance to the front car (4.57), relative speed to neighboring cars (3.28), and lane following (3.78). The average participants' responses are shown in the parentheses above. Their responses served as the engineered reward model (rx)) and are linearly combined. For example, the distance to the front car directly associated with collision and hence given larger weight. Before human data collection, we also presented the participants with the vehicle's 's behavior after braking under the regular condition when the front car slows down suddenly. Then, given the information that it is driving on sleety day, we selected segments that showcased different vehicle behaviors after braking (i.e., braking behaviors under different effectiveness). The participants were asked to select which one matched their expectations the most. Their responses were used to validate the availability biases in this experiment. Then, we collected human data by actively selecting pairs of trajectory segments for the participants to compare with. The collected data was used to train our method and the baselines. Then, we showed rollouts of the learned policy for each method new participants and asked them to rate those rollouts in the testing phase.
5.2.1 Results and Discussion
We published the experiments on Amazon Mechanical Turk (MTurrk). To sift out invalid responses, a sanity check demonstration was added that showed a collision, which should not be preferred under any situation. We recruited 15 participants for training, and one failed the sanity check. For the bias validation question, 12 out of 14 valid participants chose that the ego-vehicle would slow down effectively on a sleety day when braking. This reflects the availability bias that the participants had, which can lead to collisions on a slippery road if ignored. We also noticed that the participants preferred to brake than steer in general during training, which accords with the participants responses that attached a high importance rating to distance to the front car and lane following.
The testing phase occurs on a sleety day. The behavior of EPS, SAC, and DRLHP agents are illustrated in Fig. 4 SAC agent steers immediately when the front car slows down. It is the most efficient behavior with the most return on slippery road since braking would not be effective. DRI.HP agent, on the other hand, chooses to brake while staying in the same lane. Such a behavior, however, is more likely to lead to a collision (risk to human passengers). As we can see from Fig. 4 the agent is getting dangerously close to the vehicle in the front. EPS agent first chooses to brake (to be explicable), and then steers to the other lane (to stay safe and continue moving forward), which maintains both explicability and task efficiency. We demonstrated these rollouts in the testing phase to new participants. We informed the participants that the vehicle was running on a sleety day. Each participant was required to provide ratings to all the demonstrations ranging from 1 to 10. We obtained 11 valid responses. Interestingly, the participants rated EPS agent the most preferable, followed by DRLHP and SAC agents. The average rating for each agent and its standard deviation are shown in Fig. We note that the results contradict with the importance ratings solicited for the engineered reward model, which should have led to preferring the steering behavior as chosen by the
9

---

(a) EPS (y 7.6, = 2.4)
(b) SAC (j 5.0, 2.3)
(c) DRLHP (p
5.9, = 2.2)
Figure 4: Agents' behaviors in the autonomous driving domain illustrated in three characteristic steps from the top to bottom for each agent. The agent's ' name is followed by its average rating and standard deviation. We set the significance level the one-way ANOVA test on the data. The result [F 30) = 3.665, p = 0.037] shows significant difference among the three groups. We conducted the Shapiro-WWile and Levene's tests for checking the normality and homoscedasticity assumptions, respectively. The results show that our data satisfies the normality (33) 0.948, p p= 0.119] and homoscedasticity (F, 0.168, p 0.846] assumptions of one-way ANOVA [42].
0.05 for all statistical tests. We performed
SAC agent on slippery roads. This further suggests that the participants had a biased belief about the vehicle's dynamics (i.e., braking is efficient under the regular condition under the sleety condition). Regardless, our agent chose the behavior that was both explicable and safe.
Discussions and Conclusions
In this paper, we introduced and formulated the problem of explicable policy search that considers model differences between the learning agent and its human observer. We developed efficient solution by learning a surrogate reward function that was then used to recover an explicable policy. Our method significantly extends explicable planning stochastic environments an RL setting with continuous state and action spaces. We evaluated in simulations and with human subjects. Results showed that our approach could better handle situations under model differences than several baselines and thus contributed a critical tool to achieving explainable human-agent interaction.
Limitations and Future Work. The assumptions underlying work include that the human generates expectation based on her objective and belief about the domain dynamics a noisily rational way, and the human has softmax preference model. While such assumptions in prior work, it would be useful to consider situations where such assumptions do not hold, e.g., when the human is biased in the computational model. Furthermore, belief about the domain dynamics does not easily update as observations of the agent are made. For scenarios where inaccuracies stem from intrinsic cognitive biases that are difficult change, this is a reasonable assumption. In other cases where discrepancies were due to, e.g., information asymmetries, the belief can change dynamically and needs to be actively monitored 222. Other possible directions include more complex (such as state dependent) reconciliation of the expected return and policy explicability score, as well weighted policy explicability scores incorporate trajectory similarity as considered in prior work 48 25
commonly made
assume that the human's 's inaccurate
Our approach covers many model inevitably different from the ground truth in real-world scenarios. As a result, EPS may sometimes cause the user preferences to be ignored, which may be "unexplainable" but is necessary cost to pay in many scenarios. In particular, EPS combines the surrogate reward with an engineered reward from the agent's designer maintain critical features (e.g., safety) while respecting user preferences whenever feasible. Note that the combination of such two reward functions should not he replaced by the user's reward even if can be perfectly estimated due to complexities such human biases and misunderstandings of the domain. However, real-world deployment of EPS would require more work, such as an interactive design that explains to the user why her preferences are ignored.
applications since the human understanding of the agent's
Societal Impacts. Our research fundamental in nature without notable negative societal impacts. However, it is conceivable that approach may be used to misguide RL agents to ignore human preferences in adversarial settings. Further investigation needed to understand such a possibility.
Acknowledgement, This research is supported in part by the NSF grants 1844524, 2047186 86, the NASA grant NNX17AD06G, and the AFOSR grant FA9550-18- 1-0667 The authors would also like to thank the anonymous reviewers for their helpful comments and suggestions.
10

---

References
[1] Abbeel and Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICMI, 2004.
[2] C. Baker, R. Saxe, and Tenenbaum. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
[3] D. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. International conference on machine learning, pages 783-792. PMLR, 2019.
[4] D. S. Brown, W. Goo, and S. Niekum. automatically-ranke demonstrations. Conference on robot learning, pages 330-359. PMLR, 2020.
imitation learning via
[5] D. S. Brown, J. Schneider, A. Dragan, and S. Nickum. Value alignment verification. In International Conference on Machine Learning, pages 1105-1115. PMLR, 2021.
[6] T. Chakraborti, S. Kambhampati, M. Scheutz, and Y. Zhang. Ai challenges in human-robot cognitive teaming. arXiv preprint arXiv:1007.04775. 2017.
[7] T. Chakraborti, A. Kulkarni, S. Sreedharan, D. E. Smith, and S. Kambhampati. Explicability? legibility? predictability? transparency? privacy? security? the emerging landscape of interpretable agent behavior. In ICAPS, 2019.
[8] T. Chakraborti, S. Sreedharan, Y. Zhang, and S. Kambhampati. Plan explanations as model reconciliation: moving beyond explanation as soliloquy. In IJCAI, 2017.
[9] E. Cheung, A. Bera, E. Kubin, K. Gray, and D. Manocha. Identifying driver behaviors using trajectory features for vehicle navigation. In IROS, 2018.
[10] Christiano, Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
[11] C. Daniel, M. Viering, J. Metz, O. Kroemer, and J. Peters. Active reward learning. In RSS, 2014.
[12] A. D. Dragan, K. C. Lee, and S. Srinivasa. Legibility and predictability of robot motion. In 2013 8th ACM/IEEE International Conference Humun-Robot Interaction (HRI), pages 301-308. IEEE, 2013.
[13] B. Erdem, M. Palan, user-friendly approach to active reward learning. In CoRL, 2020.
Landolfi, D. P. Losey, D. Sadigh, et al. Asking easy questions:
[14] B. Eysenbach, Asawa, S. Chaudhari. S. Levine, and R. Salakhutdinov. Off-dynamics reinforce- ment learning: Training for transfer with domain classifiers. arXiv preprint 21t::2000.139616 2020.
[15] M. Fox and D. Long. Pdd12. 1: An extension to pddl for expressing temporal planning domains. Journal of artificial intelligence research, 20:61-124, 2003.
[16] Z. Gong and Zhang. Behavior explanation intention signaling in human-robot teaming. In RO-MAN, 2018
[17] Z. Gong and Y. Zhang. What is it you really want of me? generalized reward learning with biased beliefs about domain dynamics. In AAAI, 2020.
[18] S. Griffith, K. Subramanian, J. Scholz, C. L. Isbell, and A. Thomaz. Policy shaping: integrating human feedback with reinforcement learning. In NeurlPS, 2013.
[19] D. Gunning. Explainable artificial intelligence (xai). DARPA, 2017.
[20] Haarnoja, H. Tang, Abbeel, and Levine. Reinforcement learning with deep energy-based policies. In ICML, 2017.

---

[21] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 18X1:18001.12200 2018.
[22] A. Hanni and Y. Zhang. Generating active explicable plans in human-robot teaming. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2993-2998. IEEE, 2021.
[23] M. Herman, T. Gindele, Wagner, F. Schmitt, and W. Burgard. Inverse reinforcement learning with simultancous estimation of rewards and dynamics. In AISTATS, 2016.
[24] W. B. Knox and P. Stone. Interactively shaping agents via human reinforcement: The tamer framework. In K- K-CAP, 2009.
[25] A. Kulkarni, Y. Zha, T. Chakraborti, S. G. Vadlamudi, Y. Zhang, and S. Kambhampati, Explica- ble planning as minimizing distance from expected behavior. In AAMAS, 2019.
[26] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. In ICLR, 2018.
[27] K. Lee, L. Smith, and P. Abbeel. Pebble: Feedback- interactive reinforcement learning via relabeling experience and unsupervised pre-trate training. JCML, 2021.
[28] K. Lee, L. Smith, A. Dragan, and Abbeel. B- B-pref: Benchmarking peffrence-basee reinforce- ment learning. arXiv preprint arXiv:211 1.03026, 2021.
[29] E. Leurent. An environment for autonomous driving decision-making, https://github. com
gleurent/highway--eeee env 2018.
[30] Y. Li, J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. Advances in Neural Information Processing Systems, 30, 2017.
[31] C. Lucas, T. Griffiths, F. Xu, and C. Fawcett. A rational model of preference learning and choice prediction by children. Advances in neural information processing systems, 21, 2008
[32] J. MacGlashan, M. K. Ho, R. Loftin, B. Peng, G. Wang, D. L. Roberts, M. E. Taylor, and M. L. Littman. Interactive learning from policy-dependent human feedback. In ICML, 2017.
[33] M. S. A. Nahian, S. Frazier, B. Harrison, and M. Riedl. Training value-aligned reinforcement learning agents using normative prior. arXiv preprint arXiv:2104, 09469, 2021.
[34] M. Oaksford and N. Chater. Bayesian rationality: The probabilistic approach to human reasoning. Oxford University Press, 2007.
[35] R. Paleja, Y. Niu, A. Silva, C. Ritchie, S. Choi, and M. Gombolay. Learning interpretable, high-performing policies for continuous control problems. arXiv preprint arXiv:2202.02352. 2022.
[36] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In IJCAI, 2007.
[37] S. Reddy, A. D. Dragan, and S. Levine. Where do you think you're going? inferring beliefs about dynamics from behavior. In NeurlPS, 2018.
[38] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-21) 215, 2019.
[39] C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1-85, 2022.
[40] Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pages 101-103, 1998.
[41] D. Sadigh, A. Dragan, S. Sastry, and S. A. Seshia. Active preference-basee learning of reward functions. In RSS, 2017.
12

---

[42] M. L. Schrum, M. Johnson, M. Ghuy, and M. C. Gombolay. Four years in review: Statisti- cal practices of likert scales in human-robot interaction studies. In Companion of the 2020 ACM/IEFF International Conference Human-Robot Interaction, pages 43-52, 2020.
[43] R. Shah, N. Gundotra, P. Abbeel, and Dragan.
On the feasibility of learning, rather than assuming, human biases for reward inference. In ICML, 2019.
[44] A. Silva, M. Gombolay, T. Killian, I. Jimenez, and S.-H. Son. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference on artificial intelligence and statisties, pages 1855-1866 PMLR, 2020.
[45] H. A. Simon. Bounded rationality. In Utility and probability, pages 15-18. Springer, 1990.
[46] C. Wirth, R. Akrour, G. Neumann, J. Fürnkrazz, al. A survey of preference-based reinforce- ment learning methods. JMLR, 2017.
[47] M. Zakershahrak, A. Sonawane, Z. Gong, and Y. Zhang. Interactive plan explicability in human-robot teaming. RO-MAN, 2018.
[48] Y. Zhang, S. Sreedharan, A. Kulkarni, T. Chakraborti, H. H. Zhuo, and S. Kambhampati. Plan explicability and predictability for robot task planning. ICRA, 2017.
[49] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A] You strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL))? [Yes] See supplemen- tal material.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See results in table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See supplemental
material.
(b) Did you mention the license of the assets? [Yes] See supplemental material.
(c) Did you include any assets either in the supplemental material URL? [Yes] See supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See supplemental material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] No such info or content was used eurated.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See supplemental material.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] See supplemental material.
14


Reviews:
[{"review_id": "sGx---8VEvV", "reviewer": "Reviewer_5GPj", "summary": "This paper presents a way to make policy search \"explicable\" to humans, by eliciting their preferences over trajectories and combining a conventional \"engineered\" reward function with those trajectory preferences. The method is validated in empirical experiments as well as a user study in a small autonomous driving example.\n", "questions": "1) Is the actual contribution here the more efficient way of estimating the human beliefs over dynamics + policy by adding them together into a \"surrogate reward function\" directly learned by preference learning over trajectories, or the concept itself of using a linear combination learned preferences from [25,7] with a regular (given) RL reward function, or both? Please clarify positioning vs. earlier work in RL.\n\n2) Shouldn't the entropies in Eq. 6 be outside the expectation over p_A? Is this just a typo?\n\n3) Estimating the entropy over the true dynamics seems potentially very difficult in the general case for more complex and higher-dimensional state distributions. It is just briefly mentioned that this is done by learning the true dynamics via a NN with a Gaussian head. For more complex state distributions than narrow trajectory distributions, I wonder how reliable this would be. Any error here seems like it could also affect tuning of lambda.\n\n4) How to set lambda? The first part of the objective is a reward, the second part is a messy approximation to a KL divergence. It is not obvious to me that even if you elicit trajectory preferences from people that making a good trade off here in the end won't still require human input.\n", "limitations": "Sufficiently adequate (pending questions above).", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Appears to be a novel niche in a relevant problem.\n- Inclusion of a user study is a nice addition to this kind of paper.\n\nWeaknesses:\n- The initial motivation of the approach is weak and confusing. The positioning against related work in RL could also be better. It wasn't clear until the end of the paper that this might be a valid (if rather narrow) problem and that the approach might work. \n- The mathematical derivation seems needlessly complex and seems to have a typo or mistake (see below). The technical contributions do not appear very significant.\n\nDetails:\n- Motivation: The \"explicable\" solution in the motivating example (Fig 1a). is actually inexplicable to me. Since the proposed approach attempts to plan w.rt. wind, I assume the wind speed is constant (non-constant wind gusts are almost unpredictable and hence unplannable). Having worked with planning for drones, if the wind is constant, but less than the control authority of the drone (here drawn as a quadrotor), it would deviate some depending on the type/parameters of the controller,  strength of wind, and length of the trajectory. However, without knowing those variables, either of the drawn scenarios could happen. In addition, the \"expected\" dashed line is not straight either but curves in the opposite direction? This example could use a rework as the supposedly explicable and optimal solutions do not appear more explicable or optimal respectively.\n\n- As you appear to use the preference estimation machinery from [25,7], you might want to compare your work against those in the related work. Other RL works have also combined conventional rewards with a KL-divergence objective on trajectories (e.g. variants of Guided Policy Search). Here you are essentially learning this distribution by human preferences learned elicited as in [25,7]?\n\n- The mathematical derivation in 4.1 seems needlessly complicated, you already defined u_H on line 172 to be identical to the log of the human trajectory distribution, so Prop 1 and its proof seems redundant (c.f. e.g. line 172 vs. 188-189).\n\n- Eq. 6 does not appear to be a reformulation of Eq. 2 as claimed. As the authors note they suddenly made the KL-divergence term use the RL discount factor, but more importantly, if you convert the KL-divergence terms from (2) into entropies, shouldn't they be outside the expectation over p_A? \n\n- Eliciting reward functions from human observers appears like a slow process, and the empirical benchmarks are all very small toy examples. \n\n- The autonomous driving user study is a nice inclusion that perhaps should have been given more attention earlier on. \n\nMinor: \n- l46: calling it \"business\" seems informal\n- l53: \"we assume that the human is nosily rational at generating her expectation of the agent to accommodate her computational limitation\" - noisily rational? I am not familiar with this particular model of rationality, it might be good to explain what this means.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hgbfbH7mpw8", "reviewer": "Reviewer_TX5X", "summary": "This paper develops both a formulation and solution to the problem of what the authors term Explicable Policy Search (EPS), i.e. how to compute policies that maximize a tradeoff between task reward and how expected (or \"explicable\") a policy is to a human teammate, in order to improve human-robot teaming. The authors formalize this as an objective that combines expected cumulative reward with an \"explicability score\" for the robot's policy, which is defined as the KL divergence between the robot's policy and the policy that would be expected under the human's model of the environment and the reward function they impute to the robot.\n\nNaively, solving this problem would require access to or inference of the human's model of the environment, and the policy they expect the robot to perform. Instead, the authors show that this latent information can be encoded into an auxiliary reward function, which can be learned from human pairwise comparisons about whether some robot trajectories are more expected than others. Solving the modified MDP with these auxiliary rewards via standard RL methods yields policies that achieve high reward while being \"explicable\". The authors demonstrate this on a continuous RL test domain using simulated human data where each \"human\" has a different belief about the domain dynamics, as well as a self-driving car domain with real human feedback. The EPS algorithm outperformed non-\"explicable\" baselines in terms of the explicability score while still achieving a high amount of cumulative reward, and generated self-driving car trajectories that were more preferred by human raters.", "questions": "1. As highlighted above, why call formalize \"explicability\" as negative KL-divergence, when that more directly captures \"expectedness under the human model\"? It'd be great if the authors could do more to explain this.\n\n2. Per the discussion above, have the authors considered using negative cross-entropy $E_{p_A(\\tau)}[\\log p_A^H(\\tau)]$ as an explicability score, instead of negative KL-divergence? It's not obvious to me that the incentive to maximize environment entropy or policy entropy (Lines 208-215) is always desired, since this means the agent's behavior will be more unpredicatable!\n\n3. On Line 130, setting the true domain dynamics $\\mathcal{T}_A$ to be unknown seems unnecessarily strong a requirement. Surely explicable policy search can be applied in the case where the domain dynamics known as well -- you could just use model-based RL / value iteration instead of model-free methods like SAC.\n\n4. In equation 6, why approximate the original objective by discounting the surrogate reward $u_H$, along with the other entropy terms? It seems like you could still treat them as additional non-discounted rewards, as long as you're optimizing over finite horizons.\n\n5. Also, the phrase \"ignoring the influence of the discount factor on the surrogate reward function and entropy term\" on Lines 197-198 was confusing to me -- aren't you doing the opposite, by applying discounting to the surrogate reward?\n\n6. Are the standard deviations shown in Table 1 taken over the full set of 100 rollouts (i.e. the sample standard deviation)? Or is that the standard error of the mean? I ask because if it's the latter, then the higher explicability of EPS relative to SAC wouldn't be statistically significant. (No doubt, average-case behavior isn't everything, but is still important).\n\n7. Relatedly, I'm surprised that SAC does so well in terms of explicability score on average. Is there a reason why this is the case? Is it just because in most case, the most optimal behavior happens to be mostly explicable?", "limitations": "Authors address a number of limitations which are relevant to extending EPS to real settings where human beliefs about the world may change. Another limitation comes to mind is that EPS is currently restricted to the case where agents have a fixed goal known to the human. How to extend EPS to the case where the agent's goal is not predetermined, and the human may have uncertainty with respect to the agent's goal. Can some combination of explicable and legible planning be used here, and how?\n\nAs for potential negative societal impact, the authors claim their work in fundamental, but in my mind it is fairly close to being applied (in e.g. human robot collaboration contexts). I would suggest adding a few words about potential negative impacts from failing to adequately learn and encode the human model through the surrogate reward, or the downsides of trading off too much performance or safety for explicability (which seems to arise in the autonomous car driving scenario).\n\n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper builds upon prior work on \"explicable\" and legible planning by introducing a clear and intuitive formulation of \"planning in a way that is expected by humans\" in the context of stochastic domains where autonomous agents compute policies (distributions over actions given states) and not just plans (ordered sequences of actions). Given that most real-world domains are in fact stochastic, this extension of explicable planning is undoubtedly a useful and significant one.\n\nThe mathematical insight used to solve this problem -- showing that human expectations can be encoded as auxiliary rewards -- is also sound, original, clearly explained. The resulting EPS algorithm is well-evaluated on the whole, with clear qualitative demonstrations of how EPS-computed policies lead to more expected behavior given human assumptions about the domain, a user study, and average quantitative performance that is higher in terms of explicability score than a number of baselines (though I have questions about the statistical significance and effect size).\n\nThe main concern I have about this paper is surrounding the term \"explicable\", and how \"explicability\" is formalized as a (negative) KL-divergence, even though negative KL is better understood as something like \"expectedness\". I understand that this is similar to how the term \"explicable\" has been used in prior work in planning, but I initially found it confusing and poorly chosen, because \"explicable\" evokes the term \"explainable\", and it's not at all obvious how minimizing KL-divergence (or difference from expected behavior in general) makes behavior more explainable.\n\nUpon further thought however, I do think there *is* an intuitive sense in which the term \"explicable\" means something like \"minimize KL divergence\", but it requires a bit of justification / explanation: Typically, when we have a (probabilistic) model of some observations or behavior, we say that the observations are \"well-explained\" by the model when the observations are assigned high-probability by the model (i.e. their log-likelihood is high). This sort of language is especially apt when the model in question is causal model of the world, and so assigning high probability to the observations corresponds to having found a good causal explanation of the data. In contrast, when the observations have low probability, we might say that they are \"inexplicable\" with respect to the model.\n\nIn the context of EPS, the model in question is the human's intuitive causal model of how the world works $\\mathcal{T}_A^H$, and how an approximately rational agent would act in that world to achieve its goals. And so, if a trajectory $\\tau$ pursued by an agent has high-probability under the human's model, it is \"well-explained\" under the model, hence \"explicable\". Otherwise it is \"inexplicable\". Hence, the log-likelihood of an agent's trajectory $\\tau$ under the human's model $p_A^H(\\tau)$ can be treated as an explicability metric.\n\nNow as the authors point out, we don't just want to evaluate specific agent trajectories or plans, but agent policies $\\pi_A$, which (when combined with the true environment dynamics) are distributions over trajectories. This motivates taking an expectation over this distribution over how explicable each trajectory is:\n  - If the explicability of a single trajectory is defined as $\\log p_A^H(\\tau)$, then expectation is the negative cross-entropy $E_{p_A(\\tau)}[\\log p_A^H(\\tau)]$ of the human's model over trajectories, relative to the true model.\n  - If the explicability of a single trajectory is instead defined as the log odds ratio $\\log \\frac{p_A^H(\\tau)}{p_A(\\tau)}$, then the resulting expectation is the negative KL divergence $-D_{KL}(p_A(\\tau)||p_A^H(\\tau))$.\n\nEither of these gives us a way to evaluate how explicable a policy is relative to the human's model. It's interesting to consider the differences -- and I'd be curious what the authors think about using cross-entropy instead -- but I think the more important point is that this sort of justification for using the term \"explicable\" is missing from the paper as currently written. I would highly encourage the authors to reframe how their explicability score is introduced and defined, in order to incorporate something like the above. Otherwise, the conceptual gap between \"expected by the human\" and \"explicable\" is not an obvious one for readers to bridge. In case helpful, [1] is a paper that connects the quantity of log-likelihood to the concept of \"inexplicability\", which is formalized alongside other measures of human surprise.\n\nApart from the above, I only have a number of other minor questions and suggestions, which I'll list in the next section.\n\n[1] Zhi-Xuan, T., Gothoskar, N., Pollok, F., Gutfreund, D., Tenenbaum, J. B. & Mansinghka, V. K.  (2022). [Solving the Baby Intuitions Benchmark with a Hierarchically Bayesian Theory of Mind.](https://social-intelligence-human-ai.github.io/docs/camready_13.pdf) Social Intelligence in Humans and Robots, RSS 2022 Workshop.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ds_DLOQM3Re", "reviewer": "Reviewer_VqJ9", "summary": "This paper poses a problem for “explicable policy search,” which is defined as the problem of developing a planner (i.e., via reinforcement learning) that produces behavior that are aligned with the expectations of a human observer. The paper develops an approach that infers the human’s latent objective function through preference-based RL (though a Behavior Cloning objective is described first). The paper evaluates the approach in a computational environment with synthetic humans showing promising results. The paper then conducts a human-subject experiment showing that human’s subjectively rate the proposed approach higher than baselines.", "questions": "-Line 14, are the human’s expectations truly “hidden” or just partially observable, latent random variable?\n-How is proposition 1 fundamentally different than claims of Maximum Entropy Inverse Reinforcement Learning for learning a unique reward function? How does Proposition 1 eliminate the reward ambiguity problem?\n-Could the authors provide more information about why EPS did not conform to the human’s behavior in D3?\n", "limitations": "- The authors explicitly have a section on limitations and future work, which is helpful. \n- The paper states that the research is “fundamental in nature without notable negative societal impacts.” I appreciate the response, but the response does not seem to be in-line with the expectations for NeurIPS. At one point, GANs might have seemed to be fundamental research, but Deep Fake videos offer the world a tool for disinformation. I would recommend spending additional time pondering whether explicability in planning – as defined here – has benefits and potential harms to society and whether the benefits outweigh the harms. For the record, this reviewer believes the research is ethical. The reviewer is only asking for a more full attempt to fulfill the expectations of NeurIPS.\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "trengths:\n+ The paper’s writing is generally clear with some grammatical issues. The figures were helpful.\n+ The topic is timely, as researchers express increasing interest in explainable artificial intelligence in general. Policy explicability is an under-studied topic.\n+ The problem formulation is clear, and the approach is well-reasoned.\n+ The computational results (Table 1) show competitive returns and superior explicability scores.\n+ The visualization in Figure 2 helps provide insight into the behavior of the planners.\n+ The paper conducts a human-subject experiment to evaluate the approach, showing that it outperforms baselines with statistically significant results. The study was IRB-approved.\n\nWeaknesses:\n-Line 85 states that the surrogate reward function encodes the “necessary information.” Is that true? Is the information learned truly “necessary” or are components of the surrogate reward only approximately correct, superfluous, and/or incongruent with the truly-necessary information? What would the sufficient information be?\n-In this paper, the term “explicable” in regards to planning is defined as a plan that aligns with the human’s expectations for what a plan should look like. This is but one form of explicability, or, colloquially, explainability. For example, the paper seems to miss prior work on value-aligned reinforcement learning, such as:\n\nNahian, M.S.A., Frazier, S., Harrison, B. and Riedl, M., 2021. Training value-aligned reinforcement learning agents using a normative prior. arXiv preprint arXiv:2104.09469.\n\nBrown, D.S., Schneider, J., Dragan, A. and Niekum, S., 2021, July. Value alignment verification. In International Conference on Machine Learning (pp. 1105-1115). PMLR.\n\nThe value-aligned RL work seems to be highly similar in idea: creating RL agents that align with human expectations. The paper is also missing prior work on interpretable reinforcement learning, such as: \n\nSilva, A., Gombolay, M., Killian, T., Jimenez, I. and Son, S.H., 2020, June. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International Conference on Artificial Intelligence and Statistics (pp. 1855-1865). PMLR.\n\nPaleja, R., Niu, Y., Silva, A., Ritchie, C., Choi, S. and Gombolay, M., 2022. Learning interpretable, high-performing policies for continuous control problems. arXiv preprint arXiv:2202.02352.\n\nWhile these other approaches are not “explicable” in the sense that they do not align with a human’s expectations, they are interpretable, i.e. directly “explainable.”\n\nIt would have been helpful to include the Brown et al. (2021) method as a baseline and argue why explicable behavior is better than interpretable behavior (relatively to Silva et al., 2020), considering the position of Rudin (2019).\n\nRudin, C., 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), pp.206-215.\n\nRudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L. and Zhong, C., 2022. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16, pp.1-85.\n\n-The approach described in Equation 1 seems quite similar to the mechanisms explored in InfoGAIL (Li et al., 2017). In their paper, Equation 4 shows a discriminator loss for a generative-adversarial approach to essentially Behavior Cloning/Dagger, an entropy bonus, a mutual information term, and the reward function. If we throw out the entropy bonus and mutual information term (which is used for aligning with heterogeneous humans), then we basically recover what we have in Equation 2 of this paper: one term for the expected return (or reward function) and one for aligning with human behavior (behavior cloning).\n\nLi, Y., Song, J. and Ermon, S., 2017. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems, 30.\n\n- Following up on the previous point, it is fair to note that replacing the divergence term in Equation 2 and arriving at Equation 6 is literally a different formulation (e.g., replacing imitation learning with a preference-based RL, exponential distribution-type reward function inference setup) than in InfoGAIL. However, the improvement seems incremental to replace the KL divergence with U_H, and it would have been helpful to acknowledge this connection.\n\n- Further, the inclusion of entropy (as described in Lines 205-207 further recreates the setup from InfoGAIL without attribution. \n\n- The equation in referenced from [7] goes back at least a decade further to Lucas et al. (2008) and was leveraged again by Brown et al. (2019 and 2020) for preference-aligned RL.\n\nLucas, C., Griffiths, T., Xu, F. and Fawcett, C., 2008. A rational model of preference learning and choice prediction by children. Advances in neural information processing systems, 21.\n\nBrown, D., Goo, W., Nagarajan, P. and Niekum, S., 2019, May. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning (pp. 783-792). PMLR.\n\nBrown, D.S., Goo, W. and Niekum, S., 2020, May. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning (pp. 330-359). PMLR.\n\n- The design of the Likert scales and analysis could be improved. For example, results of test for the ANOVA assumptions (e.g., normality and homoscedasticity) would be helpful. A useful reference would be Schrum et al. (2020).\n\nSchrum, M.L., Johnson, M., Ghuy, M. and Gombolay, M.C., 2020, March. Four years in review: Statistical practices of likert scales in human-robot interaction studies. In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (pp. 43-52).\n\nWriting:\n-There are two uses of the word “it” in Line 4 that may refer to different subjects. Please try to avoid using pronouns with ambiguous antecedents.\n-“behavior, due” does not need a comma\n-Line 16, “they” is ambiguous\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YrlpZEXW6hi", "reviewer": "Reviewer_PBHQ", "summary": "The authors propose Explicable Policy Search (EPS), a method that aligns the behavior of an RL agent with the one expected by a human observer. More precisely, the goal of the method is to learn a policy that reconciles between maximizing the long-term return and minimizing the deviation from the expected behavior. This is achieved by considering a linear combination of two obectives, namely the cumulative reward and a policy explicability score. Since the human's belied about the domain dynamics is hidden to the agent and cannot be queried, the authors demonstrate that it is possible to use a surrogate reward function that encodes the information needed for EPS.\n", "questions": "* The paper mentions that the actions space in the authonomous driving domain consists of 5 actions, but the actions are only reported in the Appendix. I think it would be helpful to the reader if these actions where listed in the main paper in order to better understand the task.", "limitations": "The authors properly described the limitations.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n\nOverall, the paper describes a nice and interesting work.\nI found the paper very well written and easy to follow. The content is presented in a clear and effective way. \n\nThe method is sound, simple and elegant.\n\nThe paper provides a nice formalism to extend explicable planning to a RL setting with continuos state and action spaces.\n\n\n#### Weaknesses\n\n\nThe paper is interesting but in the current version the experimental evaluation is a bit lacking. Experiments are mainly performed on synthetic tasks, which are good for a preliminary assessment of the soundness and feasibility of the method, but should be complemented with more analyses. The study with human subjects is interesting but it is only performed on one toy task.\n\nMoreover, the evaluation assesses the deviation from the human's expected behavior in Table 1 based on the explicability score. However, it looks obvious that EPS achieves a better explicability score compared to the other approaches as it is designed to optimize the explicability score. I understand that it is interesting that the averaged return is still high compared to the baselines, but it would be more interesting to perform a human evaluation of the explicability of the approaches.\n\nOverall, I think the paper would benefit from more experiments on real-world data, but even more ablation studies would be interesting. For instance, the appendix includes an experiment on the choice of $\\lambda$ for the synthetic domains. I think these experiment could be in the main paper and it would be nice if the paper could include a similar experiment for the autonomous driving task too. Also, more experiments aimed at evaluating the role of the environment entropy (as described in lines 210-215) would be interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3TaHhTw-T_l", "reviewer": "Reviewer_gaQq", "summary": "The paper proposes an approach for incorporating human preferences into policy search by shaping the 'original' reward function with a 'preference-based' reward function for the human (other than learning the user belief via a respective policy and transition model). The authors learn the human preference-based reward based on available techniques. The approach is evaluated on environments with cont. state and action spaces (navigation & autonomous driving) based on both synthetic human preferences as well as a user study. The results show that the approach works better than pure RL algorithms as well as a preference-based approach.", "questions": "* How does the approach related to multi-objective RL? E.g. Yang, R., Sun, X. and Narasimhan, K., 2019. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in Neural Information Processing Systems, 32.\n\n* How would the system perform against more recent preference-based RL approaches, improving on the used baseline? Could one (with confidence) state that even with close to optimal approximation of the user reward function, the systems would benefit from the combination with the \"original\" reward? ", "limitations": "The central limitation which is discussed deals with: The argumentation for learning a user-specific reward function for shaping states that it is unnecessary to learn the user-dependent transition function. Is this always the case? What is the underlying assumption for this? The discussion is surely userful. \n\nIt would also be good to discuss the effects of such a straightforward combination of both reward functions. What kind of real world scenarios would this cover? Under which assumptions can a system disregard user preferences (e.g. when safety is incorprated via preferences)?\n\n#### Update after author response & discussion\nAgain, I thank the reviewers for their answers and changes to the paper. After reading the answers, comments of the other reviewers and especially the changes to the paper, I am wiling to increase my score to above the acceptance threshold. Main concerns have been addressed and approriate changes are in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The topic of the paper is relevant and up-to-date, focussing on including user feedback into RL while taking into accont possible misjudgments of the end-users when giving preferences. The taken approach is sensible, shaping the agent's reward function with the user-dependent reward function. The emprical evaluation comprises a user study, which is very helpful for interactive ML.\n\nWeaknesses: While the evaluation is insightful, it does not compare to more recent (and also already referenced) work on interactive RL, e.g. [24] of the paper. Also, it would significantly help the presentation to include the algorithm (available in the suppl. material) into the main paper. Lastly, I am wondering if such combination of agent- and user- reward function is helpful for the general advancement of RL. It feels like the human should remain in control when configuring the system (which seems to be the poinf of explainable, interpretable, safe RL research), but possibly convinced to change her/his perception. So wouldn't a better way be to incrementally suggest improvements to a policy learned via preference-based RL? Maybe this is equal to the direction mentioned in the future work section (pointing to more complex combinations of both reward functions), but it might also be necessary part to generate value.\n\nBesides, a relevant related work (more into the direction of preference-based RL) is also: Reddy, S., Dragan, A., Levine, S., Legg, S. and Leike, J., 2020, November. Learning human objectives by evaluating hypothetical behavior. In International Conference on Machine Learning (pp. 8020-8029). PMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept