Generating High-Quality Explanations for Navigation in Partially-Revealed Environments
Authors: Gregory J. Stein
Venue: NeurIPS 2021 Poster (2021)
Keywords: Explainable Planning
Navigation Under Uncertainty
Explainable Artificial Intelligence (XAI)

Abstract: We present an approach for generating natural language explanations of high-level behavior of autonomous agents navigating in partially-revealed environments. Our counterfactual explanations communicate changes to interpratable statistics of the belief (e.g., the likelihood an exploratory action will reach the unseen goal) that are estimated from visual input via a deep neural network and used (via a Bellman equation variant) to inform planning far into the future. Additionally, our novel training procedure mimics explanation generation, allowing us to use planning performance as an objective measure of explanation quality. Simulated experiments validate that our explanations are both high quality and can be used in interventions to directly correct bad behavior; agents trained via our training-by-explaining procedure achieve 9.1% lower average cost than a non-learned baseline (12.7% after interventions) in environments derived from real-world floor plans.

Generating High-Quality Explanations for Navigation in Partially-Revealed Environments
Gregory J. Stein Department of Computer Science George Mason University, Fairfax, VA gjsteinggmu edu
Abstract
We present approach for generating natural language explanations of high-level behavior of autonomous agents navigating in partially-revealed environments. Our counterfactual explanations communicate changes interpratable statistics of the belief (e.g., the likelihood an exploratory action will reach the unseen goal) that are estimated from visual input via deep neural network and used (via Bellman equation variant) to inform planning far into the future. Additionally, our novel training procedure mimies explanation generation, allowing us use planning performance as an objective measure of explanation quality. Simulated experiments validate that our explanations are both high-quality and can be used in interventions to directly correct bad behavior; agents trained via our training-by-cen -h-ee-bbb-xxiiiggg procedure achieve 9.19 % lower average cost than non-learned baseline (12.7% after interventions) in environments derived from real-world floor plans.
If are welcome robots into our homes and trust them to make decisions on their own, they should be able to clearly explain both what they are doing and why they are doing it in terms that both expert and non-expert users can understand. As learning (particularly deep learning) is increasingly used to inform decisions, it is equally important to ensure that data data-driven systems are easy inspect and audit both during development and after deployment [33, 6]. The field of explanatory artificial intelligence has developed significantly in recent years to address this need in the hopes of improving reliability and combating problematic data biases that can go unnoticed when docision-making is opaque. Myriad explainability tools aim to probe deep neural networks trained via supervised learning, often with a focus on understanding perceptual tasks, like detecting objects in images [24, 30, 28, 31, 38). Yet in many real-world navigation and home care scenarios, robots are expected to make effective decisions even when their knowledge of the world incomplete or stale; in this case, planning effectively requires making predictions about unseen space and about the potential impact of actions far into the future [39, 29, 3], complicating efforts to apply existing tools to explain robot behavior in this setting. In this work, we focus the challenging task of navigation in portially-revealle environments, in which reaching an unseen goal in minimum expected time may require recognizing subtle visual cues cues-eeg., that green markings signal route likely reach the goal-and incorporating multiple such observations when deciding where to explore next.
Designing agent that both achieves state-of-the-are performance and can meaningfully explain its actions has so far proven out of reach for existing approaches to planning under uncertainty. Symbolic abstraction-in which actions correspond to interaction with symbols (e.g.. objects) or movement through the environment-is an effective tool for planning in fully-known environments and yields plans that are interpretable-by-desiee since symbolic task plans are unnaansnanndstaaaaalllee by construction [25, 15]. Yet when planning under uncertainty, is not always obvious how to abstract knowledge without problematically degrading performance, particularly when the agent perceives the world via high- high-dimensional sensor observations, like images. Owing to challenges associated with data and computation, many tfate-of-the-as art approaches to planning under uncertainty learn the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

relationship between raw sensor observations and good behavior. Deep reinforcement learning has proven incredibly effective (if somewhat brittle [7]) in this regard [41, 36, 1, 39]. However, many such strategies are opaque by their nature and are neither well-suited to explaining themselves nor compatible with most existing tools for post-hoe explanation [12, 25].
Clearly a shift in representation is necessary: we seek an agent for which decisions are expressed in terms of symbols (so that planning interpretable) yet can also make use of learning to ease the onerous computational and data requirements of planning under uncertainty. It is a key insight of this work that the learning over subgoals planning paradigm of Stein et al. [29], which blends symbolic and data-driven planning, is well-suited to meet this need. In [29], symbolic actions each correspond to revealing space beyond frontiers, boundaries between free and unknown space, while learning is used to estimate compact statistics of unseen space that capture the impact of those actions- e.g., the likelihood an exploratory action will succeed in reaching the goal. We leverage this representation to develop a new approach to generate counterfactual explanations that communicate what changes to these estimated statistics would result in a desired change in behavior.
In this work, we introduce a novel approach to navigation in partially-revealed environments by which an agent can achieve state-of-the-art performance and also generate high quality human- understandable explanations of its behavior, despite relying on learning (from visual observations) to inform decision making. Specifically, our contributions address the following three questions:
What criteria must planner satisfy if it is to be useful for generating explanations in partially revealed environments? [Sec. 2] Our criteria (motivated by those of Lipton [15]) define what it means for planning to be interpretable-byy -yy--rttbbee----eiiig for our problem setting.
2. How can we generate explenationn explanations of the agent's high-level behavior? [Sec. 3] We show that the subgoal-basee planner of Stein et al. [29] meets our conditions for interpretable planning under uncertainty and, leveraging this representation, we introduce procedure to generate counterfactual explanations. Explanations outline what changes to the agent's 's belief would result in a change in behavior, expressed in terms of estimated interpretable statistics of unknown space: e.g., the likelihood an exploratory action will reach an unseen goal.
3. How can we validate that our explanations are high quality? [Sec. 4] We introduce a novel training process in which training data is generated via the same procedure as our explanations. This allows us to use planning performance as an objective measure of explanation quality.
We evaluate the performance of our approach in two different simulated environments in which an agent tasked to navigate to an unseen point goal in minimum expected time. We generate expla- nations in two different simulated environment classes and show that the explanations qualitatively match human intuition for how to correct the agent's 's poor behavior in these domains. Our quantitative results (Sec. 5) focus on the ability of our explanations to facilitate improving the agent's behavior procedure demonstrates state-of-the of-thh-artt performance
and show that our joint meining-by-explaineg at test time, validating that our explanations accurate and informative and demonstrating that an agent can be both explainable and performant simultancously in this application domain.
Preliminaries: planning in partially-revedn
environments
common way to model planning under uncertainty is as a Partially Observable Markov Decision Process (POMDP) [9, 16. 21J. In this model, the agent's behavior at time / is function of its belief b,, a distribution over possible states of the world, which the agent updates as it receives new observations through interaction with the world. In navigation context, the agent's 's objective is to minimize expected cost (travel distance) to reach an unseen goal. Belicf-space planning in general is incredibly computationally intensive. Planning (typically via the Bellman equation [21]) requires that the agent iteratively envision how it might take action, gain new knowledge, update its belief, and then take additional actions. In the worst case, the tree of possible actions and subsequent outcomes stemming from the current belief state grows exponentially with planning horizon; planning far into the future is fraught with challenges. In this work, we study a particular class of POMDP recently coined as a Locally Observable Markov Decision Process [18], in which the agent's local perception is assumed be perfect though limited by range and occlusions. This is common problem setting for agents equipped with either depth sensors or LIDAR and considerably simplifies planning through observed space-space the agent has seen and is therefore assumed to know with perfect precision. At every timestep, the agent receives its location, an collusiionlimitted local occupancy map (as if
Code available at https: // http//ggttttttcc///e cow.RAIL BEEiLLL-rrup//e ctaupp/xiiiia madi-nav-onn under www.-ueeattttt-ooriippeeeee
2

---

from a laser range finder), and an egocentric panoramic image of its surroundings and tasked to reach a goal, specified as world-space coordinate, in minimum time. Even with perfect local perception. planning that requires the agent to reason about unseen space is still incredibly complex.
2 Inferpretable----eeiiin planning in partially-revealle environments
Criterion for interpretable planning There are myriad choices of representations suitable for easing the challenges of planning under uncertainty-at evidenced by our earlier discussion- -yet we seek in particular strategies that are said to be interpretable, a precondition for generating explanations [6]. number of recent works consider what is implied by interpretability and propose taxonomies and criteria by which we can understand how interpretable a model is [6, 15, 25, 33]. Yet these works are broad in scope, requiring that we contextualize their criteria for our problem of interest. We propose the following criteria for evaluating the interpretability of model that relies on learning to plan under uncertainty, motivated by those of Lipton [15], which will later serve as desiderata:
Decisions available the agent should reflect how human might describe their own recision-makinn given the same information. Low-level actions like motion primitives are not typically conducive for interpreting the behavior of an agent an environment that requires thousands of steps to reach the goal. For long-horizon planning, asking Why did the agent decide to turn 5 versus 10 degrees left? is unlikely to yield useful explanations of the agent's sociision-maiing process. High-level actions instead define temporally extended behaviors (e.g.. entering a classroom or following a hallway) and often correspond to human- uunnetdddddrt symbols, with which planning can better match human intuition. This condition is a prerequisite to Lipton's idea of decomposabilltty, that each step of planning be intuitively understandable, and is consistent with a significant body of work leveraging symbolic abstraction [5, 10].
2. High-level planning should rely on model and explicitly consider the impact of the agent's actions into the future. Model-free planning strategies for planning under uncertainty, like the impressive MERLIN agent [39], directly predict the goodness of an action, implicitly estimating the impact that action will have on the agent's belief and thereby complicating efforts to understand the agent's behavior. Yet in many planning scenarios, humans seem to explain their decisions by explicitly enumerating the possible outcomes of actions that enter unknown space: e.g., whether a hallway will lead to faraway goal. If we are to understand why an agent chose one action over another, its pecision-making should match this structure; for planning under uncertainty to be interpretable and to be useful for generating explanations, the planner should (like human) explicitly consider the different ways in which executing an action might accrue cost and update the belief. This condition is consistent with Lipton's 's algorithmic transparency.
3. The impact of actions, particularly those that enter unknown space, should be expressible via a small number of muman-understandable quantities. That the plan be compuct is important for human understanding. Yet the computational and memory requirements of belief-space planning for large long- morizoo tasks are prohibitive even for a computer, and such planning fails meet the interpretability criterion Lipton refers to as simulatability. The model employed by the agent must therefore dramatically simplify the belief and the process of imagining the future.
Subgoal-based abstractions for interpretable navigation under uncertainty is key observation of this work that the learning over subgoals planning paradigm of Stein et al. [29] satisfies our criteria for interpretable-by-dese
goal-directed navigation under uncertainty; we provide an overview of this approach here. Under this abstraction, subgoals each correspond to high-level action to reveal unknown space by exploring beyond a frontier, a boundary between free and unseen space. The high-level outcome of each action is binary: an action either succeeds, if the agent discovers route to the unseen goal, or fails, if the region is a dead-end, prompting the agent to turn back. The expected cost of each action a, (corresponding to subgoal sx) depends on its likelihood of success Ps(a:) and the expected costs associated with success (a(a)) and failure (exploration) Rg (a.).. These terms (collectively, the subgoal properties σ)) implicitly' depend the belief b, and (intractable to compute directly) are estimated via learning. The costs of navigating between subgoals involves travel through observed space and can be computed via Dijkstra's 's algorithm: D(b,, a,). The expected cost of executing high-level action a, can be expressed via an approximation of the Bellman Equation
We note that Stein et al. [29] make the dependency of the subgoal properties Ps. Rs. and Rg on the belief' by explicit. Eq. (1) reflects simplified notation.
3

---

Original Plan
Modified Plan
The robot R plans to explore beyond subgoals in numerical order in an effort to reach the goal G.
The robot plans to explore beyond subgoals in alphabetical order effort reach the goal G.
8
Goal
P
0
(1) The agent's original plan leads first to Subgoal 0, which the user recognizes from the provided image (center) will not lead to the goal.
(2) The planner produces the following modified plan in response to the user query: "What would need to change for the robof R to prefer Subgoal 1 (renamed: a)?"
(3) To change its behavior, the agent changes its belief about unseen space. Having computed these changes, the agent automatically generates the following counterfactual explanation:
I would have preferred Subgoal (renamed to in the modified plan) if instead believed Subgoal 0 (c) were significantly less likely to lead to the goal (36% down from 98%) and that the cost of goal through Subgoal (c) were slightly increased (6.0 meters, up from 5.7 meters); and gettingg to the that Subgoal (a) were slightly more likely to lead to the goal (94%, up from 86%) and that the cost of getting to the goal through Subgoal 1 (a) were slightly decreased (5.8 meters, down from 6.0 meters).
Figure 1: Explaining behavior for navigation under uncertianty The generated explanation is consistent with the user's expectations: to change its behavior, the robot must change its belief to reflect that Subgoal 0 is unlikely to lead to the goal and Subgoal has higher likelihood.
written in terms of the high-level actions and their binary outcomes:
Q(br, a+) = D(b,, az) + Ps(at) Ps(aa) + +11 - Ps(ai)) RE(ar) + min aEAlb.))
Q0111. (I)
where bi+1 is the approximate future belief, which incorporates the knowledge that upon executing ag and failing to reach the goal the agent has moved (and now subgoal s,) and that the region beyond 8, is now known to be dead end (even if we do not yet know precisely what it looks like). The map of observed space not directly updated during planning, avoiding many of the computational challenges of updating the belief. Critically. the agent's high-level plan defines the order in which it aims to explore the unseen regions of the environment in search of the goal specified as a list of subgoal-actions. We refer the reader to [29] for full derivation of Eq. (1). Planning with this model is iterative. Upon selecting subgoal via Eq. (1), a low-level planner plots a trajectory through the known map towards the chosen subgoal and the agent executes short motion primitive along this trajectory. At every timestep, new space may be revealed, and Eq. (1) must again used to select the subgoal with lowest expected cost. This process repeats until the agent reaches the goal, which can take thousands of steps. Since the agent is assumed to have perfect local perception, the agent's ability to reach the goal is limited only by vehicle dynamics.
Given estimates of the subgoal properties (Ps, Rs, and Rg) for each subgoal- actioon available to the agent, high-level planning via Eq. (1) satisfies our criteria for interpretable planning under uncertainty. First, the decisions available to the agent correspond to choosing between exploring different regions of space, clearly actions (whether reaches the goal and the expected cost of each outcome) are easy to interpret and relatively few in number. Finally, planning is done via model, under which the impact of each action are explicitly taken into account and can influence the agent's s behavior far into the future.
In addition, the impact of the agent's s
Estimating subgoal properties from images Too difficult to compute exactly, the subgoal properties Ps. Rs. and Rg needed compute expected cost are estimated via neural network. Consistent with the approach of [29], subgoal properties are estimated from single sensor observations; thus, the input data used to estimate the subgoal properties consists of (1) single panoramic image, (2) the relative position of the goal in the image frame, and (3) the relative position of the subgoal in the image frame, all collected at the time the subgoal was created-i.e. the time at which the subgoal's corresponding frontier (a boundary between free and unknown space) was revealed. A ferd-forward (non-recurrent) neural network outputs the three quantities of interest; a logistic sigmoid function applied to produce the probability of success Ps. so that it is valid probability. The neural network

---

Algorithm 1: Generate Explanation
Algorithm 2: Train Subgoal Property Estimator
Data: my, Lt, ae. 0b, nn_ inputs, fo
Data: dataset, fu
Result: Subgoal property changes, Ao
Result: Neural Network Weights, By
0 + NN(nn_inpure
(o)
0 +- 8g
2 AQo <- [Q([m,, qt, Jo {an, ae})
2 foreach datum E dataset do
0 <- Do. AQ <- AQo
Tit,, 4t, do. un_inputs- <- datum
while AQ > 0 do
de arg e Q(mm, o).
5
a < (N(nnninputs, 0)
a f NN(un_inpin
0)
AQ - AQ([m.. qt, ). {an, act)
AQ + AQ([m, Qt, )) {a., dct)
Eq. (2)
a( ao. 0b)
a< a< 0,)
Eq. (3)
Leump + AQ
Leamp v logsigmoid(-AQ)
0 t 0
2aam
Voo o lasam
0 x 0
[VOO o 1 assap
b Eq. (4)
de7
der
V Cupperissdd
Vo
10 O. f- 4-NN(Nn (N(nnNNinnpppnnnee (ii_n_innutst0
11 return oy σo
10 return
0
Algorithms for generating explanations and training our subgoal property estimator, shown side-by-side to emphasize their similarities, in particular lines 5-9, a key feature of our approach.
The details of explanation generation (Sec. 3) and training (Sec. 4) can be found in the text.
is a convolutional -uull-uullltt---lllllllll------ -ht--ttteee----neeeeeee connecce encoder similar to that of [3]; not the focus of this work, the network architecture is described in detail in Appendix A.I.
3 Generating counterfactual explanations of high-level behavior
Explanations should answer fundamental questions about the agent's behavior and provide recourse for how to correct that behavior necessary. Let's say that, during an audit, the agent recently chose action az. yet the human auditor may have thought that ag was preferable option. The human may want an explanation to better understand why did the agent select action de over action a₃ ? The obvious (if unhelpful) answer is that the agent believed the action it selected (a,) was of lower expected cost than was dn. Instead, since our agent's ' behavior is determined by its estimates of the subgoal properties (Ps, Rg,, and Rg for each subgoal), the explanations of its behavior should feature these properties as well. Critically, since the agent's objective is to explain a decision to select one action over another, the explanation must contextualize the agent's choice in relation to the decision boundary between the two actions. With this in mind, explanations generated by our system are counterfactual, specifying in natural language a set of changes the (interpretable) subgoal properties that would result in a change in behavior from one action to another.
Computing counterfactual explanations via gradient descent For the planning problem defined by Eq. (I), a three clement tuple consisting of (1) the observed map me (2) the robot pose qt. and (3) the list of subgoal properties o, E g3xN, (where N, is the number of subgoals) is a sufficient statistic for the belief by We therefore use the notation Qffm, 9t. a }. a²) to represent the expected cost of executing action a; from map state m, and position qe given subgoal properties d While the map m, and pose qe are provided to the agent, the subgoal properties d are estimated from sensor observations via neural network, defined by its parameters O: changing the network parameters 0 would result in a change in the subgoal properties (Ar), which could change the agent's behavior.
We use gradient descent to compute our explanations, which are defined by set of subgoal property changes Ao and therefore a change to the agent's 's belief about unknown space. The decision boundary between the two actions of interest- the agent's 's selected action a, and the human's 's query action cxissssts at the zero-crossing of the difference in the expected cost of the two actions:
AQ([m,, qt, r.),, {an, ) = Q((m,, qt, ot), an) - Q {m,, qt, ot) ar).
(2)
If we think of this difference in cost as a loss Loomp = AQ, crossing the decision boundary can be achieved via gradient descent over the parameters 0 of the neural network used to estimate the subgoal properties σ, which are therefore implicitly functions of f. We run gradient descent until the AQ crosses zero (made easy with PyTorch [20]) indicating that the robot now prefers action as az. The change in subgoal properties after optimization Ao forms the backbone of our explanation.
Ensuring explanations are compact Returning changes to every subgoal property-whie which number three-times the number of subgoals, places the agent has yet to explore- is too much information to present all at once if explanations are to be useful for understanding or auditing an agent's 's behavior
5

---

[15]. As such, we wish to determine and communicate only the most important subgoal properties when explaining behavior. We define the importance of each subgoal property as their relative contribution to a change in AQ after one step of gradient descent. In differential form:
a aAA,. TAQ ao
V.AQ IQAQQ
o Vo
[Subgoal Property Importance] (3)
where o is the Hadamard Product (elementwise multiplication). The subgoal property importance vector a satisfies Σ; aq - [|V by construction. We communicate only the top N most important subgoal properties as natural language explanations, including only a handful of changes to key properties instead of returning a potentially- overwhelming amount of information to the human.
If we are to claim that the explanations are accurate despite only communicating a handful of subgoal properties to the user, we must ensure that changing the agent's 's behavior does not rely on changes not communicated to the user. To ensure this, we use masked gradient descent to compute explanations: the gradients from less important properties multiplied by zero, that only information from the chosen most important properties are used to update the network parameters 0:
aAQ
[Voo laxau)) 0.+1 + 0, ao
[Masked Gradient Update]
(4)
where 17 is the learning rate and (n) is the nth order statistic of a;; M is defined such that 1 1030 is a boolean vector that is true in the entries corresponding to the top-N elements of a (i.e., M = |a N)
and false otherwise. See Alg. 1 for our complete explanation generation procedure.
Fig. shows an example explanation generated using our procedure; note that the explanation only includes four subgoal properties: those deemed by our agent to be the most important. We note that all subgoal properties will change during explanation generation, even when using our masked gradient descent procedure, since the network parameters @ are shared between all subgoal properties. Even so, the explanations are still an accurate reflection of the underlying decision-maker -making process: updating the network parameters 0 until the most important subgoal properties change by their specified amounts would cause the change in behavior the explanation seeks to describe, even if other subgoal properties are also updated in the process.
From subgoal property changes to natural language After generating the subgoal property changes, we turn them into a natural language explanation using a simple rule-based grammar. Subgoals are ordered by the importance of their most important property and their selected properties are inserted into template strings and concatenated. We note that the way we have chosen to present our explanations (e.g.. Fig. 1) is only one way to present the information contained within our counterfactual explanations. While in this work we focus on presenting the explanations in way that is both complete and precise, it might be that eschewing displaying exact numbers and instead including only qualitative differences improves readability. Understanding how to best present our explanations will require further study with human participants and is an important next step to more deeply understand the effectiveness of our approach; we discuss this topic further in the future work section of Sec. 7 and additional details can be found in Appendix A.3.
4 Training by explaining: validating explanation quality
high-quality counterfactual explanation is both accurate, a faithful account of the agent's 's decision- making process, and is usesul, rich with information that could be used to audit the agent and change its behavior. While our explanations are accurate by construction (see Sec. 3), the utility of an explanation can be challenging to measure. Often the measure of the quality of an explanation includes the eyeballing metric; this metric serves an important purpose- catoosss-rtttullarllyy since explaining is for communication with humans-yet Sundararajan et al. [30] point out that this approach is problematically subjective. Moreover, as one of the primary functions of explanations in this context will be to help improve or correct poor behavior, it is critically important to ensure that the explanations can be useful in this capacity. The structure of our training procedure and our choice of experimental results are therefore focused on this objective measure of explanation's 's utility: whether explanations are sufficiently information-rice to facilitate correcting undesirable behavior.
With this in mind, we train our agent via the same procedure that is used to generate explanations. This closes the loop between the two processes, allowing us to use the performance of the system as an objective measure of the expected utility of the explanations: if the explanations were insufficient, the

---

agent would be unable to learn to plan effectively from those explanations and test-time performance would be poor. During an offline data-collection phase, we rely on an oracle-an an agent with full state observability- to specify which subgoal corresponds to the shortest path of the unseen goal, thereby serving the role of the human auditor. At each step, a datum is recorded, including the oracle's action ao, the observed map m,, the agent's 's pose lt and the agent's 's observations or all the information needed to estimate the subgoal properties o and, with them, the expected cost Q.
Training relies on the generation of one-step pseudo-explanation estimated subgoal properties would need to change to decrease the expected cost of the oracle's s action ag with respect to comparison action ae. defined as the lowest-expece ceresst-epeeeedcoos action" for the current datum and network parameters 9. Stochastic gradient descent is used to iteratively update the neural network properties 0 so as to encourage the agent to select the action a, via minimization of AQ for each datum. Furthermore, so as to mirror the process of generating explanations, we also use masked gradient descent from Eq. (4), in which only the most important subgoal properties are allowed non-zero gradient during training. Each training iteration can be thought of as taking a single step towards generating counterfactual explanation for the selected datum. To train, we cycle through many different data; to generate an explanation, we instead iterate using the same datum until the decision boundary is crossed. Alg. 2 includes our full training procedure; note the similarities as compared to our explanation generation process shown in Alg. 1.
that aim to describe how the
Training-specificc loss terms and details We note that the objective function being optimized during training is different from the objective function during explanation generation: a few modifications are needed to help stabilize training, handle vanishing gradients, and impose additional priors on the known bounds on various quantities. While our primary objective still to minimize AQ, training an agent from scratch to minimize AQ does not yield the behavior we want. In particular, training should prioritize samples in which the agent is near to the decision boundary, since these will have the most direct impact on overall performance; optimization should focus less scenarios in which the agent already believes that the oracle's s action is best (&Q < 0) or where modifying behavior would require significant change (AQ >> 0). To achieve this prioritization, instead of directly minimizing AQ define alternative training comparison objective Leomp /1 logsigmoid( AQQ), which rescales AQ to (smoothly) prioritize points near to the boundary, as desired. Though a full investigation is out of scope, anecdotally find that this alternate objective is particularly important in the more complex environments in which perfectly imitating the oracle is not possible with limited perception. Applying a monotonic scalar function to AQ does not change the direction of its gradient (in (-space), and therefore using this modified objective yields identical explanations as does using AQ directly in the limit of small step size. We also add two training objectives that incorporate additional knowledge and stabilize training. We add a supervised training objective S a cross-entropy loss that incorporates labeled data (whether subgoal leads to the goal) to account for the potentially vanishing gradient associated with the logistic sigmoid function used to compute Ps. We also add Lhounds: a collection of hinge loss terms that penalize non-physical predictions (negative values of Rş and Rgg and imposes heuristic bounds on the expected cost Q, computed via a non-learned Dijkstra's 's algorithm planner during data collection. Finally, we note that in our Guided Maze environment (shown in Fig. 1) a high value of Superriisdd is sufficient to achieve good performance without our training- wwwpiinn-vi--eppliie via-explaining process, as mentioned for a similar environment used in [29]. So as to demonstrate the effectiveness of using our explanations during training, we reduce the weight of Luppervised by a factor of 40 for all learned planners trained in that environment.
Since each datum can have over a dozen panoramic images, we use a batch size of I and training for each learned planner takes roughly 12 hours on a desktop Nvidia 2060 SUPER GPU. There is considerable redundancy in the data-since many images appear in multiple fatum-and so we train for only a single epoch, yet divide the learning rate by half every time one-cighth of the data has been consumed. Full training details can be found in Appendix A.2.
5 Results: simulated navigation under uncertainty
We show an annotated example explanation in our guided maze environment in Fig. 1; an agent trained with the poor-perfon learned baseline (described below) must explain why it did not select the correct path, as indicated by a green path on the ground. The explanation matches our expectations:
II the wwwest-xppected-eee action during training is already the oracle action a.. the www--lowwsxeeeeeeeeeeeee cost action is chosen for comparison instead so that training increases the margin from the decision boundary.
7

---

Guided Maze Environment
University Buildings Env.
Planner
Average Dist.
% Improve.
Average Dist.
% Improve.
All Subgoal Properties
14.94
23.2
43.52
6.8
Subgoal Properties
14.94
23.2
42.46
9.1
No comp (Learned Baseline)
19.17
1.3
44.34
5.1
Non-Leamed Baseline
19.45
46.70
"Guided" Maze Environment
University Buildings Environment
All Subgoal Props.
Subgoal Props.
No Subgoal Props.
All Subgoal Props.
Subgoal Props.
No Subgoal Props.
aee-Fal
Zoored
Zoomee
Zoonedi
Learned Cost
Leamned Cost
3 2
RE-
29
42
Non-Learned Cost
Non-Leamed Cost
Non-Learned Cost
10e Non-Learned Cost
Non-Learned Cost
Non-Learned Cost
Figure 2: Navigation under uncertainty performance Scatterplots each show the performance of learned planner versus the non-learned baseline for 1,000 trials in each environment; darker color indicate higher data density. Planners trained via our saiiiin-via-explainiii approach outperform those that are not, validating that our explanations are useful for correcting poor behavior.
to change its behavior, the robot would need to believe that the subgoal corresponding to the only path to the goal were more likely and that its current selection were less likely (among other small changes). While this example and the explanation in Fig. 3 qualitatively show that our counterfactual explanations can help to elucidate the agent's 's Decision-making process in terms, we additionally conduct two types of quantitative experiments that validate the goodness of our experiments and the utility of our approach with an emphasis demonstrating that our explanations are sufficiently rich with information that they can support correcting poor behavior.
Training via Explaining As discussed, our training procedure mirrors how explanations are generated, and so we can use the performance of our agent as a measure of the overall quality of our explanations. We conduct experiments in two different simulated environments: (1) the Guided Maze, procedurally generated mazes in which a green path on the ground indicates the (only) route to the goal and (2) the University Buildings environment, topologically complex maps which are extruded from over one-hundred floor plans of university buildings augmented to include obstacles to simulate clutter furniture and in which long passages (that human might identify as hallways) connect faraway regions of space. Our University Buildings environment, generated with aid of data from Whiting et al. [40], is quite large compared many existing navigation benchmarks (e.g.. [27]) and its size and complexity are well-suited for studying long-horizon navigation under uncertainty. Both environments are rendered via an in-house simulation environment built with the Unity Game Engine [34]. Start and goal locations are randomly selected free-space points generated via a fixed random seed, so that all planners are given the same set of tasks. We train three models for each environment type: All Subgoal Properties, in which all subgoal properties Subgoal Properties, where the gradient is limited to select only the
Planning Performance:
are used during training; four most important subgoal properties; and No Subgoal Properties (No Lcomp) where only the auxiliary losses Lxuperviedd and Loounds are used during training. The "No Leomp planner serves as our Learned Baseline, since it is trained without the explanatory module. We additionally compare to Non-Leurned Baseline, an optimistic agent that plans (via Dijkstra's ss algorithm) as if all unseen space were unoccupied and therefore does not rely on subgoals to plan. For each environment, we train our agents via data collected over 1,000 traversals (from start to goal) by our non-learned baseline: at every step we record single datum containing all the information a decision and the option chosen by an oracle. We evaluate performance over 1,000 traversals in previously-unsee test environments for each planner to show generalization; evaluation across all planners and environments takes roughly 10 days.
agent may need to make
Our results, included in Fig. 2, show that training via our procedure yields improved results over both our learned and non-learned baseline approaches. the Guided Maze environments, is clear that our approach yields learned models that seems to understand the structure of the environment, showing 23.2% improvement over the non-learned baseline. Performance on the more complicated University Buildings environment further substantiates the goodness of our explanatory process:
8

---

agent improves upon the non-learned baseline by 6.8% when trained with all subgoal properties and by 9.1% 1% when trained using the four most important properties each iteration. Limiting the number of subgoal properties in the University Buildings environment seems to regularize the model; exploring this effect further will be topic of future work. In both environments, the Learned Baseline training-vid--xplaaadie via-explanation strategy (and therefore our explanations) adds information valuable for correcting unwanted behavior.
underperforms the other learned planners, substantiating that our training-
Quantitative Interventions via Explanations Here we show that individual explanations can be used directly update the be- havior of our agent via interven- tions. Generating explanations in- volves updating the learned model via gradient descent (see Alg. 1) until the agent's 's behavior changes and the counterfactual explana- tion therefore describes change to the agent's learned model. In an intervention, those changes accepted by the auditor: the agent then permanently relies on the newly updated learned model to estimate subgoal properties and plan for the remainder of the trial. A successful intervention changes the agent's local behav- ior match the oracle for a sin- gle time step and also results in the agent continuing to perform well until the goal is reached, as measured by its overall naviga- tion performance. For our exper- iments, we select the 50 trials in the University Buildings environ- ment which our Subgoal Prop- erties learned planner most under- performs the non-learned baseline and intervene at the start of the longest stretch of bad wehhviio-whhe the learned agent disagrees with the oracle- thus providing us with a measure of the capacity of individual explanations to improve agent's 's behavior. We run two sets of experiments, one in which explaining uses only 4 subgoal properties (as in training) and one where all are used.
Planner
Avg. Dist
Succeed
Non-Learned Baseline
85,05
No Intervention (Learned)
124.24
Intervened: 4 Subgoal Props.
93.77
37/50
Intervened: All Subgoal Props.
92.81
43/50
Before Intervention robat Rt plans tu explore beyond subguuls in mumerical order effort reach goal G. Redas
After Intervention The robot R plans tu explore beyond subguals in adviuthetical order in effort in reach the goal Rubud
0
Explanation: I would have
preferred Subgoal 10 (a) over
Subgoal 0 (c) instead
believed that Subgoal 11 (b)
Intervention
more likely lead to
the goal (29%, up from 18%);
Trajectory Cost 337.5
and believed that Subgoal 10
Trajectory Cost: 171.7
(a) were slightly more likely
lead to goal (22%, up
from 15% ); and believed that
Subyoal 0 (c) were slightly
less likely to lead to the goal
Coal
(22%, down from 30%).
Goal
Figure 3: Explanation-Drivee Intervention Results (top) & Example (bot.) By accepting changes proposed by explanations at key decision points, we correct poor behavior in many of the worst trials in the University Buildings environment.
Our intervention experiments show that individual explanations can be used to correct undesired behavior; see Fig. 3 for statisties and an example intervention in the University Buildings environment. We show an improvement of 24.5% when explaining with 4 subgoal properties and 25.3% when using all properties. The primary cause of the performance difference between the two is the increased chance of vanishing gradients when we limit the number of subgoals: which properties are most important may change as the policy is updated and so the model may be unable to cross the decision boundary, causing explaining to fail. Even when using all subgoal properties, for significant changes in behavior the neural network used to estimate subgoal propertics may exhibit vanishing gradients or may propose non-physien non-physical changes (c.g., negative values for Q. the expected cost), which serve as grounds for rejection of the explanations. We could add additional training objectives to overcome these issues, yet these may problematically bias the explanations; exploring this relationship will be the subject of future work. Even with these occasional failures, the cost savings in our University Buildings environments from these interventions is non-trivial: 3.6% absolute cost savings when intervening using four subgoal properties and 3.7% when using all properties.
'We note that in practice, human serves the role of the auditor, and will first review the agent's explanations before deciding whether or not change in behavior is necessary. Our intervention experiments are designed to focus on and quantify the capacity of our explanations to facilitate these changes in behavior if requested.

---

6 Additional related work
Explainability Our approach to generating counterfactual explanations is similar to number of others that focus on the inputs to the learned algorithm, rather than estimates of interpretable intermediate quantities. Nearly all (like ours) rely on gradient information to generate counterfactuals so that explanations remain faithful to the underlying model, with a focus on either rule-based symbolic decision systems [37] and perceptual systems [38, 28, 31, 30] and with applications to natural language processing (22, 23, 8]. Our importance [Eq. (3)) is conceptually similar to saliency used in many visual explanations [31, 30], yet saliency usually defines the explanation, rather than being used to guide explaining. Recent survey and meta-analysis papers present taxonomies of the recent explosion of explainable Al tools [32. 15, 6, 25], yet none focus in particular on planning under uncertainty or its unique challenges and opportunities.
Planning In addition to the black- box, model-free approaches to planning discussed above, there are a host of other techniques that impose more structure to make planning more interpretable; many are in the space of self-driving [11, 42, 26], in which the model estimates intermediate understandable quantities (e.g., where is safe to drive) and is trained end-to-end. These techniques are not yet well-suited for long-horizon planning or explaining high-level behavior and how to correct it. A few recent papers learn interpretable hierarchal ontologies (including so-called neural-symbolic planners) [14, 35, 2, 17] yet are so far primarily useful for improving performance rather than for generating explanations of high-level behavior. Recent work [19] generates post-hoc visual explanations for agents trained via deep reinforcement learning, yet focuses on short-horizon objectives.
7 Limitations: implications and future work
If our approach is to be broadly adopted and enable more trustworthy autonomous agents, key limitations must first be addressed. First, the model of [29] that we build upon for explaining makes strong assumptions about planning that are untrue in general, including that exploring beyond fulgoal.ullly-reeed, an arm of the environment and that unknown space is simply connected. In envi- ronments that violate these assumptions, performance and explanation quality may be prohibitively poor. Getting good planning performance may also require that the agent "cheat" and estimate (for example) artificialll-high likelihoods to bias its behavior against missing potential routes to the goal. The conflicting objectives of high-accurad performance has potentially problematic societal implications; our system's estimates may mislead the user if they differ from their implied interpretable meaning. Second, that explanation generation can sometimes fail is prohibitive for widespread adoption of our approach; adding additional objec- tives to the explanation loss function will likely overcome this limitation (see our discussion above) yet may also problematically bias the explanations. Third, while our approach explains the agent's high-level behavior, the relationship between images and the estimated subgoal properties remains opaque, presenting challenges in the event that the human who requested the explanation is uncertain if the explanation is reasonable. There exists a body of work useful for elucidating this relation or constructing networks with interpretability in mind [4], and we look forward to future work that integrates these approaches with ours, we show preliminary results supporting the potential of these strategies in See. A.4. Finally, while our quantitative results are devoted mostly to demonstrating that our generated explanations are sufficiently rich with information that they can be used to improve the agent's behavior, how effectively a human can make use of these explanations to understand and subsequently correct bad behavior remains a question for future study. Though our theoretical contri- butions and qualitative results support the idea that our explanations are amenable for communication to humans, maximizing communication may require further tuning of their presentation. Additional work must be done to address these limitations, including additional numan-in-the-loop experiments, before approach can be trusted on an agent that coexists with humans. We additionally hope that our work will also prove useful for explaining even more complex behavior and Bradley et al. [3] recently applied the learning over subgoals representation to multi-stage task planning; extending our approach to support this application domain seems a promising direction for future work.
estimation versus predicting properties that optimize
Despite these limitations, our explanation procedure represents key step towards the design of more trustworthy robotic agents, capable of explaining their behavior even when expected to act far into the future and without full knowledge of their surroundings. Overall, we have shown that our natural language counterfactual explanations of high-level behavior for navigation under uncertainty both qualitatively meet our expectations and are quantitatively high-quality, that our explanation process can help to train an agent from scratch and to directly correct unwanted behavior.
10

---

Acknowledgments and Disclosure of Funding
We thank Kevin Doherty, Leilani Gilpin, and Lucia Rafanelli for detailed feedback and thought- provoking discussions that helped hone the ideas presented here. We also acknowledge Chris Bradley, Nick Roy, and other members of the MIT Robust Robotics group, who provided helpful feedback on an early version of this work. G. Stein acknowledges funding from the Department of Computer Science at George Mason University.
References
[1] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fis- cher, S. Hashme, C. Hesse, R. Józefowicz. S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, F. Wolski, and Zhang. DOTA 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0660 2019.
[2] T. R. Besold, A. d. Garcez, S. Bader, H. Bowman, P. Domingos, P. Hitzler, K.-U. Kühnberger, L. C. Lamb. D. Lowd, P. M. V. Lima, et al. Neural-symbolic learning and reasoning: A survey and interpretation. arXiv preprint 1rXiv:171..03900 2017.
[3] C. Bradley, A. Pacheck, G. J. Stein, S. Castro, H. Kress-Gazit, and N. Roy. Learning and planning for temporally extended tasks in unknown environments. In International Conference on Robotics and Automation (ICRA), 2021.
[4] C. Chen, O. Li, A. Barnett, J. Su, and C. Rudin. This looks like that: Deep learning for interpretable image recognition. In Neural Information Processing Systems (NeurIPS), 2019.
[5] M. Fox, D. Long, and D. Magazzeni. Explainable planning. In Proceedings of IJCAI-17 Workshop on Explainable Planning, 2017.
[6] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal. Explaining explanations: An overview of interpretability of machine learning. International Conference on Data Science and Advanced Analytics (DSAA), pages 80-89, 2018.
[7] Henderson, Islam, Bachman, Pineau, D. Precup. and D. Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, 2018.
[8] A. Jacovi, S. Swayamdipta, Ravfogel, Y. Blazar, Y. Choi, and Y. Goldberg. Contrastive explanations for model interpretability, arXiv preprint arXiv:2103.01378 2021.
[9] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101:99-134. 134, 1998.
[10] H. A. Kautz and J. F. Allen. Generalized plan recognition. In Association for the Advancement of Artificial Intelligence (AAAI), volume 86, 1986.
[11] J. Kim and M. Bansal. Attentional bottleneck: Towards an interpretable deep driving network.
In Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
[12] J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238-12 -174, 2013.
[13] N. Kokhlikyan, V. Miglani, M. Martin, E. Wang. B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, and O. Rebliz-RRccardson. Captum: A unified and generic model interpretability library for PyTorch 2020.
[14] T. P. Le, N. A. Vien, and T. Chung. A deep hierarchical reinforcement learning algorithm in partially observable markov decision processes. IEEE Access, 6:49089-49102, 2018
[15] Z. C. Lipton. The mythos of model interpretabilityy Quene, 16(3):31-57,2011
[16] M. I.. Littman, A. R. Cassandra, and L.. Kaelbling. Learning policies for partially observable environments: Scaling up. In Machine Learning Proceedings 1995, pages 362-370, 1995.

---

[17] D. Lyu, F. Yang. B. Liu, and S. Gustafson. Sdrl: interpretable and data-efficient deep rein- forcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2970-2977,
[18] M. Merlin, N. Parikh, E. Rosen, and G. Konidaris. Locally observable markov decision processes. In ICRA 2020 Workshop on Perception, Action, Learning, 2020.
[19] C. Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, and A. Mordvintsev. The building blocks interpretability. Distill, 2018. https://disttillpuu//00 8/building-block.
[20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Desmaison, Kopf, E. Yang, Vito, M. Raison, A. Tejani, S. Chilamkurthy. B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-perfone perffrrefeeeee deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pages 8024 00044-003,222 2019.
[21] J. Pineau and S. Thrun. An integrated approach to hierarchy and abstraction for pomdps. Technical Report CMU-RI-TR-00-211, Carnegie Mellon University, Pittsburgh, PA, August 2002.
[22] L. Qin, A. Bosselut, A. Holtzman, C. Bhagavatula, E. Clark, and Y. Choi. Counterfactual story reasoning and generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
[23] L. Qin, V. Shwartz, P. West, C. Bhagavatula, J. D. Hwang. R. L. Bras, A. Bosselut, and Y. Choi. Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
[24] M. T. Ribeiro, S. Singh, and C. Guestrin. Why should trust you?: Explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining, pages 1135-11 1144, 2016.
[25] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 155):206-215, 2019.
[26] A. Sadat, M. Ren, A. Pokrovsky, Y--C. Lin, E. Yumer, and R. Urtasun. Jointly learnable behavior and trajectory planning for self-driving vehicles. In International Conference on Intelligent Robots and Systems (IROS), 2019.
[27] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research. In International Conference Computer Vision (ICCV), 2019.
[28] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM Visual explanations from deep networks via gradient-based localization. In International Conference on Computer Vision (ICCV), pages 618-626, 2017.
[29] G. J. Stein, C. Bradley, and N. Roy. Learning over subgoals for efficient navigation of structured, unknown environments. In Conference on Robot Learning (CoRL), 10 2018.
[30] M. Sundararajan, A. Taly, and Q. Yan. Gradients of counterfactuals. arXiv:1611.02639 2016.
arXiv preprint
[31] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML), 2017.
[32] H. Suresh and J. V. Guttag. A framework for understanding unintended consequences of machine learning. arXiv preprint /9X.v:/901.1000 2019.
[33] H. Suresh, S. R. Gomez, K. K. Nam, and A. Satyanarayan. Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In CHI Conference Human Factors in Computing Systems, 2021.
12

---

[34] Unity Technologies. Unity. https://unity:: com, 2017.
[35] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukeuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549 PMLR, 2017.
[36] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu,
Cai, Agapiou, M. Jaderberg, A. S. Vezhnevets,
D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782)350-335) 2019.
[37] S. Wachter, B. Mittelstadt, and C. Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, 31: 841, 2017.
[38] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding. P. Mardziel, and X. Hu. Score-CAM: Score-weighted visual explanations for convolutional neural networks. Computer Vision and Pultern Recognition (CVPR) Workshops, 2020.
[39] G. Wayne, C. Hung, D. Amos, M. Mirza, A. Ahuja, A. Grabska- Barwinska, J. W. Rae, Mirowski, J. Z. Leibo, A. Santoro, M. Gemici, M. Reynolds, T. Harley, J. Abramson, S. Mo- hamed, D. J. Rezende, D. Saxton, A. Cain, C. Hillier, D. Silver, K. Kavukcuoglu, M. Botvinick, D. Hassabis, and T. P. Lillicrap. Unsupervised predictive memory in a goal-directed agent. arXiv preprint arXiv: 1803. 10760, 2018
[40] E. Whiting, J. Battat, and S. Teller. Topology urban environments: Graph construction from multi-building floor plan data. In Computer-Aided Architectural Design Fulures (CAADFutures), pages 114-128, 2007.
[41] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. Decentralized distributed (ppo): Learning near-perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations (ICLR), 2020.
[42] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (ICCV), 2019.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] To the best of our ability, the abstract and introduction are relevant and accurate.
(b) Did you describe the limitations of your work? [Yes] Our final section (Sec. 7) includes discussion of the limitations of our approach; our results section (Sec. 5) also includes some additional discussion.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Our final section (Sec. 7) includes discussion of the limitations of our approach. This includes number of key ways that our explanation strategy may fail, resulting in circumstances where the agent's s behavior cannot be particularly well explained or where poor behavior cannot be corrected and where explanations may potentially mislead the user, all prohibitive negative impacts for the immediate adoption of our approach without further study.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We affirm that have read these guidelines. 2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We enumerate all assumptions made by our approach alongside the discussion that motivates its form. See Sec. 4 and Sec. 3.
13

---

(b) you include complete proofs of all theoretical results? [N/A] While we did make theoretical contributions in the form of a novel approach generating explanations, our advances did not require proofs as such. 3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL)? [Yes] We have included all code necessary to reproduce experiments as supplementary material. So as to ensure that it be easily reproduced, we provide tooling to build the code in Docker container and automatically identify and reproduce individual trials and experiments as desired. Further details and instructions
included in README provided alongside the code.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] Some of these details are mentioned the main text. These and the remaining details are also included in the appendix: Sec, A. for learning architecture and Sec. A.2 for training details. We have tried to be thorough as possible describing our design decisions.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] All learning algorithms are trained via identical configurations (with single, fixed random seed, network configuration, and hyperparameters shared between all), with the only changes between them being the different numbers of subgoal properties used during training. For our planning experiments and results, we include both the average cost across 1,000 trials for each planner and environment and also include scatter-plots that show the individual trials used to produce results (in lieu of error bars).
Dd) you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details of the computational resources used are mentioned in few places in the main text and repeated in the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curatin//releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our simulation environments and relevant data were either created in-house or have been given proper attribution.
Dbd you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We
have included code written for the purposes of this project in the supplementary materials. As our experiments are simulated, we also include a binary of our simulated environments which can be used to generate data.
(d) (d)Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Data is collected via simulation.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] Data is collected via simulation.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable??
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xeHIPp5tz4G", "reviewer": "Reviewer_5zEg", "summary": "- This paper discusses the challenges of providing high-quality explanations of agent navigation decision making in partially-observable environments.  PointGoal in particular is studied.\n- The authors proposed a transparency-by-design method based on subgoal-based actions.  The agent selects a subgoal in the form of an area to explore next and selection is based on simple criteria.\n- Given an alternative subgoal, an explanation can be generated for why the agent selected its chosen subgoal instead of alternative subgoal.\n- The efficacy of the explanations are validated by showing the performance of the agent.  Since the explanations are directly part of the agent's action selection process, showing that the agent performs well also ensures that the explanations are high-quality.  The agent performs well and training via explanation helps.\n- Finally, the authors show that the agent can be provided interventions to improve performance\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post Rebuttal\n-----------------\n\nI thank the authors for their response. I have increased my score to a 7 to indicate that I think the paper should be presented at NeurIPS. \n\nOne comment: SPL is still very useful when success is nearly 100% as then it functions as a normalized version of Average Dist. In my opinion, SPL should be reported in addition to Average Dist. as this is a metric that readers from the visual navigation community will be highly familiar with, thereby improving the readability of the paper.\n\nOriginal Review\n---------------------\n- Strengths\n    - This paper approaches a difficult topic, explanations for visual navigation agents, and provides useful and insightful discussion on the topic\n    - The agent action selection procedure and explanation procedure are both well-thought-out\n    - The paper is very well-written and easy to follow\n    - The interventions work well\n- Weaknesses\n    - Are the explanations better than \"$$a_t$$ was selected because the agent believed it was of lowest cost\" in practice?  They provided more detail, but the explanation are based on estimations from a neural network, not wrt inputs themselves.  In other words, while the actual action selection mechanism may be interpretable and transparent, the inputs to that mechanism are not.  I do not see why a human auditor would be able to determine why (using the example in Fig 1) Subgoal 1 has a 86% chance of success instead of a 94% chance.\n        - The authors do note this issue and note that work on NN's interpretability may be able to help, but I am unsure if those methods are able to generate explanations in this situation since the NN is tasked with both modeling the low-level controllers ability to \"explore beyond a frontier\" and the input to it is more indirect than in say classification -- the network is given an input image and tasked with using that to infer properties about the rest of the environment to then estimate whether or the agent will succeed.\n        - A proof of concept experiment that existing CNN interpretability techniques provide something reasonable here would greatly strengthen this paper.\n    - Non-standard evaluation procedure and dataset.  \n        - The datasets used for evaluation aren't standard for PointGoal Navigation.  Ideally the datasets from Gibson or Habitat should be used as these would allow for comparison with many other methods (albeit non-explainable ones).  \n        - The evaluation metric of Average Dist. is also not standard nor is it explained in text.  Adding in metrics such as Success and SPL (https://arxiv.org/abs/1807.06757) would improve the paper.  \n        - It is never stated if there is a train/test split or if there is no generalization be tested\n        - Will the Guided Maze and University Buildings datasets be released?\n    - Questions:\n        - How is \"explore beyond a frontier\" performed?  That seems very ill-defined.\n- Overall\n    - While I see issues with this work, I don't necessarily think they should block publication as I do believe others may find this work useful and insightful.  I look forward to discussing this work with other reviewers.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "vihHJe-V-8-", "reviewer": "Reviewer_qvY1", "summary": "This paper presents a method for deriving counterfactual explanations of model decisions (relative to the correct decision the agent should have made). The general method is that by performing gradient descent over the model parameters until its decision changes, a set of properties is identified as important features which the agent had mistakenly used to make its decision. To ensure that a minimum number of properties is identified (only the important ones), a second pass of gradient descent is used fixing all but these properties to ensure they still result in a change in the decision. The method is evaluated by training a model using these explanations.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: I am not intimately familiar with work regarding explainable systems / generated explanations, so it is difficult for me to place this in relation to prior work. However, the proposed method seems original relative to the related work described in the paper.\n\nQuality: The method seems well-supported theoretically and in the simulated experiments. I would be interested to see performance without the supervised auxiliary loss. Also, I would have been interested to see results using human judgments, although prior work has found these problematically subjective. This is  not within the scope of this work, but it is surprising that there is not a reliable way to evaluate how useful explanations are to users of a system.\n\nThe authors describe several limitations to the approach, which I appreciated. I was somewhat confused about the space of high-level actions: it seems that (from Sections 2 and 7) taking a single high-level action would fully reveal the remaining environment accessible via taking that action. I am somewhat confused why this is the case (if I am understanding correctly), and what challenges would need to be addressed to remove this assumption. Similarly, I was curious about the topologies of the maps: do they have cycles? Multiple paths to a goal? How many high-level actions are available in the start state?\n\nI appreciated the care put into the design of the method, including masked gradient descent.\n\nClarity: There were several points which I found confusing. I would have preferred a bit more formalization of the task itself and the learning setup. And a few questions below:\n- What clues does the agent have in the university environment that one location might lead to the goal? In the other environment, it seems green paths would be the clue.\n- The explanations explain a *mistake* the agent makes, conditioned on the correct action to take. However, they don't seem to explain the initial prediction made by the agent, conditioned on the image input. They also don't seem to explain why the changes in the subgoal properties result in the change in the predicted action. Is my understanding correct? (Section 7 seems to confirm this, but I'm not sure.)\n- What does the space of subgoal properties look like? Does the agent have true access to these when navigating or is this something it is estimating?\n- I was confused about how learning from explanations worked in Section 4. Is this just that upon a mistake, the gradient descent procedure is carried out, and the resulting parameters are used in the next environment rollout?\n- Typo in L175: \"boundary\"\n\nSignificance: Some of the limitations discussed in Section 7 make me wonder about this method's applicability to new environments and action spaces; however, the general idea behind this proposed method seems like it could be extended.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the limitations of the approach were well addressed. One minor limitation I was curious about is how this approach might extend to environments that are dynamic, or action spaces which including manipulating an environment.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "okeC7FMIY5", "reviewer": "Reviewer_74uJ", "summary": "This paper presents a method that generates counterfactual explanations for navigation agents. The method is based on the \"learning over subgoals\" framework of [1], where each subgoal action indicates a plan and is formed by three properties: (i) the probability of success, (ii) the cost of success, (iii) the cost of failure. The method computes a subgoal's properties changes that can cause the agent to prefer another subgoal. These changes are computed by applying gradient descent onto the agent's model until the agent's preference switches. An explanation is then generated based on the changes, using a rule-based grammar. \n\nThe authors also propose a training-based evaluation scheme for the generated explanations. Essentially, they show that training with the gradients obtained during explanation generation leads to performance improvement. \n\n[1] Learning over subgoals for efficient navigation of structured, unknown environments. http://proceedings.mlr.press/v87/stein18a/stein18a.pdf\n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The specific method is novel, even though it shares spirit with a lot of previous work (see related work section). To add to the list of related work, closely related ideas have recently been proposed in NLP [1, 2, 3]. The uniqueness of this work is to formulate the method in the context of partially-observable environment, and to specifically target navigation agents. \n\n[1] Contrastive Explanations for Model Interpretability. https://arxiv.org/pdf/2103.01378.pdf\n\n[2] Counterfactual Story Reasoning and Generation. https://aclanthology.org/2020.emnlp-main.58.pdf\n\n[3] Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning. https://aclanthology.org/2020.emnlp-main.58.pdf\n\n**Quality**: \n\nThe method for generating explanations is technically sound. It is developed on top of a well-established framework. The authors state clearly upfront the criteria that they desire from explanations and argue why their explanations satisfy these criteria. \n\nMy main concern about quality is the training-based evaluation scheme. To me, the ultimate goal of explanations is to communicate intent of agents to humans. The criteria set by the authors also strongly emphasize the human-helpfulness and human-interpretability aspects. The training-based evaluation cannot assess their aspects and thus may not be used to replace human evaluation. The scheme mainly focuses mainly on how the agent helps itself, rather than how it helps humans obtain a better understanding of its decisions. To the latter case to be true, explanations should influence decisions or knowledge of humans. This is not happening in the scheme, as the oracle (or human auditor) simply puts out a ground-truth action without consulting an explanation.\n\nSubjectively, I find the explanation wording quite convoluted. For example, the first sentence in Figure 3  talks about why 0 is preferred over subgoal 10 but gives a reason that talks about subgoal 11. It may not be trivial to a regular user why subgoal 11 should influence the preference between subgoal 10 and subgoal 0. Human evaluation is essential to assess the interpretability of these explanations. Such a study could motivate better ways to present the explanations (e.g. using visual cues).    \n\nSimilarly, it is not clear that the intervention experiment (line 321) is showing the role that the explanations are supposed to play. Here, the decisions on when to intervene are not influenced by the explanations. They are simply rule-based (i.e. choosing the first wrong actions). A more interesting that I would suggest is to let a human observe the explanations and decide when to intervene depending on whether the explanation is correct or not. For example, if the agent is not choosing a subgoal for a wrong reason (e.g. it is miscalculating its properties), the humans can give the correct reason, helping the agent alter its decision. \n\nA concern about the results: in Figure 2 top table, in the third row, the agent should also be trained with the supervised learning loss but its performance is not much better than the non-learned baseline. I am surprised that the supervised learning loss does not help at all. The L_comp is by nature also a kind of supervised learning loss but it boosts performance dramatically. What makes the difference? Could you explain this result?\n\n**Clarity**:\n\nOverall, the paper is easy to follow. I really like the structure of the paper, where the clear goal is set upfront. There are a few points that need clarification (which I also request in the rebuttal):\n- Description of the problem: since there are many types of navigation tasks in the literature, which one is studied in this paper? How are the navigation goals expressed (by language, image, x-y coordinates, distance to goal?)? How do you generate these goals (is there a static dataset?)? What are the training and testing conditions? Are the agent trained and tested on the same or different goals?\n- Description of the environment: what is given as the state of the agent (the map, the first-person view, or both?)?What are the actions of the agent? in addition to the high-level subgoals, does the agent have low-level actions? or does it simply teleport between subgoals?\n- The image of the plans are hard to understand because the annotations are not explained. What do G, R, the numbers, and the letters, the colors represent? In figure 1(2), assigning letters to subgoals while referring to them using numbers is confusing. Overall, I couldn't not understand what plans are being depicted by these figures. \n- The \"Quantitative Interventions via Explanations\" section is missing important details. What does \"accept the changes\" mean mathematically? Does that mean you update the agent's model using the explanation gradient? Importantly, does the update only last during a current time step (and the agent is reset to the old parameters) or does it permanently change the agent then on? What are the changes here (from the agent's decisions to the oracle's decisions?)?\n\n**Significance**: The paper is an attempt to extend existing explanation techniques to partially-observed navigation tasks. However, additional evaluation is required to demonstrate the effectiveness of the generated explanations in communicating the agent's intent to humans. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I suggest the authors tailor the presentation of the explanations to make them more human-friendly. Evaluating with humans is essential to keep track of the progress. The authors mention about potentially biases when evaluating with humans. However, instead avoiding human evaluation completely, the authors can also find ways to mitigate those biases through explicit instructions or obtaining a diverse pool of human judges.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for generating counterfactual explanations for the decisions of a high-level planner that navigates an agent through a partially known environment. The planner reasons over subgoals and the proposed method performs gradient descent over the parameters of these subgoals to identify changes that cause the planner to choose one subgoal over another. A subsequent round of gradient descent is performed with all but one parameter fixed in an effort to filter out false positives. The results are converted to a textual explanation using a rule-based grammar. The quality of the explanations is evaluated by using them to train the navigational agent following much the same procedure that is used to identify the explanations.

The problem of generating a valid and concise explanation for why a policy/planner chose one action over another is challenging and touches on broader problems in interpretability/explainability that are of interest to many in the Ml community. The reviewers agree that the proposed approach is carefully designed and is technically sound. The method is presented clearly and the paper, as a whole, is well written. The experimental evaluation provides a compelling demonstration of the utility of the resulting explanations. As the reviewers note, while this evaluation is deliberately not subjective, a human evaluation of the explanations would provide more insight into the method's broader utility. However, this can be left for future work.
Decision: Accept (Poster)