Generating High-Quality Explanations for Navigation in Partially-Revealed Environments

If we are to welcome robots into our homes and trust them to make decisions on their own , they should be able to clearly explain both what they are doing and why they are doing it in terms that both expert and non-expert users can understand . As learning ( particularly deep learning ) is increasingly used to inform decisions , it is equally important to ensure that data-driven systems are easy to inspect and audit both during development and after deployment [ 33 , 6 ] . The field of explanatory artificial intelligence has developed significantly in recent years to address this need in the hopes of improving reliability and combating problematic data biases that can go unnoticed when decision-making is opaque . Myriad explainability tools aim to probe deep neural networks trained via supervised learning , often with a focus on understanding perceptual tasks , like detecting objects in images [ 24 , 30 , 28 , 31 , 38 ] . Yet in many real-world navigation and home care scenarios , robots are expected to make effective decisions even when their knowledge of the world is incomplete or stale ; in this case , planning effectively requires making predictions about unseen space and about the potential impact of actions far into the future [ 39 , 29 , 3 ] , complicating efforts to apply existing tools to explain robot behavior in this setting . In this work , we focus on the challenging task of navigation in partially-revealed environments , in which reaching an unseen goal in minimum expected time may require recognizing subtle visual cues—e.g. , that green markings signal a route likely to reach the goal—and incorporating multiple such observations when deciding where to explore next . Designing an agent that both achieves state-of-the-art performance and can meaningfully explain its actions has so far proven out of reach for existing approaches to planning under uncertainty . Symbolic abstraction—in which actions correspond to interaction with symbols ( e.g. , objects ) or movement through the environment—is an effective tool for planning in fully-known environments and yields plans that are interpretable-by-design , since symbolic task plans are human-understandable by construction [ 25 , 15 ] . Yet when planning under uncertainty , it is not always obvious how to abstract knowledge without problematically degrading performance , particularly when the agent perceives the world via high-dimensional sensor observations , like images . Owing to challenges associated with data and computation , many state-of-the-art approaches to planning under uncertainty learn the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . relationship between raw sensor observations and good behavior . Deep reinforcement learning has proven incredibly effective ( if somewhat brittle [ 7 ] ) in this regard [ 41 , 36 , 1 , 39 ] . However , many such strategies are opaque by their nature and are neither well-suited to explaining themselves nor compatible with most existing tools for post-hoc explanation [ 12 , 25 ] . Clearly a shift in representation is necessary : we seek an agent for which decisions are expressed in terms of symbols ( so that planning is interpretable ) yet can also make use of learning to ease the onerous computational and data requirements of planning under uncertainty . It is a key insight of this work that the learning over subgoals planning paradigm of Stein et al . [ 29 ] , which blends symbolic and data-driven planning , is well-suited to meet this need . In [ 29 ] , symbolic actions each correspond to revealing space beyond frontiers , boundaries between free and unknown space , while learning is used to estimate compact statistics of unseen space that capture the impact of those actions—e.g. , the likelihood an exploratory action will succeed in reaching the goal . We leverage this representation to develop a new approach to generate counterfactual explanations that communicate what changes to these estimated statistics would result in a desired change in behavior . In this work , we introduce a novel approach to navigation in partially-revealed environments by which an agent can achieve state-of-the-art performance and also generate high quality humanunderstandable explanations of its behavior , despite relying on learning ( from visual observations ) to inform decision making . Specifically , our contributions address the following three questions : 1 . What criteria must a planner satisfy if it is to be useful for generating explanations in partially revealed environments ? [ Sec . 2 ] Our criteria ( motivated by those of Lipton [ 15 ] ) define what it means for planning to be interpretable-by-design for our problem setting . 2 . How can we generate explanations of the agent ’ s high-level behavior ? [ Sec . 3 ] We show that the subgoal-based planner of Stein et al . [ 29 ] meets our conditions for interpretable planning under uncertainty and , leveraging this representation , we introduce a procedure to generate counterfactual explanations . Explanations outline what changes to the agent ’ s belief would result in a change in behavior , expressed in terms of estimated interpretable statistics of unknown space : e.g. , the likelihood an exploratory action will reach an unseen goal . 3 . How can we validate that our explanations are high quality ? [ Sec . 4 ] We introduce a novel training process in which training data is generated via the same procedure as our explanations . This allows us to use planning performance as an objective measure of explanation quality . We evaluate the performance of our approach in two different simulated environments in which an agent is tasked to navigate to an unseen point goal in minimum expected time . We generate explanations in two different simulated environment classes and show that the explanations qualitatively match human intuition for how to correct the agent ’ s poor behavior in these domains . Our quantitative results ( Sec . 5 ) focus on the ability of our explanations to facilitate improving the agent ’ s behavior and show that our joint training-by-explaining procedure demonstrates state-of-the-art performance at test time , validating that our explanations are accurate and informative and demonstrating that an agent can be both explainable and performant simultaneously in this application domain . 1 Preliminaries : planning in partially-revealed environments . A common way to model planning under uncertainty is as a Partially Observable Markov Decision Process ( POMDP ) [ 9 , 16 , 21 ] . In this model , the agent ’ s behavior at time t is a function of its belief bt , a distribution over possible states of the world , which the agent updates as it receives new observations through interaction with the world . In a navigation context , the agent ’ s objective is to minimize expected cost ( travel distance ) to reach an unseen goal . Belief-space planning in general is incredibly computationally intensive . Planning ( typically via the Bellman equation [ 21 ] ) requires that the agent iteratively envision how it might take action , gain new knowledge , update its belief , and then take additional actions . In the worst case , the tree of possible actions and subsequent outcomes stemming from the current belief state grows exponentially with planning horizon ; planning far into the future is fraught with challenges . In this work , we study a particular class of POMDP recently coined as a Locally Observable Markov Decision Process [ 18 ] , in which the agent ’ s local perception is assumed to be perfect though limited by range and occlusions . This is a common problem setting for agents equipped with either depth sensors or LIDAR and considerably simplifies planning through observed space—space the agent has seen and is therefore assumed to know with perfect precision . At every timestep , the agent receives its location , an occlusion-limited local occupancy map ( as if Code available at https : //github.com/RAIL-group/xai-nav-under-uncertainty-neurips2021 from a laser range finder ) , and an egocentric panoramic image of its surroundings and tasked to reach a goal , specified as a world-space coordinate , in minimum time . Even with perfect local perception , planning that requires the agent to reason about unseen space is still incredibly complex . 2 Interpretable-by-design planning in partially-revealed environments . Criterion for interpretable planning There are myriad choices of representations suitable for easing the challenges of planning under uncertainty—as evidenced by our earlier discussion—yet we seek in particular strategies that are said to be interpretable , a precondition for generating explanations [ 6 ] . A number of recent works consider what is implied by interpretability and propose taxonomies and criteria by which we can understand how interpretable a model is [ 6 , 15 , 25 , 33 ] . Yet these works are broad in scope , requiring that we contextualize their criteria for our problem of interest . We propose the following criteria for evaluating the interpretability of a model that relies on learning to plan under uncertainty , motivated by those of Lipton [ 15 ] , which will later serve as desiderata : 1 . Decisions available to the agent should reflect how a human might describe their own decision-making given the same information . Low-level actions like motion primitives are not typically conducive for interpreting the behavior of an agent in an environment that requires thousands of steps to reach the goal . For long-horizon planning , asking Why did the agent decide to turn 5◦ versus 10◦ degrees left ? is unlikely to yield useful explanations of the agent ’ s decision-making process . High-level actions instead define temporally extended behaviors ( e.g. , entering a classroom or following a hallway ) and often correspond to human-understandable symbols , with which planning can better match human intuition . This condition is a prerequisite to Lipton ’ s idea of decomposability , that each step of planning be intuitively understandable , and is consistent with a significant body of work leveraging symbolic abstraction [ 5 , 10 ] . 2 . High-level planning should rely on a model and explicitly consider the impact of the agent ’ s actions into the future . Model-free planning strategies for planning under uncertainty , like the impressive MERLIN agent [ 39 ] , directly predict the goodness of an action , implicitly estimating the impact that action will have on the agent ’ s belief and thereby complicating efforts to understand the agent ’ s behavior . Yet in many planning scenarios , humans seem to explain their decisions by explicitly enumerating the possible outcomes of actions that enter unknown space : e.g. , whether a hallway will lead to a faraway goal . If we are to understand why an agent chose one action over another , its decision-making should match this structure ; for planning under uncertainty to be interpretable and to be useful for generating explanations , the planner should ( like a human ) explicitly consider the different ways in which executing an action might accrue cost and update the belief . This condition is consistent with Lipton ’ s algorithmic transparency . 3 . The impact of actions , particularly those that enter unknown space , should be expressible via a small number of human-understandable quantities . That the plan be compact is important for human understanding . Yet the computational and memory requirements of belief-space planning for large long-horizon tasks are prohibitive even for a computer , and such planning fails to meet the interpretability criterion Lipton refers to as simulatability . The model employed by the agent must therefore dramatically simplify the belief and the process of imagining the future . Subgoal-based abstractions for interpretable navigation under uncertainty It is a key observation of this work that the learning over subgoals planning paradigm of Stein et al . [ 29 ] satisfies our criteria for interpretable-by-design goal-directed navigation under uncertainty ; we provide an overview of this approach here . Under this abstraction , subgoals each correspond to a high-level action to reveal unknown space by exploring beyond a frontier , a boundary between free and unseen space . The high-level outcome of each action is binary : an action either succeeds , if the agent discovers a route to the unseen goal , or fails , if the region is a dead-end , prompting the agent to turn back . The expected cost of each action at ( corresponding to subgoal st ) depends on its likelihood of success PS ( at ) and the expected costs associated with success RS ( at ) and failure ( exploration ) RE ( at ) . These terms ( collectively , the subgoal properties σ ) implicitly1 depend on the belief bt and ( intractable to compute directly ) are estimated via learning . The costs of navigating between subgoals involves travel through observed space and can be computed via Dijkstra ’ s algorithm : D ( bt , at ) . The expected cost of executing high-level action at can be expressed via an approximation of the Bellman Equation 1We note that Stein et al . [ 29 ] make the dependency of the subgoal properties PS , RS , and RE on the belief bt explicit . Eq . ( 1 ) reflects our simplified notation . written in terms of the high-level actions and their binary outcomes : Q ( bt , at ) = D ( bt , at ) + PS ( at ) RS ( at ) + ( 1− PS ( at ) ) [ RE ( at ) + min a∈A ( bt ) \at Q ( b̃t+1 , a ) ] , ( 1 ) where b̃t+1 is the approximate future belief , which incorporates the knowledge that upon executing at and failing to reach the goal the agent has moved ( and is now at subgoal st ) and that the region beyond st is now known to be a dead end ( even if we do not yet know precisely what it looks like ) . The map of observed space is not directly updated during planning , avoiding many of the computational challenges of updating the belief . Critically , the agent ’ s high-level plan defines the order in which it aims to explore the unseen regions of the environment in search of the goal specified as a list of subgoal-actions . We refer the reader to [ 29 ] for a full derivation of Eq . ( 1 ) . Planning with this model is iterative . Upon selecting a subgoal via Eq . ( 1 ) , a low-level planner plots a trajectory through the known map towards the chosen subgoal and the agent executes a short motion primitive along this trajectory . At every timestep , new space may be revealed , and Eq . ( 1 ) must again used to select the subgoal with lowest expected cost . This process repeats until the agent reaches the goal , which can take thousands of steps . Since the agent is assumed to have perfect local perception , the agent ’ s ability to reach the goal is limited only by vehicle dynamics . Given estimates of the subgoal properties ( PS , RS , and RE ) for each subgoal-action available to the agent , high-level planning via Eq . ( 1 ) satisfies our criteria for interpretable planning under uncertainty . First , the decisions available to the agent correspond to choosing between exploring different regions of space , clearly human-understandable . In addition , the impact of the agent ’ s actions ( whether it reaches the goal and the expected cost of each outcome ) are easy to interpret and relatively few in number . Finally , planning is done via a model , under which the impact of each action are explicitly taken into account and can influence the agent ’ s behavior far into the future . Estimating subgoal properties from images Too difficult to compute exactly , the subgoal properties PS , RS , and RE needed to compute expected cost are estimated via a neural network . Consistent with the approach of [ 29 ] , subgoal properties are estimated from single sensor observations ; thus , the input data used to estimate the subgoal properties consists of ( 1 ) a single panoramic image , ( 2 ) the relative position of the goal in the image frame , and ( 3 ) the relative position of the subgoal in the image frame , all collected at the time the subgoal was created—i.e. , the time at which the subgoal ’ s corresponding frontier ( a boundary between free and unknown space ) was revealed . A feed-forward ( non-recurrent ) neural network outputs the three quantities of interest ; a logistic sigmoid function is applied to produce the probability of success PS , so that it is a valid probability . The neural network Algorithm 1 : Generate Explanation Data : mt , qt , ac , ah , nn_inputs , θ0 Result : Subgoal property changes , ∆σ 1 σ0 ← NN ( nn_inputs , θ0 ) 2 ∆Q0 ← ∆Q ( { mt , qt , σ0 } , { ah , ac } ) 3 θ ← θ0 , ∆Q← ∆Q0 4 while ∆Q > 0 do 5 σ ← NN ( nn_inputs , θ ) 6 ∆Q← ∆Q ( { mt , qt , σ } , { ah , ac } ) 7 α← α ( ∆Q0 , σ0 , θ0 ) 8 Lcomp ← ∆Q 9 θ ← θ−η ∂Lcomp ∂σ · [ ∇θσ ◦ 1α > α ( M ) ] 10 σf ← NN ( nn_inputs , θ ) 11 return σf − σ0 Algorithm 2 : Train Subgoal Property Estimator Data : dataset , θ0 Result : Neural Network Weights , θf 1 θ ← θ0 2 foreach datum ∈ dataset do 3 mt , qt , ao , nn_inputs← datum 4 ac ← arg mina∈A ( mt ) \ao Q ( { mt , qt , σ } , a ) 5 σ ← NN ( nn_inputs , θ ) 6 ∆Q← ∆Q ( { mt , qt , σ } , { ao , ac } ) . Eq . ( 2 ) 7 α← α ( ∆Q , σ , θ ) . Eq . ( 3 ) 8 Lcomp ← √ 1− logsigmoid ( −∆Q ) 9 θ ← θ − η ∂Lcomp ∂σ · [ ∇θσ ◦ 1α > α ( M ) ] . Eq . ( 4 ) −η∇θLsupervised − η∇θLbounds

Summary: This paper presents a method for deriving counterfactual explanations of model decisions (relative to the correct decision the agent should have made). The general method is that by performing gradient descent over the model parameters until its decision changes, a set of properties is identified as important features which the agent had mistakenly used to make its decision. To ensure that a minimum number of properties is identified (only the important ones), a second pass of gradient descent is used fixing all but these properties to ensure they still result in a change in the decision. The method is evaluated by training a model using these explanations.