Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations

1 INTRODUCTION As AI systems are increasingly being deployed in scenarios involving high-stakes decisions , the issue of interpretability of their decisions to the humans in the loop has acquired renewed urgency . Most work on interpretability has hither-to focused on one-shot classification tasks , and has revolved around ideas such as marking regions of the input instance ( e.g . image ) that were most salient in the classification of that instance ( Selvaraju et al. , 2016 ; Sundararajan et al. , 2017 ) . Such methods used for one-shot classification tasks don ’ t generalize well to sequential decision tasks–that are the focus of this paper . In particular , in sequential decision tasks , humans in the loop might ask more complex “ contrastive ” explanatory queries–such as why the system took one particular course of action instead of another . The AI system ’ s answers to such questions will often be tied to its internal representations and reasoning , traces of which are not likely to be comprehensible as explanations to humans in the loop . Some have viewed this as an argu- ment to make AI systems use reasoning over symbolic models . We don ’ t take a position on the methods AI systems use to arrive at their decisions . We do however think that the onus of making their decisions interpretable to humans in the loop in their own vocabulary should fall squarely on the AI systems . Indeed , we believe that orthogonal to the issue of whether AI systems use internal symbolic representations to guide their own decisions , they need to develop local symbolic models that are interpretable to humans in the loop , and use them to provide explanations for their decisions . Accordingly , in this work , we develop a framework that takes a set of previously agreed-upon user vocabulary terms and generates contrastive explanations in these terms Miller ( 2018 ) ; Hesslow ( 1988 ) . Specifically , we will do this by learning components of a local symbolic dynamical model ( such as PDDL ( McDermott et al. , 1998 ) ) that captures the agent model in terms of actions with preconditions and effects from the user-specified vocabulary . There is evidence that such models conform to folk psychology Malle ( 2006 ) . This learned local model is then used to explain the potential infeasibility or suboptimality of the alternative raised by the user . Figure 1 presents the flow of the proposed explanation generation process in the context of Montezuma ’ s Revenge ( Wikipedia contributors , 2019 ) . In this paper , we will focus on deterministic settings ( Section 3 ) , though we can easily extend the ideas to stochastic settings , and will ground user-specified vocabulary terms in the AI system ’ s internal representations via learned classifiers ( Section 4 ) . The model components required for explanations are learned by using experiences ( i.e . state-action-state sets ) sampled from the agent model ( Section 4 & 5 ) . Additionally , we formalize the notion of local symbolic approximation for sequential decision-making models , and introduce the idea of explanatory confidence ( Section 6 ) . The exaplanatory confidence captures the fidelity of explanations and help ensure the system only provides explanation whose confidence is above a given threshold . We will evaluate the effectiveness of the method through both systematic ( IRB approved ) user studies and computational experiments ( Section 7 ) . As we will discuss in Section 2 , our approach has some connections to ( Kim et al. , 2018 ) , which while focused on one-shot classification tasks , also advocates explanations in terms of concepts that have meaning to humans in the loop . Our approach of constructing local model approximations is akin to LIME ( Ribeiro et al. , 2016 ) , with the difference being that we construct symbolic dynamical models in terms of human vocabulary , while LIME constructs approximate linear models over abstract features that are machine generated . 2 RELATED WORK . The representative works in the direction of using concept level explanation include works like ( Bau et al. , 2017 ) , TCAV ( Kim et al. , 2018 ) and its various offshoots like ( Luss et al. , 2019 ) that have focused on one-shot decisions . These works take a line quite similar to us in that they try to create explanations for current decisions in terms of a set of user-specified concepts . While these works don ’ t explicitly reason about explanatory confidence they do discuss the possibility of identifying inaccurate explanation , and tries to address them through statistical tests . Another thread of work , exemplified by works like ( Koh et al. , 2020 ; Lin et al. , 2020 ) , tries to force decisions to be made in terms of user-specified concepts , which can then be used for explanations . There have also been recent works on automatically identifying human-understandable concepts like Ghorbani et al . ( 2019 ) and Hamidi-Haines et al . ( 2018 ) . We can also leverage these methods when our system identifies scenarios with insufficient vocabulary . Most works in explaining sequential decision-making problems either use a model specified in a shared vocabulary as a starting point for explanation or focus on saliency-based explanations ( cf . ( Chakraborti et al. , 2020 ) ) , with very few exceptions . Authors of ( Hayes & Shah , 2017 ) have looked at the use of high-level concepts for policy summaries . They use logical formulas to concisely characterize various policy choices , including states where a specific action may be selected ( or not ) . Unlike our work , they are not trying to answer why the agent chooses a specific action ( or not ) . ( Waa et al. , 2018 ) looks at addressing the suboptimality of foils while supporting interpretable features , but it requires the domain developer to assign positive and negative outcomes to each action . In addition to not addressing possible vocabulary differences between a system developer and the end-user , it is also unclear if it is always possible to attach negative and positive outcomes to individual actions . Another related work is the approach studied in ( Madumal et al. , 2020 ) . Here , they are also trying to characterize dynamics in terms of high-level concepts but assume that the full structural relationship between the various variables is provided upfront . The explanations discussed in this paper can also be seen as a special case of Model Reconciliation explanation ( c.f ( Chakraborti et al. , 2017 ) ) , where the human model is considered to be empty . The usefulness of preconditions as explanations has also been studied by works like ( Winikoff , 2017 ; Broekens et al. , 2010 ) . Our effort to associate action cost to concepts could also be contrasted to efforts in ( Juozapaitis et al. , 2019 ) and ( Anderson et al. , 2019 ) which leverage interpretable reward components . Another group of works popular in RL explanations is the ones built around saliency maps ( Greydanus et al. , 2018 ; Iyer et al. , 2018 ; Puri et al. , 2019 ) , which tend to highlight parts of the state that are important for the current decision . In particular , we used ( Greydanus et al. , 2018 ) as a baseline because many follow-up works have shown their effectiveness ( cf . ( Zhang et al. , 2020 ) ) . Readers can refer to Alharin et al . ( 2020 ) for a recent survey of explanations in RL . Another related thread of work , is that of learning models ( some representative works include ( Carbonell & Gil , 1990 ; Stern & Juba , 2017 ; Wu et al. , 2007 ) ) . To the best of our knowledge none of the works in this direction allow for noisy observations of state and none focused on identifying specific model components . While we are unaware of any works that provide confidence over learned model components , ( Stern & Juba , 2017 ) provides loose PAC guarantees over the entire learned model . 3 PROBLEM SETTING . Our setting consists of an agent , be it programmed or RL-based , that has a model of the dynamics of the task that is inscrutable to the user ( in so far that the user can ’ t directly use the model representation used by the agent ) in the loop and uses a decision-making process that is sound for this given model . Note that here the term model is being used in a very general sense . These could refer to tabular models defined over large atomic state spaces , neural network-based dynamics models possibly learned over latent representation of the states and even simulators . The only restriction we place on the model is that we can sample possible experiences from it . Regardless of the true representation , we will denote the model by the tuple M = ⟨S , A , T , C⟩ . Where S and A are the state and action sets and T : S×A → S ∪ { ⊥ } ( ⊥ the absorber failure state ) and C : S×A → R capture the transition and cost function . We will use ⊥ to capture failures that could occur when the agent violates hard constraints like safety constraints or perform any invalid actions . We will consider goal-directed agents that are trying to drive the state of the world to one of the goal states ( G being the set ) from an initial state I . The solution takes the form of a sequence of actions or a plan π . We will use symbolic action models with preconditions and cost functions ( similar to PDDL models ( Geffner & Bonet , 2013 ) ) as a way to approximate the problem for explanations . Such a model can be represented by the tuple MS = ⟨FS , AS , IS , GS , CS⟩ , where FS is a set of propositional state variables defining the state space , AS is the set of actions , IS is the initial state , GS is the goal specification . Each valid problem state is uniquely identified by the subset of state variables that are true in that state ( so for any state s ∈ SMS , where SMS is the set of states for MS , s ⊆ FS ) . Each action a ∈ AS is further described in terms of the preconditions preca ( specification of states in which a is executable ) and the effects of executing the action . We will denote the state formed by executing action a in a state s as a ( s ) . We will focus on models where the preconditions are represented as a conjunction of propositions . If the action is executed in a state with missing preconditions , then the execution results in the invalid state ⊥ . Unlike standard STRIPS models , where the cost of executing action is independent of states , we will use a state-dependent cost function of the form CS : 2F ×AS → R to capture forms of cost functions popular in RL benchmarks . 4 CONTRASTIVE EXPLANATIONS . The specific explanatory setting , illustrated in Figure 1 , is one where the agent comes up with a plan π ( to achieve one of the goals specified in G from I ) and the user responds by raising an alternate plan πf ( the foil ) that they believe should be followed instead . Now the system needs to explain why π may be preferred over πf , by showing that πf is invalid ( i.e. , πf doesn ’ t lead to a goal state or one of the action in πf results in the invalid state ⊥ ) or πf is costlier than π ( C ( I , π ) < C ( I , πf ) ) .1 To concretize this interaction , consider the modified version of Montezuma ’ s Revenge ( Figure 1 ) . The agent starts from the highest platform , and the goal is to get to the key . The specified plan π may require the agent to make its way to the lowest level , jump over the skull , and then go to the key with a total cost of 20 . Now the user raises two possible foils that are similar to π but use different strategies in place of jumping over the skull . Foil1 : instead of jumping , the agent just moves left ( i.e. , it tries to move through the skull ) and , Foil2 : instead of jumping over the skull , the agent performs the attack action ( not part of the original game , but added here for illustrative purposes ) and then moves on to the key . Using the internal model , the system can recognize that in the first case , moving left would lead to an invalid state , and in the second case , the foil is costlier , though effectively communicating this to the user is a different question . If there exists a shared 1If the foil is as good as the original plan or better , then the system could switch to the foil . visual communication channel , the agent could try to demonstrate the outcome of following these alternate strategies . Unfortunately , this would not only necessitate additional cognitive load on the user ’ s end to view the demonstration , but it may also be confusing to the user in so far that they may not be able to recognize why in a particular state the move left action was invalid and attack action costly . As established in our user study and pointed out by Atrey et al . ( 2019 ) , even highlighting of visual features may not effectively resolve this confusion . This scenario thus necessitates the use of methods that are able to express possible explanations in terms that the user may understand . Learning Concept Maps : The input to our system is the set of propositional concepts the user associates with the task . For Montezuma , this could involve concepts like agent being on a ladder , holding onto the key , being next to the skull . Each concept corresponds to a propositional fact that the user associates with the task ’ s states and believes their presence or absence in a state could influence the dynamics and the cost function . We can collect such concepts from subject matter experts as by Cai et al . ( 2019 ) , or one could just let the user interact with or observe the agent and then provide a possible set of concepts . We used the latter approach to collect the propositions for our evaluation of the Sokoban domains ( Section 7 and A.7 ) . Each concept corresponds to a binary classifier , which detects whether the proposition is present or absent in a given internal state ( thus allowing us to convert the atomic states into a factored representation ) . Let C be the set of classifiers corresponding to the high-level concepts . For state s ∈ S , we will overload the notation C and specify the concepts that are true as C ( s ) , i.e. , C ( s ) = { ci|ci ∈ C ∧ ci ( s ) = 1 } ( where ci is the classifier corresponding to the ith concept and we overload the notation to also stand for the label of the ith concept ) . The training set for such concept classifiers could come from the user ( where they provide a set of positive and negative examples per concept ) . Classifiers can be then learned over the model states or the internal representations used by the agent decision-making ( for example activations of intermediate neural network layers ) . Explanation using concepts : To explain the preference of plan π over foil πf , we will present model information to the user taken from a symbolic representation of the agent model . But rather than requiring this model to be an exact representation of the complete agent model , we will instead focus on accurately capturing a subset of the model by instead trying to learn a local approximation Definition 1 A symbolic model MCS = ⟨C , ACS , C ( I ) , C ( G ) , CCS ⟩ . is said to be a local symbolic approximation of the model MR = ⟨S , A , T , C⟩ for regions of interest Ŝ ⊆ S if ∀s ∈ Ŝ and ∀a ∈ A , we have an equivalent action aC ∈ ACS , such that ( a ) aC ( C ( s ) ) = C ( T ( s , a ) ) ( assuming C ( ⊥ ) = ⊥ ) and ( b ) CCS ( C ( s ) , a ) = C ( s , a ) and ( c ) C ( G ) = ⋂ sg∈G∩Ŝ C ( sg ) . Following Section 3 , this is a PDDL-style model with preconditions and conditional costs defined over the conjunction of positive propositional literals . A.2 establishes the sufficiency of this representation to capture arbitrarily complex preconditions ( including disjunctions ) and cost functions expressed in terms of the proposition set C. Also to establish the preference of plan does not require informing the users about the entire model MCS , but rather only the relevant parts . To establish the invalidity of πf , we only need to explain the failure of the first failing action ai , i.e. , the one that resulted in the invalid state ( for Foil1 this corresponds to move-left action at the state visualized in Figure 1 ) . We explain the failure of action in a state by pointing out a proposition that is an action precondition which is absent in the given state . Thus a concept ci ∈ C is considered an explanation for failure of action ai at state si , if ci ∈ precai \ C ( si ) . For Foil1 , the explanation would be – the action move-left failed in the state as the precondition skull-not-on-left was false in the state . This formulation can also capture failure to achieve goal by appending an additional goal action at the end of the plan , which causes the state to transition to an end state , and fails for all states except the ones in G. Note that instead of identifying all the missing preconditions , we focus on identifying a single precondition , as this closely follows prescriptions from works in social sciences that have shown that selectivity or minimality is an essential property of effective explanations ( Miller , 2018 ) . For explaining suboptimality , we inform the user about the symbolic cost function CCS . To ensure minimality , rather than provide the entire cost function , we will instead try to learn and provide an abstraction of the cost function Cabss Definition 2 For the symbolic model MCS = ⟨C , ACS , C ( I ) , C ( G ) , CCS⟩ , an abstract cost function CabsS : 2C ×ACS → R is specified as : CabsS ( { c1 , .. , ck } , a ) = min { CCS ( s , a ) |s ∈ SMCS ∧ { c1 , .. , ck } ⊆ s } . Intuitively , CabsS ( { c1 , .. , ck } , a ) = k can be understood as stating that executing the action a , in the presence of concepts { c1 , .. , ck } costs at least k. We can use CabsS in an explanation by identifying a sequence of concept set Cπf = ⟨Ĉ1 , ... , Ĉk⟩ , corresponding to each step of the foil πf = ⟨a1 , .. , ak⟩ , such that ( a ) Ĉk is a subset of concepts in the corresponding state reached by the foil and ( b ) the total cost of abstract cost function defined over the concept subsets are larger than the plan cost∑ i= { 1 .. k } CabsS ( Ĉi , ai ) > C ( I , π ) . For Foil2 , the explanation would include the information – executing the action attack in the presence of the concept skull-on-left , will cost at least 500 .

Summary: (NB: I reviewed a previous version of this paper. I was in favour of its acceptance. In my opinion, this version is far more clear and addresses the most salient concerns of the reviewers, including me. I'll keep this review self-contained and refer to this submission).  The paper proposes a methodology for providing post-hoc explanations in sequential decision settings without assuming a model of the environment. Usual methods for explainability do not translate well into sequential decision settings, as the explanation is related to the executed actions. In particular, what's the natural form for such explanations? A simple but useful query is to ask about modifications to the plan retuned by the AI agent. These are called constrastive explanations. Those modifications could be a) better, b) lead use some illegal action execution, c) be more expensive.  The method creates a symbolic approximation around the current situation. The approximation allows to show the user a single concept that is missing and makes an action fail, or to increase the cost. The validation with users reports that they are finding them usefull. A more indirect test, less sensitive to the bias of self-report, is whether they can solve a task faster. Those results are also positive.