Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations

1 INTRODUCTION As AI systems are increasingly being deployed in scenarios involving high-stakes decisions , the issue of interpretability of their decisions to the humans in the loop has acquired renewed urgency . Most work on interpretability has hither-to focused on one-shot classification tasks , and has revolved around ideas such as marking regions of the input instance ( e.g . image ) that were most salient in the classification of that instance ( Selvaraju et al. , 2016 ; Sundararajan et al. , 2017 ) . Such methods used for one-shot classification tasks don ’ t generalize well to sequential decision tasks–that are the focus of this paper . In particular , in sequential decision tasks , humans in the loop might ask more complex “ contrastive ” explanatory queries–such as why the system took one particular course of action instead of another . The AI system ’ s answers to such questions will often be tied to its internal representations and reasoning , traces of which are not likely to be comprehensible as explanations to humans in the loop . Some have viewed this as an argu- ment to make AI systems use reasoning over symbolic models . We don ’ t take a position on the methods AI systems use to arrive at their decisions . We do however think that the onus of making their decisions interpretable to humans in the loop in their own vocabulary should fall squarely on the AI systems . Indeed , we believe that orthogonal to the issue of whether AI systems use internal symbolic representations to guide their own decisions , they need to develop local symbolic models that are interpretable to humans in the loop , and use them to provide explanations for their decisions . Accordingly , in this work , we develop a framework that takes a set of previously agreed-upon user vocabulary terms and generates contrastive explanations in these terms Miller ( 2018 ) ; Hesslow ( 1988 ) . Specifically , we will do this by learning components of a local symbolic dynamical model ( such as PDDL ( McDermott et al. , 1998 ) ) that captures the agent model in terms of actions with preconditions and effects from the user-specified vocabulary . There is evidence that such models conform to folk psychology Malle ( 2006 ) . This learned local model is then used to explain the potential infeasibility or suboptimality of the alternative raised by the user . Figure 1 presents the flow of the proposed explanation generation process in the context of Montezuma ’ s Revenge ( Wikipedia contributors , 2019 ) . In this paper , we will focus on deterministic settings ( Section 3 ) , though we can easily extend the ideas to stochastic settings , and will ground user-specified vocabulary terms in the AI system ’ s internal representations via learned classifiers ( Section 4 ) . The model components required for explanations are learned by using experiences ( i.e . state-action-state sets ) sampled from the agent model ( Section 4 & 5 ) . Additionally , we formalize the notion of local symbolic approximation for sequential decision-making models , and introduce the idea of explanatory confidence ( Section 6 ) . The exaplanatory confidence captures the fidelity of explanations and help ensure the system only provides explanation whose confidence is above a given threshold . We will evaluate the effectiveness of the method through both systematic ( IRB approved ) user studies and computational experiments ( Section 7 ) . As we will discuss in Section 2 , our approach has some connections to ( Kim et al. , 2018 ) , which while focused on one-shot classification tasks , also advocates explanations in terms of concepts that have meaning to humans in the loop . Our approach of constructing local model approximations is akin to LIME ( Ribeiro et al. , 2016 ) , with the difference being that we construct symbolic dynamical models in terms of human vocabulary , while LIME constructs approximate linear models over abstract features that are machine generated . 2 RELATED WORK . The representative works in the direction of using concept level explanation include works like ( Bau et al. , 2017 ) , TCAV ( Kim et al. , 2018 ) and its various offshoots like ( Luss et al. , 2019 ) that have focused on one-shot decisions . These works take a line quite similar to us in that they try to create explanations for current decisions in terms of a set of user-specified concepts . While these works don ’ t explicitly reason about explanatory confidence they do discuss the possibility of identifying inaccurate explanation , and tries to address them through statistical tests . Another thread of work , exemplified by works like ( Koh et al. , 2020 ; Lin et al. , 2020 ) , tries to force decisions to be made in terms of user-specified concepts , which can then be used for explanations . There have also been recent works on automatically identifying human-understandable concepts like Ghorbani et al . ( 2019 ) and Hamidi-Haines et al . ( 2018 ) . We can also leverage these methods when our system identifies scenarios with insufficient vocabulary . Most works in explaining sequential decision-making problems either use a model specified in a shared vocabulary as a starting point for explanation or focus on saliency-based explanations ( cf . ( Chakraborti et al. , 2020 ) ) , with very few exceptions . Authors of ( Hayes & Shah , 2017 ) have looked at the use of high-level concepts for policy summaries . They use logical formulas to concisely characterize various policy choices , including states where a specific action may be selected ( or not ) . Unlike our work , they are not trying to answer why the agent chooses a specific action ( or not ) . ( Waa et al. , 2018 ) looks at addressing the suboptimality of foils while supporting interpretable features , but it requires the domain developer to assign positive and negative outcomes to each action . In addition to not addressing possible vocabulary differences between a system developer and the end-user , it is also unclear if it is always possible to attach negative and positive outcomes to individual actions . Another related work is the approach studied in ( Madumal et al. , 2020 ) . Here , they are also trying to characterize dynamics in terms of high-level concepts but assume that the full structural relationship between the various variables is provided upfront . The explanations discussed in this paper can also be seen as a special case of Model Reconciliation explanation ( c.f ( Chakraborti et al. , 2017 ) ) , where the human model is considered to be empty . The usefulness of preconditions as explanations has also been studied by works like ( Winikoff , 2017 ; Broekens et al. , 2010 ) . Our effort to associate action cost to concepts could also be contrasted to efforts in ( Juozapaitis et al. , 2019 ) and ( Anderson et al. , 2019 ) which leverage interpretable reward components . Another group of works popular in RL explanations is the ones built around saliency maps ( Greydanus et al. , 2018 ; Iyer et al. , 2018 ; Puri et al. , 2019 ) , which tend to highlight parts of the state that are important for the current decision . In particular , we used ( Greydanus et al. , 2018 ) as a baseline because many follow-up works have shown their effectiveness ( cf . ( Zhang et al. , 2020 ) ) . Readers can refer to Alharin et al . ( 2020 ) for a recent survey of explanations in RL . Another related thread of work , is that of learning models ( some representative works include ( Carbonell & Gil , 1990 ; Stern & Juba , 2017 ; Wu et al. , 2007 ) ) . To the best of our knowledge none of the works in this direction allow for noisy observations of state and none focused on identifying specific model components . While we are unaware of any works that provide confidence over learned model components , ( Stern & Juba , 2017 ) provides loose PAC guarantees over the entire learned model . 3 PROBLEM SETTING . Our setting consists of an agent , be it programmed or RL-based , that has a model of the dynamics of the task that is inscrutable to the user ( in so far that the user can ’ t directly use the model representation used by the agent ) in the loop and uses a decision-making process that is sound for this given model . Note that here the term model is being used in a very general sense . These could refer to tabular models defined over large atomic state spaces , neural network-based dynamics models possibly learned over latent representation of the states and even simulators . The only restriction we place on the model is that we can sample possible experiences from it . Regardless of the true representation , we will denote the model by the tuple M = ⟨S , A , T , C⟩ . Where S and A are the state and action sets and T : S×A → S ∪ { ⊥ } ( ⊥ the absorber failure state ) and C : S×A → R capture the transition and cost function . We will use ⊥ to capture failures that could occur when the agent violates hard constraints like safety constraints or perform any invalid actions . We will consider goal-directed agents that are trying to drive the state of the world to one of the goal states ( G being the set ) from an initial state I . The solution takes the form of a sequence of actions or a plan π . We will use symbolic action models with preconditions and cost functions ( similar to PDDL models ( Geffner & Bonet , 2013 ) ) as a way to approximate the problem for explanations . Such a model can be represented by the tuple MS = ⟨FS , AS , IS , GS , CS⟩ , where FS is a set of propositional state variables defining the state space , AS is the set of actions , IS is the initial state , GS is the goal specification . Each valid problem state is uniquely identified by the subset of state variables that are true in that state ( so for any state s ∈ SMS , where SMS is the set of states for MS , s ⊆ FS ) . Each action a ∈ AS is further described in terms of the preconditions preca ( specification of states in which a is executable ) and the effects of executing the action . We will denote the state formed by executing action a in a state s as a ( s ) . We will focus on models where the preconditions are represented as a conjunction of propositions . If the action is executed in a state with missing preconditions , then the execution results in the invalid state ⊥ . Unlike standard STRIPS models , where the cost of executing action is independent of states , we will use a state-dependent cost function of the form CS : 2F ×AS → R to capture forms of cost functions popular in RL benchmarks . 4 CONTRASTIVE EXPLANATIONS . The specific explanatory setting , illustrated in Figure 1 , is one where the agent comes up with a plan π ( to achieve one of the goals specified in G from I ) and the user responds by raising an alternate plan πf ( the foil ) that they believe should be followed instead . Now the system needs to explain why π may be preferred over πf , by showing that πf is invalid ( i.e. , πf doesn ’ t lead to a goal state or one of the action in πf results in the invalid state ⊥ ) or πf is costlier than π ( C ( I , π ) < C ( I , πf ) ) .1 To concretize this interaction , consider the modified version of Montezuma ’ s Revenge ( Figure 1 ) . The agent starts from the highest platform , and the goal is to get to the key . The specified plan π may require the agent to make its way to the lowest level , jump over the skull , and then go to the key with a total cost of 20 . Now the user raises two possible foils that are similar to π but use different strategies in place of jumping over the skull . Foil1 : instead of jumping , the agent just moves left ( i.e. , it tries to move through the skull ) and , Foil2 : instead of jumping over the skull , the agent performs the attack action ( not part of the original game , but added here for illustrative purposes ) and then moves on to the key . Using the internal model , the system can recognize that in the first case , moving left would lead to an invalid state , and in the second case , the foil is costlier , though effectively communicating this to the user is a different question . If there exists a shared 1If the foil is as good as the original plan or better , then the system could switch to the foil . visual communication channel , the agent could try to demonstrate the outcome of following these alternate strategies . Unfortunately , this would not only necessitate additional cognitive load on the user ’ s end to view the demonstration , but it may also be confusing to the user in so far that they may not be able to recognize why in a particular state the move left action was invalid and attack action costly . As established in our user study and pointed out by Atrey et al . ( 2019 ) , even highlighting of visual features may not effectively resolve this confusion . This scenario thus necessitates the use of methods that are able to express possible explanations in terms that the user may understand . Learning Concept Maps : The input to our system is the set of propositional concepts the user associates with the task . For Montezuma , this could involve concepts like agent being on a ladder , holding onto the key , being next to the skull . Each concept corresponds to a propositional fact that the user associates with the task ’ s states and believes their presence or absence in a state could influence the dynamics and the cost function . We can collect such concepts from subject matter experts as by Cai et al . ( 2019 ) , or one could just let the user interact with or observe the agent and then provide a possible set of concepts . We used the latter approach to collect the propositions for our evaluation of the Sokoban domains ( Section 7 and A.7 ) . Each concept corresponds to a binary classifier , which detects whether the proposition is present or absent in a given internal state ( thus allowing us to convert the atomic states into a factored representation ) . Let C be the set of classifiers corresponding to the high-level concepts . For state s ∈ S , we will overload the notation C and specify the concepts that are true as C ( s ) , i.e. , C ( s ) = { ci|ci ∈ C ∧ ci ( s ) = 1 } ( where ci is the classifier corresponding to the ith concept and we overload the notation to also stand for the label of the ith concept ) . The training set for such concept classifiers could come from the user ( where they provide a set of positive and negative examples per concept ) . Classifiers can be then learned over the model states or the internal representations used by the agent decision-making ( for example activations of intermediate neural network layers ) . Explanation using concepts : To explain the preference of plan π over foil πf , we will present model information to the user taken from a symbolic representation of the agent model . But rather than requiring this model to be an exact representation of the complete agent model , we will instead focus on accurately capturing a subset of the model by instead trying to learn a local approximation Definition 1 A symbolic model MCS = ⟨C , ACS , C ( I ) , C ( G ) , CCS ⟩ . is said to be a local symbolic approximation of the model MR = ⟨S , A , T , C⟩ for regions of interest Ŝ ⊆ S if ∀s ∈ Ŝ and ∀a ∈ A , we have an equivalent action aC ∈ ACS , such that ( a ) aC ( C ( s ) ) = C ( T ( s , a ) ) ( assuming C ( ⊥ ) = ⊥ ) and ( b ) CCS ( C ( s ) , a ) = C ( s , a ) and ( c ) C ( G ) = ⋂ sg∈G∩Ŝ C ( sg ) . Following Section 3 , this is a PDDL-style model with preconditions and conditional costs defined over the conjunction of positive propositional literals . A.2 establishes the sufficiency of this representation to capture arbitrarily complex preconditions ( including disjunctions ) and cost functions expressed in terms of the proposition set C. Also to establish the preference of plan does not require informing the users about the entire model MCS , but rather only the relevant parts . To establish the invalidity of πf , we only need to explain the failure of the first failing action ai , i.e. , the one that resulted in the invalid state ( for Foil1 this corresponds to move-left action at the state visualized in Figure 1 ) . We explain the failure of action in a state by pointing out a proposition that is an action precondition which is absent in the given state . Thus a concept ci ∈ C is considered an explanation for failure of action ai at state si , if ci ∈ precai \ C ( si ) . For Foil1 , the explanation would be – the action move-left failed in the state as the precondition skull-not-on-left was false in the state . This formulation can also capture failure to achieve goal by appending an additional goal action at the end of the plan , which causes the state to transition to an end state , and fails for all states except the ones in G. Note that instead of identifying all the missing preconditions , we focus on identifying a single precondition , as this closely follows prescriptions from works in social sciences that have shown that selectivity or minimality is an essential property of effective explanations ( Miller , 2018 ) . For explaining suboptimality , we inform the user about the symbolic cost function CCS . To ensure minimality , rather than provide the entire cost function , we will instead try to learn and provide an abstraction of the cost function Cabss Definition 2 For the symbolic model MCS = ⟨C , ACS , C ( I ) , C ( G ) , CCS⟩ , an abstract cost function CabsS : 2C ×ACS → R is specified as : CabsS ( { c1 , .. , ck } , a ) = min { CCS ( s , a ) |s ∈ SMCS ∧ { c1 , .. , ck } ⊆ s } . Intuitively , CabsS ( { c1 , .. , ck } , a ) = k can be understood as stating that executing the action a , in the presence of concepts { c1 , .. , ck } costs at least k. We can use CabsS in an explanation by identifying a sequence of concept set Cπf = ⟨Ĉ1 , ... , Ĉk⟩ , corresponding to each step of the foil πf = ⟨a1 , .. , ak⟩ , such that ( a ) Ĉk is a subset of concepts in the corresponding state reached by the foil and ( b ) the total cost of abstract cost function defined over the concept subsets are larger than the plan cost∑ i= { 1 .. k } CabsS ( Ĉi , ai ) > C ( I , π ) . For Foil2 , the explanation would include the information – executing the action attack in the presence of the concept skull-on-left , will cost at least 500 .

Summary: This paper proposes a symbolic model implemented alongside a planning AI, which can respond to human  queries about why one plan is taken by the AI, and not another.  The system takes as input a user concept vocabulary, where each concept is implemented by binary classifier indicating whether the proposition is present. This concept vocabulary needs to be initially solicited from the user -- either through user surveys or designed by researchers.  The output of the classifiers is noisy, which is interpreted as confidence over the proposed explanation. The system then uses a symbolic PPDL model with the concept vocabulary to answer queries. The system can issue two types of answers about why a proposed plan is inadmissible: (1) a pre-condition for a given action is missing or (2) the proposed plan is more costly than the chosen plan.  To identify if a concept is a precondition for action A, the system samples a number of states from the model where A is executable. If concept is always present, then it is a pre-condition. Samples are generated by a random walk from one of the states in the model's given plan. In the evaluated problems games the sampling algorithm needs ~500 samples to accurately identify the missing pre-condition. To identify the cost associated with executing a proposed action, a heuristic is used, such that tries to sample the minimal cost when executing the action given the preconditions. (This is not clearly explained )   A small user study was used to evaluate whether the subjects find the proposed explanations helpful.