ATTENTIVE EXPLAINABILITY FOR PATIENT TEMPORAL EMBEDDING

1 INTRODUCTION . The success of AI models to solve real world problems has been hindered by the inability to fully understand the complex mechanisms behind inferences produced by these models . This problem is exacerbated in healthcare where clinicians and other stakeholders consuming AI model outputs need to understand the rationales behind the underlying AI inference processes . Learning explainable representations for patient data has proved to be notoriously difficult and has slowed the adoption of powerful sequence to sequence architectures for this task . In Choi et al . ( 2016a ) , the authors describe an interesting and effective approach for the embedding of medical patient data . However , they avoid RNN like architectures probably able to produce more accurate representations but at the expense of explainability . Explainability in AI in general has been receiving a great deal of attention within the academic community with numerous methodological advances for the learning of models that are easier to digest by humans . In Ribeiro et al . ( 2016 ) and Ribeiro et al . ( 2018 ) , the authors proposed to locally learn simple linear or tree models to approximate the local performance of complex classifiers . Their approach assumes that explainability is heavily related to model complexity . In Choi et al . ( 2016b ) , the authors proposed a Recurrent Neural Network architecture able to compute attention coefficients linking the predictions of the model back to the raw features used by the model at different time points . In this case , explainability is also a local concept that relates to attention mechanisms justifying the predictions made by a model by pointing back to the source data deemed responsible for the predictions . Recently in Dhurandhar et al . ( 2018 ) , the authors proposed to relate explainability not only to the presence of input features responsible for an inference but also to the omission of input features ( e.g. , the man who does not wear glasses ) . In Koh & Liang ( 2017 ) , the authors proposed to use influence functions to understand the internals of an AI model by tracing back predictions to training data instances that have influenced it . While these papers cleverly shed light on how complex AI models are behaving , they all use different informal notions of explainability–may it be complexity , attentiveness linking predictions to input features or training data influence on predictions . They all treat explainability as a qualitative characteristic of AI models and do not rely on well defined measurable aspects of explainability . In this paper , we treat aspects of explainability as quantitative measurable properties of models and propose approaches to measure the degrees of explainability of learned models in a regression setting , with application to computational embedding . In general , formalizing explainability is quite hard . In this work , we only focus on specific aspects of it that are grounded with the following principles : • Observability : We restrict our efforts to the generation of explanations as seen by an external observer witnessing the input-output relationship for a given AI system . This principle is restrictive as it treats any AI agent as a black box . However , we deem it to be a reasonable assumption that humans commonly make , specially in physical sciences when modeling complex physical phenomena from observational data . • Communicability : We consider explainability to be a communication problem , with a dialog between an external AI observer and a rational human actor with the external AI observer explaining the behavior of the AI system to the rational human actor at the receiving end of this communication . The amount of information that needs to be produced by the AI external observer for explainability is consequently inversely proportional to the degree of explainability of the AI system . • Subjectivity : We view explainability as a subjective concept that depends on the mind set of the receiving rational human actor , in terms of his/her preferences and amount of prior knowledge that he/she may have acquired . This principle prompts us to model some notion of preferences and/or assumed prior knowledge available to these human actors with prior distributions . In these cases , AI models are deemed less explainable as they produce predictions based on facts that are harder to explain using generally accepted human prior knowledge . Based on these principles , we model explainability using information theoretic constructs measuring the amount of information that need to be transmitted by an AI agent to a human end user to explain the rationales behind the predictions . The main contributions of this paper are three fold : ( i ) an information theoretic framework to effectively measure aspects of explainability , ( ii ) the application of the framework to regression problems using attention mechanisms , and ( iii ) conclusive experimental results on patient temporal embedding problems . The rest of the paper is organized according to these three contribution areas . 2 EXPLAINABILITY WITH THE MINIMUM DESCRIPTION LENGTH PRINCIPLE . 2.1 NOTATIONS . Throughout the paper , scalars are represented with lower case letters in a non bold typeface , e.g. , x . Sets are represented with upper case letters in a calligraphic typeface , e.g. , H. Vectors are denoted in bold lower case letters e.g. , x 2 Rk from some positive k 2 N. Matrices are represented in bold upper case letters , e.g. , A 2 Rn⇥m , where both n , m 2 N. In general , we use subscripts to index elements in a vector or in a matrix . For instance , xi is a scalar corresponding to the ith element of vector x while Ai , j denotes the i , j entry of matrix A . The data elements in this paper are always attached to a specific entity ( e.g. , a patient in healthcare ) . To avoid notation clutter , we often drop these references to entities and describe the methodology at the entity level . The data for a given entity is represented by a matrix X 2 Rn⇥m . A row Xi , 1 i n represents an m dimensional feature vector collected at discrete time i . Similarly , the output of an AI model processing X is represented by the matrix Y 2 Rn⇥k corresponding to a discrete time series of k dimensional output vectors . 2.2 THE MINIMUM DESCRIPTION LENGTH PRINCIPLE . The Minimum Description Length principle is a formalization of the Occam Razor principle guiding the search for good models that explain data using description lengths ( Grünwald ( 2007 ) , Rissanen ( 1998 ) ) . A simple version of it uses a two-part coding approach to select the `` best '' model Mopt as the one that minimizes the sum of the length in bits for a description of the model ( part one of the code ) and the length in bit of the effective description of the data using using the model . Let M denote the set of all computable models in the Turing sense . The MDL defines the best model as : M opt : = min M2M ( L ( M ) + L ( X|M ) ) ( 1 ) where L is a function that returns description lengths . The MDL approach imposes preferences for models that balance their complexity with their goodness of fit . The first part of the code is essentially the model complexity while the second part measures the goodness of fit for the model on the data . An idealized version of the MDL has been defined in Vitányi & Li ( 2000 ) where L is replaced with K , the Kolmogorov complexity . However , this version is not computable in the Turing sense . In the following sections , we present practical methods to compute estimate L for explainability purposes . 2.3 EXPLAINABILITY METRICS . Following the observability and communicability assumptions described in Section 1 and the MDL principle , we measure the explainability of an AI model M with description lengths of an external observer model only able to perceive the inputs and outputs of M to model this relationship ( See Figure 1 ) . In this setting , an explanation Mo for an AI model M is essentially such an observer model describing how its inputs X and outputs Y are related in a language that is understandable by human actors . Assuming the presence of no additional knowledge for Mo to characterize this input output relationship , we measure the effectiveness of Mo with two part codes as follows : Definition 1 . The explainability provided by an observer Mo of model M is defined as : e ag ( Mo , M , X ) : = L ( Mo ) + L ( M ( X ) |Mo , X ) ( 2 ) The knowledge agnostic explainability of a model M is defined as : E ag ( M , X ) : = min Mo2M e ag ( Mo , M , X ) = min Mo2M [ L ( Mo ) + L ( M ( X ) |Mo , X ) ] ( 3 ) As stated in Section 1 , we also assume that explainability can be subjective and dependent on the mind set of receiving rational human actors , in terms of the amount of prior knowledge that they have acquired . In our framework , this prior knowledge is represented as a prior model Mp that these human actors would use to explain the ( X , Y ) relationship . For instance , an expert physician may explain the general loss of health of a critical care patient from features representing reductions in heart rate variability in the source data . For this physician , an AI model producing rationales that are inline with this prior knowledge would certainly by easier to explain than other AI models using variables that are further from her/his mental mind set . Consequently , we modify the complexity term of the previous definition to come up with the following measures of explainability in the presence of prior knowledge . Definition 2 . With a computable prior knowledge Mp , the explainability provided by an observer M o of model M is defined as : e aw ( Mo , M , Mp , X ) : = L ( Mo|Mp ) + L ( M ( X ) |Mo , X ) ( 4 ) The knowledge aware explainability of a model M is defined as : E aw ( M , Mp , X ) : = min Mo2M e aw ( Mo , Mp , M , X ) = min Mo2M [ L ( Mo|Mp ) +L ( M ( X ) |Mo , X ) ] ( 5 ) While we have not put any restrictions on the class of models for Mo , practical computability considerations prompt us to focus on models that can be algorithmically derived and with structural properties that are understandable by humans ( e.g. , regression trees , linear regression models ) . 2.4 EXPLAINABILITY WITH ATTENTION MODELS FOR REGRESSION PROBLEMS . E ag and Eaw are not computable in the general Turing sense of the word . However , with restrictions on the class M of models , approximations for these quantities can be defined . In practice , these restrictions are typically dictated by the problem at hand . In the rest of this paper , we focus exclusively on regression problems , thus allowing us to tackle the medical patient embedding problem that we are interested in . Consequently , we restrict M to the set Mreg of regression models for all models M that estimate Y from X , and all observers Mo that estimate the Ŷ = M ( X ) from X . With the popularity and effectiveness of attention mechanisms to provide input/output rationales , we further impose attention mechanisms in the architecture of our observers . This additional step not only allows us to measure the complexity of our observers but also allows us to produce effective explanations . More specifically , our observers make use of local attention models estimating at each time i the attention that M is applying to each input feature Xij . We define ↵i ( Xi ) = SoftMax ( ( Xi ) ) 2 Rm , where is an arbitrarily complex model producing attention coefficients . Dense layers are typically used to compute but more complex recurrent neural architectures to compute attention coefficients can also be used as suggested in Choi et al . ( 2016b ) . Mo leverages the ↵i ’ s in the following way : M o ( Xi ) = W↵i Xi = ( W [ : , k ] ↵i ) X T i = PX T i ( 6 ) where P = W [ : , k ] ↵i with denoting the Hadamard product and P = W [ : , k ] ↵i being a matrix obtained by stacking the results of the Hadamard product of columns of W with ↵i .

Summary: The authors propose a framework for training an external observer that tries explain the behavior of a prediction function using the minimal description principle. They extend this idea by considering how a human with domain knowledge might have different expectations for the observer. The authors test this framework on a multi-variate time series medical data (MIMIC-III) to show that, under their formulation, the external observer can learn interpretable embeddings.