Year: 2018

Abstract:
Learning explainable patient temporal embeddings from observational data has mostly ignored the use of RNN architectures that excel in capturing temporal data dependencies but at the expense of explainability. This paper addresses this problem by introducing and applying an information theoretic approach to estimate and quantify the degree of explainability of such architectures. Using a communication paradigm, we formalize metrics of explainability by estimating the amount of information that an AI model needs to convey to a human end user to explain and rationalize its outputs. A key aspect of this work is to model human preferences and prior knowledge at the receiving end and measure a lack of explainability as a deviation from these human preferences. We apply this paradigm to medical patient representation problems by regularizing loss functions of temporal autoencoders according to the derived explainability metrics to guide the learning process towards models producing explainable outputs. We illustrate the approach with convincing experimental results for the generation of explainable temporal embeddings for critical care patient data.

heading: 1 INTRODUCTION
text: The success of AI models to solve real world problems has been hindered by the inability to fully understand the complex mechanisms behind inferences produced by these models. This problem is exacerbated in healthcare where clinicians and other stakeholders consuming AI model outputs need to understand the rationales behind the underlying AI inference processes. Learning explainable representations for patient data has proved to be notoriously difficult and has slowed the adoption of powerful sequence to sequence architectures for this task. In Choi et al. (2016a), the authors describe an interesting and effective approach for the embedding of medical patient data. However, they avoid RNN like architectures probably able to produce more accurate representations but at the expense of explainability.
Explainability in AI in general has been receiving a great deal of attention within the academic community with numerous methodological advances for the learning of models that are easier to digest by humans. In Ribeiro et al. (2016) and Ribeiro et al. (2018), the authors proposed to locally learn simple linear or tree models to approximate the local performance of complex classifiers. Their approach assumes that explainability is heavily related to model complexity. In Choi et al. (2016b), the authors proposed a Recurrent Neural Network architecture able to compute attention coefficients linking the predictions of the model back to the raw features used by the model at different time points. In this case, explainability is also a local concept that relates to attention mechanisms justifying the predictions made by a model by pointing back to the source data deemed responsible for the predictions. Recently in Dhurandhar et al. (2018), the authors proposed to relate explainability not only to the presence of input features responsible for an inference but also to the omission of input features (e.g., the man who does not wear glasses). In Koh & Liang (2017), the authors proposed to use influence functions to understand the internals of an AI model by tracing back predictions to training data instances that have influenced it.
While these papers cleverly shed light on how complex AI models are behaving, they all use different informal notions of explainability–may it be complexity, attentiveness linking predictions to input features or training data influence on predictions. They all treat explainability as a qualitative characteristic of AI models and do not rely on well defined measurable aspects of explainability. In
this paper, we treat aspects of explainability as quantitative measurable properties of models and propose approaches to measure the degrees of explainability of learned models in a regression setting, with application to computational embedding. In general, formalizing explainability is quite hard. In this work, we only focus on specific aspects of it that are grounded with the following principles:
• Observability: We restrict our efforts to the generation of explanations as seen by an external observer witnessing the input-output relationship for a given AI system. This principle is restrictive as it treats any AI agent as a black box. However, we deem it to be a reasonable assumption that humans commonly make, specially in physical sciences when modeling complex physical phenomena from observational data.
• Communicability: We consider explainability to be a communication problem, with a dialog between an external AI observer and a rational human actor with the external AI observer explaining the behavior of the AI system to the rational human actor at the receiving end of this communication. The amount of information that needs to be produced by the AI external observer for explainability is consequently inversely proportional to the degree of explainability of the AI system.
• Subjectivity: We view explainability as a subjective concept that depends on the mind set of the receiving rational human actor, in terms of his/her preferences and amount of prior knowledge that he/she may have acquired. This principle prompts us to model some notion of preferences and/or assumed prior knowledge available to these human actors with prior distributions. In these cases, AI models are deemed less explainable as they produce predictions based on facts that are harder to explain using generally accepted human prior knowledge.
Based on these principles, we model explainability using information theoretic constructs measuring the amount of information that need to be transmitted by an AI agent to a human end user to explain the rationales behind the predictions. The main contributions of this paper are three fold: (i) an information theoretic framework to effectively measure aspects of explainability, (ii) the application of the framework to regression problems using attention mechanisms, and (iii) conclusive experimental results on patient temporal embedding problems. The rest of the paper is organized according to these three contribution areas.

heading: 2 EXPLAINABILITY WITH THE MINIMUM DESCRIPTION LENGTH PRINCIPLE

heading: 2.1 NOTATIONS
text: Throughout the paper, scalars are represented with lower case letters in a non bold typeface, e.g., x. Sets are represented with upper case letters in a calligraphic typeface, e.g., H. Vectors are denoted in bold lower case letters e.g., x 2 Rk from some positive k 2 N. Matrices are represented in bold upper case letters, e.g., A 2 Rn⇥m, where both n,m 2 N. In general, we use subscripts to index elements in a vector or in a matrix. For instance, xi is a scalar corresponding to the ith element of vector x while Ai,j denotes the i, j entry of matrix A. The data elements in this paper are always attached to a specific entity (e.g., a patient in healthcare). To avoid notation clutter, we often drop these references to entities and describe the methodology at the entity level. The data for a given entity is represented by a matrix X 2 Rn⇥m. A row Xi, 1  i  n represents an m dimensional feature vector collected at discrete time i. Similarly, the output of an AI model processing X is represented by the matrix Y 2 Rn⇥k corresponding to a discrete time series of k dimensional output vectors.

heading: 2.2 THE MINIMUM DESCRIPTION LENGTH PRINCIPLE
text: The Minimum Description Length principle is a formalization of the Occam Razor principle guiding the search for good models that explain data using description lengths (Grünwald (2007),Rissanen (1998)). A simple version of it uses a two-part coding approach to select the "best" model Mopt as the one that minimizes the sum of the length in bits for a description of the model (part one of the code) and the length in bit of the effective description of the data using using the model.
Let M denote the set of all computable models in the Turing sense. The MDL defines the best model as:
M opt := min M2M (L(M) + L(X|M)) (1)
where L is a function that returns description lengths. The MDL approach imposes preferences for models that balance their complexity with their goodness of fit. The first part of the code is essentially the model complexity while the second part measures the goodness of fit for the model on the data. An idealized version of the MDL has been defined in Vitányi & Li (2000) where L is replaced with K, the Kolmogorov complexity. However, this version is not computable in the Turing sense. In the following sections, we present practical methods to compute estimate L for explainability purposes.

heading: 2.3 EXPLAINABILITY METRICS
text: Following the observability and communicability assumptions described in Section 1 and the MDL principle, we measure the explainability of an AI model M with description lengths of an external observer model only able to perceive the inputs and outputs of M to model this relationship (See Figure 1). In this setting, an explanation Mo for an AI model M is essentially such an observer model describing how its inputs X and outputs Y are related in a language that is understandable by human actors. Assuming the presence of no additional knowledge for Mo to characterize this input output relationship, we measure the effectiveness of Mo with two part codes as follows: Definition 1. The explainability provided by an observer Mo of model M is defined as:
e ag(Mo,M,X) := L(Mo) + L(M(X)|Mo,X) (2)
The knowledge agnostic explainability of a model M is defined as: E
ag(M,X) := min Mo2M e ag(Mo,M,X) = min Mo2M [L(Mo) + L(M(X)|Mo,X)] (3)
As stated in Section 1, we also assume that explainability can be subjective and dependent on the mind set of receiving rational human actors, in terms of the amount of prior knowledge that they have acquired. In our framework, this prior knowledge is represented as a prior model Mp that these human actors would use to explain the (X,Y) relationship. For instance, an expert physician may explain the general loss of health of a critical care patient from features representing reductions in heart rate variability in the source data. For this physician, an AI model producing rationales that are inline with this prior knowledge would certainly by easier to explain than other AI models using variables that are further from her/his mental mind set. Consequently, we modify the complexity term of the previous definition to come up with the following measures of explainability in the presence of prior knowledge. Definition 2. With a computable prior knowledge Mp, the explainability provided by an observer M
o of model M is defined as: e aw(Mo,M,Mp,X) := L(Mo|Mp) + L(M(X)|Mo,X) (4)
The knowledge aware explainability of a model M is defined as: E
aw(M,Mp,X) := min Mo2M e aw(Mo,Mp,M,X) = min Mo2M [L(Mo|Mp)+L(M(X)|Mo,X)] (5)
While we have not put any restrictions on the class of models for Mo, practical computability considerations prompt us to focus on models that can be algorithmically derived and with structural properties that are understandable by humans (e.g., regression trees, linear regression models).

heading: 2.4 EXPLAINABILITY WITH ATTENTION MODELS FOR REGRESSION PROBLEMS
text: E ag and Eaw are not computable in the general Turing sense of the word. However, with restrictions on the class M of models, approximations for these quantities can be defined. In practice, these restrictions are typically dictated by the problem at hand. In the rest of this paper, we focus exclusively on regression problems, thus allowing us to tackle the medical patient embedding problem that we are interested in. Consequently, we restrict M to the set Mreg of regression models for all models M that estimate Y from X, and all observers Mo that estimate the Ŷ = M(X) from X.
With the popularity and effectiveness of attention mechanisms to provide input/output rationales, we further impose attention mechanisms in the architecture of our observers. This additional step not only allows us to measure the complexity of our observers but also allows us to produce effective explanations. More specifically, our observers make use of local attention models estimating at each time i the attention that M is applying to each input feature Xij . We define ↵i(Xi) = SoftMax( (Xi)) 2 Rm, where is an arbitrarily complex model producing attention coefficients. Dense layers are typically used to compute but more complex recurrent neural architectures to compute attention coefficients can also be used as suggested in Choi et al. (2016b). Mo leverages the ↵i’s in the following way:
M o(Xi) = W↵i Xi = (W[:, k] ↵i)X T i = PX T i (6)
where P = W[:, k] ↵i with denoting the Hadamard product and P = W[:, k] ↵i being a matrix obtained by stacking the results of the Hadamard product of columns of W with ↵i.

heading: 2.4.1 ESTIMATING THE GOODNESS OF FIT FOR REGRESSIVE OBSERVERS
text: With this restriction to Mreg, we model the regression error with a zero mean normal distribution. This is quite common in regression analysis and also in the application of the MDL to regression problems (Grünwald (2007)). Hence, for all Mo 2 Mreg,Y = Mo(X) + ✏, where ✏j ⇠ N(0, 2) for 1  j  k This restriction allows us to approximate the goodness of fit term L(M(X)|Mo,X) in Equations 2 and 4 for any regressive observer for any AI model without any further assumption on M . Since this term essentially measures the amount of information that cannot be explained by the model Mo, it tends to have random properties and can be approximated using its Shannon-Fano code (Cover & Thomas (2006),Grünwald (2007)):
L(M(X)|Mo,X) ⇡ log p(M(X)|Mo(X),X) = kX
i=1
log p(M(X)i|M o(X),X) (7)
using the conditional independence of each (M(X))i given X. Clearly, for each i, p(M(X)i|Mo(X),X) follows also a Gaussian distribution N(Mo(X)i, 2). Hence, the goodness of fit can be rewritten as a scaled mean squared error between M(X) and Mo(X):
L(M(X)|Mo,X) = nk
2 log(2⇡ 2) +
1
2 2
kX
j=1
nX
i=1
(M(X)ij M o(X)ij) 2 (8)

heading: 2.4.2 ESTIMATING THE COMPLEXITY OF LINEAR REGRESSIVE OBSERVERS
text: As stated in Grünwald (2007), the basic MDL principle does not define the expression of the complexity terms in Equations 2 and 4. These terms are very much dependent on the restricted class of observer models and not on any properties of the data X. From Equation 6, we are restricted to Mo(X) = PXT , a multiplication of the input with the m by k matrix P. Rows of P can be scaled into attentive distribution that we use to estimate the complexity of Mo by measuring the compactness of these rows. Intuitively, the more compact these row vectors are, the easier it is to explain the corresponding predictions produced by M since the observer would hypothetically need to communicate less bits of information to describe how inputs X relate to outputs in M . A natural
measure of compactness is the Shannon entropy and we estimate the model complexity as follows: Let Q be a matrix such that Qj = SoftMax(|Pj |), then:
L(Mo) ⇡ 1
k
kX
j=1
H(Qj) (9)
where H(p) = P
i pi log 1 pi
With a conditional on preferences or prior knowledge Mp, one may be tempted to measure L(Mo|Mp) with a conditional entropy, thus extending naturally the approach that we used to estimate L(Mo). However, the computation of such conditional entropy is ill defined as it requires an unknown joint distribution. Instead, we complement L(Mo) with the KL divergence between the attentions Qj , 0  j  k and their counterpart computed from prior preferences or knowledge j , 0  j  k.
Consequently,
L(Mo|Mp) ⇡ 1
k
kX
j=1
(H(Qj) +DKL(Qj || j)) = 1
k
kX
j=1
Qj . log j (10)
In this case, the complexity becomes a simple average cross entropy across all k dimensions of the output.

heading: 3 LEARNING EXPLAINABLE TEMPORAL EMBEDDINGS
text: Despite the large-scale adoption of Electronic Health Records (EHR) by medical institutions, the secondary re-use of these data sets and its impact in healthcare has been lukewarm at best. Beyond all the data governance challenges that need to be overcome for proper access and integration, these data sets are quite raw, with very complex data models. They always need to be refined to become ready for analysis and these data refinement steps are quite expensive, often tailored for specific applications and not reusable. Deep representation learning techniques have been proposed by several research groups to tackle these challenges. However, most of these approaches refrain from using RNN architectures in attempts to preserve the interpretability of the learned models at the expense of being able to encode efficiently the temporal patterns in the data. Using external regressive operators as observers, we address this issue and present in this section an RNN based framework for the representation learning of temporal embeddings.

heading: 3.1 AN RNN PIPELINE FOR TEMPORAL EMBEDDINGS
text: Figure 2 illustrates the proposed deep learning architecture. At a high level, this architecture is an auto-encoder consisting of three components. First, an RNN encoder Menc is used to transform the input data X into embeddings Y. We experimented with various RNN cells for the encoder including the popular LSTM and GRU cells. We eventually settled on the minimalRNN cell presented in Chen (2017) and Chen et al. (2018) because of its predictable dynamics as it is designed to rule out mixing effects across dimensions of the embeddings produced by our autoencoder. This choice allows us to improve the likelihood of developing a regressive observer model with a reasonable goodness of fit without sacrificing too much on the quality of the embeddings.
The second component is an RNN based decoder Mdec that attempts to reconstruct the original data X from the output of the encoder. We have equipped this decoder with a temporal attention scheme similar to the ones typically used in NLP sequence 2 sequence architectures for machine translation. This temporal attention allows the decoder to reconstruct Xi using embeddings already computed at previous times i 1 to i w, w being a hyper-parameter that we keep small (i.e., less than 10).
The third component of this deep learning architecture is the regressive observer Mo described in the previous section and used to track the inputs and outputs of the encoder to provide and estimate explainability. This component is taking X as input and producing an estimate Ŷ of the embedding Y. During training, this component may have access to a prior model of user preferences Mp that it may use in regularization steps to control deviations from these preferences. Mp allows us to control the contribution of input variables into each dimension of the embeddings produced by the encoder.
To train this deep learning architecture, we use a multi-task approach where we train each of these components jointly. The joint training of this autoencoder with the observer allows us to softly enforce explainability constraints in this pipeline. While the role of the observer is essentially to estimate the degree of explainability, we use it here to constrain the generation embeddings towards explainable models according to the metrics that we defined in the previous section. The encoder and decoder are trained together to minimize the mean square reconstruction error between X and X̂, according a loss Lrec expressed as:
L rec =
1
nk
nX
i=1
kX
j=1
((X)ij (X̂)ij) 2 (11)
The regressive observer is trained to provide the best explanations in the MDL sense. In the absence of prior preferences, its learning is associated with a loss Lag defined as:
L ag = eag(Mo,Menc,X) ⇡
1
k
kX
j=1
H(Qj) + 1
2 2
kX
j=1
nX
i=1
(M(X)ij M o(X)ij) 2 (12)
Please note that we dropped the term nk2 log(2⇡ 2) from the formal expression of eag(Mo,Menc) since this term does not have impact on the optimization of the loss function when is estimated. In the presence of prior knowledge, the learning of the Mo is associated with a loss Law defined as:
L aw = eaw(Mo,Menc,X) ⇡
1
k
kX
j=1
Qj . log j + 1
2 2
kX
j=1
nX
i=1
(M(X)ij M o(X)ij) 2 (13)
Putting all these terms together allows us to define the overall loss function for this architecture:
L =
⇢ L rec + agLag in the absence of prior knowledge
L rec + awLaw in the presence of prior knowledge

heading: 3.2 EXPERIMENTAL RESULTS
text: We have performed experiments to demonstrate the effectiveness of our proposed metrics for patient data embedding using the MIMIC-III Johnson et al. (2016) dataset. For these experiments, we used a
cohort of 2395 patients who experienced sepsis, resulting in a total of 109886 data samples. This data set is sampled on an hourly basis. Each data sample consists of 37 variables together with a timestamp and a patient ID. The name of these variables are shown on the Y axis of the plot shown on Figure 3.
The RNN architecture shown in Figure 2 has been implemented on tensorflow v1.5 (Abadi et al. (2015)). Experiments were performed on a cloud environment using a small cluster of K80 GPU units. Various values of learning rate were tried for the training and the results provided in the paper uses a learning rate of 0.0005 with batch size of 10. All results were obtained using a 5 fold cross validation approach with partitions computed at the patient level.
A prior model Mp was gathered manually to impose soft constraints on how input variables may be grouped together to define each of the k embedding dimensions. This prior suggests a uniformly distributed focus for each variable in a given group. This manual grouping led us to define 16 groups that are shown in Figure 3. This figure shows the heat map of a matrix with rows corresponding to input variables and columns corresponding to embedding dimensions. Each column shows a suggested distribution of input variables within an embedding dimension. For instance, the third column suggests a concentration of variables that relates to the Glasgow Coma score (a neurological score estimating the consciousness of patients) while the fourth column suggests a concentration of variables measuring various aspects of the cardiovascular system. Different prior preference models have been tested, providing results that are consistent with the evaluation that we are describing here for the specific Mp shown in Figure 3.
We focused on evaluating two aspects of the system, namely the qualitatively analysis of Mo and the analysis of the impact of explainability regularization on Lrec
Figures 4 and 5 qualitatively illustrate the explanations provided by Mo. Figure 4 focuses on the knowledge agnostic case and shows heat maps of average attention coefficients for each embedding dimensions for ag = 1 and ag = 10. It is clear from that figure that increasing ag adds sparsity on the heat maps and help provide more explainable rationales. We also looked at the actual input variables that are selected in each of the dimensions. With ag = 10, most embedding dimensions are dominated by a single variable except for the 13th dimension focusing heavily on the blood pressure variables and the first dimension focusing on respiration rate and mean arterial pressure.
Figure 5 focuses on the knowledge aware case and shows heat maps of the average attention coefficients for each embedding dimensions for aw = 1 and aw = 5. A quick look at these plots shows that the system is indeed attempting to learn explanations that are in accordance with the prior preferences imposed by regularization. As aw increases, Mo converges more and more towards M p.
To analyze the impact of explainability regularization on the overall loss and the generated embeddings, we looked at the R2 associated with the overall reconstruction error and the R2 associated with the fitness of Mo, both as a function of regularization parameters. Table 1 shows R2 results obtained for various values of the aw. We can notice that the R2 penalty for the reconstruction is maximized at aw = 0. However, as aw increases, the R2 remains mostly flat. In general, we noticed in our experiments that lower values of aw do not necessarily yield better R2 despite reduction in entropy on the columns of W . This observation could be a manifestation of the Occam Razor principle that gives preferences for simpler and less complex models that focus on less variables. However, the penalty is much more severe on the goodness of fit of Mo as we vary aw. As aw increases, the complexity of the Mo decreases as shown in Figure 5 but this reduction of complexity limits Mo’s ability to effectively estimate M . The explanations provided by the observer become trivial for the complexity of the task at hand.

heading: 4 CONCLUDING REMARKS
text: We have presented an MDL based approach to estimate the explainability of AI models while taking into account prior preferences. We have shown how these concepts can be used to regularize the
learning of temporal embeddings with regularization terms constraining the production of explainable models, according to metrics that we have defined. Experimental results on a real patient data set from MIMICIII has demonstrated the applicability of the proposed metrics for the generation of explainable models trading explanation complexity for overall accuracy. In the future, we plan to run further experiments on larger EHR data sets for other predictive tasks, with prior knowledge extracted from medical guidelines and ontologies. Furthermore, we plan to extend these concepts to reinforcement learning, in a policy-based setting where we constrain the learning of optimal policies with knowledge aware explainability metrics to produce explainable policies.

Review summaries:
The authors propose a framework for training an external observer that tries explain the behavior of a prediction function using the minimal description principle. They extend this idea by considering how a human with domain knowledge might have different expectations for the observer. The authors test this framework on a multi-variate time series medical data (MIMIC-III) to show that, under their formulation, the external observer can learn interpretable embeddings.