Pearl: Prototype lEArning via Rule Lists

1 INTRODUCTION . The rapid growth of sizes and complexities of electronic health records ( EHR ) data has motivated the use of deep learning models , which demonstrated state-of-the-art performance in many tasks , including diagnostics and disease detection ( Lipton et al. , 2016 ; Choi et al. , 2016b ; Xiao et al. , 2018a ) , medication prediction ( Zhang et al. , 2017 ; Le et al. , 2018 ) , risk prediction ( Futoma et al. , 2015 ; Xiao et al. , 2018b ) , and patient subtyping ( Baytas et al. , 2017 ; Che et al. , 2017 ) . Although deep learning models can produce accurate predictions and classifications , they are often treated as black-box models that lack interpretability and transparency of their inner working ( Lipton , 2016 ) . This is a critical problem as it can limit the adoption of deep learning in medical decision making . Recently , there have been great efforts of trying to explain black-box deep models , including via attention mechanism ( Choi et al. , 2016b ; Xu et al. , 2015 ) , visualization ( Samek et al. , 2017 ) , and explanation by examples or prototypes ( Li et al. , 2017 ) . To bring deep models into real clinical practice , clinicians often need to understand why a certain output is produced and how the model generates this output for a given input ( Nauck & Kruse , 1999 ) . Rule learning and prototype learning are two promising directions to achieve clinical model interpretability . Rule learning generates a set of rules from training data , in which its prediction is done at leaf levels via simple models such as majority vote or regression . For example , the results of rule learning are rule lists composed of multiple if-then statements ( Angelino et al. , 2018 ) . Those rules can be interpretable to domain experts as they are expressed in simple logical forms ( Rivest , 1987 ; Breiman , 2017 ) . However , because of such a simple prediction model , the accuracy of rule-based models is often lower than deep neural networks . Moreover , the interpretability can be undermined as the depth of rules becomes very large and thus incomprehensible for human with tens or hundreds of levels of the rules . Prototype learning is another interpretable model inspired by case-based reasoning ( Kolodner , 1992 ) , where observations are classified based on their proximity to a prototype point in the dataset . Many machine learning models have incorporated prototype concepts ( Priebe et al. , 2003 ; Bien & Tibshirani , 2011 ; Kim et al. , 2014 ) , and learn to compute prototypes ( as actual data points or synthetic points ) that can represent a set of similar points . However prototypes alone may not lead to interpretable models as we still need an intuitive way to represent and explain what a prototype is , especially given recent deep prototype works ( Li et al. , 2017 ) . Both approaches were explored in healthcare applications . For example , rule learning was employed to identify how likely patients were to be readmitted to a hospital after they had been released , each probability associated with a set of rules as criteria ( Wang & Rudin , 2014 ; Chen & Rudin , 2017 ) . While prototype could be selected from actual patients and genes for clinicians to make sense of large patient cohort or gene data ( Bien & Tibshirani , 2011 ) . However , there are still open challenges : How to construct simple rules with more accurate prediction and classification performance ? How to produce accurate and intuitive definitions of prototypes ? In this work , we propose Prototype lEArning via Rule List ( PEARL ) , which combines rule learning and prototype learning on deep neural networks to harness the benefits of both approaches and alleviate their shortcomings for an accurate and interpretable prediction model . In particular , we iteratively learn rule lists , via a data reweighing procedure using prototypes , and then update prototypes via neural networks with learned rules . PEARL not only generates simple and interpretable rule lists and prototypes , but also provides neural network models which can infer the similarity of a query datum to all the prototypes . To summarize , we make the following contributions in this paper . 1 . We propose an integrative method to combine rule list and prototype learning , enabling PEARL to harness the power of these methods . 2 . PEARL automatically learns prototypes corresponding to rules in a rule list , which are more concise than conventional rule list learning methods and more explainable than prototype learning methods by providing logic reasoning . 3 . On real-world electronic health record datasets , PEARL demonstrates both accurate prediction performance and simple interpretation . 2 TECHNICAL BACKGROUND . A prototype is an object that is representative of a set of similar instances ( e.g. , a patient from a cohort ) and can be a part of the observed data points or an artifact summarizing a subset of them with similar characteristics . Prototype learning is a type of case-based reasoning ( Kolodner , 1992 ) and aims to find some prototypes ( Priebe et al. , 2003 ; Bien & Tibshirani , 2011 ; Kim et al. , 2014 ) Prototypes can be seen as an alternative approach to learn centroids of clusters , and have been applied to few shot learning ( Mensink et al. , 2013 ; Rippel et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ) . Let X = { xi } ni=1 be the data set , to learn one prototype pj of many prototypes , existing works choose some pj ∈ X ( Bien & Tibshirani , 2011 ) , compute a linear combination pj = ∑n i bijxi ( Wu & Tabak , 2017 ) , or form a Bayesian generative model ( Kim et al. , 2014 ) . In this work , we follow ( Li et al. , 2017 ) and use a general representation of pj = fj ( X ) , where fj is automatically learned via deep neural networks . pj has the same dimension as the learned representation of data , which is a predefined hyperparameter . Current prototype selection methods typically select one prototype at a time , and provide limited higher-level abstraction on the reasoning side of diagnosis . Rule lists are logic statements over original features . A rule list R = ( r1 , r2 , ... , rK , r0 ) of length K is a ( K+1 ) -tuple consisting ofK distinct association rules , rk : = zk ⇒ qk for k = 1 , ... , K with an additional default rule r0 . Each rule r = z → q is an implication corresponding to the conditional statement , “ if z , then q ” where z is premise and q is conclusion . In general , rule lists are easy to understand . In this paper , each ri in R are dependent of previous rules with “ else-if ” logics . We build on existing rule list learning method ( Angelino et al. , 2018 ) to iteratively guide the prototype learning via neural networks . In addition , we use rule learning methods where each individual rule consists of logic AND clauses but not ORs . Following existing definition of interpretability ( Lakkaraju et al. , 2016 ) , there are 4 aspects of interpretability : size , length , cover , and overlap . Size . The size of a rule lists is defined as number of rules K in a rule list R. The fewer the rules in a rule list , the easier it is for a user to understand all of the conditions that correspond to a particular class . Length . We use the term length to measure the number of clauses in each rule ri . If the number of clauses in a rule is too large , it will loose its natural interpretability . Cover . Cover measures the set of data points that satisfy each ri . Cover measures how the data is divided by the rule classifiers . Overlap . Overlap between two rule ri and rj is the number of points that satisfy both rules . It measures the discriminative power of each rule and whether decision boundary is clearly defined . In this paper , we mainly investigate and provide new methods to reduce the size ( by combining rule in rule lists into prototypes ) and the length ( by replacing the clauses in each rule using a prototype ) of rule list classifiers , while improving on the accuracy of rule lists . We propose to use the cover of each rule ri to re-weight data , which forces rule learning methods to focus on more discriminative data points and hence reduce the overlap among rules . 3 PEARL : METHODOLOGY Let X = { X1 , · · · , XN } be N data samples , where each sample Xn ( e.g. , health records for patient n ) is a sequences of discrete event labels ( such as medical codes in electronic health records ) . We can represent Xn as { ein , tin } , where ein ∈ E is the i-th event label in Xn and tin is the time stamp of ein . For each Xn , there is a class label yn . For example , in health applications , y are the classification result of targeting diseases such as the onset of heart failure ( binary ) , or subtypes of diabetes ( multiclass ) . The goal of PEARL is to accurately predict y = { y1 , · · · , yN } and to provide explanation for such predictions . In this work , we assume both Xn and yn are categorical variables . In this work , we aim to do so by providing an interpretable representation of data with a deep neural network . The outputs of the network include the class label y and a set of interpretable prototypes P̂ corresponding to a rule list R. The neural network is used to performing accurate classification , under the guidance of prototypes defined by the rule lists . Formally , the overall objective of PEARL is : argmin θ1 , θ2 λ1L1 ( h ( X ; θ1 ) , P̂ ) ︸ ︷︷ ︸ distance of data to prototypes + λ2L2 ( sR ( h ( X ; θ1 ) ; θ2 ) , y ) ︸ ︷︷ ︸ classification error , where L1 ( h ( X ; θ1 ) , P̂ ) = ∑ Xn∈X min k∈ { 1 , ··· , K } d ( h ( Xn ; θ1 ) , pk ) , and P̂ = f ( h ( X ; θ1 ) , R ) . |P̂ | ≤ |R| . ( 1 ) where h ( X ; θ1 ) is the learned representation of X with parameter θ1 . h ( X ; θ1 ) is a vector and has the same predefined dimension as pk , R is the learned rule list , and P is the set of learned prototypes . A set of prototypes P̂ = { p1 , p2 , · · · , pK } contains K representation of data , which serve as prototypes . Here d is a distance measure , such as the cosine distance . f is a fixed mapping that , given R and learned h ( X ; θ1 ) , P̂ are determined without further learning . More details on f can be found in Section 3.1.2 . Each pk lies in the same space as h ( X ; θ1 ) , and should correspond to one or more rules in R. The second term L2 is the Cross Entropy loss for the final prediction target , where sR ( h ( X ; θ1 ) ; θ2 ) represents the predicted label for X and y is the ground-truth label . Here θ1 represents all the model parameters for data representation learning h ( X ; θ1 ) and θ2 represents those of classification model sR ( · ) . We will drop θs for simplicity from now on . Minimizing L1 would encourage training examples to be as close as possible to at least one prototype in the latent space , motivated by ( Li et al. , 2017 ) . However , we do not use other terms from ( Li et al. , 2017 ) and instead introduce rule lists as the guidance for prototype learning . Note that relative weights λ1 and λ2 values are chosen via hyperparameter tuning . In general we chose λ2 > λ1 to emphasize the classification performance . Since it is non-trivial to integrate rule and neural network learning , we propose a framework , PEARL , of integrating rule learning and rule-guided prototype learning together . The main intuition is to learn and produce prototypes that are closely related to rules in R , with one-to-one or many-to-one rule-prototype mapping . This serves as a constraint to make each prototype as a surrogate for clauses in each rule , transforming “ if data x satisfies z , then x = q ” to “ if x is close to a prototype p , then x = q ” . We will discuss the network structures in details next .

Summary: This paper presents a new interpretable prediction framework which combines rule based learning, prototype learning, and NNs. The method is particularly applicable to longitudinal data. While the idea of bringing together rules, prototypes, and NNs is definitely novel, the method itself has some unclear design choices. Furthermore, the experiments seem pretty rudimentary and the presentation can be significantly improved.