On the Safety of Interpretable Machine Learning: A Maximum Deviation Approach

1 INTRODUCTION . Interpretable and explainable machine learning ( ML ) has seen a recent surge of interest because it is viewed as one of the key pillars in making models trustworthy , with implications on fairness , reliability , and safety ( Varshney , 2019 ) . It is generally accepted that the ultimate measure of ML explainability is whether a human finds the explanations useful ( Doshi-Velez & Kim , 2017 ; Dhurandhar et al. , 2017 ) . However , less attention has been paid to deeper reasons behind the human desire for explainability . In this paper , we posit an important reason is toward achieving safety and preventing unexpected harms ( Varshney & Alemzadeh , 2017 ) . This reason is implicit in the dichotomy between directly interpretable models vs. post hoc explanations of black-box models . Some argue that only directly interpretable models should be used in high-risk applications ( Rudin , 2019 ) . The crux of this argument is that post hoc explanations leave a gap between the explanation and the model that is producing the predictions . Thus , unusual data points may appear to be harmless based on the explanation , but truly cause havoc . This argument however does not explicitly address the question : What does safety mean for such models , and how is it intertwined with interpretability ? Towards answering this question , we propose a mathematical definition for assessing the safety of supervised learning ( i.e . predictive ) models . Viewing these models as functions mapping an input space to an output space , a key way in which these models can cause harm is through grossly unexpected outputs , corresponding to inputs that are poorly represented in training data . Accordingly , we approach safety assessment for a model by determining its maximum deviation over a certification set from the output of a reference model . The idea of a certification set is that it is a large subset of the input space and is intended to cover all conceivable inputs to the model . The reference model could be a simple , well-understood model or an existing model that has been “ tried and tested. ” These concepts are discussed further in Section 2 . In Section 4 , we discuss the computation of the maximum deviation for different model classes and show how this is facilitated by interpretability . For model classes regarded as interpretable , including trees , rule lists , generalized linear and additive models , the maximum deviation can be computed exactly and efficiently by exploiting the model structure . For tree ensembles , which are not regarded as interpretable , discrete optimization techniques can exploit their composition in terms of trees to provide anytime bounds on the maximum deviation . The case of trees is also generalized in a different direction by considering a broader class of functions that are piecewise Lipschitz , which we argue cover many popular interpretable functions . Here we show that the benefit of interpretability is significantly tighter regret bounds on the maximum deviation compared with black box functions , by appropriately repurposing results from the multi-armed bandit literature in this context . On the other hand , it is less clear that post hoc explanations , which approximate a model locally ( Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ; Dhurandhar et al. , 2018 ) or globally ( Buciluǎ et al. , 2006 ; Hinton et al. , 2015 ) , can help with evaluating the maximum deviation and hence safety . We conduct experiments to illustrate the deviation maximization methods in Section 4 for decision trees , linear and additive models , and tree ensembles . The results in Section 5 quantify how the maximum deviation increases as the size of the certification set increases and as the smoothness of the models decreases . For tree ensembles , we find that the obtained upper bounds on the maximum deviation are informative , showing that the maximum deviation does not increase with the number of trees in the ensemble . We also study the feature combinations that maximize deviation , which can shed light on the sources of extreme model outputs and guide further investigation . Overall , our discussion suggests that a reason for preferring more interpretable models is that it is easier to assess them for unexpected and potentially unsafe outputs . 2 ASSESSING SAFETY THROUGH MAXIMUM DEVIATION . We are given a supervised learning model f , which is a function mapping an input feature space X to an output space Y . We wish to assess the safety of this model by finding its worst-case deviation from a given reference model f0 : X 7→ Y . To do this , we additionally require 1 ) a measure of deviation D : Y × Y 7→ R+ , where R+ is the set of non-negative reals , and 2 ) a certification set C ⊆ X over which the deviation is maximized . Then the problem to be solved is max x∈C D ( f ( x ) , f0 ( x ) ) . ( 1 ) The deviation is worst-case because the maximization is over all x ∈ C ; further implications of this are discussed in Appendix C. We view problem ( 1 ) as a means toward the goal of evaluating safety . In particular , a large deviation value is not necessarily indicative of a safety risk , as two models may differ significantly for valid reasons . For example , one model may capture a useful pattern that the other does not . What large deviation values do indicate , however , is a ( possibly ) sufficient reason for further investigation . Hence , the maximizing solutions in ( 1 ) ( i.e. , the arg max ) are of operational interest . Below we further discuss some elements in this problem formulation . Output space Y . In the case of regression , Y is the set of reals R or an interval thereof . In the case of binary classification , while Y could be { 0 , 1 } or { −1 , +1 } , these limit the possible deviations to binary values as well ( “ same ” or “ different ” ) . Thus to provide more informative results , we take Y to be the space of real-valued scores that are thresholded to produce a binary label . For example , y could be a predicted probability in [ 0 , 1 ] or a log-odds ratio in R. Similarly for multiclass classification withM classes , Y ⊂ RM could be aM -dimensional space of real-valued scores . Models that abstain can also be accommodated as noted in Appendix A . Reference model f0 . The premise of the reference model is that it should be “ safe ” above all . The simplest case mathematically is for f0 to be a constant function representing a baseline value , for example zero . More generally , f0 may be a simple model that can be readily grasped by a human , may be validated against domain knowledge , or may be based on a small number of expert-selected features . Such models are common in medical risk assessment , consumer finance , and predicting semiconductor yield . By simple , we mean for example a linear model with 10 non-zero coefficients or a decision tree with 10 leaves . The reference model could also be an existing model that is not necessarily simple but has been extensively tested and deployed . In this case , f could be a new version of the model , trained on more recent data or improved in some fashion , and we wish to evaluate its safety before deploying it in place of f0 . In this and more complex settings , f0 may not be globally interpretable , but may be so in local regions . The machinery developed in this work could be applied in these settings as well to assess the safety of f ( more discussion in Appendix C ) . Certification set C. The premise of the certification set is that it contains all inputs that the model might conceivably be exposed to . This may include inputs that are highly improbable but not physically or logically impossible ( for example , a severely hypothermic human body temperature of 27°C ) . Thus , while C might be based on the support set of a probability distribution or data sample , it does not depend on the likelihood of points within the support . The set C may also be a strict superset of the training data domain . For example , a model may have been trained on data for males , and we would now like to determine its worst-case behavior on an unseen population of females . For tabular or lower-dimensional data , C might be the entire input space X . For non-tabular or higher-dimensional data , the choice C = X may be too unrepresentative because the manifold of realistic inputs is lower in dimension . In this case , if we have a dataset { xi } ni=1 , one possibility is to use a union of ` p balls centered at xi , C = n⋃ i=1 Bpr [ xi ] , Bpr [ xi ] = { x ∈ X : ‖x− xi‖p ≤ r } . ( 2 ) The set C is thus comprised of points somewhat close to the n observed examples xi , but the radius r does not have to be “ small ” . In addition to determining the maximum deviation over the entire set C , maximum deviations over subsets of C ( e.g. , different age groups ) may also be of interest . For example , Appendix D.1 shows deviation values separately for leaves of a decision tree , which partition the input space . 3 RELATED WORK . Our work relates to a number of different technical directions . Varshney & Alemzadeh ( 2017 ) and Mohseni et al . ( 2021 ) give qualitative accounts suggesting that directly interpretable models are an inherently safe design because humans can inspect them to find spurious elements ; in this paper , we attempt to make those qualitative suggestions more quantitative . Furthermore , several other authors have highlighted safety as a goal for interpretability , but without much further development as done here ( Otte , 2013 ; Doshi-Velez & Kim , 2017 ; Tomsett et al. , 2018 ; Gilpin et al. , 2018 ; Rudin , 2019 ) . Moreover , there is no consensus on how to measure interpretability , which motivates the relationship explored in this paper between interpretability and the ease of evaluating safety . In the area of ML verification , robustness certification methods aim to provide guarantees that the classification remains constant within a radius of an input point , while output reachability is concerned with characterizing the set of outputs corresponding to a region of inputs ( Wong & Kolter , 2018 ; Raghunathan et al. , 2018 ; Huang et al. , 2020 ) . Our problem of deviation maximization ( 1 ) is more closely related to output reachability . The differences in our work are : 1 ) we consider two models , a model f to be assessed and a reference f0 , and are interested in their difference as measured by deviation function D ; 2 ) our focus is global , over a comprehensive set C , rather than local to small neighborhoods around input points ; 3 ) we study the role of interpretability in safety verification . Moreover , works in robust optimization applied to machine learning minimize the worst-case probability of error , but this worst case is over parameters of f rather than over individual values of x ( Lanckriet et al. , 2002 ) . Thomas et al . ( 2019 ) present a framework where during the model training , a set of safety tests is specified in order to accept or reject the possible solution . The specification of these tests is left to the model designer but the goal of the proposed solution is to provide a reusable paradigm to support safety in ML solutions . We build on related literature from model robustness and explainability areas that deals specifically with tree ensembles . Kantchelian et al . ( 2016 ) seek to find the smallest perturbation of an input instance to ‘ evade ’ a classifier using mixed-integer programming ( MIP ) . Optimization formulations are also explored by Parmentier & Vidal ( 2021 ) for the purposes of counterfactual explanations . MIP approaches are computationally intensive however . To address this Chen et al . ( 2019 ) introduce graph based approaches for verification on trees . Their central idea , which we use , is to discretize verification computations onto a graph constructed from the way leaves intersect . The verification problem is transformed to finding all maximum cliques . Devos et al . ( 2021 ) expand on this idea by providing anytime bounds by probing unexplored nodes . Safety has become a critical issue in reinforcement learning ( RL ) with multiple works focusing on making RL policies safe ( Amodei et al. , 2016 ; Zhu et al. , 2019 ; Inala et al. , 2020 ; Rupprecht et al. , 2020 ) . There are two broad themes ( Garcı́a et al. , 2015 ) : ( i ) a safe and verifiable policy is learned at the outset by enforcing certain constraints , and ( ii ) post hoc methods are used to identify bad regimes or failure points of an existing policy . Our current proposal is complementary to these works as we focus on the supervised learning setup viewed from the lens of interpretability . Nonetheless , ramifications of our work in the RL context are briefly discussed in Appendix C .

Summary: The authors posit that the demand for explainable/interpretable models in machine learning is linked to safety. In other words, domain experts trust explainable or interpretable models more. Based on this premise, they suggest to assess the safety of a model by measuring its maximum deviation of a model to a reference model, which is supposed to be safe. This deviation is computed on a certification set of data points. Additionally, the authors show that their proposed maximum derivation can be computed exactly for some interpretable models, such as decision trees, rule lists and generalized linear and additive models. Further, they discuss implications for tree ensembles and empirically evaluate their approach.