On the Safety of Interpretable Machine Learning: A Maximum Deviation Approach

1 INTRODUCTION . Interpretable and explainable machine learning ( ML ) has seen a recent surge of interest because it is viewed as one of the key pillars in making models trustworthy , with implications on fairness , reliability , and safety ( Varshney , 2019 ) . It is generally accepted that the ultimate measure of ML explainability is whether a human finds the explanations useful ( Doshi-Velez & Kim , 2017 ; Dhurandhar et al. , 2017 ) . However , less attention has been paid to deeper reasons behind the human desire for explainability . In this paper , we posit an important reason is toward achieving safety and preventing unexpected harms ( Varshney & Alemzadeh , 2017 ) . This reason is implicit in the dichotomy between directly interpretable models vs. post hoc explanations of black-box models . Some argue that only directly interpretable models should be used in high-risk applications ( Rudin , 2019 ) . The crux of this argument is that post hoc explanations leave a gap between the explanation and the model that is producing the predictions . Thus , unusual data points may appear to be harmless based on the explanation , but truly cause havoc . This argument however does not explicitly address the question : What does safety mean for such models , and how is it intertwined with interpretability ? Towards answering this question , we propose a mathematical definition for assessing the safety of supervised learning ( i.e . predictive ) models . Viewing these models as functions mapping an input space to an output space , a key way in which these models can cause harm is through grossly unexpected outputs , corresponding to inputs that are poorly represented in training data . Accordingly , we approach safety assessment for a model by determining its maximum deviation over a certification set from the output of a reference model . The idea of a certification set is that it is a large subset of the input space and is intended to cover all conceivable inputs to the model . The reference model could be a simple , well-understood model or an existing model that has been “ tried and tested. ” These concepts are discussed further in Section 2 . In Section 4 , we discuss the computation of the maximum deviation for different model classes and show how this is facilitated by interpretability . For model classes regarded as interpretable , including trees , rule lists , generalized linear and additive models , the maximum deviation can be computed exactly and efficiently by exploiting the model structure . For tree ensembles , which are not regarded as interpretable , discrete optimization techniques can exploit their composition in terms of trees to provide anytime bounds on the maximum deviation . The case of trees is also generalized in a different direction by considering a broader class of functions that are piecewise Lipschitz , which we argue cover many popular interpretable functions . Here we show that the benefit of interpretability is significantly tighter regret bounds on the maximum deviation compared with black box functions , by appropriately repurposing results from the multi-armed bandit literature in this context . On the other hand , it is less clear that post hoc explanations , which approximate a model locally ( Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ; Dhurandhar et al. , 2018 ) or globally ( Buciluǎ et al. , 2006 ; Hinton et al. , 2015 ) , can help with evaluating the maximum deviation and hence safety . We conduct experiments to illustrate the deviation maximization methods in Section 4 for decision trees , linear and additive models , and tree ensembles . The results in Section 5 quantify how the maximum deviation increases as the size of the certification set increases and as the smoothness of the models decreases . For tree ensembles , we find that the obtained upper bounds on the maximum deviation are informative , showing that the maximum deviation does not increase with the number of trees in the ensemble . We also study the feature combinations that maximize deviation , which can shed light on the sources of extreme model outputs and guide further investigation . Overall , our discussion suggests that a reason for preferring more interpretable models is that it is easier to assess them for unexpected and potentially unsafe outputs . 2 ASSESSING SAFETY THROUGH MAXIMUM DEVIATION . We are given a supervised learning model f , which is a function mapping an input feature space X to an output space Y . We wish to assess the safety of this model by finding its worst-case deviation from a given reference model f0 : X 7→ Y . To do this , we additionally require 1 ) a measure of deviation D : Y × Y 7→ R+ , where R+ is the set of non-negative reals , and 2 ) a certification set C ⊆ X over which the deviation is maximized . Then the problem to be solved is max x∈C D ( f ( x ) , f0 ( x ) ) . ( 1 ) The deviation is worst-case because the maximization is over all x ∈ C ; further implications of this are discussed in Appendix C. We view problem ( 1 ) as a means toward the goal of evaluating safety . In particular , a large deviation value is not necessarily indicative of a safety risk , as two models may differ significantly for valid reasons . For example , one model may capture a useful pattern that the other does not . What large deviation values do indicate , however , is a ( possibly ) sufficient reason for further investigation . Hence , the maximizing solutions in ( 1 ) ( i.e. , the arg max ) are of operational interest . Below we further discuss some elements in this problem formulation . Output space Y . In the case of regression , Y is the set of reals R or an interval thereof . In the case of binary classification , while Y could be { 0 , 1 } or { −1 , +1 } , these limit the possible deviations to binary values as well ( “ same ” or “ different ” ) . Thus to provide more informative results , we take Y to be the space of real-valued scores that are thresholded to produce a binary label . For example , y could be a predicted probability in [ 0 , 1 ] or a log-odds ratio in R. Similarly for multiclass classification withM classes , Y ⊂ RM could be aM -dimensional space of real-valued scores . Models that abstain can also be accommodated as noted in Appendix A . Reference model f0 . The premise of the reference model is that it should be “ safe ” above all . The simplest case mathematically is for f0 to be a constant function representing a baseline value , for example zero . More generally , f0 may be a simple model that can be readily grasped by a human , may be validated against domain knowledge , or may be based on a small number of expert-selected features . Such models are common in medical risk assessment , consumer finance , and predicting semiconductor yield . By simple , we mean for example a linear model with 10 non-zero coefficients or a decision tree with 10 leaves . The reference model could also be an existing model that is not necessarily simple but has been extensively tested and deployed . In this case , f could be a new version of the model , trained on more recent data or improved in some fashion , and we wish to evaluate its safety before deploying it in place of f0 . In this and more complex settings , f0 may not be globally interpretable , but may be so in local regions . The machinery developed in this work could be applied in these settings as well to assess the safety of f ( more discussion in Appendix C ) . Certification set C. The premise of the certification set is that it contains all inputs that the model might conceivably be exposed to . This may include inputs that are highly improbable but not physically or logically impossible ( for example , a severely hypothermic human body temperature of 27°C ) . Thus , while C might be based on the support set of a probability distribution or data sample , it does not depend on the likelihood of points within the support . The set C may also be a strict superset of the training data domain . For example , a model may have been trained on data for males , and we would now like to determine its worst-case behavior on an unseen population of females . For tabular or lower-dimensional data , C might be the entire input space X . For non-tabular or higher-dimensional data , the choice C = X may be too unrepresentative because the manifold of realistic inputs is lower in dimension . In this case , if we have a dataset { xi } ni=1 , one possibility is to use a union of ` p balls centered at xi , C = n⋃ i=1 Bpr [ xi ] , Bpr [ xi ] = { x ∈ X : ‖x− xi‖p ≤ r } . ( 2 ) The set C is thus comprised of points somewhat close to the n observed examples xi , but the radius r does not have to be “ small ” . In addition to determining the maximum deviation over the entire set C , maximum deviations over subsets of C ( e.g. , different age groups ) may also be of interest . For example , Appendix D.1 shows deviation values separately for leaves of a decision tree , which partition the input space . 3 RELATED WORK . Our work relates to a number of different technical directions . Varshney & Alemzadeh ( 2017 ) and Mohseni et al . ( 2021 ) give qualitative accounts suggesting that directly interpretable models are an inherently safe design because humans can inspect them to find spurious elements ; in this paper , we attempt to make those qualitative suggestions more quantitative . Furthermore , several other authors have highlighted safety as a goal for interpretability , but without much further development as done here ( Otte , 2013 ; Doshi-Velez & Kim , 2017 ; Tomsett et al. , 2018 ; Gilpin et al. , 2018 ; Rudin , 2019 ) . Moreover , there is no consensus on how to measure interpretability , which motivates the relationship explored in this paper between interpretability and the ease of evaluating safety . In the area of ML verification , robustness certification methods aim to provide guarantees that the classification remains constant within a radius of an input point , while output reachability is concerned with characterizing the set of outputs corresponding to a region of inputs ( Wong & Kolter , 2018 ; Raghunathan et al. , 2018 ; Huang et al. , 2020 ) . Our problem of deviation maximization ( 1 ) is more closely related to output reachability . The differences in our work are : 1 ) we consider two models , a model f to be assessed and a reference f0 , and are interested in their difference as measured by deviation function D ; 2 ) our focus is global , over a comprehensive set C , rather than local to small neighborhoods around input points ; 3 ) we study the role of interpretability in safety verification . Moreover , works in robust optimization applied to machine learning minimize the worst-case probability of error , but this worst case is over parameters of f rather than over individual values of x ( Lanckriet et al. , 2002 ) . Thomas et al . ( 2019 ) present a framework where during the model training , a set of safety tests is specified in order to accept or reject the possible solution . The specification of these tests is left to the model designer but the goal of the proposed solution is to provide a reusable paradigm to support safety in ML solutions . We build on related literature from model robustness and explainability areas that deals specifically with tree ensembles . Kantchelian et al . ( 2016 ) seek to find the smallest perturbation of an input instance to ‘ evade ’ a classifier using mixed-integer programming ( MIP ) . Optimization formulations are also explored by Parmentier & Vidal ( 2021 ) for the purposes of counterfactual explanations . MIP approaches are computationally intensive however . To address this Chen et al . ( 2019 ) introduce graph based approaches for verification on trees . Their central idea , which we use , is to discretize verification computations onto a graph constructed from the way leaves intersect . The verification problem is transformed to finding all maximum cliques . Devos et al . ( 2021 ) expand on this idea by providing anytime bounds by probing unexplored nodes . Safety has become a critical issue in reinforcement learning ( RL ) with multiple works focusing on making RL policies safe ( Amodei et al. , 2016 ; Zhu et al. , 2019 ; Inala et al. , 2020 ; Rupprecht et al. , 2020 ) . There are two broad themes ( Garcı́a et al. , 2015 ) : ( i ) a safe and verifiable policy is learned at the outset by enforcing certain constraints , and ( ii ) post hoc methods are used to identify bad regimes or failure points of an existing policy . Our current proposal is complementary to these works as we focus on the supervised learning setup viewed from the lens of interpretability . Nonetheless , ramifications of our work in the RL context are briefly discussed in Appendix C .

Summary: **Summary:** * The paper puts forth methodology for (efficiently?) computing the worst-case deviation between two fitted models (one is a "candidate" model, the other is a "reference" model), over some feasible region $\mathcal C$ (that need not be convex). * The takeaway message is that these kind of computations are useful for evaluating the safety of the candidate model, b/c if it (i.e., its predictions) deviate(s) too far from the reference model, then that is a bad sign and someone should "investigate something". * Some calculations are worked out showing that for a variety of model classes, these computations can be done efficiently-ish. * Some experimental results are presented, mainly showing how the deviation varies as the tuning parameters used to fit the candidate model are varied ...