On the Safety of Interpretable Machine Learning: A Maximum Deviation Approach
Authors: Dennis Wei
Rahul Nair
Amit Dhurandhar
Kush R. Varshney
Elizabeth M. Daly
Moninder Singh
Venue: NeurIPS 2022 Accept (2022)
Keywords: safety
interpretability
explainability

Abstract: Interpretable and explainable machine learning has seen a recent surge of interest. We focus on safety as a key motivation behind the surge and make the relationship between interpretability and safety more quantitative. Toward assessing safety, we introduce the concept of *maximum deviation* via an optimization problem to find the largest deviation of a supervised learning model from a reference model regarded as safe. We then show how interpretability facilitates this safety assessment. For models including decision trees, generalized linear and additive models, the maximum deviation can be computed exactly and efficiently. For tree ensembles, which are not regarded as interpretable, discrete optimization techniques can still provide informative bounds. For a broader class of piecewise Lipschitz functions, we leverage the multi-armed bandit literature to show that interpretability produces tighter (regret) bounds on the maximum deviation. We present case studies, including one on mortgage approval, to illustrate our methods and the insights about models that may be obtained from deviation maximization.

On the Safety of Interpretable Machine Learning: A Maximum Deviation Approach
Dennis Wei
Rahul Nair IBM Research rahul www.aiiiiiii.....m. ibm. com
Amit Dhurandhar IBM Research adhuran@us 1bm. com
IBM Research dwei@us. 1mm. com
Kush R. Varshney
Elizabeth M. Daly
Moninder Singh IBM Research moninder@us ibm com
IBM Research krvarshnCus. bmm com
IBM Research
elizabeth. daly@ie ibm. com
Abstract
Interpretable and explainable machine learning has seen recent surge of interest. We focus on safety as a key motivation behind the surge and make the relationship between interpretability and safety more quantitative. Toward assessing safety, we introduce the concept of maximum deviation via an optimization problem find the largest deviation of supervised learning model from reference model regarded as safe. We then show how interpretability facilitates this safety assessment. For models including decision trees, generalized linear and additive models, the maximum deviation can be computed exactly and efficiently. For tree ensembles, which are not regarded as interpretable, discrete optimization techniques can still provide informative bounds. For broader class of piecewise Lipschitz functions, we leverage the multi-armed bandit literature to show that interpretability produces tighter (regret) bounds on the maximum deviation. We present case studies, including one on mortgage approval, to illustrate our methods and the insights about models that may be obtained from deviation maximization.
Introduction
Interpretable and explainable machine learning (ML) has seen recent surge of interest because it is viewed as a key pillar making models trustworthy, with implications on fairness, reliability, and safety [I]. In this paper, we focus safety as a key reason behind the demand for explainability. The motivation of safety has been discussed at a qualitative level by several authors (215). Its role is perhaps clearest in the dichotomy between directly interpretable models
post hoc explanations of black-box models. The former have been called "inherently safe" B and promoted as the only alternative in high-risk applications [3]. The crux of this argument is that post hoc explanations leave a gap between the explanation and the model producing predictions. Thus, unusual data points may appear to be harmless based the explanation, but truly cause havoc. We aim to go beyond these qualitative arguments and address the following questions quantitatively: 1) What does safety mean for such models, and how exactly does interpretability aid safety?
Towards answering the first question, we make a conceptual contribution the form of an optimization problem, intended tool for assessing the safety of supervised lcarning (i.e. predictive) models. Viewing these models as functions mapping an input space to an output space, a key way in which inputs that
these models cause harm is through grossly unexpected outputs, corresponding are poorly represented in training data. Accordingly, we approach safety assessment for a model by determining where it deviates the most from the output of reference model and by how much (i.e., its maximum deviation). The reference model, which represents expected behavior and deemed to be
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

safe, could be model well-understood by domain experts or one that has been extensively "tried and tested." The maximization is done over certification set, a large subset of the input space intended to cover all conceivable inputs to the model. These concepts are discussed further in Section
Towards answering the second question, in Section 4 we discuss computation of the maximum deviation for different model classes and show how this is facilitated by interpretability. For model classes regarded as interpretable, including trees, generalized linear and additive models, the maxi- mum deviation can be computed exactly and efficiently by exploiting the model structure. For tree ensembles, which are not regarded as interpretable, discrete optimization techniques can exploit their composition in terms of trees to provide anytime bounds on the maximum deviation. The case of trees is also generalized in a different direction a broader class of piecewise Lipschitz functions, which we argue cover many popular interpretable functions. Here we show that the benefit of interpretability significantly tighter regret bounds on the maximum deviation compared with general black-box functions, leveraging results from the multi-armed bandit literature. More broadly, the development of tailored methods for additional model classes is beyond the scope of this first work the maximum deviation approach (the black-box optimization of Section 4.4 applicable to all models but obviously not tailored). We discuss in Appendix B.3 some possible approaches, and the research gaps to be overcome, for neural networks and to make use of post hoc explanations, which approximate a model locally 66 or globally ( 10).
In S we present case studies that illustrate the deviation maximization methods in Section foo decision trees, linear and additive models, and tree ensembles. It is seen that deviation maximization provides insights about models through studying the feature combinations that lead to extreme outputs. These insights can in turn direct further investigation and invite domain expert input. We also quantify how the maximum deviation depends on model complexity and the size of the certification set. For tree ensembles, we find that the obtained upper bounds on the maximum deviation are informative, showing that the maximum deviation does not increase with the number of trees in the ensemble.
Overall, our discussion provides a more quantitative basis for safety assessment of predictive models and for prefering more interpretable models due to the greater ease of performing this assessment.
2 Assessing Safety Through Maximum Deviation
We are given supervised learning model f. which is function mapping an input feature space X' to an output space y. We wish to assess the safety of this model by finding its largest deviation from given reference model Jo : X' + Y' representing expected behavior. To do this, we additionally require 1) a measure of deviation D : y x y + R2,, where R is the set of non-negative reals, and 2) certification set C C X over which the deviation is maximized. Then the problem to be solved is max ((f(z), fo())). #EC
(1)
The deviation is worst-case because the maximization is over all E C; further implications of this are discussed in Appendix C Note that (I) is different than typical robust training where the focus is to learn model that minimizes some worst case loss, as opposed to finding regions in X' where two already trained models differ significantly.
We view problem ① as only means toward the goal of evaluating safety. In particular, a large deviation value is not necessarily indicative of a safety risk, as two models may differ significantly for valid reasons. For example, one model may capture useful pattern that the other does not. We thus think that it would be overly simplistic to regard the maximum deviation as just another metric to be optimized in selecting models. What large deviation values do indicate, however, is (possibly) sufficient reason for further investigation. Hence, the maximizing solutions in (T) (i.e., the arg max) are of as much operational interest as the maximum values (this will be illustrated in Section 5)
We now elaborate on elements in problem formulation
Output space y. In the case of regression, y is the set of reals IR or an interval thereof. In the case of binary classification, while y could be {0. 1} or +11, these limit the possible deviations to binary values as well ("same" or "different"). Thus to provide more informative results, we take y to be the space of real-valued scores that are thresholded to produce binary label. For example, y could be a predicted probability in [0, 1] or a log-odds ratio in lR. Similarly for multi-class classification with M classes, y c MM could be -ddimensional space of real-valued scores. In Appendix A we discuss considerations in choosing the deviation function D as well models that abstain.

---

Reference model fo. The premise of the reference model is that it should capture expected behavior while being "safe". The simplest case is for fo be constant function representing a baseline value, for example zero mean prediction. We consider the more general case where Jo may vary with z. Below we give several examples of reference models to address the natural question of how they might be obtained. The examples can be categorized as 1) existing domain-specific models, 2) interpretable ML models validated by domain knowledge, and 3) extensively tested and deployed models. The first two categories are prevalent in high-stakes domains where interpretability is critical.
1. Existing Nomain--pecific models: These models originate from application domain and may not be based on ML at all. For example in consumer finance, several industry-standar models compute credit scores from a consumer's 's credit information (the FICO score is the best-known in the US). Similarly in medicine, scoring systems (sparse linear models with small integer coefficients) abound for assessing various risks (the CHADS2 score for stroke risk is well-known, see the "Scoring Systems: Applications and Prior Art" section of [ for list of others). These models have been used for decades by thousands of practitioners so they are well understood. They may very well be improved upon by a more ML--based model, but for such a model gain acceptance with domain experts, any large deviations from existing models need to be examined and understood.
2. Interpretable models validated by domain knowledge:
Here, an interpretable ML model is learned from data and is validated by domain experts in some way. for example by selecting important input features or by carefully inspecting the trained model. We provide two real examples: In semiconductor manufacturing, process engineers typically want decision trees 112 to model their respective manufacturing process (e.g. etching, polishing. rapid thermal processing, etc.) since they are comfortable understanding and explaining them to their superiors, which is critical especially when things go out-of-spec. Hence, a tree built from data (or any model in general) would only be allowed to make automated measurement predictions if the features it highlights (viz. pressures, gas flows, temperatures) make sense for the specific process. Similarly, in predicting failures of industrial assets such as wind turbines, some failure data is available to train models but experts in these systems (e.g. engineers) may also be consulted. They have knowledge that can help validate the model, for example which components are most likely to cause failures or which environmental variables (e.g. temperature) are most influential.
3. Extensively tested and deployed models: A reference model may also be one that is not necessarily informed by domain knowledge but has been extensively tested, deployed, and/or approved by a regulator. For medical devices that use MI. models, the US Food and Drug Administration (FDA) has instituted a risk-based regulatory system. Any system updates or changes, for instance changes in model architecture, retraining based on new data, or changes intended use (e.g. use for pediatric cases for devices approved only for adults), need to either seek new approvals or demonstrate "substantial equivalence" by providing supporting evidence that the revised model is similar previously approved device. In the latter case, the reference model is the approved device and small maximum deviation serves as evidence of equivalence. As another example, consider a ML-based recommendation model for products of an online retailer or articles on a social network, where because of the scale, a tree ensemble may be used for its fast inference time as well as its modeling flexibility [3]. In this case, a model that has been deployed for some time could be the reference model, since has been extensively tested during this time even though human validation of it may be limited. When new version of the model is trained on newer data or improved in some fashion, finding its maximum deviation from the reference model can serve as one safety check before deploying it in place of the reference model.
Certification set C. The premise of the certification set is that it contains all inputs that the model might conceivably be exposed to. This may include inputs that are highly improbable but not physically or logically impossible (for example, a severely hypothermic human body temperature of 279C). Thus, while C might be based on the support set of a probability distribution or data sample, it does not depend on the likelihood of points within the support. The set C may also be strict superset of the training data domain. For example, a model may have been trained on data for males, and we would now like determine its worst-case behavior on an unseen population of females.
For tabular or lower-dimensional data, C might be the entire input space X. For non-tabular or higher- digher-dimensional data, the choice C = X' may be too unrepresentative because the manifold of
3

---

realistic imputs is lower in dimension. In this case, if we have a dataset {2:)==1> one possibility is to use union of E balls centered at T3+
C
B
BF = {ar X || xillp S sr..
The set C is thus comprised of points somewhat close to the observed examples 24, but the radius does not have to be "small".
In addition to determining the maximum deviation over the entire set C, maximum deviations over subsets of C ge. different age groups) may also be of interest. For example, Appendix DD3 deviation values separately for leaves of decision tree, which partition the input space.
[..e shows
3 Related Work
In previous work on safety and interpretability in ML, the authors of B 14 give qualitative accounts suggesting that directly interpretable models are inherently safe design because humans can inspect them to find spurious elements; in this paper, we attempt to make those qualitative suggestions more quantitative and automate some of the human inspection. Furthermore, several other authors have highlighted safety as goal for interpretability [2] [5 16, 5) but again without quantitative development. Moreover, the lack of consensus how to measure interpretability motivates the relationship that we explore between interpretability and the ease of evaluating safety.
In the area of ML verification, robustness certification methods aim to provide guarantees that the classification remains constant within a radius of an input point, while output reachability is concerned with characterizing the set of outputs corresponding difference in our work is that we consider two models, a model f to be assessed and a reference fo. whereas the above notions of robustness and reachability involve single model. Another important difference is that our focus is global, over a comprehensive set C, rather than local to small neighborhoods around input points; a local focus is especially true of neural network verification (8) We also study the role of interpretability in safety verification. Works in robust optimization applied to ML minimize the worst-case probability of error, but this worst case is over parameters of rather than values of (28). Thomas et al. [29] present framework where during model training, a set of safety tests is specified by the model designer in order accept or reject the possible solution.
region of imputs [17. A major
We build on related literature on robustness and explainability that deals specifically with tree ensembles. Mixed-integer programming (MIP) and discrete optimization have been proposed find the smallest input perturbation to 'evade' a classifier (30) and to obtain counterfactual explanations
31. MIP approaches computationally intensive however. To address this Chen et al. [32] introduce graph based approaches for verification on trees. Their central idea, which we use, is to discretize verification computations onto a graph constructed from the way leaves intersect. The verification problem is transformed to finding all maximum cliques. Devos et al. 33 expand on this idea by providing anytime bounds by probing unexplored nodes.
Safety has become a critical issuc in reinforcement learning (RL) with multiple works focusing on making RL policies safe (74.7]] There are two broad themes 38] : (i) safe and verifiable policy is learned at the outset by enforcing certain constraints, and (ii) post hoc methods are used to identify bad regimes failure points of an existing policy. Our current proposal is complementary these works as we focus on the supervised learning setup viewed from the lens of interpretability.
Nonetheless, ramifications of our work in the RL context briefly discussed in Appendix C
4 Deviation Maximization for Specific Model Classes
In this section, we discuss approaches to computing the maximum deviation I for f belonging to various model classes. We show the benefit of interpretable model structure in different guises. Exact and efficient computation is possible for decision trees, and generalized linear and additive models in Sections 4. and 4.2) Section 4.3 the composition tree ensembles in terms of trees allows discrete optimization methods to provide anytime bounds. For general class of piecewise Lipschitz functions in Section 4.4, the application of multi-arm bandit results yields tighter regret bounds on the maximum deviation. While some of the results in this section may be less surprising, one of our

---

contributions is to identify precise properties that allow them to hold. We also show that intuitive measures of model complexity, such as the number of leaves or pieces or smoothness of functions, have an additional interpretation in terms of the complexity of maximizing deviation. More broadly, the development of methods specific to additional model classes is beyond the scope of a single work. We discuss in Appendix BB.5 possible approaches and the advances needed for neural networks (beyond applying the black-boo methods of Section (44) and to make use of post hoc explanations.
To develop mathematical results and efficient algorithms, we will sometimes assume that the reference model fo is from the same class as f. Appendix C we discuss the case where fu may not be globally interpretable, but may be so in local regions. We will also sometimes assume that the certification set C and other sets are Cartesian products. This means that C II Cj. where for a continuous feature j. C = Xx, is an interval, and for a categorical feature j. C, is set of categories. We mention relaxations of the Cartesian product assumption in Appendix B.2
4.1
Trees
We begin with the case where f and fo are both decision trees. A decision tree with L leaves partitions the input space X' into L corresponding parts, which we also refer to 'leaves'. We consider only non-oblique trees. In this case, each leaf is described by a conjunction of conditions on individual features and is therefore Cartesian product as defined above. With Lı C X' denoting the /th leaf and yi E Y' the output value assigned to it, tree f is described by the function
f()) = y if Li, = 1. L,
and similarly for tree fo with leaves Lom and outputs yom. m = 1..... Lo. As discussed in 39 rule lists where each rule is condition on single feature are one-sided trees in the above sense.
The partitioning of X' by decision trees and their piecewise-constant nature simplify the computation of the maximum deviation (I). Specifically, the maximization can be restricted to pairs of leaves (I,m) m) for which the intersection Li n Zom nC non-empty. The intersection of leaves Li C Zom is another Cartesian product, and we assume that it is tractable to determine whether C intersects a given Cartesian product (see examples in Appendix B.1
For visual representation and later use in Section 4.3 it is useful to define a bipartite graph, with L nodes representing the leaves Li of f on side and To nodes representing the leaves Lom of fo on the other. Define the edge set & = [(1, m) : L1 n Com O o 0); clearly |8] s L. Then
max
D(I(z),fo(z)) = max D(n, Nom). FEC
(4)
(L,m)EE
We summarize the complexity of deviation maximization for decision trees as follows. This a slight refinement of (32 Thm. l] in the case = 2, see Appendix B. for details.
Proposition 1. Let f and fo be decision trees as in 3 with and Lo leaves respectively, and be the bipartite edge set leaf intersections defined above. Then the maximum deviation can be computed with [E] evaluations as shown in
4.2 Linear and additive models
In this subsection, assume that f is generalized additive model (GAM) given by
-x))--- Σ Es(t))
(5)
where each fy is arbitrary function of feature zzy. In the case where fz(z) wyz, for all continuous features Tj. where Wlj is a real coefficient, (5) is a generalized linear model (GLM). We discuss the treatment of categorical features in Appendix B.2 The invertible link function g R R furthermore assumed to be monotonically increasing. This assumption is satisfied by common GAM link functions: identity, logit (g(y) log(u/(1 v))). and logarithmic.
Equation (5) implies that y C and the deviation Dyy, Mo) is function of two scalars y and yo- For this scalar case, we make the following intuitively reasonable assumption throughout the subsection.
5

---

Assumption 1. For y. yo E y S R. 1) D(y, vo) = 0 whenever y = you 2) (y, yo) ) is monotonically non-decreasing in y y 2 yo and non-inereasing
for y S yoo
Our approach is to exploit the additive form of (5) by reducing problem to the optimization
Ma (f, max/min
Σ Es(1g), ==1
(6)
xéS
for different choices of S C A' and where + corresponds to max and - to min. We discuss below how this can be done for two types of reference model fo: decision tree (which includes the constant case Lo 1) and additive. For the first case, we prove the following result in Appendix B.2
Proposition 2. Let f be a GAM as and S be a subset of X where fo(#) = yo is constant. Then if Assumption hoolss
max D(f(z). FES
( (Mo ofit.-}
max
Tree-structured fo- Since fo is piecewise constant over its leaves Com m = I,..., Lo. we take S be the intersection of C. with each Lom in and apply Proposition 2 then obtained as the maximum over the leaves,
The overall maximum is
max max max D(((s), = m- oE(t.-) #EC
D (g Com NC)), yom.
This reduces ①
to solving 20 instances of (6).
Additive fo- For this case, make the additional assumption that the link function g in (5) is the identity function, as well as Assumption 2 below. The implication of these assumptions is discussed in Appendix [B.2
Assumption 2. Dyy, = Dyy - Uo) is function only of the difference y - Voo-
Then = (z,) and the difference f(x) - (z) is also additive. Using Assumptions 2
[add similar argument in the proof of Proposition the maximum deviation is again obtained by maximizing and minimizing an additive function, resulting in two instances of (6) with S C:
max D(f(z). = max D (M - C))
DEC
oe(t.-)
Computational complexity of (6 For the case of nonlinear additive f. we additionally assume that is a Cartesian product. It follows that S = I S, is Cartesian product (see Appendix B.2 for the brief justification) and (6) separates into one-dimensional optimizations over Sj.
d max min (xz). E,
min zES
E1 Σ
(8)
The computational complexity of (8) is thus Σラ Cj, where Cj is the complexity of the jth one- dimensional optimization. We discuss different cases of Cj in Appendix B.2, the important point is that the overall complexity is linear in d.
In the GLM case where Σj= (z²) = wTa. problem 6 is simpler and it is less important that be a Cartesian product. In particular, if a convex set, so too is S (again see Appendix B.2 for justification). Hence (6) is convex optimization problem.
4.3 Tree ensembles
We now extend the idea used for single decision trees in Section to tree ensembles. This class covers several popular methods such as Random Forests and Gradient Boosted Trees. It can also cover rule ensembles [1 as a special case, as explained in Appendix B.3] We assume / a tree ensemble consisting of K trees and fo is single decision tree. Let Li, denote the /th leaf of the kth tree in f for 1 1...., T.g, and Lom the with leaf fo, for m = 1,..., Top Correspondingly let yr, and 30m denote the prediction values associated with each leaf.

---

Define graph G(VE), where there is vertex for each leaf in f and fu. i.e.
V = |4/ = 1,..., K, = 1...., La3 U {m/m = I,..., Fo7.
(9)
Construct an edge for each overlapping pair of leaves in V, i.e.
= (((()))),,, 00, v(i.)) i
(10)
This graph is K + 1-partite graph as leaves within individual tree do not intersect and an independent set. Denote M to be the adjaceney matrix of G. Following Chen et al. [32] a maximum clique S of size K 1 on such a graph provides a discrete region in the feature space with a computable deviation. A clique is a subset of nodes all connected to each other; maximum clique one that cannot be expanded further by adding a node. The model predictions ye and yoe can be ensembled from leaves in S. Denote by (SS) the deviation computed from the clique S. Maximizing over all such cliques solves D However, complete enumeration is expensive, so informative bounds, either using the merge procedure in Chen et al. [2] or the heuristic function in Devos et al. [3 can be used. We use the latter which exploits the K + 1-partite structure of G.
Specifically, we adapt the anytime bounds of Devos et al. 33 as follows. At each step of the enumeration procedure, intermediate clique S contains selected leaves from trees in [1,.--; and unexplored trees in [K + I...., K + 1]. For each unexplored tree, we select a valid candidate leaf that maximizes deviation, i.c.
Uk
arg max D(SUL))
(11)
12 VieS
Using these worst-case leaves, a heuristic function
K+1
H(S) = D(S) = D(S
U vm) m=k+1
(12)
provides an upper (dual) bound. In practice, this dual bound is tight and therefore very useful during the search procedure to prune the search space. Each K + +1 clique provides a primal bound, so the search can be terminated early before examining all trees the dual bound is less than the primal bound. We adapt the search procedure of Mirghorbani and Krokhmal 43 to include the pruning arguments. Appendix [.3 presents the full algorithm. Starting with an empty clique, the procedure adds a single node from each tree to create an intermediate clique. If the size of the clique is K + 1 the primal bound is updated. Otherwise, the dual bound is computed. A node compatibility vector is used to keep track of all feasible additions. When the search is terminated at any step, the maximum deviation is bounded by (Dn,, D.v)
The algorithm works for the entire feature space. When the certification set C is a union of balls as in (2). some additional considerations are needed. First, we can disregard leaves that do not intersect with C during the graph construction phase. A validation step to ensure that the leaves of a clique all intersect with the same ball in C also needed.
4.4 Piecewise Lipsehitz Functions
We saw the benefits of having specific (deterministic) interpretable functions as well as their extensions in the context of safety. Now consider richer class of functions that may also be randomized with finite variance. In this case let f and fo denote the mean values of the learned and reference functions respectively. We consider the case where each function is either interpretable or black box, where the latter implies that query access is the only realistic way of probing the model. This leads to three cases where either both functions are black box or interpretable, or one is black box. What we care about in all these cased js to find the maximum (and minimum) of function A(x) [(x) fo(æ). Let us consider finding only the maximum of as the other case is symmetric. Given that f and fo can be random functions A is also random function and if A is black box standard way optimize it is either using Bayesian Optimization (BO) 44 or tree search type bandit methods 45 46). We
repurpose some of the results from this latter literature in context showcasing the benefit of interpretability from a safety standpoint. To do this we first define relevant terms.
For simplicity assume D(.,) ) to be the identity function.
7

---

Definition 1 (Simple Regret (5)). If fc denotes the optimal value of the function o on the certifica-
tion set C, then the simple regret r after querying the f function times and obtaining a solution 24 is given by, rg
fC f(zy)-
Definition 2 (Order 8 c-LLipschitz). Given a (normalized) metric a function f is c-Lipschitz continuous of order B > 0 if for any two inputs x, y and for > 0 we have, (f(z) f(y)l -
c. ()
Definition 3 (Near optimality dimension 45
If E,, is the maximum number of radius balls one can in C given the metric l and C. = fo (l)(2) 2 fc -c c), then for c >> 0 the c-near
optimality dimension is given by, U = max [ii sup--o
IC. In(e
Intuitively, simple regret measures the deviation between our current best and the optimal solution. The Lipschitz condition bounds the rate of change of the function. Near optimality dimension measures the set size for which the function has close to optimal values. The lower the value of v, the easier it is find the optimum. We now define what means have an interpretable function. Assumption (Characterizing an Interpretable Function). If a function f is interpretable, then we can (easily) find 1 S m & n partitions {C(2) ...., C(m)) } of the certification set C such that the function f = {f()){ E c(0) Vi E m,] in each partition is c-Lipschitz of order B3.
Note that the (interpretable) function overall does not have to be c-Lipschitz of bounded order, rather only in the partitions. This assumption is motivated by observing different interpretable functions. For example, in the case of decision trees the m partitions could be its leaves, where typically the function is a constant in each leaf (c = 0). For rule lists well a fixed prediction is usually made by each rule. For a linear function one could consider the entire input space (i.e. m = 1), where for bounded slope the function would also satisfy our assumption (c and 8 = 1). Examples of models that are not piecewise constant or globally Lipschitz are oblique decision trees (Murthy et al., 1994), regression trees with linear functions in the leaves, and functional trees. Moreover, m is likely to be small so that the overall model is interpretable (viz. shallow trees or small rules). With the above definitions and Assumption we now provide the simple regret for the function A.
1. Both black box models: If both ff and fo are black box then it seems no gains could be made in estimating the maximum of A over standard results in bandit literature. Hence, using Hierarchical Optimistic Optimization (HOO) with assumptions such as C being compact and A being weakly Lipschitz 45 with near optimality dimension the simple regret after q queries is:
[m(q] q
(A) 0
(13)
Both interpretable models: If both I and Jo are interpretable, then for each function based on Assumption 3 we can find my and mo partitions of C respectively where the functions are ei and co -g-Lipschitz of order 31 and 9n respectively. Now if we take non-empty intersections of these partitions where we could have maximum of mo partitions, the function A in these partitions would be c = max(co. Lipssihth of order B min(Bo, 31) as stated next (proof in appendix). Proposition 3. functions he and 1ı are cu and c1 Lipschitz of order 3o and 31 respectively, then the function ho h1 is c- L-iischtiiz of order B, where c 2 max(cu, c1) and B min 81).
Given that A is smooth in these partitions with underestimated smoothness of order 3, the simple regret after qi queries in the 7th partition ((i) with near optimality dimension v; based on HOO is: pec0))
AA) O g0 M((xt))), where U S d' If we divide the overall query budget q across the
q
S 7p non- non-empty partitions equally, then the bound will be scaled by π +/((++2) when expressed as function of Moreover, the regret for the entire C can then be bounded by the maximum regret across these partitions leading to
((
2 4438
4 C( 5o
(14)
Notice that for model to be interpretable ma and my are likely to be small (i.e. shallow trees small rule lists or linear model where m 1) leading to "smallish" и and v can be much >> $ in case 1. Hence, interpretability reduces the regret in estimating the maximum deviation.
8

---

3. Black box and interpretable model: Making no further assumptions on the black box model and assuming A satisfies properties mentioned in case I, the simple regret has the same behavior as (33 This is expected as the black box model could be highly non-smooth.
Case Studies
We present case studies to serve three purposes: 1) show that deviation maximization can lead to insights about models, 2) illustrate the maximization methods developed in Section 4 and 3) quantify the dependence the maximum deviation on model complexity and certification set size (mostly in Appendix D). Two datasets are featured: sample of US Home Mortgage Disclosure Act (HMDA) data (see Appendix D.2 for details), meant as a proxy for mortgage approval scenario, and the UCI Adult Income dataset I471. a standard tabular dataset with mixed data types. A subset of results is shown in this section with full results, experimental details, and an additional Lending Club dataset in Appendix D Since these binary classification datasets, we take the deviation function D to be the absolute difference between predicted probabilities of class 1. For the certification set C, we consider union of & balls (2) centered at test set instances. While we have used the test set here, any not necessarily labelled dataset would suffice. The case r = 0 yields a finite set consisting only of the test set, while r + oo corresponds to C being the entire domain X. We reiterate that the dependence of the certification set on a chosen dataset is only on (an expanded version of) the support of the dataset and not on other aspects of the data distribution.
To demonstrate insights from deviation maximization, we study the solutions that maximize deviation (the arg max in and discuss three examples below.
Identification of an artifact: The first example comes from the Adult Income dataset, where the reference model fo is decision tree (DT) and ff is an Explainable Boosting Machine (EBM) [48], a type of GAM (plots of both in Appendix [D.3). Here, the capital loss feature the largest contributor to the maximum deviation (the discussion below explenex how this is determined), and Table in Appendix [D3 shows that as the certification set radius r increases, the maximizing values of capital loss converge to the interval [1598, 1759]. The plot of the GAM shape function Sn for capital loss in Figure F shows that this interval corresponds to a curiously low value of the function. This low region may be artifact warranting further investigation since it seems anomalous compared to the rest of the function, and since individuals who report capital losses on their income tax returns to offset capital gains usually have high income (hence high log-odds score). Note that this potential artifact was automatically identified through deviation maximization.
For the next two examples, we consider a simplified mortgage approval scenario using the HMDA dataset. Suppose that a DT fo (shown in Figure Appendix (.2) has been trained to make final decisions on mortgage applications. Domain experts have determined that this DT is sensible and safe and are now looking to improve upon by exploring EBMs. (Logistic regression (LR) models are deferred to Appendix D.2 because they do not have higher balanced accuracy than fo-)
Conflict between fo: We first examine solutions that result in the most positive difference between the predicted probabilities of an EBM f with parameter max max_bins=32 and the DT fo- These all occur leaf of fo (leaf 2 in Figure 3 where the applicant's 's debt-to-income (DTT) ratio is too high (> 52%) and fo predicts low probability of approval. The other salient feature of the solutions that they all have 'reapprovall= 11, indicating that preapproval was requested, which is given a large weight by the EBM I in favor of approval (see Figure in Appendix [.2] and Table 3 for more feature values). Thus I and fo are in conflict. Among different ways in which the conflict could be resolved, domain expert might decide that fo remains correct in rejecting risky applicants with high DTI, even if there is a preapproval request and the new EBM model puts a high weight on it.
Trend toward extreme points, deviation can be good: We now look at solutions that yield the most negative difference between the predicted probabilities of f and fo. for r S 0.8. Table shows the 6 features that contribute most to the deviation (again see Appendix D.2 for details). All of these points lie in a leaf of fo (leaf 14 in Figure 3] denoted L14) that excludes several clearer-cut cases, with the result being less confident predicted probability of 0.652 from fo. The feature values that maximize deviation tend toward extreme points of the region L1.. Specifically, the values of the continuous features debt-to-income ratio, loan-to-value ratio, property value, and income all move in the direction of application denial. For the latter three features, the boundary of L14 is reached soon as 0.1, whereas for tebt-to-income ratio, this occurs at 0.4. The movement

---

debt to income (%)
state
loan
value (%)
prop value (000S)
income (000S)
0.0
46.0
CA
95.0
3
415
77.0
0.1
[45.9 46.5]
none
[100. 100,92]
[120 1201
$28.5
0.2
[45.5 46.5]
none
[100. 100.92]
[120 120]
$28.5
0.4
[S2. 52.]
none
[100. 100.92]
1
[120 120]
$28.5
0.6
[52. 52.]
none
[100. 100.92]
1
[120 120]
$28.5
0.8
[52. 52.]
[100. 100.92]
1
[120 120]
$28.5
Table 1: Values of top features that maximize difference in predicted probabilities between a decision reference model fo and Explainable Boosting Machine f (max_ bins 32) on the HMDA dataset. For radius r > O, the maximizing values continuous features form an interval because the corresponding EBM shape functions fj are piecewise constant.
1.0
pttmll bound
0.8
Stone
stnole
1000
Figure 1: EBM shape function fy for capital loss feature, showing anomalously low interval identi- fied by deviation maximization.
Forest
Figure 2: Maximum deviation D the Adult Income dataset as a function of number of estimators in Random Forest.
toward extremes is expected for this / since its relevant shape functions Jj are mostly increasing or decreasing, as seen in Figure in Appendix D.2 The same behavior is observed in other GAM In this example, domain expert might conclude that the large
and LR examples deviation is in fact desirable because f is providing varying predictions in C14 in ways that make sense, as opposed to the constant given by fo- This shows that deviation maximization can work in both directions, identifying where the reference model and its assumptions may be too simplistic and giving an opportunity to improve the reference model.
Appendix [
Maximum deviation vs. number of frees in RF: In Figure 2 we highlight one result from set of such results in Appendix [ showing maximum deviation as function model complexity, here quantified by the number of estimators (trees) in Random Forest (RF). This is demonstration of the method Section 44.] which general provides bounds on the maximum deviation. In this case, the upper ("dual") bound is informative enough actually show a decrease as the number of estimators increases. The larger number of estimators increases averaging and may serve to make the model smoother.
6 Conclusion
We have considered the relationship between interpretability and safety in supervised learning through two main contributions: First, the proposal maximum deviation as means toward assessing safety, and second, discussion of approaches to computing maximum deviation and how these simplified by interpretable model structure. We believe that there is much more to explore in this relationship. Appendices C and B.5 provide further discussion of several topics and future directions.
Acknowledgements
We thank Michael Hind for several early discussions the topic of ML model risk assessment that inspired this work, and for his overall leadership on this topic. We also thank Dhaval Patel for a discussion the industrial assets example in Section 2
10

---

References
[1] Kush R. Varshney. Trustworthy Machine Learning. Independently Published, Chappaqua, NY, USA, 2022.
[2] Clemens Otte. Safe and interpretable machine learning: A methodological review. Computational Intelligence in Intelligent Data Analysis, pages 111-122. 2013.
[3] Kush R. Varshney and Homa Alemzadeh. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big Data, 5(3):246-255, September 2017.
[4] Finale Doshi Doshi-Velez and Been Kim. Towards rigorous science of interpretable machine learning arXiv preprint arXiv: 2017.
[5] Cynthia Rudin. Stop explaining black box machine lcarning models for high stakes decisions and usc interpretable models instead. Nature Mach. Intell., 155):206-215, May 2019.
[6] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?": Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1135- 1144, 2016.
[7] Scott M. Lundberg and Su-In Lec. A unified approach to interpreting model prodictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.
[8] Amit Dhurandhar, Pin- Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, pages 192-603, 2018.
[9] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2006.
[10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. arXiv:1503 005311 2015.
Distilling the knowledge in neural network.
[11] Cynthia Rudin and Berk Ustun. Optimized scoring systems: Toward trust in machine leamning for healthcare and criminal justice. Interfaces, 48(5):399-48) October 2018.
[12] Amit Dhurandhar, Karthikeyan Shanmugam, Ronny Luss, and Peder Olsen. Improving simple models with confidence profiles. In Advances in Neural Information Processing Systems, pages 10296-10306, 2018.
[13] Aleksandar Ilie and Oleksandr Kuvshynov. Evaluating boosted decision trees for bil- lions of users, 2017. URL btttp::/eeggneeriing fb. com 2017/03/27/mm http:////mmmppppiaaiio.n.s..
wraluating-boose wwwpeetddddeeiioonn--e trees fowffforbllbllee www.bbilllllbboeee.eeeeeeeael.we of uusess Last accessed: 2021-10.
[14] Sina Mohseni, Zhiding Yu, Chaowei Xiao, Jay Yadawa, Haotao Wang, and Zhangyang Wang. Practical machine learning safety: A survey and primer. arXiv:2106.0482,, 2021.
[15] Richard Tomsett, Dave Braines, Dan Harborne, Alun Preece, and Supriyo Chakraborty. Interpretable to whom? A role-based model for analyzing interpretable machine learning systems. In Proc. ICML Workshop Human Interpret. Mach. Learn., pages 8-1,, 2018.
[16] Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In Proc. IEEE Int. Conf. Data Sci. Adv. Anal., pages 80-89, 2018.
[17] Xiaowei Huang. Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. survey of safety and trustworthiness deep neural networks: Verification, testing. adversarial attack and defence, and interpretability. Computer Science Review, 37000220), 2020.
[18] Rüdiger Ehlers. Formal verification of piece-wise linear -orward neural networks. In Automated Technology for Verification and Analysis, pages 269-286, 2017.
[19] Lily Weng. Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation certified robustness for ReLU networks. In Proceedings of the International Conference on Machine Learning, pages 5276-5285, July 2018.
[20] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Proceedings of the International Conference Machine Learning, pages 5286-5295 2018
11

---

[21] Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy Mann, and Pushmeet Kohli. A dual approach scalable verification of deep networks. In Proceedings the Conference on Uncertainty in Artificial Intelligence, August 2018
[22] Gagandeep Singh. Timon Gehr, Matthw Mirman, Markus Püschel. and Martin Vechev. Fast and effective robustness certification. Advances Neural Information Processing Systems, volume 31, 2018.
[23] Aditi Raghunathan, Jacob Steinhardt, and Percy S. Liang. Semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems, volume 31, 2018.
[24] Guy Katz, Derek A. Huang, Duligur Ibeling. Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Alcksandar Zeljić, David L. Dill, Mykel J. Kochendcrfer, and Clark Barrett. The Marabou framework for verification and analysis of deep neural networks. In Computer Aided Verification, pages 443-452, 2019.
[25] Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In Procedings of the International Conference on Learning Representations, 2019.
[26] Ross Anderson, Joey Huchette, Will Ma, Christian Tjandraatmadja, and Juan Pablo Vielma. Strong mixed-integer programming formulations for trained neural networks. Mathematical Programmingg 183:
3-39, 2020.
[27] Sumanth Dathathri, Krishnamurthy Dvijotham, Alexey Kurakin, Aditi Raghunathan, Jonathan Uesato, Rudy R. Bunel, Shreya Shankar. Jacob Steinhardt, Ian Goodfellow, Percy S. Liang, and Pushmeet Kohli. Enabling certification of verification-agnosiic networks via memory-efficienn semidefinite programming. In Advances in Neural Information Processing Systems, volume 33, pages 5318 5331, 2020.
[28] Gert R. G. Lanckriet, Laurent El Ghaoui, Chiranjib Bhattacharyya, and Michael I. Jordan. A robust minimax approach to classification. J. Mach. Learn. Res., 3:555-582, 582, December 2002.
[29] Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. Preventing undesirable behavior of intelligent machines. Science, 366(6468):999 1004, 2019.
[30] Alex Kantchelian. J. Doug Tygar, and Authony Joseph. Evasion and hardening of tree ensemble classifiers. In Proceedings of the International Conference on Machine Learning, pages 2387-2396, 2016.
[31] Axel Parmentier and Thibaut Vidal. Optimal counterfactual explanations in tree ensembles. In Proceedings of the International Conference Machine Learning, pages 1422-8431, 2021.
[32] Hongge Chen, Huan Zhang. Si Si, Yang Li, Duane Boning, and Cho-Jui Hsich. Robustness verification of tree-based models. Advances in Neural Information Processing Systems, 32:12317 123288, 2019.
[33] Laurens Devos, Wannes Meert, and Jesse Davis. Versatile verification of tree ensembles. In Proceedings of the International Conference on Machine Learning, pages 2654-2664, 2021.
[34] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems AI safety. 100.:1666..0665,, 2016.
[35] He Zhu, Zikang Xiong, Stephen Magill, and Suresh Jagannathan. An inductive synthesis framework for verifiable reinforeement learning. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 686 686-701, 2019.
[36] Jeevana Priya Inala, Oshert Bastani, Zenna Tavares, and Armando Solar-L.czama. Synthesizing program- matic policies that inductively generalize. Proceedings of the International Conference on Learning Representations, 2020.
[37] Christian Rupprecht, Cyril Ibrahim, and Christopher J. Pal. Finding and visualizing weaknesses of deep reinforcement learning agents. In Proceedings of the International Conference on Learning Representations. 2020.
[38] Javier Gareía and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437 1480, 2015.
[39] Hongyu Yang. Cynthia Rudin, and Margo Seltzer. Scalable Bayesian rule lists. In Proceedings of the International Conference on Machine Learning, page 3921-3930. 2017.
[40] Elaine Angelino, Nicholas Larus-Stone Daniel Alabi, Margo Seltzer, and Cynthia Rudin. Learning certifiably optimal rule lists for categorical data. Journal of Machine Learning Research, 18(234):1-78. 2018.
12

---

[41] Jerome H. Friedman and Bogdan E. Popescu. Predictive learning via rule ensembles. Annals of Applied Statistics, 2(3):916-954, July 2008.
[42] Krzysztof Dembezyíski, Wojeiech Kotlowski, and Roman Slowinski. ENDER: statistical framework for boosting decision rules. Data Mining and Knowledge Discovery, 21(1):52-90, July 2010.
[43] M. Mirghorbani and P. Krokhmal. On finding k-cliques in k-partite graphs. Optimization Letters, 7(6): 1555-116,, 2013.
[44] Peter Auer. Using confidence bounds for exploitation- trade-offs. Journal of Machine Learning Research, 3:397-422, 2002.
[45] Sebastien Bubeck, Remi Munos, Gilles Stoltz, and Csaba Szepesvari. arame bandits. Journal of Machine Learning Research, 12:1655 1695, 2011.
[46] Alexandra Carpentier and Michal Valko. Simple regret for infinitely many armed bandits. In Proceedings of the International Conference on Machine Learning, pages 1133-1141, 2015.
[47] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL hwtp::ttatc/iieee..... iis.ucc.uu edu/m1
[48] Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: unified framework for machine learning interpretability, 2019.
[49] Peter Bartlett and Marten H. Wegkamp, Classification with reject option using hinge loss. Journal of Machine Learning Research, ((59)18823-1440 2008.
[50] William W. Cohen and Yoram Singer. A simple, fast, and effective rule learner. In Proc. Conf. Artif. Intell., pages 335 335-342, 1999.
[51] Ulrich Rückert Stefan Kramer. statistical approach to rule learning. Proc. Int. Conf. Mach. Learn.. pages 785-792, 2006.
[52] Dennis Wei, Sanjeeb Dash, Tian Gao, and Oktay Günlük. Generalized linear rule models. In Proceedings of the International Conference on Machine Learning, pages 6687-6696, June 2019.
[53] Roger Koenker. Quantile Regression. Cambridge University Press, 2005.
[54] Nicolai Meinshausen. Quantile regression forests. Journal of Machine Learning Research, 7(6):983 999, 2006.
[55] Gábor Petneházi. QCNN: Quantile convolutional neural network. arXiv:1908.0m 077978, 2019.
[56] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey F. Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, volume 34, pages 4699-4711, 2021.
[S7] Isha Puri, Amit Dhurandhar, Tejaswini Pedapati, Karthikeyan Shanmugam, Dennis Wei, and Kush Varshney. CoFrNets: Interpretable neural architecture inspired by continued fractions. In Advances in Neural Information Processing Systems, volume 34, pages 21668-21680, 2021.
[58] Laurent Meunier, Blaise J Delattre, Alexandre Araujo, and Alexandre Allauzen. A dynamical system perspective for Lipschitz neural networks. In Proceedings of the International Conference on Machine Learning, pages 15484-15500, July 2022.
[59] Ronny Luss, Pin-Yu Chen, Amit Dhurandhar, Prasanna Sattigeri, Karthik Shanmugam, and Chun-Chen Tu. Leveraging latent features for local explanations. In Proceedings of the ACM SIGKDD International COnference on Knowledge Discovery and Data Mining, 2021.
[60] Adrian Weller. Transparency: Motivations and challenges. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pages 23-40 40-40.20 2019.
[61] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems, 2018
[62] High Level Expert trustworthy AI for
Group on Artificial Intelligence. self assessment. URL https::/ddigitaa-ssraate ec europa. ew/en/1ibrary/ assessment- list- artptrootthy-aatttiiiial---a-e-e
Assessment list for Technical report, European Commis-
sion. 2020.
altai self assessment
13

---

[63] Mona Sloane. Emanuel Moss, and Rumman Chowdhury. A silicon valley love triangle: Hiring algorithms. pseudo-science, and the quest for auditability. arXix:2106.12410 2021.
[64] Matthew Amold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mchta, Aleksandra Mojsilović, Ravi Nair, Karthikeyan Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, Darrell Reimer, John Richards, Jason Tsay, and Kush R. Varshney. FactSheets: Increasing trust in AI services through supplier's & declarations of conformity. IBM Journal of Research and Development, 63(4/5):6, 2019.
[65] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Griscl, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, Cournapeau, M. Brucher, M. Perrot, and E. Duchesmay. Scikit-learn Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[66] Navdeep Gill, Patrick Hall, Kim Montgomery, and Nicholas Schmidt. A responsible machine learning workflow with focus on interpretable models, post-hoc explanation, and discrimination testing. Information, 11)),, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix C in particular.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix [
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [No] The code is proprietary at this time due
institutional obligations.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [N/A] We report case studies assessing the safety of fixed models.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] The HMDA data comes from the US
government and does not appear to have a license.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'ne using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] None of the data contain personally identifiable information, and the US Consumer Finance Protection Bureau indicates that they have modified the HMDA data to protect applicant and borrower privacy.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "jWlf8KNIA7s", "reviewer": "Reviewer_V5qC", "summary": "This paper defines a new \"safety\" measure for a ML model given a good reference model (normally an interpretable model). The safety metric is the maximum of outputs deviation between the target model and the reference model for some input sets. The authors then analyze the \"safety\" metric for some common ML models, including decision trees, generalized linear and additive models, tree ensembles, and piecewise Lipschitz functions. The authors then show how \"safety\" metric on two real world datasets.", "questions": "- An interpretable model may sound good, but it won't get 100% correctness. Often interpretable model will sacrifice performance for interpretability. What if the baseline model is wrong on some inputs but the target model is actually correct. In that case, how would the safety measure address this issue? \n- In practice, how do you choose the certification set? You may not have access to the test set in reality and domain experts' inputs may just be a few data samples with label. In that case, how would one define the certification set based on the limited inputs?\n- This question might be less related. But I'm wondering, how would this safety metric be used by real-world ML applications? Do the users train a few models and choose the safest one by the metric regardless of the model's overview performance? If the certification set contains a few data samples, can the user just run a sanity check for each model and drop those models with unsafe outputs?", "limitations": "- I think this paper describes how to compute the safety score for some sets of common ML models. Extra work is required to generalize this metric to all ML models. \n- Depends on the certification set, it may not be practical to compute the safety score for some ML models.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of using a reference model to measure how safe another model is looks quite novel. \n- The authors describe the motivation and the actual idea well. Most parts of this paper are relatively easy to follow.\n\nWeaknesses:\n- The discussions on how the baseline model and the certification set are chosen are quite limited. It was not very convincing that the current form is what the idea of two model safety measure would be. See my questions in the section below.\n- The experimental section is not easy to follow. There are too many discussions/figures or other information on the appendix. Readers have to go back and forth multiple times between the main body and appendix. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "bTSPZzDOBxI", "reviewer": "Reviewer_yACw", "summary": "This paper proposes to use maximum deviation, defined by the difference between a reference model and a trained model over a superset of the training data,  achieving safety and interpretability for supervised learning models.  Such a measure can be efficiently computed for decision trees, generalized linear and additive models. The idea of using maximum deviation to measure the model safety is novel.  However, the proposed approach is somewhat too restrictive to generalize. ", "questions": "1. How to specify a reference model f_0 in practice? Since  we do not have any prior on the relationship between input and output,  an improper specification on f_0 may result in misleading interpretable result. \n\n2. The theoretical part is not surprise to me. In particular, the authors argue that when f and f_0 are both decision trees, the complexity of computing maximum deviation cost  in O(\\epsilon), which is actually stated in [1] for verifying the robustness of single tree. Besides, a pioneering work [2] proposed a verification of trees using anytime searching algorithm to address multiple verification tasks on ensemble trees. So the method described in section 4.3 is basically a generalization of [2]. \n\n3. Consider that modern machine learning models are usually with complex structure (e.g. deep neural networks) when dealing with big data. The proposed framework is too restrict to generalize. My concern lies with when the reference model f_0 is a complex DNN, is there an efficient way to compute the corresponding maximum deviation, especially when f is also a DNN. How about the corresponding computational complexity. \n\n4. It would be better to add some baseline comparisons to illustrate the safety of a specific model. \n\n[1] Hongge Chen, Huan Zhang, Si Si, Yang Li, Duane Boning, Cho-Jui Hsieh, robustness verification of tree-based models, NeurIPS, 2019. \n[2] L. Devos, W. Meert and J. Davis, Versatile verification of tree ensembles, ICML, 2021. \n\n", "limitations": "Please refer to Questions", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The topic is quite interesting and important. \n2. Rich experimental results.  \n3. Well written and organized. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1o7fqyJAmbw", "reviewer": "Reviewer_aVuR", "summary": "This paper proposes a new conceptual framework to assess the safety of model via maximum deviation from a reference model over a pre-specified certification dataset. The paper also motivates how such reference model along with certification dataset might be chosen depending on the problem. With this quantification of safety, the paper then demonstrates the computation of this deviation for interpretable models like trees and generalized additive models. They also discuss discrete optimization techniques to compute this deviation metric in case of tree ensembles which are not considered interpretable. Finally they discuss the case of a more general piecewise Lipschitz functions and show how interpretable models can lead to more efficient computation of their metric. \n\nThrough cases studies, the paper demonstrates how this framework can be used to identify regions within feature space where the deviation is large. This can be further subjected to investigation, helping in improving safety of models. They also demonstrate the case where deviation bounds may be used to study the effect of number of estimators in the random forest on safety of the model as defined within their framework. ", "questions": "1) It would be interesting to have a set of thumbrules/guidelines on the selection of deviation function D(y, y_0) under different scenarios. Have the authors generated any results on this?\n2) I would like to know if the authors have tried to assess models like deep neural networks (since they are not intepretable according to the assumptions in the paper - will this reduce to worst case scenario when computing maximum deviation?). \n\n", "limitations": "Yes, addressed in the appendix. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The conceptual framework proposed in this paper is intuitive and helps reason about the safety of a predictive model. \n2) The paper demonstrates the computation of deviation or bounds on deviation for trees, GAMs and tree ensembles along with a broader class of piecewise Lipschitz functions. This helps in understanding the practicality of this framework under different situations\n3) The case studies show how this framework can be useful not only in the assessment of regions within feature space that are problematic, but also in choosing hyperparameters such that the models are safe w.r.t this deviation metric\n\nWeaknesses:\n1) Choosing a reference model, certification dataset and the deviation function D(y, y_0) may not be a trivial exercise while assessing the safety of non-interpretable model. It is not straightforward to reason about the tightness and usefulness of bounds computed under these circumstances. \n2) In the case where the model is interpretable, we still need to make several assumptions (e.g. C is cartesian product while optimizing for non-linear additive f). While many of these assumptions made in the paper look reasonable, it may limit the applicability of the method to several practical scenarios involving interpretable models. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept