Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: **Summary:**
* The paper puts forth methodology for (efficiently?) computing the worst-case deviation between two fitted models (one is a "candidate" model, the other is a "reference" model), over some feasible region $\mathcal C$ (that need not be convex).
* The takeaway message is that these kind of computations are useful for evaluating the safety of the candidate model, b/c if it (i.e., its predictions) deviate(s) too far from the reference model, then that is a bad sign and someone should "investigate something".
* Some calculations are worked out showing that for a variety of model classes, these computations can be done efficiently-ish.
* Some experimental results are presented, mainly showing how the deviation varies as the tuning parameters used to fit the candidate model are varied ...
main_review: **Strong points:**
* Great topic.
* Good collection of results (to be blunt: I'm not sure the results are all that surprising, but it is nonetheless good to have them written down somewhere).
* Paper easy to read.

**Weak points:**
* I think the experimental evaluation is not that convincing.  I get that it's kind of challenging to evaluate the methodology presented here, b/c (i) there seem to not be any real baselines, and (ii) it's hard to come up with an objective evaluation criteria.  But I do think there are a couple of things the authors could do to get around these issues:
    * Can you generate a synthetic data set (based on some specification of $\mathcal C$) that you essentially use as a hold out set?  I.e., evaluate both the reference and candidate model on this hold out set, and compare the two accuracies.  Now how does this number compare with the deviation you compute?  This is probably the simplest possible baseline that might still serve as "check" on your method, i.e., that it is doing something sensible.
    * You cite Wong and Kolter (2018) in your related work section.  That paper produces bounds on the robust error.  If you compare those numbers for two models, how do they compare to the deviations you get?  Perhaps you could utilize other related works in a similar way.
* As an alternative to the above, it would probably be worthwhile (and a good contribution) to carry out some kind of case study, where you run your methodology on a real-world data set, and draw out some interesting insights/findings.  (I didn't really see this being done in the appendix, but maybe I missed it.)

**Questions:** see my "Weak points" and "Additional feedback" sections.

**Additional feedback:**
* If you use the black box methodology (Sec 4.4) even when you have "closed-form" methodology (Sec 4.1) available, what happens?  How well/poorly does the black box methodology do?  Is it a bad idea to just always use the black box methodology, by default?
* To handle neural networks, there is some interesting recent work that reparametrizes a neural network in a certain way, winding up with a linear model.  I guess this could conceivably put neural networks into the more tractable part of your framework?  That seems like it could be pretty useful ...
summary_of_the_review: **Recommendation:** reject.  I like the paper and topic a lot.  But it's a bit hard to objectively evaluate the paper as it is currently written.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In the paper the authors proposed the maximum deviation approach to study the safety of a predictive model by measuring its distance to an interpretable, "safe" function on a certification set. The authors explicitly showed how to compute such measure between function classes of linear, generalized additive, tree based, and piecewise Lipschitz models. 
main_review: The paper raises an important question regarding the assessment of model safety, defined by the authors as the minimax rate over a certification set, usually a superset of the input space, with respect to a safe and interpretable alternative model. However, the reviewer has some major concerns regarding the novelty and the development of the proposed methods. 

Major issues:
1. The definition and motivation of "safety" could use more justification. Though the proposed minimax rate is a legit distance measure between a given model and a interpretable model, it is also a direct measure for the extrapolation behavior / generalizability / robustness, of the model, while this will trivialize the novelty of the "safety" aspect and question the necessity of defining a new term. The idea of using an interpretable function as a reference should also be better motivated and supported theoretically and empirically. It is unclear why we need a two-function approach, especially: 
- The examples (existing practices / real world cases) that exercise it, which seem not available in existing literature. 
- If the intended maximum deviation is large, why not directly compare to a constant function / measure the minimax.
- If the intended maximum deviation is small, what is the purpose of forcing another model to behavior almost identically w.r.t. the safe one.

2. The discussion regarding how to compute the maximum deviation can be rebalanced. The linear and the tree parts are relatively simple, whereas the more interesting piecewise Lipschitz scenario with regrets could be further extended. The derivation in its current shape leaves a couple of questions.
- The linear models and the tree models both have tractable extrapolation patterns, therefore it appears weird why to use them as f.
- The setup of a cartesian set as the certification set implies compactness, which seems to result all continuous functions to piecewise Lipschitz of any order \beta (m \to \infty when \beta \to 0). The optimal minimax rate in (14) is then likely to be dominated by the constant term overall. On the other hand, it would be useful to have examples corresponding to the functions discussed here, especially pertaining to realistic model that is discontinuous and not piece-wise constant (e.g. functional trees).

3. The empirical study is also insufficient since it only covers one dataset and one f in the main body. It therefore is hard to argue the practical value of the proposed method. 


summary_of_the_review: While the authors proposed a maximum deviation approach to measure model safety, the overall research question remains ambiguous and both theoretical and empirical supports for the proposed method and its claims are insufficient.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed inspecting the deviation between the reference model (e.g., black-box model) and its approximation (e.g., white-box model).
The major motivation is on inspecting the safety of the reference black-box model through the approximated white-box model.
The safety, such as the possible maximum output within the prescribed input domain, of white-box model, such as decision tree and linear model, are easier to inspect.
Thus, if we can evaluate the maximum deviation between the black-box model and the white-box model, we can evaluate the safety of the black-box model as well.
Based on this idea, the paper considers some possible approaches for inspecting the maximum deviation between the models.
In the experiments, the authors demonstrated that the maximum deviation are considerably large in practice.
main_review: ### Strength
#### [Strength1]
The idea of inspecting the safety of black-box models through the approximated white-box model wills be novel and interesting.
This idea will be able to relax the difficulty of directly inspecting the safety of black-box models, where some ideal assumptions are required on the black-box models under consideration.
By putting the approximated white-box model in the middle of the inspection, the task turns into inspecting the gap between the two models, where we will need some assumptions only on the difference of the models but not the models themselves.

#### [Strength2]
The authors tried to inspect the maximum deviation under a piece-wise Lipschitzness assumptions.
In particular, the authors adopted Assumption 3 trying to characterize interpretable white-box models as piece-wise Lipschitz functions.
Although this may not always be true, nevertheless this assumption itself would be an interesting idea in the area of interpretable machine learning where interpretability is ambiguous.

### Weakness
#### [Weakness1]
The theoretical results on specific model classes in Section 4.1 and 4.2 on the deviation between decision trees and generalized additive models are straightforward and trivial.
The results on tree ensembles in Section 4.3 are less trivial, although the proposed method is a simple extension of the pioneering studies of Chen et al. (2019) and Devos et al. (2021).

I think the most important results are on Section 4.4 where we do not require specific model classes except that the models are piece-wise Lipschitz.
In practical machine learning, it is not realistic to restrict the model class to be deployed: the models can be complex deep neural networks or ensemble of several different models.
The current result in Section 4.4 is just borrowed from Bubeck et al., (2011).
I would expect to see further investigations over piece-wise Lipschitz models.
For example, with what assumptions (without specifying model classes), we can derive nontrivial better results than the general result of Bubeck et al., (2011).

#### [Weakness2]
The experimental results on Adult Income dataset (binary classification) showing that the maximum deviation on the positive class probability close to one seems to be trivial.
If the two classification models are not identical, it is not surprising that the two models have conflicted predictions for some inputs.

This observation leads to a natural question, in what circumstances we can obtain non trivial maximum deviation (e.g., the maximum deviation less than 0.1)?
For example, suppose the reference model is the depth 5 decision tree, and the approximation is the depth 3 decision tree.
Do we need the depth 5 reference model to be sufficiently close to the depth 3 approximated model to obtain the non trivial maximum deviation?
Or, is there a possibility that the maximum deviation can be small for some depth 5 reference model distinct from the depth 3 approximated model?
summary_of_the_review: I like the idea of inspecting the safety of black-box models through approximate white-box models, and I think some ideas (Assumption 3 in particular) are interesting and novel.

However, the main theoretical results are trivial or simple modifications of the existing results.
Moreover, the experimental results showing only trivial deviations lead to a question whether it is possible to obtain non trivial maximum deviation in practice.

Overall, I think the paper is interesting in its underlying idea.
However, the technical novelty is marginal, and there remains a fundamental question when we can have non trivial maximum deviation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors posit that the demand for explainable/interpretable models in machine learning is linked to safety. In other words, domain experts trust explainable or interpretable models more. Based on this premise, they suggest to assess the safety of a model by measuring its maximum deviation of a model to a reference model, which is supposed to be safe. This deviation is computed on a certification set of data points. Additionally, the authors show that their proposed maximum derivation can be computed exactly for some interpretable models, such as decision trees, rule lists and generalized linear and additive models. Further, they discuss implications for tree ensembles and empirically evaluate their approach.
main_review: Strong points of the paper include the following:

+ the main idea of the paper, i.e. using the maximum deviation to assess the safety of a model w.r.t. a baseline model, is well explained
+ the authors show that for some simple models, e.g. trees and rule lists, their proposed measure can be computed exactly
+ the authors propose a way to characterize interpretable functions via piecewise Lipschitz functions and give an interpretation in terms of tree models
+ the contributions seem to be technically correct
+ the approach is empirically validated and details to parameters and data sets are provided

Weaknesses include:
- When discussing related work, the authors mention that as opposed to previous work of Mohseni et al. (2021), their proposed approach can quantify safety through interpretability.  This is a quite bold statement, since the approach needs a reference model f_0 and judges safety of a new model f based on the difference compared to that model (f_0 needs to be provided). Then the question is, whether the baseline model is safe, which is difficult to answer by itself (especially if the baseline model is not very simple).
- Also, to me the proposed maximum deviation measure seems a bit conservative for evaluating the safety risk of a model. A large deviation between the prediction of two models could also mean that the new model is an improved estimator---i.e. it can model parts of the data that the old model did not model (put into the same leaf, for the example of decision trees). 
- Last, it would be nice if the obtained score provides a guidance for domain experts to retrain a safer model in terms of the score.

Minor remark: Using H(S) in Eq. 12 is a bit confusing since H usually corresponds to entropy.

summary_of_the_review: In summary, the technical contributions and the evaluation of the approach seem solid. However, I do not see how exactly the proposed dissimilarity (the maximum deviation) can help a domain expert to retrain a safer model based in its output, which seems like the goal of such a metric. Inspecting every subset of data points on which the new model and the baseline differ substantially might take a lot of time.
review_title: null

Meta-review: This paper proposes a metric for the safety and interpretability of supervised learning models based on the maximum deviation from interpretable white-box models. The safety and interpretability of black-box models is an important topic, and many reviewers agree that the approach proposed by the authors is interesting. However, the maximum deviation from popular models such as decision trees, generalized linear and additive models have been intensively studied in the context of robust statistics/learning. Without explicit discussion on the connections with these existing studies, the novelty of the proposed approach cannot be properly evaluated. We thus have to conclude that the paper cannot be accepted in its current form.
Decision: Reject