Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new conceptual framework to assess the safety of model via maximum deviation from a reference model over a pre-specified certification dataset. The paper also motivates how such reference model along with certification dataset might be chosen depending on the problem. With this quantification of safety, the paper then demonstrates the computation of this deviation for interpretable models like trees and generalized additive models. They also discuss discrete optimization techniques to compute this deviation metric in case of tree ensembles which are not considered interpretable. Finally they discuss the case of a more general piecewise Lipschitz functions and show how interpretable models can lead to more efficient computation of their metric. 

Through cases studies, the paper demonstrates how this framework can be used to identify regions within feature space where the deviation is large. This can be further subjected to investigation, helping in improving safety of models. They also demonstrate the case where deviation bounds may be used to study the effect of number of estimators in the random forest on safety of the model as defined within their framework. 
strengths_and_weaknesses: Strengths:
1) The conceptual framework proposed in this paper is intuitive and helps reason about the safety of a predictive model. 
2) The paper demonstrates the computation of deviation or bounds on deviation for trees, GAMs and tree ensembles along with a broader class of piecewise Lipschitz functions. This helps in understanding the practicality of this framework under different situations
3) The case studies show how this framework can be useful not only in the assessment of regions within feature space that are problematic, but also in choosing hyperparameters such that the models are safe w.r.t this deviation metric

Weaknesses:
1) Choosing a reference model, certification dataset and the deviation function D(y, y_0) may not be a trivial exercise while assessing the safety of non-interpretable model. It is not straightforward to reason about the tightness and usefulness of bounds computed under these circumstances. 
2) In the case where the model is interpretable, we still need to make several assumptions (e.g. C is cartesian product while optimizing for non-linear additive f). While many of these assumptions made in the paper look reasonable, it may limit the applicability of the method to several practical scenarios involving interpretable models. 
questions: 1) It would be interesting to have a set of thumbrules/guidelines on the selection of deviation function D(y, y_0) under different scenarios. Have the authors generated any results on this?
2) I would like to know if the authors have tried to assess models like deep neural networks (since they are not intepretable according to the assumptions in the paper - will this reduce to worst case scenario when computing maximum deviation?). 


limitations: Yes, addressed in the appendix.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper defines a new "safety" measure for a ML model given a good reference model (normally an interpretable model). The safety metric is the maximum of outputs deviation between the target model and the reference model for some input sets. The authors then analyze the "safety" metric for some common ML models, including decision trees, generalized linear and additive models, tree ensembles, and piecewise Lipschitz functions. The authors then show how "safety" metric on two real world datasets.
strengths_and_weaknesses: Strengths:
- The idea of using a reference model to measure how safe another model is looks quite novel. 
- The authors describe the motivation and the actual idea well. Most parts of this paper are relatively easy to follow.

Weaknesses:
- The discussions on how the baseline model and the certification set are chosen are quite limited. It was not very convincing that the current form is what the idea of two model safety measure would be. See my questions in the section below.
- The experimental section is not easy to follow. There are too many discussions/figures or other information on the appendix. Readers have to go back and forth multiple times between the main body and appendix. 
questions: - An interpretable model may sound good, but it won't get 100% correctness. Often interpretable model will sacrifice performance for interpretability. What if the baseline model is wrong on some inputs but the target model is actually correct. In that case, how would the safety measure address this issue? 
- In practice, how do you choose the certification set? You may not have access to the test set in reality and domain experts' inputs may just be a few data samples with label. In that case, how would one define the certification set based on the limited inputs?
- This question might be less related. But I'm wondering, how would this safety metric be used by real-world ML applications? Do the users train a few models and choose the safest one by the metric regardless of the model's overview performance? If the certification set contains a few data samples, can the user just run a sanity check for each model and drop those models with unsafe outputs?
limitations: - I think this paper describes how to compute the safety score for some sets of common ML models. Extra work is required to generalize this metric to all ML models. 
- Depends on the certification set, it may not be practical to compute the safety score for some ML models.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to use maximum deviation, defined by the difference between a reference model and a trained model over a superset of the training data,  achieving safety and interpretability for supervised learning models.  Such a measure can be efficiently computed for decision trees, generalized linear and additive models. The idea of using maximum deviation to measure the model safety is novel.  However, the proposed approach is somewhat too restrictive to generalize. 
strengths_and_weaknesses: Strengths:
1. The topic is quite interesting and important. 
2. Rich experimental results.  
3. Well written and organized. 

questions: 1. How to specify a reference model f_0 in practice? Since  we do not have any prior on the relationship between input and output,  an improper specification on f_0 may result in misleading interpretable result. 

2. The theoretical part is not surprise to me. In particular, the authors argue that when f and f_0 are both decision trees, the complexity of computing maximum deviation cost  in O(\epsilon), which is actually stated in [1] for verifying the robustness of single tree. Besides, a pioneering work [2] proposed a verification of trees using anytime searching algorithm to address multiple verification tasks on ensemble trees. So the method described in section 4.3 is basically a generalization of [2]. 

3. Consider that modern machine learning models are usually with complex structure (e.g. deep neural networks) when dealing with big data. The proposed framework is too restrict to generalize. My concern lies with when the reference model f_0 is a complex DNN, is there an efficient way to compute the corresponding maximum deviation, especially when f is also a DNN. How about the corresponding computational complexity. 

4. It would be better to add some baseline comparisons to illustrate the safety of a specific model. 

[1] Hongge Chen, Huan Zhang, Si Si, Yang Li, Duane Boning, Cho-Jui Hsieh, robustness verification of tree-based models, NeurIPS, 2019. 
[2] L. Devos, W. Meert and J. Davis, Versatile verification of tree ensembles, ICML, 2021. 


limitations: Please refer to Questions
review_title: null

Meta-review: metareview: The authors propose to inspect learned models based on their maximum deviation to a reference model. They evaluate the feasibility of computing this deviation for a number of widely used model classes, including generalised linear models and decision trees. The idea is illustrated in cases studies. 

Reviewers all appreciated the novelty and importance of the proposed problem and the contributions made to examine the feasibility of solving it. Their main concerns were regarding the limited discussion on choosing the reference model and the certification set. The authors expanded greatly on this in their rebuttal and in a revision to the paper. 

The technical novelty is rather lower but the paper should not have trouble finding an audience in the NeurIPS community due to the broad applicability of the problem under consideration the clarity of the manuscript. I think the added discussion asked for by reviewers is well within what could be expected to be added for a camera-ready version (and indeed, this is already in the Appendix of the revision). 
confidence: Less certain
Decision: Accept