Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed a new method to learn predictive clinical checklists for classification tasks by solving an integer programming problem.The method was validated and evaluated with 8 benchmark clinical datasets with consideration of ML fairness. 
main_review: The paper proposed an interesting method to learn a short-form checklists from clinical datasets with the potential application of clinical diagnosis, treatment decision making, etc. The paper showed clear problem constructions, originality of the work with comparison to related literatures and decent considerations on model fairness as well as social impact.
Technical analysis might require more experiments to improve the soundness and confirm the utilities of the proposed method. Detailed questions/suggestions are as follows:
1. As indicated in Authors' checklist, no confidence intervals are reported for experiment results. Considering the oversampling on minority class, boostrapping on training data might be necessary to obtain confidence intervals to evaluate performance robustly. Cross validation could also be applied to test generalization ability of different models. 
2. The paper chose accuracy as the main performance comparison metric across different datasets, which could be optimized considering the real-world scenarios. However, depending on the nature of the task, the cost of false positive (FP) and the cost of false negative (FN) prediction might vary a lot. e.g. For PTSD diagnosis, missing a single positive case might lead to severe outcome in delayed treatment, FNR at a fixed TPR might be a better evaluation metric. But for the mortality task, TPR given a fixed FPR is a more important metric since we need to be very cautious in decision-making on withdrawal of life supporting treatment when a ML classifier makes prediction of "positive" in mortality. 
3. A fundamental question: For each of the classification task, feature selection and feature importance could be easily obtained, which could be used as a potential checklist as well. What will be the advantages of the proposed method with comparison to get checklists from feature selection?  
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work presents an extensive evaluation of checklist models (linear models with binary features and coefficients) for clinical predictions.  Such models are quite transparent relative to more complex families, and can be potentially be extremely simple to implement in clinic.  The learning problem is framed as mixed integer programming, which has other virtues - solvers can provide bounds on how far our current solution is to the optimal solution wrt training set performance, and the inclusion of desirable properties as constraints.  This work also provides an evaluation of binarization methods for continuous variables (recall that features must be binary), and proposes a pair of methods for improving performance of the learned models.  Extensive evaluations are performed on 8 datasets; the checklist models very often achieve quite good performance on par with more complex model families.  
main_review: This work presents a study of a very restricted class of linear models - binary features and coefficients - for clinical prediction tasks.  Such models are basically checklists - you evaluate some predicates and add up the true values and see if the sum >= some threshold.  Such models have many virtues - the chief ones being that they are potentially very simple to deploy - you can print them out, and clinicians can run them with minimal arithmetic (though evaluating the predicates can still be non-trivial). They are also transparent in the sense of how they arrive at scores (it should be noted that this is not the only important sense of transparent - it does not, as the authors point out, imply causal mechanisms - but lack of this sort of transparency can sometimes be a barrier to adoption).  The formulation finding optimal checklists as a mixed integer program admits the use of off the shelf solvers, and the encoding of some desirable properties (e.g., keeping the number of predicates small, bounding the difference in performance between subgroups small) as constraints on the optimization.  It can also provide bounds on the gap between the current solution and the best possible solution with respect to training set error.  The authors also propose two methods that are claimed to enable better solutions (with respect to time required and quality).  Evaluations are performed on 8 prediction tasks, each using a different dataset, and the checklist models are for the most part on par with some more complex model classes suggesting plenty of room for trading off marginal gains in performance vs ease of use and deployment, etc.  

Overall, I liked this work a lot and _really_ appreciated the clarity of the writing.  I am not aware of previous uses of MIP for finding optimal checklist models for clinical prediction, though of course there is plenty of prior work (NB - this work was cited appropriately by the current work) on linear models with integer coefficients, etc.  Although this work does not present much very new in the way of core methods - it does provide a useful evaluation of a far simpler model class, and the benefits admitted by framing it as an MIP; I think it would be useful for the NeuRIPS community, particularly the subset focused on clinical prediction tasks, to see this work.  There are, however, some ways in which the work could be improved.  

First, I strongly recommend doing an ablation of the sub-modularity heuristic and the path algorithm presented in sections 4.1 and 4.2.  The authors demonstrate the benefit of the adaptive binarization scheme (this scheme is not novel but does seem to have a quite substantial effect on bringing the performance of the checklist models on par with the other baselines!) but not these two components of their method. 

Second, in order for the lasso and xgb baselines to be trusted as reasonable lower bounds on error, it is important to provide some detail on hyperparameter tuning.  It doesn't have to be a lot, and can reasonably be relegated to supplementary materials as long as readers are directed there in the main manuscript.  

Third, I recommend being a bit more circumspect in the language describing the utility of the optimality gaps.  These are bounds regarding the training set behavior, not generalization error.  The authors argue that the extremely simple model class prevents large divergences between training and generalization performance most of the time, and this is borne out for the most part in the evaluations.  Perhaps the authors could emphasize how precisely the gap would be used in practice (i.e., as they implied, to help guide the developer in, say, relaxing constraints on number of predicates, etc, to achieve better performance)?

In summary, I think this is a very clearly written, well organized paper presenting a useful perspective on how to build clinically useful diagnostic or risk prediction models. There are some gaps in the evaluation I would like to see filled.  



limitations_and_societal_impact: The authors are commendably straight forward about some of the technical limitations of their work but it could also be pointed out that the path to clinical utility is a bit more complicated than printing out a checklist and posting it on exam room walls - effective adoption does not end at deployment.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Method to train classifiers based on integer programing optimization of binarized variables.
Evaluate and multiple public healthcare related datasets, comparing accuracy vs. other baseline methods.
Offers some optimality gap guarantees of models chosen within time constraints (vs. exhaustive search).

main_review: Seems to be confusing the term "checklist" here. What is described are more analogous to clinical risk scores (see MDCalc.com for implemented example).
This is quite different than the checklists "widely used to promote safety and reliability" eluded to in the Intro such as surgical pre-op checklists. Most of those are about consistently executing a process and have nothing to do with prediction, risk assessment, diagnosis, or calculation.

The construction and evaluation of the test models developed overall seem fair, but as eluded to in the Intro, these seem to function very similarly to many existing methods for sparse linear classifiers with small integer coefficients. Neither based on performance or interpretability did there seem to be a dramatic advantage to the proposed approach.
The "certifiably optimal result" was an interesting contribution to define an optimality gap between the current best performing model vs. best conceivably possible given the available data variables.

Not clear there's much value in reporting train set performance throughout manuscript. Is mostly distracting and even confusing as I first thought the XGBoost models were far outperforming the other models, but that's just because they were overfit to the training data. Potential generalizable accuracy on the test sets is much more relevant.

limitations_and_societal_impact: As above, no major concerns beyond existing tools and technology, but also not clear a substantial difference/impact beyond existing ones.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposed a formulation and learning algorithms for learning diagnostic checkslists.  A checklist is a set of binary items where the prediction is positive if the number of checked items exceeds a threshold.  

The paper proposes two techniques to solve the resultant integer program optimization problem.  The first is a submodular heuristic
The second is a sequential algorithm that reduces the size of the branch and bound tree by eliminating checklists that would be subsets of checklists that do not perform well. 

The submission includes evaluation on multiple datasets and under different binarization schemes. For one dataset, there is an evaluation of group-wise bias in the predictions based on the learned lists. There is also a more detailed case study for PTSD that demonstrates how to translate a realistic goal into the setup needed for the learning problem. 
main_review: # Originality
This work provides a formulation for a task that has not been adopted in machine learning, but is highly relevant to healthcare. 

# Quality
This work is thorough and well described in the submission. The experiments support the technique's performance boradly and suggest where future improvements may lie.  The case study on PTSD works very well to illustrate how to take a variety of real-world goals and formulate a checklist search problem. 



# Clarity
This paper is overall understandable.  There are minor clarity issues that make some sections require multiple attempts or later context to understand, but they do not interfere with understanding the main conclusions or methods. 

The abstract doesn't stand well on its own.  After reading the paper it reads like a good summary, but before reading, too many parts read as undefined terms. 

Section 4.2 is a little hard to parse. Parts of it read as if it's the introduction to a forthcoming more detailed description of the algorithm, but there is no longer description section. 

The fairness evaluation is less strong than the rest of the paper and could be improved some. It's not clear how "a false negative is worse than a false positive" (line249) supports minimizing FPR and holding FNR as a constraint and evaluating the FPR gap. This may be a clarity issue, so please explain that connection more clearly, otherwise, your setup seems to suggest that the FNR should be monitored more closely. If false negatives are worse than false positives, it would seem to follow that a gap in the FNR would also be worse- or does setting it as a constraint also make the FNR equal across groups?



# Significance
This work represents a highly interpretable model that presents the predictive task in a similar form to what clinicians already use.  Checklists are widely adopted and trusted in medicine for many reasons and learning the from data presents an opportunity to improve real practice. The performance as is with the current adaptive binarization is comparable to more complex learning algorithms. 


# Minor Issues 


- line 22, is it that there are no standardized *automated* approaches or truly no standards in how checklists are developed? expert panels often have strict protocols that they use that do serve as a type of of standardization, though it's less strict than computerized standardization
- line 47 uses PCL without it being defined yet.
- line 71: "reduce computation" here is not clear, a more specific phrase would help. Because that part is talking about interpretable models, computation could refer to either at the time of learning or prediction. 
- Table 1 is difficult to read small
- section heading 4.2, should that be plural or singular?
- line 171: "The algorithms exploits" -> "The algorithm exploits"
- line 188: "... by using one.." to "...by using each..." as  it reads, I was looking for how you chose which one to use until I saw the results
- the table 3 caption should state that the "lower bound" columns are a baseline and the way they're referred to in the table should match the text (line 212)
- on line 225 it says the checklists performs similarly to more complex classifiers, this could use a citation, 
- line 279, define ED
limitations_and_societal_impact: The authors address the social impact of their work very well, throughout the paper.  The motivation in the introduction makes it clear.  The authors evaluate for bias in one dataset and show how their framework allows for a context-specific mitigation to reduce the most important gaps in performance. The conclusion includes specific cautions for the interpretation and application of their work both with respect to fairness and its role as an interpretable model.
review_title: null

Meta-review: This work proposes an IP-based solution to learning predictive clinical checklists. Overall, the paper was well received, but reviewers make a number of recommendations to help improve the paper that should be incorporated prior to presentation:
-including error bars/confidence intervals
-including an ablation of the sub-modularity heuristic and the path algorithm
-compare to the strongest baselines possible (thorough baseline hyperparameter turning)
-framing of fairness results
I encourage authors to carefully consider reviewer feedback when working on their revision.

Decision: Accept (Poster)