Learning Optimal Predictive Checklists

1 Introduction . Checklists are simple tools that are widely used to assist humans when carrying out important tasks or making important decisions [ 8 , 13 , 18 , 35 , 44 , 50 , 60 , 61 , 64 , 65 , 66 ] . These tools are often used as predictive models in modern healthcare applications . In such settings , a checklist is a set of Boolean conditions that predicts a condition of interest – e.g. , a list of symptoms flag a patient for a critical illness when M out N symptoms are checked . These kinds of “ predictive checklists ” are often used for clinical decision support because they are easy to use and easy to understand [ 33 , 54 ] . In contrast to other kinds of predictive models , clinicians can easily scrutinize a checklist , and make an informed decision as to whether they will adopt it . Once they have decided to use a checklist , they can integrate the model into their clinical workflow without extensive training or technology ( e.g. , as a printed sheet [ 52 ] ) . Considering these benefits , one of the key challenges in using predictive checklists in healthcare applications is finding a reliable way to create them [ 33 ] . Most predictive checklists in medicine are either hand-crafted by panels of experts [ 30 , 41 ] , or built by combining statistical techniques and heuristics [ e.g. , logistic regression , stepwise feature selection , and rounding 39 ] . These approaches * Equal supervision . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . make it difficult to develop checklists that are sufficiently accurate – as panel or pipeline will effectively need to specify a model that performs well under stringent assumptions on model form . Given the simplicity of the model class , it is entirely possible that some datasets may never admit a checklist that is sufficiently accurate to deploy in a clinical setting – as even checklists that are accurate at a population-level may perform poorly on a minority population [ 57 , 70 ] . In this paper , we introduce a machine learning method to learn checklists from data . Our method is designed to streamline the creation of predictive checklists in a way that overcomes specific challenges of model development in modern healthcare applications . Our method solves an integer programming problem to return the most accurate checklist that obeys user-specified constraints on model form and/or model performance . This approach is computationally challenging , but provides specific functionality that simplifies and streamlines model development . First , it learns the most accurate checklist by optimizing exact measures of model performance ( i.e. , accuracy rather than a convex surrogate measure ) . Second , it seeks to improve the performance of checklists by adaptively binarizing features at training time . Third , it allows practitioners to train checklists that obey custom requirements on model form or on prediction , by allowing them to encode these requirements as constraints in the optimization problem . Finally , it provides practitioners with an optimality gap , which informs them when a sufficiently accurate checklist does not exist . The main contributions of this paper are : 1 . We present a machine learning method to learn checklists from data . Our method allows practitioners to customize models to obey a wide range of real-world constraints . In addition , it pairs checklists with an optimality gap that can inform practitioners in model development . 2 . We develop specialized techniques that improve the ability of our approach to train a checklist that performs well , and to pair this model with a small optimally gap . One of these techniques can be used to train checklists heuristically . 3 . We conduct a broad empirical study of predictive checklists on clinical classification datasets [ 24 , 32 , 37 , 51 , 58 ] . Our results show that predictive checklists can perform as well as state-of-the-art classifiers on some datasets , and that our method can provide practitioners with an optimality gap to flag when this is not the case . We highlight the ability of our method to handle real-world requirements through applications where we enforce group fairness constraints on a mortality prediction task , and where we build a short-form checklist to screen for PTSD . 4 . We provide a Python package to train and customize predictive checklists with open-source and commercial solvers , including CBC [ 29 ] and CPLEX [ 20 ] ( see https : //github.com/ MLforHealth/predictive_checklists ) . 2 Related Work . The use of checklist-style classifiers – e.g. , classifiers that assign coefficients of ±1 to Boolean conditions – dates to the work of Burgess [ 9 ] . The practice is often referred to as “ unit weighting , '' and is motivated by observations that it often performs surprisingly well [ see e.g. , 7 , 19 , 23 , 27 ] . Our work is strongly related to methods that learn sparse linear classifiers with small integer coefficients [ see e.g. , 4 , 11 , 31 , 38 , 68 , 69 , 75 ] . We learn checklists from data by solving an integer program . Our problem can be viewed as a special case of an IP proposed by Ustun and Rudin [ 68 ] , Zeng et al . [ 77 ] to learn sparse linear classifiers with small integer coefficients . The problem that we consider is considerably easier to solve from a computational standpoint because it restricts coefficients to binary values and does not include ` 0-regularization . Here , we make use of this “ slack '' in computation to add functionality that improves the performance of checklists , namely : ( 1 ) constraints to binarize continuous and categorical features into items that can be included in a checklist [ see also 10 , 12 , 71 ] ; and ( 2 ) develop specialized techniques to reduce computation during the learning process . Checklists are M -of-N rules — i.e. , classifiers that predict y = +1 when M out of N conditions are true . Early work in machine learning used M -of-N rules in an auxiliary capacity – e.g. , to explain the predictions of neural networks [ 67 ] , or to use as features in decision trees [ 78 ] . More recent work has focused on learning M -of-N rules for standalone prediction – e.g . Chevaleyre et al . [ 16 ] describe a method to learn M -of-N rules by rounding the coefficients of linear SVMs to { −1 , 0 , 1 } using a randomized rounding procedure [ see also 79 ] . Methods to learn disjunctions and conjunctions are also relevant since they correspond to 1-of-N rules and N -of-N rules , respectively . These methods learn models by solving a special case of the ERM problem in ( 1 ) where M = 1 or M = N . Given that this problem is NP-hard , existing methods often reduce computation by using algorithms that return approximate solutions – e.g. , simulated annealing [ 72 ] , set cover [ 48 , 49 ] , or ERM with a convex surrogate loss [ 21 , 22 , 46 , 47 ] . These approaches could be used to learn checklists by solving the ERM problem in ( 1 ) . However , they would not be able to handle the discrete constraints needed for binarization and customization . More generally , they would not be guaranteed to recover the most accurate checklists for a given dataset , nor pair models with a certificate of optimality that can be used to inform model development . 3 Methodology . We start with a dataset of n examples ( xi , yi ) ni=1 where xi = [ xi1 , . . . , xid ] ∈ { 0 , 1 } d is a vector of d binary variables and yi ∈ { 0 , 1 } is a label . Here , xi are Boolean variables that could be used as items in a checklist . We denote the indices of positive and negative examples as I+ = { i | yi = +1 } and I− = { i | yi = −1 } respectively , and let n+ = |I+| and n− = |I−| . We denote the set of positive integers up to k as [ k ] = { 1 , . . . , k } . We use the dataset to learn a predictive checklist – i.e. , a Boolean threshold rule that predicts ŷ = +1 when at least M of N items are checked . We represent a checklist as a linear classifier with the form : ŷi = sign ( λ > xi ≥M ) . Here λ = [ λ1 , . . . , λd ] ∈ { 0 , 1 } d is a coefficient vector and λj = 1 iff the checklist contains item j . We denote the number of items in a checklist with coefficients λ as N = ∑d j=1 λj , and denote the threshold number of items that must be checked to assign a positive prediction as M . We learn checklists from data by solving an empirical risk minimization problem with the form : min λ , M l ( λ , M ) + NN + MM s.t . N = ‖λ‖ M ∈ [ N ] λ ∈ { 0 , 1 } d ( 1 ) Here , l ( λ , M ) = ∑n i=1 1 [ yi 6= ŷi ] counts the number of mistakes of a checklist with parameters λ and M . The parameters N > 0 and M > 0 are small penalties used to specify lexicographic preferences . We set N < 1nd and M < N d . These choices ensure that optimization will return the most accurate checklist , breaking ties between checklists that are equally accurate to favor smaller N , and breaking ties between checklists that are equally accurate and sparse to favor smaller M . Smaller values of N and M are preferable , as checklists with smaller N users check fewer items , and checklists with smaller M let users stop checking as soon as M items are checked ( see Figure 3 ) . We recover a globally optimal solution to ( 1 ) by solving the integer program ( IP ) : min λ , z , M l+ +W−l− + NN + MM s.t . Bizi≥M − d∑ j=1 λjxi , j i ∈ I+ ( 2a ) Bizi≥ d∑ j=1 λjxi , j −M + 1 i ∈ I− ( 2b ) l+ = ∑ i∈I+ zi l−= ∑ i∈I− zi N = d∑ j=1 λj M ∈ [ N ] zi ∈ { 0 , 1 } i ∈ [ n ] λj ∈ { 0 , 1 } j ∈ [ d ] Here , l+ and l− are variables that count the number of mistakes on positive and negative examples . The values of l+ and l− are computed using the mistake indicators zi = 1 [ ŷi 6= yi ] , which are set to 1 through “ Big-M '' Constraints in ( 2a ) and ( 2a ) .These constraints depend on the “ Big-M '' parameters Bi , which can be set to its tightest possible value maxλM − λ > xi . W− is a user-defined parameter that reflects the relative cost of misclassifying a negative example . By default , we set W− = 1 , so that the objective minimizes training error . Customization The IP in ( 2 ) can be customized to fit checklists that obey a wide range of realworld requirements on performance and model form . We list examples of constraints in Table 1 . Our method provides two ways of handling classification problems with different costs of False Negatives ( FNs ) and False Positives ( FPs ) . First , the user can specify the relative misclassification costs for FPs and FNs in the objective function by setting W− . Here , optimizing ( 2 ) corresponds directly to minimizing the weighted error . Second , the user can instead specify limits on FPR and FNR - e.g . minimize the training FPR ( by setting W− = n− ) subject to training FNR ≤ 20 % . Adaptive Binarization Practitioners often work with datasets that contain real-valued and categorical features . In such settings , rule-learning methods often require practitioners to binarize such features before training . Our approach allows practitioners to binarize features into items during training . This allows all items to be binarized in a way that maximizes accuracy . This approach requires practitioners to first define Tj candidate items for each non-binary feature ui , j . For example , given a real-valued feature ui , j ∈ R , the set of candidate items could take the form { xi , j,1 , xi , j,2 , . . . , xi , j , Tj } where xi , j , t = 1 [ ui , j ≥ vj , t ] and vj , t is a threshold . We would add the following constraint to IP ( 2 ) to ensure that the checklist only uses one of the Tj items : ∑ t∈ [ Tj ] λtj ≤ 1 . In this way , the IP would then choose a binary item that is aligned with the goal of maximizing accuracy . In practice , the set of candidate items can be produced automatically [ e.g. , using binning or information gain 53 ] or specified on the basis of clinical standards ( as real-valued features like blood pressure and BMI have established thresholds ) . This approach is general enough to allow practitioners to optimize over all possible thresholds , though this may lead to overfitting . Our experiments in Section 5.2 show that adaptive binarization produces a meaningful improvement in performance for our model class . Optimality Gap We solve IP ( 2 ) with a MIP solver , which finds a globally optimal solution through exhaustive search algorithms like branch-and-bound [ 73 ] . Given an instance of IP ( 2 ) and a time limit for when to stop the search , a solver returns : ( i ) the best feasible checklist found within the time limit – i.e. , values of λ and M that achieve the best objective value V max = l ( λ , M ) ; and ( ii ) a lower bound on the objective value – i.e. , V min , the minimal objective value for any feasible solution . These quantities are used to compute the optimality gap ε : = 1− V min V max . When the upper bound V max matches the lower bound V min , the solver returns a checklist with an optimality gap of ε = 0 – i.e. , one that is certifiably optimal . In our setting , the optimality gap reflects the worst-case difference in training error between the checklist that a solver returns and the most accurate checklist that we could hope to obtain by solving IP ( 2 ) . Given a solution to IP ( 2 ) with an objective value of L and an optimality gap of ε , any feasible checklist has a training error of at least d ( 1− ε ) Le . When ε is small , the most accurate checklist has a training error ≈ L. Thus , if we are not satisfied with the performance of our model , we know that no checklist can perform better than L , and can make an informed decision to relax constraints , fit a classifier from a more complex hypothesis class , or not fit a classifier at all . When ε is large , our problem may admit a checklist with training error far smaller than L. If we are not satisfied with the performance of the checklist , we can not determine whether this is because the dataset does not admit a sufficiently accurate model or because the solver has not been able to find it .

Summary: This work presents an extensive evaluation of checklist models (linear models with binary features and coefficients) for clinical predictions.  Such models are quite transparent relative to more complex families, and can be potentially be extremely simple to implement in clinic.  The learning problem is framed as mixed integer programming, which has other virtues - solvers can provide bounds on how far our current solution is to the optimal solution wrt training set performance, and the inclusion of desirable properties as constraints.  This work also provides an evaluation of binarization methods for continuous variables (recall that features must be binary), and proposes a pair of methods for improving performance of the learned models.  Extensive evaluations are performed on 8 datasets; the checklist models very often achieve quite good performance on par with more complex model families.