FasterRisk: Fast and Accurate Interpretable Risk Scores
Authors: Jiachang Liu
Chudi Zhong
Boxuan Li
Margo Seltzer
Cynthia Rudin
Venue: NeurIPS 2022 Accept (2022)
Keywords: interpretability
scoring system
risk scores
rashomon set

Abstract: Over the last century, risk scores have been the most popular form of predictive model used in healthcare and criminal justice. Risk scores are sparse linear models with integer coefficients; often these models can be memorized or placed on an index card. Typically, risk scores have been created either without data or by rounding logistic regression coefficients, but these methods do not reliably produce high-quality risk scores. Recent work used mathematical programming, which is computationally slow. We introduce an approach for efficiently producing a collection of high-quality risk scores learned from data. Specifically, our approach  produces a pool of almost-optimal sparse continuous solutions, each with a different support set, using a beam-search algorithm. Each of these continuous solutions is transformed into a separate risk score through a "star ray" search, where a range of multipliers are considered before rounding the coefficients sequentially to maintain low logistic loss. Our algorithm returns all of these high-quality risk scores for the user to consider. This method completes within minutes and can be valuable in a broad variety of applications. 

FasterRisk: Fast and Accurate Interpretable Risk Scores
Jiachang Liu
Chudi Zhong' Boxuan Li'
Margo Seltzer2 Cynthia Rudin'
Duke Univeristy University of British Columbia (jiachang.live chudi zhong, boxuan @duke edu edu mseltzerecs. ubc. cynthia@cs duke. edu
Abstract
Over the last century, risk scores have been the most popular form of predictive model used healthcare and criminal justice. Risk scores are sparse linear models with integer coefficients; often these models be memorized placed index card. Typically, risk scores have been created either without data or by rounding logistic regression coefficients, but these methods do reliably produce high-quality risk scores. Recent work used mathematical programming, which is computationally slow. We introduce an approach for efficiently producing a collection of high sugh-tuality risk scores learned from data. Specifically, our approach produces a pool of almost-optimal sparse continuous solutions, each with different support set, using beam- beam-search algorithm. Each of these continuous solutions is transformed into separate risk score through "star ray" search, where range multipliers are considered before rounding the coefficients sequentially to maintain low logistic loss. Our algorithm returns of these high-quality risk scores for the user consider. This method completes within minutes and can be valuable in broad variety of applications.
Introduction
Risk scores are sparse linear models with integer coefficients that predict risks. They are arguably
the most popular form of predictive model for high stakes decisions through the last century and
are the standard form of model used in criminal justice
4 22
and medicine [9
Their history dates back to at least the
1. Oval Shape
points
criminal justice work of Burgess [ where, based on their criminal history
2. Irregular Shape
4 points
Cireumscribed Margin
points
and demographics, individuals were assigned integer point scores between and 21 that determined the probabil- ity of their "making good of fail- ing upon parole." Other famous risk scores arguably the most widely- used predictive models healthcare. These include the APGAR score [3].
4. Spiculated Margin
2 points
Age 2 60
3 points
SCORE
SCORE
-3
RISK
6.0%
10.6%
13.8%
17.9%
22.8%
28.6%
SCORE
1
3
2: 5
RISK
35.2%
42.4%
50.0%
57.6%
64.8%
71.4%
developed in 1952 and given to new- borns, and the CHADS, score [18]. which estimates stroke risk for atrial fibrillation patients. Figures [ and show example risk scores, which es-
Figure 1: Risk score on the mammo dataset 13], whose pop- ulation is biopsy patients. It predicts the risk of malignancy of a breast lesion. Risk score is from FasterRisk on fold of 5-CV split. The AUCs the training and sets 0.854 and 0.853, respectively.
"Thsee authors contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

timate risk of a breast lesion being malignant.
1. Irregular Shape
4 points
2. Cireumscribed Margin
-5 points
Risk scores have the benefit of being easily memorized; usually their names reveal the full model- for instance, the factors in cHADS² are past Chronic heart failure, Hypertension, Age275 years, Diabetes, and past Stroke (where past stroke receives 2 points and the others each receive 1 point). For risk scores, counterfactu- als are often trivial to compute, even without a calculator. Also, checking that the data and calculations are cor- rect is easier with risk scores than with other approaches. In short, risk scores have been created by humans for century to support a huge spectrum of applications [2]
3.
SpiculatedMargin
2 points
4.
Age 2 45
point
5. Age 2 60
3 points
SCORE
SCORE
-5
-4
-3
-2
-1 21.9%
RISK
7.3%
9.7%
12.9%
16.9%
27.8%
SCORE
2
3
5
34.6%
42.1%
50.0%
57.9%
65.4%
72.2%
Figure 2: A second risk score on the mammo dataset on the same fold as in Figure [ The AUCs on the training and test sets are 0.855 and 0.859, respectively. FasterRisk can produce a diverse pool of high-quality models. Users can choose a model that best fits with their domain knowledge.
43 44 47
because humans find them easy to understand.
Traditionally, risk scores have been created in two main ways: (1) without data, with expert knowledge only (and validated only afterwards on data), and (2) using a semi-manull process involving manual feature selection and rounding of logistic regression coefficients. That is, these approaches rely heavily on domain expertise and rely little on data. Unfortunately, the alternative of building a model directly from data leads to computationally hard problems: optimizing risk scores over a global objective on data is NP-hard, because in order to produce integer-valued scores, the feasible region must be the integer lattice. There have been only a few approaches to design risk scores automatically
40,, but each of these has flaw that limits its use in practice: the optimization basedd approaches use mathematical programming solvers (which require license) that slow and scale poorly, and the other methods are randomized greedy algorithms, producing fast but much lower-quality solutions. We need an approach that exhibits the best of both worlds: speed fast enough to operate in few minutes laptop and optimization/search capability that of the mathematical programming tools. Our method, FasterRisk, lies at this intersection. is fast enough to enable interactive model design and rapidly produce large pool of models from which users can choose rather than producing only a single model.
5
16
2
33
38
39
powerful as
One may wonder why simple rounding of f--regularized logistic regression coefficients does not yield sufficiently good risk scores. Past works 37 39 explain this as follows: the sheer amount of & regularization needed to get very sparse solution leads to large biases and worse loss values, and rounding goes against the performance gradient. For example, consider the following coefficients from & regularization: [1.45, .87, .83, .47, .23, .15, ... J. This model is worse than its unregularized counterpart due to the bias induced by the large ( term. Its rounded solution is [1,1,,,0,0,0,..,, which leads to even worse loss. Instead, one could multiply all the coefficients by constant and then round, but which constant is best? There are an infinite number of choices. Even if some value of the multiplier leads to minimal loss due to rounding, the bias from the &, term still limits the quality of the solution.
The algorithm presented here does not have these disadvantages. The steps are: (1) Fast subset search with lo optimization (avoiding the bias from f1). This requires the solution of an NP-hard problem, but our fast subset selection algorithm is able to solve this quickly. We proceed from this accurate sparse continuous solution, preserving both sparseness and accuracy in the next steps. (2) Find a pool of diverse continuous sparse solutions that almost as good as the solution found in (1) but with different support sets. (3) A "star ray" search, where we search for feasible integer-valued solutions along multipliers of each item in the pool from (2). By using multipliers, the search space resembles the rays of a star, because it extends each coefficient in the pool outward from the origin to search for solutions. To find integer solutions, we perform a local search (a form of sequential rounding). This method yields high performance solutions: we provide a theoretical upper bound on the loss difference between the continuous sparse solution and the rounded integer sparse solution.
2

---

Through extensive experiments, we show that our proposed method is computationally fast and produces high-quality integer solutions. This work thus provides valuable and novel tools create risk scores for professionals in many different fields, such as healthcare, finance, and criminal justice.
Contributions: Our contributions include the three-step framework for producing risk scores, beam-search-bere based algorithm for logistic regression with bounded coefficients (for Step 1), the search algorithm to find pools of diverse high-quality continuous solutions (for Step 2), the star ray search technique using multipliers (Step 3), and a theorem guaranteeing the quality of the star ray search.
2 Related Work
Optinization-buta based approaches: Risk scores, which model P(y = 1/2;), are different from threshold classifiers, which predict either y = or y = -1 given æ. Most work in the area of optimization of integer-valued sparse linear models focuses on classifiers, not risk scores [5] 6 y 32
6%5
57
40 46
This difference is important, because a classifier generally cannot be calibrated well for use risk scoring: only its single decision point is optimized. Despite this, several works use the hinge loss algorithms use mathematical programming solvers (i.e., integer programming solvers), which tend to be slow and cannot be used on larger problems. However, they can handle both feature selection and integer constraints.
to calibrate predictions 6 | All of these optimization-
To directly optimize risk scores, typically the logistic loss is used. The RiskSLIM algorithm 139 optimizes the logistic loss regularized with lg regularization, subject to integer constraints on the coefficients. RiskSLIM uses callbacks to MIP solver, alternating between solving linear programs and using branch-and-cre to divide and reduce the search space. The branch-and-cut procedure needs to keep track of unsolved nodes, whose number increases exponentially with the size of the feature space. Thus, RiskSLIM's major challenge scalability.
Local search- baarch-based approaches: As discussed carlier, natural way to produce a scoring system or risk score by selecting features manually and rounding logistic regression coefficients or hinge-loss solutions to integers [T0 39]. While rounding is fast, rounding errors can cause the solution quality to be much worse than that of the optimization-basse
approaches. Several works have proposed
improvements over traditional rounding. In Randomized Rounding (IO). each coefficient is rounded up or down randomly, based on its continuous coefficient value. However, randomized rounding does not seem to perform well in practice. Chevaleyre [10] also proposed Greedy Rounding, where coefficients are rounded sequentially. While this technique aimed to provide theoretical guarantees for the hinge loss, we identified a serious flaw in the argument, rendering the bounds incorrect (see Appendix B The RiskSLIM paper 339 proposed SequentialRounding, which, at each iteration, chooses coefficient to round up down, making the best choice according to the regularized logistic loss. This gives better solutions than other types of rounding, because the coefficients are considered together through their performance on the loss function, not independently.
drawback of SequentialRounding is that it considers rounding up or down only to the nearest integer from the continuous solution. By considering multipliers, we consider a much larger space of possible solutions. The idea of multipliers (i.e., "scale and round") is used for medical scoring systems [II] though, as far as we know, it has been used only with traditional rounding rather than SequentialRounding which could easily lead to poor performance, and we have seen no previous work that studies how to perform scale-and-round in systematic, computationally efficient way. While the general idea of scalc-and-roumt seems simple, it is not: there are an infinite number of possible multipliers, and, for each one, number of possible nearby integer coefficient vectors that is the size of a hypercube, expanding exponentially in the search space.
Sampling Methods: The Bayesian method of Ertekin et al. [6 samples scoring systems, favoring those that are simpler and more accurate, according to a prior. "Pooling" 69 creates multiple models through sampling along the regularization path of ElasticNet. As discussed, when regularization is tuned high enough to induce sparse solutions, it results in substantial bias and low-quality solutions (see
77 39 for numerous experiments on this point). Note that there is a literature on finding diverse solutions to mixed-integer optimization problems [e.g.. I. but it focuses only on linear objective functions.
3

---

Algorithm 1 FaaceERssK(..C............ + {(wt(, w .e))
Input: dataset D (consisting of feature matrix X TR= and labels y IR"), sparsity constraint
coefficient constraint C C== 5, beam search size B = 10, tolerance level e = 0.3, number of attempts
T = 50, number of multipliers to try Nn 20. Output: a pool P of scoring systems {(w², (o6).m") where t is the index enumerating all found scoring systems with | lo - k and 1lo0 S C. and m² is the corresponding multiplier.
1: Call Algorithm 2 SparseBeaml.R(22 k, C, to find high-quality solution (w", wH) to the sparse logistic regression problem with continuous coefficients satisfying a box constraint, (Algorithm SparseBeaml.K will call Algorithm ExpandSuppByl as subroutine, which grows the solution by beam search.)
i.e., solve Problem O
2: Call Algorithm CoaeiiimmmmSCooSSSStSt S DiverssoDivesc (w", wi), e, T). which solves Problem 4 Place its output (w" }e in pool / = {w", w. P 4- PU [(w", w/))..
3: Send each member in the pool P. which is (w",,()) to Algorithm 3 StarRaySearch
(D, (w', wy), (m) to perform line search among possible multiplier values and obtain an integer solution (w++, " with multiplier my, Algorithm calls Algorithm Auxiliary-
LossRounding which conducts the rounding step.
4: Return the collection of risk scores {ww*,, Wo .m)).. If desired, return only the best model according to the logistic loss.
3 Methodology
Define dataset D = (1,2, v. (1 is static feature corresponding to the intercept) and scaled dataset 1 D | ai W/ In J=z1 for real-valued m. Our goal is to produce high-quality risk scores within a few minutes on small personal computer. We start with an optimization problem
similar to RiskSLIM's (39]. which minimizes the logistic loss subject to sparsity constraints and integer coefficients:
min L(w, Wo, D), where L(w, Wp. D) -E log(1 + exp( (2 w + wo))) W. w.
(1)
such that ||w/|o < k and w E Z2, Vj E [I....) wy E (-5,5,, wo E Z.
In practice, the range of these box constraints [-5, 5] is user-defined and can be different for each coefficient. (We use for case of exposition.) The sparsity constraint || S & or integer constraints E 7P make the problem NP-hard, and this is difficult mixed-integer nonlinear program. Transforming the original features to all possible dummy variables, which is a standard type of preprocessing [e.g.. 24I. changes the model into (flexible) generalized additive model; such models can be as accurate as the best machine learning models 39 42 Thus, we generally process variables in æ to be binary.
To make the solution space substantially larger than [-5, -4, ..., 4,5), we use multipliers. The problem becomes:
1 m D)) where L w. Wo, ( -D)==X Σ log 1 + exp (-u: x w + wo mm )) (2)
min L (ww, wa. 11
i=1
such that ||wl|o S k. w EP, Vj [L.... pl, wy E [-5,5], wi E Σ, m > 0.
Note that the use of multipliers does not weaken the interpretability of the risk score: the user still sees integer risk scores composed of values wy E (-5, -4, .., 4, 5). wo E Σ. Only the risk conversion table calculated differently, as P(Y = 1/x) 1/(1 + -(())) where f(x) = |/4 (w + wo).
Our method proceeds in three steps, as outlined in Algorithm D In the first step, it approximately solves the following sparse logistic regression problem with box constraint (but not integer constraints), detailed in Section 3.1 and Algorithm 2
(w", w E argmin L(w, wo. D), ||l]|o < k, w E RP, Vje [L,..., p). wy E 1-5,5], wo E R. 8e,, 100g
The algorithm gives accurate and sparse real-valued solution (w* w())
The second step produces many near-optimal sparse logistic regression solutions, again without integer constraints, detailed in Section 3.2 and Algorithm 5 Algorithm 5 uses (w", wy) from the

---

first step to find a set {(w", wilte h such that for all t and given threshold Ew=
(w=, ve)) obeys L(w", wh, D) S L(w", w6: D) x (1 (w)) ||2" lo S k w' E ER, Vj [1....pp. pl. w/ |-5, 5], wp R.
(4)
After these steps, we have pool of almost-optimal sparse logistic regression models. In the third step, for each coefficient vector in the pool, we compute risk score. It is feasible integer solution (witf, wn to the following, which includes positive multiplier m > 0:
1 D) S L(w", w6 D) + E22 w w m:D) L
wt ZP, Vj EjE [L...., pl, with E 1 5]. w0 Z.
where we derive tight theoretical upper bound on et. A detailed solution to (5) shown in Algorithm in Appendix A We solve the optimization problem for large range of multipliers Algorithm for each coefficient vector in the pool, choosing the best multiplier for each coefficient vector. This third step yields a large collection of risk scores, all of which are approximately as accurate as the best sparse logistic regression model that can be obtained. All steps in this process are fast and scalable.
Algorithm 2
SparseBBamm..KD......B
(w, wo)
Input: dataset D, sparsity constraint k. coefficient constraint C, and beam search size Output: sparse continuous coefficient vector (w, wo) with ||w]lo llo S k; || S C.
I: Define N+ and N_ numbers of positive and negative labels, respectively.
Initialize the intercept and coefficients. >lnitialize the collection of found supports as empty set 4: (W,F)+ + ExpandSuppBy1(1. (w, Wo),
2: We + Iog((o++NN]] w
3: F 0
>Returns S B models of support Beaam search expand the support
5: for to........ do
6: Wimp <-
7:
for (w'' (w',wh) W do
DEach of these has support 1 >Returns S. B models with supp. t.
8:
(W Ex (w', F,, B).
9: Wimp t Wmm UW
10: end for
Reset W be the B solutions in Wump with the smallest logistic loss values.
12: end for
Pick (w, wo) from W with the smallest logistic loss.
14:
Return (w, Wo).
3.1 High-quality Sparse Continuous Solution
There are many different approaches for sparse logistic regression, including & regularization
h5
ElasticNet 48), regularization [ 24] and orthogonal matching pursuit (OMP) [4 25). but none of these approaches seem to be able to handle both the box constraints and the sparsity constraint in Problem S so we developed a new approach. This approach, in Algorithm 2 SparseBeaml RR, uses beam search for best subset selection: each iteration contains several coordinate descent steps to determine whether new variable should be added the support, and clips coefficients to the box [-5,5] 5] as it proceeds. Hence the algorithm is able to determine, before committing to the new variable, whether likely to decrease the loss while obeying the box constraints. This beam search algorithm for solving implicitly uses the assumption that one of the best models of size k implicitly contains variables of one of the best models of size h - 1. This type of assumption has been studied in the sparse learning literature [14] (Theorem 5). However, we not aware of any other work that applics box constraints or beam search for sparse logistic regression. In Appendix E show that method produces better solutions than the OMP method presented in [4].
Algorithm 2 calls the ExpandSuppByl Algorithm, which has two major steps. The detailed algorithm can be found in Appendix A For the first step, given a solution we perform optimization on each single coordinate j outside of the current support supp(w):
d7 arginnn L(w dey, wo, D) for Vjj where wy 0.
(6)
de[-5,5)
5

---

Vector e; is 1 for the jth coordinate and 0 otherwise. We find d: for each j through an iterative thresholding operation, which is done on all coordinates in parallel, iterating several (~ 10) times: for iteration d &- Threshold(j dj, w, wo, D) = min(max((a3) -5), 5):
where Cd, = d, + V - d wo, D). and i is a Lipschitz constant on coordinate j [E4 Importantly, we can perform Equation / on all j where wi = 0 in parallel using matrix form.
For the second step, after the parallel single coordinate optimization is done, we pick the top B indices (j's) with the smallest logistic losses L(w + dze;) and fine tune on the new support:
w new When
argmin L(a, 6, D) with supp(a) supp(w) {i}. 6
(8)
This can be done again using a variant of Equation iteratively on all the coordinates in the new support. We get B pairs of (whew, Wanew) through this ExpandSuppBy1 procedure, and the collection of these pairs form the set W in Line of Algorithm 2
At the end, Algorithm 2 (SparseBeamL returns the best model with the smallest logistic loss found by the beam search procedure. This model satisfies both the sparsity and box constraints.
3.2 Collect Sparse Diverse Pool (Rashomon Set)
We now collect the sparse diverse pool. In Section 3.1 our goal was to find a sparse model (w" wi) with the smallest logistic loss. For high dimensional features or in the presence of highly correlated features, there could exist many sparse models with almost equally good performance 77. This set of models is also known as the Rashomon set. Let us find those and turn them into risk scores. We first predefine a tolerance gap level e (hyperparameter, usually set to 0.3). Then, we delete a feature with index j- in the support supp (=)) and add a new feature with index f+. We select each new index to be f+ whose logistic loss is within the tolerance gap:
Find all j+ S.L. min L(w" - w w + aej+, Wo: D) S L(w", wi, D) +e).
aE|-5,5]
We fine-tune the coefficients each of the new supports and then save the solution in pool. Details can be found in Algorithm 5 Swapping one feature at a time is computationally efficient, and our experiments show it produces sufficiently diverse pools over many datasets. We call this method the CollectSpareeDiverssPoo Algorithm.
3.3 "Star Ray" Search for Integer Solutions
The last challenge is how to get an integer solution from a continuous solution. To achieve this, we use a "star ray" search that searches along each "ray" of the star, extending each continuous solution outward from the origin using many values of multiplier, star ray search provides much more flexibility in finding good integer solution than simple rounding. The largest multiplier mmaa is set to 5/ max, ([wāl) which will take one of the coefficients to the boundary of the box constraint at 5. We set the smallest multiplier to be 1.0 and pick Nm (usually 20) equally spaced points from mmi, mmax] If maax 1, we set mmii 0.5 to allow shrinkage of the coefficients. We scale the coefficients and datasets with each multiplier and round the coefficients to integers using the sequential rounding technique in A Algorithm 6 For each continuous solution (each "ray" of the "star"), we report the integer solution and multiplier with the smallest logistic loss. This process yields multiplier does not work, because the rounding error is highly non-convex.
shown in Algorithm 3 The
collection of risk scores. Note here that a standard line search along the
We briefly discuss how the sequential rounding technique works. Details of this method can be found in Appendix AA] We initialize w+ = w. Then we round the fractional part of w one coordinate at a time. At each step, some of the w 's are integer- valued (so w, wy is nonzero) and we pick the coordinate and rounding operation (cither floor or ceil) based on which can minimize the following objective function, where we will round to integer at coordinate r":
r", x² E argmin T.
Σ 1
Zv - wr) + Σ 2ms(w5 - w5)
(10)
subject to r E {il w & 2) and v E {( Tw

---

Algorithm 3 StarRaySearch(D, (w, wo). C, Nm) - (wt w m
Input: dataset D. a sparse continuous solution (w, wo), coefficient constraint C, and number of multipliers to try Nm-
Output: sparse integer solution (w wx)) with w| || S C and multiplier
1: Define Mmax + C/ max/w] discussed in Section 3.3 If Mmmax
set Mmin 0.5; if
Mmma > 1, set M1mmn f 1.
2: Pick Nm equally spaced multiplier values my E [ mmax] for E [1, Nm) and call this
set M {
3: Use each multiplier scale the good continuous solution (w, wo). to obtain (myw, miwo). which is a good continuous solution to the rescaled dataset mi D.
4: Send each rescaled solution (mzw, m.wo) and its rescaled dataset 1 to Algorithm AuxiliaryLossRouddinggg 1 (w- " is close to (mrw, mywo), and where (w-1, wo +() on , DD has a small logistic loss.
my m.m,w, m.wa) for rounding. It returns (w ,wg' mm), where
5: Evaluate the logistic loss to pick the best multiplier 1" E argminy L(w 1 w D)
6: Return (w+*", wwt" ) and my-
where 4 is the Lipschitz constant restricted to the rounding interval and can be computed as 4 1/(1 + exp(y, 7:)) with Ny [wy] if yeiig > and Ki =[wp] otherwise. (The Lipschitz constant here is much smaller than the one in Section 3.1 due to the interval restriction.) After we select r" and find value v", we update w+ by setting w. v². We repeat this process until w" is on the integer lattice: w E 2?, The objective function in Equation 10 can be understood as an auxiliary upper bound of the logistic loss. Our algorithm provides an upper bound on the difference between the logistic losses of the continuous solution and the final rounded solution before we start the rounding algorithm (Theorem [.T beloww. Additionally, during the sequential rounding procedure, we do not need perform expensive operations such logarithms or exponentials required by the logistic loss function; the bound and auxiliary function require only sums of squares, not logarithms or exponentials. Its derivation and proof are in Appendix C
Theorem 3.1. Let t be the real-valued coefficients for the logistic regression model with objective function L(w) = Ei= log(1 + exp(-y;xt w)) (the intercept is incorporated). Let w be the integer-valued coefficienss returned by the AuxiliaryLossRonnde
method. Furthermore, let uj Wi Lwy]. Let l4 1/(1 + exply;æf x3) with 7aj Lw;] if yawig > 0 and Tisj -wwi] otherwise. Then, we have an upper bound on the difference between the loss L(w) and the loss L(w+)::
I(w=)) - L(w) S
n
(Lan;)) 2u, uj). (1)
(11)
==1=11
Note. Our method has higher prediction capacity than RiskSLIM: its search space is much larger Compared to RiskSLIM, our use of the multiplier permits a number of solutions that grows exponen- tially in k as we increase the multiplier. To see this, consider that for each support of & features, since logistic loss is convex, it contains a hypersphere in coefficient space. The volume of that hypersphere is (as usual) V +1)"
2h/2 pk where is the radius of the hypersphere. If we increase the multiplier to
2, the grid becomes finer by a factor of 2, which is equivalent to increasing the radius by a factor of 2. Thus, the volume increases by a factor of 2k, In general, for maximum multiplier m, the search space is increased by a factor of over RiskSLIM.
4 Experiments
We experimentally focus on two questions: (1) How good is FasterRisk's solution quality compared to baselines? ( (2) How fast is FasterRisk compared with the state-of-the-aa ($4.2) In the appendix, we address three more questions: (3) How much do the sparse beam search, diverse pools,
and multipliers contribute to our solution quality? E.4 (4) How well-calibrated are the models
produced by FasterRisk?
(5) How sensitive is FisterRisk to each of the hyperparameters the
algorithm? (E.10)
7

---

train auc vs model size
test auc vs model size [adult) (n-37 2-36)
train auc vs model size (bank) (n-41188, p=55)
test auc vs model size
(adult) in-622 p636)
(bank) (n-41188 5555)
80
0.80
00
20
60
Sest Aa
5..00
3.700
70
0.40
0.60
moddl Slzs
model Sine
mudel sioe
madel size
traiin auc vs model sine
test auc vs model size
train auc vs model size
test auc vs model size
Imammmol (n-911. 141
Imammel (n-961, p=14)
(uushroom (n B124, p 1=31
[mush room) (n- BB.24, 113)
0.85
0.83
vaain gaz.
80
taxk site
2.80
0.80
0.80
6660
0.60
10
model
traiin auc madel size (compasl (n-6907, p=134)
test ouc vs model 5:ze (compas) (n=7907, p=134)
train auc VS model size
test aMC vs model size
(fico) (n= 10459.99 9, 19177)
(fico) [n=10459, p=1917)
80
0.80
0.70
0.70
0.40
30550
60
0.50
0.40
0.40
16
Pooled PLR RD Pooles--PLR-RRRD
Posted PLR RDSP Posted-PLR-RDP
Pooled PLR ROU
RISKSUM FasterRisk (ours)
Rooled-PPLRRand
Figure 3: Performance comparison. FasterRisk outperforms all baselines due to its larger hypothesis space. On the datasets with nighl--correlate variables such as COMPAS and FICO (both in the bottom row), FasterRisk outperforms other methods by a large margin.
train auc model size (adult) In- 32561, 22561_pm33 pm36)
test auc vs model size [adult) In- 32561_ p-36)
train auc vs model size (bankl (1-41186. 1188_p-55) p-55)
test aut vs model size (bank) (n-4) 188. p=55)
ab
76
0.00
20
074
0.74
10
10
6
10
10
train model size
test model size [mammel 061, p-14)
train auc model size
test model size
(mammol) (n-961, a- 14)
(mushroom) (n- B124, 1=13
(mushroom) (n-8124, p=113)
0.20
0.86
00
1.00
0.80
3032
1034
trin
96
toxt aik
2.95
0.82
0:30
10
10
10
size
model sine
sioe
model size
model
train model size
test model
(compas) (n=6907, 1-134)
(compas) (n=5907, p=134]
(fico) [n=10459, p=1717)
(fico) In=10459, p=19771
train us
75
0.70
0.68
0.50
0.70
10
model Fize
model Fibe
model cloc
model size
RISKSUM
Festeriiikk (ours)
Figure 4: Performance comparison between FasterRisk and RiskSLIM.
We compare with RiskSLIM (the current state-of-the-art), as well as algorithms Pooled-PLR-RO RD, Cooled-PLR-RSRD. PRL- Pooled-PLR- and Pooled-PRL-RDD These algorithms were all previously shown to be inferior to RiskSLIM 39 These methods first find a pool of sparse continuous solutions using different regularizations of ElasticNet (hence the name "Pooled Penalized Logistic Regression" - PPoled-PLLP PLR) and then round the coefficients with different techniques. Details are Appendix [33] The best solution is chosen from this pool of integer solutions that obeys the sparsity and box constraints and has the smallest logistic loss. We also compare with the baseline AutoScore 4 However, on some datasets, the results produced by AutoScore are so poor that
they distort the AUC scale, show those results only in Appendix E.IT As there is publicly
8

---

available code for any of TO, 16 32 [3] they do not appear in the experiments. For each dataset, we perform 5-fold cross validation and report training and test AUC. Appendix DJrresents details of the datasets, experimental setup, evaluation metrics, loss values, and computing platform/enviroomeent. More experimental results appear Appendix
4.1 Solution Quality
We first evaluate FasterRisk's s solution quality. Figure shows the training and test AUC on six datasets (results for training loss appear in Appendix E FasterRisk (the red line) outperforms all baselines, consistently obtaining the highest AUC scores on both the training and test sets. Notably, our method obtains better results than RiskSLIM, which uses mathematical solver and the current state-of-the-ar method for scoring systems. This superior performance is due to the use of multipliers, which increases the complexity of the hypothesis space. Figure provides a more detailed comparison between FasterRisk and RiskSLIM. One may wonder whether running RiskSLIM longer would make this MIP-based method comparable to our FasterRisk, since the current running time limit for RiskSLIM is only 15 minutes. We extended RiskSLIM's running time limit up to I hour and show the comparison in Appendix E.8 FasterRisk still outperforms RiskSLIM by a large margin.
FasterRisk performs significantly better than the other baselines for two reasons. First, the continuous sparse solutions produced by ElasticNet are low quality for very sparse models. Second, it difficult to obtain an exact model size by controlling & regularization. For example, Pooled-PI.R PLRRDD and Pooled-PL.R R-RDSP do not have results for model size 10 on the mammo datasets, because no such model size exists in the pooled solutions after rounding.
run time vs model size (adult) [n=32561, p-36) 18 14 10 10
run time vs model size
run time vs model size (mammo) (n=961, p=14)
(bank) (n=41188, p=55)
14 120 to 16 10 1.
800
800
800
run time
600
run time
600
600
400
400
run
400
200
10
made size
model size
model size
run time modell size
run time vs model size (compas) pn=6907_p==34 p=134)
time model size
(mushroom) (n=8124, p=113)
(fico) (n=10459, p=1917)
800
800
800
run time
600
run time
600
time
600
200
200
200
0
2
8
10
2
8
1.0
10
mode size
model size
model size
FasterRisk (ours)
RiskSLIM
Run time
900ss
Figure 5: Runtime Comparison. Runtime (in seconds) versus model size for our method FasterRisk (in red) and the RiskSLIM (in blue). The shaded blue bars indicate cases that timed out (T.O.) at 900 seconds.
4.2 Runtime Comparison
The major drawback of RiskSLIM is its limited scalability. Runtime is important to allow interactive model development and to handle larger datasets. Figure [euure3]show shows that FasterRisk (red bars) is significantly faster than RiskSLIM (blue bars) in general. We ran these experiments with a 900 second (15 minute) timeout. RiskSLIM finishes running on the small dataset mammo, but it times out on the larger datasets, timing out on models larger than 4 features for adult, larger than 3 features for bank, larger than 7 features for mushroom, larger than 2 features for COMPAS, and larger than I

---

feature for FICO. RiskSLIM times out early on COMPAS and FICO datasets, suggesting that the MIP-based method struggles with high -ish-simessinal and highly-correlated features. Thus, we see that FasterRisk tends to be both faster and more accurate than RiskSLIM.
4.3 Example Scoring Systems
The main benefit of risk scores is their interpretability. We place few example risk scores in Table T to allow the reader to judge for themselves. More risk scores examples can be found in Appendix provide pool of solutions for the top 12 models the bank, mammo, and Netherlands datasets in Appendix F22 Prediction performance generally not the only criteria users consider when deciding deploy a model. Provided with pool of solutions that perform equally
F.T
Additionally,
well, user can choose the one that best incorporates domain knowledge [45]. After the pool of models generated, interacting with the pool essentially computationally instantaneous, Finally, we can reduce some models to relatively prime coefficients or transform some features for better interpretability. Examples of such transformations are given in Appendix G.1
no high school diploma
-4 points
1.
odor=almond
-5 points
2. high school diploma only
-2 points
+
2.
odor=anise
-5 points
3. age 22 to 29
-2 points
3.
odor=none
-5 points
any capital gains married
points
4.
odor=foul
points
points
5.
gill broad
-3 points
SCORE
SCORE
<-4
3
SCORE
3
>2
RISK <1.3%
2.4%
4.4%
7.8%
13.6%
RISK
1.62%
26.4%
73.6%
>99.8%
SCORE
2
3
7
RISK
22.5%
35.0%
50.5%
65.0%
92.2%
(a) FasterRisk models for the adult dataset, predicting salary>> 50K.
(h) FasterRisk model for mushroom dataset, predicting whether a mushroom is poisonous.
Table 1: Example FasterRisk models
Conclusion
FasterRisk produces collection of high-quality risk scores within minutes. Its performance owes to three key ideas: a new algorithm for sparsity- and box-constrained continuous models, using a pool of diverse solutions, and the of the ray search, which leverages multipliers and sequential rounding technique. FasterRisk suitable for high-stakes decisions, and permits domain experts collection of interpretable models to choose from.
Code Availability
Implementations of FasterRisk discussed in this paper are available at com/ jiachanglit/FPatteriiss
Acknowledgements
The authors acknowledge funding from the National Science Foundation under grants IIS-2147061 and IIS-2130255 National Institute on Drug Abuse under grant RO1 DA054994, Department of En- ergy under grants DE-SC0021358 and DE-SC0023194 and National Research Traineeship Program under NSF grants DGF--222040 and CCF-1934964. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). Nous remercions Conseil de recherches sciences naturelles et génic du Canada (CRSNG) de son soutien.
10

---

References
[1] Izuwa Ahanor, Hugh Medal, and Andrew C. Trapp. Diversitree: Computing diverse sets of near-optimal solutions to mixed-integer optimization problems. arXiv, 2022.
[2] Mohamed Farouk Allam. Scoring system for the diagnosis of COVID-19. The Open Public
Health Journal, 13(1), 2020.
[3] Virginia Apgar. A proposal for new method of evaluation of the newborn infant. Current Researches in Anesthesia and Analgesia, 1953(32):260-267, 1953.
[4] James Austin, Roger Ocker, and Avi Bhati. Kentucky pretrial risk assessment instrument validation. Bureau of Justice Statistics, 2010.
[5] Lieven Billiet, Sabine Van Huffel, and Vanya Van Belle. Interval coded scoring extensions for larger problems. In Proceedings of the IEEE Symposium on Computers and Communications, pages 198-203. IEEE, 2017.
[6] Lieven Billiet, Sabine Van Huffel, and Vanya Van Belle. Interval Coded Scoring: A toolbox for interpretable scoring systems. PeerJ Computer Science, 4:e150, 04 2018.
[7] Leo Breiman. Statistical modeling: The two cultures. Statistical Science, 16(3); (293):99-231, 2001.
[8] Ernest W Burgess. Factors determining success or failure on parole. Illinois Committee on Indeterninate-Sentenee Law and Parole Springfield, II, 1928.
[9] E. Carrizosa, A. rogales-Gómez' and D. Romero Morales. Strongly agree or strongly disagree?: Rating features in support vector machines. Technical report, Said Business School, University of Oxford, UK, 2013.
[10] Yann Chevaleyre, Frédéerick Koriche, and lean-Daniel Zucker. Rounding methods for discrete linear classification. In International Conference on Machine Learning, pages 651-659. PMLR, 2013.
[1]] TJ Cole. Scaling and rounding regression coefficients to integers. Journal of the Royal Statistical Society: Series (Applied Statistics), 42:261-268, 1993.
[12] Lorrie Faith Cranor and Brian A LaMacchia. Spam! Communications of the ACM, 41(8):74-83, 1998.
[13] Antoine Dedieu, Hussein Hazimeh, and Rahul Mazumder. Learning sparse classifiers: Continu- ous and mixed integer optimization perspectives. Journal of Machine Learning Research, 22:
135-1, 2021.
[14] Ethan R Elenberg, Rajiv Khanna, Alexandros G Dimakis, and Sahand Negahban. Restricted strong convexity implies weak submodularity. The Annals of Statistics, 46(6B))3539-35)) -368,, 2018.
[15] Matthias Elter, Rüdiger Schulz-Wendtland and Thomas Wittenberg. The prediction of breast cancer biopsy outcomes using two cad approaches that both emphasize an intelligible decision process. Medical Physics, 34(1 (4(11):416444177 2007.
[16] Söyda Ertekin and Cynthia Rudin. bayesian approach to learning scoring systems. Big Data,
3(4):267-276, Dec 2015.
[17] FICO, Google, Imperial College London, MIT, University of Oxford, UC Irvine, and UC Berkeley. Explainable Machine Learning Challenge. https:://oommun:: fico. com/s. s/ explainable- --- chaigg-challenege. haaiichhh---arrnnng-caaaaaaegeg 2018.
[18] Brian F Gage, Amy D Waterman, William Shannon, Michael Boechler, Michael W Rich, and Martha Radford. Validation of clinical classification schemes for predicting stroke. The Journal of the American Medical Association, 285(22)28644-2870 2001.
[19] Ronald C Kessler, Lenard Adler, Minnie Ames, Olga Demler, Steve Faraone, EVA Hiripi, Mary J Howes, Robert Jin, Kristina Secnik, Thomas Spencer, and et al. The world health organization adult ADHD self-report scale (ASRS): a short screening scale for use in the general population. Psychological Medicine, (5(02):245-256, 2005.
[20] Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Proceedings Knowledge Discovery and Data Mining (KDD), volume 96, pages 202-207, August 1996.
[21] Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. How we analyzed the COM- PAS recidivism algorithm. ProPublica. May 23, httph:ttp::tt http:/::///wwwpppbbbbbccaa org/article/ lov-we-analyyzd- wwwpzzeed-the-emeeeee www.eerrrriiiiiiialaeeereee
2016.
[22] Edward Latessa, Paula Smith, Richard Lemke, Matthew Makarios, and Christopher Lowenkamp. Creation and validation of the Ohio risk assessment system: Final report. httpp: : : /wwj///w.jj.......oo...///////..
ohio. http://wwss.iiaaaeeerrrt.r.. 2009.

---

[23]
Ji Yeon Lee, Byung-Ho Ho Nam, Mhinjine Kim, Jongmin Hwang, Jin Young Kim, Miri Hyun, Hyun Ah Kim, and Chi-Heum Cho. A risk scoring system to predict progression to severe pneumonia in patients with 19.. Scientific Reports, 12(1):1-8, 2022.
[24] Jiachang Liu, Chudi Zhong. Margo Seltzer, and Cynthia Rudin. Fast sparse classification for generalized linear and additive models. In Proceedings of International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
[25] Aurelie Lozano, Grzegorz Swirszez, and Naoki Abe. Group orthogonal matching pursuit for logistic regression. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 452-460, 2011.
[26] Olvi L Mangasarian, W Nick Street, and William H Wolberg. Breast cancer diagnosis and prognosis via linear programming. Operations Research, 43(4):570- 1995.
[27] Rui P Moreno, Philipp GH Metnitz, Eduardo Almeida, Barbara Jordan, Peter Bauer, Ri- cardo Abizanda Campos, Gaetano lapichino. David Edbrooke, Maurizia Capuzzo, and Jean- Roger Le Gall. SAPS3 - from evaluation of the patient to evaluation of the intensive care unit. part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive 2005.
Care Medicine,
[28] Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22-31, 2014.
[29] Jeffrey Curtis Schlimmer. Concept acquisition through representational adjustment. PhD thesis, University of California, Irvine, 1987.
[30] Yufeng Shang, Tao Liu. Yongchang Wei, Jingfeng Li, Liang Shao, Minghui Liu, Yongxi Zhang, Zhigang Zhao, Haibo Xu, Zhiyong Peng, et al. Scoring systems for predicting mortality for severe patients with COVID-19. EClinicalMedicine, 24:100426, 2020.
[31] A. Jacob. Six, Barbra E. Backus, and Johannes C. Kelder. Chest pain in the emergency room:
value of the heart score. Netherlands Heart Journal, 16(6):191-196 2008
[32] Nataliya Sokolovska, Yann Chevaleyre, Karine Clément, and Jean-Daniel Zucker. The fused lasso penalty for learning interpretable medical scoring systems. In 2017 International Joint Conference Neural Networks (IJCNN), pages 4504-4511. IEEE, 2017.
[33] Nataliya Sokolovska, Yann Chevaleyre. and
Zucker. A provable algorithm for learning interpretable scoring systems. In Proceedings of International Conference on Artificial Intelligence and Statistics, pages 566-574. PMLR, 2018
[34] Martin Than, Dylan Flaws, Sharon Sanders, Jenny Doust, Paul Glasziou, Jeffery Kline, Sally Aldous, Richard Troughton, Christopher Reid, and William A Parsonage. Development and validation of the emergency department assessment of chest pain score and 2h accelerated diagnostic protocol. Emergency Medicine Australasia, 26(1 [[ 134444, 2014.
[35] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):667-288 1996.
[36] Nikolaj Tollenaar and P.G.M. van der Heijden. Which method predicts recidivism best?:
comparison of statistical, machine learning and data mining predictive models. Journal of the Royal Statistical Society: Series A (Statistics in Society), 176(2):565-544 2013.
[37] Berk Ustun and Cynthia Rudin. Supersparse linear integer models for optimized medical scoring systems. Machine Learning, pages 1 1-43, 2015. ISSN 0885-6125 doi: 10. 1007 URL http:/t/://d doi org/ 1007/
s $10994-015-5522
[38] Berk Ustun and Cynthia Rudin. Optimized risk scores. In Proceedings of the 23rd SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1125-1134,201
[39] Berk Ustun and Cynthia Rudin. Learning optimized risk scores. Journal of Maching Learning Research, 20:150-1, 2019.
[40] Berk Ustun, Stefano Traca, and Cynthia Rudin. Supersparse linear integer models for predictive scoring systems. In Workshops at the Twenty-Sevente AAAI Conference on Artificial Intelligence. 2013.
[41] Berk Ustun, Lenard A Adler, Cynthia Rudin, Stephen V Faraone, Thomas J Spencer, Pa- tricia Berglund, Michael J Gruber, and Ronald C Kessler. The world health organization adult attention-defici / hyperactivity disorder self-report screening scale for DSM-5. JAMA Psychiarry, 74(5):520-52) 526, 2017.
[42] Caroline Wang, Bin Han, Bhrij Patel, and Cynthia Rudin. In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction. Journal of Quantitative Criminology, pages 1-63, 2022. ISSN 0748-4518. doi: 10. 10011000220-2-9955555 [43] Piotr Wasilewski, Bartosz Mruk, Samuel Mazur, Gabriela Póttorak- Syymmakk,-- Katarzyna Sklinda, and Jerzy Walecki. COVID-19 19 severity scoring systems in radiological imaging-a
12

---

review. Polish Journal of Radiology, 85(1):3613368, 2020.
[44] Feng Xie. Bibhas Chakraborty, Mareus Eng Hock Ong. Benjamin Alan Goldstein, Nan Liu, et al. Autoscore: A machine learning- based automatic clinical score generator and its application to mortality prediction using electronic health records. JMIR Medical Informatics, 8(10):21778. 2020.
[45] Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Explor- ing the whole rashomon set of sparse decision trees. In Proceedings of Neural Information Processing Systems, 2022.
[46] Jiaming Zeng, Berk Ustun, and Cynthia Rudin. Interpretable classification models for recidivism prediction. Journal of the Royal Statistical Society: Series A (Statistics Society), 180(3):
689-722. 2017.
[47] Chi Zhang. Ling Qin, Kang Li, Qi Wang, Yan Zhao, Bin Xu, Lianchun Liang, Yanchao Dai, Yingmei Feng. Jianping Sun, al. A novel scoring system for prediction of disease severity in COVID-19. Frontiers in Cellular and Infection Microbiology, 10:318, 2020.
[48] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301-320, 2005.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the Limitations section in Appendix H
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the
Limitations section in Appendix H
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide public GitHub repository link under the section Code Availability. Data links included the Appendix [..]]
(b) Did you specify all the training details (e.g., data splits, byperparameters, how they were chosen)? [Yes] We perform 5-fold CV as specified in Section 4 Hyperparameters are already specified (default values) in Algorithm of Section 5
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Section CV.
Error bars are included for the 5-fold
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix D.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Appendix D.3 and the References
(b) Did you mention the license of the assets? [Yes] See Appendix D.3
(c) Did you include any assets either the supplemental material
URL? [Yes]
Our code is provided through a public GitHub repository link under the section Code Availability.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "x2z82euYSrP", "reviewer": "Reviewer_aCoP", "summary": "This paper aims to provide a fast algorithm to derive sparse risk scores that scales to high-dimensional datasets. The authors identified a few major limitations in current methods, and described how these were addressed by the three components in their proposed algorithm. In several experiments with low and high dimension data, the authors showed that their method outperformed the current state-of-the-art and several other baseline methods. The algorithm is implemented in stand-alone Python code, which is advantageous over competitors that rely on mathematical programming solvers.", "questions": "1. The authors focused on developing scores by finding integer sparse solutions, and showed some advantages of FasterRisk over the state-of-the-art. But I find the discussion of alternative approaches inadequate, therefore I could not fully appreciate the contribution of this work. For example, the authors pointed out two major limitations of building scores by rounding logistic regression coefficients: (i) $l_1$ and $l_0$ regularizations not able to get sparse solutions, and (ii) rounding of coefficients worsens performance by making the scores too coarse. But (i) may be resolved by using alternative variable selection methods and (ii) by using smallest non-zero coefficient to scale all coefficients and then rounding to larger integer values (e.g., total score ranging from 0 to 100). For example, a 2020 paper (https://doi.org/10.2196/21798) describes such an alternative approach that worked reasonably well in several clinical applications, and by separating variable selection from score development, domain experts are more easily engaged in the development process to ensure clinical meaningfulness and fairness. I find this paper lacking in discussion on this general approach. Could the authors include such more recent works in their discussion and method evaluation?\n\n2. The authors stated in appendix that the choice of hyperparameters does not have much impact on performance, but did not provide empirical evidence. I am particularly concerned with the choice of tolerance gap level $\\{epsilon}=0.3$ (equation (9)), meaning we are willing to tolerate up to 30% increase in loss when expanding to “equally good” scores. Without detailed explanation, 30% seems too large to me. Can the authors justify their choice empirically or by citing related literature? I would also like to see some empirical results regarding change in other hyperparameters.\n\n3. Although the authors generated a pool of “equally good” scores, they did not seem to make use of them other than selecting the best-performing one to report. This pool of scores could be useful for users to select well-performing AND fair scores. This is related to Limitations below.", "limitations": "Fairness of scores developed from the proposed algorithm is not adequately discussed. Related to Question 3 above, a naïve application of the proposed method may lead to unfair risk scores. For example, in Table 3 of Appendix F, the method generated a 3-variable risk score to predict salary>30K using education level and marital status, and this direct link of being married with salary level is highly debatable. Marital status might represent a mixed effect of age and socio-economic status, and it may be better to use the latter in the score for more meaningful interpretation. Since the authors have generated a pool of “equally good” scores that make use of alternative predictors in a step of the algorithm, I suggest the authors make use of this pool to help users balance performance and fairness.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper describes the authors’ original work to resolve several methodological difficulties in current development of risk scores. The writing is clear in defining several major challenges the authors aimed to address, and the proposed algorithm consists of separate components to address them. In addition to evaluating the algorithm with respect to baselines, the authors also showed the importance of each component by assessing the reduction in performance without them.\n\nMy major concern is the scope of this study, which affects its quality and significance. The authors aim to develop a fast and well-performing algorithm, FasterRisk, to develop sparse risk scores, which, if successful, would be very useful to healthcare applications. But when discussing related work, the authors did not include some recent works that have partially addressed some of the limitations the authors proposed to address (elaborated in Question 1 below). When evaluating FasterRisk, there is practically only one competitor algorithm, and FasterRisk only had marginal advantage in performance (in most experiments). FasterRisk is indeed much faster than the competitor, but by timing out the run time at 15 minutes, I am not convinced that the competitor is slow enough to be concerning in practice.\n\nThe clarity in mathematics notations can be improved. The equations became difficult to follow when the author used some notations without introducing them. For example, $\\{epsilon}_w$ in equation (4) and $\\{epsilon}_t$ in equation (5) lack bounds, and it is difficult to understand what the arbitrary a, b, c, d, e in equations (6)-(9) stand for. These affect my trust in the work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "i-i0eHnRLJ", "reviewer": "Reviewer_DdPk", "summary": "The paper is focused on risk scores learning which are simple but efficient (in terms of performance) models. The main idea is to produce a pool of almost-optimal sparse continuous solutions with different support sets using a beam-search algorithm. Each of these solutions is explored: the real-values models are transformed into feasible integer-valued solutions along multipliers (what allows for a large space of possible solutions). The method is computationally efficient. ", "questions": "It seems that m is defined quite late in text, and it is not clear from the beginning of Section 3 that it is the multiplier. Why did you decide to divide by m and not to multiply (if it is a multiplier)?\n\nIn Section 3.2. it is mentioned that \"swapping one feature at a time is computationally efficient\". I would say rather not efficient, if there are a lot of features. \n\nI am not sure whether Section 4.3. is necessary. It underlines that there are not any results on real scores in the current submission. I guess the example provided in the Introduction (and Appendix) is enough. \n\nI am curious to know what the range of m (multiplier) values was in your experiments?\nHow many intermediate (pool) models did you generate? And what is the percentage of reasonable final (integer) models?\n", "limitations": "The authors provide the limitations on page 3. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths. The paper clearly describes a novel three step framework to learn simple interpretable models. The numerical results are convincing. \n\nWeaknesses. The method has three separate steps what can lead to some inconveniences (some kind of error cumulation is possible; coordinate descent can be long as well as the line search).  \nAs also mentioned by the authors, real scores are not considered in the current contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hDYcngHdlgq", "reviewer": "Reviewer_ntN1", "summary": "The authors propose a method for efficiently automatically generating a pool of “risk scores” (sparse linear models with integer coefficients), involving (1) a beam search algorithm to identify a sparse set of features, (2) given the original set of features, identify a pool of sparse solutions with similar performance (but “diverse” set of features), and (3) “star ray” search to choose integer coefficients. They evaluate both the speed and accuracy of their approach on multiple benchmark datasets. ", "questions": "- You mention  (line 54)  “We need an approach that exhibits the best of both worlds: speed fast enough to operate in a few  minutes on a laptop and optimization and search capability as powerful as that of the mathematical programming tools. Our method, FasterRisk, lies at this intersection.” — Of course, it makes sense to have a goal of developing an accurate model, but I’m wondering why it is important that risk scores must operate in a few minutes on a laptop. If these are risk scores for a high-impact situation, why can’t we run an analysis for an hour? Or even a week? \n\n- Figure 4:   I find this figure a bit frustrating, because it’s not really allowing me to see how the baseline method scales compared to yours.  A 15 minute time-out seems kind of silly/arbitrary (in the real world, I’d expect people to be willing to train their methods for quite a while if it’s for a high stakes setting), and I would highly recommend allowing your baselines to run for longer (at least several hours) to show how the times actually scale (and then possibly display with a log scale as needed). It would also be helpful to share the number of rows and columns in each of the datasets to give a sense of scale. \n\n- Section 4.3 Example scoring systems: how were these examples selected?   Also, you mention that risk scores offer interpretability – is this something that you argue is unique to your approach and not your baselines?\n\n- Your algorithm is often described to return a “pool” of solutions, but I didn’t see much discussion of what  that pool actually looks like. For example, I would want to see how many solutions were produced, how different they are from each other, etc.   In a use case, would you expect the user to look through all of them and then choose one, or just defer to the lowest-error one?\n", "limitations": "The authors describe some limitations related to their algorithms, which is appreciated. However, they describe the potential negative societal impacts as “[N/A]” which seems like a huge oversight to me. As they say, “[risk scores] are possibly the most popular form of predictive model for high stakes decisions through the last century and are the standard form of model used in criminal justice  and medicine,” it seems obvious that any contribution they make to this field could have serious societal impacts, for better or worse.\n\nHere’s an example of how this approach could be used problematically:  Let’s say we have a criminal justice scenario in which we have access to race and some other features that are essentially proxies for race (e.g., zip code). \tNow let’s assume a user is given a pool of “diverse” solutions by the FasterRisk approach, and they know that they don’t want a model that’s “racist”. They may notice one risk score has a highest accuracy but relies on race, so they decide they shouldn’t use that model. They then notice an almost identical model that has all the same features except race has been replaced by zip code, and they choose this model instead and deploy it in some real world scenario (e.g., recommending whether someone should be placed on parole).\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors describe an interesting method for quickly identifying risk scores in a diverse range of settings. The main improvement seems to be speed (which wasn’t very well quantified with respect to baselines), since performance-wise  it was similar to a previous approach – and I wonder how useful this speed would actually be in this high-stakes offline setting. \n\nOriginality/clarity: this seems to be a creative combination of past algorithms, and they described their algorithms in detail\n\nQuality/significance: the authors evaluated their methods and baselines along both accuracy and speed metrics, and also shared extensive additional experiments in their supplement. One aspect that seemed problematic was that for speed plots, they cut off algorithms after 15 minutes, and it’s unclear exactly how their baselines scale because they tend to be censored after just one or two points along the x-axis. It would also be helpful to see if quantitatively, there are any significant differences in AUC between FasterRisk and alternative methods. Based on these two results, it would be easier to assess whether their method provides a meaningful contribution to real-world use cases of risk scores.\n\nThe authors also do not describe any considerations of societal impact which is an important factor given their suggested use cases. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "X3fZeJvDzA7", "reviewer": "Reviewer_ifPK", "summary": "This study proposed a novel method to accurately and efficiently generate a collection of high-quality risk scores based on the integration of the beam-search-based algorithm for LR, the generation of diverse high-quality solutions with different support sets, and the star search for integer solutions. It achieved SOTA performance with less time cost in some datasets. ", "questions": "\nNone", "limitations": "None", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe proposed three-step framework includes a beam-search-based algorithm for logistic regression with box constraints and L0 regularization, the search algorithm to collect the sparse diverse pool with different support set, and the star search technique using multipliers, and a theorem guaranteeing the quality of the star search results. The whole methodology was solid and efficient. \n2.\tThe introductions of the research context and related work were well-organized and clear. \n3.\tThe proposed method achieved the SOTA performance with significantly less time (as shown in figure 4), showing its reliability and efficiency. \n4.\tTheir theoretical discussion and supplement material were abundant, Moreover, they also conducted extensive experiments on performance, including performance comparison, efficiency, and ablation experiments. \n\nWeaknesses:\n1.\tThe examples of scoring systems in the Introduction seem out of date, there are many newer and recognized clinical scoring systems. It also should briefly introduce the traditional framework of the scoring system and its difference in methodology and performance with the proposed method. \n2.\tAs shown in figure 3, the performance improvement of proposed methods seems not so significant, the biggest improvement in the bank dataset was ~0.02. Additionally, using some tables to directly show the key improvements may be more intuitive and detailed. \n3.\tAlthough extensive experiments and discussion on performance, in my opinion, its most significant improvement would be efficiency, and there are few discussions or ablation experiments on efficiency. \n4.\tThe model AUC can assess the model discriminant ability, i.e., the probability of a positive case is bigger than that of a negative case, but may be hard to show its consistency between predicted score and actual risk. However, this consistency may be more crucial to the clinical scoring system (differentiated with classification task). Therefore, the related studies are encouraged to conduct calibration curves to show the agreement. It would be better to prove the feasibility of the generated scoring system?  The difference between the traditional method and our method can also be discussed in this paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept