Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a method for efficiently automatically generating a pool of “risk scores” (sparse linear models with integer coefficients), involving (1) a beam search algorithm to identify a sparse set of features, (2) given the original set of features, identify a pool of sparse solutions with similar performance (but “diverse” set of features), and (3) “star ray” search to choose integer coefficients. They evaluate both the speed and accuracy of their approach on multiple benchmark datasets. 
strengths_and_weaknesses: The authors describe an interesting method for quickly identifying risk scores in a diverse range of settings. The main improvement seems to be speed (which wasn’t very well quantified with respect to baselines), since performance-wise  it was similar to a previous approach – and I wonder how useful this speed would actually be in this high-stakes offline setting. 

Originality/clarity: this seems to be a creative combination of past algorithms, and they described their algorithms in detail

Quality/significance: the authors evaluated their methods and baselines along both accuracy and speed metrics, and also shared extensive additional experiments in their supplement. One aspect that seemed problematic was that for speed plots, they cut off algorithms after 15 minutes, and it’s unclear exactly how their baselines scale because they tend to be censored after just one or two points along the x-axis. It would also be helpful to see if quantitatively, there are any significant differences in AUC between FasterRisk and alternative methods. Based on these two results, it would be easier to assess whether their method provides a meaningful contribution to real-world use cases of risk scores.

The authors also do not describe any considerations of societal impact which is an important factor given their suggested use cases. 

questions: - You mention  (line 54)  “We need an approach that exhibits the best of both worlds: speed fast enough to operate in a few  minutes on a laptop and optimization and search capability as powerful as that of the mathematical programming tools. Our method, FasterRisk, lies at this intersection.” — Of course, it makes sense to have a goal of developing an accurate model, but I’m wondering why it is important that risk scores must operate in a few minutes on a laptop. If these are risk scores for a high-impact situation, why can’t we run an analysis for an hour? Or even a week? 

- Figure 4:   I find this figure a bit frustrating, because it’s not really allowing me to see how the baseline method scales compared to yours.  A 15 minute time-out seems kind of silly/arbitrary (in the real world, I’d expect people to be willing to train their methods for quite a while if it’s for a high stakes setting), and I would highly recommend allowing your baselines to run for longer (at least several hours) to show how the times actually scale (and then possibly display with a log scale as needed). It would also be helpful to share the number of rows and columns in each of the datasets to give a sense of scale. 

- Section 4.3 Example scoring systems: how were these examples selected?   Also, you mention that risk scores offer interpretability – is this something that you argue is unique to your approach and not your baselines?

- Your algorithm is often described to return a “pool” of solutions, but I didn’t see much discussion of what  that pool actually looks like. For example, I would want to see how many solutions were produced, how different they are from each other, etc.   In a use case, would you expect the user to look through all of them and then choose one, or just defer to the lowest-error one?

limitations: The authors describe some limitations related to their algorithms, which is appreciated. However, they describe the potential negative societal impacts as “[N/A]” which seems like a huge oversight to me. As they say, “[risk scores] are possibly the most popular form of predictive model for high stakes decisions through the last century and are the standard form of model used in criminal justice  and medicine,” it seems obvious that any contribution they make to this field could have serious societal impacts, for better or worse.

Here’s an example of how this approach could be used problematically:  Let’s say we have a criminal justice scenario in which we have access to race and some other features that are essentially proxies for race (e.g., zip code). 	Now let’s assume a user is given a pool of “diverse” solutions by the FasterRisk approach, and they know that they don’t want a model that’s “racist”. They may notice one risk score has a highest accuracy but relies on race, so they decide they shouldn’t use that model. They then notice an almost identical model that has all the same features except race has been replaced by zip code, and they choose this model instead and deploy it in some real world scenario (e.g., recommending whether someone should be placed on parole).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This study proposed a novel method to accurately and efficiently generate a collection of high-quality risk scores based on the integration of the beam-search-based algorithm for LR, the generation of diverse high-quality solutions with different support sets, and the star search for integer solutions. It achieved SOTA performance with less time cost in some datasets. 
strengths_and_weaknesses: Strengths:
1.	The proposed three-step framework includes a beam-search-based algorithm for logistic regression with box constraints and L0 regularization, the search algorithm to collect the sparse diverse pool with different support set, and the star search technique using multipliers, and a theorem guaranteeing the quality of the star search results. The whole methodology was solid and efficient. 
2.	The introductions of the research context and related work were well-organized and clear. 
3.	The proposed method achieved the SOTA performance with significantly less time (as shown in figure 4), showing its reliability and efficiency. 
4.	Their theoretical discussion and supplement material were abundant, Moreover, they also conducted extensive experiments on performance, including performance comparison, efficiency, and ablation experiments. 

Weaknesses:
1.	The examples of scoring systems in the Introduction seem out of date, there are many newer and recognized clinical scoring systems. It also should briefly introduce the traditional framework of the scoring system and its difference in methodology and performance with the proposed method. 
2.	As shown in figure 3, the performance improvement of proposed methods seems not so significant, the biggest improvement in the bank dataset was ~0.02. Additionally, using some tables to directly show the key improvements may be more intuitive and detailed. 
3.	Although extensive experiments and discussion on performance, in my opinion, its most significant improvement would be efficiency, and there are few discussions or ablation experiments on efficiency. 
4.	The model AUC can assess the model discriminant ability, i.e., the probability of a positive case is bigger than that of a negative case, but may be hard to show its consistency between predicted score and actual risk. However, this consistency may be more crucial to the clinical scoring system (differentiated with classification task). Therefore, the related studies are encouraged to conduct calibration curves to show the agreement. It would be better to prove the feasibility of the generated scoring system?  The difference between the traditional method and our method can also be discussed in this paper. 
questions: 
None
limitations: None
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to provide a fast algorithm to derive sparse risk scores that scales to high-dimensional datasets. The authors identified a few major limitations in current methods, and described how these were addressed by the three components in their proposed algorithm. In several experiments with low and high dimension data, the authors showed that their method outperformed the current state-of-the-art and several other baseline methods. The algorithm is implemented in stand-alone Python code, which is advantageous over competitors that rely on mathematical programming solvers.
strengths_and_weaknesses: This paper describes the authors’ original work to resolve several methodological difficulties in current development of risk scores. The writing is clear in defining several major challenges the authors aimed to address, and the proposed algorithm consists of separate components to address them. In addition to evaluating the algorithm with respect to baselines, the authors also showed the importance of each component by assessing the reduction in performance without them.

My major concern is the scope of this study, which affects its quality and significance. The authors aim to develop a fast and well-performing algorithm, FasterRisk, to develop sparse risk scores, which, if successful, would be very useful to healthcare applications. But when discussing related work, the authors did not include some recent works that have partially addressed some of the limitations the authors proposed to address (elaborated in Question 1 below). When evaluating FasterRisk, there is practically only one competitor algorithm, and FasterRisk only had marginal advantage in performance (in most experiments). FasterRisk is indeed much faster than the competitor, but by timing out the run time at 15 minutes, I am not convinced that the competitor is slow enough to be concerning in practice.

The clarity in mathematics notations can be improved. The equations became difficult to follow when the author used some notations without introducing them. For example, $\{epsilon}_w$ in equation (4) and $\{epsilon}_t$ in equation (5) lack bounds, and it is difficult to understand what the arbitrary a, b, c, d, e in equations (6)-(9) stand for. These affect my trust in the work.

questions: 1. The authors focused on developing scores by finding integer sparse solutions, and showed some advantages of FasterRisk over the state-of-the-art. But I find the discussion of alternative approaches inadequate, therefore I could not fully appreciate the contribution of this work. For example, the authors pointed out two major limitations of building scores by rounding logistic regression coefficients: (i) $l_1$ and $l_0$ regularizations not able to get sparse solutions, and (ii) rounding of coefficients worsens performance by making the scores too coarse. But (i) may be resolved by using alternative variable selection methods and (ii) by using smallest non-zero coefficient to scale all coefficients and then rounding to larger integer values (e.g., total score ranging from 0 to 100). For example, a 2020 paper (https://doi.org/10.2196/21798) describes such an alternative approach that worked reasonably well in several clinical applications, and by separating variable selection from score development, domain experts are more easily engaged in the development process to ensure clinical meaningfulness and fairness. I find this paper lacking in discussion on this general approach. Could the authors include such more recent works in their discussion and method evaluation?

2. The authors stated in appendix that the choice of hyperparameters does not have much impact on performance, but did not provide empirical evidence. I am particularly concerned with the choice of tolerance gap level $\{epsilon}=0.3$ (equation (9)), meaning we are willing to tolerate up to 30% increase in loss when expanding to “equally good” scores. Without detailed explanation, 30% seems too large to me. Can the authors justify their choice empirically or by citing related literature? I would also like to see some empirical results regarding change in other hyperparameters.

3. Although the authors generated a pool of “equally good” scores, they did not seem to make use of them other than selecting the best-performing one to report. This pool of scores could be useful for users to select well-performing AND fair scores. This is related to Limitations below.
limitations: Fairness of scores developed from the proposed algorithm is not adequately discussed. Related to Question 3 above, a naïve application of the proposed method may lead to unfair risk scores. For example, in Table 3 of Appendix F, the method generated a 3-variable risk score to predict salary>30K using education level and marital status, and this direct link of being married with salary level is highly debatable. Marital status might represent a mixed effect of age and socio-economic status, and it may be better to use the latter in the score for more meaningful interpretation. Since the authors have generated a pool of “equally good” scores that make use of alternative predictors in a step of the algorithm, I suggest the authors make use of this pool to help users balance performance and fairness.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper is focused on risk scores learning which are simple but efficient (in terms of performance) models. The main idea is to produce a pool of almost-optimal sparse continuous solutions with different support sets using a beam-search algorithm. Each of these solutions is explored: the real-values models are transformed into feasible integer-valued solutions along multipliers (what allows for a large space of possible solutions). The method is computationally efficient. 
strengths_and_weaknesses: Strengths. The paper clearly describes a novel three step framework to learn simple interpretable models. The numerical results are convincing. 

Weaknesses. The method has three separate steps what can lead to some inconveniences (some kind of error cumulation is possible; coordinate descent can be long as well as the line search).  
As also mentioned by the authors, real scores are not considered in the current contribution.
questions: It seems that m is defined quite late in text, and it is not clear from the beginning of Section 3 that it is the multiplier. Why did you decide to divide by m and not to multiply (if it is a multiplier)?

In Section 3.2. it is mentioned that "swapping one feature at a time is computationally efficient". I would say rather not efficient, if there are a lot of features. 

I am not sure whether Section 4.3. is necessary. It underlines that there are not any results on real scores in the current submission. I guess the example provided in the Introduction (and Appendix) is enough. 

I am curious to know what the range of m (multiplier) values was in your experiments?
How many intermediate (pool) models did you generate? And what is the percentage of reasonable final (integer) models?

limitations: The authors provide the limitations on page 3.
review_title: null

Meta-review: metareview: Thank you for submitting your paper to NeurIPS! This paper makes a valuable contribution to the scoring model literature, providing a fast and scalable algorithm to derive sparse risk scores. The reviewers uniformly appreciated the methodological approach (integrating beam search with logistic regression, diverse feature selection, and star search for choosing integer coefficients), and noted that the stand-alone Python implementation is also advantageous over competitors that rely on mathematical programming solvers. I am pleased to recommend acceptance of this practically relevant work.
confidence: Certain
Decision: Accept