Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: This work investigates potential improvements to the “AutoScore” Framework (Xie et al. 2020) in variable selection and automatic model selection. They suggest multiple variable selection methods and propose a simple automatic model selection based on a threshold of the relative model performance to the maximum.
strengths_and_weaknesses: Strengths:

- Figure 1 is very clear on what the paper is looking at. The paper is well written overall.
- Overall this work seems reproducible. The code has an excellent readme which clearly describes the method.

Weaknesses:

- It isn’t clear to me what curve the “AUC” metric is referring to. I’m guessing either receiver operator characteristic or precision recall. Given that an automatic score is based on thresholding this value it would be good to discuss which score this is and when you might want to threshold based on it or another score (I can see even just false positive or false negative rate being reasonable in some circumstances).
- One thing that I feel is missing in the scoring is the model complexity and not only the number of input features. A complex model built on a few features may be less desirable than a simple model built on a more features. For instance SHAP with a large MLP may be less useful overall than a random forest or linear model. This is not an issue when only comparing random forest models, but becomes an issue when comparing models of different types.
suggested_changes: - Clarify the score used and why it was chosen as well as its benefits and limitations.

- Clarify the issues with model selection (which is in general a very difficult task) that confounds selection based on the number of variables. 

- Clarity of the abstract: The “… AutoScore leaves room for enhancement: variable ranking via the random forest and manual model selection. In this work, we improved them…” bit of the abstract was a bit unclear to me, but I understand it after reading the next sentence. I Would suggest: “… AutoScore leaves room for enhancement. In this work, we improved AutoScore…”
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: In this paper, the authors investigate methods to augment AutoScore, an existing framework for generating clinical scores. Specifically, the authors seek to improve and expand two modules within the framework: (1) variable ranking using random forest, and (2) model selection, transitioning from manual to automatic, in order to achieve comparable discriminating ability while utilizing fewer variables.
strengths_and_weaknesses: - **Strengths**
    - The authors use a comprehensive approach in experimenting with multiple techniques i.e. five alternative variable ranking methods utilizing both statistical and machine learning methods, as well as an ensemble ranking method via majority voting. This broad exploration of techniques helps to advance research towards a rigorous and comprehensive evaluation of the AutoScore framework.
    - The authors have shared their code, which promotes reproducibility, transparency, and collaboration within the research community.
    - The paper presents a method for moving towards an automatic option for model selection within AutoScore, by aggregating the outputs of individual ranking methods, which contrasts with the current manual model selection process.
- **Weakness**
    - The paper mentions that - “AutoScore applies random forest (RF) (Breiman, 2001) to rank variables, which is biased in some scenarios (Strobl et al., 2007; Nicodemus, 2011)”. I would encourage the authors to expand on the rationale behind their selection of the five alternative variable ranking methods, as well as any potential limitations or drawbacks of these methods in certain scenarios. This could help to provide a more comprehensive and nuanced evaluation of the overall framework.
suggested_changes: None
review_title: null

Meta-review: metareview: This work aims to improve the AutoScore framework by focusing on two aspects: variable ranking using random forest and model selection, transitioning from manual to automatic. The authors experiment with multiple techniques, including five alternative variable ranking methods and an ensemble ranking method using majority voting. Overall, the paper is well-written, clear, and reproducible, with the authors providing their code to promote transparency.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to present