Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Develop a data-driven policy learning framework to infer clinical decision making processes around the applied question of organ transplant offers and acceptances (organ donation data). Use this approach to better understand the underlying decision processes with mroe interpretability than "black box" methods, but more personalization and accuracy than standard logistic regression methods.

main_review: Note that I already reviewed a prior version of this paper that was submitted to ICML. I overall rated the manuscript favorably then and will mostly reiterate my assessment here:

Good description of a high-stakes but variable clinical decision problem and a policy modeling framework to better understand sources of that variability.

Since the goal is not achieving a particular measurable outcome (e.g., prediction accuracy), not as clear how to judge whether the interpretability of suggestions here offers decision makers much more value than they could get from logistic regression models.

Argues that logistic regression is too general a model vs. a personalized one, but couldn't local site/physician features and interaction terms be added to a basic logistic regression model to achieve similar benefits?

Decent survey of related studies understanding clinical decision making factors, though with varying purposes.
In addition to review of different technical approaches, useful review of the live decision making process. I was aware of the MELD prioritization scheme for liver transplant offers, but not aware of the variability in organ offer acceptance rates that the authors focus their modeling on here.

Well laid out description of decision policy modeling framework, and reference to how that fits in the real-world process.

limitations_and_societal_impact: Hard to objectively compare the proposed method vs. existing methods.
Better understanding of decision drivers and risk assessment could significantly influence organ transplant practices. Misinterpreting correlations found through these approaches as causal could negatively impact transplant decisions without clear guidance on interpretation of findings.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a ML procedure (or framework) for understanding donor organ acceptance decisions for patients on an organ waiting list. Their framework uses NNs for both learning a latent space (encoder/decoder), and for identifying policy criteria for the patient decision function. They include a variety of constraints (consistency, partial monotonicity, model complexity) that are salient to a real organ allocation setting. They lightly discuss their results on real data from UNOS.
main_review: This paper is very well written and mostly easy to follow. Their main contributions are relatively novel, and they do improve on prior work in some meaningful ways. In particular: Their model is useful for illustrating how individual-level (and hospital-level) organ accept/reject decisions are made, and which latent features are most salient to these decisions. This extends prior work with "black-box" models, which provide no intuition as to the decision making process. This interpretability comes at very little cost in accuracy, since their model performs nearly as well as the best "black box" model they compare against.


Some general comments & questions follow:

- The latter part of the pipeline is somewhat unclear: what role does the MoE module play, and why use an MoE rather than a single network? (It seems that this module learns the transform \mathcal{T} and weight vector(?) \rho, however this is not clear.)

- In general, there are several design decisions that seem arbitrary, yet are very important to the model design. I think it would substantially improve the readability and impact of this paper if you clarified these design choices when introducing the model. Some of these choices include:

    - Why choose the Boltzman distribution / logit model in (1), as opposed to some other model? (This model is similar or identical to a multinomial logit, which would be natural appropriate if a patient needs to choose between multiple organs. But in your case, a patient is choosing whether or not to accept a single organ, or wait for another option. So what is the intuition behind this model?)

    - Why use the MoE module as opposed to something else?

    - Why is consistency an important criteria? (Eq. (2)). There are many reasons my preferences for donor organs would not be smooth -- for example, I may be likely to reject an organ from a donor with a particular antibody that I am incompatible with, even though this donor may be a good fit in all other aspects. (So this organ is close to a desirable organ in feature space, but very far from the desirable organ in decision space.)

To be clear, I'm not saying that these choices are bad. I just think you need to justify them.

- Some comments on the application & related literature. Overall, since this is mainly an applications paper, I think there needs to be more discussion on related application-focused literature as well as more explanation of the experiments.

    - There is a large body of literature on this topic, and I suggest you engage with it more in your paper. For example, this paper (https://aasldpubs.onlinelibrary.wiley.com/doi/full/10.1002/lt.24113) has a very similar goal to your paper, with a more applied focus. They discuss two main criteria of interest to organ patients: graft failure and disease transmission. It is likely that these factors also play a major role in the decisions you are aiming to model in your paper. If you do not use this domain knowledge, you're just making the task of predicting donor behavior harder. Even if you do not use this domain knowledge, you should certainly cite it.

    - The organ accept/reject decisions are strongly dependent on both the waitlist prioritization (who has the highest priority to receive an organ) and the compatibility criteria (when is an organ compatible with a patient). At very least, these policies will make certain features in your dataset very dependent. For example, certain patient types will never be offered certain types of donor organ. Please discuss how these policies impact your analysis.


limitations_and_societal_impact: I'm a bit concerned about the interpretation and use of the methods proposed in this paper, and I think more discussion is needed. 
- What is the intended use of this model? 
- What is the interpretation of your results? (Especially the normalized policy signature in Fig. 3 and the patient clusters.)
- Is this analysis meant to influence future decisions? And if so, how does this differ from state-of-the-art decision tools for organ allocation? 
    - If so, what are the risks for using this method?

These are issues because of the sensitive and applied nature of this paper, where the prediction task at hand is literally tied to human life and death.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a policy learning strategy that is patient-driven and interpretable. They focus on the case of organ transplant, not predicting matching but rather whether an organ will be accepted by a clinician for their patient. The authors show that their method perform on par with other approaches, with the added benefit that the decision space is interpretable and can be investigated.
main_review: Originality
---------------
This is an interesting combination of different ML components to provide personalized policies. Technically, those components are not novel, but I find their combination interesting. The overall idea is quite similar (in philosophy) to that of Koh et al., 2020 (https://arxiv.org/abs/2007.04612). This parallel could be made explicit in the text.

Quality
----------
The experiments are mostly described in the supplementary materials. I have a few questions regarding some of the data and modeling choices:
- "For the experiments in this paper, we select rejected organ offers of twelve cases, covering 87.7% of all rejected offers". It feels like the data is selected based on parameters that are well represented in the criteria space. While I understand the reasons for not including all offers, this makes me question the broader applicability of the method. Can the authors comment on a potential circularity here, that might reduce the application to other settings or new patients? 
- In the same way, the features for donors and patients are selected based on the criteria.
- The criteria space was selected based on domain knowledge. Do you mind expanding on who was consulted to provide those criteria? Was this set defined a priori and then used, or were multiple sets tried? If the latter, was there an effect on performance of the selected set?
- In the selection of hyper-parameters, the authors mention that they impose models to have similar learning capacities. This seems like an unnecessary constraint, especially given the diversity of approaches considered. Have the authors tried to optimize each technique, and maybe then compared the required numbers of parameters or CPU/GPU resources? While I understand the reasoning, a more fair comparison for overall usability would optimize each technique "independently" of the others, and then other parameters (such as computational expenses) can be taken into account in the comparison.
- Can the authors comment on the lower performance for center 6820?
- Supplementary Table 2 includes interesting results, showing that LASSO can perform similarly to the selected method when using (X,O) as input. Can the authors comment on their choice of including results in the main vs supplementary table?


Clarity
---------
The paper is overall well written. I would encourage the authors to revise some of the phrasings, that I found a bit convoluted. In general, simpler sentences would probably help the reader.
I also found that many of the experimental details were missing in the main text. I understand that these are part of the Supplementary Materials, but splitting the main text from the supplemental is a critical part of the writing, that I believe could be improved here (e.g. remove or make a bit more compact some of the intro and related works, bring in data descriptors, split and training/testing strategies at the high-level).
Another main source of confusion for me was that the space of criteria was not directly described as manually crafted based on domain knowledge. This seems to be critical for the interpretability of the method: a limited set of criteria, that are human-understandable. This information should be available early in the text, and discussed for its limitations.

Significance
-----------------
The authors mention that their work could help understand clinical decisions, and understand variations across centers. This seems promising but I was wondering whether those were desired by the clinical community? I also missed examples of other problems that could benefit from the method.
limitations_and_societal_impact: The authors provide a brief conclusion in the main text, and a short paragraph on limitations in the supplementary. I believe that the work is making a few assumptions and I would encourage the authors to explicitely discuss these in the main text. I have identified the following limitations (in addition to previous comments):
- The authors do not comment on a validation with clinicians as to whether the proposed approach would be useful for them to understand the decision making process.
- There is no validation that the proposed interpretability is useful. While SHAP and LIME are shown to provide results that are not consistent (which is not surprising given that these methods are not robust to perturbations), it is unclear whether clinicians would benefit from interpretability (in the sense that it helps them in achieving their goal), or that this technique in particular is answering their needs in terms of interpretability.
- Is the method able to handle missing data? How about other methods?
- Can the authors comment on some of the assumptions made, including linearity in the criteria space?
- The clustering and the investigation per center are interesting and convincing. However, there is little validation of the results, which are susceptible to confirmation bias. Can the authors comment or add as a limitation?
review_title: null

Meta-review: Reviewers' scores and sentiments were remarkably consistent and positive toward this work.  The approach taken to address a high-stakes problem seems appropriate, and reviewers appreciated both that approach and its validation.  Still, reviewers -- especially Q9SD and oN9u -- did have outstanding questions regarding details about the methodology and about the experimental validation, and I would encourage the authors to update their work to reflect answers to those questions (many of which were given in the rebuttal).
Decision: Accept (Poster)