Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper touches on a very interesting and important problem in the applied machine learning field, the trustworthiness of ML models, how to measure it, and its effect on user experience. The paper presents a very nice introduction to the topic and opens the problem in the context of out-of-distribution detection. Three different approaches for out-of-distribution detection are benchmarked on several datasets and ML methods. Furthermore, a confidence score is proposed to translate the out-of-distribution scores into more interpretable scores for the users. A user study is further conducted to evaluate the effect of using confidence scores on the trust level of human participants.
main_review: The paper discusses a significant problem in the field and the text is very clear. I have few concerns regarding the experimental setups and results:
1) It seems from the text that the user study was performed on random participants. In order to truly measure the effect of using the confidence score on the perceived trustworthiness, the experiments must be performed on expert participants (as true users of the ML model) with domain knowledge about the experimental data. 
2) It is not clear from the text, how the confidence score is computed.
3) The authors may also discuss the advantage of their proposed confidence score with commonly used metrics in the field such as uncertainty measures.

limitations_and_societal_impact: NA
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This work applies known out-of-distribution (OOD) detection techniques on various medical datasets and confirms that the OOD detection techniques indeed works. Specifically, the medical prediction models tend to make more incorrect predictions for samples that OOD detection algorithms deem OOD.
main_review: Pros
- This paper focuses on an important problem; the reliability of prediction models in healthcare. They specifically employ OOD detection methods to test if they can help increase the trustworthyness of prediction models.
- The authors conduct extensive experiments using medical datasets of multiple modalities (e.g. images, audio) and multiple conditions (e.g. skin cancer, lung sound) to provide reliable conclusion

Cons
- The focus of this work is to study whether the existing OOD detection methods can work in various medical datasets. Therefore there is limited methodological novelty, which is the main interest of the NeurIPS community.

Summary
- The work itself is valuable, but probably not in NeurIPS. This work would be much more appreciated in relevant venues just such as MLHC, CHIL or JAMIA, JMIR.

Post Rebuttal
- I appreciate the authors taking the time to describe their thoughts in detail. However, I do not belive that simply applying well-known OOD detection methods on publicly available datasets justifies an acceptance to this venue, as that is not the main interest of this community. Therefore I'll keep my initial rating. (The user study is an interesting aspect, and will be of huge relevance to the healthcare community)
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors provide a small scale study of out of distribution detection methods on three medical datasets and perform a survey of Turkers to see how these OODs impact trustworthiness and medical decision making. They find that Gram and Manhabolis based methods detect OOD data well and that these scores translate into increased trustworthiness and medical decision making in their survey.
main_review: ### Main comments:
I would like to commend the author's for taking the time to conduct a survey to see how their experimental results translate into user perception and behavior. A very rare thing for computational papers! 

In addition to my detail comments below, my primary concern about this paper stems from the scope of its evaluations. 3 datasets and 3 methods is good, but still modest by most standards. Furthermore, the "difficulty" of the OOD scenarios tested is unlikely to be reflective of real clinical scenarios. Realistic OOD in healthcare is partially captured by some of the examples, but a practitioner in this space will likely still have many questions about which if any OOD detection methods will work in practice. Much more subtle forms of distributional shift occur in practice and inclusion of these would strengthen the conclusions.

I also think that the author's should more formally define what they mean by OOD, i.e. are they testing covariate shift, label shift, or something else? Providing this additional specificity will help users know if the authors' results are applicable to their wrork.

### Detailed comments:

__Originality__: The work is original and includes both a small computational evaluation and user survey.

__Quality__:  The experimental sections are sound and the survey is as well done, if a bit small and artificial. It would be great to include the distribution of softmax probabilities on the OOD data. I am curious to know if the models are extremely confident on data that has nothing to do with their training task. 

- I do not believe _CONFIDENCE SCORE_ is ever formally define. Please include this definition.

__Clarity__: The work is clearly presented and I found text and experiments easy to follow.

__Significance__: The work looks at a very important topic that is understudied. Further experiments would increase the significance of this work.
limitations_and_societal_impact: Yes. This research involved human subjects and the authors state they received appropriate IRB approval for it.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work investigates the effect of out-of-distribution data on machine learning models on healthcare applications. The authors first display a decrease in model performance on near and far out-of-distribution samples for 3 health applications. They then assess the performance of 3 techniques for out-of-distribution detection and build a confidence score from this binary classifier. The confidence score is assessed in a user-study as a complementary piece of information to be presented to the user.
main_review: Originality
---------------
Out-of-distribution detection is not a novel problem, and technically the methods proposed are not new. The authors mention that their application to healthcare data makes this piece different, however I found no evidence that the considered healthcare applications affected the techniques or their use. Similarly, a decrease of model performance on ood data has been highlighted in several works (see works referring to 'robustness' to e.g. new diseases, time shifts or local perturbations). On the other hand, user studies are rarely conducted when claiming better ood detection methods. I would therefore encourage the authors to highlight the user-study more.

Quality
----------
The work is overall well executed and convicing. I have a couple of questions for the authors that would help me to better assess the impact, especially of the user study:
- In the Appendix, the questions refer to the user's trust in the system, and whether they would see a doctor if given this diagnosis. Is the assumption that there is a deterministic relationship between the model's prediction of e.g. malignancy with whether they would consult a doctor? Can a decision be factorized into 2 components: how the user/patient combines the model's prediction (+ confidence score) with their own priors to make a diagnosis themselves, and whether they decide to see a doctor based on this diagnosis? Or is it assumed that the model is only informational and that the user/patient task is to decide whether they see a doctor or not? Was there any investigation into which mental model of the classifier the user were referring to? If the former, it could have been interesting to add a question about what their diagnosis was. I understand this is probably a broader issue about the perception of a classifier as a diagnosis vs referral vs informational tool, but I was wondering whether the authors had any thoughts?
- On the increased user trust: Is the additional information increasing trust, independently of its quality? One way to test this would be to provide randomized confidence scores to images, to understand whether the added trust comes from the quality of the score and the correct understanding of the user, or whether it comes from a perception of increased information.

Clarity
---------
The paper is well written and easy to follow.

Significance
-----------------
I believe this is a nice application of existing techniques, well executed. I think the most interesting part is the user study, and I think it could be improved to have even more impact (see questions above).


References:
- Does Your Dermatology Classifier Know What It Doesn’t Know? Detecting the Long-Tail of Unseen Conditions https://arxiv.org/pdf/2104.03829.pdf
- Feature Robustness in Non-stationary Health Records, http://proceedings.mlr.press/v106/nestor19a/nestor19a.pdf
limitations_and_societal_impact: The authors acknowledge the 'theoretical' setup of the healthcare applications. This remains a limitation, as it is unclear whether predictions in themselves are useful in clinical settings. However, I think that the proposed framework has merits in terms of ood detection and transparency with the user, that can lead to more realistic studies.

I would encourage the authors to highlight a bit more the user study and discuss its limitations.
review_title: null

Meta-review: Thanks to the authors for an interesting and thorough study of OOD methods in the healthcare context. 

The reviewers all agreed that this work was well done and interesting, but the main concern is relevance for the NeurIPS community.  While I agree with reviewer Ku9s that the main interest of the this community is in ML methodology and theory, I do believe it is important to investigate real use cases of methods --- two of which were published recently in NeurIPS, and one in ICML.  MLHC, CHIL, JAMIA, and JMIR may also be appropriate venues, but the NeurIPS/ICML community developed these methods, and are an important audience for this kind of evaluation.  I tend to agree with the authors that their work falls squarely within the "Social Aspects of Machine Learning" (and "Applications"), so I am not as concerned about the relevance of this work for the ML community. 

Beyond the relevance concern, all reviewers were quite positive about the work itself.  Reviewer hRAn detailed some concerns about definitions (OOD, confidence score), which should be clarified in the main text.  Reviewer QZBK also describes an improvement to the user study, which should be incorporated into the discussion. 



Decision: Accept (Poster)