Enhancing Risk Aware Decision in Healthcare through Probabilistic Modeling of Uncertainty Rahul Vishwakarma∗ rahuldeo.vishwakarma-sa@csulb.edu California State University Long Beach, California, USA Jinha Hwang∗ jinha.hwang01@student.csulb.edu California State University Long Beach, California, USA Benyamin Ahmadnia benyamin.ahmadnia@csulb.edu California State University Long Beach, California, USA ABSTRACT In domains with high stakes, like healthcare and medicine, trust- worthy and robust decision-making is crucial due to the potential risks associated with misclassification. However, many traditional machine learning classifiers lack calibrated predictions, and reliable uncertainty estimates for new unseen data. This paper addresses the challenge of uncertainty quantification in text classification in healthcare and proposes a three-fold approach to support robust and trustworthy decision-making by medical practitioners. To evaluate our solution, we implement it on a multi-label medical transcrip- tion dataset from Kaggle. Our study demonstrates three significant results: the ability of our model to reject uncertain predictions by providing a null set, the provision of set predictions with guaran- teed coverage for further investigation, and the prioritization of decision-making based on confidence levels of predictions with the same label. Additionally, we tackle the issue of imbalanced datasets in the medical domain by employing the Mondrian Conformal Pre- dictor with a Naïve Bayes classifier. Our findings are expected to enhance the risk-aware decision-making process in the medical field. CCS CONCEPTS • Computing methodologies →Probabilistic reasoning. KEYWORDS uncertainty, conformal inference, risk-aware decision 1 INTRODUCTION Machine learning has emerged as a powerful tool in the medical domain, providing healthcare professionals with a means to make more informed decisions and improve patient outcomes [1]. Medical transcript analysis, in particular, holds the potential to assist in diagnostics [2], treatment planning [3], and patient monitoring [4]. However, traditional natural language processing techniques have limitations that can result in inaccuracies in classification [5, 6]. Natural Language Processing (NLP) techniques, specifically text classification, can be leveraged in various industry applications to overcome these limitations. Several methods have been proposed to achieve this goal. [3, 7] However, they are often under-utilized due to the criticality of the decisions involved and a lack of confidence in individual decisions. Currently, with text classification, we don’t have a mechanism to ‘tell’ the model to be more strict or lenient while making a decision. Hence, the accuracy of the model solely depends on the classifier. ∗Both authors contributed equally to this research. 1.1 Related Work Text classification has been widely explored in the field of NLP, and it has found applications in various domains such as finance [8], military [9], and medical [10, 11], among others. Most of the research in this field has focused on developing algorithms that can improve accuracy while keeping the computational cost low [12]. A few works [13] have used aleatory and epistemic uncertainty. Still, they do not quantify the prediction of each new prediction, while [14] focuses on the profound learning aspect of text classification. However, we still see a gap in the practical realization and the applicability of the metrics for confident decision-making for a text classification system. 2 NOVEL CONTRIBUTIONS The novelty of this work is that the proposed mechanism not only gives classification out, but two measures, i.e., confidence and cred- ibility, to tune those decisions based on the criticality and control the algorithmic decision-making. The main contributions of this paper are as follows: • Uncertainty quantification for each prediction for improved and robust decision-making for the medical domain with imbalance