MedDec: A Dataset for Extracting Medical Decisions from Discharge Summaries (2024)

Abstract: Medical decisions directly impact individuals’ health and well-being. Extracting decision spans from clinical notes plays a crucial role in understanding medical decision-making processes. In this paper, we develop a new dataset called “MedDec,” which contains clinical notes of eleven different phenotypes (diseases) annotated by ten types of medical decisions. We introduce the task of medical decision extraction, aiming to jointly extract and classify different types of medical decisions within clinical notes. We provide a comprehensive analysis of the dataset, develop a span detection model as a baseline for this task, evaluate recent span detection approaches, and employ a few metrics to measure the complexity of data samples. Our findings shed light on the complexities inherent in clinical decision extraction and enable future work in this area of research. The dataset and code are available through https://github.com/CLU-UML/MedDec.

heading: 1 Introduction
text: Clinical notes contain rich information about medical decision-making. Such notes document patient conditions, medications, laboratory and diagnostic results, assessments and plans, prognoses, and follow-up information, among other crucial data points. However, automatic knowledge extraction from clinical notes has been challenged by imprecise clinical descriptions, heterogeneous data, and the need for data annotation. In particular, although there exist comprehensive and manually verified taxonomies of medical decisions (Braddock et al., 1997; Ofstad et al., 2018), and successful information extraction techniques in medical (Mullenbach et al., 2021; Miwa and Sasaki, 2014; Islamaj Doğan et al., 2011; He et al., 2017; Uzuner et al., 2010), and general (Dethlefs et al., 2012; Goodman, 2002; Frampton et al., 2009; Bui and Peters, 2010; Hsueh and Moore, 2007) domains,
there is currently no dataset for extracting (i.e. detecting and classifying) medical decisions in clinical narratives. A medical decision is defined as a particular course of clinically relevant actions and/or a statement concerning the assessment of a patient’s health as defined in the Decision Identification and Classification Taxonomy for Use in Medicine (DICTUM) (Ofstad et al., 2016).
Automatic extraction of medical decisions from clinical notes has the potential to transform clinical practice. It can inform the development of evidence-based decision-making guidelines and stewardship programs, identify potential deviations from best decision-making practices, and determine potential risks to patient based on prior medical decisions and their outcomes. In addition, beyond clinical applications, understanding clinical decision patterns can inform health policy development and refinement, especially when evaluating the impact of particular interventions or policies.
This paper develops the first expert-annotated dataset for medical decision extraction and classification within discharge summaries (MedDec). It is developed using patient data sourced from the Medical Information Mart for Intensive Care (MIMIC-
III) (Pollard and Johnson III, 2016), which is a publicly available dataset of de-identified clinical data of patients who were treated in intensive care units (ICUs). MedDec contains annotated decisions in 451 discharge summaries, covering more than 54k sentences and containing diverse patient groups based on sex, race, and English proficiency. In addition, 187 out of the 451 discharge summaries were previously classified into eleven phenotypic (main disease) categories through manual annotation by Gehrmann et al. (2018). We extend the dataset as follows: all medical decisions in the discharge summaries are annotated by domain experts according to the Decision Identification and Classification Taxonomy for Use in Medicine (DICTUM) (Ofstad et al., 2016). DICTUM covers ten (10) medical decision categories listed in Table 1; we add the residual category “None” for spans of texts that do not contain any medical decision. Two expert annotators independently label all text spans of medical decisions in each discharge summary according to the DICTUM guidelines. Disagreements between the annotators are adjudicated by a senior third annotator. Figure 1 shows an excerpt from a de-identified discharge summary annotated with several categories of medical decisions.
In addition, we introduce the new task of clinical decision extraction, which involves identifying and classifying spans of medical decisions within relatively long clinical notes. This focused information extraction task contributes to the advancement of bioNLP techniques and has the potential to improve healthcare. We develop several baselines including span detection and named entity recognition models, and evaluate and analyze their performance on MedDec. In addition, we introduce a framework for medical decision extraction to set a baseline for future research.
The contributions of this paper are:
• to the best of our knowledge, MedDec is the first expert-annotated dataset for research on medical decision extraction and classification from clinical notes;
• we provide a comprehensive analysis of the dataset, including distribution reports for protected variables, including sex, race, and English proficiency; and
• we evaluate existing span detection approaches on MedDec, and develop a baseline model to lay the foundation for future research in this area.

heading: 2 MedDec

heading: 2.1 Taxonomy of Medical Decisions
text: Table 1 provides descriptions of different types of medical decisions in clinical notes, adapted from DICTUM (Ofstad et al., 2016). The “Contact related" category involves decisions related to patient admissions, discharges, follow-ups, and referrals within the healthcare system. “Gathering information" decisions pertain to acquiring data from sources other than patient interviews or charts, such as ordering tests. “Defining problem" decisions involve complex assessments that define medical issues, including diagnoses, etiological inferences, and prognostic judgments. “Treatment goal" decisions specify treatment objectives beyond general advice. “Drug" decisions pertain to initiation, alteration, cessation, or maintenance of drug regimens. “Therapeutic procedure" decisions involve interventions or therapeutic procedure management. “Evaluating test result" decisions are those that evaluate clinical findings and test outcomes. “Deferment" decisions delay or reject medical decision-making, often due to insufficient information or the need to await test results. “Advice and precaution" decisions involve providing advice or precautions to patients and transferring responsibility for actions to them. Finally, “Legal/insurance-related" decisions deal with medical matters related to legal regulations or financial arrangements.
These categories provide a comprehensive grouping of medical decisions in clinical notes. They can be used for systematic classification and structured analysis of medical decisions, and for understanding the complex processes involved in clinical decision making.

heading: 2.2 Data Collection
text: MedDec is created using patient data sourced from the MIMIC-III (Pollard and Johnson III, 2016). It contains annotated decisions in 451 discharge summaries, representing diverse patient groups based on sex, race, and English proficiency. All medical decisions in each discharge summary are annotated according to the 10 medical decision categories introduced in DICTUM (Ofstad et al., 2016).
The token-level inter-annotator agreement, measured by Cohen’s Kappa between the first two annotators is substantial, k = 0.74, indicating that it is fairly easy for domain experts to identify medical decisions in discharge summaries. A similar agreement level was reported in DICTUM (Ofstad et al.,
2016). We note that token-level agreement provides a lower bound for true inter-annotator agreement as it may sometimes underestimate agreement. This occurs, for instance, when minor variations such as the inclusion or exclusion of less relevant tokens (e.g. stopwords) at the start or end of decision spans are considered as disagreements.

heading: 2.3 MedDec Novelty
text: The novelty of MedDec is in its focused annotation of medical decisions based on an expert-verified and comprehensive taxonomy of medical decisions, its diversity across sex, race, and language proficiency patient groups and phenotypes (diseases), and its potential to drive advancements in both bioNLP research and clinical decision-making. To our knowledge, MedDec is the first dataset specifically developed for extracting medical decisions in clinical notes. This diversity in MedDec enables investigations on potential disparities in medical decisions across the above-mentioned protected variables, which can provide insights for addressing healthcare inequities. These features make MedDec an asset in bioNLP.

heading: 2.4 MedDec Statistics
text: Table 2 reports the percentage of decision spans for each decision category and each protected variable
in MedDec. The total counts of decision spans are reported in the last row. Medical decisions related to defining problems, drugs, evaluation, and therapeutic procedures are categories with the highest prevalence, while legal, deferment, treatment goal, gathering information, advice, and contact have considerably lower prevalence. In MedDec, 42.6% of summaries are related to Female patients, 75.9% belong to white patients (of patients with known race), 9.7% to African American, and 85.2% to patients (with known language proficiency) identified as proficient in the English language.
In addition, Table 3 shows the distribution of patients across phenotypic (disease) categories. Patients with psychiatric disorders (including schizophrenia, bipolar, and anxiety disorders), depression, chronic neurologic dystrophies, and chronic pain are more prominently represented in MedDec. Conversely, patients associated with substance abuse, lung conditions, cancer, and obesity are less prevalent in the dataset.
The discharge summaries in MedDec contain 1.4M tokens, with 879K tokens forming part of a span, while 37K tokens belong to more than one span (accounting for 4.2% of labeled tokens). However, the majority of overlaps are minor, where a token marks the end of a span and the start of another.

heading: 3 Learning to Extract Medical Decisions
text: We present an overview of our approach and describe its two key components in subsequent sections. Figure 2 shows our approach for extracting and classifying medical decisions. First, a long note is chunked into segments of acceptable length to the model. Each segment is fed into the model to generate hidden representations and token classification probabilities (this step can be batched for efficiency). Then, the resulting labeled text sequences are concatenated to obtain classification results for the entire clinical note. Finally, we postprocess the results to convert them into spans, defined by a start position, end position, and category.

heading: 3.1 Sequence Labeling Framework
text: We develop a multi-class sequence labeling approach that fine-tunes a pre-trained model for span detection. The data consists of a sequence of n tokens t = {t1, . . . , tn} and token labels y = {y1, . . . , yn}, and each label indicates a set of k+1
categories indicating k decision types and the none category, yi ∈ {C1, .., Ck, O}. Practically, the labels follow the BIO (beginning-inside-outside) token labeling scheme (Ramshaw and Marcus, 1995).
We use a pre-trained bidirectional transformer encoder f to encode the tokens and generate ddimensional hidden states. Formally, the latent representation of the i-th token is computed as:
hi = f(t)i, (1)
where hi ∈ Rd. We then employ a fully-connected layer g(.) on top of the hidden representation that maps each hidden state to obtain the logits across all classes of decision categories:
zik = g(hi)k. (2)
To realize multi-class classification, the logits are fed into a softmax function, where the class with the maximum predicted probability is considered the predicted label. This approach does not take
into account the categorical similarity of neighboring tokens. Previous studies indicate that token classification remains competitive as the most effective span detection and named entity recognition method (Aarsen, 2023; Jurkiewicz et al., 2020; Chhablani et al., 2021; Gu et al., 2022).
There are several alternative methods for span detection, such as detecting the start and end positions, classifying a pair of tokens to check if they constitute a span’s boundary, detecting each span category using a separate query process (Devlin et al., 2019; Shen et al., 2022), and conditional random fields (Panchendrarajan and Amaresan, 2018) that compute the maximum probable span assignment from logits.1

heading: 3.2 Sequence Chunking
text: Clinical notes are typically thousands of tokens long, and transformer models are computationally
1Our experiments show that CRF does not improve performance; so, we do not include it in our architecture.
restricted and can typically process a maximum of 512 tokens at once. To overcome this challenge, we develop a data sampling function that samples segments of 512 tokens or fewer from random starting points at each training iteration. Therefore, a unique set of text segments is seen at each iteration. This sampling method acts as a data augmentation method by sampling different segments of the same clinical note with different start and end positions, different sentence compositions, and different lengths. At inference time, the input is chunked into segments of 512 tokens with no overlap, each segment is tagged, and the results are concatenated.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Models We evaluate the following models:
• Binder (Zhang et al., 2023): employs encoders for tokens and token types, optimizes a contrastive objective, with a dynamic threshold loss for negative sampling.
• PIQN (Shen et al., 2022): uses NER pointer mechanism (Yang and Tu, 2022) for span boundary detection and an entity classifier for classification. A dynamic label assignment objective is proposed to assign gold labels to instance queries. It dynamically learns query semantics for instance queries and extracts all types of entities simultaneously.
• DyLex (Wang et al., 2021): a sequence labelingbased approach that incorporates lexical knowledge with an efficient matching algorithm to generate word-agnostic tag embeddings for NER.
• Instance-based NER (Ouchi et al., 2020): formulates NER as instance-based learning, where model assigns labels based on a nearestneighbor approach.
The following BERT-based models employ the token classification approach described in Devlin et al. (2019). All experiments use the base-size version of the models.
• DeBERTa v3 (He et al., 2022): uses advanced training strategies, primarily disentangled attention and mask decoder.
• ALBERT (Lan et al., 2020): implements shared weights across layers, leading to a greatly reduced memory footprint.
• ELECTRA (Clark et al., 2020): a bidirectional encoder employing a new pre-training objective. It learns to discriminate between real and fake (but plausible) input tokens.
• RoBERTa (Liu et al., 2019): a BERT-based model with an improved training objective, hyperparameters, and increased data.
• BioClinicalBERT (Alsentzer et al., 2019): a BERT-based model pre-trained on PubMed abstracts and MIMIC-III clinical notes.
Evaluation We use the standard evaluation metrics for NER, span exact match, and token accuracy.2 The correctness of a span is determined by an exact match (both boundaries and category). We report results in terms of micro-F1 score. Tokenlevel accuracy is a more flexible metric, allowing partial overlap with the true spans.
Difficulty Score We use span length and number of UMLS medical concepts to divide medical decisions into into three difficulty levels, namely Easy, Medium, and Hard. Example of a span with a single medical concept and low cognitive load: “you will continue taking two antibiotics.” Example of a span with a high number of medical concepts (underlined) and high cognitive load: “CT abdomen with intravenous contrast: The heart size is at the upper limits of normal. Dense coronary calcifications are identified. In the lung bases, there is bibasilar atelectasis. There are also chronic pleural inflammatory changes including fat deposition and fibrotic changes, left greater than right. Bilateral small pleural effusions are also identified, right greater than left. No focal pulmonary nodules or opacities are identified
2https://huggingface.co/spaces/ evaluate-metric/seqeval
in the lung bases.” These difficulty scores provide insights into the difficulty of learning medical decisions and can also inform curriculum discovery (Elgaar and Amiri, 2023).

heading: 4.2 Main Results
text: We compare recent span detection and classification models using our training framework.3 Table 4 shows the results where we observe that RoBERTa achieves the best performance with a 34.8 F1 score, followed by ELECTRA and BioClinicalBERT.
Span and token accuracy are not perfectly correlated. Although ALBERT performs lower than BINDER in span exact-match, it achieves a higher token accuracy, meaning that it is more effective in partial span detection. BINDER, PIQN, and DyLex achieve higher span-level but lower token-level accuracy than ALBERT, stemming from their design that emphasizes span exact matching.
Our approach focuses on training a model to reliably label segments in clinical notes, irrespective of their boundaries. This strategy results in improved span boundary prediction at inference time.
3We note that the performance of current models without our training framework is significantly lower.

heading: 4.2.1 Span Complexity
text: We suggest two factors that offer insights into the difficulty of learning spans of medical decisions: (a): the number of medical concepts in each span according to the UMLS ontology, and (b) the length of spans as a heuristic metric; the longer spans may contain more diverse semantic content and can involve more complex sentence structures, which requires the model to maintain more contextual information. We divide the spans into three groups based on their complexity: low, medium, and high.
The results show significant performance disparity across three complexity levels and two metrics. In particular, performance varies considerably based on span length, where predictions are most accurate on shorter spans and least accurate on longer ones. The performance reaches up to 17.7 on easy samples and as low as 10.0 on hard samples. However, we do not observe such a decreasing trend in performance using the number of UMLS concepts, see Figure 3. Span detection performance drops when there is either a very low or very high number of medical concepts in a span. A span with a low number of medical concepts is likely ambiguous and hard to interpret and classify, while a span with a large number of medical concepts is complex and hard to understand. Therefore, sentence length is a better measure of sample complexity, perhaps due to the use of broader contextual information.
We note that the difficulty analysis is the average of the following three models, ELECTRA (Clark et al., 2020), RoBERTa (Liu et al., 2019), and BioClinicalBERT (Alsentzer et al., 2019). Comparing model performance across varying levels of sample complexity provides a better understanding of each model’s strengths and weaknesses.

heading: 4.2.2 Insights Across Patient Phenotypes
text: Figure 4 shows a significant disparity in the performance of span detection at the phenotype level, with a standard deviation of 7.5. The highest F1 score is achieved for “Heart Disease” (44.7), while the lowest is for “Substance Abuse” (19.1). Across 11 phenotypes, performance exceeds the average performance (from Table 4) for six phenotypes but falls under the average for the remaining five phenotypes: “Depression,” “Neurological Dystrophies,” “Substance Abuse,” “Fibromyalgia” and “Alcohol Abuse”. The variability in the performance of decision extraction across different patient groups identified by different phenotypes highlights the challenges associated with the practical use of the system. We attribute this variance in performance to two potential factors: (a): MedDec contains more training data for the high-performing phenotypes and less training data for the low-performing phenotypes, (b): the intrinsic characteristics of phenotypes affect their difficulty in learning. For example, “Heart Disease” is identifiable through clear clinical markers like abnormal ECG findings or chest pain. Conversely, “Fibromyalgia” is a more complex condition due to its complex nature and subjective symptoms like widespread pain and fatigue. The subjective nature of these symptoms and their overlap with other conditions make it challenging to precisely classify “Fibromyalgia" cases.

heading: 4.2.3 Generalizability to Unseen Phenotypes
text: Results in Figure 4 show a performance drop for 7/11 phenotypes when they are not present in the training data. Conversely, Substance Abuse, Fibromyalgia, Depression, and Neurological do not suffer from being unseen, as perhaps they are sufficiently informed by transferred knowledge from available phenotypes in the training data. Heart
disease and Psychiatric disorders are among the most affected phenotypes, perhaps due to specific domain knowledge related to their decisions that differentiate them from others.

heading: 4.2.4 IFT for Span Extraction
text: Large language models (LLMs) have shown effectiveness in performing a wide variety of tasks through instruction-tuning (IFT) (Zhao et al., 2023; Li et al., 2024; Tran et al., 2024). In tasks such as medical question answering, recent works have shown that LLMs show comparable performance to extensively fine-tuned models (Nori et al., 2023a; Singhal et al., 2023; Thirunavukarasu et al., 2023) using domain-specific prompting methods, such as the retrieval of relevant medical queries to serve as demonstrations (Nori et al., 2023b).
We evaluate LLama-3-8B-Instruct (AI@Meta, 2024) on 10 discharge summaries. We prompt the LLM to extract decision spans for each decision category separately, prompting it ten times for each clinical note. We experiment with the zero-shot and one-shot settings both using the following prompt:
[[[System]]] Extract all sub-strings from the following clinical note that contains medical decisions within the specified category. Print each sub-string in a new line.

heading: If no such sub-string exists, output "None". [Clinical Note]: {Discharge summary here}
text: # IF: one-shot setting [[[User]]] [Category]: {Decision category here}
[[[Assistant]]] {Demonstrations} # End IF
[[[User]]] [Category]: {Decision category here}
[[[Assistant]]] {Response}
In the one-shot setting, we present as demonstrations all decisions of a single category other than the one being asked for. The demonstration category is selected as the category with the most number of decisions in the clinical note. The results are shown in Table 5. The LLM returns the extracted spans without token-level annotations, therefore it is not possible to calculate token-level accuracy. We compute the performance of span exact match and fuzzy match F1-score. The span fuzzy match is a substitute for token-level accuracy used in Table 4, as the span may be partially detected by the LLM but not be accounted for by the exact match score. To compute the fuzzy span match, we check if either of the extracted span and the true span are a substring of the other, and that they differ by no more than 10 words. These preliminary results show that decision extraction might be challenging for LLMs compared to fine-tuned models.
The low performance of the IFT models can be attributed to the lower efficacy of LLMs in processing long contexts (An et al., 2023). Moreover, the output of LLMs is in free form, which can result in correct responses that do not precisely match the content of medical decisions in notes. For example, a documented decision can be “the patient has high blood pressure,” whereas the generated decision can be “the patient is experiencing elevated blood pressure.” While semantically correct and relevant, such responses make accurate evaluation of LLM responses a challenging task.

heading: 5 Related Work

heading: 5.1 BioNLP Datasets
text: Nye et al. (2018) developed a dataset of 5K abstracts annotated with {population, interventions, compared, outcomes} (PICO), to inform personalize patient care. Lehman et al. (2019) developed a dataset of intervention, comparator, and outcome labels of more than 10K randomized controlled trial articles. Patel et al. (2018) developed the clinical entity recognition (CER) corpus, which consists of 5.1K clinical notes annotated by experts with entities such as anatomical structures, body functions, lab devices and medical problems. , and findings. The extracted concepts correspond to a selected group of UMLS semantic types. CLIP (Mullenbach et al., 2021) is a dataset of 718 discharge summaries from MIMIC-III, annotated with seven types of action items: {Appointment, Lab, Procedure, Medication, Imaging, Patient Instructions,
Other} at sentence level. covering more than 107K sentences. Stupp et al. (2022) introduced a dataset of 579 admission and progress notes from MIMICIII, annotated with diseases, assessments, and categories of action items. PHEE (Sun et al., 2022) consists of 5K events from case reports and literature, annotated for pharmacovigilance {Subject, Drug, Effect} for drug safety monitoring. Recently, Cheng et al. (2023) developed MDACE, a dataset of clinical notes annotated with ICD codes and their rationales for computer-assisted coding.

heading: 5.2 NER and Span Detection
text: Existing NER and span detection approaches can be divided into sequence labeling-based (tagging) approaches and span-based approaches. Sequence labelling approaches (Aarsen, 2023; Tjong Kim Sang, 2002; Gu et al., 2022) classify every token in the sequence to their corresponding class(es). This formulation is challenged by nested entities. Span-based approaches (Sohrab and Miwa, 2018; Luan et al., 2019; Zheng et al., 2019; Tan et al., 2020; Shen et al., 2021), however, identify and classify spans. First, the spans are either extracted through enumeration or boundary identification and then classified to their corresponding classes.
Du et al. (2019) developed the relational spanattribute tagging (R-SAT) model for extracting clinical entities, their properties, and relations. It employed a method similar to ours, however, the tasks are different as we tackle medical decisions. Ouchi et al. (2020) formulated the NER task as an instance-based learning task, where the NER model was trained to learn the similarity between spans of the same class. DyLex (Wang et al., 2021) retrieved lexicon knowledge for input sequence, applied a denoising module to remove noisy matches, and encoded and fused the lexicon knowledge into the sequence embeddings with column-wise attention for NER.
Abaho et al. (2021) jointly detected and classified spans of health outcomes in clinical notes. Most prior methods decoupled the detection and classification phases. Sent2Span (Liu et al., 2021) was developed for the extraction of PICO information from clinical trial reports. It is designed to work with non-expert sentence-level annotations on the presence of PICO information, without the need of expert span-level annotations and is able to achieve higher recall than comparable methods. PIQN (Shen et al., 2022) developed entity pointer for localization and entity classifier for classifica-
tion. A dynamic label assignment objective was proposed to extract different types of entities simultaneously.
Recently, Zhang et al. (2023) proposed Binder, which optimized two encoders for NER, one for tokens and one for token types. For each span associated with a class, Binder sampled negative spans based on their loss, and optimized model parameters using a contrastive learning objective. Mirror (Zhu et al., 2023) was an information extraction framework based on graph decoding, where entities were nodes in the graph and the relations of interest were edges. Mirror allowed for extracting all entities and relations in a single step. DICE (Ma et al., 2023) adapted sequenceto-sequence models for structured event extraction from clinical text, using PubMed documents in MACCROBAT dataset (Caufield et al., 2019). Raza and Schwartz (2023) introduced a model consisting of BioBERT (Lee et al., 2019), and BiLSTM (Huang et al., 2015) and Conditional Random Field (CRF) (Lafferty et al., 2001) layers to extract clinical (diseases, conditions, symptoms, and drugs) and non-clinical (social factors) entities from clinical notes.

heading: 6 Conclusion
text: We developed MedDec for the extraction and classification of ten types of documented medical decisions in discharge summaries of eleven different phenotypes (diseases). We demonstrate several baseline models to tackle this task. Through extensive experiments and analysis, we find that the task is challenging, the performance of the bestperforming model significantly varies across phenotypes and the spectrum of sample complexity. The dataset can be useful in studying population statistics, biases in medical treatment, analysis of medical decisions for different phenotypes, and understanding medical decision-making processes.
Limitations
The present work has several limitations, which form the basis of our future work: the distribution of the ten classes of medical decisions within the dataset is considerably imbalanced. Table 2 highlights these data imbalances across protected variables. Class imbalance may lead to biases in the model training process and affect the model’s ability to accurately predict less represented classes. Addressing these data imbalances can prevent com-
putational models from learning and perpetuating such biases in the data. In addition, we note that, these imbalances reflect the challenges of working with real-world data and can inform future research in healthcare equity and the development of systems that perform well across all patient groups. The models have been applied to the notes in MIMIC-III dataset, and the extent of their generalizability to other datasets has not been evaluated. While our best classifier is performant, it may fail to identify and classify certain medical decisions, such as those pertinent to deferment. This limitation could be partly due to the effect of longer decisions, which can inversely affect the model’s performance due to the potentially higher linguistic complexity of longer texts (see relevant results in Figure 3). Finally, while discharge summaries contain rich information about patient care, it’s important to acknowledge their limitations in fully capturing the breadth of medical decisions made during a patient’s hospital stay. Nevertheless, common medical decision-making patterns and clinical reasoning processes are expected to make models trained on these summaries generalize to other types of clinical documents.
Ethic and Broader Impact Statements
This project adheres to ethical considerations and safeguards to ensure the responsible and ethical handling of medical data and its implications. All results have been presented in aggregate, and we have made and will make every effort to protect human subject information and minimize the potential risk of loss of patient privacy and confidentiality (all authors with access to the data have successfully completed a training program in the protection of human subjects and privacy protection). In addition, our work is transformational in nature, and its broader impacts are first and foremost the potential to improve the well-being of individual patients in the society and support clinicians in their medical decision-making efforts.

heading: A Examples of Model Predictions
text: Table 6 shows examples where the model partially or fully fails to capture the underlying medical decision from the clinical note.

heading: Future Work
text: Limitations
The present work has several limitations, which form the basis of our future work: the distribution of the ten classes of medical decisions within the dataset is considerably imbalanced. Table 2 highlights these data imbalances across protected variables. Class imbalance may lead to biases in the model training process and affect the model’s ability to accurately predict less represented classes. Addressing these data imbalances can prevent com-
putational models from learning and perpetuating such biases in the data. In addition, we note that, these imbalances reflect the challenges of working with real-world data and can inform future research in healthcare equity and the development of systems that perform well across all patient groups. The models have been applied to the notes in MIMIC-III dataset, and the extent of their generalizability to other datasets has not been evaluated. While our best classifier is performant, it may fail to identify and classify certain medical decisions, such as those pertinent to deferment. This limitation could be partly due to the effect of longer decisions, which can inversely affect the model’s performance due to the potentially higher linguistic complexity of longer texts (see relevant results in Figure 3). Finally, while discharge summaries contain rich information about patient care, it’s important to acknowledge their limitations in fully capturing the breadth of medical decisions made during a patient’s hospital stay. Nevertheless, common medical decision-making patterns and clinical reasoning processes are expected to make models trained on these summaries generalize to other types of clinical documents