Robust Conformal Prediction for Infrequent Classes
Authors: Jens-Michalis Papaioannou
Sebastian Jäger
Alexei Figueroa
David Stutz
Betty van Aken
Keno Bressem
Wolfgang Nejdl
Felix Gers
Alexander Löser
Felix Biessmann
Venue: Accepted by TMLR ()

Abstract: Many real-world classification tasks involve datasets with large and imbalanced label spaces, making class-specific uncertainty quantification particularly challenging. Conformal Prediction (CP) provides a model-agnostic framework, which formally guarantees coverage, meaning that its prediction sets contain the true label with a user-defined probability (confidence level). However, standard class-conditional methods often fail when data is scarce for some classes. We propose a method that uses domain knowledge or label hierarchies to dynamically group semantically related classes to meet the desired coverage for a given confidence threshold. Our method maintains class-conditioned calibration when possible and provides group-conditioned guarantees where necessary. We evaluate our method on outcome diagnoses prediction, an important clinical task that does not only benefit from robust uncertainty estimation, but also presents a very imbalanced label distribution. We conduct experiments using three clinical datasets employing two medical taxonomies (ICD-10 and CCSR) and label spaces of varying sizes with up to more than 1,000 classes. Our results show that the proposed approach is able to successfully exploit the label hierarchy and consistently improves class-conditional coverage for infrequent diagnoses. By improving coverage for underrepresented classes, our method enhances the reliability and trustworthiness of predictive models. This improvement is especially valuable in clinical applications, where failure to detect rare but serious conditions can lead to harmful consequences.

Published in Transactions on Machine Learning Research (01/2026)
Robust Conformal Prediction for Infrequent Classes
Jens-Michalis Papaioannou
michalis. bat.iias..pppccoooooooebbbbebebo
Berlin University of Applied Sciences and Technology Leibniz University Hanmover
Sebastian Jäger*
scbestion.jeeger@@ht..bbeb..
Berlin University of Applied Sciences and Technology
Alexei Figueroa
Berlin University of Applied Sciences and Technology Leibniz University Hannover
David Stutz Max Planck Institate for Informaties
dstutz@mmpi-nff.mpp. de
Betty van Aken Grammarly
Keno Bressem
keno. bressem@tum,
Technical University of Munich, School of Medicine and Health
Wolfgang Nejdl
nejdl
Leihniz University Honnover
Felix Gers
FelieAleranderrGesS Gerserttt-ereeerreeeeeeee
Berlin University of Applied Sciences and Technology
Alexander Löser
aloeser@üht-herlin. de
Berlin University of Applied Sciences and Technology
Felix Biessmann
felúz. www.eeessmmmmnnttbbeebeeeeee
Berlin University of Applied Sciences and Technology
Einstein Center Digital Future
Reviewed on OpenReview: https: openreviee.. net/ forun? id=nJ4p8rh.31
"Equal contribution.
Work done while associated with Berlin University of Applied Sciences and Technology
Abstract
Many real-world classification tasks involve datasets with large and imbalanced label spaces, making class-specific uncertainty quantification particularly challenging. Conformal Pre- diction (CP) provides a model-agnostic framework, which formally guarantees coverage, meaning that its prediction sets contain the true label with user-defined probability (con- fidence level). However, standard class-conditional methods often fail when data is scarce for some classes. We propose method that uses domain knowledge or label hicrarchics lo dynamically group semantically related classes to meet the desired coverage for a given confidence threshold. Our method maintains consiioondi calibration when possible and provides group-conditioned guarantees where necossary. We evaluate our method on outcome diagnoses prediction, an important clinical task that does not only benefit from robust. uncertainty estimation, but also presents very imbalanced label distribution. We

---

Published in Transactions on Machine Learning Research (01/2026)
conduct experiments using three clinical datasets employing two medical taxonomies (ICD- 10 and CCSR) and label spaces of varying sizes with up to more than 1,000 classes. Our results show that the proposed approach is able to successfully exploit the label hierarchy and consistently improves class-conditioaal coverage for infrequent diagnoses. By improving coverage for underrepresented classes, our method enhances the reliability and trustworthi- ness of predictive models. This improvement is especially valuable in clinical applications, where failure to detect rare but serious conditions can lead to harmful consequences.
1 Introduction
In this work, we address class calibration in challenging settings with large mumber of classes and limited available samples. We focus tasks involving hierarchically organized label spaces, where classes are struc- tured according to the relationships between the classes, e.g., a taxonomy. Such hierarchies capture semantic relationships between labels and are common in many real-world domains, inchuding product categorization, biological classification of bacteria, or diagnoses in healthcare. More broadly, our approach applies lo any setting where domaín knowledge enables meaningful grouping of classes, whether through formal hierarchies or alternative semantic relationships. In settings where structured domain knowledge is limited, our method is complementary to data-driven approaches that automatically discover groupings from data.
We focus on the medical domain, specifically on the task of outcome diagnosis prediction as a canonical example of this setting. A key challenge of outcome diagnosis prediction is the large and imbalanced label space that exhibits a pronounced long-tail. Clinical decision support systems (CDSS) must not only show strong performance, but also be well-calibrated, as miscalibration can lead to harmful misdiagnoses (Alkan ct al. 2025). At the same time, clinical models are usually designed to yield point predictions (Miotto et al., 2018 2016 which offer no measure of uncerlainty. This is especially problematic in diagnosis lasks, where overlapping symptoms (Wagan et al., 2024] are common and models may struggle to distinguish between similar conditions, especially for underrepresented classes. Thus, there is an important requirement for models to provide reliable predictions, as well as uncertainty estimates.
The medical domain is particularly suitable for demonstrating the impact of improvements in class conditional coverage, as it offers well-established taxonomies such as ICD-codes and presents high-stakes scenarios where calibration failures can directly affect patient safety. For a successful clinical deployment of AI technology, medical staff and patients need to trust its predictions. A central assumption underlying a substantial body of literature in eXplainable Al (XAI) is that trust can be fostered by rendering model predictions more
transparent (Ribeiro et al. 2016 Lundberg & Lee 2017 Samek et al. 2019 Schmidt & Biessmann 2019
especially in the medical domain HHam et al. 2023 van Aken et al. 2022].
While transparency addresses an important dimension of trustworthiness (Wang Yin 2021 another key aspect lies in understanding the uncertainty of AT system predictions (Dhuliawala et al. 2023 In light of the growing capacity of models, which has been associated with poor calibration of uncertainty estimates (Snoek ct al. 2019 Guo et al. 2017), improving uncertainty calibration is a fundamental prerequisite for trust in AI systems. This is especially true in safely-critical AI applications that fall under the high-risk calegory of the EU AI act. (Council of European Union 2024 such as AT healthcare products, where understanding
uncertainty (Grote & Berens 2023 Seoni et al. 2023 and communicating it (Banerji et al. 2023) are
essential to improve trust.
Uncertainty Calibration with Conformal Prediction. A popular model-agnostic calibration method is Conformal Prediction (CP)(Vovk et al. 2005), which provides prediction sets instead of point predictions. These sets offer formal coverage guarantees, indicating how often the true label is expected to be included on average. This is especially important in clinical settings, where overconfident point predictions can be misleading. However, ensuring reliable coverage is challenging in imbalanced, long-tailed label distributions [Ksaa & Taylor 2023). This because rare classes are often absent or severely underrepresentedd when calibration sets are very limited in size, making difficult estimate uncertainty reliably or guarantee valid coverage for those classes.
2

---

Published in Transactions on Machine Learning Research (01/2026)
Diseases of the Circu- latory System (100-199)
32 patienta
Chronic rheumatic heart diseases (105-109)
Acute rheumatic fever (100-102)
12 patients
20 patients
Rheumatic mitral valve discases (I05) patients
Rheumatic fever with heart involvement (I01) patients
Rheumatic chorea (102)
palients
Figure ICD-10 hierarchy of selected diseases the circulatory system. 1CD-10 codes are shown in parentheses and the number of patients in the calibration data per class is indicated the bottom of cach box. When the number of samples for class smaller than certain threshold (here, m 10), our method, Dynamically Grouped Conformal Prediction (DGCP), groups this class with semantically related ones, using domain knowledge such as the ICD-10 hierarchy. In this example, the leaf node Rheumatic chorea (I02) with only 3 patients (highlighted in light red) is grouped with I01 and 105 because they share a common higher-level ICD calegory (indicated by gray shading), which used to define the grouping. However, 101 and 105 exceed the threshold (highlighted in light green), which is why they are calibrated on the class level.
Contribution: Dynamically Grouped Conformal Prediction. We address these challenges by proposing post-hoc and model-agnostic method called dynamically grouped CP (DGCP). DGCP intro- duces a hyperparameter m, which defines the minimum number of calibration samples required for class-level calibration. This is motivated by preliminary experiments, which show that classes with no or only a few calibration samples cannot be reliably calibrated. Therefore, the idca to relax the strict guarantocs class- conditional conformal prediction and dynamically group a class with fewer than m samples logether with semantically related classes using domain knowledge. As our experimental evaluation shows, this allows us balance strong class-conditional guarantees, while increasing coverage for underrepresented classes. Figure [ illustrates example that uses ICD-10 codes as labels and m 10. In this case, the number of calibration samples for the diagnosis of rheumatic chorea (T02) does not exceed the threshold m. Thus, DGCP combines all patients diagnosed with 102 together with other diagnoses, using domain knowledge. For hierarchical label spaces, a natural grouping is given by a higher level in the hierarchy. However, it is also possible to use any other attribute or grouping function. Our experiments demonstrate that the proposed method is robust the choice of hyperparancter m.
In summary, we propose dynamically grouped conformal prediction that maintains class-conditioned cali- bration if sufficient data are available and provides group-conditioned guarantees if not. We evaluate approach on three clinical datascts and show that it consistently improves class-conditional coverage for the underrepresented classes.
2 Related Work
Outcome Diagnoses Prediction from Text. Transformer models have demonstrated remarkable per- formance across various domains, including the medical ficld. yaan Aken et al. (2021) pre-train transformers using a modified next-sentence prediction objective between admission and discharge sentences to improve outcome diagnoses prediction. [aik et al. (2022) augment clinical notes with medical literature and Ji & Marttinen (2023) adopts multitask approach for unsoen diagnoses categories. van Aken et al. (2022) ad- dressed the problem of rare diagnoses codes by combining prototypical classifier with Transformer improve prediction performance.
3

---

Published in Transactions on Machine Learning Research (01/2026)
Uncertainty Quantification and Conformal Prediction. Uncertainty quantification in deep learning
has gained considerable attention in recent years (Fakour et al. 2024 Tyralis & Papacharalampous 2022;
Abdar et a 2021). Conformal prediction (CP) has emerged as a principled framework for producing prodiction sets with rigorous coverage guarantees, even when the underlying models are imperfect. Notably, Straitouri & Rodriguez (2024); ; Straitouri et al. (2023) demonstrate that conformal prediction can assist domain experts reduce their workload, lead to better docisions, and increase trust (Dhuliawala et al.) 2023
Conformal prediction has been successfully applicd across a wide range of domains, inchuding natural lan-
guage processing (Mohri & Hashimoto 2024 Campos et al. 2024 clinical medicine (Hirsch & Goldberger
2024
Grote & Berens 2023 Banerji et al., [023] Lu et al.) 2022, Olsson et al. 2022 Vazquez & Facelli 2022
Kompa et al.) 2021), and drug discovery Alvarsson et al. 2021), underscoring its broad utility. Further, a
substantial amount of literature has focused on improving set efficiency (Dhiloon et al.) 2024 Stutz et al.
2022 Fisch et al. 2021 Romano et al. 2020 Angelopoulos et al.
2021
generalizing beyond coverage
other monotonic loss functions (Angclopoulos et al.) 2024p, tackling hicrarchical classification (Mortier et al.,
2025), distribution shifts (Gibbs & Candès 2024 Barber et al.) 2023 Bhal.agarr el. al., 2023) or structured
output prediction (Zhang et al. 2025). In this work, we enhance the standard split conformal prediction
framework (Angclopoulos & Bates 2023) and propose an approach to improve class-conditional coverage for infrequent classes by incorporating domain knowledge.
The work most closely related to ours is by Ding et al. 2023), who address multiclass classification with up to 1,000 labels by clustering classes based on similar nou-conformitty scores. Like our work, their goal is to overcome the limitations of cladsicon conformal prediction in low-data regimes. Our approach is complementary and differs in two key aspects. First, while their method uses data-driven clustering to discover groupings from calibration data, we leverage domain knowledge such as label taxonomies to define semantically meaningful groups. This makes our approach particularly effective in domains where established hierarchical structures exist; However, data-driven methods are necessary when such knowledge is unavailable. Second, we apply grouping selectively: we maintain class-level calibration for well represented classes (2 m samples) and resort to group-level calibration only for underrepresented classes. This allows us to preserve fine-grained class-level guarantees where data are sufficient while ensuring valid coverage for rare classes through domain-informed grouping.
3 Task and Datasets
Outcome Diagnoses Prediction. We evaluate our approach on the task of predicting the primary dis- charge diagnosis from unstructured clinical admission notes, as introduced van Aken et al. (2021). J. Unlike multi-label settings that consider multiple diagnoses per patient, this task focuses solely on the main diagnosis determined at dischargo. Following van Aken et al. (2021), only information available at the time of admis- sion is used for prediction, simulating a realistic early-decision support scenario. The task is formulated as a multiclass classification problem with up to over 1,000 possible labels, most of which are underrepresented in the training data.
setting
Datasets. Large-scale, publicly available medical datasets for general use are rare. We use the MIMIC datasets, which are the most comprehensive clinical datasets that are publicly available. These contain anonymized patient records from the Intensive Care Unit (ICU) of the Beth Israel Deaconess Medical Center in Boston. MIMIC-II (Johnson et al. 2016) consists of data between 2001 and 2012, and MIMIC-IV (John- son et al. 2 between 2001 and 2019, respectively. To create datasets with label spaces of different sizes, we split MIMIC-IV randomly into halves. Each of the splits contains different patients. For the first dataset, we use three-digit ICD-10 codes (compare Choi et al. (2017)). We map the labels of the second MIMIC-IV dataset and MIMIC-111 to CCSR codes, which are clinically meaningful groupings of ICD-10 codes (Health- care Cost and Utilization Project (HCUP). 2024). We remove notes that directly mention the correct main diagnosis using MedCAT (Kraljevic el al.] 2021). Additionally, we extract two attributes from each primary diagnosis. The first attribute, body system, is derived based on the classification framework provided by Healtheare Cost and Utilization Project (HCUP) (2024), which categorizes diagnoses into 21 clinically rele- vant groups, such as neoplasms (NEO), respiratory conditions (RSP), and injuries (INJ). Second, with the help of medical professionals, we assign severity score per diagnosis, reflecting its level of life-threatening
4

---

Published in Transactions on Machine Learning Research (01/2026)
MIMIC-III CCSR Codes (356)
MIMIC-IV CCSR Codes (423)
MIMIC-IV ICD Codes (1054)
1,000
Head
Long Tail
scale)
100
10
100 200
300
100
200
300
400
200
400
600
800
1000
Enumerated Diagnoses
Diagnoses distribution in MIMIC-III (CCSR), MIMIC-IV (CCSR), and MIMIC-IV (ICD). Diag- noses sorted by frequency within each dataset. The Head includes the most frequent. diagnoses covering 80% of patients (e.g., 163 diagnoses in MIMIC-IV ICD), while the Tail comprises the remaining, less fre- quent diagnoses (e.g., 891 diagnoses in MIMIC-IV ICD) that account for the remaining 20% of cases. This highlights the extreme class imbalance present in clinical datasets.
Figure 2:
risk. Severity level 1 corresponds to the most critical diagnoses such sepsis. In contrast, severity level 5 represents non-critical diagnoses. This results in the following datasets: MIMIC-III (CCSR) with 356 diag- noses and x 4.000 records, MIMIC-IV (CCSR) with 423 diagnoses and = 44.000 records, and MIMIC-IV (ICD) with 1054 diagnoses and x 44.000 records. Note that the MIMIC-IV (CCSR) and MIMIC-IV (ICD) datasots not only differ in thoir label spaces, but also include different patients, resulting from a random split of MIMI-IV in order to simulate two separate hospitals. In Figure we present the label distribution of all training datasets.
We split cach of the three datasets into training, validation, and test sets using stratified sampling. We keep each patient's 's first visit and ensure that all diagnoses appear at least once in the training set.
4 Background: Conformal Prediction
Conformal prediction (CP), originally presented by Vovk et al. (2005) is distribution and model- agnostic uncertainty quantification method. It turns any black-box point predictor into a set predictor, which statistically guarantees to cover the correct label with a user-defined grobabiltyycconfidee level. Assume f is a fitted classification model that outputs softmax scores: f(x) E R [0.1]- For point predictions, the predicted class ÿ E K} is the index of the highest softmax score.
Non-conformity. To build confidence sets C(Xtest), CP uses non-conformity scores, computed on cali- bration set not used for training: Seati- = { Vi E (1,..., Noata.. Non-conformity scores 8; represent how (X,, y1) differs from the model prediction (X,, ÿ.). For this, we use a non-conformity score function, e.g., one minus the softmax output of the true class: = f(Xi),, Next, compute the k-th empirical quantile of Senti as follows:
(n + 1)(1 a1] k
(1)
n 9 quantile(Seatit k),
where (1 (1-a) a) the user-defined confidence level and n is the number of calibration points. For a new unseen test data point Xtest (yeest is unknown), CP includes all classes in C for which 8; does not exceed the threshold q. Formally, C(Xtest) = {y f(Xtest)y S g). which is guaranteed satisfy (Equation
independently of the model and the data distribution Zeni et al. 2020: Angelopoulos & Bates 2023
P(yhest E C(Xtest)) 2 (1 a)
(2)
5

---

Published in Transactions on Machine Learning Research (01/2026)
Marginal
Conditional
Group 1
Group 2
Group 1
Group 2
Covered
Not Covered
Figure 3: Comparison of marginal (left) and conditional (right) conformal prediction. Both examples rep- resent, test set of 100 data points, where 90% of the true labels exist. in the conformal prediction sets, visualized as light gray points. Black points represent samples for which their true label was not included in the prediction set. Although both methods achieve 90% overall coverage, marginal prediction distributes coverage unevenly across groups. Where the groups are typically defined by the target class, but can the- orctically depend on any attribute. On the other hand, conditional prediction guarantees 90% coverage for each group individually. This figure is adapted from Angelopoulos Bates (2023).
(Marginal) Coverage. This property, referred as marginal coverage Lei Wasserman 2014), ensures that approximately (1 a3% of the test data points are correctly included in the prediction sets. When the model f effectively fits the data, these sets C' tend to be small. Conversely, if f does not fit the data well Xtest is ambiguous, C will be greater in size (Lii et al., 2013). As Figure g (left) shows, although marginal coverage gives statistical guaranlees the groups are typically defined by the target class, but can theoretically depend on any attribute. In many cases, it is desirable to obtain the coverage guarantee of Equation for each group, known as conditional coverage.
average, it may neglect the existence of groups in the data. Where
Conditional Coverage. As illustrated in Figure 3 the left side (marginal coverage) shows that coverage is achieved for Group 1, while Group 2 achieves no coverage at all. However, because marginal coverage guarantees are only average across all samples, the overall 90% confidence level is satisfied. To achieve more balanced coverage, the formulation in Equation [1 is modified to define multiple group-specific thresholds g(æ), each corresponding to single group E g. As a result, associated variables such as n(()) k(0), and calib are also indexed by group. Finally, the confidence sets constructed follows:
C(Xrest) = {y ((X(2)) S G(9)),
where I denotes the group to which the test sample belongs. These sets C satisfy the stronger group- conditional guarantee defined in Equation (3), ensuring that each group individually meets the target cover- age level. Note that in contrast to marginal, the right side (conditional coverage) of Figure 3 demonstrates balanced coverage across both groups, each achieving 90% coverage.
Plytest E (( (( 2 21 - a), YgEg
(3)
Class-conditional coverage. If the groups are defined by label attribute follows:
g g=(k), Vgg G and {L,.., K}.
(4)
Equation 3 referred to as class-conditional coverage H In many applications with small and balanced label spaces, applying class-conditional CP has shown good results. For further details and proofs, we refer the reader to Angelopoulos & Bates (2023) Vovk et al. (2005).
5 Methods
Dynamical Grouping with Limited Calibration Samples per Class. Reliable class-conditional cov- erage is challenging on datasets with large and imbalanced label spaces. To address this issue, we propose
1 CClass-condiiin conditionl conformal prediction also known as mondrian conformal prediction Vork et 2005

---

Published in Transactions on Machine Learning Research (01/2026)
dynamically grouping classes with insufficient calibration data into domain-specific groups. To this end, we introduce a new hyperparameter m, which denotes the minimum number of calibration samples required for class to be considered sufficiently represented. If [X calih 2 m, class k is treated as having sufficient calibration data, and we apply class-conditional conformal prediction with g = { (cf. Equation ) Con- versely, if [x(k < m, class is grouped together with other classes, and we perform group-based calibration according to Equation 3 The grouping function user-defined and, given the calibration set size and a specific class, returns the appropriate group (e.g., based on additional features used organize the data), For instance, in hicrarchical label spaces, it may return coarscr-lovel category another attribute describ- ing, for example, the body system or severity of a disease. A formal definition of this procedure is provided in Algorithm 1
Algorithm Calibration with Dynamically Grouped Conformal Prediction (DGCP)
Require: fitted classification model calibration set Xeatis confidence level (1-a), sufficient data threshold m, grouping function group()), non-conformity score function nonconformity())
Ensure: fitted classification model f is calibrated f
Scalib < nonconformity(XX (Xallb, f)
for each class l (I,...,k) do
if |x cath. 2 m then
g(k) + Quantile (saai [(n +1((-o))
b class-level calibration
else
g grogruu((Xculr k)
q(h) Quantile (5(0) (100))))((11)))
group-level calibration
n(2)
end if end for return calibrated model f
Similarity to other Conformal Prediction Methods. In contrast to Ding et al. (023) who first cluster the calibration data based on their non-conformity scores, our method directly uses domain knowledge to assign samples to each group. In the best case, where every class has enough data for a reliable calibration, our method is equivalent to class-conditional CP (ef. Equation 4 and fulfills the predefined class-conditional guarantees. If class has fewer than m calibration points, we group samples with a common attribute to estimate a group-specific threshold. This threshold is then applied to the underrepresented class. For example, diagnoses related to cardiovascular conditions can be grouped to estimate shared threshold, which is then used for classes with limited data of that group (c.g., the rare Takotsubo cardiomyopathy diagnosis). This approach enables the calibration of classes that are absent from the calibration set.
6 Experimental Setup
Through a comprehensive evaluation, we validate empirically whether our methodology improves class- conditional coverage. Our primary objective is to improve upon existing methods that provide formal class-conditional coverage guarantees. While we also compare to non-conformal baselines that achieve strong empirical performance, these methods provide no formal guarantees on class-conddiional coverage. Their good performance in our experiments reflects favorable empirical conditions rather than theoretical assurances and may not hold under other scenarios. Therefore, guarantecs. We train domain-specific model (Section 6.1 on each datasct, and compare the performance (Section (6.4) of different calibration methods (Section 6.2 and Section 6.3). For calibration, we use the following hyperparameters: calibration set sizes n E {1000, 2000} and confidence levels (1 - a) e (0.8, 0.9] and keep the underlying prodictors fixed. In addition, for DGCP, we evaluate the effect of m E (10, 20). To ensure the robustness of our results, we repeat each experiment 50 times. In each repetition, we randomly re-sample the calibration set from the test set and the remaining data points for testing: 4,094 for
key comparisons focus methods with formal
7

---

Published in Transactions on Machine Learning Research (01/2026)
MIMIC-III 20,900 for MIMIC-IV(CCSR)) and 20,722 MIMIC-IV(ICD).. We report the metrics detailed in Section 6.4
6.1 Prediction Model
2022), a tramsformer-based architecture that has demonstrated strong performance in diagnosis prediction, especially for rare diagnosis codes. The model combines a biomedical transformer encoder (Gu et al., 2020) with prototypical layer. This layer consists of one prototype vector and one attention vector per diagnosis. Each patient admission note is encoded and projected into lower-dimensional space, where it is weighed by diagnosis-specife attention vectors to map the note to the latent metric space. The model then computes the softmax over the negative distances between the resulting reprosentation Upat and each diagnosis prototype Udiag, using the Euclidean distance: d = Uppt - Mddingl2 During training, the model minimizes the binary cross-entropy (BCE) loss over all patients = Epa Eding BCE(ssftmax((4) diag,, where ypat,dbag E {0,1} } is the ground-truth. This loss encourages the representation of each input to move closer to the prototype of the correct class and farther from those of the incorrect ones. Finally, for inference, the prediction corresponds to the diagnosis with the closest protolype.
For our experiments, we use ProtoPatient (aan Aken et al.
We choose ProtoPatient for evaluation because of its strong performance rare diagnosis codes, and because its distance-based classification provides a natural non-conformity score. In addition, ProtoPatient offers inherent interpretability and justification for predictions, which complements our goals of transparency and trustworthiness, critical factors in clinical decision support. For further findings and results, we refer to yan Aken al. (022). However, DGCP is model-agnostic and can be applied post hoc to any base predictor.
6.2 Conformal Calibration Methods
For model calibration, we define the non-conformity score the distance between the patient. encoding and the prototype of the true class: 8; d; f(X,)x, Vi E {1,..., Neati) (ef. Section 6.1 where f is the fitted ProtoPatient model. We apply CP as described in Section 4 Although efficiency is important for practical deployment, in this study our focus is on improving cass-conditional coverage. This choice reflects priorities in high-stakes tasks such as medical diagnosis, where including the true diagnosis (coverage) is critical.
acknowledge that set size
Marginal and Class-conditional CP. To calibrate with marginal or class- condiiioniii CP, we procoed as described in Section 5 Marginal CP provides coverage guarantees averaged over all labels (cf. Equation ( Class-condiiional CP provides coverage guarantees for each class, accounting for imbalances that marginal coverage neglects (ef. Equation (
Clustered CP. Clustered conformal prediction (Ding et al. 2023 improves class-conditional coverage in settings with limited data. clusters classes with similar non con-conformity scores and calibrates at cluster- level.
Dynamically Grouped CP (ours). We apply DGCP as described in Section 5 and use the following naming scheme: DGCP/Grouping Method. For example, when we use the body system related to a diagnosis for grouping, we refer to this method: DGCP/Body System.
6.3 Non-conformal Calibration Methods
For completeness, in addition to the above CP methods, we use baselines that construct prediction sets from model outputs, but do not provide any coverage guarantces.
Adaptive top-k. A simple approach that returns set predictions draws inspiration from the top-k clas- sification metrics. Where is not fixed, but classes are included in the prediction set until the cumulative sum of probabilities exceeds the predefined confidence level (1 11-)). a).
8

---

Published in Transactions on Machine Learning Research (01/2026)
Calibrated Adaptive top-k. Empirical evidence suggests that modern neural networks poorly cali- brated (Guo et al. 2017). To account for this, use temperature scaling (TS) to calibrate ProtoPatient's probabilities and then follow the same approach as Adaptive top-k to construct prediction sets. Tempera- ture scaling is a simple method to calibrate point prediction models by introducing a single scalar parameter T > O, which scales the logit values and the calibrated logit defined as:
exp(zx/T)
2 = max
vk (I,..,,,
(5)
Σ eXp(2)/T)
Note that in contrast CP methods that are applied post-hoc, fitting T requires gradient computation. We follow Angelopoulos et al. [2021 for the implementation.
6.4 Conformal Metrics
In our analysis, we use common metrics to assess the validity of our methods. Macro coverage measures the mean of the average empirical coverage per class and weights them equally. Formally,
IX (K)
coueragCmacro
Σ
1(k C(X(ed.)))
(6)
K
(K) test
where K is the number of classes under consideration, typically the entire data X or a subset thereof, | is the number of test samples available for class k, and C(x(()),)) is the prediction of the i-th example of class k. Idcally, the empirical macro coverage reaches the specified confidence level. Additionally, measure the mean prediction set size:
Xted
Xtevel
set
size
(((ttstt,a
(7)
[Xeest
Smaller prediction sets are preferable due to their increased efficiency.
7 Results
Although we trained and evaluated two variants of the ProtoPatient model, one with pre-initialization of the protolypical layer following van Aken el al. (2022) and one with random initialization, we elaborate on their differences Appendix A Since the pre-initialized version consistently yields stronger conformal calibration metrics across all methods, we use it the basis for all subsequent experiments. As described in Section G calibrate the model with different approaches and compare them experimentally. If not stated otherwise, we show the results for m 10, confidence level (1 a) 0.9, and calibration set size of 1000. We expand on the results of the different calibration hyperparameter settings in Appendix B
Table I compares the performance of the calibration methods. We report the macro coverage over all classes in the dataset, as defined in Equation (G), which assigns equal weight each class, including those lacking calibration or test samples, and the average prediction set size, as defined Equation 7 Because all datasets exhibit pronounced long-tailed distributions, methods that neglect rare classes achieve lower macro coverage but smaller prediction set sizes. In this work, we primarily focus improving class-conditional coverage. Enhancing the efficiency of prediction sets is left for future work.
Adaptive top-k and Calibrated Adaptive top-k show surprisingly good results and are generally second and third best in terms of macro coverage, but do not provide formal coverage guarantees. Clustered CP and Marginal CP perform almost equally in both class-conditional coverage and prediction set size. Class- conditional CP achieves the lowest macro coverage, as many classes do appear in the calibration set, preventing the prediction of these classes.
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Experiment Results. Calibrated using m 10, confidence level (1 - a²) 0.9, and calibration set size of 1000. Conformal calibration methods (with guarantoes) are above, and non-conformal methods (without guarantees) are below the horizontal line. Macro coverage is computed over the entire dataset X and accounts for all classes present in the data. The corresponding number of classes is provided below each dataset name. Our approach, DGCP with body system fallback, consistently improves and outperforms other methods in terms of class-conditiona coverage on all datasets. + represents standard deviation over 50 repetitions.
Macro Coverage (1)
Prediction Set Size (J)
MIMIC-1II
MIMIC-IV
MIMIC-IV
MIMIC-III
MIMIC-IV
MIMIC-IV
Method
(356 CCSR)
(423 CCSR)
(1,054 ICD)
(356 CCSR)
(423 CCSR)
(1,054 ICD)
DGCP/Sevvrityy Score (ours)
60.2 2.9
67.0 + 1.6
2..0
18.4 2.0
33.9 6.7
DGCP Body System (ours)
65.5
71.8 + 2.4
67.0
3.6
31.0
67.8
14.4 14.4
Clustered CP
55.2+2.3
+.1
61.7+28
12.0+1.1
16.9+144
27.8 4.4
clndttaon CP
34.9 1.5
38.9
23.7
16.3
20.3
21.4
Marginal CP
55.1 + 2.1
66.6+1.7
61.7+255
12.0+1.0
17.0
27.9 + 3.9
Calibrated Adaptive top-k
64.1 + 1.6
1.5
6.
170
21.9
34.6 3.0
Adaptive top-k
62.1 + 0.9
67.4+0.2
0.3
15.3
1833+00
27.5 + 0.0
Dynamically Grouped CP Improves Class-conditional Coverage. DGCP/Body System is consis- tently the best method in terms of macro coverage, closely followed by DGCP Severity Score. Both methods improve their class-conditional coverage over Class-conditional CP by a factor between 1.7 and 2.9. However, this (ypically comes at the cost of increased prediction set sizes; in our case, they range between 0.9 and 3.1. This highlights the trade-off between efficiency (small prediction set sizes) and reliability (high macro coverage). In our experiments, this relationship is roughly lincar.
A Closer Look at the Label Distribution in the Calibration Dataset. Across 50 repetitions, only 166 6.4 of the 356 classes in MIMIC-III are present in the calibration dataset, with 15.5% 1.4 containing ten or more calibration samples. For MIMIC-IV (CCSR), 255 + 6.2 out of 423 classes are represented in the calibration data, with 13.8% + 1.2 having at least ten samples. In MIMIC-IV (ICD), 313 + 9.3 out of 1,054 classes appear in the calibration data, of which 5.4% + 0.7 include ten more samples. In this experiment, the calibration dataset contains 1,000 samples, which explains why larger label spaces reduce these fractions. In Table 2 we show the macro coverage separately computed for classes in the calibration data that have ten or more calibration points and less than ten calibration points. Note that here we focus on the calibration dataset and compute the macro coverage over subsets that do not include all classes existing in the entire dataset. Consequently, these results must be interpreted independently of Table []] As expected, for classes with sufficient calibration data, all conformal calibration methods consistently exceed the expected coverage of 90%. In contrast, for classes with fewer than ten calibration samples, coverage degrades. However, compared to Class-condiiional CP all methods achieve improvements.
Class-conditional Coverage under Data Scarcity. Figure f shows the class-conditional coverage as function of class size in the calibration data for all methods that provide class-lovel guarantocs. Class- conditional CP yields the lowest conditional coverage, reaching the target confidence level (1 - a) 0.9 only once approximately ten calibration samples per class are available. For classes with fewer than ten samples, all methods exhibit increased class-conditional macro coverage. As discussed above, when sufficient calibration data are available (here m 10), DGCP is equivalent to Class-conditional CP. Since Clustered CP behaves almost identically to Marginal CP, it attains close to 100% coverage for frequent classes.
To complement the analysis in Figure A and to extend the findings reported Table 1 we additionally examine in Appendix C how coverage varies with respect to the class distribution encountered at test time. This provides further insight. into the behavior of the methods when applied in practice.
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Experiment Results separated for classes with at least ten calibration points ((x catal 2 10) vs. less than ten (ix 10). Macro coverage is computed relative to these subsets in the calibration set and, therefore, represents only fraction of all classes. Calibrated using and calibration set size of 1000. For classes with sufficient calibration data (Ixx(a) catasl 2 10), all conformal calibration methods consistently exceed the expected coverage of 90%. However, with less calibration data, all methods achieve improvements Class-conditional CP. DGCP [ooyy System followed by DGCP Body System achieve the best improvements.
10, confidence level (1 a) = 0.9,
Macro Coverage (1) for [x catibl 2 10
Macro Coverage (1) for [x 10
MIMIC-III (356 CCSR)
MIMIC-IV
MIMIC-IV
MIMIC-III
MIMIC-IV
MIMIC-IV (1,054 ICD)
Method
(423 CCSR)
(1,054 ICD)
(356 CCSR)
(423 CCSR)
JGCPP/eeeertg Score (ours)
92.1 + 1.3
92.8 1.4
92.6 1.7
77.0 #
77.5 + 1.9
80.6 10.6:1 1.7
DGCP DGdy/Body System (ours)
92.1 + 1.3
9.
9..6
77.7
81.6 1.8
82.6 1.8
Clustered CP
95.9 0.9
96.2 0.7
98.2 0.5
72.2
76.6 2.3
79.3 + 2.2
Clnsscconiiiional CP
92.1 + 1.3
9.48
92.6 12.6117 1.7
66.8 + 2.9
65.5 2.2
62.6 2.1
Marginal CP
95.8 + 0.8
96.2 + 0.5
98.2 + 0.5
72.1 2.3
76.8 + 1.8
79.3 + 1.9
MIMIC-III CCSR Codes (356)
MIMIC-IV CCSR Codes (423)
MIMIC-I ICD Codes (1054)
Class- Class-Condee Conditional Coverage
0.9
0.8
0.7
0.6
0.5
10
10
10a
10a
10a
10
Label Frequency in Calibration Data
Body|Body System
DGCP|Severity Score
CaassCoonllional CP
Clustered CP
Figure 4: Class-conditional coverage function of class size in the calibration data for conformal methods. The horizontal dashed line marks the target confidence level. Calibrated using 10, confidence level (1 a) = 0.9, and calibration set size repetitions. If a class contains fewer than ten calibration samples, Class-conditional CP does not attain the larget confidence level (1 a) 0.9. In this regime, both DGCP variants and Clustered CP increase the class-conditional coverage. For classes with ten or more calibration samples, DGCP becomes equivalent to Class-conditional CP and reaches the target confidence level (1 a) 0.9.
1,000. Error bars show the standard deviation over the 50
8 Discussion
Non-conformal Calibration Methods. Adaptive top-k and Calibrated Adaptive top-k achieve sur- prisingly strong macro coverage in our experiments, comparable to exceeding some conformal methods. However, this empirical performance comes with an important caveat: these methods provide no formal guarantees. Under different conditions, their coverage may degrade arbitrarily, whereas conformal methods maintain their guarantees.
Marginal CP and Class-conditional CP. Class- conditionl CP yields the lowest Mass-conditionns cov- erage because it calibrates each class independently and cannot. generate predictions for classes that
11

---

Published in Transactions on Machine Learning Research (01/2026)
absent from the calibration set. Given the highly imbalanced distribution of diagnoses and limited cal- ibration set of 1,000 samples, many classes remain unrepresented, leading to significantly degraded macro coverage, In contrast, Marginal CP aggregates across all samples during calibration, weighting classes ac- cording to their frequency in the calibration set. Unlike Class-conditional CP, it produces prediction sets for all classes, improving macro coverage. However, this comes at the cost of class-level guarantees. Since Marginal CP focuses on the majority classes (i.e., the head of the label distribution), it achieves relatively small prediction set sizes, but provides poorer coverage for infrequent classes.
Clustered CP is Almost Equivalent to Class-conditional CP. In terms of macro coverage and average set size, in our experiments, Clustered CP's results are almost equivalent to Marginal CP. This occurs because of the high imbalancod label space and the limited calibration set size. For the Clustered CP all classes that cannot be confidently assigned to any learned cluster are assigned to a NULL cluster, which uses marginal CP. Given the pronounced long-tail distribution of diagnoses and limited calibration data, almost all of the classes are assigned to this NULL cluster. Consequently, the NULL cluster dominates the overall behavior, causing the method to effectively behave similar to Marginal CP, which simply groups all classes togeterr. In contrast, our domain knowledge- besee based grouping forms semantically meaningful groups, avoiding the need for a generic NULL category and enabling better calibration for underropresented classes.
Using Domain Knowledge Improves Class-conditional Coverage. As shown in Table I leverag- ing domain knowledge to group diagnoses by body system or severity score based on medical laxonomies improves dlass-conditional coverage. In Table 2 and Figure | it is shown that improvements stem from the underrepresented classes rather than the majority classes. These results suggest that hierarchical label structures, which capture meaningful semantic relationships, can make calibration more robust, particularly for underrepresented classes. We argue that although this approach trades fine-grained class-level guarantees for more stable group-level estimation, the resulting increase in conditional coverage makes it a worthwhile compromise in imbalancod settings. Although not all domains offer expert-defied hicrarchies or may intro- duce noise that complicates grouping, it could benefit domains with taxonomic label structures, such as the biological, legal, or financial domains.
9 Conclusion
In this work, we introduce dynamically grouped conformal prediction (DGCP). We empirically demonstrate that our approach improves class conditional coverage in settings with limited data availability. By leveraging domain knowledge to group underrepresented classes, DGCP enables more robust threshold estimation for rare classes while preserving class-level guarantees for well represented ones. We demonstrated its effective- ness across three clinical datasots and two different label spaces: MIMIC-III (CCSR), MIMIC-IV (CCSR) and MIMIC-IV (ICD), showing consistent improvements in conditional coverage.
Limitations and Future Work. Our investigation focuses mainly on improving class conditional cov- erage for infrequent classes since these are critical for high-stakes domains. However, other aspects of conformal prediction are also important for safe clinical deployment. These include optimizing prediction set efficiency (Romano et al. 2020; Angelopoulos et al.) 2021 Stutz et al.) 2022) or controlling for alter- native metrics such as Fl-score (Angelopoulos et al., 2024) among others. Future work should explore the integration of these ideas into our approach.
10 Acknowledgments
We thank the reviewers and the Action Editor for their constructive leedback, which improved this work. This work is funded by the Deutsche Forschungsgeneinschatt (DFG, German Research Foundation) Project-ID 528483508 FIP 12, European Union under the grant project 101079894 (COMFORT Improving Urologic Cancer Care with Artificial Intelligence Solutions), German Federal Ministry of Research, Technology and Space grant number 16SV8856, well by BMWe SOOFT, Grant Agreement 13IPC040D. The views
12

---

Published in Transactions on Machine Learning Research (01/2026)
expressed are solely those of the authors and do not necessarily reflect those of the European Union European Health and Digital Executive Agency (HaDEA); neither is responsible for them.
References
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Sacid Na- havandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Pusion, 76:243-291 297, 2021. ISSN 1566-2535. doi:
http:///..........ff.sss... wofffuss.221.000 005.008.
Muhammet Alkan, Idris Zakariyya, Samuel Leighton, Kaushik Bhargav Sivangi, Christos Anagnostopoulos, and Fani Deligianni. Artificial intelligenne--driven climical decision support systems. CoRR, abs 25501.9968,, 2025. doi: 10.48550 ARX www.A..w.......... 009628. URL https:///oi...... http://ti..org//// org/ 10 48550/arXi1s 2501 09628
Jonathan Alvarsson, Staffan Arvidsson McShane, Ulf Norinder, and Ola Spjuth. Predicting With Confidence: Using Conformal Prediction in Drug Discovery. Journal of Pharmaceutical Sciences, 110(1):42 49, January 2021. ISSN 1520-6017. doi: 10.1016/j.xppss.22000
Anastasios N. Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Found. Trends Mach. Learn., 16(4):494 591, 2023. doi: 10. 10.1561 1215612220000101..1 URL https://bbd..orr//O..66/// org/10.15611
22000001011
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael I. Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nett net, 2021. URL https://openrevies
het/forum?id=eNdil DbM9
Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
Christopher R. S. Banerji, Tapabrata Chakraborti, Chris Harbron, and Ben D. MacArthur. Clinical ai tools must convey predictive uncertainty for each individual patient. Nature Medicine, 29(12):299- 2998, Dee 2023. ISSN 1546-170X. doi: 10.1038/s41 1101///111111-0222566633 URL https://doi http::::/doi.org/1...111
041591-023-02566-7
Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51 (2):8116 845, 2023. doi: 10.1214/23-A05227 URL
fttpp://doi.org/111 2ttp://000..rr111112222222-2222
Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, 1CML 2023, 23-20 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 2337 2363. PMLR, 2023. URL
wtt.::ssww..rrr.e./eemmmmmmeeeee ppress/v202/ pttpess//22//aaaaaaarrrrrr bhattagar html
Margarida M. Campos, António Farinhas, Chrysoula Zerva, Mário A. Figueiredo, and André F. T. Martins. Conformal prediction for natural language processing: A survey. CoRR, abs//2405001766 2024. doi: 10.48550/ (05550//RXV.2220 2405.019766 URL httLhhttt::/ddd org/ 485500/arXii 2405.01976
Edward Choi, Siddharth Biswal, Bradley A. Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. Gen- erating multi-label discrete patient records using generative adversarial networks. In Finale Doshi-Velez, Jim Fackler, David C. Kale, Rajesh Ranganath, Byron C. Wallace, and Jenna Wiens (eds.), Proceedings of the Machine Learning for Health Care Conference, MLHC 2017, Boston, Massachusetts, USA, 18-19 August 2017, volume 68 of Proceedings of Machine Learning Research, pp. 286 286-308 305. PMLR, 2017. URL http://proceediing mlr wwwps/ v68/ choi 1ha. html htmm
13

---

Published in Transactions on Machine Learning Research (01/2026)
Council of European Union. Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence and amending regulations (ec), oj 1, 2024/1689, 2024.
https://eur- -ex.x 1753118234752
Guneet S. Dhillon, George Deligiannidis, and Tom Rainforth. On the expected size of conformal prediction sets. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 of Proceedings of Machine Leurning Research, pp. 1549 1557. PMLR, 2024. URL Inttpa:/ .oaiiiiinnnmmm.......mm mlr press/ v238/ 1738/dhillon24p html
Shehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. A diachronic per- spective on user trust in AT under uncertainty. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 5567 5580. Association for Computational Linguistics, 2023. doi: 11/86665//22233/// /ttp///22222.2222/2222 WWW.EEENNN---MN...EE URL https://doot org/oo//// 10. 18663/v1/22223 emnlp-main 339
Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael I. Jordan, and Ryan J. Tibshirani. Class- conditional conformal prediction with many classos. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL htttp://papess... nips cc/ www.ppppppppppp... ltllllppppppaaaplllllllllaaaeeee pttss/ppper/222... html
cb931
Fahimeh Fakour, Ali Mosleh, and Ramin Ramezani. A structured review of literature on uncertainty in machine learning & deep learning. ArXiv, abs/24b6. 24066.003,, 2024. URL https://api. semantccscholar.
org/CorpusID 2702118844
Adam Fisch, Tal Schuster, Tommi S. Jaakkola, and Regina Barzilay. Few-shot conformal prediction with auxiliary tasks, In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine
Learning Research, pp. 3329 3339. PMLR, 2021. URL httlhttt///prooceingss mlr. httpp/ceeengsal...pppee//////aad ppresss/113/ffsshh2lll
html
Isaac Gibbs and Emmanuel Candès. Conformal inference for online prediction with arbitrary distribution shifts. J. Mach. Learn. Res., 25:162:1-162:36 2024. URL httpp: http:ppprrs//v//2222222222 h2m1.
Thomas Grote and Philipp Berens. Uncertainty, evidence, and the integration of machine learning into medical practice. The Journal of medicine and philosophy, 2023. URL http:///app..emaaticcch..oo.o.oo
Oxg/CorpusID. : 255593366
Yu Gu, Robert Timn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specifie language model pretraining for biomedical natural language processing. CoRR, abs/2007.157799, 2020. URL htttt://aaxiiii wtttrxii..org/bb//0000//// 15779
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Tch (ods.), Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volune 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 2017. URL http://procceedings mlr. W....press/v00. gress//70//uuo7a. html
Charlie A Hamm, Georg I. Baumgärtner, Felix Biessmann, Nick I. Beetz, Alexander Hartenstein, Lynn J Savie, Konrad Froböse, Franziska Dräger, Simon Schallenberg, Madhuri Rudolph, et al. Interactive ex- plainable deep learning model informs prostate cancer diagnosis at mri. Radiology, 307(4):e222276, 2023.
Healtheare Cost and Utilization Project (HCUP). Clinical Classifications Software Refined (CCSR), 2024. URL
ref
14

---

Shaoxiong Ji and Pekka Marttinen. Patient outcome and zero-shot diagnosis prediction with hypernetwork. guided multitask learning. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubroenik, Croatia, May 2-6, 2023, pp. 589 598. Association for Computational Linguistics, 2023. doi: 10.18653/V1 1122023.EACL-MAAI Published in Transactions on Machine Learning Research (01/2026)
Roy Hirsch and Jacob Goldberger. A conformalized learning of prediction set with applications to medical imaging classification. 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1-5, 2024. URL
org//orppsID: 271843498
URL https://dol org/ 100..01116//////...ooo wawp:///2222....mmaaaaaaa.a.aaaa mane-mmmin... 43
Alistair F. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark, Mimic-iv, freely accessible electronic health record datasct. Scientific Data, 10(1):1, Jan 2023. ISSN 2052-4463. doi: 10. ///////4219--22--189095
URL htttps://do: http:///ooiooo////// 1010///411770022001199999
Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. Mimic-iii, freely accessible critical care database. Scientific Data, 3(1):160035, May 2016. ISSN 2052-4463. doi: 10.1038/sdata.200 35. URL htttps://do..rr:::ee
10w800008//aaa 2016.33 35
Kevin Kasa and Graham W. Taylor Empirically validating conformal prediction on modern vision ar- chitectures under distribution shift and long-tailed data. ArXiv, 0os/2307.01088, 2023. URL https:
seapaiicmmnticscholaa waapeacchooaaooorroruuuuuse
org/Corpus 259317042
Benjamin Kompa, Jasper Snoek, and Andrew L. Beam. Second opinion needed: communicating uncertainty in medical machine learning. upj Digital Medicine, 4(1):4, Jan 2021. ISSN 2398-6352. doi: 10.1038/
s41746-024 077746--000-066--.. URL https:///ot: http:////ooooorg///// 00000//008174000000000000000 00367-3
Zeljko Kraljevic, Thomas Searle, Anthony Shek, Lukasz Roguski, Kawsar Noor, Daniel Bean, Aurelie Mascio, Leilei Zhu, Amos A Folarin, Angus Roberts, Rebecca Bendayan, Mark P Richardson, Robert Stewart, Anoop D Shah, Wai Keong Wong, Zina Ibrahim, James T Teo, and Richard J B Dobson. Multi-domain clinical natural language processing with MedCAT: The medical concept annotation toolkit. Artif. Intell. Med., 117:102083, July 2021. ISSN 0933-3657. doi: (..00//j..rtmed2021....
Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(1):71-96 Jamuary 2014. ISSN 13697412. doi: 10.1111/rssb.12 12021. URL 12021
wiley. http://wwww.ll...oo///o/ooooo com//665/t0.111//rss.
Jing Lei, James Robins, and Larry Wasserman. Distribution-Free Prediction Sets. Journal of the Amer-
ican Statistical Association, 108(501):278 287, March 2013. ISSN 0162-1459, 1537-274X. doi: 10.1080/
01221459.202.751888 URL http: http://www.taafoollliicc
http::/wwa...t......cccccooaaa// 1000://100000000000101000000 2012. 751873,
Charles Lu, Andréanne Lemay, Ken Chang, Katharina Höbel, and Jayashree Kalpathy-Cramer Fair conformal predictors for applications in medical imaging. In Thirty-Siath AAAI Conference on Arti- ficial Intelligence, AAAT 2022, Shirty-Fourthh Conference on Innovative Applications of Artificial In- telligence, IAAI 2022, The Twelveth Symposium on Educational Advonces in Artificial Intelligence, FAAT 2022 Virtual Event, February 22 March 1, 2022, pp. 12008-12016. AAAT Press, 2022. doi: 10. N..1609/AAAL,, V36.11...1.21445 URL httpps///di.. org/ 10. 1609//aaii wwt.aaaiiivvii....
Scott M. Lundberg and Su-In Lec. A unified approach to interpreting model predictions. In Is- abelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advonces in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4765-4774, 2017. URL
html
15

---

Published in Transactions on Machine Learning Research (01/2026)
Riccardo Miotto, Li Li, Brian A. Kidd, and Joel T. Dudley. Deep patient: An unsupervised representation to predict the future of patients from the electronic health records. Scientific Reports, 6, 2016.
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T. Dudley, Deep learning for healthcare: review, opportanities and challenges. Briefings Bioinform., 19(6):1236-122 12466 2018. doi: 10. /BBB/ BBX044. URL https:///oi org/ 10. wtt.19999bbbbxxx44
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL httpps://openreeieer wtt.://wwww...tooooooooo/o//////
Thomas Mortier, Alireza Javanmardi, Yusuf Sale, Eyke Hüllermeier, and Willem Waegeman. Conformal prediction in hierarchical classification. CoRR, abs/2501.19033 2025. doi: 10.48550 ARXIV 2500.11938.
URL https: org 10. 48550/arXiv 2501. 19038
Aakanksha Naik, Sravanthi Parasa, Sergey Feldman, Lucy Lu Wang, and Tom Hope. Literatuur--angmented clinical outcome prediction. In Marine Carpuat, MlarieCCatherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. URL
Association for Computational Linguistics, 2022. doi: 10. 10.18653/ 181.3/1111666////2/2222 wwwd:fissnninnnnnnnnnnn
V1/2022. 11/202.FINDINGG--AAACL...
Henrik Olsson, Kimmo Kartasalo, Nita Mulliqi, Marco Capuccini, Pekka Ruusuvuori, Hemamali Sama- ratunga, Brett Delahunt, Cecilia Lindskog, Emiel A. M. Janssen, Anders Blilie, Lars Egevad, Ola Spjuth, Martin Eklund, and ISUP Prostate Imagebase Expert Panel. Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prodiction. Nature Communications, 13(1): 7761, Dec 2022. ISSN 2041-1723. doi: 10.1038//411167-022349445 URL https:///doi.rr//000 241467-022-349455
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should I trust you?": Explaining the pre- dictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA,, USA, August 13-17, 2016, pp. URL htttt::///i..oorreteee org/10.
ACM, 2016. doi: 021444/999672..0
Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL https://prooceedings /ttp:/2444ddd7775555150100
peurips.cc/ppppr//
Wojeiech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Rober Müller, Explain- able AI: interpreting, erplaining and visualizing deep learning, volume 11700. Springer Nature, 2019.
Philipp Schmidt and Felix Biessmann. Quantifying interpretability and trust in machine learning systems. In AAAI-19 Workshop on Network Interpretability for Deep Learning, 2019.
Silvia Seoni, V. Jahmunah, Massimo Salvi, Prabal Datta Barua, Filippo Molinari, and Usha R. Acharya. Ap- plication of uncertainty quantification to artificial intelligence in healtheare: A review of last decade (2013- 2023). Computers in biology and medicine, 165:107441, 2023. URL httpp://app semanticscholar. org/
Corpus IrrpusID:200 :D:114882214 261483812
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché- Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Pro- cessing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouter, BC, Canada, pp. 13969- 13980, 2019. URL https://proceedings. hattppp..ccappe/e////aasssssssaa http:///eaaa.aaeaaaaaaaaaaaaaaae
16

---

Published in Transactions on Machine Learning Research (01/2026)
Eleni Straitouri and Manuel Gomez Rodriguez. Designing decision support systems using counterfactual pre- diction sets. In Forly-first International Conference on Machine. Learning, TCML, 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openrevier: wtt..//oofooooiii/rg/uuuubaebe
Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with conformal prediction. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, 1CML 2023, 23-29 July 2023, Honolulu, Hawaii USA, volume 202 of Proceedings of Machine Learning Research, pp. 32633 32653. PMLR, 2023. URL https::/prrcceedingg mlr 2ress//v202/straitoo/
html
David Stutz, Krishnamurthy Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet.. Learning optimal conformal classifiers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nn net, 2022. httpp:///ppparrrreee.eee/errrrra
Hristos Tyralis and Georgia Papacharalampous. A review of predictive uncertainly estimation with ma- chine learning. Artif. Intell. Rev., 57:94, 2022. URL https://epp..emmaticcccoooo wtwp:llaaa.rrrooopsooii...
252367346
Betty van Aken, Jens-Michalis Papaioannou, Manuel Mayrdorfer, Klemens Budde, Felix A. Gers, and Alexander Löser. Clinical outcome prediction from admission notes using self-supervised knowledge in- tegration. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Confer- ence of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 991-893. Association for Computational Linguisties, 2021. doi: 10.18653/v1 /ttp://///221///// http:/2221.cclmmmee www.-main...... URL https://doi. wwwp//////....... 186....16611111222220 wwtp:/eeeaacmmmmmm sacl-main.75.
Betty van Aken, Jens-Michalis Papaioannou, Marcel Ganesh Naik, Georgios Eleftheriadis, Wolfgang Nejdl, Felix A. Gers, and Alexander Löser. This patient looks like that patient: Prototypical networks for interpretable diagnosis prediction from clinical text. In Yulan He, Heng Ji, Yang Liu, Sujian Li, Chia- Hui Chang, Soujanya Poria, Chenghua Lin, Wray L. Buntine, Maria Liakata, Hanqi Yan, Zonghan Yan, Sebastian Ruder, Xiaojun Wan, Miguel Arana-Calania, Zhongyu Wei, Hen-Hsen Huang, Jheng-Long Wu, Min- Yuh Day, Pengfei Liu, and Ruifeng Xu (eds.), Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, AACL/IJNNLP 2022 Volume 1: Long Papers, Online Only, November 20 23, 2022, pp. 172 184. Association for Computational Linguistics, 2022. URL htslattts///ccaatthlleg org/2022. maa.cmmmman.11. 14
Janette Vazquez and Julio C. Facelli. Conformal prediction in clinical medical sciences. Journal of Healthcare Informalics Research, 6(3):241 2522, Sep 2022. ISSN 2509-498X. doi: 10.1007 /411/6660-00100001333 10.1000////44416600000000000
URL
Vladimir Vovk, A. Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer, New York, 2005. ISBN 978-0-387-00152-1 978-0-387-25061.5
Asif Ali Wagan, Shahnawaz Talpur, and Sanam Narejo. Clustering uncertain overlapping symptoms of multiple discases in clinical diagnosis. PeerJ Comput. Sei., 10:c2315, 2024. doi: 10.7717/PERRJ-S.231
URL https:://aa org/10 7717/peerj cs. 23311
Xinru Wang and Ming Yin. Are explanations helpful? A comparative study of the elfects of explanations in ai- assisted decision decision-making, In Tracy Hammond, Katrien Verbert, Dennis Parra, Bart P. Knijnenburg, John O'Donovan, and Paul Teale (ods.), IUI '21: 26th International Conference on Intelligent User Interfaces, College Station, TX, USA, April 13-17, 2021, pp. 318 328. ACM, 2021. doi: URL https://doi...... org/10.11100/ 10. 11011114/3399/0 3450650
Gianluca Zeni, Matteo Fontana, and Simone Vantini. Conformal Prediction: a Unified Review of Theory and New Challenges. Technical Report arXiv:2005.07972 arXiv, May 2020. URL http:///rxxv..rrg abs/2005.0797) 07972 a:Xiiv2000.07972 (os, econ, stat] type: article.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Botong Zhang, Shuo Li, and Osbert Bastani. Conformal structured prediction. In The Thirteenth Inter- national Conference on Learning Representations, TCLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net, 2025. URL attp:://ooaaunoss.weeeeerrrrrrrr
18

---

Published in Transactions on Machine Learning Research (01/2026)
A Influence of Pre-initialization of Prototypical Layer on Calibration
van Aken et al. 2022) introduced ProtoPatient and explored two initialization strategics: one in which prototypes and attention vectors are pre-initialized, and another using random initialization. Table $ reports calibration results for all methods using model predictions from both initialization strategies, indicated in the Init column. The results demonstrate that pro-initializing the prototypical layer yiclds higher macro coverage, smaller average prediction set sizes, and lower standard deviations across 50 runs with different random seeds, indicating more stable and reliable calibration. As shown Table 3 pre-initialization also improves macro-AUC, consistent with the findings of van Aken al. (2022), although results in slightly lower accuracy. We argue that models with higher macro-AUC achieve better calibration results than those with higher accuracy.
Table 3: Performance Metries of ProtoPatient all datasets for the main clinical outcome prediction task. While AUC score consistently increases when pre-initializing is used, accuracy slightly degrades.
Init
Dataset
# Classes
Accuracy
Macro-AUC
MIMIC-111 (CCSR)
356
47.81
93.24
False
MIMIC-IV (CCSR)
123
45.36
95.22
MIMIC-IV (ICD)
1054
48.39
93.82
MIMIC-I1I (CCSR)
356
45.39
94.83
True
MIMIC-IV (CCSR)
423
42.82
95.75
MIMIC-IV (ICD)
1054
43.43
95.41
Table 4: Calibration results for ProtoPatient models trained with different pre-initialization methods (Init), using a calibration set size of 1,000 and a confidence level (1 ax) 0.9 and m 10. Each row reports macro coverage (higher is better) and prediction set size (lower better) for three clinical classification tasks: MIMIC-III (CCSR), MTMIC-IV (CCSR), and MTMIC-IV (TCD). Conformal methods with formal coverage guarantees appear above the single horizontal line; non-conformal baselines are shown below. Results are shown as mean + standard deviation over 50 repetitions. Pre-initialization consistently improves calibration performance, yielding higher coverage, smaller prediction sets, and reduced variance.
Macro Coverage (t)
Prediction Set Size (4)
MIMIC-III
MIMIC-IV
MIMIC-IV (ICD)
MIMIC-1II
MIMIC-IV (CCSR)
MIMIC-IV
Init
Method
(CCSR)
(CCSR)
(CCSR)
(ICD)
DGCP Severity Score (ours)
2.92
67.0 1.6
64.024
17.0
18.4 2.0
DGCP Body System (ours)
3.2
71.8 1..8220 2.4
2.5
33.2
31.0 - 5.7
67.8 1.8810 14.4
Clustered CP
2.3
66.5
61.7+2.8
12.0
16.9 1.9
27.8 4.4
Yes
Class-conditional CP
1.5
38.9 1.5
2.97+
1..3
20.3 1.9
21.4
Marginal CP
55.1
66.6
6..7
12.0 1.0
17.0 1.6
27.9 3.9
Calibrated Adaptive top-k
64.1 + 1.6
70.2 + 1.5
64.5 1.6
17.3 + 1.0
21.9 + 1.8
34.6 + 3.0
Adaptive top-k
62.1 = 0.9
67.4 + 0.2
60.9 1 0.3
15.3
18.3 0.0
27.5
DGCP Severity Score (ours)
53.2
66.6 + 2.9
62.8 2.5
20.0 + 2.6
19.3 + 2.7
38.6 - 8.8
DGCP Body System (ours)
60.3
71.7
25.1424
38.8 8.3
34.4
85.1 1.112219
Clustered CP
3.6
65.3 4.0
2.7
1..1
16.4
29.9 -
No
consicondiiional CP
1.7
1.4
2.88
18.7
2.1
23.9 2.4
Marginal CP
3.2
65.3 3.3
1.9
14.1
16.3 1.
30.2 3.1
Adaptive top-k
1.4
66.7 1.6
61.5
18.6 0.9
17.5 1.3
34.9 - 2.9
Calibrated Adaptive top-k
51.00:: 0.7
53.9 4: 0.2
47.8
15.0 1.10
9.9 0.0
16.8
19

---

Published in Transactions on Machine Learning Research (01/2026)
B Effect of Conformal Prediction Hyperparameters on Calibration
Influence of Calibration Threshold m. The parameter specifies the minimum number of calibration samples required for class-level calibration. If class contains fewer than m samples in the calibration set, we group it with semantically related classes present in the set for threshold estimation. As shown in Table 5 increasing m from 10 to 20 has minimal impact on conditioni coverage and prediction set size across all datasets. A threshold of 20 leads more frequent use of dynamic grouping, since in a calibration set of 1,000 samples it becomes suggest that dynamic grouping introduces almost no effect (within the standard deviation) on coverage and prediction set efficiency, demonstrating the robustness of our method to modest changes in m.
likely that individual classes will not meet the sample requirement. These results
Influence of Confidence Level (1 a). In Table 6 we present calibration results at a confidence level (1 (1-a) a) = 0.8. To investigate the effect of varying the confidence level, we compare these results with those presented Table 5 which uses the same calibration set size of 1,000 samples but a higher confidence level (1 (1-x) a+) 0.9. As expected, lowering the confidence level leads consistently smaller prediction set sizes across all datasets and methods. However, this reduction comes the cost of decreased macro coverage, reflecting the fundamental trade-off between precision and reliability in conformal prediction. Despite the drop in coverage, the relative ranking of methods remains consistent, suggesting that method robustness is preserved across confidence levels. These findings emphasize the importance of choosing an appropriate con- fidence level based on task requirements, whether minimizing prediction ambiguily or maximizing empirical coverage with guarantees.
Influence of Calibration set Size n. We present calibration results for two calibration set sizes: 1,000 samples in Table 3 and 2,000 samples in Table We analyze the impact of calibration set size and compare the results at fixed confidence level (1 a) 0.9. The most notable gain in cass-conditional coverage is observed for Class-conditional CP which shows substantial increase in coverage across all datasets. However, this improvement comes at the cost of significantly larger prediction sets. In contrast, DGCP variants (based on severity and body system) maintain comparable coverage while showing a reduction in prediction set size and a decrease in standard deviation across repetitions. This suggests that our method benefits from the increase in calibration data by becoming more efficient and stable without sacrificing reliability. For other basclines, including Marginal CP, Clustered CP, and non-conformal methods, both coverage and set size remain largely unchanged, indicating limited sensitivity to calibration size. Therefore, in this study, focus our in-depth analysis on the scenario with only 1,000 calibration samples to better understand performance in terms of class-conditional coverage under more limited data conditions.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Calibration results for different values of m. All models use the pre-initialized model, confidence level (1 -1-00 a) 0.9 and calibration size n I, 000. Results show that decreasing m from 20 to 10 maintains stable performance in terms of macro coverage and prodiction set size. Conformal methods with formal coverage guarantees are listed above the single horizontal line, while non-conformal baselines are shown below. + indicates standard deviation over 50 repetitions.
Macro Coverage (t)
Prediction Set Size (4)
MIMIC-II1 (CCSR)
MIMIC-IV (CCSR)
MIMIC-IV (ICD)
MIMIC-III (CCSR)
MIMIC-IV
MIMIC-IV (ICD)
m
Method
(CCSR)
DGCP Severity Score (ours)
60.2
67.0 + 1.6
64.0 2.4
17.0 2.8
33.9 + 6.7
10
DGCP|Bodyy System (ours)
65.5 +: 3.2
71.8 + 2.4
67.0 + 2.5
33.2 8.6
31.0 + 5.7
67.8 + 14.4
CGCC//Sevrrity Score (ours)
60.3 1: 2.9
67.2 + 1.6
64.0 2.4
16.5
18.1 + 1.9
34.8 1 7.0
20
DGCC|/Bodh System (ours)
65.5 1: 3.2
71.9 2.4
67.0 1 2.5
33.2 8.7
31.3 1 5.9
68.6 1: 14.5
Clustered CP
55.2 + 2.3
66.5 + 2.1
61.7 + 2.8
1.
16.9 1.9
27.8 + 4.4
dlassconnitiinnll CP
+
38.9 1.5
2.9
16.3
21.4 1.9
Marginal CP
55.1 + 2.1
1.7
61.7 2.5
1.00
1..0
27.9 3.9
Adaptive top-k
64.1 + 1.6
70.2
61.5 1.6
17.3+
2..9
64.6 3.0
Calibrated Adaptive top-k
62.1 +: 0.9
0.2
60.9
1.13
18.3 0.0
0.0
Table 6: Calibration results for calibration set size n 0000 and a confidence level (1 (1-a) 0.8. Each row presents the macro coverage (higher is better) and prediction set size (lower is better) across three clinical classification tasks: MIMIC-III (CCSR), MIMIC-IV (CCSR), and MIMIC-IV (ICD). Conformal methods with formal coverage guarantees are listed above the single horizontal line, while non-conformal baselines are shown below. The threshold m indicates the minimum number of samples required for class-level calibration. Reported values are mean + standard deviation over 50 repetitions.
Macro Coverage (t)
Prediction Set Size (1)
MIMIC-III (CCSR)
MIMIC-IV (CCSR)
MIMIC-IV
MIMIC-III
MIMIC-IV
MIMIC-IV
m
Method
(ICD)
(CCSR)
(CCSR)
(ICD)
DGCP (Sverity Score (ours) DGCP/Body System (ours)
42.5 2.0
50.6 1.8
46.1 1 2.2
7.3 0.8
11.2 1.8
10
49.0 + 2.9
56.5 + 2.3
49.9 + 2.4
14.5 + 4.4
12.4 2.5
26.1 + 8.7
DGCP Severity Score (ours) DGCP Body System (ours)
42.5 +
50.9 1.8
46.2 + 2.2
+ 0.7
7.4 0.8
11.7 4 2.0
20
48.9 + 2.9
56.7 + 2.4
50.0 + 2.4
14.1 4.3
12.7 2.5
26.5 4 8.7
Clustered CP
37.7 +: 1.5
50.1 + 2.8
43.9 2.6
5.1 + 0.5
7.3 4: 1.0
10.3 4 1.6
Clus--conditional CP
34.0 + 1.5
38.1 +: 1.5
23.5
13.6
17.0
19.7 2.2
Marginal CP
37.6 + 1.4
49.9 1.0 2.0
1.7
5.0
7.2
10.2 1 1.0
Adaptive top-k
51.1 + 1.3
55.8
4..7
8.0 0.4
9.7
13.9 1.0
Calibrated Adaptive top-k
48.9+00
0.2
0.3
7.3
8.4
11.6
21

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Calibration results for calibration set size n = 2, 000 and a confidence level (1 (1-a) = 0.9. Each row presents the macro coverage (higher is better) and prediction set size (lower is better) across three clinical classification tasks: MIMIC-III (CCSR), MIMIC-IV (CCSR), and MIMIC-IV (ICD). Conformal hods with formal coverage guarantees are listed above the single horizontal line, while non-conformal baselines are shown below. The threshold m indicates the minimum number of samples roquired for class-level calibration. Reported values are mean + standard deviation over 50 repetitions.
Macro Coverage (1)
Prediction Set Size (L)
MIMICIIII (CCSR)
MIMIC-IV (CCSR)
MIMIC-IV (ICD)
MIMIC-III (CCSR)
MIMIC-IV (CCSR)
MIMIC-IV (ICD)
m
Method
DGCP Severity Score (ours)
62.3 + 2.2
67.3 + 1.1 71.8 1.98
1.8
15.5 1.3
18.9 1.4 27.0+3..
30.4
10
DGdP/Body System (ours)
68.1 2.6
66.3 2.0
28.1 +.3
51.7 9.3
DGCP\Severity Score (ours)
62.3 + 2.2
67.2 1.1
63.9 1.8
14.6 1.2
16.5 0.9
30.5 + 3.4
20
DGCP|Body System (ours)
68.0 = 2.6
71.8 + 1.9
66.4 + 2.0
27.1 + 4.2
25.1 + 2.8
52.6 + 9.5
Clustered CP
58.3 + 2.5
67.0 1.6 1.6
61.7 2.4
0.9
15.2+1.7
25.6 + 4.0
Class-conditional CP
17.8+200
51.2
3.2
1..6
2..
34.3 2.8
Marginal CP
57.9 2.1
67.1 1.2
62.1
11.9
17.2+1.0
28.0 + 3.2
Adaptive top-k
64.5 + 1.6
6.11
1.3
1.66
20
32.1 1 222
Calibrated Adaptive top-k
64.2 + 1.5
67.6
0..2
1.2
18.3
27.5 + 0.1
C
Analysis of Alass-Conditional Methods at Inference Time
MIMIC-III CCSR Codes (356)
MIMIC-IV CCSR Codes (423)
MIMIC-I ICD Codes (1054)
Confidence a)
Class- ClassConditionall Coverage
0.8-
0.6
0.4
0.2
0.0
10"
10:
102
10a
100
10a
10a
10a
100
10=
102
10a
Label Frequency in Training
DGCP/ System
Class-Conditional CP
DGCP|Severity Score
Clustered CP
Figure 5: Class-conditional coverage as function of training-set. label frequency. The dashed horizontal
line marks the target confidence level (1 a1) a) = 0.9. Results shown for (1 (1-00 -1-a) a) = 0.9, calibration size n = 1,000, and threshold m = 10. Frequent classes typically have sufficient calibration support to reach the target coverage, while rare classes often lack enough calibration points. Grouping methods (DGCP)Bodd System, DGCP\Sevvrity Score, Clustered CP) shift the coverage curves leftward, indicating more efficient use of limited calibration data and improved coverage for low-frequency classes.
To complement the analysis in Figure and exlend the findings reported in Table we additionally examine how coverage behaves with respect to the true distribution of classes encountered at inference time. The figure illustrates the expected class-conditional coverage under the application distribution and shows patterns consistent with Figure however, with generally lower absolute coverage values due to the increased difficulty of achieving llass- validity under the true long-tailed training distribution.
22

---

Published in Transactions on Machine Learning Research (01/2026)
Moreover, even in datasets where the label space is smaller than the calibration set (e.g., MIMIC-IIIIV CCSR), individual calibration splits may still contain fewer than ten examples for classes that are frequent in the training data. Thus, calibration-set frequency should not be interpreted as the true class frequency; it reffects only the calibration support available in a particular split. Coverage plotted against calibration Irequency therefore captures the behaviour of a method under limited calibration data, whereas coverage plotted against training-set frequency reflects the expected reliability across head and tail classes at inference time, where grouping-based methods provide the greatest benefit due to the underlying long- long-tailed label distribution.
23


Reviews:
[{"review_id": "TO0Gp1zLqU", "reviewer": "Reviewer_ZwB8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\nThis paper develops conformal prediction methods that achieve (approximate) class-conditional coverage. The main methodological idea is to use auxiliary domain information to group together semantically similar classes for which limited calibration data is available. Experiments on clinical diagnosis datasets show that this method improves class-conditional coverage on rare classes. \n\n**Strengths:**\nThe main method is a new proposal that performs well empirically.  \n\n**Weaknesses:** \nThe overall novelty of the methodological contribution is small. Closely related techniques for obtaining (approximate) class-conditional coverage have been proposed in Ding et al. (2023). With that said, the proposed methods do offer some small empirical improvements over existing approaches. Additionally, I found some the specifics of the experimental set-up and results to be difficult to interpret (see below).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claim of the article is that DGCP provides better class-conditional coverage than existing approaches. This is supported empirically in Table 1 and Figure 4.", "audience_yn": "Yes", "audience_explanation": "I believe the results of this paper would be of interest to readers working in conformal inference or other topics in uncertainty quantification.", "requested_changes": "**Major Changes**\n- I found the presentation of the empirical results and method to be somewhat imprecise. I believe the paper would benefit from a more explicit definition of the main method as well as the various quantities shown in Table 1 and Figure 4 (e.g. macro coverage, definition of the x-axis in Figure 4).\n- I find the results of Figure 4 to be somewhat surprising. Most notably, if I am understanding the plot correctly, the authors find that for classes in which there are ~100 training samples available, class-conditional conformal prediction obtains only 60-80% coverage (versus a target coverage of 90%). This seems quite strange to me as at that sample size I would expect this method to deliver near-exact coverage. \n- Related to the above, the text below Figure 4 suggests that much of the undercoverage is due to rare classes not being well represented in the training data. It would be helpful to have more specific quantitative information regarding the class frequencies and the sources of miscoverage. Additionally, it would be beneficial to have more information on the groupings output by DGCP in this experiment. \n\n**Minor Changes**\n- The definition of the conformal prediction set above equation (2) is stated with strict inequality. Typically, one would define this set to allow for equality (so it is $\\hat{C}(X_{\\text{test}}) =\\\\{y : \\hat{f}(X_{\\text{test}})_y \\leq q\\\\}$). This can be important for classification tasks where the equality cases is needed to ensure coverage. \n- In the equation display at the top of page 6 it would be helpful to explicitly notate that the group, g depends on y. \n- What clustering method was used in the comparison to Ding et al. (2023)? Does the choice of clustering method effect the comparisons?\n- Why were the top-k based methods excluded from figure 4?", "broader_impact_concerns": "I have no concerns on the broader ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gsSYu7SQgT", "reviewer": "Reviewer_YHgb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper proposes Dynamically Grouped Conformal Prediction (DGCP), a model-agnostic framework to improve conformal prediction for datasets that have large and imbalanced label spaces.\n- The key idea is to retain class-conditional calibration when sufficient calibration samples exist, while dynamically grouping underrepresented classes using domain knowledge (e.g., ICD hierarchies, body systems, severity levels) when data are scarce. This approach improves class-conditional coverage for rare classes, an especially important property in clinical diagnosis prediction tasks.\n- The method is evaluated on three clinical datasets and consistently outperforms other baselines in terms of coverage for infrequent classes.", "claims_and_evidence_yn": "No", "claims_explanation": "While the proposed method (DGCP) is evaluated on three clinical datasets with repeated experiments, the evidence presented is not sufficient to convincingly support the broad claims. The improvements in class-conditional coverage seem to have largely resulted from increased prediction set sizes rather than fundamentally better calibration, and this trade-off is not analyzed in depth. Furthermore, the experimental scale is limited to a few datasets, despite claims that the method is model- and data-agnostic and could generalize to other domains. Key methodological questions, such as how to handle domains without hierarchies or how to automatically identify semantically related classes, remain unanswered. Overall, the claims are only partially substantiated, and stronger experimental validation and methodological clarity are needed.", "audience_yn": "Yes", "audience_explanation": "TMLR’s audience includes researchers in uncertainty quantification and applied ML in safety-critical domains. Improving conformal prediction for infrequent classes is directly relevant to these communities, particularly in healthcare applications where rare but high-risk conditions matter. Beyond clinical settings, the idea of dynamically grouping classes based on hierarchies could be valuable in other domains with structured label spaces (e.g., biology, legal or product taxonomies). Thus, the findings would be of interest to both methodological researchers and practitioners", "requested_changes": "**Method**\n- The technical contribution is relatively limited, as the main novelty lies in applying simple hierarchical grouping to conformal prediction.\n- It is unclear how the method would operate in domains where no semantically related classes or expert-defined hierarchies exist.\n  - While the authors note that some domains (biology, law, finance) offer hierarchical structures, the method’s reliance on such structures limits its generality.\n- The paper does not address whether semantically related classes could be identified automatically, which would improve applicability beyond domains with curated taxonomies.\n\n**Experiments**\n- The experimental scale is relatively small: only three clinical datasets are used, all from MIMIC. Evaluations on broader domains or non-hierarchical datasets would make the claims of generalizability more convincing.\n- The paper states that model- and data-agnostic guarantees suggest generalization to other datasets and architectures, but such evidence should have been included rather than deferred to future work.\n- Repeating experiments 50 times is a strong point, but the improvements in class-conditional coverage may stem mainly from larger prediction set sizes rather than more effective calibration. A deeper analysis of this trade-off is needed.\n\n**Writing**\n- Some notations are not properly defined; for example, $S_{calib} = \\{ \\hat{f}(X_{calib})_k: k = y_{calib} \\}$ is firstly presented as undefined.\n- Citations are inconsistently formatted. For example, “The authors of (van Aken et al., 2021)” should use `\\citet{}` instead of `\\citep{}`.\n- Similarly, phrasing like “Naik et al. (2022), augment” should be revised as \"Naik et al. (2022) augment\" to maintain grammatical consistency.", "broader_impact_concerns": "The work has positive societal implications: improving uncertainty estimates for rare conditions could strengthen clinical decision support and reduce missed critical diagnoses. Risks include over-reliance on potentially biased or incomplete taxonomies and the possibility that larger prediction sets may overwhelm clinicians. These concerns could be mitigated through careful grouping choices, human-in-the-loop evaluation, and attention to set-size efficiency.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PMApRMPK2G", "reviewer": "Reviewer_9sGw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript proposes Dynamically Grouped Conformal Prediction (DGCP), a post-hoc calibration scheme for multi-class classification tasks with long-tailed, hierarchically organized label spaces. If a class has $<m$ calibration samples, DGCP merges it with its sibling classes from an existing taxonomy and calibrates on the group; otherwise, it reduces to class-conditional CP. Empirically, DGCP shows higher macro class-conditional coverage and reduced efficiency relative to marginal CP, class-conditional CP, clustered CP (Ding et al. 2023), and two top-k baselines. \n\n**Strengths**\n- The authors address an important practical gap and aim at improving coverage for rare classes in high-risk medical applications.\n- The curated MIMIC III/IV data represents well the long-tailed classification setting considered here. Two different hierarchies (severity score, body system) are used in the experiments.\n\n**Weaknesses**\n- Macro coverage often falls short of the nominal 0.9 (even after grouping) on the test set, while the method's theoretical guarantees are emphasized up front. Additional metrics and analysis would help understand the results better (please see \"suggested revisions\").\n- The method assumes that a semantic hierarchy is provided.\n- Hyperparameter _m_ is fixed a priori; no data-driven or adaptive strategy is explored.\n- Prediction sets for DGCP are 2–3× larger than baselines for the body system hierarchy.\n- There are a few minor errors/inconsistencies and presentation issues throughout where additional details would strengthen the claim (please see \"suggested revisions\").", "claims_and_evidence_yn": "No", "claims_explanation": "The central claim is that DGCP \"consistently improves class-conditional coverage for infrequent [classes], outperforming strong baselines in all settings in terms of class-conditional coverage.\" While it makes sense to assume access to a hierarchy/grouping of classes informed by domain knowledge, the extra information does not seem to help much -- DGCP improves macro coverage only slightly, at the expense of hurting prediction efficiency quite significantly. It would be helpful to see a more in-depth analysis backed by additional metrics, stratified by head/tail groups, to better understand the results. I believe this may uncover opportunities to improve the results further.", "audience_yn": "Yes", "audience_explanation": "This approach would be of interest to those working on uncertainty quantification for risk-sensitive many-class classification settings. Rare classes / long tails are underexplored. Hierarchical label spaces are common beyond medicine (ImageNet, product taxonomies, biological taxonomies, legal codes).", "requested_changes": "While the empirical improvements are not very dramatic, the proposed approach in the specific setting considered is interesting. I would recommend acceptance provided that the following points are addressed. I've marked as \"optional\" suggestions that would simply strengthen the work in my view. \n\n### Analysis\n[Adaptive top-k baselines]\n* There are several possible reasons adaptive top-k baselines show strong performance: (1) choice of the inverse quantile conformity score function (as opposed to the correct-class model probability that the authors used), (2) lack of the conformal calibration layer (does this step hurt somehow, as the calibration set is sparse for some classes?), and (3) $k$ chosen adaptively rather than determined a priori for all test samples, like $m$. It would be clarifying to disentangle these effects. \n* Even though adaptive top-k was not wrapped in a conformal layer, I believe [a] should be cited. Given its strong performance, the inverse quantile conformity score of [a] may yield better performance than the score used by authors. Switching to this score for all the conformal methods considered in the experiments may enable \"fairer comparisons,\" as all the baseline methods would be using the same \"effective score\" and we would be isolating the effect of (1). The current \"marginal CP\" would be replaced by [a].\n* If adaptive top-k baselines are to be included without conformal calibration as is, it's fairer to include the calibration set into the training set of the classifier, so that they get access to the same amount of data.\n* (Optional) Is there an explanation for why the \"body system\" grouping method leads to such a large increase in prediction set size?\n\n```\n[a] Romano, Sesia, and Candes. (2020). Classification with Valid and Adaptive Coverage. NeurIPS.\n```\n\n[Additional experimental detail]\n- What is varied across the 50 repetitions? Initialization for the underlying predictor or just the train/calib/test splits?\n- Eq 6 in Section 6.4 defines marginal coverage, but the text says that \"We measure these metrics as macro-averages to highlight changes impacting all classes and not only the majority class.\" Please make the equations consistent with the metrics actually used. \n- To understand why the coverage values reported in Table 1 are so low compared to the nominal level of 0.9, it would be helpful to supplement the macro average coverage metric with the worst-class coverage and/or report the head-only and tail-only macro average coverage metrics separately (using the head/tail definition in Fig 2). The idea is to get a better sense of the coverage distribution across head/tail groups; please also consider including a figure. The same goes for the efficiency metric, to understand why the prediction sizes are so large.\n- Approximately what fraction of the test samples were assigned to the NULL cluster? \n- How many points are in the test set? Is it possible the test set is too small for minority class evaluation?\n\n### Methods\n* The proposed approach assumes access to a hierarchy of classes. Could the authors please elaborate on what is meant by \"possible to use any other variable or grouping function?\" The text seems to suggest that all that's required is a way to group a given (rare) class to other semantically related classes -- in the absence of a full hierarchy, how can we construct a semantic grouping function for medical diagnosis applications?\n* The following text is potentially misleading: \"While [Clustered CP] relies on unsupervised clustering to group samples, we instead leverage semantic similarity of labels to dynamically aggregate samples only for underrepresented classes while using class-specific data for frequent classes.\" Clustered CP operates in settings with less information (i.e., where a hierarchy or semantic grouping function is not available).\n* (Optional) As future extensions, could the authors discuss ways the hyperparameter $m$ can be determined in a data-driven way or adaptively per class depending on model performance?\n\n### Minor errors and inconsistencies\n* BNNs offer subjective uncertainty at best, and aren't \"calibrated\" in frequentist sense. So to state that \"a broad spectrum of methods to calibrate uncertainty estimates\" exists and only mention BNNs is awkward, especially as BNNs aren't used as baselines. Other non-conformal post-hoc calibration methods for multi-class classification include temperature/vector/matrix scaling (Guo et al. 2017 used alongside adaptive top-k), Dirichlet calibration (Kull et al. 2019)\n* Conformity and non-conformity scores are mixed up throughout -- please correct the relevant definitions and equations\n- Fixing citep and citet throughout", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)